Due to the striking gap in agricultural labor productivity (ALP) between China and developed countries, improving agricultural labor productivity is critically important. This study investigates the effect of land rental on agricultural labor productivity from two levels including household agricultural labor productivity and aggregated village-level agricultural labor productivity by employing propensity matching methods and instrumental variables method. Mediation analysis is also applied to explore the influencing mechanism and underlying paths for household agricultural labor productivity improvement. The results analyses based on the nation-wide data sets in 2017 reveal that renting-in land has a significant positive impact on promoting household agricultural labor productivity and renting-out land has significant opposite effects. The mediation analysis indicates that renting-in land affects household ALP indirectly through the land-labor ratio, intermediate inputs, and agricultural assets investment. Moreover, from the village perspective, we further found that the development of the land rental market positively affects the improvement of aggregated agricultural labor productivity of the village. Land rental is an effective channel to stimulate land transferred from lower agricultural labor productivity to higher ones and then promote the resource allocation within the village.