The rapid growth of China’s economy since the reform in 1978 should be largely attributed to urbanization. Nonetheless, in terms of farmland productivity, urbanization may lead to perverse incentives and thus threaten food security. On the one hand, the requisition–compensation balance of farmland (RCBF) policy could reduce farmland productivity because of a “superior occupation and inferior compensation”; on the other hand, urbanization promotes the transfer of the younger labor force and thus reduces the productivity of the agricultural labor force. To investigate the undesirable effects, based on some stylized facts, this study selects 29,415 county-level samples in a Chinese county from 2000–2014 to construct an empirical model. With a new stochastic frontier analysis method that eliminates the classical econometric issues of endogeneity and heterogeneity, the empirical results show that there is a U-shaped relationship between the farmland use efficiency (productivity) and urbanization rate, indicating that only when the urbanization rate is relatively low would urbanization decrease the farmland use efficiency; in contrast, when the urbanization rate is relatively high, technical progress would obviously be accompanied by urbanization, and thus, the undesirable effects are fully offset. Furthermore, the U-shaped relationship is robust after considering the endogeneity of the urbanization rate and total-factor farmland use efficiency. With these findings, recommendations to implement sustainable management and conservation policies regarding farmland resources are made.