To date, scholars have increasingly focused on the reduction in crop yields caused by cultivated land fragmentation, yet its effects on the ecological efficiency of cultivated land use are often overlooked. This oversight leads to land resource waste and environmental pollution. It is essential to explore this problem to achieve moderate-scale farming operations and promote the green transformation of agricultural land. This study theoretically analyzed the mechanisms by which cultivated land fragmentation and management scales influence the ecological efficiency of cultivated land use. Based on 2023 household data from Changde and Shaoyang, China, empirical tests were conducted using the stochastic frontier analysis method, Tobit model, and structural equation model. The research results indicate that: (1) The mean ecological efficiency of cultivated land use among the total sample households was 0.822, and the eco-efficiency in the plains was slightly lower than that in the hilly areas. (2) The scale of cultivated land management played a moderating role in the impact of cultivated land fragmentation on ecological efficiency, with differences observed between topographical types. The scale of management can offset part of the negative impact of cultivated land fragmentation on the ecological efficiency of cultivated land use. (3) Regarding the impact of cultivated land fragmentation on the ecological efficiency of cultivated land use, cultivated land management scale changes play a complete mediating role. These findings help provide policy implications to improve the ecological efficiency of cultivated land use. Policy support should be strengthened by promoting moderate-scale cultivated land operations, enhancing the comprehensive remediation of cultivated land fragmentation, and developing skilled farmers for long-term environmental sustainability.