The growth of rural residential land (RRL) areas has led to the encroachment of cultivated land, which has seriously reduced cultivated land use efficiency (CLUE). This paper takes 42 cities in the middle reaches of the Yangtze River (MRYR) as an example, using the kernel density estimation method, the Super-SBM model, and mediating effect test methods to explore the impact of RRL change on CLUE during 2000–2020. Specifically, based on the analysis of the spatiotemporal distribution characteristics of RRL and CLUE, this paper attempts to further explore the influence path of RRL change on CLUE and test whether there is a mediating effect. The results show that (1) the overall RRL area increased by 30,386.34 hm², except for the decrease in RRL area in a few regions of Hunan Province, and the RRL area in other regions increased. (2) The hot-spot and sub-hot-spot regions of CLUE in the MRYR were mainly concentrated in northwestern Hubei Province and eastern Hunan Province, and the hot-spot and sub-hot-spot regions in Hunan Province are the highest among the three provinces. (3) Under the control of socioeconomic variables, the change in RRL has a significant negative impact on CLUE. (4) The area of cultivated land occupied by rural residential land (CLRRL) has a mediating role during 2000–2020, while the per capita cultivated land area (PCLA) and the rural permanent population (RPP) only have a mediating role during 2000–2010. In the future, the government should strictly prohibit the occupation of cultivated land by RRL and to improve the CLUE.