The Yangtze-Huaihe rivers region is an important production base of rice and wheat in China. It is necessary to clarify the differences and formation mechanism between different yield levels in the region, and to explore the limiting factors for regional grain production, which can provide scientific basis and reference for the management practices to reduce the annual yield gap in the Yangtze-Huaihe rivers region. Based on annual rice–wheat production situation, crop yield was divided into three different levels, farmer yields, experimental yields and high record yields. Yield gaps and the climate factors of different yield levels were quantified. Results showed that there were significant differences between farmer yields and experimental yields, high record yields of rice, wheat and annual in the Yangtze-Huaihe rivers region, which were 3315.9, 1537.5, and 4645.6 kg hm–2, 7498.6, 3977.9, and 9840.9 kg hm–2, respectively. Compared with the experimental yields, the farmer yields of rice, wheat and annual had yield increase potential of 46.2%, 29.7% and 37.3%, and 104.5%, 77.0% and 79.0% in comparison with the high record yields, respectively. The number of grains per spike was the main factor resulting in the yield difference in rice, and the grain numbers per spike and the number of spikes contributed to the yield difference in wheat. Compared with the farmer average yield, the grain numbers per spike of rice in experimental yields and high record yields were increased by 30.4% and 116.1%, respectively; the spikes and grain numbers per spike of wheat were increased by 40.9%, 70.0% and 21.8%, 19.6%, respectively. Reducing the yield gaps mainly depended on increasing the grain numbers per spike for rice, and synergistic improvement in the number of spikes and the grain numbers per spike for wheat. Cumulative radiation and low accumulated temperature during the growth period were the main climatic factors affected rice production, while excessive rainfall was the main climatic factor affected wheat production. The results suggested that in order to strengthen rice production and stabilize wheat production was an effective way to increase the annual grain production in the Yangtze-Huaihe rivers region.