Closing yield (YG) and nitrogen use efficiency (NUE) gaps amongst smallholder farmers are critical to ensuring food security and environmental sustainability. Here we report a comprehensive analysis of a data set derived from farm-surveys and previously published studies in a typical high N input area of China. Using scenario analysis, farmers from both sources were divided into four rice production levels to assess the YG and NUE of smallholder farmers. Farm surveys showed that rice yield and partial factor productivity of applied N (PFP_N) averaged 8273 kg ha⁻¹ and 23.1 kg kg⁻¹ at regional-scale, with a potential increase of 21–43% and 33–52%, respectively. A wide variation in yield and PFP_N across different types of farmer was observed. Optimized N management significantly narrowed YGs by 3.7% and PFP_N gap by 63.5% on average based on the published literature. Multiple factors, including excessive N rates, small rice planting area (0.33 ha), the aging (51 years old) and low education level (6 years of education) of the major labor force, can partly explain causes of yield and NUE gaps. These findings provide farm-based evidence that centralized management, good agronomy plus advances in knowledge and technologies are essential for future agricultural development.