Rapid urbanization of China has brought lifestyle changes resulting in a continuous decline in children’s physical fitness (PF) and out-of-school physical activity (PA). To date, studies have been focused on correlates of PF and out-of-school PA, and patterns and trends based on geographic diversity and urban-rural contrasts. Western China, with a large rural population, has substantial urban-rural differences, but little work has been done to compare its children’s physical fitness (PF) and out-of-school physical activity (PA) at a county level. A total of 715 primary school students (grades 3–6) were surveyed from one urban school (n = 438) and four rural schools (n = 277) in a county-level administrative unit, Yangling District, Shaanxi, in western China. Physical fitness index (PFI) was measured and calculated based on the revised Chinese Student Physical Fitness Standards. Out-of-school PA and other variables of demographics, behavior and perception were collected using questionnaires. Statistical analyses explored urban-rural differences and correlates of PFI and out-of-school PA. We found that the PFI (72.86 vs. 79.67) and weekly moderate-to-vigorous physical activity (MVPA) duration (167.57 vs. 220.08) of urban students were significantly lower than those of rural students. Weekly MVPA duration had the largest positive impact on PFI. Perceived availability of PA spaces was positively associated with both the urban and rural students’ PF and PA, while screen time was negatively associated with PF and PA, especially for rural students. Facilitators of PA frequency include the perceived availability of PA time and parental educational level. Parents’ PA habits had a positive impact on urban students’ PA. No association between active school commuting and PF or PA was found. Our findings revealed that PF and out-of-school PA of urban students were clearly lower than among rural students. The health of rural children at the county level in western China should be paid much more attention during the process of rapid urbanization.