Background: Cognitive function generally declines with the aging process. Although the association of physical fitness with cognitive function has been proved, how many and how well the physical fitness components are linked to cognitive function is not clear. This study aimed to examine the association of physical fitness with cognitive function, and find out which aspects of physical fitness components are the most closely related to cognitive function in community-dwelling older adults. Methods: This cross-sectional study was conducted from March to July 2019. The sample consisted of 107 older people in the community with a mean age of 71.7 ± 5.0 years. The cognitive function of the participants was measured by a Chinese version of the Mini-Mental State Examination (MMSE). Several physical fitness items including grip strength, 5-repetition sit-to-stand, timed up and go, sit and reach, one-leg balance with the eye open, and 6-min walk were measured to reflect muscle strength, muscle endurance, agility, flexibility, balance, and cardiopulmonary endurance, respectively. Results: The correlation analysis showed that the grip strength and the 6-min walk were positively related to cognitive function (r = 0.42 and 0.35, P < 0.05), while the 5-repetition sit-to-stand was negatively associated with cognitive function (r = -0.43, P < 0.01) adjusting for sex, age and years of education. It was also found that the mean values of physical fitness items including grip strength and 6-min walk were significantly lower, and 5-repetition sit-to-stand and timed up and go were significantly greater in the older adults with cognitive impairment (MMSE score < 27) than those in the normal older adults (MMSE score ≥ 27) (P < 0.05). Stepwise regression analysis revealed that age, together with physical fitness items including grip strength and 6-min walk can explain the cognitive function in older adults. Conclusion: The findings suggest that there is an association between physical fitness and cognitive function, and the grip strength and 6-min walk can help explain the cognitive function in community-dwelling older adults. More attention needs to be paid to the increase in physical fitness for preventing or improving the cognitive dysfunction of older persons in the community, and further longitudinal study is warranted.