As it is not easy to modify lifestyle, it is important to examine the effect of social capital (SC), which does not require behavior modifications, on dementia prevention. This study aimed to clarify gender differences in the relationship between cognitive function and individual SC among people living in a rural area in Japan. We used the Shimane Center for Community-based Healthcare Research and Education (CoHRE) study data from 2011 to conduct a cross-sectional analysis. The analysis included 491 participants, aged 40 years or older, who had undergone medical examinations in two rural towns in Japan. Both cognitive SC and structural SC were measured. Multivariate logistic regression analysis was conducted to estimate the odds ratios (OR) and 95% confidence interval (CI) for cognitive function levels as binary outcomes. We found a significant association between cognitive function and individual cognitive SC in men (OR: 3.11, 95% CI: 1.43–6.78), and found that cognitive function was associated with structural SC in women (OR: 1.89, 95% CI: 1.08–3.31). This study showed that the relationship between cognitive function and individual SC differed by gender. These results suggest that it is important to approach dementia prevention differently in men and women.