Background: Mineral intake may protect against cognitive impairment (CI) and all-cause dementia, which affects a large number of adults worldwide. The aim of this study was to investigate the association between mineral intake and Montreal Cognitive Assessment (MoCA), which is a sensitive and specific test. Methods: In total, 201 adults were included in a cross-sectional study. They completed a three-day dietary record to estimate their average daily intake of minerals. Contributions to dietary reference intakes (DRIs) were also calculated. The participants were divided into tertiles according to their mineral intake. CI classifications were determined via the MoCA (score < 26). Apolipoprotein E (APOE) genotyping was carried out, and the patients’ anthropometric measurements and physical activity, health and personal data were collected. Results: The prevalence of CI in this selective sample was 54.2% (34.3% females and 19.9% males). In women, being in the third tertiles of iron and manganese intake was associated with lower odds of having CI (OR [95% CI]: 0.32 [0.11 ± 0.93]; 0.33 [0.12 ± 0.93], p < 0.05). No significant differences were observed for any of the nutrients studied in men. Conclusions: These findings suggest that a low mineral intake, especially low iron and manganese intake in women, is associated with a worse cognition as assessed by MoCA.