Background: Older cancer survivors in general are at greater risk for cancer-related cognitive impairment (CRCI), yet few studies have explored its association with health outcomes. This study examined the association between subjective and objective measures of cognitive function and physical function, frailty, and quality of life (QoL) among older breast cancer survivors. Materials and Methods: Older breast cancer survivors who reported cognitive concerns completed surveys on patient-reported cognitive function, physical function, frailty, and QoL as well as objective tests of visuospatial working memory and sustained attention. Data were analyzed using descriptive statistics and separate linear regression models. Results: A total of 219 female breast cancer survivors completed the study. Perceived cognitive abilities were associated with better physical function, frailty, and QoL (p ≤ 0.001) while cognitive concerns were negatively related with these metrics (p ≤ 0.001). Poorer visuospatial working memory and sustained attention were linked to increased frailty (p ≤ 0.001–0.01), whereas poorer sustained attention was associated with poorer physical function (p < 0.01). Conclusions: Older breast cancer survivors with perceived cognitive impairment and poorer cognitive performance reported poorer physical functioning, increased frailty, and poorer QoL. These findings underscore the importance of assessing cognitive concerns and their associated outcomes in older breast cancer survivors.