As survival rates increase, more emphasis has gone to possible cognitive sequelae in older cancer patients, which could be explained by accelerated brain aging. In this review, we provide a complete overview of studies investigating neuroimaging, neurocognitive, and neurodegenerative disorders in older cancer survivors (>65 years), based on three databases (Pubmed, Web of Science and Medline). Ninety-six studies were included. Evidence was found for functional and structural brain changes (frontal regions, basal ganglia, gray and white matter), compared to healthy controls. Cognitive decline was mainly found in memory functioning. Anti-hormonal treatments were repeatedly associated with cognitive decline (tamoxifen) and sometimes with an increased risk of Alzheimer’s disease (androgen deprivation therapy). Chemotherapy was inconsistently associated with later development of cognitive changes or dementia. Radiotherapy was not associated with cognition in patients with non-central nervous system cancer but can play a role in patients with central nervous system cancer, while neurosurgery seemed to improve their cognition in the short-term. Individual risk factors included cancer subtypes (e.g., brain cancer, hormone-related cancers), treatment (e.g., anti-hormonal therapy, chemotherapy, cranial radiation), genetic predisposition (e.g., APOE, COMT, BDNF), age, comorbidities (e.g., frailty, cognitive reserve), and psychological (e.g., depression, (post-traumatic) distress, sleep, fatigue) and social factors (e.g., loneliness, limited caregiver support, low SES). More research on accelerated aging is required to guide intervention studies.