Anxiety and depression are frequent among patients with breast cancer (BCa). Evidence of the persistence and recovery from these conditions and their determinants is scarce. We describe the occurrence of clinically significant anxiety and depression symptoms and their associated factors among BCa patients. A total of 506 women admitted in 2012 at the Portuguese Institute of Oncology of Porto were evaluated before treatment and after one, three, and five years (7.9% attrition rate). The five-year prevalence of anxiety and/or depression (Hospital Anxiety and Depression Scale, subscores ≥ 11) was 55.4%. The peak prevalence for anxiety was before treatment (38.0%), and after one year for depression (13.1%). One in five patients with anxiety/depression at baseline had persistent anxiety/depression over time, while only 11% and 22% recovered permanently from anxiety and depression, respectively, during the first year. Higher education, higher income, practicing physical activity, and adequate fruit and vegetable intake were protective factors against anxiety and/or depression. Loss of job and income, anxiolytics and antidepressants, cancer-related neuropathic pain, and mastectomy were associated with higher odds of anxiety and/or depression. These results highlight the importance of monitoring anxiety/depression during the first five years after cancer diagnosis and identify factors associated with these conditions.