Breast tumors with neuroendocrine (NE) differentiation comprise an uncommon and heterogeneous group of tumors, including invasive breast cancer of no special type (IBC-NST) with NE features, neuroendocrine tumors (NETs), and neuroendocrine carcinoma (NEC). The most recent World Health Organization (WHO) classification in 2019 defined neuroendocrine neoplasms (NENs) of the breast (Br-NENs) as tumors in which >90% of cells show histological evidence of NE differentiation, including NETs (low-grade tumors) and NEC (high-grade). Due to the low prevalence of these tumors and successive changes in their diagnostic criteria over the years, only limited evidence of these tumors exists, derived mainly from case reports and retrospective case series. Breast tumors with NE differentiation are usually treated like the more commonly occurring IBC-NSTs. Immunohistochemistry (IHC) of breast tumors with NE differentiation usually shows a hormone receptor (HR)-positive and human epidermal growth factor type 2 (HER2)-negative profile, so that hormonal therapy with cyclin-dependent kinase (CDK)4/6 inhibitors or other targeted agents would be reasonable treatment options. Herein, we present a review of the literature on breast tumors with NE differentiation as defined in the latest WHO 2019 classification, and discuss the clinical management of these tumors.