Aldehyde dehydrogenase 1A1 (ALDH1A1) is a cancer stem cell (CSC) marker related to clinical outcomes in breast cancer (BC). The aim of this study was to analyze the relationship between ALDH1A1, programmed death ligand 1 (PD-L1) and tumor-infiltrating lymphocytes (TILs) in triple negative (TN) and human epidermal growth factor receptor 2-positive (HER2+) BC tumors, and its association with clinicopathological characteristics and outcomes. A retrospective, historical cohort study of patients diagnosed with early or locally advanced BC treated with neoadjuvant chemotherapy was conducted. ALDH1A1, PD-L1 expression and TILs were assessed using immunohistochemistry. A total of 75 patients were analyzed (42.7% TN, 57.3% HER2+ tumors). ALDH1A1+ was related to HTILs (p = 0.005) and PD-L1+ tumors (p = 0.004). ALDH1A1+ tumors presented higher CD3+ (p = 0.008), CD4+ (p = 0.005), CD8+ (p = 0.003) and CD20+ (p = 0.006) TILs. ALDH1A1+ (p = 0.018), PD-L1+ (p = 0.004) and HTILs (p < 0.001) were related to smaller tumors. ALDH1A1+ was related to pathologic complete response (pCR) (p = 0.048). At the end of the follow-up (54.4 [38.3–87.6] months), 47 patients (62.7%) remained disease-free, and 20 (26.7%) had died. HTILs were related to improved disease-free survival (p = 0.027). ALDH1A1+ was related to PD-L1+ and HITLs, that might be related to higher pCR rates with neoadjuvant therapy.