Tumor-associated macrophages (TAMs) are associated with a poor outcome in breast cancer (BC), but their prognostic value in different BC subtypes has remained somewhat unclear. Here, we investigated the prognostic value of M2-like TAMs (CD163+) and all TAMs (CD68+) in a patient cohort of 278 non-metastatic BC patients, half of whom were HER2+ (n = 139). The survival endpoints investigated were overall survival (OS), breast cancer-specific survival (BCSS) and disease-free survival (DFS). In the whole patient cohort (n = 278), a high CD163+ TAM count and a high CD68+ TAM count were associated with a worse outcome (p ≤ 0.023). In HER2+ BC, a high CD163+ TAM count was an independent factor for a poor prognosis across all the investigated survival endpoints (p < 0.001). The prognostic effect was evident in both the HER2+/hormone receptor-positive (p < 0.001) and HER2+/hormone receptor-negative (p ≤ 0.012) subgroups and regardless of the provision of adjuvant trastuzumab (p ≤ 0.002). In HER2-negative BC, the CD163+ TAM count was not significantly associated with survival. These results suggest that a high CD163+ TAM count predicts an inferior outcome, especially in HER2+ BC patients, and as adjuvant trastuzumab did not overcome the poor prognostic effect, combination treatments including therapies targeting the macrophage function could represent an effective therapeutic approach in HER2+ BC.