Background/Objectives: Breast cancer is the leading cause of cancer-related mortality in women, highlighting the urgent need for robust prognostic tools to enable individualized risk stratification. Methods: Transcriptomic data from 1075 breast cancer and 113 adjacent normal tissues in The Cancer Genome Atlas (TCGA) were integrated with clinical information. Differential expression analysis identified 531 immune-related genes, which were further selected by univariate Cox regression and Least Absolute Shrinkage and Selection Operator (LASSO) regression to construct a 13-gene prognostic signature. The model was validated in an independent cohort (n = 327). Tumor immune microenvironment and single-cell RNA sequencing data were analyzed to explore underlying biological differences. Results: The 13-gene signature effectively stratified patients into low- and high-risk groups with significantly different overall survival in both the TCGA cohort (log-rank p < 0.0001; C-index = 0.678; 5-year AUC = 0.72) and the validation cohort (log-rank p < 0.0001; C-index = 0.703; 3-year AUC = 0.81). Low-risk tumors exhibited an antitumor immune microenvironment enriched in CD8⁺ T cells, T follicular helper (Tfh) cells, and M1 macrophages, whereas high-risk tumors were dominated by immunosuppressive regulatory T cells and M2 macrophages (all p < 0.0001). Single-cell analysis revealed expansion of malignant epithelial cells and inflammatory cancer-associated fibroblasts (iCAFs) in high-risk tumors, with higher iCAF scores significantly associated with poorer survival (log-rank p = 0.00036). Conclusions: Collectively, this study delivers a rigorously validated 13-gene immune signature whose prognostic utility is rooted in distinct immune microenvironmental features, while unveiling iCAF-targeted therapeutic strategies as a promising intervention avenue.