Background/Objectives: Recent clinical trials in breast cancer have demonstrated that some patients benefit from immune checkpoint blockade, though better predictive markers are needed. The accumulation of the immunomodulatory matrix proteoglycan versican (VCAN) can predict the exclusion of CD8⁺ tumor-infiltrating lymphocytes (TILs) in some settings and, thus, is evaluated in breast cancer here. Methods: A total of 230 breast cancers were analyzed for VCAN accumulation, VCAN proteolysis, and CD8⁺ TILs. CD8⁺ TILs were categorized based on their localization in the tumor epithelial or stromal compartments. Results: VCAN accumulation was detected in 90% of breast cancers, more commonly in ER+ tumors (93% vs. 77%; p < 0.001). MCF7 cells treated with estrogen upregulate VCAN without an enhanced expression of ADAMTS-proteases. VCAN-undetectable tumors demonstrate greater CD8⁺ TILs compared to VCAN-detectable tumors (p = 0.012). CD8⁺ T cells within TNBC tumors with high VCAN proteolysis infiltrated the epithelial compartment more often than in tumors with low VCAN proteolysis (91% vs. 42% respectively; p = 0.008). In the TCGA cohort, a strong inverse correlation between CD8A and VCAN expression was observed across subtypes. Conclusions: VCAN accumulation correlates with the exclusion of CD8⁺ TILs across subtypes of breast cancer, warranting further validation of VCAN accumulation and proteolysis as predictive biomarkers for breast cancer immunotherapy.