Invasive lobular carcinoma (ILC) is the second most common histological subtype of breast cancer, and it exhibits a number of clinico-pathological characteristics distinct from the more common invasive ductal carcinoma (IDC). We set out to identify alterations in the tumor microenvironment (TME) of ILC. We used laser-capture microdissection to separate tumor epithelium from stroma in 23 ER+ ILC primary tumors. Gene expression analysis identified 45 genes involved in regulation of the extracellular matrix (ECM) that were enriched in the non-immune stroma of ILC, but not in non-immune stroma from ER+ IDC or normal breast. Of these, 10 were expressed in cancer-associated fibroblasts (CAFs) and were increased in ILC compared to IDC in bulk gene expression datasets, with PAPPA and TIMP2 being associated with better survival in ILC but not IDC. PAPPA, a gene involved in IGF-1 signaling, was the most enriched in the stroma compared to the tumor epithelial compartment in ILC. Analysis of PAPPA- and IGF1-associated genes identified a paracrine signaling pathway, and active PAPP-A was shown to be secreted from primary CAFs. This is the first study to demonstrate molecular differences in the TME between ILC and IDC identifying differences in matrix organization and growth factor signaling pathways.