Solid cancer progression is dictated by neoplastic cell features and pro-tumoral crosstalks with their microenvironment. Stroma modifications, such as fibroblast activation into cancer-associated fibroblasts (CAFs) and extracellular matrix (ECM) remodeling, are now recognized as critical events for cancer progression and as potential therapeutic or diagnostic targets. The recent appreciation of the key, complex and multiple roles of the ECM in cancer and of the CAF diversity, has revolutionized the field and raised innovative but challenging questions. Here, we rapidly present CAF heterogeneity in link with their specific ECM remodeling features observed in cancer, before developing each of the impacts of such ECM modifications on tumor progression (survival, angiogenesis, pre-metastatic niche, chemoresistance, etc.), and on patient prognosis. Finally, based on preclinical studies and recent results obtained from clinical trials, we highlight key mechanisms or proteins that are, or may be, used as potential therapeutic or diagnostic targets, and we report and discuss benefits, disappointments, or even failures, of recently reported stroma-targeting strategies.