Cancer cells rely on the tumor microenvironment (TME), a composite of non-malignant cells, and extracellular matrix (ECM), for survival, growth, and metastasis. The ECM contributes to the biomechanical properties of the surrounding tissue, in addition to providing signals for tissue development. Cancer-associated fibroblasts (CAFs) are stromal cells in the TME that are integral to cancer progression. Subtypes of CAFs across a variety of cancers have been revealed, and each play a different role in cancer progression or suppression. CAFs secrete signaling molecules and remodel the surrounding ECM by depositing its constituents as well as degrading enzymes. In cancer, a remodeled ECM can lead to tumor-promoting effects. Not only does the remodeled ECM promote growth and allow for easier metastasis, but it can also modulate the immune system. A better understanding of how CAFs remodel the ECM will likely yield novel therapeutic targets. In this review, we summarize the key factors secreted by CAFs that facilitate tumor progression, ECM remodeling, and immune suppression.