Cancer is the second leading cause of death worldwide and the global cancer burden rises rapidly. The risk factors for cancer development can often be attributed to lifestyle factors, of which an unhealthy diet is a major contributor. Dietary fat is an important macronutrient and therefore a crucial part of a well-balanced and healthy diet, but it is still unclear which specific fatty acids contribute to a healthy and well-balanced diet in the context of cancer risk and prognosis. In this review, we describe epidemiological evidence on the associations between the intake of different classes of fatty acids and the risk of developing cancer, and we provide preclinical evidence on how specific fatty acids can act on tumor cells, thereby modulating tumor progression and metastasis. Moreover, the pro- and anti-inflammatory effects of each of the different groups of fatty acids will be discussed specifically in the context of inflammation-induced cancer progression and we will highlight challenges as well as opportunities for successful application of fatty acid tailored nutritional interventions in the clinic.