Background: Immune checkpoint inhibitors (ICIs) represent a new therapeutic standard for an increasing number of tumor entities. Nevertheless, individual response and outcome to ICI is very heterogeneous, and the identification of the ideal ICI candidate has remained one of the major issues. Sarcopenia and the progressive loss of muscle mass and strength, as well as muscular fat deposition, have been established as negative prognostic factors for a variety of diseases, but their role in the context of ICI therapy is not fully understood. Here, we have evaluated skeletal muscle composition as a novel prognostic marker in patients undergoing ICI therapy for solid malignancies. Methods: We analyzed patients with metastasized cancers receiving ICI therapy according to the recommendation of the specific tumor board. Routine CT scans before treatment initialization and during ICI therapy were used to assess the skeletal muscle index (L3SMI) as well as the mean skeletal muscle attenuation (MMA) in n = 88 patients receiving ICI therapy. Results: While baseline L3SMI and MMA values were unsuitable for predicting the individual response and outcome to ICI therapy, longitudinal changes of the L3SMI and MMA (∆L3SMI, ∆MMA) during ICI therapy turned out to be a relevant marker of therapy response and overall survival. Patients who responded to ICI therapy at three months had a significantly higher ∆L3SMI compared to non-responders (−3.20 mm²/cm vs. 1.73 mm²/cm, p = 0.002). Moreover, overall survival (OS) was significantly lower in patients who had a strongly decreasing ∆L3SMI (<−6.18 mm²/cm) or a strongly decreasing ∆MMA (<−0.4 mm²/cm) during the first three month of ICI therapy. Median OS was only 127 days in patients with a ∆L3SMI of below −6.18 mm²/cm, compared to 547 days in patients with only mildly decreasing or even increasing ∆L3SMI values (p < 0.001). Conclusion: Both progressive sarcopenia and an increasing skeletal muscle fat deposition are associated with poor response and outcome to ICI therapy, which might help to guide treatment decisions during ICI therapy.