Cancer-related fatigue (CRF) is prevalent and onerous for cancer survivors. Not all survivors respond equally to interventions, but the characteristics distinguishing responders and non-responders are often unknown. This secondary analysis study compared baseline characteristics for responders (CRF reduction ≥2 points), non-responders, and those lost to follow-up using data from a two-group pre-test/post-test trial of a four-week exercise intervention compared to usual care. Included were 278 adult cancer survivors, with a mean age of 52.2 ± 11.9, 65% (180/278) female, and 90% (250/278) Caucasian. Of these, 77 (28%) were responders, 153 (55%) were non-responders, and 48 (17%) were lost to follow-up. At baseline, participants completed the 6-item Schwartz Cancer Fatigue Scale, with responses from 1 (not at all) to 5 (extremely fatigued) and a total score ranging 6–30. In the intervention group, 35% (49/141) reported decreased fatigue, 24% (34/141) reported increased fatigue, 25% (35/141) had minimal change, and 16% (23/141) were lost to follow-up. In the control group, 20% (28/137) reported decreased fatigue, 39% (53/137) reported increased fatigue, 23% (31/137) had minimal change, and 18% (25/137) were lost to follow-up. Responders in both groups reported higher baseline fatigue than non-responders, with mean differences of 5.2 (95% CI: 3.6–6.8) and 5.4 (95% CI: 3.4–7.3) for intervention and usual care, respectively. Higher baseline fatigue was found in responders compared to non-responders, regardless of group assignment, suggesting that those with a greater fatigue burden may have derived more benefit from exercise for CRF or a regression to the mean effect.