Purpose We investigated the impact of body composition on outcomes of patients with early breast cancer. Skeletal muscle mass, visceral adipose tissue (VAT), subcutaneous adipose tissue (SAT), and muscle fat infiltration or inter-muscular adipose tissue areas (IMAT), obtained by computed tomography (CT), were assessed. Methods A total of 119 female patients who had breast cancer were included in this retrospective study. The total skeletal muscle and fat tissue areas were evaluated in two adjacent axial slices obtained at the third lumbar vertebra by CT used for disease staging. The women were assigned to either a sarcopenia or non-sarcopenia group based on their skeletal muscle index (cut-off 41.0 cm²/m²). They also were classified into high and low VAT/SAT ratio groups and assigned to either the high or low IMAT index group. The association of the body composition parameters and prognosis was statistically analyzed. Results Among the 119 evaluable patients, 58 were sarcopenic (48.8%), 55 (46.2%) had a high VAT/SAT ratio, and 62 (52.1%) had a high IMAT index. Median follow-up was 52.4 months. Multivariate analysis revealed sarcopenia and IMAT index as independent prognostic factors for disease-free survival (p = 0.02 and p = 0.04, respectively) and overall survival (p = 0.05 and p = 0.02, respectively). BMI was not significantly associated with disease-free survival, but a trend was observed (p = 0.09). Conclusions Sarcopenia and IMAT index are independent prognostic factors in early breast cancer; therefore, assessing body composition could be a simple and useful approach to integrate into patient management.