Being physically active as part of everyday life reduces breast cancer risk. Less is known whether the benefits of an active lifestyle differ depending on the timing of physical activity in life or anthropometric characteristics. The aim of this study was to bring further insights to the association of physical activity in relation to menopausal status and body composition with breast cancer risk by making use of a prospective Swedish cohort (Malmö Diet and Cancer Study) with long-term follow-up. Physical activity information of 15,983 participants for the past 12 months prior to study entry was assessed according to metabolic equivalent task (MET)-hours/week to integrate duration and intensity of reported activities. During 23.2 years median follow-up, 1302 invasive breast cancers occurred. Women reporting a high physical activity at study baseline, corresponding to >1 h daily walking/week (≥28.5 MET-h/week), had a 23% lower long-term breast cancer risk (HR_adj = 0.77, 95% CI 0.66–0.90) than those reporting low physical activity, being most pronounced among perimenopausal and postmenopausal women, and women with waist circumference, body fat percentage, or BMI in the upper-normal and overweight range. For premenopausal women or women having obesity or the largest body composition, high physical activity alone did not modify the breast cancer risk, suggesting additional preventive measures indicated in these groups to reduce the long-term risk of breast cancer.