Background Low back pain (LBP) may originate from different sources such as intervertebral disc degeneration (IVDD), end-plate and paraspinal muscle changes. Our aim is to explore the relevance of paraspinal muscles’ fat-infiltration in women with LBP and its association with IVDD and Modic changes. Methods Consecutive female patients presenting with chronic LBP to the outpatient clinics were included. Patients were evaluated in terms of IVDD, vertebral end-plate changes, and fatty infiltration in the paraspinal muscles at all lumbar levels on lumbar spine magnetic resonance imaging (MRI). Visual Analogue Scale (VAS) scores were recorded using our prospectively collected database. Results Patients with higher VAS scores were significantly more likely to have more fatty infiltration in the multifidus and less fatty infiltration in the psoas at L4-L5 level when compared to those with lower VAS scores (69.1 vs. 31.8%, p = 0.003). To predict LBP, fatty infiltration in the multifidus and psoas had odds ratio (OR) of 4 (p = 0.010), and 0.3 (p = 0.013), respectively; whereas disc degeneration had an OR of 0.5 (p = 0.028). Conclusion This is the first clinical cross-sectional study suggested that women with chronic low back pain could have less fat-infiltrated psoas to compensate more fat-infiltrated multifidus at L4-L5 disc level.