Except for aging, APOE ε4 allele ( APOE4 ) is the most important risk factor for sporadic Alzheimer’s disease. APOE4 carriers may have reduced capacity to recycle lipids, resulting in reduced white matter integrity. Here, we evaluated if white matter impairment measured by diffusion tensor imaging (DTI) differs between healthy individuals with different number of APOE4 alleles and if white matter impairment associates with brain beta-amyloid (Aβ) load and serum levels of neurofilament light chain (NfL). We studied 96 participants ( APOE4/4 , N = 20; APOE4/3 , N = 39; APOE3/3 , N = 37; mean age 70.7 (SD 5.22) years, 63% females) with brain MRI including a DTI sequence (N = 96), Aβ-PET (N = 89) and a venous blood sample for serum NfL concentration measurement (N = 88). Fractional anisotropy (FA), mean diffusivity (MD), radial diffusivity (RD) and axial diffusivity (AxD) in six a priori -selected white matter regions-of-interest (ROIs) were compared between the groups using ANCOVA, with sex and age as covariates. A voxel-weighted average of FA, MD, RD and AxD was calculated for each subject, and correlations with Aβ-PET and NfL levels were evaluated. APOE4/4 carriers exhibited higher MD and higher RD in the body of corpus callosum than APOE4/3 (p = 0.0053 and p = 0.0049, respectively) and APOE3/3 (p = 0.026 and p = 0.042). APOE4/4 carriers had higher AxD than APOE4/3 (p = 0.012) and APOE3/3 (p = 0.040) in the right cingulum adjacent to cingulate cortex. In the total sample, composite MD, RD and AxD positively correlated with cortical Aβ load (r = 0.26 to 0.33, p < 0.013 for all) and with serum NfL concentrations (r = 0.31 to 0.36, p < 0.0028 for all). In conclusion, reduced white matter integrity was detected in cognitively unimpaired APOE4/4 homozygotes compared to APOE4/3 and APOE3/3 carriers and reduced white matter integrity correlated with biomarkers of Alzheimer’s disease and neurodegeneration. White matter impairment seems to be an early phenomenon in the Alzheimer’s disease pathologic process in APOE4/4 homozygotes. Highlights: Cognitively normal elderly APOE4 homozygotes showed impaired white matter integrity APOE4 homozygotes showed higher diffusivity in corpus callosum and right cingulum MD, RD and AxD correlate with amyloid load assessed by PET in APOE4 carriers MD, RD and AxD correlate with serum neurofilament light chain in APOE4 carriers White matter impairment may be an early phenomenon in the AD pathologic process