The APOE ε4 allele is a risk factor for late-life pathological changes that is also associated with anatomical and functional brain changes in middle-aged and elderly healthy subjects. We investigated structural and functional effects of the APOE polymorphism in 18 young healthy APOE ε4-carriers and 18 matched noncarriers (age range: 20–35 years). Brain activity was studied both at rest and during an encoding memory paradigm using blood oxygen level-dependent fMRI. Resting fMRI revealed increased “default mode network” (involving retrosplenial, medial temporal, and medial-prefrontal cortical areas) coactivation in ε4-carriers relative to noncarriers. The encoding task produced greater hippocampal activation in ε4-carriers relative to noncarriers. Neither result could be explained by differences in memory performance, brain morphology, or resting cerebral blood flow. The APOE ε4 allele modulates brain function decades before any clinical or neurophysiological expression of neurodegenerative processes.