The neural organization of semantic memory remains much debated. A ‘distributed-only’ view contends that semantic knowledge is represented within spatially distant, modality-selective primary and association cortices. Observations in semantic variant primary progressive aphasia have inspired an alternative model featuring the anterior temporal lobe as an amodal hub that supports semantic knowledge by linking distributed modality-selective regions. Direct evidence has been lacking, however, to support intrinsic functional interactions between an anterior temporal lobe hub and upstream sensory regions in humans. Here, we examined the neural networks supporting semantic knowledge by performing a multimodal brain imaging study in healthy subjects and patients with semantic variant primary progressive aphasia. In healthy subjects, the anterior temporal lobe showed intrinsic connectivity to an array of modality-selective primary and association cortices. Patients showed focal anterior temporal lobe degeneration but also reduced physiological integrity throughout distributed modality-selective regions connected with the anterior temporal lobe in healthy controls. Physiological deficits outside the anterior temporal lobe correlated with scores on semantic tasks and with anterior temporal subregion atrophy, following domain-specific and connectivity-based predictions. The findings provide a neurophysiological basis for the theory that semantic processing is orchestrated through interactions between a critical anterior temporal lobe hub and modality-selective processing nodes.