Some soil heavy metal pollution, such as As (Arsenic) and Cd (cadmium), in the black shale areas of western Zhejiang, exhibits significant geological background characteristics, yet the migration patterns and bioavailability are unclear. This study systematically integrated geochemical investigations of the rock-weathered soil–water–soil system to reveal the migration mechanisms and the species of the potentially toxic elements (PTEs) in black shale regions. The results showed that strongly acidic drainage (pH = 3.9) released from black shale weathering led to significant enrichment of Cd and As in soils. The mean Cd concentration (0.84 mg/kg) was 3.3 times higher than the Zhejiang background value, with active speciation (exchangeable fraction and humic acid-bound fraction) dominating during migration. This research provides a scientific basis for PTE prevention and control in geologically high-background regions.