Heavy metal poisoning has caused serious and widespread human tragedies via the food chain. To alleviate heavy metal pollution, particular attention should be paid to low accumulating vegetables and crops. In this study, the concentrations of five hazardous heavy metals (HMs), including copper (Cu), chromium (Cr), lead (Pb), cadmium (Cd), and arsenic (As) were determined from soils, vegetables, and crops near four typical mining and smelting zones. Nemerow’s synthetical pollution index (P_n), Potential ecological risk index (RI), and Geo-accumulation index (I_geo) were used to characterize the pollution degrees. The results showed that soils near mining and metal smelting zones were heavily polluted by Cu, Cd, As, and Pb. The total excessive rate followed a decreasing order of Cd (80.00%) > Cu (61.11%) > As (45.56%) > Pb (32.22%) > Cr (0.00%). Moreover, sources identification indicated that Cu, Pb, Cd, and As may originate from anthropogenic activities, while Cr may originate from parent materials. The exceeding rates of Cu, Cr, Pb, Cd, and As were 6.7%, 6.7%, 66.7%, 80.0%, and 26.7% among the vegetable and crop species, respectively. Particularly, vegetables like tomatoes, bell peppers, white radishes, and asparagus, revealed low accumulation characteristics. In addition, the hazard index (HI) for vegetables and crops of four zones was greater than 1, revealing a higher risk to the health of local children near the mine and smelter. However, the solanaceous fruit has a low-risk index (HI), indicating that it is a potentially safe vegetable type.