The concentrations of the heavy metals Pb, Cd, Cr, Hg, As, Cu and Zn in soil and locally produced grain (rice) were determined in paddy soil and rice around an abandoned high-arsenic coal mine area of Xingren county, southwest China. The health risk assessment was used to assess the multimedia and multipathway health risks of HM exposure in the study area. The results showed that the concentrations of As, Pb and Cd in soil were all higher than the corresponding limits for HMs in China. In terms of the accumulation and transfer capacity, Cd was more likely to transfer from the roots to rice, and its strong mobility may pose potential risks to local residents. The non-carcinogenic risks and carcinogenic risks of HM exposure in different media and exposure pathways were higher in children than adults. The total non-carcinogenic risks and carcinogenic risks in adults and children were higher than the standard limit values because of the HM exposure through ingesting rice husk. Among the exposure pathways evaluated, the contribution of diet was the largest, and As was the most important heavy metal in terms of the non-carcinogenic risk and carcinogenic risk factors. The total non-carcinogenic risks and carcinogenic risks caused by As in dietary crop (rice) accounted for 52% of the total in both adults and children. In order to maintain the health of residents in the study area, it is necessary to strictly strengthen the monitoring of heavy metal pollution in the study area and find effective soil improvement methods to reduce the health risks caused by heavy metal exposure.