In this study, we identified the sources of trace metals, investigated their spatial distribution in topsoil and assessed their potential ecological risk in the area surrounding a typical manganese mining area in Xiangtan, Hunan Province, China. The concentrations of Mn, Cu, Pb, Zn, Cd, Ni, Cr and Hg in the topsoil of the study area were measured. Except for Cr and Hg, all trace metals exceeded the corresponding soil background values for Hunan Province. The spatial variation in trace metals was visualized by GIS, and the results show that trace metals in topsoil are enriched mainly around mines and smelters. Two groups of trace metals were identified using the spatial distribution, trend analysis, Pearson’s correlation and principal component analysis: Mn, Cu, Pb, Zn, Cd and Ni can be attributed to industrial and mining activities, whereas Cr and Hg are of natural origin. The results also revealed the extent of the influence of secondary processes such as the prevailing wind direction, erosion of mine tailings and rainwater runoff play significant roles in the wider dispersal and transfer of trace metals. In addition, the environmental risk of metal pollution was evaluated by applying the geoaccumulation index and potential ecological risk index (PERI) to the study area. The accumulated PERI for metals of concern is at highest risk level in the main manganese mine area. This decreases to a moderate risk around the manganese mine area, highlighting locations for further risk management concern. Furthermore, nearly 80% of the potential ecological risk was from Cd across the study area.