Trace metal element contamination in mining areas is always a huge environmental challenge for the global mining industry. In this study, an abandoned sphalerite mine near the Yanshan Mountains was selected as subject to evaluate the soil and water contamination caused by small-scale mining. The results show that (1) Pearson correlation matrix and principal component analysis (PCA) results reveal that Zn, Cu, Cd, and Pb were greatly affected by the operation of mines, especially mineral tailings. The contents of trace metal elements decrease with the increase of the distance from the mining area. Zinc, Pb, and Cd were discovered in almost all soil samples, and Zn accounted for about 80% of pollution of the topsoil. (2) The trace element pollution levels in the topsoil of the three villages were ranked as follows: Cd > Cu > Pb~Zn. The potential ecological risk of farmland around the mine ranges from lower to higher, with Cd being the most harmful. (3) Human health risk assessment results show that trace elements in the mining area pose obvious non-carcinogenic health risks to children while the risks to adults are not equally obvious. The carcinogenic risk of Cd and Cr is within a safe range and does not pose an obvious cancer risk to the population.