Chromium has long been known as an enhancer of insulin action. However, the role of chromium in the development of type 2 diabetes mellitus (T2DM) in humans remains controversial. The current study aimed to examine the associations of plasma chromium levels with T2DM and pre-diabetes mellitus (pre-DM). We conducted a case-control study involving 1471 patients with newly diagnosed T2DM, 682 individuals with newly diagnosed pre-DM, and 2290 individuals with normal glucose tolerance in a Chinese population from 2009 to 2014. Plasma chromium was measured by inductively coupled plasma mass spectrometry. Plasma chromium levels were lower in the T2DM and pre-DM groups than in the control group (median: 3.68 μg/L, 3.61 μg/L, 3.97 μg/L, respectively, p < 0.001). After adjustment for potential confounding factors, the odds ratios (95% confidence interval) for T2DM across increasing quartiles of plasma chromium levels were 1 (referent), 0.67 (0.55–0.83), 0.64 (0.51–0.79), and 0.58 (0.46–0.73), respectively (p for trend <0.001). The corresponding odds ratios (95% confidence interval) for pre-DM were 1 (referent), 0.70 (0.54–0.91), 0.67 (0.52–0.88), and 0.58 (0.43–0.78), respectively (p for trend < 0.001). Our results indicated that plasma chromium concentrations were inversely associated with T2DM and pre-DM in Chinese adults.