Dendrobium officinale polysaccharide (DOP), which serves as a prebiotic, exhibits a variety of biological activities, including hypoglycemic activities. However, the effects of DOP on diabetes prevention and its hypoglycemic mechanisms are still unclear. In this study, the effects of DOP treatment on the prediabetic mice model were studied and the mechanism was investigated. The results showed that 200 mg/kg/d of DOP reduced the relative risk of type 2 diabetes mellitus (T2DM) from prediabetes by 63.7%. Meanwhile, DOP decreased the level of LPS and inhibited the expression of TLR4 by regulating the composition of the gut microbiota, consequently relieving the inflammation and alleviating insulin resistance. In addition, DOP increased the abundance of SCFA (short chain fatty acid)-producing bacteria in the intestine, increased the levels of intestinal SCFAs, promoted the expression of short-chain fatty acid receptors FFAR2/FFAR3, and increased the secretion of the intestinal hormones GLP-1 and PYY, which helped to repair islet damage, suppress appetite, and improve insulin resistance. Our results suggested that DOP is a promising functional food supplement for the prevention of T2DM.