The quality of carbohydrates has metabolic consequences in people with prediabetes. However, the causality of short-chain fermentable carbohydrate intakes and metabolic parameters has not been explored in the prediabetic or diabetic population. We investigated associations between different types of carbohydrates, including fermentable oligosaccharides, disaccharides, monosaccharides, polyols (FODMAPs), and polysaccharides (dietary fibre), and body composition and glucose/insulin responses in subjects with prediabetes. In this prospective cross-sectional study, 177 subjects with impaired glucose tolerance (IGT) (mean age: 60 (54–62) years, 41% men) underwent an assessment of body composition and completed six-point oral glucose tolerance tests (OGTT), Homeostatic Model Assessment of Insulin Resistance (HOMA2-IR), insulin sensitivity, detailed 3-day food records, and physical activity questionnaire. Daily habitual FODMAP intake decreased progressively with increasing BMI, ranging from 7.9 (6.2–12.7) g/d in subjects with normal BMI and 6.6 (4.6–9.9) g/d in subjects with overweight to 5.8 (3.8–9.0) g/d in subjects with obesity (p = 0.038). After adjustment for age and gender, galactooligosaccharides (GOSs) were negatively correlated with body fat (Standardised Beta coefficient β = −0.156, p = 0.006) and positively associated with insulin sensitivity (β = 0.243, p = 0.001). This remained significant after adjustment for macronutrients, fibre, and physical activity (p = 0.035 and p = 0.010, respectively). In individuals with IGT, higher dietary GOS intake was associated with lower body fat and higher insulin sensitivity independent of macronutrients and fibre intake, calling for interventional studies to evaluate the effect of FODMAP intake in prediabetes.