Background: Inflammatory bowel diseases (IBD), including ulcerative colitis (UC) and Crohn’s disease (CD), are chronic conditions with complex etiologies that may involve dietary factors. This study investigates the association between fruit and vegetable consumption and the risk of UC and CD, focusing on gender-specific differences. Methods: A hospital-based case–control study, comprising 158 UC patients (93 men and 65 women), 245 CD patients (167 men and 78 women), and 395 controls without IBD (256 men and 139 women), was conducted in Riyadh, Saudi Arabia. Fruit and vegetable consumption data were gathered through a self-administered questionnaire distributed before diagnosis. Logistic regression analysis was applied to estimate the odds ratios (ORs) and 95% confidence intervals (95% CIs) for UC and CD among the participants reporting the daily consumption of fruits and vegetables. Results: Among men, daily vegetable consumption was associated with higher odds of UC in the age-adjusted model [OR (95% CI): 1.78 (1.02, 3.10)], but this association became non-significant after further adjustment for body mass index, smoking, anemia, and elevated liver enzymes [OR (95% CI): 1.70 (0.91, 3.18)]. No significant associations were observed between vegetable consumption and CD. In contrast, the women who consumed vegetables every day had a non-significant inverse association with UC and a significant inverse association with CD in both the age-adjusted and multivariable-adjusted models [ORs (95% CIs): 0.44 (0.23, 0.87) and 0.41 (0.20, 0.84), respectively]. Fruit consumption was neither associated with UC nor CD in either sex. Conclusions: Daily vegetable consumption was significantly associated with decreased odds of CD among women, but not men, highlighting potential sex-specific dietary influences on IBD risk.