Fine-grained maturity assessment of Sichuan pepper (Zanthoxylum bungeanum cv. Hanyuan) is challenging: CNNs are data-hungry and opaque, and vision-language models (VLMs) achieve only chance-level accuracy ($16.67\%$) on this six-way task when used end-to-end. We present KG-RAG, a retrieval-augmented classification framework that combines CNN visual features with a 25,881-triplet knowledge graph built from VLM-extracted structured attributes. KG-RAG introduces three key components: (i) a hierarchical attribute consistency score (HACS) that generalizes Jaccard re-ranking via mutual-information weighting and family-level regularization; (ii) a confidence-guided retrieval gate for adaptive parametric/non-parametric fusion, with calibration as a secondary benefit; and (iii) a two-stage VLM curriculum that repurposes a VLM—inaccurate as an end-to-end classifier but reliable as an attribute extractor—into a structured knowledge provider. On 2114 expert-annotated images (inter-annotator Cohen’s $\kappa =0.89$), $10\times 5$-fold cross-validation shows consistent gains across four backbones (+1.25 to +4.65 percentage points), with the largest gain in the low-data regime (+14.89 pp at 10% training data). Statistical significance is assessed via paired t-tests with repeated-CV caveats.