This study proposes a deep metric learning-based pavement distress classification method to address critical limitations in conventional approaches, including their dependency on large training datasets and inability to incrementally learn new categories. To resolve high intra-class variance and low inter-class distinction in distress images, we design a CNN head with multi-cluster centroins trained via SoftTriple loss, simultaneously maximizing inter-class separation while establishing multiple intra-class centers. An adaptive weighting strategy combining sample similarity and class priors mitigates data imbalance, while soft-label techniques reduce labeling noise by evaluating similarity against support-set exemplars. Evaluations on the UAV-PDD2023 dataset demonstrate superior performance—3.2% higher macro-recall than supervised learning, and 6.7%/8.5% improvements in macro-F1/weighted-F1 over iCaRL incremental learning—validating the method’s effectiveness for real-world road inspection scenarios with evolving distress types and limited annotation.