Traditional road structural detection and evaluation is inefficient, imprecise, and destructive. To address these issues, a feature-enhanced multiscale vision transformer (FeMViT) for road distress classification from ground penetrating radar (GPR) images was proposed. FeMViT model used the feature-enhanced feature pyramid network (FPN) and feature enrichment module (FEM) to extract the distress better features on GPR images. The pooling attention was also modified using the residual pooling connection to reduce computational complexity and memory usage. Experimental results showed that this model further realized the comprehensive improvement of classification indexes for road distresses. The accuracy and $\bm{ F}_{1}$ score of the overall classification result was 91.9% and 90.8%, improved by 10.4% and 7.1% compared to the original Transformer, respectively. Misattribution and visualization analysis provided ideas for improvement directions. The internal distress rate ( $\bm{IDR}$ ) and internal pavement structural integrity score ( $\bm{IPSI}$ ) indexes of structural integrity were determined based on GPR images. Field tests suggested a good correlation between the structural strength and integrity indexes of asphalt pavement. This illustrates that the proposed method is reliable and could provide a more comprehensive approach to the structural condition assessment of asphalt pavement.