Fiber-reinforced polymers (FRPs) are increasingly being used as substitutes for traditional metallic materials across various industries due to their exceptional strength-to-weight ratio. However, their orthotropic properties make them prone to multiple forms of damage, posing significant challenges in their design and application. During the design process, FRPs are subjected to various loading conditions to study their microscopic damage behavior, typically assessed through scanning electron microscopy (SEM). While SEM provides detailed insights into fracture surfaces, the manual analysis of these images is labor-intensive, time-consuming, and subject to variability based on the observer’s expertise. To address these limitations, this research proposes a deep learning-based approach for the autonomous microscopic damage assessment of FRPs. Several computationally efficient pre-trained deep learning models, such as DenseNet121, NasNet Mobile, EfficientNet, and MobileNet, were evaluated for their performance in identifying different damage modes autonomously, thus reducing the need for manual interpretation. SEM images of FRPs with five distinct failure modes were used to validate the proposed method. These failure modes include three fiber-based failures such as fiber breakage, fiber pullout, and mixed-mode failure, and two matrix-based failures such as matrix brittle failure and matrix ductile failure. The entire dataset is divided into train, validation, and test sets. Deep learning models were established by training on train and validation sets for five failure modes, while the test set was used as the unseen data to validate the models. The models were assessed using various evaluation metrics on an unseen test dataset. Results indicate that the EfficientNet model achieved the highest accuracy of 97.75% in classifying the failure modes. The findings demonstrate the effectiveness of employing deep learning techniques for microscopic damage assessment, offering a more efficient, consistent, and scalable solution compared to traditional manual analysis.