Carbon fiber reinforced plastics inevitably develop defects such as delamination, inclusions, and impacts during manufacturing and usage, which can adversely affect their performance. Ultrasonic phased array inspection is the most effective method for conducting nondestructive testing to ensure their quality. However, the diversity of defects within carbon fiber reinforced plastics makes it challenging for the current ultrasonic phased array inspection techniques to accurately identify these defects. Therefore, this paper presents a method for the ultrasonic phased array nondestructive testing and classification of various internal defects in carbon fiber reinforced plastics based on convolutional neural networks. We prepared an ultrasonic C-scan dataset containing multiple types of internal defects, analyzed the defect features in the ultrasonic C-scan images, and established an autoencoded classifier network to recognize manufacturing defects and impact defects of varying sizes. The experiments showed that the proposed method demonstrates superior defect feature extraction capabilities and can more accurately identify both impact and manufacturing defects.