Vision-based inspection systems are essential for quality control in manufacturing industries, and advances in artificial intelligence (AI) have significantly enhanced their accuracy. However, the high-precision requirements of products such as contact lenses demand even more robust inspection methods. This paper introduces a novel defect-adaptive hierarchical structure convolution neural network (DHS-CNN) model based on InceptionV4. The proposed model architecture reflects the manufacturing process and defect types, and we developed a custom loss function to suit this multi-output hierarchical design. Experimental results on a dataset of 2800 contact lens images revealed that the proposed model improved accuracy by 2.08% over the baseline model. These findings suggest that the defect-adaptive hierarchical structure and customized loss function offer substantial improvements in the vision-based inspection of contact lenses and may enhance AI-driven quality control processes in other manufacturing sectors.