In the semiconductor industry, especially in the manufacturing of Through-Glass Vias (TGVs), there is an increasing need to improve the quality and efficiency of manufacturing processes. To address the challenges such as lack of efficiency, requiring substantial manual labor, and falling short in precision of traditional methods in meeting high standards for TGVs manufacturing, the approach that combines deep learning and optimization techniques was introduced to achieve automatic quality assessment and refine laser drilling parameters for TGVs manufacturing. We have developed a residual U-Net model with an accuracy of up to 87.9% by training high-resolution scanning electron microscope (SEM) images of TGVs for automatic assessment of TGVs quality, closely matching the assessments made by human experts. We used Bayesian optimization to iteratively adjust the laser drilling parameters that are crucial for TGVs manufacturing, and the quality scores obtained by residual U-Net model enhanced 13.2% after 50 iterations, which confirms the effectiveness of the integration of U-Net architecture with Bayesian optimization in achieving optimal manufacturing results.