To address the issue of low accuracy and stability in traditional Convolutional Neural Networks (CNN)-based defect depth prediction for civil aircraft composites, we propose an improved Feature Enhancement Network (FEN)-CNN-Bidirectional Long Short-Term Memory (BiLSTM) impact damage depth prediction method. By integrating terahertz (THz) time-domain, frequency-domain, and absorbance spectroscopy with Confocal Laser Scanning Microscopy (CLSM) depth measurements, the correlation between THz spectral features and impact damage defect depth is systematically elucidated, thereby constructing a “THz features-depth” dataset. Furthermore, by leveraging the FEN model’s feature enhancement and denoising capabilities, along with the BiLSTM model’s bidirectional sequence modeling capability, the underlying relationship between terahertz spectral features and defect depth is deeply learned. This approach improves the stability and accuracy of spectral feature extraction by the CNN model under complex conditions. Ablation experiments revealed the improved model, compared to traditional CNN, reduced Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) by 43.08%, 44.4%, 57.18%, and 34.56%, respectively. Additionally, it decreased the Relative Standard Deviation (RSD) by 32.14%, and increased the Coefficient of Determination (R²) by 6.8%. Comparative experiments demonstrated the proposed model achieved an MSE of 0.0075 and an R² of 0.9539, outperforming other models. This study provides a novel method for precise low-velocity impact damage assessment in carbon fiber reinforced composites, enhancing safety evaluation for civil aircraft composite structures and contributing to aviation safety.