Bridge health assessment and deterioration prediction are essential for traffic safety and maintenance planning. However, conventional bridge evaluation still relies heavily on expert judgment and heuristic rules, limiting objectivity and long-term forecasting capability. This study proposes a physics-constrained multi-task learning–semi-Markov framework for bridge condition assessment, multi-year deterioration forecasting, and risk-aware maintenance prioritization. Guided by the Highway Bridge Technical Condition Rating Code, the proposed model jointly predicts health grade, defect type, and severity from inspection item/defect entry records, improving assessment robustness through cross-task information sharing. The predicted health states are then incorporated into a physics-constrained semi-Markov model to forecast bridge deterioration over a three-year horizon, and the current and predicted states are further used for maintenance prioritization. Experiments on real bridge inspection data show that the proposed model achieves test accuracies of 95.20%, 99.67%, and 99.72% for health grade, defect type, and severity, respectively, with a Cohen’s kappa of 0.872, while the proposed semi-Markov model achieves a three-year prediction accuracy of 95.4% and a weighted accuracy of 99.1%. Comparative and ablation studies further demonstrate the effectiveness of the proposed framework for bridge lifecycle management.