Surrounding rock deformation in soft rock tunnels is controlled by complex nonlinear interactions among geological conditions, construction parameters, and support measures, making accurate prediction challenging. In this study, a project-scale deformation prediction dataset was constructed using field monitoring data from a sandy shale tunnel project. Eight engineering factors were selected as input variables, including excavation method, initial support strength, closure time, tunnel burial depth, lithology, rock integrity, groundwater condition, and the relative orientation between major structural planes and the tunnel. A hybrid prediction framework integrating a novel whale optimization algorithm (NWOA) and a long short-term memory (LSTM) network was developed. The proposed NWOA improves the standard whale optimization algorithm by introducing a nonlinear convergence strategy, an adaptive weight coefficient, and a dynamic spiral position updating mechanism to enhance the hyperparameter search process of the LSTM model. Model performance and stability were further assessed using repeated and nested cross-validation. The corresponding RMSEs were 0.2294 ± 0.0734 and 0.2219 ± 0.0751 percentage points, the MAEs were 0.1427 ± 0.0350 and 0.1505 ± 0.0585 percentage points, and the R² values were 0.8660 ± 0.0548 and 0.8732 ± 0.0599, respectively. These comparable results support project-specific predictive performance for the investigated tunnel sections.