It is necessary but difficult to accurately predict the water levels in front of the pumping stations of an open-channel water transfer project because of the complex interactions among hydraulic structures. In this study, a novel GRA-NARX (gray relation analysis—nonlinear auto-regressive exogenous) model is proposed based on a gray relation analysis (GRA) and nonlinear auto-regressive exogenous (NARX) neural network for 2 h ahead for the prediction of water levels in front of pumping stations, in which an improved algorithm of the NARX neural network is used to obtain the optimal combination of the time delay and the hidden neurons number, and GRA is used to reduce the prediction complexity and improve the prediction accuracy by filtering input factors. Then, the sensitivity to changes of the training algorithm is analyzed, and the prediction performance is compared with that of the NARX and GRA-BP (gray relation analysis back-propagation) models. A case study is performed in the Tundian pumping station of the Miyun project, China, to demonstrate the reliability and accuracy of the proposed model. It is revealed that the GRA-NARX-BR (gray relation analysis—nonlinear auto-regressive exogenous—Bayesian regularization) model has higher accuracy than the model based only on a NARX neural network and the GRA-BP model with a correlation coefficient (R) of 0.9856 and a mean absolute error (MAE) of 0.00984 m. The proposed model is effective in predicting the water levels in front of the pumping stations of a complex open-channel water transfer project.