Due to randomness factors in the machine learning model construction process, reproducibility is compromised. This study investigates the impact of randomness on model stability and evaluates techniques for reducing this impact using the widely adopted shallow neural network model as a testbed. Randomness in this neural network model arises from three events: randomly initializing model parameters, randomly selecting a validation subset, and randomly sampling batches for parameter updates. Among these, batch randomness exerts a much weaker impact than the other two factors. In this study, the model training is stopped when the validation performance fails to improve or when a preset threshold for loss or epoch number is met. The final model stability is considerably better when using threshold criteria than when using validation criterion, as the former avoids the randomness associated with selecting a validation subset. Sensitivity experiments show that scaling the model’s initial parameters (i.e., weights) to 0.1 times their original values can mitigate the impact of initialization randomness, thereby markedly improving model stability while also substantially enhancing predictive skill. Furthermore, weight decay and multi-model ensembles, which are two commonly used techniques, can also markedly enhance model stability. From the perspective of this case study, the compression of model initial parameters yields better improvements in stability compared to weight decay, and unlike multi-model ensemble methods that entail substantial increases in computational cost, it serves as a preferable technique for improving model stability.

This study introduced a pre-training method for machine learning-based climate prediction models. The method leverages the advantage of climate events (some theoretical knowledge) to address their limitation (small sample size). It consists of the following steps: generating artificial samples via composite analysis of high and low anomaly events, pre-training predictive models with these samples, and selecting an optimal pre-trained model that most closely matches the observational training set from numerous repeated experiments with only the model’s random number seeds being varied. Sensitivity experiments demonstrate that this pre-training method not only substantially improves predictive skills but also significantly reduces prediction instability. This simple and practical pre-training method is applicable not only to the climate prediction events in this study but also to all climate events for which composite analysis is applicable.

Due to randomness factors in the machine learning model construction process, reproducibility is compromised. This study investigates the impact of randomness on model stability and evaluates techniques for reducing this impact, using the widely adopted shallow neural network (NN) model as a testbed. Randomness in this NN model arises from three events: randomly initializing model parameters, randomly selecting a validation subset, and randomly sampling batches for parameter updates. Among these, batch randomness exerts a significantly weaker impact than the other two factors. In this study, the model training is stopped when the validation performance fails to improve or when a preset threshold for loss or epoch-number is met. The final model stability is significantly better using threshold criteria than using validation criterion, as the former avoids the randomness associated with validation subset. Sensitivity experiments show that scaling the model's initial parameters to 0.1 times their original values can mitigate the impact of initialization randomness, thereby significantly improving model stability while also markedly enhancing predictive skill. Furthermore, weight decay and multi-model ensembles, which are two commonly used techniques, can also significantly enhance model stability. Moreover, the inherent instability of individual sub-models may actually benefit the overall predictive skill of a multi-model ensemble.