This research focuses on developing models to accurately predict student’s cumulative grade point average (CGPA) in the early stages of their study to tackle the problem of dropout rates in educational institutions. The state-of-the-art methods address CGPA prediction as a classification problem, providing only an approximate prediction where precise prediction is essential. In this research, six regression models, namely linear regression, support vector regression (SVR), decision tree (DT), random forest (RF), lasso regression (LR), and ridge regression (RR) are developed without optimization and later fine-tuned using Bayesian optimization (BO) and GridSearchCV. BO efficiently searches the hyper-parameter space using probabilistic distribution’s function, whereas GridSearchCV exhaustively searches the hyper-parameter space. These techniques significantly improved the model's performance; SVR achieved an R² score of 94.11% through BO. Ensemble techniques, such as stacking, voting, and boosting, can further enhance the predictive capability of the model. The stacking ensemble model achieved the highest R² score of 94.45%, providing a 0.50% improvement in the R2 score. The findings of this study suggest that advanced optimization and ensemble techniques can substantially enhance the predictive capability of the model, thus enabling institutions to support students at risk of academic probation proactively.