This paper investigates a novel reinforcement learning (RL)-based quadratic programming (QP) method for the safety-critical trajectory tracking control of autonomous underwater vehicles (AUVs). The proposed approach addresses the substantial challenge posed by model uncertainty, which may hinder the safety and performance of AUVs operating in complex underwater environments. The RL framework can learn the inherent model uncertainties that affect the constraints in Control Barrier Functions (CBFs) and Control Lyapunov Functions (CLFs). These learned uncertainties are subsequently integrated for formulating a novel RL-CBF-CLF Quadratic Programming (RL-CBF-CLF-QP) controller. Corresponding simulations are demonstrated under diverse trajectory tracking scenarios with high levels of model uncertainties. The simulation results show that the proposed RL-CBF-CLF-QP controller can significantly improve the safety and accuracy of the AUV’s tracking performance.