Preventive maintenance (PM) activities in battery energy storage systems (BESSs) aim to achieve a better status in long-term operation. In this article, we develop a reinforcement learning-based PM method for the optimal PM management of BESSs equipped with prognostics and health management capabilities. A multilevel PM framework is established to generate a PM action strategy considering costs, capacity, and reliability simultaneously. Finite costs are the constraints, and reliability is the objective according to capacity degradation, respectively. The proposed PM agent with an integrated Monte Carlo tree search and a deep neural network (DNN) utilizes the state-of-health information of large-scale batteries in the BESS and selects optimal maintenance actions. The DNN is used as the state-action value function to extend the ability to address PM problems with large state-action spaces. The case of a BESS with 9 × 12 × 4 batteries in a fleet is simulated via Python. The results show that the PM agent can achieve efficient and steady decision-making proficiency in BESS PM management.