With the increasing integration of renewable energy sources (RES), the electrical grid is facing an amplified uncertainty in the energy supply. Transmission System Operators (TSOs) are offering remuneration in exchange for energy exchanges that reduce system imbalances. Helping stabilize the grid frequency is hence an economically viable endeavor, but it requires strategies that can properly manage stochasticities. To tackle this, we analyze the participation of grid-scale batteries in Frequency Containment Reserve (FCR) using a Reinforcement Learning (RL) control strategy. Acting in a multi-market scenario, the RL agent learns to effectively leverage imbalance prices for a high-quality energy recovery strategy. We trained the agent to maximize the imbalance settlement profit while ensuring conforming participation in the FCR service. The agent is also trained to keep the battery yearly cycles below a planned value. In our simulations, we demonstrated the efficacy of RL when dealing with different FCR participation magnitudes, obtaining an average improvement of + 9% in profit compared to a rule-based controller baseline.
The growing share of Renewable Energy Sources (RES) in modern power systems increases both grid imbalances and frequency deviations, reinforcing the need for ancillary services such as Frequency Containment Reserve (FCR) and passive balancing. Battery Energy Storage Systems (BESS) are well-suited for these services, but prior research typically relies on uniform FCR bids that remain constant throughout the control period. Such static bids fail to fully exploit BESS flexibility, as they do not balance the trade-off between reserving energy for FCR delivery and using it for imbalance arbitrage, limiting the achievable value in value-stacking settings. To address this limitation, we propose a two-stage control framework for the European context that introduces non-uniform FCR bids. In the first stage, we derive a time-varying bid sequence using data-driven Monte Carlo (MC) optimization. In the second stage, a Deep Reinforcement Learning (DRL) agent leverages the residual flexibility for real-time imbalance trading while proactively managing the State of Energy (SoE) to ensure compliance with FCR requirements. The framework is presented as a proof of concept, highlighting the potential benefits of time-varying bidding strategies. By incorporating daily cycle budgets and time-varying reserve commitments, our approach achieves a 7.56% profit increase compared to uniform baselines. These results show that non-uniform bidding can unlock additional value by more effectively aligning reserve obligations with rapidly changing imbalance opportunities.