Power markets are becoming increasingly complex as they move towards (i) integrating higher amounts of variable renewable energy, (ii) shorter trading intervals and lead times, (iii) stronger interdependencies between related markets, and (iv) increasing energy system integration. For designing them appropriately, an enhanced understanding of the dynamics in interrelated short-term physical power and energy markets is required, which can be supported by market simulations. In this paper, we present an agent-based power market simulation model with rule-based bidding strategies that addresses the above-mentioned challenges, and represents market participants individually with a high level of technical detail. By allowing agents to participate in several interrelated markets, such as the energy-only market, a procurement platform for control reserve and a local heat market representing district heating systems, cross-market opportunity costs are well reflected. With this approach, we were able to reproduce EPEX SPOT market outcomes for the German bidding zone with a high level of accuracy (mean absolute percentage error of 8 €/MWh for the years 2016–2019). We were also able to model negative market prices at the energy-only market realistically, and observed that the occurrence of negative prices differs among data inputs used. The simulation model provides a useful tool for investigating different short-term physical power/energy market structures and designs in the future. The modular structure also enables extension to further related markets, such as fuel, CO${}_2$, or derivative markets.

Modeling energy storage units realistically is challenging as their decision-making is not governed by a marginal cost pricing strategy but relies on expected electricity prices. Existing electricity market models often use centralized rule-based bidding or global optimization approaches, which may not accurately capture the competitive behavior of market participants. To address this issue, we present a novel method using multi-agent deep reinforcement learning to model individual strategies in electricity market models. We demonstrate the practical applicability of our approach using a detailed model of the German wholesale electricity market with a complete fleet of pumped hydro energy storage units represented as learning agents. We compare the results to widely used modeling approaches and demonstrate that the proposed method performs well and can accurately represent the competitive behavior of market participants. To understand the benefits of using reinforcement learning, we analyze overall profits, aggregated dispatch, and individual behavior of energy storage units. The proposed method can improve the accuracy and realism of electricity market modeling and help policymakers make informed decisions for future market designs and policies.

Modeling electricity markets realistically plays a crucial role for understanding complex emerging market dynamics and guiding policy making. In systems with a high share of variable renewable generation, accurately modeling the behavior of storage units can be particularly challenging, as their bidding strategies depend on expected electricity prices. While there exist a wide variety of electricity market models, they typically rely on rule-based bidding strategies or optimization approaches, which may not be sufficient to represent competitive and strategic behavior on the market. In this paper, we present a multi-agent deep reinforcement learning modeling framework that allows representing competitive and strategic behavior of energy storage units. This framework can be executed in large-scale electricity market models, thus facilitating market design analyses. We show that the proposed approach performs very well when compared with widely used modeling approaches, and its computational efficiency makes its use in energy market modeling practical.