Bi-level optimization and reinforcement learning (RL) constitute the state-of-the-art frameworks for modeling strategic bidding decisions in deregulated electricity markets. However, the former neglects the market participants’ physical non-convex operating characteristics, while conventional RL methods require discretization of state and / or action spaces and thus suffer from the curse of dimensionality. This paper proposes a novel deep reinforcement learning (DRL) based methodology, combining a deep deterministic policy gradient (DDPG) method with a prioritized experience replay (PER) strategy. This approach sets up the problem in multi-dimensional continuous state and action spaces, enabling market participants to receive accurate feedback regarding the impact of their bidding decisions on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics. Case studies demonstrate that the proposed methodology achieves a significantly higher profit than the alternative state-of-the-art methods, and exhibits a more favourable computational performance than benchmark RL methods due to the employment of the PER strategy.

Multi-energy systems (MES) are attracting increasing attention driven by its potential to offer significant flexibility in future smart grids. At the residential level, the roll-out of smart meters and rapid deployment of smart energy devices call for autonomous multi-energy management systems which can exploit real-time information to optimally schedule the usage of different devices with the aim of minimizing end-users’ energy costs. This paper proposes a novel real-time autonomous energy management strategy for a residential MES using a model-free deep reinforcement learning (DRL) based approach, combining state-of-the-art deep deterministic policy gradient (DDPG) method with an innovative prioritized experience replay strategy. This approach is tailored to align with the nature of the problem by posing it in multi-dimensional continuous state and action spaces, facilitating more cost-effective control strategies to be devised. The superior performance of the proposed approach in reducing end-user’s energy cost while coping with the MES uncertainties is demonstrated by comparing it against state-of-the-art DRL methods as well as conventional stochastic programming and robust optimization methods in numerous case studies in a real-world scenario.

Peer-to-peer (P2P) transactive energy trading has emerged as a promising paradigm towards maximizing the flexibility value of prosumers’ distributed energy resources (DERs). Despite reinforcement learning constitutes a well-suited model-free and data-driven methodological framework to optimize prosumers’ energy management decisions, its application to the large-scale coordinated management and P2P trading among multiple prosumers within an energy community is still challenging, due to the scalability, non-stationarity and privacy limitations of state-of-the-art multi-agent deep reinforcement learning (MADRL) approaches. This paper proposes a novel P2P transactive trading scheme based on the multi-actor-attention-critic (MAAC) algorithm, which addresses the above challenges individually. This method is complemented by a P2P trading platform that incentivizes prosumers to engage in local energy trading while also penalizes each prosumer’s addition to rebound peaks. Case studies involving a real-world, large-scale scenario with 300 residential prosumers demonstrate that the proposed method significantly outperforms the state-of-the-art MADRL methods in reducing the community’s cost and peak demand.

The effective pricing of electric vehicles (EV) charging by aggregators constitutes a key problem towards the realization of the significant EV flexibility potential in deregulated electricity systems, and has been addressed by previous work through bi-level optimization formulations. However, the solution approach adopted in previous work cannot capture the discrete nature of the EV charging / discharging levels. Although reinforcement learning (RL) can tackle this challenge, state-of-the-art RL methods require discretization of state and / or action spaces and thus exhibit limitations in terms of solution optimality and computational requirements. This paper proposes a novel deep reinforcement learning (DRL) method to solve the examined EV pricing problem, combining deep deterministic policy gradient (DDPG) principles with a prioritized experience replay (PER) strategy, and setting up the problem in multi-dimensional continuous state and action spaces. Case studies demonstrate that the proposed method outperforms state-of-the-art RL methods in terms of both solution optimality and computational requirements, and comprehensively analyze the economic impacts of smart-charging and vehicle-to-grid (V2G) flexibility on both aggregators and EV owners.

Local electricity markets (LEM) have recently attracted great interest as an effective solution to the challenging problem of distributed energy resources’ (DER) management. However, LEM designs combining the market functions of local energy trading and flexibility services (FS) provision to wider system operators have not attracted sufficient attention. In the context of addressing this research gap, this paper firstly provides a new model-based system-centric formulation for the coordination of such a LEM, which provides a theoretical optimality benchmark. Compared to previous formulations, it considers the time-coupling operating characteristics of flexible DERs, and optimizes the two market functions simultaneously. Furthermore, this paper explores for the very first time a model-free prosumer-centric coordination approach for such a LEM, in order to address the practical limitations of model-based system-centric approaches. This is achieved through a new multi-agent deep reinforcement learning method which combines the beneficial properties of the multi-actor-attention-critic and the prioritized experience replay approaches. Case studies on a real-world, large-scale setting validate that the proposed LEM design successfully encapsulates the economic benefits of both local energy trading and FS provision functions, and demonstrate that the proposed learning method outperforms previous methods.