High-impact and low-probability extreme events have occurred more frequently than before because of rapid climate change, which can seriously damage distribution systems. However, conventional distribution management can be dysfunctional after an event, destroying its centralized supervision towards resilience enhancement. In this context, networked microgrids (NMGs) with distributed energy resources provide a viable solution for the resilience enhancement of distribution systems. Existing literature tends to employ model-based optimization approaches for resilient operations of NMGs, which require complete system models and can be time-consuming. To address these challenges, this article suggests a decentralized framework for resilience-oriented coordination of NMGs and proposes a novel multi-agent reinforcement learning (MARL) method to solve it. Specifically, the proposed MARL method develops an efficient credit assignment scheme for NMGs to learn their contributions to the distribution system resilience via the Shapley Q-value technique with more efficient resilience enhancement. Case studies based on two modified IEEE 15- and 69-bus distribution networks are conducted to validate the effectiveness of the proposed MARL method in enabling effective coordination among NMGs and providing a high resilience level.
23 December 2022 in IEEE Transactions on Smart Grid
Fast and accurate detection of cyberattacks is a key element for a cyber-resilient power system. Recently, data-driven detectors and physics-based Moving Target Defences (MTD) have been proposed to detect false data injection (FDI) attacks on state estimation. However, the uncontrollable false positive rate of the data-driven detector and the extra cost of frequent MTD usage limit their wide applications. Few works have explored the overlap between these two areas. To fill this gap, this paper proposes blending data-driven and physics-based approaches to enhance the detection performance. To start, a physics-informed data-driven attack detection and identification algorithm is proposed. Then, an MTD protocol is triggered by the positive alarm from the data-driven detector. The MTD is formulated as a bilevel optimisation to robustly guarantee its effectiveness against the worst-case attack around the identified attack vector. Meanwhile, MTD hiddenness is also improved so that the defence cannot be detected by the attacker. To guarantee feasibility and convergence, the convex two-stage reformulation is derived through duality and linear matrix inequality. The simulation results verify that blending data and physics can achieve extremely high detection rate while simultaneously reducing the false positive rate of the data-driven detector and the extra cost of MTD. All codes are available at https://github.com/xuwkk/DDET-MTD .
Reinforcement learning (RL) is a powerful tool for market agents solving decision-making problems in electricity markets. Vanilla RL enables agents to learn optimal policies in dynamic and uncertain market environments via trial and error. However, uncertainties in state transitions are often treated as exogenous state features with statistical errors. This approach can result in policies that are sensitive to perturbations of these uncertainties, potentially leading to performance degradation. This sensitivity is particularly critical in electricity markets, where the penetration of renewable energy and demand variability are increasing. To address this issue, this paper proposes a robust adversarial RL algorithm aimed at learning a robust optimal policy that accounts for market uncertainties in state transitions to systematically mitigate sensitivity to perturbations in uncertain environments. Specifically, we leverage the uncertainty set regularizer technique to define uncertainty sets within the parametric space of state transitions. Furthermore, we introduce a novel adversarial approach to generate unknown uncertainty sets using the value function as a basis. We finally conduct a comprehensive assessment of the robust adversarial RL algorithm across three electricity market applications: strategic bidding, retail pricing, and peer-to-peer energy trading, demonstrating significant improvements in robustness performance against various uncertainties.