This article proposesa novel federated reinforcement learning (FRL) approach for the energy management of multiple smart homes with home appliances, a solar photovoltaic system, and an energy storage system. The novelty of the proposed FRL approach lies in the development of a distributed deep reinforcement learning (DRL) model that consists of local home energy management systems (LHEMSs) and a global server (GS). Using energy consumption data, DRL agents for LHEMSs construct and upload their local models to the GS. Then, the GS aggregates the local models to update a global model for LHEMSs and broadcasts it to the DRL agents. Finally, the DRL agents replace the previous local models with the global model and iteratively reconstruct their local models. Simulation results obtained under heterogeneous home environments indicate the advantage of the proposed approach in terms of convergence speed, appliance energy consumption, and number of agents.
This paper presents a data-driven approach that leverages reinforcement learning to manage the optimal energy consumption of a smart home with a rooftop solar photovoltaic system, energy storage system, and smart home appliances. Compared to existing model-based optimization methods for home energy management systems, the novelty of the proposed approach is as follows: (1) a model-free Q-learning method is applied to energy consumption scheduling for an individual controllable home appliance (air conditioner or washing machine), as well as the energy storage system charging and discharging, and (2) the prediction of the indoor temperature using an artificial neural network assists the proposed Q-learning algorithm in learning the relationship between the indoor temperature and energy consumption of the air conditioner accurately. The proposed Q-learning home energy management algorithm, integrated with the artificial neural network model, reduces the consumer electricity bill within the preferred comfort level (such as the indoor temperature) and the appliance operation characteristics. The simulations illustrate a single home with a solar photovoltaic system, an air conditioner, a washing machine, and an energy storage system with the time-of-use pricing. The results show that the relative electricity bill reduction of the proposed algorithm over the existing optimization approach is 14%.
This paper presents a hierarchical deep reinforcement learning (DRL) method for the scheduling of energy consumptions of smart home appliances and distributed energy resources (DERs) including an energy storage system (ESS) and an electric vehicle (EV). Compared to Q-learning algorithms based on a discrete action space, the novelty of the proposed approach is that the energy consumptions of home appliances and DERs are scheduled in a continuous action space using an actor–critic-based DRL method. To this end, a two-level DRL framework is proposed where home appliances are scheduled at the first level according to the consumer’s preferred appliance scheduling and comfort level, while the charging and discharging schedules of ESS and EV are calculated at the second level using the optimal solution from the first level along with the consumer environmental characteristics. A simulation study is performed in a single home with an air conditioner, a washing machine, a rooftop solar photovoltaic system, an ESS, and an EV under a time-of-use pricing. Numerical examples under different weather conditions, weekday/weekend, and driving patterns of the EV confirm the effectiveness of the proposed approach in terms of total cost of electricity, state of energy of the ESS and EV, and consumer preference.
This paper proposes a distributed optimization algorithm for scheduling the energy consumption of multiple smart homes with distributed energy resources. In the proposed approach, the centralized optimization problem for home energy management is decomposed into a two-level optimization problem, corresponding to the local home energy management system (LHEMS) at the first level and the global home energy management system (GHEMS) at the second level. The controllable household appliances (e.g., air conditioner and washing machine) are scheduled in the LHEMS within the consumer's preferred appliance scheduling and comfort level, while the energy storage system and power trading between households are scheduled in the GHEMS. In the simulation study, the proposed distributed algorithm shows almost equivalent performance to the centralized algorithm in terms of the electricity cost and the consumer's comfort level. The impact of different network topologies on the proposed algorithm is also analyzed, and the result provides insight into the selection of the optimal network configuration in view of the consumer's electricity cost saving.
This paper proposes a distributionally robust model predictive control (DRMPC) for energy management of a vehicle-to-grid (V2G)/vehicle-to-vehicle (V2V)-enabled smart electric vehicle charging station (EVCS) with a photovoltaic (PV) system and an energy storage system. The proposed DRMPC method aims to reduce the operational cost of the EVCS while ensuring the desired charging demands of electric vehicle (EV) users under uncertainties in electricity buying/selling prices, PV generation outputs, and future EV charging demands. To cope with these uncertainties, the proposed method includes the following three features: i) tractable reformulation of the worst-case expected buying cost and selling revenue using a Wasserstein metric and duality theory, ii) determination of a distributionally robust bound on the random PV generation output using its support information, and iii) a scenario-based approach to predicting the future EV charging demand. To improve computational efficiency, a penalty method is proposed to relax the complementarity constraints, while still ensuring nonsimultaneous charging and discharging of EVs under the derived sufficient conditions. Numerical examples using a real-world operational dataset of the EVCS are provided to demonstrate the effectiveness of the proposed DRMPC method under uncertain environments in terms of the EVCS cost saving via V2G/V2V capability, data utilization, and computational complexity.