Deep Reinforcement Learning Based Approach for Multi-Agent Control of Residential Electric Water Heaters for Distribution Load Management

Abstract

The push towards decarbonization and electrification of the society is leading to increased electricity demand. Many countries, including Canada, are utilizing non-greenhouse gas (GHG) emitting sources and renewable energy sources (RES) to meet this increasing demand. Many of the RES, however, are intermittent and uncertain, and are non-load following sources of electricity. Technologies supporting demand flexibility are being increasingly used to respond to intermittent changes in RES supply and meet the power grid requirements by modifying the energy consumption patterns of residential loads.

The work presented in this thesis discusses the application of electric water heaters (EWHs) as flexible and controllable loads. EWHs, accounting for a significant portion (44%) of water heaters in the Canadian residential sector, and being the second largest consumer of electricity in the household sector (20%), are becoming a viable source for providing load flexibility.

This thesis presents a multi-agent reinforcement learning (MARL) approach to address the energy management problem of EWHs. Two agents, the residential aggregator agent (RAA)- for EWH control and the utility agent (UA)- to represent the role of a utility, are designed to interact with each other and the (reinforcement learning) environment to maximize their respective rewards. A novel control algorithm using a binning process is employed by the RAA to control operations of certain groups of EWHs. The multi-agent deep deterministic policy gradient (MADDPG) algorithm is implemented for this problem and used in training the RAA and UA to follow the optimal policy.

The proposed EWH energy management approach is tested for consumers in Ontario, New Brunswick and Quebec which have varying consumer tariff rates. The results demonstrate the ability of the proposed RAA and UA to control the behaviour of EWHs via price incentive signals, thus providing benefits for the consumers and the utility.