Energy management of buildings with energy storage and solar photovoltaic: A diversity in experience approach for deep reinforcement learning agents

Abstract

Deep reinforcement learning (DRL) is a suitable approach to handle uncertainty in managing the energy consumption of buildings with energy storage systems. Conventionally, DRL agents are trained by randomly selecting samples from a data set, which can result in overexposure to some data categories and under/no exposure to other data categories. Thus, the trained model may be biased towards some data groups and underperform (provide suboptimal results) for data groups to which it was less exposed. To address this issue, diversity in experience-based DRL agent training framework is proposed in this study. This approach ensures the exposure of agents to all types of data. The proposed framework is implemented in two steps. In the first step, raw data are grouped into different clusters using the K-means clustering method. The clustered data is then arranged by stacking the data of one cluster on top of another. In the second step, a selection algorithm is proposed to select data from each cluster to train the DRL agent. The frequency of selection from each cluster is in proportion to the number of data points in that cluster and therefore named the proportional selection method. To analyze the performance of the proposed approach and compare the results with the conventional random selection method, two indices are proposed in this study: the flatness index and the divergence index. The model is trained using different data sets (1-year, 3-year, and 5-year) and also with the inclusion of solar photovoltaics. The simulation results confirmed the superior performance of the proposed approach to flatten the building's load curve by optimally operating the energy storage system.