Twin-Delayed Deep Deterministic Policy Gradient Algorithm to Control a Boost Converter in a DC Microgrid

Twin-Delayed Deep Deterministic Policy Gradient Algorithm to Control a Boost Converter in a DC Microgrid Rifqi Firmansyah Muktiadji Department of Electrical and Computer Engineering, King Abdulaziz University, Jeddah 21589, Saudi Arabia http://orcid.org/0000-0002-4843-8621 Makbul A. M. Ramli Department of Electrical and Computer Engineering, King Abdulaziz University, Jeddah 21589, Saudi Arabia Ahmad H. Milyani Department of Electrical and Computer Engineering, King Abdulaziz University, Jeddah 21589, Saudi Arabia Center of Research Excellence in Renewable Energy and Power Systems, K.A.CARE Energy Research and Innovation Center, King Abdulaziz University, Jeddah 21589, Saudi Arabia http://orcid.org/0000-0002-1926-9486

A stable output voltage of a boost converter is vital for the appropriate functioning of connected devices and loads in a DC microgrid. Variations in load demands and source uncertainties can damage equipment and disrupt operations. In this study, a modified twin-delayed deep deterministic policy gradient (TD3) algorithm is proposed to regulate the output voltage of a boost converter in a DC microgrid. TD3 optimizes PI controller gains, which ensure system stability by employing a non-negative, fully connected layer. To achieve optimal gains, multi-deep reinforcement learning agents are trained. The agents utilize the error signal to obtain the desired output voltage. Furthermore, a new reward function used in the TD3 algorithm is introduced. The proposed controller is tested under load variations and input voltage uncertainties. Simulation and experimental results demonstrate that TD3 outperforms PSO, GA, and the conventional PI. TD3 exhibits less steady-state error, reduced overshoots, fast response times, fast recovery times, and a small voltage deviation. These findings confirm TD3’s superiority and its potential application in DC microgrid voltage control. It can be used by engineers and researchers to design DC microgrids.
01 20 2024 433 electronics13020433 State University of Surabaya http://dx.doi.org/10.13039/ No. B/43105/UN38.III.1/LK.04.00/2023 https://creativecommons.org/licenses/by/4.0/ 10.3390/electronics13020433 https://www.mdpi.com/2079-9292/13/2/433 https://www.mdpi.com/2079-9292/13/2/433/pdf Zhang The Voltage Stabilizing Control Strategy of Off-Grid Microgrid Cluster Bus Based on Adaptive Genetic Fuzzy Double Closed-Loop Control J. Electr. Comput. Eng. 2021 2021 5515362 Abbas Optimal Placement and Sizing of Distributed Generation and Capacitor Banks in Distribution Systems Using Water Cycle Algorithm IEEE Syst. J. 2018 10.1109/JSYST.2018.2796847 12 3629 10.3390/electronics11233886 Zishan, F., Akbari, E., Montoya, O.D., Giral-Ramírez, D.A., and Molina-Cabrera, A. (2022). Efficient PID Control Design for Frequency Regulation in an Independent Microgrid Based on the Hybrid PSO-GSA Algorithm. Electronics, 11. Bastos Power-Sharing for Dc Microgrid with Composite Storage Devices and Voltage Restoration without Communication Int. J. Electr. Power Energy Syst. 2022 10.1016/j.ijepes.2021.107928 138 107928 Esmaeili Robust Power Management System with Generation and Demand Prediction and Critical Loads in DC Microgrid J. Clean. Prod. 2023 10.1016/j.jclepro.2022.135490 384 135490 10.1109/ICIMIA48430.2020.9074930 Mahajan, T., and Potdar, M.S. (2020, January 5–7). An Improved Strategy for Distributed Generation Control and Power Sharing in Islanded Microgrid. Proceedings of the 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), Bangalore, India. Badar An Autonomous Hybrid DC Microgrid with ANN-Fuzzy and Adaptive Terminal Sliding Mode Multi-Level Control Structure Control Eng. Pract. 2022 10.1016/j.conengprac.2021.105036 121 105036 Sarangi A Comprehensive Review of Distribution Generation Integrated DC Microgrid Protection: Issues, Strategies, and Future Direction Int. J. Energy Res. 2021 10.1002/er.6245 45 5006 10.3390/en14144308 Ali, S., Zheng, Z., Aillerie, M., Sawicki, J.P., Péra, M.C., and Hissel, D. (2021). A Review of Dc Microgrid Energy Management Systems Dedicated to Residential Applications. Energies, 14. Naik Adaptive Energy Management Strategy for Sustainable Voltage Control of PV-Hydro-Battery Integrated DC Microgrid J. Clean. Prod. 2021 10.1016/j.jclepro.2021.128102 315 128102 Liu Robust Stability Analysis of DC Microgrids With Constant Power Loads IEEE Trans. Power Syst. 2018 10.1109/TPWRS.2017.2697765 33 851 Aluisio Planning and Reliability of DC Microgrid Configurations for Electric Vehicle Supply Infrastructure Int. J. Electr. Power Energy Syst. 2021 10.1016/j.ijepes.2021.107104 131 107104 Elwarraki Intelligent Perturb and Observe Based MPPT Approach Using Multilevel DC-DC Converter to Improve PV Production System J. Electr. Comput. Eng. 2021 2021 6673022 10.3390/electronics12194032 Liu, X., Zhang, Y., Suo, Y., Song, X., and Zhou, J. (2023). Large-Signal Stability Analysis for Islanded DC Microgrids with N+1 Parallel Energy-Storage Converters. Electronics, 12. 10.1109/ISCAS.2018.8351078 Al-Baidhani, H., Kazimierczuk, M.K., and Reatti, A. (2018, January 27–30). Nonlinear Modeling and Voltage-Mode Control of DC-DC Boost Converter for CCM. Proceedings of the IEEE International Symposium on Circuits and Systems, Florence, Italy. Alipour Observer-Based Backstepping Sliding Mode Control Design for Microgrids Feeding a Constant Power Load IEEE Trans. Ind. Electron. 2022 10.1109/TIE.2022.3152028 70 465 Guo Model Predictive Control and Linear Control of DC–DC Boost Converter in Low Voltage DC Microgrid: An Experimental Comparative Study Control Eng. Pract. 2023 10.1016/j.conengprac.2022.105387 131 105387 Borase A Review of PID Control, Tuning Methods and Applications Int. J. Dyn. Control 2020 10.1007/s40435-020-00665-4 9 818 10.1109/ICIAS.2016.7824044 Ibrahim, O., Yahaya, N.Z., and Saad, N. (2016, January 15–17). Comparative Studies of PID Controller Tuning Methods on a DC-DC Boost Converter. Proceedings of the International Conference on Intelligent and Advanced Systems, ICIAS 2016, Kuala Lumpur, Malaysia. 10.3390/electronics12214540 Zehra, S.S., Dolara, A., Amjed, M.A., and Mussetta, M. (2023). Implementation of Nonlinear Controller to Improve DC Microgrid Stability: A Comparative Analysis of Sliding Mode Control Variants. Electronics, 12. Slamet A Robust Maximum Power Point Tracking Control for PV Panel Using Adaptive PI Controller Based on Fuzzy Logic Telkomnika (Telecommun. Comput. Electron. Control) 2020 10.12928/telkomnika.v18i6.17271 18 2999 Hasanien A Taguchi Approach for Optimum Design of Proportional-Integral Controllers in Cascaded Control Scheme IEEE Trans. Power Syst. 2013 10.1109/TPWRS.2012.2224385 28 1636 10.3390/electronics8111249 Li, H., Liu, X., and Lu, J. (2019). Research on Linear Active Disturbance Rejection Control in Dc/Dc Boost Converter. Electronics, 8. 10.3390/math9070712 Gupta, D.K., Soni, A.K., Jha, A.V., Mishra, S.K., Appasani, B., Srinivasulu, A., Bizon, N., and Thounthong, P. (2021). Hybrid Gravitational-Firefly Algorithm-Based Load Frequency Control for Hydrothermal Two-Area System. Mathematics, 9. 10.3390/en13040866 Faisal, S.F., Beig, A.R., and Thomas, S. (2020). Time Domain Particle Swarm Optimization of PI Controllers for Bidirectional VSC HVDC Light System. Energies, 13. Wongkhead Implementation of a Dsp- Tms320f28335 Based State Feedback with Optimal Design of Pi Controller for a Speed of Bldc Motor by Ant Colony Optimization Prz. Elektrotech. 2021 10.15199/48.2021.07.02 97 9 Belgaid Optimal Tuning of PI Controller Using Genetic Algorithm for Wind Turbine Application Indones. J. Electr. Eng. Comput. Sci. 2019 18 167 Darshi Decentralized Reinforcement Learning Approach for Microgrid Energy Management in Stochastic Environment Int. Trans. Electr. Energy Syst. 2023 10.1155/2023/1190103 2023 1190103 Kolodziejczyk Real-Time Energy Purchase Optimization for a Storage-Integrated Photovoltaic System by Deep Reinforcement Learning Control Eng. Pract. 2021 10.1016/j.conengprac.2020.104598 106 104598 Arwa Reinforcement Learning Techniques for Optimal Power Control in Grid-Connected Microgrids: A Comprehensive Review IEEE Access 2020 10.1109/ACCESS.2020.3038735 8 208992 Fu Distributed Economic Droop Control for DC Microgrid Based on Reinforcement Learning Dianli Zidonghua Shebei/Electric Power Autom. Equip. 2021 41 1 Kosaraju Reinforcement Learning Based Distributed Control of Dissipative Networked Systems IEEE Trans. Control Netw. Syst. 2022 10.1109/TCNS.2021.3124896 9 856 Hajihosseini DC/DC Power Converter Control-Based Deep Machine Learning Techniques: Real-Time Implementation IEEE Trans. Power Electron. 2020 10.1109/TPEL.2020.2977765 35 9971 Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 4–9). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2–4). Continuous Control with Deep Reinforcement Learning. Proceedings of the 4th International Conference on Learning Representations, ICLR, San Juan, Puerto Rico. Shamsudin Twin Delayed Deep Deterministic Policy Gradient-Based Target Tracking for Unmanned Aerial Vehicle with Achievement Rewarding and Multistage Training IEEE Access 2022 10.1109/ACCESS.2022.3154388 10 23545 10.3390/en15072392 Nicola, M., Nicola, C.I., and Selișteanu, D. (2022). Improvement of the Control of a Grid Connected Photovoltaic System Based on Synergetic and Sliding Mode Controllers Using a Reinforcement Learning Deep Deterministic Policy Gradient Agent. Energies, 15. Joshi Twin Actor Twin Delayed Deep Deterministic Policy Gradient (TATD3) Learning for Batch Process Control Comput. Chem. Eng. 2021 10.1016/j.compchemeng.2021.107527 155 107527 Muktiadji Control of Boost Converter Using Observer-Based Backstepping Sliding Mode Control for DC Microgrid Front. Energy Res. 2022 10.3389/fenrg.2022.828978 10 8978 10.1109/ICVEE57061.2022.9930441 Muktiadji, R.F., Ramli, M.A.M., Seedahmed, M.M.A., and Uswarman, R. (2022, January 10–11). Endryansyah Power Sharing Control and Voltage Restoration in DC Microgrid Using PI Fuzzy. Proceedings of the 2022 Fifth International Conference on Vocational Education and Electrical Engineering (ICVEE), Surabaya, Indonesia. Ortega Passivity-Based Controllers for the Stabilization of DC-to-DC Power Converters Automatica 1997 10.1016/S0005-1098(96)00207-5 33 499 10.3390/electronics10040493 Chincholkar, S., Jiang, W., Chan, C.Y., and Rangarajan, S.S. (2021). A Simplified Output Feedback Controller for the Dc-dc Boost Power Converter. Electronics, 10. Nguyen Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications IEEE Trans. Cybern. 2020 10.1109/TCYB.2020.2977374 50 3826 10.1145/3387168.3387199 Dankwa, S., and Zheng, W. (2019, January 26–28). Twin-Delayed DDPG: A Deep Reinforcement Learning Technique to Model a Continuous Movement of an Intelligent Robot Agent. Proceedings of the 3rd International Conference on Vision, Image and Signal Processing, Vancouver, BC, Canada.