Iterative learning control: from model-based to reinforcement learning

Abstract

High-performance control systems that repeat the same tasks have a wide range of applications. Iterative learning control (ILC) is a control method that enables such systems to achieve high-performance tracking by updating the input based on historical data. Based on whether a system model is required or not, ILC algorithms can be divided into model-based and model-free algorithms. Model-based ILC uses system dynamics (though not necessarily accurate) to update the inputs, whereas model-free ILC only uses input-output data to update. However, model-free ILC techniques typically converge slower than model-based ILC techniques, despite retaining high performance. It is noteworthy that the idea of adapting actions based on past information is also at the core of reinforcement learning (RL). More excitingly, RL provides a number of model-free methods for determining an optimal action. Despite being in different subject areas, RL and ILC have many similarities. This motivates the author to present the first in-depth study of the relationship between ILC and RL from the viewpoint of high-performance tracking problems and propose several novel model-free ILC designs. This thesis starts with a quantitative comparison of ILC and RL techniques in both model-based and model-free scenarios from a control perspective. ILC is shown to be more data-efficient when model information is unavailable, which suggests that research into the structure of the problem may lead to improved performance and opens the door for the development of RL-based ILC (RILC) algorithms. Policy gradient methods and Q-learning from reinforcement learning are used to develop new model-free ILC algorithms. The proposed algorithms achieve high-performance tracking without any model information. Their convergence performance is comparable to their model-based counterparts under certain conditions. Moreover, these algorithms are shown to have the potential to handle nonlinear dynamics. Numerical simulations are presented to demonstrate the effectiveness of the proposed design.

More information

Identifiers

ORCID for Yueqing Zhang: orcid.org/0000-0003-2304-6151

ORCID for Bing Chu: orcid.org/0000-0002-2711-8717

Catalogue record

Export record