Optimal control of networked systems using reinforcement learning

Abstract

The trend of using wireless communication channel in network control system increases a lot, because of its flexibility and mobility. Improving system performance with simple devices, such as low storage capacity sensors and low transmission power channel, is very important to ensure long life time. Hence, there is interest in system communication and controller design to optimize the information used by devices, so as to maintain overall system performance. This thesis explores an approach to co-design of communication and control. First of all, the design of encoder and controller pair for feedback control systems over binary symmetric channels is concerned. An iterative design method based on Q-learning is proposed to obtain a pair of encoder and controller that can optimize a finite-horizon linear quadratic cost function. Three encoder strategies, memoryless encoder, memory encoder and predictive encoder, are considered. The proposed design can be implemented online, and has the potential to provide better performance. Compared with traditional control optimization method, the proposed design method is model-free, only data measured along with the system trajectories is utilized. Simulations are provided to show the effectiveness and the merits of the proposed method. Only finite channel inputs and finite outputs is considered in previous work, while there are some infinite channel output models in practical. Hence, we studies how the generalization to infinite-output channels affected the optimization of the encoder-controller, theoretically and practically, by studying one special type of infinite output channels, namely, Gaussian channel. Since the infinite-channel outputs mainly affect the controller design, we devote to controller design, which are soft controller design, hard controller design and the combination. From above considerations, all the research works are based on iterative design method, which means the encoder is optimized with fixed controller and the controller is optimized with fixed encoder. However, only local optimal solutions can be got by iterative design. Therefore, distributed encoder and controller design is proposed. Both encoder and controller learn independently with their own local information, and both of them can be optimized simultaneously. Obviously, the system performance is better than iterative design. In addition, distributed Qlearning can be applied into complex networked control systems.

More information

Identifiers

ORCID for Christopher Freeman: orcid.org/0000-0003-0305-9246

Catalogue record

Export record