The University of Southampton
University of Southampton Institutional Repository

Optimal control of networked systems using reinforcement learning

Optimal control of networked systems using reinforcement learning
Optimal control of networked systems using reinforcement learning
The trend of using wireless communication channel in network control system increases a lot, because of its flexibility and mobility. Improving system performance with simple devices, such as low storage capacity sensors and low transmission power channel, is very important to ensure long life time. Hence, there is interest in system communication and controller design to optimize the information used by devices, so as to maintain overall system performance. This thesis explores an approach to co-design of communication and control. First of all, the design of encoder and controller pair for feedback control systems over binary symmetric channels is concerned. An iterative design method based on Q-learning is proposed to obtain a pair of encoder and controller that can optimize a finite-horizon linear quadratic cost function. Three encoder strategies, memoryless encoder, memory encoder and predictive encoder, are considered. The proposed design can be implemented online, and has the potential to provide better performance. Compared with traditional control optimization method, the proposed design method is model-free, only data measured along with the system trajectories is utilized. Simulations are provided to show the effectiveness and the merits of the proposed method. Only finite channel inputs and finite outputs is considered in previous work, while there are some infinite channel output models in practical. Hence, we studies how the generalization to infinite-output channels affected the optimization of the encoder-controller, theoretically and practically, by studying one special type of infinite output channels, namely, Gaussian channel. Since the infinite-channel outputs mainly affect the controller design, we devote to controller design, which are soft controller design, hard controller design and the combination. From above considerations, all the research works are based on iterative design method, which means the encoder is optimized with fixed controller and the controller is optimized with fixed encoder. However, only local optimal solutions can be got by iterative design. Therefore, distributed encoder and controller design is proposed. Both encoder and controller learn independently with their own local information, and both of them can be optimized simultaneously. Obviously, the system performance is better than iterative design. In addition, distributed Qlearning can be applied into complex networked control systems.
University of Southampton
Sun, Xiaoru
f023210d-4ef1-4272-a08d-8d47f2d8ba47
Sun, Xiaoru
f023210d-4ef1-4272-a08d-8d47f2d8ba47
Freeman, Christopher
ccdd1272-cdc7-43fb-a1bb-b1ef0bdf5815

Sun, Xiaoru (2019) Optimal control of networked systems using reinforcement learning. Doctoral Thesis, 127pp.

Record type: Thesis (Doctoral)

Abstract

The trend of using wireless communication channel in network control system increases a lot, because of its flexibility and mobility. Improving system performance with simple devices, such as low storage capacity sensors and low transmission power channel, is very important to ensure long life time. Hence, there is interest in system communication and controller design to optimize the information used by devices, so as to maintain overall system performance. This thesis explores an approach to co-design of communication and control. First of all, the design of encoder and controller pair for feedback control systems over binary symmetric channels is concerned. An iterative design method based on Q-learning is proposed to obtain a pair of encoder and controller that can optimize a finite-horizon linear quadratic cost function. Three encoder strategies, memoryless encoder, memory encoder and predictive encoder, are considered. The proposed design can be implemented online, and has the potential to provide better performance. Compared with traditional control optimization method, the proposed design method is model-free, only data measured along with the system trajectories is utilized. Simulations are provided to show the effectiveness and the merits of the proposed method. Only finite channel inputs and finite outputs is considered in previous work, while there are some infinite channel output models in practical. Hence, we studies how the generalization to infinite-output channels affected the optimization of the encoder-controller, theoretically and practically, by studying one special type of infinite output channels, namely, Gaussian channel. Since the infinite-channel outputs mainly affect the controller design, we devote to controller design, which are soft controller design, hard controller design and the combination. From above considerations, all the research works are based on iterative design method, which means the encoder is optimized with fixed controller and the controller is optimized with fixed encoder. However, only local optimal solutions can be got by iterative design. Therefore, distributed encoder and controller design is proposed. Both encoder and controller learn independently with their own local information, and both of them can be optimized simultaneously. Obviously, the system performance is better than iterative design. In addition, distributed Qlearning can be applied into complex networked control systems.

Text
Thesis - Version of Record
Available under License University of Southampton Thesis Licence.
Download (5MB)
Text
PTD
Restricted to Repository staff only

More information

Published date: September 2019

Identifiers

Local EPrints ID: 455659
URI: http://eprints.soton.ac.uk/id/eprint/455659
PURE UUID: 78abb559-e9c3-42f0-bf1f-6e955a2d362f

Catalogue record

Date deposited: 30 Mar 2022 16:37
Last modified: 16 Mar 2024 16:43

Export record

Contributors

Author: Xiaoru Sun
Thesis advisor: Christopher Freeman

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×