Unmanned Aerial Vehicle Autonomous Navigation using Semi-Prior Deep Reinforcement Learning

Abstract

The rapid development of unmanned aerial vehicles (UAV) in recent years has attracted widespread attention from academia and industry. In the current context, the urgent problems that drones need to solve are endurance and autonomous navigation. Autonomous navigation is the main barrier for UAV technology. There is a wealth of research on UAV autonomous navigation and path planning, but most of it focuses on path planning problems in 2D environments, which sacrifices the flexibility of UAV movement in 3D space. Deep reinforcement learning (DRL) algorithm is considered one of the most suitable algorithms for unmanned aerial vehicle path planning problems. It can learn the dynamic changes of the environment, and use deep neural networks to reduce the spatial complexity of 3D environment with geometric growth of environment size.

However, DRL algorithms still faces problems such as low training efficiency, sparse training data, and the tendency to fall into local optimal traps in 3D environment path planning tasks. In order to address the above issues, I propose an cumulated reward model in this thesis to overcome the sparsity of the training data for DRL algorithms, and propose a region segmentation algorithm to reduce the probability of DRL algorithms UAV agents falling into the local optimal trap. Furthermore, I propose the 3D spatial information compression (3DSIC) algorithm, which can compress 3D spatial environment information into 2D, greatly reducing the search space of UAVs in 3D environments, and then extend 3DSIC to unknown partially observable Markov environments, enabling the algorithm to work in more complex application scenarios.

Firstly, I propose a cumulative reward model that considers the distance between UAV and destination, as well as the obstacle density near the next state to construct a reward function, greatly solving the sparsity problem of the effective experience in the experience pool of the DRL algorithm with experience replay. This greatly improves the training efficiency of the UAV agents. Additionally, this reward model can be applied to various DRL algorithms such as Deep Q-Network (DQN) and deep deterministic policy gradient (DDPG). Simulation results show that compared to traditional reward models, the DRL algorithm using cumulative reward models can attain an improved training efficiency by 30.8%. Additionally, I propose a region segmentation algorithm that divides a large region into multiple smaller regions and detects the connectivity between the boundaries of the small regions to establish a connectivity table. Then, based on the connectivity information, the reward value can be re-evaluated to reduce the likelihood of UAV agents falling into local optimal traps. The simulation results show that the region segmentation algorithm can reduce the probability of DQN falling into the local optimal trap by 99% and the probability of DDPG falling into the local optimal trap by 92%.

Then, I propose a 3DSIC algorithm, which can compress a limited number of drone flight altitude layers into a 2D environment. Furthermore, the UAV path planning problem in the 3D environment is transformed into several path planning problems in the 2D environment. The simulation results show that the training efficiency of DQN can be improved by 4.028 times after applying the 3DSIC algorithm. After applying the 3DSIC algorithm, the training efficiency of the DDPG algorithm increased by 3.9 times.

Afterwards, I extend the 3DSIC algorithm to the 3D partially observable Markov decision process (POMDP) environment, so that even if UAVs have no knowledge of environmental information, they can perceive the environment through sensor data and apply the 3DSIC algorithm to compress spatial information. More specifically, I introduce an emulator that not only compresses spatial information, but also performs Monte Carlo simulations in the emulator to obtain mapping probabilities between observations and states. The simulation results show that the extended 3DSIC algorithm can improve the training efficiency of DDPG algorithms by 95.9% in the POMDP environment.

Finally, a dual-directional 3D spatial information compression (DD3DSIC) algorithm is proposed to reduce the problem of collisions that drones are prone to during vertical motion. This method compresses both horizontal and vertical spatial information, providing a more comprehensive assessment of drone movements. The simulation results show that the probability of collision between DD3DSIC algorithm and vertical obstacles is 24.13% lower than the original 3DSIC algorithm.

More information

Identifiers

ORCID for Zhipeng Wang: orcid.org/0009-0004-1940-1047

ORCID for Michael Ng: orcid.org/0000-0002-0930-7194

ORCID for Mohammed El-Hajjar: orcid.org/0000-0002-7987-1401

Catalogue record

Export record