Beyond the rainbow: high performance deep reinforcement learning on a desktop PC
Beyond the rainbow: high performance deep reinforcement learning on a desktop PC
Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent’s performance. In this paper, we present 'Beyond The Rainbow' (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures. Code is available at https://github.com/VIPTankz/BTR.
Reinforcement Learning, Atari, Control, Vision, deep learning (DL)
Clark, Tyler
cfbf31c0-1d37-4736-9d55-67e71fd3c861
Towers, Mark
18e6acc7-29c4-4d0c-9058-32d180ad4f12
Evers, Christine
93090c84-e984-4cc3-9363-fbf3f3639c4b
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Clark, Tyler
cfbf31c0-1d37-4736-9d55-67e71fd3c861
Towers, Mark
18e6acc7-29c4-4d0c-9058-32d180ad4f12
Evers, Christine
93090c84-e984-4cc3-9363-fbf3f3639c4b
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Clark, Tyler, Towers, Mark, Evers, Christine and Hare, Jonathon
(2025)
Beyond the rainbow: high performance deep reinforcement learning on a desktop PC.
International Conference on Machine Learning 2025, Vancouver, Canada, Vancouver, Canada.
11 - 19 Jul 2025.
28 pp
.
(In Press)
Record type:
Conference or Workshop Item
(Paper)
Abstract
Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent’s performance. In this paper, we present 'Beyond The Rainbow' (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures. Code is available at https://github.com/VIPTankz/BTR.
Text
Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
- Accepted Manuscript
More information
Accepted/In Press date: 1 May 2025
Venue - Dates:
International Conference on Machine Learning 2025, Vancouver, Canada, Vancouver, Canada, 2025-07-11 - 2025-07-19
Keywords:
Reinforcement Learning, Atari, Control, Vision, deep learning (DL)
Identifiers
Local EPrints ID: 502630
URI: http://eprints.soton.ac.uk/id/eprint/502630
PURE UUID: f6bfb37a-0b85-480e-9c1c-a26c166b4938
Catalogue record
Date deposited: 02 Jul 2025 16:39
Last modified: 22 Aug 2025 02:31
Export record
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
