IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.

References in zbMATH (referenced in 9 articles )

Showing results 1 to 9 of 9.
Sorted by year (citations)

  1. Martin, Sergio M.; Wälchli, Daniel; Arampatzis, Georgios; Economides, Athena E.; Karnakov, Petr; Koumoutsakos, Petros: Korali: efficient and scalable software framework for Bayesian uncertainty quantification and stochastic optimization (2022)
  2. Sha, Xingyu; Zhang, Jiaqi; You, Keyou; Zhang, Kaiqing; Başar, Tamer: Fully asynchronous policy evaluation in distributed reinforcement learning over networks (2022)
  3. Dulac-Arnold, Gabriel; Levine, Nir; Mankowitz, Daniel J.; Li, Jerry; Paduraru, Cosmin; Gowal, Sven; Hester, Todd: Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021)
  4. Schlegel, Matthew; Jacobsen, Andrew; Abbas, Zaheer; Patterson, Andrew; White, Adam; White, Martha: General value function networks (2021)
  5. Bachrach, Yoram; Everett, Richard; Hughes, Edward; Lazaridou, Angeliki; Leibo, Joel Z.; Lanctot, Marc; Johanson, Michael; Czarnecki, Wojciech M.; Graepel, Thore: Negotiating team formation using deep reinforcement learning (2020)
  6. Bard, Nolan; Foerster, Jakob N.; Chandar, Sarath; Burch, Neil; Lanctot, Marc; Song, H. Francis; Parisotto, Emilio; Dumoulin, Vincent; Moitra, Subhodeep; Hughes, Edward; Dunning, Iain; Mourad, Shibl; Larochelle, Hugo; Bellemare, Marc G.; Bowling, Michael: The Hanabi challenge: a new frontier for AI research (2020)
  7. Lazaridis, Aristotelis; Fachantidis, Anestis; Vlahavas, Ioannis: Deep reinforcement learning: a state-of-the-art walkthrough (2020)
  8. Zhang, Ji; Tan, Leonard; Tao, Xiaohui; Pham, Thuan; Chen, Bing: Relational intelligence recognition in online social networks -- a survey (2020)
  9. Marzen, Sarah E.: Novelty detection improves performance of reinforcement learners in fluctuating, partially observable environments (2019)