Portrait of Johan Samir Obando Ceron

Johan Samir Obando Ceron

PhD - Université de Montréal
Supervisor
Aaron Courville
Co-supervisor
Pablo Samuel Castro
Research Topics
Deep Learning
Reinforcement Learning
Website
Google Scholar
GitHub

Publications

The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning
Johan Samir Obando Ceron
Marc Gendron-Bellemare
Pablo Samuel Castro
2023-01-01
Tiny Papers @ ICLR (published)
openreview.net
openreview.net
Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning
Johan Samir Obando Ceron
Marc Gendron-Bellemare
Pablo Samuel Castro
In deep reinforcement learning, multi-step learning is almost unavoidable to achieve state-of-the-art performance. However, the increased va… (see more)riance that multistep learning brings makes it difficult to increase the update horizon beyond relatively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning. It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments to better understand what we call the variance doubledown phenomenon.
2022-12-09
NeurIPS.cc/2022/Workshop/DeepRL (unknown)
openreview.net
openreview.net