Portrait of Pierluca D'Oro

Pierluca D'Oro

Affiliate Member
Research Scientist, Meta
Research Topics
AGI (Artificial General Intelligence)
Human-Centered AI
Large Language Models (LLM)
Reinforcement Learning
Website
Google Scholar
LinkedIn
X

Publications

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Nathan Rahn
Pierluca D'Oro
Harley Wiltzer
Pierre-Luc Bacon
Marc Gendron-Bellemare
Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In th… (see more)is work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.
openreview.net
Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
Pierluca D'Oro
Max Schwarzer
Evgenii Nikishin
Pierre-Luc Bacon
Marc Gendron-Bellemare
Aaron Courville
Increasing the replay ratio, the number of updates of an agent's parameters per environment interaction, is an appealing strategy for improv… (see more)ing the sample efficiency of deep reinforcement learning algorithms. In this work, we show that fully or partially resetting the parameters of deep reinforcement learning agents causes better replay ratio scaling capabilities to emerge. We push the limits of the sample efficiency of carefully-modified algorithms by training them using an order of magnitude more updates than usual, significantly improving their performance in the Atari 100k and DeepMind Control Suite benchmarks. We then provide an analysis of the design choices required for favorable replay ratio scaling to be possible and discuss inherent limits and tradeoffs.
2023-02-01
ICLR.cc/2023/Conference (notable)
openreview.net
openreview.net
The Primacy Bias in Deep Reinforcement Learning
Evgenii Nikishin
Max Schwarzer
Pierluca D'Oro
Pierre-Luc Bacon
Aaron Courville
2022-06-28
Proceedings of the 39th International Conference on Machine Learning (published)
doi.org
arxiv.org