Portrait of Pablo Samuel Castro

Pablo Samuel Castro

Core Industry Member
Adjunct professor, Université de Montréal, Department of Computer Science and Operations Research
Research Scientist, Google DeepMind
Research Topics
Reinforcement Learning
Website
Google Scholar
LinkedIn
X

Biography

Pablo Samuel Castro was born and raised in Quito, Ecuador, and moved to Montréal after high school to study at McGill University. For his PhD, he studied reinforcement learning with Doina Precup and Prakash Panangaden at McGill. Castro has been working at Google for over eleven years. He is currently a staff research scientist at Google DeepMind in Montreal, where he conducts fundamental reinforcement learning research and is a regular advocate for increasing LatinX representation in the research community.

He is also an adjunct professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. In addition to his interest in coding, AI and math, Castro is an active musician.

Current Students

Roger Creus-Castanyer
PhD - Université de Montréal
Principal supervisor :
Glen Berseth
roger.creus-castanyer@mila.quebec
Website
Github
Google Scholar
Nicola Gauthier
Master's Research - Université de Montréal
nicola.gauthier@mila.quebec
Olya Mastikhina
PhD - Université de Montréal
olya.mastikhina@mila.quebec
Walter Mayor
Collaborating researcher
waltermayor@gmail.com
Website
Github
Google Scholar
Johan Samir Obando Ceron
PhD - Université de Montréal
Principal supervisor :
Aaron Courville
johan.ceron@mila.quebec
Website
Github
Google Scholar
Gandharv Patil
PhD - McGill University
Principal supervisor :
Doina Precup
patilgan@mila.quebec
Website
Github
Google Scholar
Ali Saheb Pasand
PhD - McGill University
Principal supervisor :
Blake Richards
ali.sahebpasand@mila.quebec
Dhruv Sreenivas
PhD - Université de Montréal
dhruv.sreenivas@mila.quebec
Website
Github

Publications

Distributional reinforcement learning with linear function approximation
Bellemare Marc-Emmanuel
Nicolas Roux
Pablo Samuel Castro
Subhodeep Moitra
Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited.… (see more) One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cramer distance, but their results only apply to the tabular setting and ignore C51's use of a softmax to produce normalized distributions. In this paper we adapt the Cramer distance to deal with arbitrary vectors. From it we derive a new distributional algorithm which is fully Cramer-based and can be combined to linear function approximation, with formal guarantees in the context of policy evaluation. In allowing the model's prediction to be any real vector, we lose the probabilistic interpretation behind the method, but otherwise maintain the appealing properties of distributional approaches. To the best of our knowledge, ours is the first proof of convergence of a distributional algorithm combined with function approximation. Perhaps surprisingly, our results provide evidence that Cramer-based distributional methods may perform worse than directly approximating the value function.
2019-04-10
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (published)
proceedings.mlr.press
proceedings.mlr.press
Dopamine: A Research Framework for Deep Reinforcement Learning
Pablo Samuel Castro
Subhodeep Moitra
Carles Gelada
Saurabh Kumar
Bellemare Marc-Emmanuel
Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provid… (see more)e stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact and reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.
2018-09-26
ArXiv (preprint)
openreview.net
openreview.net