Pablo Samuel Castro

Google Scholar

Biography

Pablo Samuel Castro was born and raised in Quito, Ecuador, and moved to Montréal after high school to study at McGill University. For his PhD, he studied reinforcement learning with Doina Precup and Prakash Panangaden at McGill. Castro has been working at Google for over eleven years. He is currently a staff research scientist at Google DeepMind in Montreal, where he conducts fundamental reinforcement learning research and is a regular advocate for increasing LatinX representation in the research community.

He is also an adjunct professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. In addition to his interest in coding, AI and math, Castro is an active musician.

Current Students

Roger Creus-Castanyer

PhD - Université de Montréal

Principal supervisor :

Independent visiting researcher - RWTH Aachen University

Collaborating Alumni - Université de Montréal

Nicola Gauthier

Master's Research - Université de Montréal

Olya Mastikhina

PhD - Université de Montréal

Walter Mayor

Collaborating researcher

Github

Google Scholar

Johan Samir Obando Ceron

PhD - Université de Montréal

Principal supervisor :

PhD - McGill University

Principal supervisor :

PhD - McGill University

Principal supervisor :

Blake Richards

Dhruv Sreenivas

PhD - Université de Montréal

RLiable: Towards Reliable Evaluation & Reporting in Reinforcement Learning

Github

Blog Posts

May 12, 2022

Rishabh Agarwal

Pablo Samuel Castro

Read the article

Publications

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Rishabh Agarwal

Marlos C. Machado

Bellemare Marc-Emmanuel

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generali… (see more)zation, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite.

2020-12-31

ICLR (published)

Deep Reinforcement Learning at the Edge of the Statistical Precipice

Bellemare Marc-Emmanuel

Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. M… (see more)ost published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.

2020-12-31

Advances in Neural Information Processing Systems 34 (NeurIPS 2021) (published)

MICo: Improved representations via sampling-based state similarity for Markov decision processes

Tyler Kastner

Prakash Panangaden

Mark Rowland

We present a new behavioural distance over the state space of a Markov decision process, and demonstrate the use of this distance as an effe… (see more)ctive means of shaping the learnt representations of deep reinforcement learning agents. While existing notions of state similarity are typically difficult to learn at scale due to high computational cost and lack of sample-based algorithms, our newly-proposed distance addresses both of these issues. In addition to providing detailed theoretical analyses, we provide empirical evidence that learning this distance alongside the value function yields structured and informative representations, including strong results on the Arcade Learning Environment benchmark.

2020-12-31

Advances in Neural Information Processing Systems 34 (NeurIPS 2021) (published)

MICo: Learning improved representations via sampling-based state similarity for Markov decision processes

Tyler Kastner

Prakash Panangaden

Mark Rowland

We present a new behavioural distance over the state space of a Markov decision process, and demonstrate the use of this distance as an eﬀ… (see more)ective means of shaping the learnt representations of deep reinforcement learning agents. While existing notions of state similarity are typically diﬃcult to learn at scale due to high computational cost and lack of sample-based algorithms, our newly-proposed distance addresses both of these issues. In addition to providing detailed theoretical analysis

2020-12-31

arXiv.org (preprint)

dblp.uni-trier.de

Autonomous navigation of stratospheric balloons using reinforcement learning

Bellemare Marc-Emmanuel

S. Candido

Jun Gong

Marlos C. Machado

Subhodeep Moitra

Sameera S. Ponda

Ziyun Wang

2020-11-30

Nature (published)

An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

Felipe Petroski Such

Vashisht Madhavan

Rosanne Liu

Rui Wang

Yulun Li

Jiale Zhi

Ludwig Schubert

Bellemare Marc-Emmanuel

Jeff Clune

Joel Lehman

Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the … (see more)Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.

2019-08-09

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (published)

arxiv.org

A Comparative Analysis of Expected and Distributional Reinforcement Learning

Clare Lyle

Bellemare Marc-Emmanuel

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results re… (see more)lative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the reasons behind the improvements distributional RL provides. In this paper we begin the investigation into this fundamental question by analyzing the differences in the tabular, linear approximation, and non-linear approximation settings. We prove that in many realizations of the tabular and linear approximation settings, distributional RL behaves exactly the same as expected RL. In cases where the two methods behave differently, distributional RL can in fact hurt performance when it does not induce identical behaviour. We then continue with an empirical analysis comparing distributional and expected RL methods in control settings with non-linear approximators to tease apart where the improvements from distributional RL methods are coming from.

2019-07-16

Proceedings of the AAAI Conference on Artificial Intelligence (published)

arxiv.org

Distributional reinforcement learning with linear function approximation

Bellemare Marc-Emmanuel

Nicolas Roux

Subhodeep Moitra

Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited.… (see more) One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cramer distance, but their results only apply to the tabular setting and ignore C51's use of a softmax to produce normalized distributions. In this paper we adapt the Cramer distance to deal with arbitrary vectors. From it we derive a new distributional algorithm which is fully Cramer-based and can be combined to linear function approximation, with formal guarantees in the context of policy evaluation. In allowing the model's prediction to be any real vector, we lose the probabilistic interpretation behind the method, but otherwise maintain the appealing properties of distributional approaches. To the best of our knowledge, ours is the first proof of convergence of a distributional algorithm combined with function approximation. Perhaps surprisingly, our results provide evidence that Cramer-based distributional methods may perform worse than directly approximating the value function.

2019-04-10

Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

A Geometric Perspective on Optimal Representations for Reinforcement Learning

Bellemare Marc-Emmanuel

Will Dabney

Robert Dadashi

Adrien Ali Taiga

Nicolas Roux

Dale Schuurmans

Tor Lattimore

Clare Lyle

2018-12-31

NeurIPS (published)

Dopamine: A Research Framework for Deep Reinforcement Learning

Subhodeep Moitra

Carles Gelada

Saurabh Kumar

Bellemare Marc-Emmanuel

Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provid… (see more)e stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact and reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.

2018-09-26

ArXiv (preprint)