Alexandre Piché

Implicit Offline Reinforcement Learning via Supervised Learning

Rafael Pardinas

David Vazquez

Igor Mordatch

Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset of varied b… (see more)ehaviors. It is as simple as supervised learning and Behavior Cloning (BC) but takes advantage of the return information. On BC tasks, implicit models have been shown to match or outperform explicit ones. Despite the benefits of using implicit models to learn robotic skills via BC, Offline RL via Supervised Learning algorithms have been limited to explicit models. We show how implicit models leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets. Furthermore, we show how closely related our implicit methods are to other popular RL via Supervised Learning algorithms.

2022-12-09

NeurIPS.cc/2022/Workshop/DeepRL (unknown)

doi.org

openreview.net

Unsupervised Model-based Pre-training for Data-efficient Reinforcement Learning from Pixels

Sai Rajeswar

Pietro Mazzaglia

Tim Verbelen

Alexandre Piché

Bart Dhoedt

Aaron Courville

Alexandre Lacoste

Reinforcement learning (RL) aims at autonomously performing complex tasks. To this end, a reward signal is used to steer the learning proces… (see more)s. While successful in many circumstances, the approach is typically data hungry, requiring large amounts of task-specific interaction between agent and environment to learn efficient behaviors. To alleviate this, unsupervised RL proposes to collect data through self-supervised interaction to accelerate task-specific adaptation. However, whether current unsupervised strategies lead to improved generalization capabilities is still unclear, more so when the input observations are high-dimensional. In this work, we advance the field by closing the performance gap in the Unsupervised RL Benchmark, a collection of tasks to be solved in a data-efficient manner, after interacting with the environment in a self-supervised way. Our approach uses unsupervised exploration for collecting experience to pre-train a world model. Then, when fine-tuning for downstream tasks, the agent leverages the learned model and a hybrid planner to efficiently adapt for the given tasks, achieving comparable results to task-specific base-lines, while using 20x less data. We extensively evaluate our work, comparing several exploration methods and improving the fine-tuning process by studying the interactions between the learned components. Furthermore, we investigate the limitations of the pre-trained agent, gaining insights into how these influence the decision process and shedding light on new research directions.

2022-06-14

ICML.cc/2022/Workshop/DARL (accepted)

openreview.net

A Probabilistic Perspective on Reinforcement Learning via Supervised Learning

Alexandre Piché

Rafael Pardinas

David Vazquez

Chris Pal

2022-04-27

ICLR.cc/2022/Workshop/GPL (poster)

openreview.net

Towards good validation metrics for generative models in offline model-based optimisation

Christopher Beckham

Alexandre Piché

David Vazquez

Chris Pal

In this work we propose a principled evaluation framework for model-based optimisation to measure how well a generative model can extrapolat… (see more)e. We achieve this by interpreting the training and validation splits as draws from their respective ‘truncated’ ground truth distributions, where examples in the validation set contain scores much larger than those in the training set. Model selection is performed on the validation set for some prescribed validation metric. A major research question however is in determining what validation metric correlates best with the expected value of generated candidates with respect to the ground truth oracle; work towards answering this question can translate to large economic gains since it is expensive to evaluate the ground truth oracle in the real world. We compare various validation metrics for generative adversarial networks using our framework. We also discuss limitations with our framework with respect to existing datasets and how progress can be made to mitigate them. 1

2022-01-01

arXiv.org (preprint)

doi.org

Probabilistic Planning with Sequential Monte Carlo methods

Alexandre Piché

Valentin Thomas

Cyril Ibrahim

Yoshua Bengio

Chris Pal

2018-09-27

International Conference on Learning Representations (published)

openreview.net