Publications

Galaxies on graph neural networks: towards robust synthetic galaxy catalogs with deep generative models
Yesukhei Jagvaral
François Lanusse
Sukhdeep Singh
Rachel Mandelbaum
Siamak Ravanbakhsh
Duncan Campbell
The future astronomical imaging surveys are set to provide precise constraints on cosmological parameters, such as dark energy. However, pro… (see more)-duction of synthetic data for these surveys, to test and validate analysis methods, suffers from a very high computational cost. In particular, generating mock galaxy catalogs at sufficiently large volume and high resolution will soon become computa-tionally unreachable. In this paper, we address this problem with a Deep Generative Model to create robust mock galaxy catalogs that may be used to test and develop the analysis pipelines of future weak lensing surveys. We build our model on a custom built Graph Convolutional Networks, by placing each galaxy on a graph node and then connecting the graphs within each gravitationally bound system. We train our model on a cosmological simulation with realistic galaxy populations to capture the 2D and 3D orientations of galaxies. The samples from the model exhibit comparable statistical properties to those in the simulations. To the best of our knowledge, this is the first instance of a generative model on graphs in an astrophysical/cosmological context.
2022-12-10
ArXiv (preprint)
arxiv.org
arxiv.org
Neural Bandits for Data Mining: Searching for Dangerous Polypharmacy
Alexandre Larouche
Audrey Durand
Richard Khoury
Caroline Sirois
2022-12-09
ArXiv (preprint)
doi.org
arxiv.org
Bayesian Q-learning With Imperfect Expert Demonstrations
Fengdi Che
Xiru Zhu
Doina Precup
David Meger
Gregory Dudek
Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expe… (see more)rt information. We propose a novel algorithm to speed up Q-learning with the help of a limited amount of imperfect expert demonstrations. The algorithm avoids excessive reliance on expert data by relaxing the optimal expert assumption and gradually reducing the usage of uninformative expert data. Experimentally, we evaluate our approach on a sparse-reward chain environment and six more complicated Atari games with delayed rewards. With the proposed methods, we can achieve better results than Deep Q-learning from Demonstrations (Hester et al., 2017) in most environments.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
doi.org
openreview.net
Energy efficiency as a normative account for predictive coding
Shahab Bakhtiari
2022-12-08
Patterns (published)
doi.org
Imitation from Observation With Bootstrapped Contrastive Learning
Medric Sonwa
Johanna Hansen
Eugene Belilovsky
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
doi.org
openreview.net
Implicit Offline Reinforcement Learning via Supervised Learning
Alexandre Piché
Rafael Pardinas
David Vázquez
Igor Mordatch
Christopher Pal
Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset of varied b… (see more)ehaviors. It is as simple as supervised learning and Behavior Cloning (BC) but takes advantage of the return information. On BC tasks, implicit models have been shown to match or outperform explicit ones. Despite the benefits of using implicit models to learn robotic skills via BC, Offline RL via Supervised Learning algorithms have been limited to explicit models. We show how implicit models leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets. Furthermore, we show how closely related our implicit methods are to other popular RL via Supervised Learning algorithms.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
doi.org
openreview.net
Informing the development of an outcome set and banks of items to measure mobility among individuals with acquired brain injury using natural language processing
Rehab Alhasani
Mathieu Godbout
Audrey Durand
Claudine Auger
Anouk Lamontagne
Sara Ahmed
The banks of items of mobility domains represent a first step toward establishing a comprehensive outcome set and a common language of mobil… (see more)ity to develop the ontology. It enables researchers and healthcare professionals to begin exposing the content of mobility measures as a way to assess mobility comprehensively.
2022-12-08
BMC Neurology (published)
doi.org
Learning Successor Feature Representations to Train Robust Policies for Multi-task Learning
Melissa Mozifian
Dieter Fox
David Meger
Fabio Ramos
Animesh Garg
The deep reinforcement learning (RL) framework has shown great promise to tackle sequential decision-making problems, where the agent learns… (see more) to behave optimally through interactions with the environment and receiving rewards. The ability of an RL agent to learn different reward functions concurrently has many benefits, such as the decomposition of task rewards and promoting skill reuse. In this paper, we consider the problem of continuous control for robot manipulation tasks with an explicit representation that promotes skill reuse while learning multiple tasks with similar reward functions. Our approach relies on two key concepts: successor features (SFs), a value function representation that decouples the dynamics of the environment from the rewards, and an actor-critic framework that incorporates the learned SFs representation. SFs form a natural bridge between model-based and model-free RL methods. We first show how to learn a decomposable representation required by SFs as a pre-training stage. The proposed architecture is able to learn decoupled state and reward feature representations for non-linear reward functions. We then evaluate the feasibility of integrating SFs into an actor-critic framework, which is more tailored for tasks solved with deep RL algorithms. The approach is empirically tested on non-trivial continuous control problems with compositional structure built into the reward functions of the tasks.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
openreview.net
openreview.net
Locally Constrained Representations in Reinforcement Learning
Somjit Nath
S Ebrahimi Kahou
The success of Reinforcement Learning (RL) heavily relies on the ability to learn robust representations from the observations of the enviro… (see more)nment. In most cases, the representations learned purely by the reinforcement learning loss can differ vastly across states depending on how the value functions change. However, the representations learned need not be very specific to the task at hand. Relying only on the RL objective may yield representations that vary greatly across successive time steps. In addition, since the RL loss has a changing target, the representations learned would depend on how good the current values/policies are. Thus, disentangling the representations from the main task would allow them to focus not only on the task-specific features but also the environment dynamics. To this end, we propose locally constrained representations, where an auxiliary loss forces the state representations to be predictable by the representations of the neighboring states. This encourages the representations to be driven not only by the value/policy learning but also by an additional loss that constrains the representations from over-fitting to the value loss. We evaluate the proposed method on several known benchmarks and observe strong performance. Especially in continuous control tasks, our experiments show a significant performance improvement.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
doi.org
openreview.net
Rethinking Learning Dynamics in RL using Adversarial Networks
Ramnath Kumar
Tristan Deleu
Yoshua Bengio
Recent years have seen tremendous progress in methods of reinforcement learning. However, most of these approaches have been trained in a st… (see more)raightforward fashion and are generally not robust to adversity, especially in the meta-RL setting. To the best of our knowledge, our work is the first to propose an adversarial training regime for Multi-Task Reinforcement Learning, which requires no manual intervention or domain knowledge of the environments. Our experiments on multiple environments in the Multi-Task Reinforcement learning domain demonstrate that the adversarial process leads to a better exploration of numerous solutions and a deeper understanding of the environment. We also adapt existing measures of causal attribution to draw insights from the skills learned, facilitating easier re-purposing of skills for adaptation to unseen environments and tasks.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
openreview.net
openreview.net
On The Fragility of Learned Reward Functions
Lev E McKinney
Yawen Duan
David M. Krueger
Adam Gleave
Reward functions are notoriously difficult to specify, especially for tasks with complex goals. Reward learning approaches attempt to infer … (see more)reward functions from human feedback and preferences. Prior works on reward learning have mainly focused on the performance of policies trained alongside the reward function. This practice, however, may fail to detect learned rewards that are not capable of training new policies from scratch and thus do not capture the intended behavior. Our work focuses on demonstrating and studying the causes of these relearning failures in the domain of preference-based reward learning. We demonstrate with experiments in tabular and continuous control environments that the severity of relearning failures can be sensitive to changes in reward model design and the trajectory dataset composition. Based on our findings, we emphasize the need for more retraining-based evaluations in the literature.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
doi.org
openreview.net
Training Equilibria in Reinforcement Learning
Lauro Langosco
David M. Krueger
Adam Gleave
In partially observable environments, reinforcement learning algorithms such as policy gradient and Q-learning may have multiple equilibria-… (see more)--policies that are stable under further training---and can converge to equilibria that are strictly suboptimal. Prior work blames insufficient exploration, but suboptimal equilibria can arise despite full exploration and other favorable circumstances like a flexible policy parametrization. We show theoretically that the core problem is that in partially observed environments, an agent's past actions induce a distribution on hidden states. Equipping the policy with memory helps it model the hidden state and leads to convergence to a higher reward equilibrium, \emph{even when there exists a memoryless optimal policy}. Experiments show that policies with insufficient memory tend to learn to use the environment as auxiliary memory, and parameter noise helps policies escape suboptimal equilibria.
2022-12-08
NeurIPS.cc/2022/Workshop/DeepRL (accepted)
openreview.net
openreview.net