David Meger

Valliappan Chidambaram Adaikkappan

Doctorat - McGill

Google Scholar

Wesley Chung

Doctorat - McGill

Co-superviseur⋅e :

Doina Precup

Farnoosh Faraji

Doctorat - McGill

Co-superviseur⋅e :

Maîtrise recherche - McGill

Co-superviseur⋅e :

Hsiu-Chin Lin

Zina Kamel

Maîtrise recherche - McGill

Co-superviseur⋅e :

Hsiu-Chin Lin

Sahand Rezaei-Shoshtari

Doctorat - McGill

Superviseur⋅e principal⋅e :

Doctorat - McGill

Maîtrise recherche - McGill

Steven Wang

Maîtrise recherche - McGill

Harley Wiltzer

Doctorat - McGill

Co-superviseur⋅e :

Marc Gendron-Bellemare

Doctorat - McGill

Publications

Tractable Representations for Convergent Approximation of Distributional HJB Equations

Julie Alhosh

Harley Wiltzer

2025-02-28

arXiv (publié)

Programmable Shape‐Preserving Soft Robotics Arm via Multimodal Multistability (Adv. Funct. Mater. 6/2025)

Benyamin Shahryari

Hossein Mofatteh

Arian Sargazi

Armin Mirabolghasemi

Abdolhamid Akbarzadeh

2025-01-31

Advanced Functional Materials (publié)

Learning active tactile perception through belief-space control

Jean-François Tremblay

Johanna Hansen

Francois Hogan

Robot operating in an open world can encounter novel objects with unknown physical properties, such as mass, friction, or size. It is desira… (voir plus)ble to be able to sense those property through contact-rich interaction, before performing downstream tasks with the objects. We propose a method for autonomously learning active tactile perception policies, by learning a generative world model leveraging a differentiable bayesian filtering algorithm, and designing an information- gathering model predictive controller. We test the method on three simulated tasks: mass estimation, height estimation and toppling height estimation. Our method is able to discover policies which gather information about the desired property in an intuitive manner.

2024-12-31

ICRA (publié)

Fairness in Reinforcement Learning with Bisimulation Metrics

Sahand Rezaei-Shoshtari

Ensuring long-term fairness is crucial when developing automated decision making systems, specifically in dynamic and sequential environment… (voir plus)s. By maximizing their reward without consideration of fairness, AI agents can introduce disparities in their treatment of groups or individuals. In this paper, we establish the connection between bisimulation metrics and group fairness in reinforcement learning. We propose a novel approach that leverages bisimulation metrics to learn reward functions and observation dynamics, ensuring that learners treat groups fairly while reflecting the original problem. We demonstrate the effectiveness of our method in addressing disparities in sequential decision making problems through empirical evaluation on a standard fairness benchmark consisting of lending and college admission scenarios.

2024-12-21

ArXiv (prépublication)

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

Harley Wiltzer

Bellemare Marc-Emmanuel

Patrick Shafto

Yash Jhaveri

When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In… (voir plus) turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents are sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the superiority as a probabilistic generalization of the advantage -- the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

2024-09-24

NeurIPS.cc/2024/Conference (poster)

Parseval Regularization for Continual Reinforcement Learning

Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequenc… (voir plus)es of tasks -- all referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.

2024-09-24

NeurIPS.cc/2024/Conference (poster)

Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models

Lucas Berry

Axel Brando

Generative diffusion models, notable for their large parameter count (exceeding 100 million) and operation within high-dimensional image spa… (voir plus)ces, pose significant challenges for traditional uncertainty estimation methods due to computational demands. In this work, we introduce an innovative framework, Diffusion Ensembles for Capturing Uncertainty (DECU), designed for estimating epistemic uncertainty for diffusion models. The DECU framework introduces a novel method that efficiently trains ensembles of conditional diffusion models by incorporating a static set of pre-trained parameters, drastically reducing the computational burden and the number of parameters that require training. Additionally, DECU employs Pairwise-Distance Estimators (PaiDEs) to accurately measure epistemic uncertainty by evaluating the mutual information between model outputs and weights in high-dimensional spaces. The effectiveness of this framework is demonstrated through experiments on the ImageNet dataset, highlighting its capability to capture epistemic uncertainty, specifically in under-sampled image classes.

2024-09-11

Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence (publié)

proceedings.mlr.press

Programmable Shape‐Preserving Soft Robotics Arm via Multimodal Multistability

Benyamin Shahryari

Hossein Mofatteh

Arian Sargazi

Armin Mirabolghasemi

Abdolhamid Akbarzadeh

Inflatable multistable materials have significantly advanced the design of shape‐preserving soft robotic arms, offering substantial benefi… (voir plus)ts in terms of shape adaptability, energy efficiency, and safety, ensuring operational reliability even in the event of sudden power loss. However, existing strategies for realizing multistable arms often limit themselves to a single mode of multistability, commonly with rotationally symmetric designs favoring extension stability and asymmetric designs inducing bending stability. To address the limitation, this study introduces a pioneering platform termed multimodal multistability that utilizes geometrical frustration. A single cylindrical symmetric cell, designed for extension bistability, could achieve frustrated multistable states in bending by controlling the cell with multiple degrees of freedom incorporated pneumatic actuator. This platform extends the spectrum of attainable stable trajectories while preserving essential attributes of arms, such as load‐bearability, programmability, and reversibility of shape changes. Leveraging a pneumatic system with four degrees of freedom for pressure control, not only enables capturing previously unexplored stable configurations in mechanical metastructures but also allows for the control of their deformation modes. With applications spanning space exploration, medical instruments, and rescue missions, the multimodal multistability promises unparalleled flexibility and efficiency in the design and operation of soft robots.

2024-08-28

Advanced Functional Materials (publié)

Imitation Learning from Observation through Optimal Transport

Wei-Di Chang

Scott Fujimoto

2024-05-13

rl-conference.cc/RLC/2024/Conference (publié)

Constrained Robotic Navigation on Preferred Terrains Using LLMs and Speech Instruction: Exploiting the Power of Adverbs

Faraz Lotfi

Farnoosh Faraji

Nikhil Kakodkar

Travis Manderson

2024-04-01

ArXiv (prépublication)

Policy Gradient Methods in the Presence of Symmetries and State Abstractions

Prakash Panangaden

Sahand Rezaei-Shoshtari

Rosie Zhao

Doina Precup

Reinforcement learning (RL) on high-dimensional and complex problems relies on abstraction for improved efficiency and generalization. In th… (voir plus)is paper, we study abstraction in the continuous-control setting, and extend the definition of Markov decision process (MDP) homomorphisms to the setting of continuous state and action spaces. We derive a policy gradient theorem on the abstract MDP for both stochastic and deterministic policies. Our policy gradient results allow for leveraging approximate symmetries of the environment for policy optimization. Based on these theorems, we propose a family of actor-critic algorithms that are able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. Finally, we introduce a series of environments with continuous symmetries to further demonstrate the ability of our algorithm for action abstraction in the presence of such symmetries. We demonstrate the effectiveness of our method on our environments, as well as on challenging visual control tasks from the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance, and the visualizations of the latent space clearly demonstrate the structure of the learned abstraction.

2023-12-31

J. Mach. Learn. Res. (publié)

Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model

Faraz Lotfi

Khalil Virji

Farnoosh Faraji

Lucas Berry

Andrew Holliday

In this paper, we investigate a hybrid scheme that combines nonlinear model predictive control (MPC) and model-based reinforcement learning … (voir plus)(RL) for navigation planning of an autonomous model car across offroad, unstructured terrains without relying on predefined maps. Our innovative approach takes inspiration from BADGR, an LSTM-based network that primarily concentrates on environment modeling, but distinguishes itself by substituting LSTM modules with transformers to greatly elevate the performance our model. Addressing uncertainty within the system, we train an ensemble of predictive models and estimate the mutual information between model weights and outputs, facilitating dynamic horizon planning through the introduction of variable speeds. Further enhancing our methodology, we incorporate a nonlinear MPC controller that accounts for the intricacies of the vehicle's model and states. The model-based RL facet produces steering angles and quantifies inherent uncertainty. At the same time, the nonlinear MPC suggests optimal throttle settings, striking a balance between goal attainment speed and managing model uncertainty influenced by velocity. In the conducted studies, our approach excels over the existing baseline by consistently achieving higher metric values in predicting future events and seamlessly integrating the vehicle's kinematic model for enhanced decision-making. The code and the evaluation data are available at https://github.com/FARAZLOTFI/offroad_autonomous_navigation/).

2023-12-31

ICRA (publié)