Hsiu-Chin Lin

Doctorat - McGill

Co-superviseur⋅e :

elham.daneshmand@mila.quebec

Site web

Mohamad H. Danesh

Doctorat - McGill

mohamad.danesh@mila.quebec

Maîtrise recherche - McGill

Superviseur⋅e principal⋅e :

David Meger

anas.houssaini@mila.quebec

Github

Publications

SafeExplorer: An Unbiased Mixed-Policy Gradient for Reinforcement Learning with Recovery Interventions

Majid Khadiv

Training agents directly on the task they will be deployed on often leads to the best policy performance, but training on physical robots ca… (voir plus)n cause damage through falls, so minimizing training falls is critical. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region. The resulting mixed-policy rollouts bias the on-policy update, and importance sampling cannot remove this bias when the recovery policy is deterministic. We propose SafeExplorer, a drop-in modification of proximal policy optimization (PPO) built on an unbiased policy-gradient estimator of the mixed-policy return. The estimator uses the score function only inside the safe region and never evaluates the recovery policy's density, so deterministic recovery policies pose no problem. On final reward, the estimator matches or outperforms truncated importance sampling, which is itself biased for that same gradient. SafeExplorer also speeds up learning near the safe-region boundary with a closed-form critic target at recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only from successful recoveries. On three simulated locomotion environments with ten seeds each, and given an already available recovery controller, SafeExplorer incurs

2026-10-17

Transactions on Machine Learning Research (accepté)

CopyQuad: Quadrupedal Skill Transfer from a Single Side-View Video Without a Depth Model

Mohamad H. Danesh

Raina Hu

Mengxue Xu

Video-to-robot pipelines for quadrupedal imitation insert a learned lifting stage, either a monocular depth model or a per-clip 3D reconstru… (voir plus)ction, to recover the one axis a side-view camera cannot see. We present CopyQuad, which omits that stage and pins the missing axis to the robot's own stance width instead. The setting is side-view footage of sagittally dominated motion, where limb motion is almost entirely in the image plane. A frozen 2D keypoint detector and a fixed analytic retargeting then turn a single uncalibrated clip into a robot reference whose foot targets are reachable by construction, so no per-clip trajectory optimization is needed. One phase-indexed reward structure, whose only per-behavior freedom is three scalars from a shared sweep, learns a dog walk, a horse walk, a horse trot and two non-periodic gestures, all deployed zero-shot on hardware. We then measure what omitting the axis costs. In a training-free rigidity test, a video depth model varies the distance between the animal's hips by

2026-10-11

arXiv (prépublication)

Imitation Learning via Discrete and Rhythmic Primitive Composition

Rafid Saif

Auke Ijspeert

Long-horizon robotic manipulation is inherently composed of semantically meaningful sub-tasks. While Movement Primitives (MPs) can model the… (voir plus)se components, their composition typically relies on manual engineering or sample-inefficient reinforcement learning. Existing supervised methods are limited by the lack of MP labels in standard datasets and often fail to generalize across discrete and rhythmic movements. We propose Neural Dynamic Composition (NDC), a supervised framework to compose Dynamic Movement Primitives (DMPs) using only raw trajectories for supervision, given a single example of each primitive. Our approach exploits parallel combinations of DMPs as a differentiable relaxation of categorical primitive selection, and utilizes a warp-invariant loss formulation to identify primitives despite spatial or temporal variations. A staged training curriculum is employed to learn the selection policy and DMP parameters, unifying discrete and rhythmic movements in one framework. Evaluated on 2D handwriting and 3D robotic tasks, our method outperforms baselines on most tasks and metrics. An ablation study demonstrates how each design choice contributes to the system's performance.

2026-10-10

arXiv (prépublication)

Gated Transformer World Models for Model-Based Reinforcement Learning

Model-based reinforcement learning has emerged as a promising approach for training sample-efficient policies by learning compact latent dyn… (voir plus)amics models that simulate, plan, and act through imagination. Recent work has shown that transformer-based world models outperform recurrent architectures due to their ability to capture long-range temporal dependencies. However, standard self-attention mechanisms apply the attention score uniformly across all dimensions of the token's representation, lacking an explicit mechanism to regulate information flow across dimensions. We introduce a gating mechanism for transformer-based world models that implements a feature-wise modulation mechanism within the attention computation. The gating module adaptively scales dimensions in a data-dependent way, enabling the model to selectively control contextual information. Our gating mechanism achieves greater sample-efficient policy learning through normalized mean performance improvements for the Atari 100k benchmark when applied to the STORM and TWISTER transformer world models.

2026-10-02

WM_PAI @ Advances in Neural Information Processing Systems (poster)

Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion

Chenhao Li

Marco Hutter

World models promise a paradigm shift in robotics, where an agent learns the physics of its environment once and then acquires behaviors eff… (voir plus)iciently. Yet the learned dynamics models at their core are typically morphology locked. In legged locomotion, a dynamics model trained on an ANYmal-D quadruped fails on a Unitree Go1 because it overfits to one robot's embodiment rather than capturing the locomotion dynamics shared across robots, so even a small change in actuator dynamics or limb length forces retraining from scratch. However, if we formalize a robot's unique physical traits into a morphology specification, a controller for a family of robots can utilize this blueprint in two ways. It can feed the specification to a model-free policy, or it can feed the specification to a learned dynamics model and extract the policy in imagination. We argue for the second route and introduce the Quadrupedal World Model (QWM), which conditions a single generative dynamics model on scale-invariant physical features and trains policies entirely inside it, through a physical morphology encoder, an adaptive reward normalizer, and morphology conditioning in the latent dynamics. Holding the morphology information identical, a model-free policy matches QWM on the training cohort but degrades on unseen morphologies, while QWM transfers zero-shot with no fine-tuning, adaptation, or warm-up in such cases. To our knowledge, this is the first world model to demonstrate zero-shot cross-embodiment transfer within the quadrupedal family.

2026-09-03

Conference on Robot Learning (poster)

VIDP: Variable Impedance Diffusion Policy for Compliant Robot Manipulation from Diverse Demonstrations

Hisham Khalil

Neil Fernandes

Thomas M. Kwok

Yue Hu

Contact-rich manipulation requires precise tracking and mechanical compliance, where variable impedance control can improve robustness in ta… (voir plus)sk success, whereas static compliance cannot adapt to varying contact constraints. Variable impedance skills can be learned from demonstrations, avoiding complex modeling, but compliance is a hidden variable in force-agnostic kinematic data. While existing methods infer compliance from trajectory variations, these variations may reflect geometric adaptation and not intentional compliance when subject to changing spatial layouts. Therefore, this letter introduces Variable Impedance Diffusion Policy (VIDP), an imitation learning-based variable impedance control framework leveraging a Task-Parameterized Directionality-Aware Mixture Model (TP-DAMM) to extract physically consistent trajectory distributions from diverse demonstrations. By mapping distributions to stiffness profiles, VIDP jointly predicts pose actions and task compliance without force sensors. Real-world experiments show that VIDP significantly outperforms fixed-impedance baselines in task success rate while reducing interaction forces with respect to high stiffness controllers and tracking errors with respect to low stiffness baselines.

2026-08-05

arXiv (prépublication)

SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

Majid Khadiv

Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot … (voir plus)be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.

2026-07-08

arXiv (prépublication)

SLowRL: Safe Low-Rank Adaptation for Bridging the Sim-to-Real Gap in Legged Locomotion

Shafeef Omar

Majid Khadiv

A simulator is, at best, a coarse low-fidelity model of the real world the agent eventually has to act in. Closing this residual gap on hard… (voir plus)ware is a canonical instance of operating in a big world: the real environment exposes contact dynamics, latencies, and disturbances that the agent was never given the capacity (parameters or data) to model during pretraining. Naive on-hardware fine-tuning is risky --- the policy can damage the robot before it improves --- and full-parameter updates require prohibitive interaction time. We propose SLowRL, a continual fine-tuning framework that confronts this big-world adaptation problem with two complementary forms of capacity limitation: (i) a rank-1 LoRA adapter applied per layer to both actor and critic, restricting each layer's update to a single direction in its image space (

2026-06-09

rl-conference.cc/RLC/2026/Workshop/RL_in_Big_Worlds (poster)

Drift Q-Learning

Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value … (voir plus)estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/

2026-05-28

arXiv (prépublication)

Partial sympatry of two interspersed Uroteuthis edulis (Loliginidae) clades reveals cryptic stock structure in the southern Taiwan Strait

Rui-Yi Hong

Lorenzo C. Halasan

Ke-Yang Chang

Chia-Hui Wang

2026-04-30

Fisheries Research (publié)

SAD-Flower: Flow Matching for Safe, Admissible, and Dynamically Consistent Planning

Tzu-Yuan Huang

Armin Lederer

Dai-Jie Wu

Xiaobing Dai

Sihua Zhang

Shao-Hua Sun

Stefan Sosnowski

Sandra Hirche

Flow matching (FM) has shown promising results in data-driven planning. However, it inherently lacks formal guarantees for ensuring state an… (voir plus)d action constraints, whose satisfaction is a fundamental and crucial requirement for the safety and admissibility of planned trajectories on various systems. Moreover, existing FM planners do not ensure the dynamical consistency, which potentially renders trajectories inexecutable. We address these shortcomings by proposing SAD-Flower, a novel framework for generating

2026-04-29

International Conference on Machine Learning (accepté)

Genetic connectivity of Uroteuthis sibogae (Cephalopoda: Loliginidae) in the Sulu Sea with notes on morphology and statolith microchemistry

Jessica M. Legaspi

Lorenzo C. Halasan

Kris Angeli S. Sanchez-Delos Reyes

Tomoyo Okumura

Hsiu‐Chin Lin

2026-04-25

Regional Studies in Marine Science (publié)