Portrait de Elham Daneshmand

Elham Daneshmand

elham.daneshmand@mila.quebec
Doctorat - McGill
Superviseur⋅e principal⋅e
Hsiu-Chin Lin
Co-supervisor
Glen Berseth
Sujets de recherche
Apprentissage par renforcement
Site web

Publications

SafeExplorer: An Unbiased Mixed-Policy Gradient for Reinforcement Learning with Recovery Interventions
Elham Daneshmand
Majid Khadiv
Glen Berseth
Hsiu-Chin Lin
Training agents directly on the task they will be deployed on often leads to the best policy performance, but training on physical robots ca… (voir plus)n cause damage through falls, so minimizing training falls is critical. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region. The resulting mixed-policy rollouts bias the on-policy update, and importance sampling cannot remove this bias when the recovery policy is deterministic. We propose SafeExplorer, a drop-in modification of proximal policy optimization (PPO) built on an unbiased policy-gradient estimator of the mixed-policy return. The estimator uses the score function only inside the safe region and never evaluates the recovery policy's density, so deterministic recovery policies pose no problem. On final reward, the estimator matches or outperforms truncated importance sampling, which is itself biased for that same gradient. SafeExplorer also speeds up learning near the safe-region boundary with a closed-form critic target at recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only from successful recoveries. On three simulated locomotion environments with ten seeds each, and given an already available recovery controller, SafeExplorer incurs
2026-10-17
Transactions on Machine Learning Research (accepté)
openreview.net
openreview.net
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Elham Daneshmand
Majid Khadiv
Glen Berseth
Hsiu-Chin Lin
Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot … (voir plus)be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.
2026-07-08
arXiv (prépublication)
doi.org
arxiv.org
SLowRL: Safe Low-Rank Adaptation for Bridging the Sim-to-Real Gap in Legged Locomotion
Elham Daneshmand
Shafeef Omar
Glen Berseth
Majid Khadiv
Hsiu-Chin Lin
A simulator is, at best, a coarse low-fidelity model of the real world the agent eventually has to act in. Closing this residual gap on hard… (voir plus)ware is a canonical instance of operating in a big world: the real environment exposes contact dynamics, latencies, and disturbances that the agent was never given the capacity (parameters or data) to model during pretraining. Naive on-hardware fine-tuning is risky --- the policy can damage the robot before it improves --- and full-parameter updates require prohibitive interaction time. We propose SLowRL, a continual fine-tuning framework that confronts this big-world adaptation problem with two complementary forms of capacity limitation: (i) a rank-1 LoRA adapter applied per layer to both actor and critic, restricting each layer's update to a single direction in its image space (
2026-06-09
rl-conference.cc/RLC/2026/Workshop/RL_in_Big_Worlds (poster)
openreview.net
openreview.net
SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion
Elham Daneshmand
Shafeef Omar
Glen Berseth
Majid Khadiv
Hsiu-Chin Lin
Sim-to-real transfer of locomotion policies often leads to performance degradation due to the inevitable sim-to-real gap. Naively fine-tunin… (voir plus)g these policies directly on hardware is problematic, as it poses risks of mechanical failure and suffers from high sample inefficiency. In this paper, we address the challenge of safely and efficiently fine-tuning reinforcement learning (RL) policies for dynamic locomotion tasks. Specifically, we focus on fine-tuning policies learned in simulation directly on hardware, while explicitly enforcing safety constraints. In doing so, we introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy. We evaluate our method both in simulation and on a real Unitree Go2 quadruped robot for jump and trot tasks. Experimental results show that our method achieves a
2026-03-16
arXiv (prépublication)
doi.org
arxiv.org