Portrait de Dhruv Sreenivas

Dhruv Sreenivas

dhruv.sreenivas@mila.quebec
Doctorat - UdeM
Superviseur⋅e principal⋅e
Pablo Samuel Castro
Sujets de recherche
Apprentissage de représentations
Apprentissage par renforcement
Apprentissage profond
Modèles génératifs
Site web
GitHub

Publications

Imagining Goals with Geometric Horizon Models for Policy Generalization
Daniel Lawson
Dhruv Sreenivas
Adriana Hugessen
Khimya Khetarpal
Glen Berseth
Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be… (voir plus) easily applied zero-shot or efficiently finetuned for downstream tasks. However, these policies can still have poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require combining its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is by generating new experience with world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, which samples from a geometric horizon model and allows for directly imagining novel outcomes that can be achieved by composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC policies significantly improves generalization in stitching-based OGBench tasks.
2026-10-12
arXiv (prépublication)
doi.org
arxiv.org
Imagining Goals for Policy Generalization with Geometric Horizon Models
Daniel Lawson
Dhruv Sreenivas
Adriana Hugessen
Khimya Khetarpal
Glen Berseth
Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be… (voir plus) easily applied zero-shot or efficiently finetuned for downstream tasks. However, training policies on offline data can lead to poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require reusing its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is through generating new experience through world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, sampling from a geometric horizon model, which allows for directly imagining novel outcomes that can be achieved through composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC and offline RL policies improves generalization in stitching-based OGBench tasks.
2026-07-20
RLBRew @ Reinforcement Learning Conference (publié)
openreview.net
openreview.net
Optimistic critics can empower small actors
Olya Mastikhina
Dhruv Sreenivas
Pablo Samuel Castro
Actor-critic methods have been central to many of the recent advances in deep reinforcement learning. The most common approach is to use _sy… (voir plus)mmetric_ architectures, whereby both actor and critic have the same network topology and number of parameters. However, recent works have argued for the advantages of _asymmetric_ setups, specifically with the use of smaller actors. We perform broad empirical investigations and analyses to better understand the implications of this and find that, in general, smaller actors result in performance degradation and overfit critics. Our analyses suggest _poor data collection_, due to value underestimation, as one of the main causes for this behavior, and further highlight the crucial role the critic can play in alleviating this pathology. We explore techniques to mitigate the observed value underestimation, which enables further research in asymmetric actor-critic methods.
2025-05-08
rl-conference.cc/RLC/2025/Conference (accepté)
openreview.net
openreview.net