Portrait of Yihao Sun

Yihao Sun

yihao.sun@mila.quebec
PhD - Université de Montréal
Supervisor
Pierre-Luc Bacon
Research Topics
Deep Learning
Reinforcement Learning
Website
Google Scholar
GitHub

Publications

The Three Regimes of Offline-to-Online Reinforcement Learning
Lu Li
Tianwei Ni
Yihao Sun
Pierre-Luc Bacon
Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online… (see more) interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online-fine tuning that work well in one setting can fail completely in another. We propose a stability--plasticity principle that can explain this inconsistency: we should preserve the knowledge of pretrained policy or offline dataset during online fine-tuning, whichever is better, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 of 63 cases. This work provides a principled framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.
2026-06-15
ContinualRL @ Reinforcement Learning Conference (published)
doi.org
openreview.net
Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
Zhilong Zhang
Haoxiang Ren
Yihao Sun
Yifei Sheng
Haonan Wang
Haoxin Lin
Zhichao Wu
Pierre-Luc Bacon
Yang Yu
Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is … (see more)constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across multimodal models and model-based RL, we propose **VLA-MBPO**, a practical world model-based RL framework to tackle these problems in VLA finetuning. Our approach is guided by three key design choices: (i) adapting *unified multimodal models (UMMs)* to VLA settings, leveraging rich multimodal priors to enable world modeling with limited data; (ii) introducing an *interleaved view decoding* mechanism to enforce consistency across views; and (iii) employing *chunk-level branched rollout* to limit rollout horizons and mitigate error compounding during policy optimization. Our theoretical analysis shows a reduction in value gap of VLA-MBPO, and experiments in both simulated and real-world tasks demonstrate that our method effectively improves policy performance and sample efficiency for VLA finetuning.
2026-03-01
World Models @ International Conference on Learning Representations (published)
doi.org
openreview.net
Hierarchical Value-Decomposed Offline Reinforcement Learning for Whole-Body Control
Zhilong Zhang
Yunpeng Mei
Xinghao Du
Hongjie Cao
Haonan Wang
Pengyuan Min
Chenyu Wang
Pengfei Chen
Chenbo Xin
Yijie Wang
Wenyu Luo
Yihao Sun
Yidi Wang
Lei Yuan
Gang Wang
Yang Yu
Scaling imitation learning to high-DoF whole-body robots is fundamentally limited by the \textbf{curse of dimensionality} and the prohibitiv… (see more)e cost of collecting expert demonstrations. We argue that the core bottleneck is paradigmatic: real-world supervision for whole-body control is inherently imperfect, yet most methods assume expert data. To overcome this, we propose \textbf{HVD} (Hierarchical Value-Decomposed Offline Reinforcement Learning), an offline reinforcement learning framework that learns effective policies directly from suboptimal, reward-labeled trajectories. HVD structures the value function along the robot’s kinematic hierarchy and over temporal chunks, enabling precise credit assignment in long-horizon, high-dimensional tasks. Built on a Transformer-based architecture, HVD supports \textit{multi-modal} and \textit{multi-task} learning, allowing flexible integration of diverse sensory inputs for policy learning. To enable realistic evaluation and training, we further introduce \textbf{WB-50}, a 50-hour dataset of teleoperated and policy rollout trajectories annotated with rewards and preserving natural imperfections — including partial successes, corrections, and failures. Experiments show HVD significantly outperforms existing baselines in success rate across complex whole-body tasks. Our results suggest that effective policy learning for high-DoF systems can emerge not from perfect demonstrations, but from structured learning over realistic, imperfect data.
2025-12-31
International Conference on Learning Representations (Accept (Poster))
openreview.net
openreview.net
Speedup Patch: Learning a Plug-and-Play Policy to Accelerate Embodied Manipulation
Zhichao Wu
Junyin Ye
Zhilong Zhang
Yihao Sun
Haoxin Lin
Jiaheng Luo
Haoxiang Ren
Lei Yuan
Yang Yu
While current embodied policies exhibit remarkable manipulation skills, their execution remains unsatisfactorily slow as they inherit the ta… (see more)rdy pacing of human demonstrations. Existing acceleration methods typically require policy retraining or costly online interactions, limiting their scalability for large-scale foundation models. In this paper, we propose **S**peed**u**p **P**atch (**SuP**), a lightweight, policy-agnostic framework that enables **plug-and-play acceleration** using solely offline data. SuP introduces an external scheduler that adaptively downsamples action chunks provided by embodied policies to eliminate redundancies. Specifically, we formalize the optimization of our scheduler as a Constrained Markov Decision Process (CMDP) aimed at maximizing efficiency without compromising task performance. Since direct success evaluation is infeasible in offline settings, SuP introduces **World Model based state deviation** as a surrogate metric to enforce safety constraints. By leveraging a learned world model as a virtual evaluator to predict counterfactual trajectories, the scheduler can be optimized via offline reinforcement learning. Empirical results on simulation benchmarks (Libero, Bigym) and real-world tasks validate that SuP achieves an overall
2025-12-31
International Conference on Machine Learning (Accept (regular))
doi.org
openreview.net
Planning with Unified Multimodal Models
Yihao Sun
Zhilong Zhang
Yang Yu
Pierre-Luc Bacon
With the powerful reasoning capabilities of large language models (LLMs) and vision-language models (VLMs), many recent works have explored … (see more)using them for decision-making. However, most of these approaches rely solely on language-based reasoning, which limits their ability to reason and make informed decisions. Recently, a promising new direction has emerged with unified multimodal models (UMMs), which support both multimodal inputs and outputs. We believe such models have greater potential for decision-making by enabling reasoning through generated visual content. To this end, we propose Uni-Plan, a planning framework built on UMMs. Within this framework, a single model simultaneously serves as the policy, dynamics model, and value function. In addition, to avoid hallucinations in dynamics predictions, we present a novel approach self-discriminated filtering, where the generative model serves as a self-discriminator to filter out invalid dynamics predictions. Experiments on long-horizon planning tasks show that Uni-Plan substantially improves success rates compared to VLM-based methods, while also showing strong data scalability, requiring no expert demonstrations and achieving better performance under the same training-data size. This work lays a foundation for future research in reasoning and decision-making with UMMs.
2025-09-26
ArXiv (preprint)
doi.org
arxiv.org