Portrait de Michael Rabbat n'est pas disponible

Michael Rabbat

Membre industriel associé
Chercheur scientifique, AMI Labs
Sujets de recherche
Apprentissage de représentations
Optimisation
Systèmes distribués
LinkedIn

Biographie

Mike Rabbat est membre affilié de Mila et cofondateur et vice-président des « World Models » chez Advanced Machine Intelligence (AMI). Ses intérêts de recherche portent sur l'apprentissage de représentations efficace et robuste, l'apprentissage auto-supervisé, ainsi que la planification et le contrôle basés sur des modèles. Il s'intéresse également à l'optimisation pour l'entraînement efficace de modèles.

Publications

Interpreting Physics in Video World Models: Toward Safe Embodied AI
Sonia Joseph
Quentin Garrido
Randall Balestriero
Matthew Kowal
Thomas Fel
Shahab Bakhtiari
Blake Aaron Richards
Michael Rabbat
Physical AI agents acting in the real world fail differently from language models: a misjudgment of trajectory, force, or contact can have i… (voir plus)mmediate and potentially irreversible consequences. As video world models are increasingly deployed as the perception and dynamics layer of such agents, understanding what physical structure they actually represent internally be- comes a precondition for trustworthy deployment. Yet today these models are largely studied as black boxes, evaluated only through behavioral benchmarks. We argue that this should change: world models can be opened up, and the latent variables they encode can be measured, interpreted, and eventually controlled. We take a first step in this direction across two state-of-the-art video encoders (V-JEPA 2 and VideoMAE-v2), using layerwise probing, subspace geometry, patch-level decoding, and targeted attention ablations. We identify a sharp intermediate-depth transition we call the Physics Emergence Zone, at which physical variables become accessible. Decomposing motion into explicit variables, we find that scalar quantities such as speed and acceleration are available from early layers onwards, whereas motion direction becomes accessible only at the Physics Emergence Zone, encoded as a high-dimensional circular population code that requires coordinated multi-feature intervention to steer. These findings argue against compact, reusable latent physics state and in favor of distributed, task-specific representations supported by a shared local-attention circuit. We discuss implications for safe deploy- ment of video world models in embodied physical AI, including why intermediate-layer features, multi-feature monitors, and the local-attention circuit at the Physics Emergence Zone are natural targets for runtime verification.
2026-07-19
SPAI @ International Joint Conference on Artificial Intelligence - European Conference on Artificial Intelligence (publié)
openreview.net
openreview.net
Towards Unified Multimodal Pretraining
Shengbang Tong
David Fan
John Nguyen
Ellis Brown
Gaoyue Zhou
Shengyi Qian
Boyang Zheng
Théophane Vallaeys
Rob Fergus
Naila Murray
Marjan Ghazvininejad
Mike Lewis
Jakob Verbeek
Nicolas Ballas
Amir Bar
Michael Rabbat
Yann Lecun
Luke Zettlemoyer
Saining Xie
Koustuv Sinha
Unified multimodal models aim to input and output both vision and language data within a single system. In this work, we explore the design … (voir plus)space of Unified Multimodal Pretraining through a controlled, from-scratch study. We find that leveraging a single high-dimensional semantic encoder (e.g. SigLIP 2) achieves the best combined performance for both visual understanding and generation. Furthermore, we observe that integrating diverse visual data---including raw video and image-text pairs---has minimal impact on language capabilities, suggesting that vision and text are compatible within a single unified model. We identify positive synergy where joint pretraining enhances downstream capabilities such as Visual Question Answering (VQA) and World Modeling. Turning to architecture, we investigate Mixture-of-Experts (MoE) design choices, such as granularity and sparsity, to identify an effective training recipe. Finally, we quantify scaling dynamics via IsoFLOP analysis and uncover a scaling asymmetry: language scaling is parameter-hungry, while vision scaling is significantly more data-hungry. We demonstrate that MoE architectures help address this imbalance by decoupling total parameter capacity from active compute, enabling the high capacity required for language while also accommodating the data-intensive nature of vision.
2025-12-31
International Conference on Machine Learning (Accept (spotlight))
openreview.net
openreview.net
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
Mahmoud Assran
Quentin Duval
Ishan Misra
Piotr Bojanowski
Pascal Vincent
Michael Rabbat
Yann Lecun
Nicolas Ballas
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. W… (voir plus)e introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.
2023-06-16
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (publié)
doi.org
arxiv.org