Shahab Bakhtiari

nastaran.darjani@mila.quebec

Laura Chen

Research Intern - McGill University University

laura.chen@mila.quebec

Nastaran Darjani

PhD - Université de Montréal

hafez.ghaemi@umontreal.ca

Google Scholar

Hafez Ghaemi

PhD - Université de Montréal

Principal supervisor :

Eilif B. Muller

PhD

imane.hamzaoui@mila.quebec

Google Scholar

Lianghuan Huang

Collaborating researcher

lianghuan.huang@mila.quebec

Pelin Moradi Haghgou

Research Intern - Université de Montréal

peivand.moradi-haghgou@mila.quebec

Ladan Shahshahani

Postdoctorate - Université de Montréal

ladan.shahshahani@umontreal.ca

elizaveta.sycheva@mila.quebec

Google Scholar

Liza Sycheva

PhD - Université de Montréal

Ehsan Tousi

Collaborating researcher - Western University

ehsan.tousi@mila.quebec

Doris Voina

Postdoctorate - Université de Montréal

doris.voina@mila.quebec

Charlotte Volk

PhD - McGill University

Principal supervisor :

Blake Richards

charlotte.volk@mila.quebec

Publications

Divergent specializations for motion-driven representations in higher lateral and dorsal visual areas

Nastaran Darjani

Sophia Robert

Maryam Vaziri-Pashkam

The human visual system integrates both static and dynamic information to support form and shape perception, yet the computational principle… (see more)s underlying the integration of motion for object recognition remain unclear. Artificial neural networks (ANNs) offer a computational framework for developing and testing hypotheses about these principles: if ANNs trained on motion-related tasks develop representations that align with brain activity and support object categorization, this would suggest that the training objectives and architectural constraints of these networks may capture key aspects of motion processing in biological visual systems in general, and motion processing for object recognition, in particular. Here, we investigated this question using “object kinematograms”, stimuli in which object form is conveyed solely through motion cues. We measured neural responses of two higher regions of the lateral and the dorsal visual pathways, respectively, with strong sensitivity to dynamic cues from objects: lateral occipitotemporal cortex (LOT bio ), and left supramarginal gyrus (SMG lh ), as well as primary visual cortex (V1). We compared brain responses to representations extracted from two neural networks: SlowFast, a dual-pathway architecture trained on action recognition that processes slow- and fast-varying visual information with cross-pathway integration, and DorsalNet, a model of the primate dorsal visual pathway trained on embodied self-motion estimation. Representational similarity analysis revealed distinct representational profiles across brain areas, demonstrating functional specialization in motion-based form processing. LOT bio was best characterized by the slow pathway of the SlowFast model, whereas SMG lh showed strong similarity to both models. Critically, we found that representations aligned with brain activity also better supported behavioral function: the full SlowFast model, incorporating both slow and fast pathways, outperformed other models in few-shot categorization of object kinematograms and showed the highest similarity to human perceptual judgments. These findings demonstrate that with appropriate inductive biases, specifically, dual-pathway architectures for multi-scale motion processing and training objectives focused on dynamic visual tasks, ANNs can develop functionally useful representations of motion-defined forms that exhibit better alignment with the visual regions involved in processing dynamic visual signals.

2026-08-11

bioRxiv (preprint)

Interpreting Physics in Video World Models: Toward Safe Embodied AI

Sonia Joseph

Quentin Garrido

Randall Balestriero

Matthew Kowal

Thomas Fel

Blake Aaron Richards

Michael Rabbat

Physical AI agents acting in the real world fail differently from language models: a misjudgment of trajectory, force, or contact can have i… (see more)mmediate and potentially irreversible consequences. As video world models are increasingly deployed as the perception and dynamics layer of such agents, understanding what physical structure they actually represent internally be- comes a precondition for trustworthy deployment. Yet today these models are largely studied as black boxes, evaluated only through behavioral benchmarks. We argue that this should change: world models can be opened up, and the latent variables they encode can be measured, interpreted, and eventually controlled. We take a first step in this direction across two state-of-the-art video encoders (V-JEPA 2 and VideoMAE-v2), using layerwise probing, subspace geometry, patch-level decoding, and targeted attention ablations. We identify a sharp intermediate-depth transition we call the Physics Emergence Zone, at which physical variables become accessible. Decomposing motion into explicit variables, we find that scalar quantities such as speed and acceleration are available from early layers onwards, whereas motion direction becomes accessible only at the Physics Emergence Zone, encoded as a high-dimensional circular population code that requires coordinated multi-feature intervention to steer. These findings argue against compact, reusable latent physics state and in favor of distributed, task-specific representations supported by a shared local-attention circuit. We discuss implications for safe deploy- ment of video world models in embodied physical AI, including why intermediate-layer features, multi-feature monitors, and the local-attention circuit at the Physics Emergence Zone are natural targets for runtime verification.

2026-07-19

SPAI @ International Joint Conference on Artificial Intelligence - European Conference on Artificial Intelligence (published)

Beyond Sensory Summation: How Expectations and Sensory Evidence Shape Multisensory Perception

Elizaveta Sycheva

Léa St-Gelais

Jérémy Brunel

Karim Jerbi

Franco Lepore

Vanessa Hadid

Perceptual decisions arise through the interplay of incoming sensory evidence and prior expectations. However, it remains unclear how this i… (see more)nteraction shapes multisensory integration during the accumulation of decision evidence over time. Using dynamic audiovisual (AV) scenes in a semantic decision task, we examined how sensory reliability and semantic expectations influence decision-making. AV signals that were both coherent and congruent accelerated responses relative to unimodal conditions. This facilitation was strongest when visual input was degraded, consistent with increased reliance on joint AV contributions, as indicated by race-model violations demonstrating multisensory coactivation. Diffusion modeling revealed increased drift rates alongside longer non-decision times, indicating stronger evidence accumulation despite additional sensory processing, and resulting in faster overall responses. Together, these findings reveal that multisensory integration is not a fixed sensory-gain mechanism but a context-dependent coactivation process that selectively enhances evidence accumulation when signals converge on a shared semantic interpretation.

2026-05-13

Cognitive Computational Neuroscience (poster)

Shaped by meaning, weighted by reliability: New insights into multisensory integration

Elizaveta Sycheva

Léa St-Gelais

Karim Jerbi CoCo Lab

Franco Lepore

Vanessa Hadid

2026-01-04

bioRxiv (preprint)

Interpreting Physics in Video World Models

Sonia Joseph

Quentin Garrido

Randall Balestriero

Matthew Kowal

Thomas Fel

Blake Richards

Mike Rabbat

A long-standing question in physical reasoning is whether video-based models need to rely on factorized representations of physical variable… (see more)s in order to make physically accurate predictions, or whether they can implicitly represent such variables in a distributed manner. While modern video world models achieve strong performance on intuitive physics benchmarks, it remains unclear which of these representational regimes they implement internally. Here, we present the first interpretability study to directly examine physical representations inside large-scale video encoders. Using layerwise probing, subspace geometry, patch-level decoding, and targeted attention ablations, we characterize where physical information becomes accessible and how it is organized within encoder-based video transformers. Across architectures, we identify a sharp intermediate-depth transition— which we call the \emph{Physics Emergence Zone}—at which physical variables become accessible. Physics-related representations peak shortly after this transition and degrade toward the output layers. Decomposing motion into explicit variables, we find that scalar quantities such as speed and acceleration are available from early layers onwards, whereas motion direction becomes accessible only at the Physics Emergence Zone. Notably, we find that direction is encoded through a high-dimensional population structure with circular geometry, requiring coordinated multi-feature intervention to control. These findings suggest that modern video models do not use factorized representations of physical variables like a classical physics engine. Instead, they use a distributed representation that is nonetheless sufficient for making physical predictions.

2025-12-31

International Conference on Machine Learning (Accept (regular))

seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models

Hafez Ghaemi

Eilif B. Muller

Current self-supervised algorithms commonly rely on transformations such as data augmentation and masking to learn visual representations. T… (see more)his is achieved by enforcing invariance or equivariance with respect to these transformations after encoding two views of an image. This dominant two-view paradigm often limits the flexibility of learned representations for downstream adaptation by creating performance trade-offs between high-level invariance-demanding tasks such as image classification and more fine-grained equivariance-related tasks. In this work, we proposes \emph{seq-JEPA}, a world modeling framework that introduces architectural inductive biases into joint-embedding predictive architectures to resolve this trade-off. Without relying on dual equivariance predictors or loss terms, seq-JEPA simultaneously learns two architecturally segregated representations: one equivariant to specified transformations and another invariant to them. To do so, our model processes short sequences of different views (observations) of inputs. Each encoded view is concatenated with an embedding of the relative transformation (action) that produces the next observation in the sequence. These view-action pairs are passed through a transformer encoder that outputs an aggregate representation. A predictor head then conditions this aggregate representation on the upcoming action to predict the representation of the next observation. Empirically, seq-JEPA demonstrates strong performance on both equivariant and invariant benchmarks without sacrificing one for the other. Furthermore, it excels at tasks that inherently require aggregating a sequence of observations, such as path integration across actions and predictive learning across eye movements.

2025-12-02

Conference on Neural Information Processing Systems (Accept (poster))

Context-Aware World Models for Task-Agnostic Control

Busra Tugce Gurbuz

Hafez Ghaemi

Christopher C. Pack