Derek Nowrouzezahrai

Informatique visuelle différentiable : passer de la deuxième à la troisième dimension dans les applications d’apprentissage automatique

Collaborateur·rice de recherche - McGill

Co-superviseur⋅e :

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Doctorat - McGill

Xiangyu (Daniel) Kong

Doctorat - McGill

Github

Joey Marten

Maîtrise recherche - McGill

Co-superviseur⋅e :

Noam Aigerman

Github

Hossam Mohamed S M Ahmed

Doctorat - McGill

Github

Somjit Nath

Doctorat - McGill

Superviseur⋅e principal⋅e :

Samira Ebrahimi Kahou

Site web

Google Scholar

Yasaman Sabbagh Ziarani

Doctorat - McGill

Yuanyuan Tao

Doctorat - McGill

Mattie Tesfaldet

Collaborateur·rice de recherche - McGill

Co-superviseur⋅e :

Junha Yoo

Maîtrise recherche - McGill

Co-superviseur⋅e :

James Richard Forbes

Billets de blogue

An illustration of the forward and backward processes of a differentiable renderer.

29 janvier 2025

par

Jordan J. Bannister

Derek Nowrouzezahrai

Lire l'article

Publications

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

Parsa Esmati

Somjit Nath

Katja Hofmann

Samira Ebrahimi Kahou

Majid Mirmehdi

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simula… (voir plus)tors. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.

2026-06-02

arXiv (prépublication)

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion

Somjit Nath

Jackson J Cone

Samira Ebrahimi Kahou

Neuroscientific research has revealed that the brain encodes complex behaviors by leveraging structured, low-dimensional manifolds and dynam… (voir plus)ically fusing multiple sources of information through adaptive gating mechanisms. Inspired by these principles, we propose a novel reinforcement learning (RL) framework that encourages the disentanglement of dynamics-specific and reward-specific features, drawing direct parallels to how neural circuits separate and integrate information for efficient decision-making. Our approach leverages locally linear embeddings (LLEs) to capture the intrinsic, locally linear structure inherent in many environments—mirroring the local smoothness observed in neural population activity—while concurrently deriving reward-specific features through the standard RL objective. An attention mechanism, analogous to cortical gating, adaptively fuses these complementary representations on a per-state basis. Experimental results on benchmark tasks demonstrate that our method, grounded in neuroscientific principles, improves learning efficiency and overall performance compared to conventional RL approaches, highlighting the benefits of explicitly modeling local state structures and adaptive feature selection as observed in biological systems.

2026-05-24

Transactions on Machine Learning Research (accepté)

Modelling Customer Trajectories with Reinforcement Learning for Practical Retail Insights

Ken Ming Lee

Maxime C. Cohen

Understanding customer movement within retail spaces is essential for optimizing store layouts. Real-world trajectory data can provide highl… (voir plus)y accurate insights, but collecting it is costly and often infeasible for many retailers. Heuristics such as Travelling Salesman Problem (TSP) and Probabilistic Nearest Neighbours (PNN) are commonly used as inexpensive approximations, but actual customer trajectories deviate by an average of 28% from shortest paths, highlighting a tradeoff between accuracy and practicality. We propose an agent-based modelling framework that casts customer trajectory prediction as a maximum entropy reinforcement learning (RL) problem, balancing reward maximization with stochasticity to better reflect customers with bounded rationality. Using real-world trajectory data from a convenience store, we show that RL-generated trajectories align more closely with customer behaviour than TSP and PNN, providing more accurate estimates of impulse purchase rates and shelf traffic densities. Furthermore, only RL-based predictions yield repositioning decisions for impulse products that align with those derived from actual trajectory data, resulting in comparable estimated profit gains. Our work demonstrates that RL provides a practical, behaviourally grounded alternative that bridges the gap between oversimplified heuristics and data-intensive approaches, making accurate layout optimization more accessible. To encourage further research, the source code is available on GitHub.

2026-05-17

arXiv (prépublication)

Hierarchical Differentiable Fluid Simulation

Xiangyu Kong

Arnaud Schoentgen

Damien Rioux‐Lavoie

Paul G. Kry

Differentiable simulation is an emerging field that offers a powerful and flexible route to fluid control. In grid‐based settings, high me… (voir plus)mory consumption is a long‐standing bottleneck that constrains optimization resolution. We introduce a two‐step algorithm that significantly reduces memory usage: our method first optimizes for bulk forces at reduced resolution, then refines local details over sub‐domains while maintaining differentiability. In trading runtime for memory, it enables optimization at previously unattainable resolutions. We validate its effectiveness and memory savings on a series of fluid control problems.

2025-10-16

Computer Graphics Forum (publié)

Spherical Harmonic Exponentials for Efficient Glossy Reflections

Ari Silvennoinen

Peter‐Pike Sloan

Michaƚ Iwanicki

Abstract We propose a high‐performance and compact method for computing glossy specular reflections. Commonly‐used prefiltered environme… (voir plus)nt maps have large storage requirements and high error due to constrained treatment of view‐dependence. We propose a factorized spherical harmonic exponential representation that exploits new observations of the benefits of log‐space reconstruction for reflectance. Our method is compact, properly accounts for view‐dependent reflections, and is more accurate than the state‐of‐the‐industry solutions. We achieve higher quality results with an order of magnitude less memory, all with efficient and alias‐free reconstruction of glossy reflections from environment lights and continuously‐varying material roughness.

2025-09-18

Computer Graphics Forum (publié)

Minimax Exploiter: A Data Efficient Approach for Competitive Self-Play

Gabriel Robert

Recent advances in Competitive Self-Play (CSP) have achieved, or even surpassed, human level performance in complex game environments such a… (voir plus)s Dota 2 and StarCraft II using Distributed Multi-Agent Reinforcement Learning (MARL). One core component of these methods relies on creating a pool of learning agents -- consisting of the Main Agent, past versions of this agent, and Exploiter Agents -- where Exploiter Agents learn counter-strategies to the Main Agents. A key drawback of these approaches is the large computational cost and physical time that is required to train the system, making them impractical to deploy in highly iterative real-life settings such as video game productions. In this paper, we propose the Minimax Exploiter, a game theoretic approach to exploiting Main Agents that leverages knowledge of its opponents, leading to significant increases in data efficiency. We validate our approach in a diversity of settings, including simple turn based games, the arcade learning environment, and For Honor, a modern video game. The Minimax Exploiter consistently outperforms strong baselines, demonstrating improved stability and data efficiency, leading to a robust CSP-MARL method that is both flexible and easy to deploy.

2023-12-31

AAMAS (publié)

A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

Jakob Foerster

Amy Zhang

Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences. H… (voir plus)owever, most existing Multi-Agent Reinforcement Learning (MARL) methods are online and thus impractical for real-world applications in which collecting new interactions is costly or dangerous. While these algorithms should leverage offline data when available, doing so gives rise to what we call the offline coordination problem. Specifically, we identify and formalize the strategy agreement (SA) and the strategy fine-tuning (SFT) coordination challenges, two issues at which current offline MARL algorithms fail. Concretely, we reveal that the prevalent model-free methods are severely deficient and cannot handle coordination-intensive offline multi-agent tasks in either toy or MuJoCo domains. To address this setback, we emphasize the importance of inter-agent interactions and propose the very first model-based offline MARL method. Our resulting algorithm, Model-based Offline Multi-Agent Proximal Policy Optimization (MOMA-PPO) generates synthetic interaction data and enables agents to converge on a strategy while fine-tuning their policies accordingly. This simple model-based solution solves the coordination-intensive offline tasks, significantly outperforming the prevalent model-free methods even under severe partial observability and with learned world models.

2023-12-31

AAMAS (publié)

Neural Implicit Reduced Fluid Simulation

Yuanyuan Tao

Ivan Puhachov

Paul Kry

High-fidelity simulation of fluid dynamics is challenging because of the high dimensional state data needed to capture fine details and the … (voir plus)large computational cost associated with advancing the system in time. We present neural implicit reduced fluid simulation (NIRFS), a reduced fluid simulation technique that combines an implicit neural representation of fluid shapes and a neural ordinary differential equation to model the dynamics of fluid in the reduced latent space. The latent trajectories are computed at very little cost in comparison to simulations for training, while preserving fine physical details. We show that this approach can work well, capturing the shapes and dynamics involved in a variety of scenarios with constrained initial conditions, e.g., droplet-droplet collisions, crown splashes, and fluid slosh in a container. In each scenario, we learn the latent implicit representation of fluid shapes with a deep-network signed distance function, as well as the energy function and parameters of a damped Hamiltonian system, which helps guarantee desirable properties of the latent dynamics. To ensure that latent shape representations form smooth and physically meaningful trajectories, we simultaneously learn the latent representation and dynamics. We evaluate novel simulations for conservation of volume and momentum conservation, discuss design decisions, and demonstrate an application of our method to fluid control.

2023-12-31

SIGGRAPH Asia (publié)

MeshDiffusion: Score-Based Generative 3D Mesh Modeling

Zhen Liu

Yao Feng

Michael J. Black

Liam Paull

Weiyang Liu

We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and… (voir plus) physical simulation. Compared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which are mostly optimized for meshes. Previous scalable methods for generating meshes typically rely on sub-optimal post-processing, and they tend to produce overly-smooth or noisy surfaces without fine-grained geometric details. To overcome these shortcomings, we take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization. We demonstrate the effectiveness of our model on multiple generative tasks.

2023-01-31

ICLR.cc/2023/Conference (notable)

Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests

Christopher Pal

Different types of mental rotation tests have been used extensively in psychology to understand human visual reasoning and perception. Under… (voir plus)standing what an object or visual scene would look like from another viewpoint is a challenging problem that is made even harder if it must be performed from a single image. We explore a controlled setting whereby questions are posed about the properties of a scene if that scene was observed from another viewpoint. To do this we have created a new version of the CLEVR dataset that we call CLEVR Mental Rotation Tests (CLEVR-MRT). Using CLEVR-MRT we examine standard methods, show how they fall short, then explore novel neural architectures that involve inferring volumetric representations of a scene. These volumes can be manipulated via camera-conditioned transformations to answer the question. We examine the efficacy of different model variants through rigorous ablations and demonstrate the efficacy of volumetric representations.

2022-11-29

Pattern Recognition (inconnu)

Learning to Guide and to Be Guided in the Architect-Builder Problem

Tristan Karch

Clément Moulin-Frier

Christopher Pal

Pierre-Yves Oudeyer

We are interested in interactive agents that learn to coordinate, namely, a …

2022-01-27

ICLR.cc/2022/Conference (poster)

Attention-based Neural Cellular Automata

Mattie Tesfaldet

Christopher Pal

Recent extensions of Cellular Automata (CA) have incorporated key ideas from modern deep learning, dramatically extending their capabilities… (voir plus) and catalyzing a new family of Neural Cellular Automata (NCA) techniques. Inspired by Transformer-based architectures, our work presents a new class of

2021-12-31

Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (publié)