Derek Nowrouzezahrai

Core Academic Member

derek@mila.quebec

Canada CIFAR AI Chair

Full Professor, McGill University, Department of Electrical and Computer Engineering

Research Topics

Computational Photography

Computer Vision

Deep Learning

Dynamical Systems

Generative Models

Reinforcement Learning

Representation Learning

Google Scholar

Biography

Derek Nowrouzezahrai is a full professor at McGill University, where he directs the Centre for Intelligent Machines and co-directs the Graphics Lab.

He is also a Canada CIFAR AI Chair and holds the Ubisoft–Mila research Chair, Scaling Game Worlds with Responsible AI.

Nowrouzezahrai’s research tackles the simulation of various physical phenomena, such as the dynamics of moving objects and the simulation of lighting for realistic image synthesis, which have applications in virtual reality, video games, fluid simulation and control, digital manufacturing, computationally augmented optics and geometry processing. He is also interested in the development of differentiable simulators of these dynamical systems and their applications to inverse problems in robotics and vision.

This work relies fundamentally on developing high performance and sample efficient (Markov chain) Monte Carlo-based methods, high-order statistics and computational methods for complex multi-dimensional integration problems, differentiable physics-based simulators and numerical methods for dynamical systems, and on applying machine learning to 3D, visual and interactive media.

Current Students

Daniel Bairamian

PhD - McGill University

daniel.bairamian@mila.quebec

Paul Barde

Collaborating researcher - McGill University

Co-supervisor :

Chris Pal

paul.b.barde@gmail.com

PhD - McGill University

Co-supervisor :

Kaleem Siddiqi

shubham.chopra@mail.mcgill.ca

Github

Chris Emezue

PhD - Université de Montréal

Principal supervisor :

Chris Pal

chris.emezue@mila.quebec

PhD - McGill University

chenbo.jiang@mail.mcgill.ca

Xiangyu (Daniel) Kong

PhD - McGill University

xiangyu.kong@mila.quebec

Github

Joey Marten

Master's Research - McGill University

Co-supervisor :

Noam Aigerman

joey.marten@mila.quebec

Github

Hossam Mohamed S M Ahmed

PhD - McGill University

hossam.mohamed@mila.quebec

Github

Somjit Nath

PhD - McGill University

Principal supervisor :

Samira Ebrahimi Kahou

somjit.nath@mila.quebec

Website

Google Scholar

Yasaman Sabbagh Ziarani

PhD - McGill University

yasaman.sabbagh@mila.quebec

Yuanyuan Tao

PhD - McGill University

yuanyuan.tao@mila.quebec

Mattie Tesfaldet

Collaborating researcher - McGill University

Co-supervisor :

Chris Pal

mattie.tesfaldet@mila.quebec

Website

Github

Google Scholar

Junha Yoo

Master's Research - McGill University

Co-supervisor :

James Richard Forbes

junha.yoo@mila.quebec

Blog Posts

An illustration of the forward and backward processes of a differentiable renderer.

January 29, 2025

Differentiable Visual Computing: Bridging 2D and 3D in Machine Learning Applications

Jordan J. Bannister

Derek Nowrouzezahrai

Read the article

Publications

Overcoming Challenges in Leveraging GANs for Few-Shot Data Augmentation

Pau Rodríguez

Christopher Pal

In this paper, we explore the use of GAN-based few-shot data augmentation as a method to improve few-shot classification performance. We per… (see more)form an exploration into how a GAN can be fine-tuned for such a task (one of which is in a class-incremental manner), as well as a rigorous empirical investigation into how well these models can perform to improve few-shot classification. We identify issues related to the difficulty of training such generative models under a purely supervised regime with very few examples, as well as issues regarding the evaluation protocols of existing works. We also find that in this regime, classification accuracy is highly sensitive to how the classes of the dataset are randomly split. Therefore, we propose a semi-supervised fine-tuning approach as a more pragmatic way forward to address these problems.

2022-11-27

Proceedings of The 1st Conference on Lifelong Learning Agents (published)

doi.org

proceedings.mlr.press

Learning Latent Structural Causal Models

Jithendaraa Subramanian

Yashas Annadani

Ivaxi Sheth

Nan Rosemary Ke

Tristan Deleu

Stefan Bauer

Derek Nowrouzezahrai

Samira Ebrahimi Kahou

Causal learning has long concerned itself with the accurate recovery of underlying causal mechanisms. Such causal modelling enables better e… (see more)xplanations of out-of-distribution data. Prior works on causal learning assume that the high-level causal variables are given. However, in machine learning tasks, one often operates on low-level data like image pixels or high-dimensional vectors. In such settings, the entire Structural Causal Model (SCM) -- structure, parameters, \textit{and} high-level causal variables -- is unobserved and needs to be learnt from low-level data. We treat this problem as Bayesian inference of the latent SCM, given low-level data. For linear Gaussian additive noise SCMs, we present a tractable approximate inference method which performs joint inference over the causal variables, structure and parameters of the latent SCM from random, known interventions. Experiments are performed on synthetic datasets and a causally generated image dataset to demonstrate the efficacy of our approach. We also perform image generation from unseen interventions, thereby verifying out of distribution generalization for the proposed causal model.

2022-10-23

arXiv (preprint)

doi.org

openreview.net

Learning to Guide and to Be Guided in the Architect-Builder Problem

Paul Barde

Tristan Karch

Derek Nowrouzezahrai

Clément Moulin-Frier

Christopher Pal

Pierre-Yves Oudeyer

We are interested in interactive agents that learn to coordinate, namely, a …

2022-04-24

International Conference on Learning Representations (Accept (Poster))

doi.org

openreview.net

Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Christopher Pal

Adversarial Imitation Learning alternates between learning a discriminator -- which tells apart expert's demonstrations from generated ones … (see more)-- and a generator's policy to produce trajectories that can fool this discriminator. This alternated optimization is known to be delicate in practice since it compounds unstable adversarial training with brittle and sample-inefficient reinforcement learning. We propose to remove the burden of the policy optimization steps by leveraging a novel discriminator formulation. Specifically, our discriminator is explicitly conditioned on two policies: the one from the previous generator's iteration and a learnable policy. When optimized, this discriminator directly learns the optimal generator's policy. Consequently, our discriminator's update solves the generator's optimization problem for free: learning a policy that imitates the expert does not require an additional optimization loop. This formulation effectively cuts by half the implementation and computational burden of Adversarial Imitation Learning algorithms by removing the Reinforcement Learning phase altogether. We show on a variety of tasks that our simpler approach is competitive to prevalent Imitation Learning methods.

2020-12-06

Conference on Neural Information Processing Systems (Accept (poster))

doi.org

arxiv.org

Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments

Martin Weiss

Simon Chamorro

Roger Girgis

Margaux Luck

Samira E. Kahou

Joseph P. Cohen

Millions of blind and visually-impaired (BVI) people navigate urban environments every day, using smartphones for high-level path-planning a… (see more)nd white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavor to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.

2020-05-11

Proceedings of the Conference on Robot Learning (published)

doi.org

proceedings.mlr.press

Pix2Shape – Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation

Sai Rajeswar

Fahim Mannan

Florian Golemo

Jérôme Parent-lévesque

David Vázquez

Derek Nowrouzezahrai

Aaron Courville

We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-expl… (see more)ored, with most prior work relying on supervision from, e.g., 3D ground-truth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four components: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfel-based reconstruction of a scene from the latent code (iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed, called 3D-IQTT, to evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape's ability to solve scene reconstruction, generation, and understanding tasks.

2020-03-19

International Journal of Computer Vision (unknown)

doi.org

arxiv.org

Sidewalk Environment for Visual Navigation

Martin Weiß

Simon Chamorro

Roger Girgis

Margaux Luck

Samira Ebrahimi Kahou

Joseph Cohen

This dataset contains low and high resolution panoramic images, coordinates, labels and a connectivity graph. In order to run this simulated… (see more) environment, you will need at least one copy of the panoramic images, which are available in low (84x224 pixels) or high resolution (1280x3840 pixels). For more information, visit https://mweiss17.github.io/SEVN/.

2019-11-06

Polytechnique Montréal (accepted)

doi.org

Pix2Scene: Learning Implicit 3D Representations from Images

Sai Rajeswar

Fahim Mannan

2018-09-26

(published)