Derek Nowrouzezahrai

Core Academic Member

derek@mila.quebec

Canada CIFAR AI Chair

Associate Professor, McGill University, Department of Electrical and Computer Engineering

Research Topics

Computational Photography

Computer Vision

Deep Learning

Dynamical Systems

Generative Models

Reinforcement Learning

Representation Learning

Google Scholar

Biography

Derek Nowrouzezahrai is a full professor at McGill University, where he directs the Centre for Intelligent Machines and co-directs the Graphics Lab.

He is also a Canada CIFAR AI Chair and holds the Ubisoft–Mila research Chair, Scaling Game Worlds with Responsible AI.

Nowrouzezahrai’s research tackles the simulation of various physical phenomena, such as the dynamics of moving objects and the simulation of lighting for realistic image synthesis, which have applications in virtual reality, video games, fluid simulation and control, digital manufacturing, computationally augmented optics and geometry processing. He is also interested in the development of differentiable simulators of these dynamical systems and their applications to inverse problems in robotics and vision.

This work relies fundamentally on developing high performance and sample efficient (Markov chain) Monte Carlo-based methods, high-order statistics and computational methods for complex multi-dimensional integration problems, differentiable physics-based simulators and numerical methods for dynamical systems, and on applying machine learning to 3D, visual and interactive media.

Current Students

Daniel Bairamian

PhD - McGill University

Paul Barde

Collaborating researcher - McGill University

Co-supervisor :

PhD - Université de Montréal

Principal supervisor :

PhD - McGill University

Xiangyu (Daniel) Kong

PhD - McGill University

Github

Joey Marten

Master's Research - McGill University

Co-supervisor :

Noam Aigerman

Github

Hossam Mohamed S M Ahmed

PhD - McGill University

Github

Somjit Nath

PhD - McGill University

Principal supervisor :

Samira Ebrahimi Kahou

Website

Google Scholar

Yasaman Sabbagh Ziarani

PhD - McGill University

Yuanyuan Tao

PhD - McGill University

Mattie Tesfaldet

Collaborating researcher - McGill University

Co-supervisor :

Junha Yoo

Master's Research - McGill University

Co-supervisor :

James Richard Forbes

Blog Posts

An illustration of the forward and backward processes of a differentiable renderer.

January 29, 2025

Differentiable Visual Computing: Bridging 2D and 3D in Machine Learning Applications

Jordan J. Bannister

Derek Nowrouzezahrai

Read the article

Publications

Overcoming Challenges in Leveraging GANs for Few-Shot Data Augmentation

Pau Rodríguez

Christopher Pal

In this paper, we explore the use of GAN-based few-shot data augmentation as a method to improve few-shot classification performance. We per… (see more)form an exploration into how a GAN can be fine-tuned for such a task (one of which is in a class-incremental manner), as well as a rigorous empirical investigation into how well these models can perform to improve few-shot classification. We identify issues related to the difficulty of training such generative models under a purely supervised regime with very few examples, as well as issues regarding the evaluation protocols of existing works. We also find that in this regime, classification accuracy is highly sensitive to how the classes of the dataset are randomly split. Therefore, we propose a semi-supervised fine-tuning approach as a more pragmatic way forward to address these problems.

2021-12-31

CoLLAs (published)

doi.org

proceedings.mlr.press

Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments

Martin Weiss

Simon Chamorro

Roger Girgis

Margaux Luck

Samira E. Kahou

Joseph P. Cohen

Millions of blind and visually-impaired (BVI) people navigate urban environments every day, using smartphones for high-level path-planning a… (see more)nd white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavor to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.

2020-05-11

Proceedings of the Conference on Robot Learning (published)

doi.org

proceedings.mlr.press

Pix2Shape – Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation

Sai Rajeswar

Fahim Mannan

Florian Golemo

Jérôme Parent-lévesque

David Vázquez

Derek Nowrouzezahrai

Aaron Courville

We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-expl… (see more)ored, with most prior work relying on supervision from, e.g., 3D ground-truth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four components: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfel-based reconstruction of a scene from the latent code (iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed, called 3D-IQTT, to evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape's ability to solve scene reconstruction, generation, and understanding tasks.

2020-03-19

International Journal of Computer Vision (unknown)

doi.org

arxiv.org

Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Christopher Pal

Adversarial Imitation Learning alternates between learning a discriminator -- which tells apart expert's demonstrations from generated ones … (see more)-- and a generator's policy to produce trajectories that can fool this discriminator. This alternated optimization is known to be delicate in practice since it compounds unstable adversarial training with brittle and sample-inefficient reinforcement learning. We propose to remove the burden of the policy optimization steps by leveraging a novel discriminator formulation. Specifically, our discriminator is explicitly conditioned on two policies: the one from the previous generator's iteration and a learnable policy. When optimized, this discriminator directly learns the optimal generator's policy. Consequently, our discriminator's update solves the generator's optimization problem for free: learning a policy that imitates the expert does not require an additional optimization loop. This formulation effectively cuts by half the implementation and computational burden of Adversarial Imitation Learning algorithms by removing the Reinforcement Learning phase altogether. We show on a variety of tasks that our simpler approach is competitive to prevalent Imitation Learning methods.

2019-12-31

Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (published)

doi.org

arxiv.org

Pix2Scene: Learning Implicit 3D Representations from Images

Sai Rajeswar

Fahim Mannan

2018-09-26