Portrait of Yoshua Bengio

Yoshua Bengio

Core Academic Member

Canada CIFAR AI Chair

Full Professor, Université de Montréal

Research Topics

Causality

Computational Neuroscience

Deep Learning

Generative Models

Graph Neural Networks

Machine Learning Theory

Medical Machine Learning

Molecular Modeling

Natural Language Processing

Probabilistic Models

Reasoning

Recurrent Neural Networks

Reinforcement Learning

Representation Learning

Biography

*For media requests, please write to medias@mila.quebec.

For more information please contact Cassidy MacNeil, Senior Assistant and Operation Lead at cassidy.macneil@mila.quebec.

Yoshua Bengio is recognized worldwide as a leading expert in AI. He is most known for his pioneering work in deep learning, which earned him the 2018 A.M. Turing Award, “the Nobel Prize of computing,” with Geoffrey Hinton and Yann LeCun.

Bengio is a full professor at Université de Montréal, and the founder and scientific advisor of Mila – Quebec Artificial Intelligence Institute. He is also a senior fellow at CIFAR and co-directs its Learning in Machines & Brains program, serves as special advisor and founding scientific director of IVADO, and holds a Canada CIFAR AI Chair.

In 2019, Bengio was awarded the prestigious Killam Prize and in 2022, he was the most cited computer scientist in the world by h-index. He is a Fellow of the Royal Society of London, Fellow of the Royal Society of Canada, Knight of the Legion of Honor of France and Officer of the Order of Canada. In 2023, he was appointed to the UN’s Scientific Advisory Board for Independent Advice on Breakthroughs in Science and Technology.

Concerned about the social impact of AI, Bengio helped draft the Montréal Declaration for the Responsible Development of Artificial Intelligence and continues to raise awareness about the importance of mitigating the potentially catastrophic risks associated with future AI systems.

Current Students

Jamal Abou Haibeh

Collaborating Alumni - McGill University

jamal.abouhaibeh@mila.quebec

Collaborating researcher - Cambridge University

Principal supervisor :

berkes.anais@mila.quebec

PhD - Université de Montréal

assouelr@mila.quebec

PhD - Université de Montréal

xiaoyin.chen@mila.quebec

Collaborating researcher - KAIST

sanghyeok.choi@mila.quebec

Aniket Didolkar

PhD - Université de Montréal

aniket.didolkar@mila.quebec

PhD - Université de Montréal

Co-supervisor :

Guillaume Lajoie

eric.elmoznino@mila.quebec

PhD - Université de Montréal

lena-nehale.ezzine@mila.quebec

Jean-Pierre Falet

PhD - Université de Montréal

jean-pierre.falet@mila.quebec

PhD

pietro.greiner@mila.quebec

PhD - Université de Montréal

mohsin.hasan@mila.quebec

PhD - Université de Montréal

moksh.jain@mila.quebec

Thomas Jiralerspong

PhD - Université de Montréal

Principal supervisor :

Guillaume Lajoie

thomas.jiralerspong@mila.quebec

Younesse Kaddar

Collaborating Alumni - Université de Montréal

younesse.kaddar@mila.quebec

Postdoctorate - Université de Montréal

Principal supervisor :

Alex Hernández-García

hyeonah.kim@mila.quebec

Collaborating researcher - Université de Montréal

minsu.kim@mila.quebec

Collaborating researcher - KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY

jeongjae.lee@mila.quebec

Collaborating researcher - s.o.

liusong299@gmail.com

Collaborating researcher

liucheng@mila.quebec

Cristian Dragos Manta

PhD - Université de Montréal

Co-supervisor :

cristian-dragos.manta@mila.quebec

PhD - Université de Montréal

Principal supervisor :

Guillaume Lajoie

mittalsa@mila.quebec

Independent visiting researcher - Université de Montréal

nguyenph@mila.quebec

Collaborating researcher - Ying Wu Coll of Computing

ali.parviz@mila.quebec

Collaborating researcher - University of Waterloo

Principal supervisor :

elena.podina@mila.quebec

Jarrid Rector-Brooks

PhD - Université de Montréal

jarrid.rector-brooks@mila.quebec

Postdoctorate - Université de Montréal

danyal.rehman@mila.quebec

Postdoctorate - Université de Montréal

oliver.richardson@mila.quebec

Camille Rochefort-Boulanger

PhD - Université de Montréal

Principal supervisor :

rochefoc@mila.quebec

Postdoctorate - Université de Montréal

Co-supervisor :

Alex Hernández-García

divya.sharma@mila.quebec

Mélisande Astrid Crystal Teng

Collaborating Alumni - Université de Montréal

Co-supervisor :

Hugo Larochelle

tengmeli@mila.quebec

Collaborating Alumni - Université de Montréal

alexander.tong@mila.quebec

Collaborating researcher - University of Cambridge

oszkar.urban@mila.quebec

Collaborating Alumni - Université de Montréal

Co-supervisor :

donna.vakalis@mila.quebec

Siddarth Venkatraman

PhD - Université de Montréal

Principal supervisor :

siddarth.venkatraman@mila.quebec

Collaborating researcher

joseph.viviano2@mila.quebec

Esmeralda Whitammer

Collaborating researcher - Université de Montréal

esmeralda.whitammer@mila.quebec

Collaborating Alumni

omar.younis@mila.quebec

Collaborating researcher - Université de Montréal

taeyoung.yun@mila.quebec

PhD - Université de Montréal

tianyu.zhang@mila.quebec

PhD - McGill University

Principal supervisor :

Mathieu Blanchette

xi.zhang@mila.quebec

Collaborating Alumni - McGill University

Principal supervisor :

zhaoming@mila.quebec

Blog Posts

Generic thumbnail for Mila Blog articles.

February 22, 2024

Skipper: Combining Spatial and Temporal Abstraction for Better Generalization

by

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Read the article

Scaling in the service of reasoning & model-based ML

April 4, 2023

Scaling in the Service of Reasoning & Model-Based ML

by

Read the article

A collaboration between Mila and Relation Therapeutics to discover novel synergistic combinations of drugs in vitro

March 23, 2022

A collaboration between Mila and Relation Therapeutics to discover novel synergistic combinations of drugs in vitro

by

Jake P. Taylor-King

Read the article

Generative Flow Networks

March 15, 2022

Generative Flow Networks

by

Read the article

Publications

Quantized Guided Pruning for Efficient Hardware Implementations of Deep Neural Networks

Ghouthi Boukli hacene

Matthieu Arzel

Nicolas Farrugia

Deep Neural Networks (DNNs) in general and Convolutional Neural Networks (CNNs) in particular are state-of-the-art in numerous computer visi… (see more)on tasks such as object classification and detection. However, the large amount of parameters they contain leads to a high computational complexity and strongly limits their usability in budget-constrained devices such as embedded devices. In this paper, we propose a combination of a pruning technique and a quantization scheme that effectively reduce the complexity and memory usage of convolutional layers of CNNs, by replacing the complex convolutional operation by a low-cost multiplexer. We perform experiments on CIFAR10, CIFAR100 and SVHN datasets and show that the proposed method achieves almost state-of-the-art accuracy, while drastically reducing the computational and memory footprints compared to the baselines. We also propose an efficient hardware architecture, implemented on Field Programmable Gate Arrays (FPGAs), to accelerate inference, which works as a pipeline and accommodates multiple layers working at the same time to speed up the inference process. In contrast with most proposed approaches which have used external memory or software defined memory controllers, our work is based on algorithmic optimization and full-hardware design, enabling a direct, on-chip memory implementation of a DNN while keeping close to state of the art accuracy.

2020-06-15

IEEE New Circuits and Systems Conference (published)

Learning Causal Models Online

Martha White

2020-06-11

ArXiv (preprint)

A Large-Scale, Open-Domain, Mixed-Interface Dialogue-Based ITS for STEM

Iulian V. Serban

Varun Gupta

Ekaterina Kochmar

Dung D. Vu

Robert Belfer

Aaron Courville

Laurent Charlin

2020-06-09

Artificial Intelligence in Education (published)

BigBrain 3D atlas of cortical layers: Cortical and laminar thickness gradients diverge in sensory and motor cortices

Konrad Wagstyl

Stéphanie Larocque

Guillem Cucurull

Claude Lepage

Joseph Paul Cohen

Sebastian Bludau

Nicola Palomero-Gallagher

Lindsay B. Lewis

Thomas Funck

Hannah Spitzer

Timo Dickscheid

Paul C. Fletcher

Adriana Romero

Karl Zilles

Katrin Amunts

Alan C. Evans

Histological atlases of the cerebral cortex, such as those made famous by Brodmann and von Economo, are invaluable for understanding human b… (see more)rain microstructure and its relationship with functional organization in the brain. However, these existing atlases are limited to small numbers of manually annotated samples from a single cerebral hemisphere, measured from 2D histological sections. We present the first whole-brain quantitative 3D laminar atlas of the human cerebral cortex. This atlas was derived from a 3D histological model of the human brain at 20 micron isotropic resolution (BigBrain), using a convolutional neural network to segment, automatically, the cortical layers in both hemispheres. Our approach overcomes many of the historical challenges with measurement of histological thickness in 2D and the resultant laminar atlas provides an unprecedented level of precision and detail. We utilized this BigBrain cortical atlas to test whether previously reported thickness gradients, as measured by MRI in sensory and motor processing cortices, were present in a histological atlas of cortical thickness, and which cortical layers were contributing to these gradients. Cortical thickness increased across sensory processing hierarchies, primarily driven by layers III, V and VI. In contrast, fronto-motor cortices showed the opposite pattern, with decreases in total and pyramidal layer thickness. These findings illustrate how this laminar atlas will provide a link between single-neuron morphology, mesoscale cortical layering, macroscopic cortical thickness and, ultimately, functional neuroanatomy.

2020-06-04

Public Library of Science (published)

On the interplay between noise and curvature and its effect on optimization and generalization

Valentin Thomas

Fabian Pedregosa

Bart Van Merriënboer

Pierre-Antoine Mangazol

Nicolas Le Roux

The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the v… (see more)ariance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ultimate goal is good generalization performance, we clarify how both curvature and noise are relevant to properly estimate the generalization gap. Realizing that the limitations of some existing works stems from a confusion between these matrices, we also clarify the distinction between the Fisher matrix, the Hessian, and the covariance matrix of the gradients.

2020-06-02

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

The need for privacy with public digital contact tracing during the COVID-19 pandemic

Richard Janda

Yun William Yu

Daphne Ippolito

Max Jarvie

Dan Pilat

Brooke Struck

Sekoul Krastev

Abhinav Sharma

2020-06-01

The Lancet Digital Health (published)

Training End-to-End Analog Neural Networks with Equilibrium Propagation

Jack D. Kendall

Ross D. Pantone

Kalpana Manickavasagam

Benjamin Scellier

We introduce a principled method to train end-to-end analog neural networks by stochastic gradient descent. In these analog neural networks,… (see more) the weights to be adjusted are implemented by the conductances of programmable resistive devices such as memristors [Chua, 1971], and the nonlinear transfer functions (or `activation functions') are implemented by nonlinear components such as diodes. We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models: they possess an energy function as a consequence of Kirchhoff's laws governing electrical circuits. This property enables us to train them using the Equilibrium Propagation framework [Scellier and Bengio, 2017]. Our update rule for each conductance, which is local and relies solely on the voltage drop across the corresponding resistor, is shown to compute the gradient of the loss function. Our numerical simulations, which use the SPICE-based Spectre simulation framework to simulate the dynamics of electrical circuits, demonstrate training on the MNIST classification task, performing comparably or better than equivalent-size software-based neural networks. Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning.

2020-06-01

ArXiv (preprint)

Multi-Image Super-Resolution for Remote Sensing using Deep Recurrent Networks

Md Rifat Arefin

Vincent Michalski

Pierre-Luc St-Charles

Alfredo Kalaitzis

Sookyung Kim

Samira E. Kahou

High-resolution satellite imagery is critical for various earth observation applications related to environment monitoring, geoscience, fore… (see more)casting, and land use analysis. However, the acquisition cost of such high-quality imagery due to the scarcity of providers and needs for high-frequency revisits restricts its accessibility in many fields. In this work, we present a data-driven, multi-image super resolution approach to alleviate these problems. Our approach is based on an end-to-end deep neural network that consists of an encoder, a fusion module, and a decoder. The encoder extracts co-registered highly efficient feature representations from low-resolution images of a scene. A Gated Re-current Unit (GRU)-based module acts as the fusion module, aggregating features into a combined representation. Finally, a decoder reconstructs the super-resolved image. The proposed model is evaluated on the PROBA-V dataset released in a recent competition held by the European Space Agency. Our results show that it performs among the top contenders and offers a new practical solution for real-world applications.

2020-05-31

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (published)

Multi-Task Self-Supervised Learning for Robust Speech Recognition

Mirco Ravanelli

Santiago Pascual

Pawel Swietojanski

Joao Monteiro

Jan Trmal

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To t… (see more)ake a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation. Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.

2020-05-03

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (published)

Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation

Maxence Ernoult

Julie Grollier

Damien Querlioz

Benjamin Scellier

Equilibrium Propagation (EP) is a biologically inspired alternative algorithm to backpropagation (BP) for training neural networks. It appli… (see more)es to RNNs fed by a static input x that settle to a steady state, such as Hopfield networks. EP is similar to BP in that in the second phase of training, an error signal propagates backwards in the layers of the network, but contrary to BP, the learning rule of EP is spatially local. Nonetheless, EP suffers from two major limitations. On the one hand, due to its formulation in terms of real-time dynamics, EP entails long simulation times, which limits its applicability to practical tasks. On the other hand, the biological plausibility of EP is limited by the fact that its learning rule is not local in time: the synapse update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically. Our work addresses these two issues and aims at widening the spectrum of EP from standard machine learning models to more bio-realistic neural networks. First, we propose a discrete-time formulation of EP which enables to simplify equations, speed up training and extend EP to CNNs. Our CNN model achieves the best performance ever reported on MNIST with EP. Using the same discrete-time formulation, we introduce Continual Equilibrium Propagation (C-EP): the weights of the network are adjusted continually in the second phase of training using local information in space and time. We show that in the limit of slow changes of synaptic strengths and small nudging, C-EP is equivalent to BPTT (Theorem 1). We numerically demonstrate Theorem 1 and C-EP training on MNIST and generalize it to the bio-realistic situation of a neural network with asymmetric connections between neurons.

2020-04-28

ArXiv (preprint)

Equilibrium Propagation with Continual Weight Updates

Maxence Ernoult

Julie Grollier

Damien Querlioz

Benjamin Scellier

Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching… (see more) those of Backpropagation Through Time (BPTT), but with a learning rule local in space. Given an input

2020-04-28

arXiv (preprint)

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims

Miles Brundage

Shahar Avin

Jasmine Wang

Haydn Belfield

Gretchen Krueger

Gillian Hadfield

Heidy Khlaaf

Jingying Yang

Helen Toner

Ruth Fong

Pang Wei Koh

Sara Hooker

Jade Leung

Andrew Trask

Emma Bluemke

Jonathan Lebensold

Cullen O’Keefe

Mark Koren

Théo Ryffel … (see 39 more)

JB Rubinovitz

Tamay Besiroglu

Federica Carugati

Jack Clark

Peter Eckersley

Sarah de Haas

Maritza Johnson

Ben Laurie

Alex Ingerman

Igor Krawczuk

Amanda Askell

Rosario Cammarota

Andrew Lohn

David Krueger

Charlotte Stix

Peter Henderson

Logan Graham

Carina Prunkl

Bianca Martin

Elizabeth Seger

Noa Zilberman

Seán Ó hÉigeartaigh

Frens Kroeger

Girish Sastry

Rebecca Kagan

Adrian Weller

Brian Tse

Elizabeth Barnes

Allan Dafoe

Paul Scharre

Ariel Herbert-Voss

Martijn Rasser

Carrick Flynn

Thomas Krendl Gilbert

Lisa Dyer

Saif Khan

Markus Anderljung

With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and … (see more)recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development. In order for AI developers to earn trust from system users, customers, civil society, governments, and other stakeholders that they are building AI responsibly, they will need to make verifiable claims to which they can be held accountable. Those outside of a given organization also need effective means of scrutinizing such claims. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems and their associated development processes, with a focus on providing evidence about the safety, security, fairness, and privacy protection of AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.

2020-04-14

arXiv (preprint)