Yoshua Bengio

jamal.abouhaibeh@mila.quebec

Biography

*For media requests, please write to medias@mila.quebec.

For more information please contact Cassidy MacNeil, Senior Assistant and Operation Lead at cassidy.macneil@mila.quebec.

Yoshua Bengio is recognized worldwide as a leading expert in AI. He is most known for his pioneering work in deep learning, which earned him the 2018 A.M. Turing Award, “the Nobel Prize of computing,” with Geoffrey Hinton and Yann LeCun.

Bengio is a full professor at Université de Montréal, and the founder and scientific advisor of Mila – Quebec Artificial Intelligence Institute. He is also a senior fellow at CIFAR and co-directs its Learning in Machines & Brains program, serves as special advisor and founding scientific director of IVADO, and holds a Canada CIFAR AI Chair.

In 2019, Bengio was awarded the prestigious Killam Prize and in 2022, he was the most cited computer scientist in the world by h-index. He is a Fellow of the Royal Society of London, Fellow of the Royal Society of Canada, Knight of the Legion of Honor of France and Officer of the Order of Canada. In 2023, he was appointed to the UN’s Scientific Advisory Board for Independent Advice on Breakthroughs in Science and Technology.

Concerned about the social impact of AI, Bengio helped draft the Montréal Declaration for the Responsible Development of Artificial Intelligence and continues to raise awareness about the importance of mitigating the potentially catastrophic risks associated with future AI systems.

Current Students

Jamal Abou Haibeh

Collaborating Alumni - McGill University

Collaborating researcher - Cambridge University

Principal supervisor :

David Rolnick

berkes.anais@mila.quebec

sanghyeok.choi@mila.quebec

Rim Assouel

PhD - Université de Montréal

assouelr@mila.quebec

Xiaoyin Chen

PhD - Université de Montréal

xiaoyin.chen@mila.quebec

Sanghyeok Choi

Collaborating researcher - KAIST

PhD - Université de Montréal

aniket.didolkar@mila.quebec

PhD - Université de Montréal

Co-supervisor :

Guillaume Lajoie

eric.elmoznino@mila.quebec

PhD - Université de Montréal

lena-nehale.ezzine@mila.quebec

Jean-Pierre Falet

PhD - Université de Montréal

jean-pierre.falet@mila.quebec

PhD

pietro.greiner@mila.quebec

Mohsin Hasan

PhD - Université de Montréal

mohsin.hasan@mila.quebec

thomas.jiralerspong@mila.quebec

Moksh Jain

PhD - Université de Montréal

moksh.jain@mila.quebec

PhD - Université de Montréal

Principal supervisor :

Guillaume Lajoie

Collaborating Alumni - Université de Montréal

younesse.kaddar@mila.quebec

Hyeonah Kim

Postdoctorate - Université de Montréal

Principal supervisor :

Alex Hernández-García

hyeonah.kim@mila.quebec

Minsu Kim

Collaborating researcher - Université de Montréal

minsu.kim@mila.quebec

Collaborating researcher - KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY

jeongjae.lee@mila.quebec

cristian-dragos.manta@mila.quebec

Song LIU

Collaborating researcher - s.o.

Collaborating researcher

Cristian Dragos Manta

PhD - Université de Montréal

Co-supervisor :

Dhanya Sridhar

Sarthak Mittal

PhD - Université de Montréal

Principal supervisor :

Independent visiting researcher - Université de Montréal

nguyenph@mila.quebec

Ali Parviz

Collaborating researcher - Ying Wu Coll of Computing

ali.parviz@mila.quebec

Lena Podina

Collaborating researcher - University of Waterloo

Principal supervisor :

David Rolnick

elena.podina@mila.quebec

jarrid.rector-brooks@mila.quebec

Jarrid Rector-Brooks

PhD - Université de Montréal

Danyal REHMAN

Postdoctorate - Université de Montréal

danyal.rehman@mila.quebec

oliver.richardson@mila.quebec

Oli RICHARDSON

Postdoctorate - Université de Montréal

Camille Rochefort-Boulanger

PhD - Université de Montréal

Principal supervisor :

Postdoctorate - Université de Montréal

Co-supervisor :

Alex Hernández-García

divya.sharma@mila.quebec

Mélisande Astrid Crystal Teng

Collaborating Alumni - Université de Montréal

Co-supervisor :

Alex Tong

Collaborating Alumni - Université de Montréal

alexander.tong@mila.quebec

Collaborating researcher - University of Cambridge

oszkar.urban@mila.quebec

donna.vakalis@mila.quebec

Donna Vakalis

Collaborating Alumni - Université de Montréal

Co-supervisor :

David Rolnick

PhD - Université de Montréal

Principal supervisor :

Glen Berseth

siddarth.venkatraman@mila.quebec

Collaborating researcher

joseph.viviano2@mila.quebec

Collaborating researcher - Université de Montréal

esmeralda.whitammer@mila.quebec

Collaborating Alumni

omar.younis@mila.quebec

Collaborating researcher - Université de Montréal

taeyoung.yun@mila.quebec

Nicole Zhang

PhD - McGill University

Principal supervisor :

PhD - Université de Montréal

tianyu.zhang@mila.quebec

Skipper: Combining Spatial and Temporal Abstraction for Better Generalization

Harry Zhao

Collaborating Alumni - McGill University

Principal supervisor :

Blog Posts

Generic thumbnail for Mila Blog articles.

February 22, 2024

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Scaling in the Service of Reasoning & Model-Based ML

April 4, 2023

Yoshua Bengio

Edward J. Hu

A collaboration between Mila and Relation Therapeutics to discover novel synergistic combinations of drugs in vitro

March 23, 2022

Paul Bertin

Jake P. Taylor-King

Yoshua Bengio

March 15, 2022

Generative Flow Networks

Yoshua Bengio

Publications

h-detach: Modifying the LSTM Gradient Towards Better Optimization

Nan Rosemary Ke

Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident i… (see more)n tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithm\footnote{Our code is available at this https URL.} prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.

2018-12-31

ICLR.cc/2019/Conference (poster)

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Devansh Arpit

Vı́ctor Campos

Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initial… (see more)ization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy. Similarly, initialization for ResNets have also been studied for un-normalized networks and often under simplified settings ignoring the shortcut connection. To address these issues, we propose a novel parameter initialization strategy that avoids explosion/vanishment of information across layers for weight normalized networks with and without residual connections. The proposed strategy is based on a theoretical analysis using mean field approximation. We run over 2,500 experiments and evaluate our proposal on image datasets showing that the proposed initialization outperforms existing initialization methods in terms of generalization performance, robustness to hyper-parameter values and variance between seeds, especially when networks get deeper in which case existing methods fail to even start training. Finally, we show that using our initialization in conjunction with learning rate warmup is able to reduce the gap between the performance of weight normalized and batch normalized networks.

2018-12-31

NeurIPS (published)

arxiv.org

InfoBot: Structured Exploration in ReinforcementLearning Using Information Bottleneck

D. Strouse

Matthew Botvinick

Sergey Levine

2018-12-31

(published)

www.semanticscholar.org

InfoBot: Transfer and Exploration via the Information Bottleneck

Daniel Strouse

Matthew Botvinick

Sergey Levine

A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed. We postula… (see more)te that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}. These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions. We propose to learn about decision states from prior experience. By training a goal-conditioned policy with an information bottleneck, we can identify decision states by examining where the model actually leverages the goal state. We find that this simple mechanism effectively identifies decision states, even in partially observed settings. In effect, the model learns the sensory cues that correlate with potential subgoals. In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision states and through new regions of the state space.

2018-12-31

ICLR.cc/2019/Conference (poster)

doi.org

Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Accuracy

Alex Lamb

Vikas Verma

Juho Kannala

Adversarial robustness has become a central goal in deep learning, both in theory and practice. However, successful methods to improve adver… (see more)sarial robustness (such as adversarial training) greatly hurt generalization performance on the clean data. This could have a major impact on how adversarial robustness affects real world systems (i.e. many may opt to forego robustness if it can improve performance on the clean data). We propose Interpolated Adversarial Training, which employs recently proposed interpolation based training methods in the framework of adversarial training. On CIFAR-10, adversarial training increases clean test error from 5.8% to 16.7%, whereas with our Interpolated adversarial training we retain adversarial robustness while achieving a clean test error of only 6.5%. With our technique, the relative error increase for the robust model is reduced from 187.9% to just 12.1%.

2018-12-31

arXiv.org (preprint)

dblp.uni-trier.de

Lagrangian neurodynamics for real-time error-backpropagation across cortical areas

Dominik Dold

Akos F. Kungl

João Sacramento

Mihai A. Petrovici

Kaspar Anton Schindler

Jonathan Binas

Walter Senn

2018-12-31

(published)

www.semanticscholar.org

Learning deep representations by mutual information estimation and maximization

R Devon Hjelm

Alex Fedorov

Samuel Lavoie-Marchildon

Karan Grewal

Adam Trischler

Phil Bachman

This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep n… (see more)eural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

2018-12-31

ICLR (published)

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Kundan Kumar

Rithesh Kumar

Thibault de Boissiere

Lucas Gestin

Wei Zhen Teoh

Jose Sotelo

Alexandre De Brébisson

Aaron Courville

2018-12-31

Neural Information Processing Systems (published)

doi.org

arxiv.org

Modeling the Long Term Future in Model-Based Reinforcement Learning

Nan Rosemary Ke

Amanpreet Singh

Ahmed Touati

Anirudh Goyal

Devi Parikh

Dhruv Batra

2018-12-31

International Conference on Learning Representations (poster)

Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

Christopher Pal

Neural generative models have been become increasingly popular when building conversational agents. They offer flexibility, can be easily ad… (see more)apted to new domains, and require minimal domain engineering. A common criticism of these systems is that they seldom understand or use the available dialog history effectively. In this paper, we take an empirical approach to understanding how these models use the available dialog history by studying the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that commonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diagnostic tool for evaluating dialog systems in the future.

2018-12-31

Association for Computational Linguistics (published)

doi.org

arxiv.org

Probabilistic Planning with Sequential Monte Carlo Methods

Alexandre Piché

Valentin Thomas

Cyril Ibrahim

Chris Pal

2018-12-31

International Conference on Learning Representations (poster)

Quaternion Recurrent Neural Networks

Titouan Parcollet

Mirco Ravanaelli

Mohamed Morchid

Georges Linarès

Chiheb Trabelsi

Renato De Mori

Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term d… (see more)ependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN), alongside with a quaternion long-short term memory neural network (QLSTM), that take into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition. Finally, we show that QRNN and QLSTM reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and LSTMs to reach better results, leading to a more compact representation of the relevant information.

2018-12-31

ICLR.cc/2019/Conference (poster)