Portrait de Yoshua Bengio

Yoshua Bengio

Membre académique principal
Chaire en IA Canada-CIFAR
Professeur titulaire, Université de Montréal, Département d'informatique et de recherche opérationnelle
Fondateur et Conseiller scientifique, Équipe de direction
Sujets de recherche
Apprentissage automatique médical
Apprentissage de représentations
Apprentissage par renforcement
Apprentissage profond
Causalité
Modèles génératifs
Modèles probabilistes
Modélisation moléculaire
Neurosciences computationnelles
Raisonnement
Réseaux de neurones en graphes
Réseaux de neurones récurrents
Théorie de l'apprentissage automatique
Traitement du langage naturel
Site web
Google Scholar
LinkedIn
X

Biographie

*Pour toute demande média, veuillez écrire à medias@mila.quebec.

Pour plus d’information, contactez Marie-Josée Beauchamp, adjointe administrative à marie-josee.beauchamp@mila.quebec.

Reconnu comme une sommité mondiale en intelligence artificielle, Yoshua Bengio s’est surtout distingué par son rôle de pionnier en apprentissage profond, ce qui lui a valu le prix A. M. Turing 2018, le « prix Nobel de l’informatique », avec Geoffrey Hinton et Yann LeCun. Il est professeur titulaire à l’Université de Montréal, fondateur et conseiller scientifique de Mila – Institut québécois d’intelligence artificielle, et codirige en tant que senior fellow le programme Apprentissage automatique, apprentissage biologique de l'Institut canadien de recherches avancées (CIFAR). Il occupe également la fonction de conseiller spécial et directeur scientifique fondateur d’IVADO.

En 2018, il a été l’informaticien qui a recueilli le plus grand nombre de nouvelles citations au monde. En 2019, il s’est vu décerner le prestigieux prix Killam. Depuis 2022, il détient le plus grand facteur d’impact (h-index) en informatique à l’échelle mondiale. Il est fellow de la Royal Society de Londres et de la Société royale du Canada, et officier de l’Ordre du Canada.

Soucieux des répercussions sociales de l’IA et de l’objectif que l’IA bénéficie à tous, il a contribué activement à la Déclaration de Montréal pour un développement responsable de l’intelligence artificielle.

Étudiants actuels

Jamal Abou Haibeh
Collaborateur·rice alumni - McGill
Github
Google Scholar
Mohammed Abukalam
Collaborateur·rice alumni - UdeM
Github
Berkes Anaïs
Collaborateur·rice de recherche - Cambridge University
Superviseur⋅e principal⋅e :
David Rolnick
Site web
Github
Google Scholar
Rim Assouel
Doctorat - UdeM
Stefan Bauer
Visiteur de recherche indépendant
Co-superviseur⋅e :
Guillaume Lajoie
Google Scholar
Paul Bertin
Doctorat - UdeM
Shahana Chatterjee
Collaborateur·rice de recherche - N/A
Superviseur⋅e principal⋅e :
David Rolnick
Google Scholar
Xiaoyin Chen
Doctorat - UdeM
Sanghyeok Choi
Collaborateur·rice de recherche - KAIST
Site web
Github
Google Scholar
Tristan Deleu
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Aniket Didolkar
Doctorat - UdeM
Site web
Github
Google Scholar
Abdessamad EL KABID
Stagiaire de recherche - UdeM
Co-superviseur⋅e :
Loubna Benabbou
Github
Eric Elmoznino
Doctorat - UdeM
Co-superviseur⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Léna Ezzine
Doctorat - UdeM
Github
Jean-Pierre Falet
Doctorat - UdeM
Co-superviseur⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Leo Feng
Doctorat - UdeM
leo.feng@mila.quebec
Site web
Google Scholar
Ivan Grega
Stagiaire de recherche - UdeM
Github
Pietro Greiner
Doctorat
Google Scholar
Mohsin Hasan
Doctorat - UdeM
mohsin.hasan@mila.quebec
Site web
Github
Google Scholar
Edward Hu
Doctorat - UdeM
Moksh Jain
Doctorat - UdeM
moksh.jain@mila.quebec
Site web
Github
Google Scholar
Thomas Jiralerspong
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Younesse Kaddar
Collaborateur·rice alumni - UdeM
Site web
Github
Hyeonah Kim
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Alex Hernandez
Yaroslav KIVVA
Collaborateur·rice de recherche - UdeM
Github
Google Scholar
Salem Lahlou
Collaborateur·rice alumni - UdeM
Github
Tabitha Edith Lee
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Seanie Lee
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Chenghao Liu
Collaborateur·rice alumni
Site web
Github
Google Scholar
Zhen Liu
Collaborateur·rice alumni - UdeM
Superviseur⋅e principal⋅e :
Liam Paull
Kanika Madan
Doctorat - UdeM
Nikolay Malkin
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Cristian Dragos Manta
Doctorat - UdeM
Co-superviseur⋅e :
Dhanya Sridhar
Github
Sören Mindermann
Collaborateur·rice de recherche - UdeM
Site web
Google Scholar
Sarthak Mittal
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Jama Mohamud
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Mirco Ravanelli
Site web
Github
Google Scholar
Cyrus Neary
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Phong Nguyen
Visiteur de recherche indépendant - UdeM
Github
Padideh Nouri
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Doina Precup
Site web
Github
Google Scholar
Ali Parviz
Collaborateur·rice de recherche - Ying Wu Coll of Computing
Github
Google Scholar
Lena Podina
Doctorat - University of Waterloo
Superviseur⋅e principal⋅e :
David Rolnick
Github
Google Scholar
Nassim Rahaman
Collaborateur·rice alumni - Max-Planck-Institute for Intelligent Systems
Amine RAZIG
Stagiaire de recherche - UdeM
Co-superviseur⋅e :
Loubna Benabbou
Site web
Jarrid Rector-Brooks
Doctorat - UdeM
Danyal REHMAN
Postdoctorat - UdeM
Github
James Requeima
Visiteur de recherche indépendant - UdeM
Oli RICHARDSON
Postdoctorat - UdeM
Site web
Github
Google Scholar
Camille Rochefort-Boulanger
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Julie Hussin
Github
Luca Scimeca
Postdoctorat - UdeM
Site web
Github
Google Scholar
Dragos Secrieru
Maîtrise recherche - UdeM
Marcin Sendera
Collaborateur·rice alumni - UdeM
Github
Google Scholar
Divya Sharma
Postdoctorat
Site web
Github
Marco Stock
Visiteur de recherche indépendant - Technical University of Munich
marco.stock@tum.de
Github
Mélisande Astrid Crystal Teng
Doctorat - UdeM
Co-superviseur⋅e :
Hugo Larochelle
Site web
Github
Alex Tong
Postdoctorat - UdeM
alexander.tong@mila.quebec
Site web
Github
Google Scholar
Donna Vakalis
Postdoctorat - UdeM
Co-superviseur⋅e :
David Rolnick
Site web
Github
Google Scholar
Siddarth Venkatraman
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Omar G. Younis
Collaborateur·rice de recherche
Site web
Github
Google Scholar
Taeyoung YUN
Collaborateur·rice de recherche - UdeM
Github
Tianyu Zhang
Doctorat - UdeM
Site web
Github
Google Scholar
Nicole Zhang
Doctorat - McGill
Superviseur⋅e principal⋅e :
Mathieu Blanchette
Github
Google Scholar
Dinghuai Zhang
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Aaron Courville
Site web
Github
Harry Zhao
Collaborateur·rice alumni - McGill
Superviseur⋅e principal⋅e :
Doina Precup
Site web
Github
Google Scholar

Publications

A108 AUTOMATED DETECTION OF ILEOCECAL VALVE, APPENDICEAL ORIFICE, AND POLYP DURING COLONOSCOPY USING A DEEP LEARNING MODEL
Mahsa Taghiakbari
Sina Hamidi Ghalehjegh
E Jehanno
Tess Berthier
Lisa Di Jorio
Alan Barkun
Eric Deslandres
Simon Bouchard
Sacha Sidani
Yoshua Bengio
Daniel von Renteln
2023-03-01
Journal of the Canadian Association of Gastroenterology (publié)
doi.org
Proactive Contact Tracing
Prateek Gupta
Tegan Maharaj
Martin Weiss
Nasim Rahaman
Hannah Alsdurf
Nanor Minoyan
Soren Harnois-Leblanc
Joanna Merckx
Andrew Williams
Victor Schmidt
Pierre-Luc St-Charles
Akshay Patel
Yang Zhang
David Buckeridge
Chris Pal
Bernhard Schölkopf
Yoshua Bengio
The COVID-19 pandemic has spurred an unprecedented demand for interventions that can reduce disease spread without excessively restricting d… (voir plus)aily activity, given negative impacts on mental health and economic outcomes. Digital contact tracing (DCT) apps have emerged as a component of the epidemic management toolkit. Existing DCT apps typically recommend quarantine to all digitally-recorded contacts of test-confirmed cases. Over-reliance on testing may, however, impede the effectiveness of such apps, since by the time cases are confirmed through testing, onward transmissions are likely to have occurred. Furthermore, most cases are infectious over a short period; only a subset of their contacts are likely to become infected. These apps do not fully utilize data sources to base their predictions of transmission risk during an encounter, leading to recommendations of quarantine to many uninfected people and associated slowdowns in economic activity. This phenomenon, commonly termed as “pingdemic,” may additionally contribute to reduced compliance to public health measures. In this work, we propose a novel DCT framework, Proactive Contact Tracing (PCT), which uses multiple sources of information (e.g. self-reported symptoms, received messages from contacts) to estimate app users’ infectiousness histories and provide behavioral recommendations. PCT methods are by design proactive, predicting spread before it occurs. We present an interpretable instance of this framework, the Rule-based PCT algorithm, designed via a multi-disciplinary collaboration among epidemiologists, computer scientists, and behavior experts. Finally, we develop an agent-based model that allows us to compare different DCT methods and evaluate their performance in negotiating the trade-off between epidemic control and restricting population mobility. Performing extensive sensitivity analysis across user behavior, public health policy, and virological parameters, we compare Rule-based PCT to i) binary contact tracing (BCT), which exclusively relies on test results and recommends a fixed-duration quarantine, and ii) household quarantine (HQ). Our results suggest that both BCT and Rule-based PCT improve upon HQ, however, Rule-based PCT is more efficient at controlling spread of disease than BCT across a range of scenarios. In terms of cost-effectiveness, we show that Rule-based PCT pareto-dominates BCT, as demonstrated by a decrease in Disability Adjusted Life Years, as well as Temporary Productivity Loss. Overall, we find that Rule-based PCT outperforms existing approaches across a varying range of parameters. By leveraging anonymized infectiousness estimates received from digitally-recorded contacts, PCT is able to notify potentially infected users earlier than BCT methods and prevent onward transmissions. Our results suggest that PCT-based applications could be a useful tool in managing future epidemics.
2023-03-01
PLOS Digital Health (publié)
doi.org
Proactive Contact Tracing
Prateek Gupta
Tegan Maharaj
Martin Weiss
Nasim Rahaman
Hannah Alsdurf
Nanor Minoyan
Soren Harnois-Leblanc
Joanna Merckx
Andrew Williams
Victor Schmidt
Pierre-Luc St-Charles
Akshay Patel
Yang Zhang
David L Buckeridge
Chris Pal
Bernhard Schölkopf
Yoshua Bengio
The COVID-19 pandemic has spurred an unprecedented demand for interventions that can reduce disease spread without excessively restricting d… (voir plus)aily activity, given negative impacts on mental health and economic outcomes. Digital contact tracing (DCT) apps have emerged as a component of the epidemic management toolkit. Existing DCT apps typically recommend quarantine to all digitally-recorded contacts of test-confirmed cases. Over-reliance on testing may, however, impede the effectiveness of such apps, since by the time cases are confirmed through testing, onward transmissions are likely to have occurred. Furthermore, most cases are infectious over a short period; only a subset of their contacts are likely to become infected. These apps do not fully utilize data sources to base their predictions of transmission risk during an encounter, leading to recommendations of quarantine to many uninfected people and associated slowdowns in economic activity. This phenomenon, commonly termed as “pingdemic,” may additionally contribute to reduced compliance to public health measures. In this work, we propose a novel DCT framework, Proactive Contact Tracing (PCT), which uses multiple sources of information (e.g. self-reported symptoms, received messages from contacts) to estimate app users’ infectiousness histories and provide behavioral recommendations. PCT methods are by design proactive, predicting spread before it occurs. We present an interpretable instance of this framework, the Rule-based PCT algorithm, designed via a multi-disciplinary collaboration among epidemiologists, computer scientists, and behavior experts. Finally, we develop an agent-based model that allows us to compare different DCT methods and evaluate their performance in negotiating the trade-off between epidemic control and restricting population mobility. Performing extensive sensitivity analysis across user behavior, public health policy, and virological parameters, we compare Rule-based PCT to i) binary contact tracing (BCT), which exclusively relies on test results and recommends a fixed-duration quarantine, and ii) household quarantine (HQ). Our results suggest that both BCT and Rule-based PCT improve upon HQ, however, Rule-based PCT is more efficient at controlling spread of disease than BCT across a range of scenarios. In terms of cost-effectiveness, we show that Rule-based PCT pareto-dominates BCT, as demonstrated by a decrease in Disability Adjusted Life Years, as well as Temporary Productivity Loss. Overall, we find that Rule-based PCT outperforms existing approaches across a varying range of parameters. By leveraging anonymized infectiousness estimates received from digitally-recorded contacts, PCT is able to notify potentially infected users earlier than BCT methods and prevent onward transmissions. Our results suggest that PCT-based applications could be a useful tool in managing future epidemics.
2023-03-01
PLOS Digital Health (publié)
doi.org
Proactive Contact Tracing
Prateek Gupta
Tegan Maharaj
Martin Weiss
Nasim Rahaman
Hannah Alsdurf
Nanor Minoyan
Soren Harnois-Leblanc
Joanna Merckx
Andrew Williams
Victor Schmidt
Pierre-Luc St-Charles
Akshay Patel
Yang Zhang
David L Buckeridge
Chris Pal
Bernhard Schölkopf
Yoshua Bengio
The COVID-19 pandemic has spurred an unprecedented demand for interventions that can reduce disease spread without excessively restricting d… (voir plus)aily activity, given negative impacts on mental health and economic outcomes. Digital contact tracing (DCT) apps have emerged as a component of the epidemic management toolkit. Existing DCT apps typically recommend quarantine to all digitally-recorded contacts of test-confirmed cases. Over-reliance on testing may, however, impede the effectiveness of such apps, since by the time cases are confirmed through testing, onward transmissions are likely to have occurred. Furthermore, most cases are infectious over a short period; only a subset of their contacts are likely to become infected. These apps do not fully utilize data sources to base their predictions of transmission risk during an encounter, leading to recommendations of quarantine to many uninfected people and associated slowdowns in economic activity. This phenomenon, commonly termed as “pingdemic,” may additionally contribute to reduced compliance to public health measures. In this work, we propose a novel DCT framework, Proactive Contact Tracing (PCT), which uses multiple sources of information (e.g. self-reported symptoms, received messages from contacts) to estimate app users’ infectiousness histories and provide behavioral recommendations. PCT methods are by design proactive, predicting spread before it occurs. We present an interpretable instance of this framework, the Rule-based PCT algorithm, designed via a multi-disciplinary collaboration among epidemiologists, computer scientists, and behavior experts. Finally, we develop an agent-based model that allows us to compare different DCT methods and evaluate their performance in negotiating the trade-off between epidemic control and restricting population mobility. Performing extensive sensitivity analysis across user behavior, public health policy, and virological parameters, we compare Rule-based PCT to i) binary contact tracing (BCT), which exclusively relies on test results and recommends a fixed-duration quarantine, and ii) household quarantine (HQ). Our results suggest that both BCT and Rule-based PCT improve upon HQ, however, Rule-based PCT is more efficient at controlling spread of disease than BCT across a range of scenarios. In terms of cost-effectiveness, we show that Rule-based PCT pareto-dominates BCT, as demonstrated by a decrease in Disability Adjusted Life Years, as well as Temporary Productivity Loss. Overall, we find that Rule-based PCT outperforms existing approaches across a varying range of parameters. By leveraging anonymized infectiousness estimates received from digitally-recorded contacts, PCT is able to notify potentially infected users earlier than BCT methods and prevent onward transmissions. Our results suggest that PCT-based applications could be a useful tool in managing future epidemics.
2023-03-01
PLOS Digital Health (publié)
doi.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli
Stefano Massaroli
Eric Nguyen
Daniel Y Fu
Tri Dao
Stephen Baccus
Yoshua Bengio
Stefano Ermon
Christopher Re
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the c… (voir plus)ore building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.
2023-02-21
ArXiv (prépublication)
doi.org
arxiv.org
Stochastic Generative Flow Networks
Ling Pan
Dinghuai Zhang
Moksh J. Jain
Longbo Huang
Yoshua Bengio
Generative Flow Networks (or GFlowNets for short) are a family of probabilistic agents that learn to sample complex combinatorial structures… (voir plus) through the lens of"inference as control". They have shown great potential in generating high-quality and diverse candidates from a given energy landscape. However, existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with stochastic dynamics, which can limit their applicability. To overcome this challenge, this paper introduces Stochastic GFlowNets, a new algorithm that extends GFlowNets to stochastic environments. By decomposing state transitions into two steps, Stochastic GFlowNets isolate environmental stochasticity and learn a dynamics model to capture it. Extensive experimental results demonstrate that Stochastic GFlowNets offer significant advantages over standard GFlowNets as well as MCMC- and RL-based approaches, on a variety of standard benchmarks with stochastic dynamics.
2023-02-19
ArXiv (prépublication)
doi.org
arxiv.org
DEUP: Direct Epistemic Uncertainty Prediction
Moksh J. Jain
Salem Lahlou
Hadi Nekoei
Victor I Butoi
Paul Bertin
Jarrid Rector-Brooks
Maksym Korablyov
Yoshua Bengio
Epistemic Uncertainty is a measure of the lack of knowledge of a learner which diminishes with more evidence. While existing work focuses on… (voir plus) using the variance of the Bayesian posterior due to parameter uncertainty as a measure of epistemic uncertainty, we argue that this does not capture the part of lack of knowledge induced by model misspecification. We discuss how the excess risk, which is the gap between the generalization error of a predictor and the Bayes predictor, is a sound measure of epistemic uncertainty which captures the effect of model misspecification. We thus propose a principled framework for directly estimating the excess risk by learning a secondary predictor for the generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. We discuss the merits of this novel measure of epistemic uncertainty, and highlight how it differs from variance-based measures of epistemic uncertainty and addresses its major pitfall. Our framework, Direct Epistemic Uncertainty Prediction (DEUP) is particularly interesting in interactive learning environments, where the learner is allowed to acquire novel examples in each round. Through a wide set of experiments, we illustrate how existing methods in sequential model optimization can be improved with epistemic uncertainty estimates from DEUP, and how DEUP can be used to drive exploration in reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic image classification and predicting synergies of drug combinations.
2023-02-13
TMLR (accepté)
openreview.net
openreview.net