Portrait de Yoshua Bengio

Yoshua Bengio

Membre académique principal
Chaire en IA Canada-CIFAR
Professeur titulaire, Université de Montréal, Département d'informatique et de recherche opérationnelle
Fondateur et Conseiller scientifique, Équipe de direction
Sujets de recherche
Apprentissage automatique médical
Apprentissage de représentations
Apprentissage par renforcement
Apprentissage profond
Causalité
Modèles génératifs
Modèles probabilistes
Modélisation moléculaire
Neurosciences computationnelles
Raisonnement
Réseaux de neurones en graphes
Réseaux de neurones récurrents
Théorie de l'apprentissage automatique
Traitement du langage naturel
Site web
Google Scholar

Biographie

*Pour toute demande média, veuillez écrire à medias@mila.quebec.

Pour plus d’information, contactez Marie-Josée Beauchamp, adjointe administrative à marie-josee.beauchamp@mila.quebec.

Reconnu comme une sommité mondiale en intelligence artificielle, Yoshua Bengio s’est surtout distingué par son rôle de pionnier en apprentissage profond, ce qui lui a valu le prix A. M. Turing 2018, le « prix Nobel de l’informatique », avec Geoffrey Hinton et Yann LeCun. Il est professeur titulaire à l’Université de Montréal, fondateur et conseiller scientifique de Mila – Institut québécois d’intelligence artificielle, et codirige en tant que senior fellow le programme Apprentissage automatique, apprentissage biologique de l'Institut canadien de recherches avancées (CIFAR). Il occupe également la fonction de conseiller spécial et directeur scientifique fondateur d’IVADO.

En 2018, il a été l’informaticien qui a recueilli le plus grand nombre de nouvelles citations au monde. En 2019, il s’est vu décerner le prestigieux prix Killam. Depuis 2022, il détient le plus grand facteur d’impact (h-index) en informatique à l’échelle mondiale. Il est fellow de la Royal Society de Londres et de la Société royale du Canada, et officier de l’Ordre du Canada.

Soucieux des répercussions sociales de l’IA et de l’objectif que l’IA bénéficie à tous, il a contribué activement à la Déclaration de Montréal pour un développement responsable de l’intelligence artificielle.

Étudiants actuels

Jamal Abou Haibeh
Collaborateur·rice alumni - McGill
Github
Google Scholar
Mohammed Abukalam
Collaborateur·rice alumni - UdeM
Github
Berkes Anaïs
Collaborateur·rice de recherche - Cambridge University
Superviseur⋅e principal⋅e :
David Rolnick
Site web
Github
Google Scholar
Rim Assouel
Doctorat - UdeM
Stefan Bauer
Visiteur de recherche indépendant
Co-superviseur⋅e :
Guillaume Lajoie
Google Scholar
Paul Bertin
Doctorat - UdeM
Shahana Chatterjee
Collaborateur·rice de recherche - N/A
Superviseur⋅e principal⋅e :
David Rolnick
Google Scholar
Xiaoyin Chen
Doctorat - UdeM
Sanghyeok Choi
Collaborateur·rice de recherche - KAIST
Site web
Github
Google Scholar
Tristan Deleu
Doctorat - UdeM
Site web
Github
Google Scholar
Aniket Didolkar
Doctorat - UdeM
Site web
Github
Google Scholar
Abdessamad EL KABID
Stagiaire de recherche - UdeM
Co-superviseur⋅e :
Loubna Benabbou
Github
Eric Elmoznino
Doctorat - UdeM
Co-superviseur⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Akram Erraqabi
Doctorat - UdeM
Léna Ezzine
Doctorat - UdeM
Github
Jean-Pierre Falet
Doctorat - UdeM
Co-superviseur⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Leo Feng
Doctorat - UdeM
leo.feng@mila.quebec
Site web
Google Scholar
Ivan Grega
Stagiaire de recherche - UdeM
Github
Pietro Greiner
Doctorat
Mohsin Hasan
Doctorat - UdeM
mohsin.hasan@mila.quebec
Site web
Github
Google Scholar
Edward Hu
Doctorat - UdeM
Moksh Jain
Doctorat - UdeM
moksh.jain@mila.quebec
Site web
Github
Google Scholar
Thomas Jiralerspong
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Younesse Kaddar
Collaborateur·rice alumni - UdeM
Site web
Github
Minsu Kim
Stagiaire de recherche - UdeM
Site web
Github
Google Scholar
Hyeonah Kim
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Alex Hernandez
Yaroslav KIVVA
Collaborateur·rice de recherche - UdeM
Github
Google Scholar
Salem Lahlou
Collaborateur·rice alumni - UdeM
Github
Tabitha Edith Lee
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Seanie Lee
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Chenghao Liu
Collaborateur·rice alumni
Site web
Github
Google Scholar
Zhen Liu
Collaborateur·rice alumni - UdeM
Superviseur⋅e principal⋅e :
Liam Paull
Kanika Madan
Doctorat - UdeM
Nikolay Malkin
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Cristian Dragos Manta
Doctorat - UdeM
Co-superviseur⋅e :
Dhanya Sridhar
Github
Sören Mindermann
Collaborateur·rice de recherche - UdeM
Site web
Google Scholar
Sarthak Mittal
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Guillaume Lajoie
Site web
Github
Google Scholar
Jama Mohamud
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Mirco Ravanelli
Site web
Github
Google Scholar
Cyrus Neary
Postdoctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Phong Nguyen
Visiteur de recherche indépendant - UdeM
Padideh Nouri
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Doina Precup
Site web
Github
Google Scholar
Ali Parviz
Collaborateur·rice de recherche - Ying Wu Coll of Computing
Google Scholar
Lena Podina
Doctorat - University of Waterloo
Superviseur⋅e principal⋅e :
David Rolnick
Github
Google Scholar
Nassim Rahaman
Collaborateur·rice alumni - Max-Planck-Institute for Intelligent Systems
Jarrid Rector-Brooks
Doctorat - UdeM
Danyal REHMAN
Postdoctorat - UdeM
Github
James Requeima
Visiteur de recherche indépendant - UdeM
Oli RICHARDSON
Postdoctorat - UdeM
Site web
Github
Google Scholar
Camille Rochefort-Boulanger
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Julie Hussin
Github
Victor Schmidt
Collaborateur·rice alumni - UdeM
Site web
Github
Google Scholar
Luca Scimeca
Postdoctorat - UdeM
Site web
Github
Google Scholar
Dragos Secrieru
Maîtrise recherche - UdeM
Marcin Sendera
Collaborateur·rice alumni - UdeM
Github
Google Scholar
Vedant Shah
Maîtrise recherche - UdeM
Site web
Github
Google Scholar
Divya Sharma
Postdoctorat
Site web
Github
Marco Stock
Visiteur de recherche indépendant - Technical University of Munich
marco.stock@tum.de
Github
Mélisande Astrid Crystal Teng
Doctorat - UdeM
Co-superviseur⋅e :
Hugo Larochelle
Site web
Github
Alex Tong
Postdoctorat - UdeM
alexander.tong@mila.quebec
Site web
Github
Google Scholar
Donna Vakalis
Postdoctorat - UdeM
Co-superviseur⋅e :
David Rolnick
Site web
Github
Google Scholar
Siddarth Venkatraman
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Glen Berseth
Site web
Github
Google Scholar
Leah WAIRIMU
Collaborateur·rice de recherche - UdeM
Github
Omar G. Younis
Collaborateur·rice de recherche
Site web
Github
Google Scholar
Taeyoung YUN
Collaborateur·rice de recherche - KAIST
Github
Nicole Zhang
Doctorat - McGill
Superviseur⋅e principal⋅e :
Mathieu Blanchette
Github
Google Scholar
Dinghuai Zhang
Doctorat - UdeM
Superviseur⋅e principal⋅e :
Aaron Courville
Site web
Github
Tianyu Zhang
Doctorat - UdeM
Site web
Github
Google Scholar
Harry Zhao
Doctorat - McGill
Superviseur⋅e principal⋅e :
Doina Precup
Site web
Github
Google Scholar

Publications

Integrating Generative and Experimental Platforms for Biomolecular Design
Cheng-Hao Liu
Jarrid Rector-Brooks
Soojung Yang
Sidney L Lisanza
Francesca-Zhoufan Li
Hannes Stärk
Jacob Gershon
Lauren Hong
Pranam Chatterjee
Tommi Jaakkola
Regina Barzilay
David Baker
Frances H. Arnold
Yoshua Bengio
Biomolecular design, through artificial engineering of proteins, ligands, and nucleic acids, holds immense promise in addressing pressing me… (voir plus)dical, industrial, and environmental challenges. While generative machine learning has shown significant potential in this area, a palpable disconnect exists with experimental biology: many ML research efforts prioritize static benchmark performance, potentially sidelining impactful biological applications. This workshop seeks to bridge this gap by bringing computationalists and experimentalists together, catalyzing a deeper interdisciplinary discourse. Together, we will explore the strengths and challenges of generative ML in biology, experimental integration of generative ML, and biological problems ready for ML. To attract high-quality and diverse research, we partnered with Nature Biotechnology for a special collection, and we created dedicated tracks for in-silico ML research and hybrid ML-experimental biology research. Our lineup features emerging leaders as speakers and renowned scientists as panelists, encapsulating a spectrum from high-throughput experimentation and computational biology to generative ML. With a diverse organizing team and backed by industry sponsors, we dedicate the workshop to pushing the boundaries of ML's role in biology.
2025-01-01
ICLR.cc/2025/Workshop_Proposals (publié)
openreview.net
openreview.net
Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
Brian R. Bartoldson
Siddarth Venkatraman
James Diffenderfer
Moksh J. Jain
Tal Ben-Nun
Seanie Lee
Minsu Kim
Johan Samir Obando Ceron
Yoshua Bengio
Bhavya Kailkhura
2025-01-01
arXiv.org (prépublication)
doi.org
A neuronal least-action principle for real-time learning in cortical circuits
Walter Senn
Dominik Dold
Akos F. Kungl
Benjamin Ellenberger
Jakob Jordan
Yoshua Bengio
João Sacramento
Mihai A. Petrovici
One of the most fundamental laws of physics is the principle of least action. Motivated by its predictive power, we introduce a neuronal lea… (voir plus)st-action principle for cortical processing of sensory streams to produce appropriate behavioural outputs in real time. The principle postulates that the voltage dynamics of cortical pyramidal neurons prospectively minimize the local somato-dendritic mismatch error within individual neurons. For motor output neurons, it implies minimizing an instantaneous behavioural error. For deep network neurons, it implies a prospective firing to overcome integration delays and correct for possible output errors right in time. The neuron-specific errors are extracted in the apical dendrites of pyramidal neurons through a cortical microcircuit that tries to explain away the feedback from the periphery, and correct the trajectory on the fly. Any motor output is in a moving equilibrium with the sensory inputs and the motor feedback during the whole sensory-motor trajectory. Ongoing synaptic plasticity reduces the somato-dendritic mismatch error within each cortical neuron and performs gradient descent on the output cost at any moment in time. The neuronal least-action principle offers an axiomatic framework to derive local neuronal and synaptic dynamics for global real-time computation and learning in the brain and in physical substrates in general.
2024-12-20
eLife (publié)
doi.org
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
Zhen Liu
Tim Z. Xiao
Weiyang Liu
Yoshua Bengio
Dinghuai Zhang
While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetun… (voir plus)e pretrained diffusion models on some reward functions that are either designed by experts or learned from small-scale datasets. Existing methods for finetuning diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as
2024-12-10
ArXiv (prépublication)
arxiv.org
arxiv.org
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
Zhen Liu
Tim Z. Xiao
Weiyang Liu
Yoshua Bengio
Dinghuai Zhang
While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetun… (voir plus)e pretrained diffusion models on some reward functions that are either designed by experts or learned from small-scale datasets. Existing methods for finetuning diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as
2024-12-10
ArXiv (prépublication)
doi.org
arxiv.org
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte
Franccois Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
M. L. Richter
Saverio Vadacchino
Shubbam Agarwal
Sanket Biswas … (voir 23 de plus)
Sara Shanian
Ying Zhang
Noah Bolger
Kurt MacDonald
Simon Fauvel
Sathwik Tejaswi
Srinivas Sunkara
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharagani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Chris Pal
Issam Hadj Laradji
Spandanna Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows,… (voir plus) extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .
2024-12-05
ArXiv (prépublication)
arxiv.org
arxiv.org
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte
Franccois Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
M. L. Richter
Saverio Vadacchino
Shubbam Agarwal
Sanket Biswas … (voir 23 de plus)
Sara Shanian
Ying Zhang
Noah Bolger
Kurt MacDonald
Simon Fauvel
Sathwik Tejaswi
Srinivas Sunkara
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharagani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Chris Pal
Issam Hadj Laradji
Spandanna Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows,… (voir plus) extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .
2024-12-05
ArXiv (prépublication)
arxiv.org
arxiv.org
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte
Franccois Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
Mats Leon Richter
Saverio Vadacchino
Shubbam Agarwal
Sanket Biswas … (voir 23 de plus)
Sara Shanian
Ying Zhang
Noah Bolger
Kurt MacDonald
Simon Fauvel
Sathwik Tejaswi
Srinivas Sunkara
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharaghani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Chris Pal
Issam Hadj Laradji
Spandanna Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows,… (voir plus) extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .
2024-12-05
ArXiv (prépublication)
arxiv.org
arxiv.org
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte
Franccois Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
Mats Leon Richter
Saverio Vadacchino
Shubbam Agarwal
Sanket Biswas … (voir 23 de plus)
Sara Shanian
Ying Zhang
Noah Bolger
Kurt MacDonald
Simon Fauvel
Sathwik Tejaswi
Srinivas Sunkara
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharaghani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Chris Pal
Issam Hadj Laradji
Spandanna Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows,… (voir plus) extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .
2024-12-05
ArXiv (prépublication)
doi.org
arxiv.org
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte
Franccois Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
Mats Leon Richter
Saverio Vadacchino
Shubbam Agarwal
Sanket Biswas … (voir 23 de plus)
Sara Shanian
Ying Zhang
Noah Bolger
Kurt MacDonald
Simon Fauvel
Sathwik Tejaswi
Srinivas Sunkara
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharaghani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Chris Pal
Issam Hadj Laradji
Spandanna Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows,… (voir plus) extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .
2024-12-05
ArXiv (prépublication)
doi.org
arxiv.org
Imagining and building wise machines: The centrality of AI metacognition
Samuel G. B. Johnson
Amir-Hossein Karimi
Yoshua Bengio
Nick Chater
Tobias Gerstenberg
Kate Larson
Sydney Levine
Melanie Mitchell
Iyad Rahwan
Bernhard Schölkopf
Igor Grossmann
Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks. … (voir plus)However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack of transparency in their reasoning (explainability), challenges in communication and commitment (cooperation), and risks due to potential harmful actions (safety). We argue that these shortcomings stem from one overarching failure: AI systems lack wisdom. Drawing from cognitive and social sciences, we define wisdom as the ability to navigate intractable problems - those that are ambiguous, radically uncertain, novel, chaotic, or computationally explosive - through effective task-level and metacognitive strategies. While AI research has focused on task-level strategies, metacognition - the ability to reflect on and regulate one's thought processes - is underdeveloped in AI systems. In humans, metacognitive strategies such as recognizing the limits of one's knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety. By focusing on developing wise AI, we suggest an alternative to aligning AI with specific human values - a task fraught with conceptual and practical difficulties. Instead, wise AI systems can thoughtfully navigate complex situations, account for diverse human values, and avoid harmful actions. We discuss potential approaches to building wise AI, including benchmarking metacognitive abilities and training AI systems to employ wise reasoning. Prioritizing metacognition in AI research will lead to systems that act not only intelligently but also wisely in complex, real-world situations.
2024-11-04
ArXiv (prépublication)
doi.org
arxiv.org
Imagining and building wise machines: The centrality of AI metacognition
Samuel G. B. Johnson
Amir-Hossein Karimi
Yoshua Bengio
Nick Chater
Tobias Gerstenberg
Kate Larson
Sydney Levine
Melanie Mitchell
Iyad Rahwan
Bernhard Schölkopf
Igor Grossmann
Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks. … (voir plus)However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack of transparency in their reasoning (explainability), challenges in communication and commitment (cooperation), and risks due to potential harmful actions (safety). We argue that these shortcomings stem from one overarching failure: AI systems lack wisdom. Drawing from cognitive and social sciences, we define wisdom as the ability to navigate intractable problems - those that are ambiguous, radically uncertain, novel, chaotic, or computationally explosive - through effective task-level and metacognitive strategies. While AI research has focused on task-level strategies, metacognition - the ability to reflect on and regulate one's thought processes - is underdeveloped in AI systems. In humans, metacognitive strategies such as recognizing the limits of one's knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety. By focusing on developing wise AI, we suggest an alternative to aligning AI with specific human values - a task fraught with conceptual and practical difficulties. Instead, wise AI systems can thoughtfully navigate complex situations, account for diverse human values, and avoid harmful actions. We discuss potential approaches to building wise AI, including benchmarking metacognitive abilities and training AI systems to employ wise reasoning. Prioritizing metacognition in AI research will lead to systems that act not only intelligently but also wisely in complex, real-world situations.
2024-11-04
ArXiv (prépublication)
doi.org
arxiv.org