Portrait de Doina Precup

Doina Precup

Membre académique principal
precupdo@mila.quebec
Chaire en IA Canada-CIFAR
Professeure agrégée, McGill University, École d'informatique
Chef d'équipe de recherche, Google DeepMind
Sujets de recherche
Apprentissage automatique médical
Apprentissage par renforcement
Modèles probabilistes
Modélisation moléculaire
Raisonnement

Biographie

Doina Precup enseigne à l'Université McGill tout en menant des recherches fondamentales sur l'apprentissage par renforcement, notamment les applications de l'IA dans des domaines ayant des répercussions sociales, tels que les soins de santé. Elle s'intéresse à la prise de décision automatique dans des situations d'incertitude élevée.

Elle est membre de l'Institut canadien de recherches avancées (CIFAR) et de l'Association pour l'avancement de l'intelligence artificielle (AAAI), et dirige le bureau montréalais de DeepMind.

Ses spécialités sont les suivantes : intelligence artificielle, apprentissage machine, apprentissage par renforcement, raisonnement et planification sous incertitude, applications.

Étudiants actuels

Safa Alver
Doctorat - McGill
alversaf@mila.quebec
Samin Yeasar Arnob
Doctorat - McGill
Site web
Google Scholar
Sumana Basu
Doctorat - McGill
Co-superviseur⋅e :
Adriana Romero Soriano
Site web
Github
Google Scholar
Veronica Chelu
Doctorat - McGill
Github
Lynn Cherif
Maîtrise recherche - McGill
Co-superviseur⋅e :
Khimya Khetarpal
Site web
Google Scholar
Raymond Chua
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
Site web
Google Scholar
Wesley Chung
Doctorat - McGill
Superviseur⋅e principal⋅e :
David Meger
Jonathan Colaço Carr
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Prakash Panangaden
Élodie Coté-Gauthier
Stagiaire de recherche - McGill
Franco Del Balso
Stagiaire de recherche - UdeM
Github
Jesse Farebrother
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Site web
Github
Google Scholar
Tugce Gurbuz
Doctorat - McGill
Superviseur⋅e principal⋅e :
Eilif Benjamin Muller
Site web
Github
Google Scholar
Howard Huang
Doctorat - McGill
Haque Ishfaq
Doctorat - McGill
Site web
Github
Google Scholar
Mohammad Sami Nur Islam Islam
Maîtrise recherche - McGill
Github
Google Scholar
Arushi Jain
Doctorat - McGill
Site web
Github
Google Scholar
Martin Klissarov
Doctorat - McGill
Site web
Github
Google Scholar
Flemming Kondrup
Postdoctorat - McGill
Google Scholar
Elaine Lau
Maîtrise recherche - McGill
Jonathan Lebensold
Collaborateur·rice alumni - McGill
Site web
Github
Google Scholar
Ray Luo
Doctorat - McGill
Superviseur⋅e principal⋅e :
Xujie Si
Site web
Github
Google Scholar
G McCracken
Doctorat - McGill
Google Scholar
Nazanin Mohammadi Sepahvand
Doctorat - McGill
Github
Shahrad Mohammadzadeh
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Reihaneh Rabbany
Site web
Github
Google Scholar
Gabriela Moisescu-Pareja
Maîtrise recherche - McGill
Padideh Nouri
Doctorat - UdeM
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar
Charles Onu
Doctorat - McGill
Site web
Github
Google Scholar
Gandharv Patil
Doctorat - McGill
Co-superviseur⋅e :
Pablo Samuel Castro
Site web
Github
Google Scholar
Nate Rahn
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Sahand Rezaei-Shoshtari
Doctorat - McGill
Co-superviseur⋅e :
David Meger
Site web
Github
Google Scholar
Manoosh Samiei
Doctorat - McGill
Co-superviseur⋅e :
Paul Masset
Site web
Github
Google Scholar
Mandana Samiei
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
samiemandana@gmail.com
Site web
Github
Google Scholar
Shishir Sharma
Doctorat - McGill
Site web
Github
Google Scholar
Nishanth Anand Vemgal
Doctorat - McGill
Site web
Github
Google Scholar
David Anthony Venuto
Doctorat - McGill
Priyesh Vijayan
Doctorat - McGill
Co-superviseur⋅e :
Samira Ebrahimi Kahou
Site web
Github
Google Scholar
Guangyuan Wang
Stagiaire de recherche - McGill
Github
Steve Wen
Maîtrise recherche - McGill
Co-superviseur⋅e :
Gregory Dudek
Github
Zijing Wu
Doctorat - McGill
Co-superviseur⋅e :
Paul Masset
Site web
Github
Google Scholar
Shuyuan Zhang
Doctorat - McGill
Github
Harry Zhao
Doctorat - McGill
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar

Publications

Policy Gradient Methods in the Presence of Symmetries and State Abstractions
Prakash Panangaden
Sahand Rezaei-Shoshtari
Rosie Zhao
David Meger
Doina Precup
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
R'emi Munos
Michal Valko
Daniele Calandriello
M. G. Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Learning domain-invariant classifiers for infant cry sounds
Charles Onu
Hemanth K. Sheetha
Arsenii Gorin
Doina Precup
2023-11-30
ArXiv (prépublication)
arxiv.org
arxiv.org
MUDiff: Unified Diffusion for Complete Molecule Generation
Chenqing Hua
Sitao Luan
Minkai Xu
Zhitao Ying
Rex Ying
Jie Fu
Stefano Ermon
Doina Precup
2023-11-18
logconference.io/LOG/2023/Conference (poster)
doi.org
openreview.net
DGFN: Double Generative Flow Networks
Elaine Lau
Nikhil Murali Vemgal
Doina Precup
Emmanuel Bengio
2023-10-27
NeurIPS.cc/2023/Workshop/GenBio (poster)
doi.org
openreview.net
Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search
Abbas Mehrabian
Ankit Anand
Hyunjik Kim
Nicolas Sonnerat
Matej Balog
Gheorghe Comanici
Tudor Berariu
Andrew Lee
Anian Ruoss
Anna Bulanova
Daniel Toyama
Sam Blackwell
Bernardino Romera Paredes
Petar Veličković
Laurent Orseau
Joonkyung Lee
Anurag Murty Naredla
Doina Precup
Adam Zsolt Wagner
2023-10-27
NeurIPS.cc/2023/Workshop/MATH-AI (poster)
doi.org
openreview.net
Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels
Thomas Jiralerspong
Flemming Kondrup
Doina Precup
Khimya Khetarpal
The ability to plan at many different levels of abstraction enables agents to envision the long-term repercussions of their decisions and th… (voir plus)us enables sample-efficient learning. This becomes particularly beneficial in complex environments from high-dimensional state space such as pixels, where the goal is distant and the reward sparse. We introduce Forecaster, a deep hierarchical reinforcement learning approach which plans over high-level goals leveraging a temporally abstract world model. Forecaster learns an abstract model of its environment by modelling the transitions dynamics at an abstract level and training a world model on such transition. It then uses this world model to choose optimal high-level goals through a tree-search planning procedure. It additionally trains a low-level policy that learns to reach those goals. Our method not only captures building world models with longer horizons, but also, planning with such models in downstream tasks. We empirically demonstrate Forecaster's potential in both single-task learning and generalization to new tasks in the AntMaze domain.
2023-10-27
NeurIPS.cc/2023/Workshop/GenPlan (publié)
doi.org
openreview.net
A cry for help: Early detection of brain injury in newborns
Charles Onu
Samantha Latremouille
Arsenii Gorin
Junhao Wang
Uchenna Ekwochi
P. Ubuane
O. Kehinde
Muhammad A. Salisu
Datonye Briggs
Yoshua Bengio
Doina Precup
2023-10-12
ArXiv (prépublication)
doi.org
arxiv.org