Portrait de Doina Precup

Doina Precup

Membre académique principal
precupdo@mila.quebec
Chaire en IA Canada-CIFAR
Professeure agrégée, McGill University, École d'informatique
Chef d'équipe de recherche, Google DeepMind
Sujets de recherche
Apprentissage automatique médical
Apprentissage par renforcement
Modèles probabilistes
Modélisation moléculaire
Raisonnement

Biographie

Doina Precup enseigne à l'Université McGill tout en menant des recherches fondamentales sur l'apprentissage par renforcement, notamment les applications de l'IA dans des domaines ayant des répercussions sociales, tels que les soins de santé. Elle s'intéresse à la prise de décision automatique dans des situations d'incertitude élevée.

Elle est membre de l'Institut canadien de recherches avancées (CIFAR) et de l'Association pour l'avancement de l'intelligence artificielle (AAAI), et dirige le bureau montréalais de DeepMind.

Ses spécialités sont les suivantes : intelligence artificielle, apprentissage machine, apprentissage par renforcement, raisonnement et planification sous incertitude, applications.

Étudiants actuels

Safa Alver
Doctorat - McGill
alversaf@mila.quebec
Samin Yeasar Arnob
Doctorat - McGill
Site web
Google Scholar
Sumana Basu
Doctorat - McGill
Co-superviseur⋅e :
Adriana Romero Soriano
Site web
Github
Google Scholar
Veronica Chelu
Doctorat - McGill
Github
Lynn Cherif
Maîtrise recherche - McGill
Co-superviseur⋅e :
Khimya Khetarpal
Site web
Google Scholar
Raymond Chua
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
Site web
Google Scholar
Wesley Chung
Doctorat - McGill
Superviseur⋅e principal⋅e :
David Meger
Jonathan Colaço Carr
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Prakash Panangaden
Élodie Coté-Gauthier
Stagiaire de recherche - McGill
Derek Kweku DEGBEDZUI Degbedzui
Maîtrise recherche - McGill
Github
Google Scholar
Manuel Del Verme
Doctorat - McGill
Jesse Farebrother
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Site web
Github
Google Scholar
Tugce Gurbuz
Doctorat - McGill
Superviseur⋅e principal⋅e :
Eilif Benjamin Muller
Site web
Github
Google Scholar
Will Hua
Maîtrise recherche - McGill
Co-superviseur⋅e :
Guy Wolf
Github
Howard Huang
Doctorat - McGill
Haque Ishfaq
Doctorat - McGill
Site web
Github
Google Scholar
Mohammad Sami Nur Islam Islam
Maîtrise recherche - McGill
Github
Google Scholar
Riashat Islam
Doctorat - McGill
Site web
Github
Google Scholar
Arushi Jain
Doctorat - McGill
Site web
Github
Google Scholar
Thomas Jiralerspong
Maîtrise recherche - UdeM
Superviseur⋅e principal⋅e :
Yoshua Bengio
Site web
Github
Google Scholar
Martin Klissarov
Doctorat - McGill
Site web
Github
Google Scholar
Flemming Kondrup
Postdoctorat - McGill
Google Scholar
Elaine Lau
Maîtrise recherche - McGill
Jonathan Lebensold
Collaborateur·rice alumni - McGill
Site web
Github
Google Scholar
Sitao Luan
Doctorat - McGill
Github
Google Scholar
Ray Luo
Doctorat - McGill
Superviseur⋅e principal⋅e :
Xujie Si
Site web
Github
Google Scholar
G McCracken
Doctorat - McGill
Google Scholar
Shahrad Mohammadzadeh
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Reihaneh Rabbany
Site web
Github
Google Scholar
Gabriela Moisescu-Pareja
Maîtrise recherche - McGill
Girdhar Neil Girdhar
Collaborateur·rice de recherche - McGill
Padideh Nouri
Doctorat - UdeM
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar
Charles Onu
Doctorat - McGill
Site web
Github
Google Scholar
Gandharv Patil
Doctorat - McGill
Co-superviseur⋅e :
Pablo Samuel Castro
Site web
Github
Google Scholar
Nate Rahn
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Janarthanan Rajendran
Collaborateur·rice alumni - UdeM
Superviseur⋅e principal⋅e :
Sarath Chandar
Site web
Google Scholar
Sahand Rezaei-Shoshtari
Doctorat - McGill
Co-superviseur⋅e :
David Meger
Site web
Github
Google Scholar
Mandana Samiei
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
samiemandana@gmail.com
Site web
Github
Google Scholar
Manoosh Samiei
Doctorat - McGill
Co-superviseur⋅e :
Paul Masset
Site web
Github
Google Scholar
Shishir Sharma
Doctorat - McGill
Site web
Github
Google Scholar
Nishanth Anand Vemgal
Doctorat - McGill
Site web
Github
Google Scholar
David Anthony Venuto
Doctorat - McGill
Priyesh Vijayan
Doctorat - McGill
Co-superviseur⋅e :
Samira Ebrahimi Kahou
Site web
Github
Google Scholar
Guangyuan Wang
Stagiaire de recherche - McGill
Keyu Wang
Stagiaire de recherche - McGill
Site web
Github
Steve Wen
Baccalauréat - McGill
Co-superviseur⋅e :
Gregory Dudek
Shuyuan Zhang
Doctorat - McGill
Harry Zhao
Doctorat - McGill
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar

Publications

Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo
Haque Ishfaq
Qingfeng Lan
Pan Xu
A. Rupam Mahmood
Doina Precup
Animashree Anandkumar
Kamyar Azizzadenesheli
We present a scalable and effective exploration strategy based on Thompson sampling for reinforcement learning (RL). One of the key shortcom… (voir plus)ings of existing Thompson sampling algorithms is the need to perform a Gaussian approximation of the posterior distribution, which is not a good surrogate in most practical settings. We instead directly sample the Q function from its posterior distribution, by using Langevin Monte Carlo, an efficient type of Markov Chain Monte Carlo (MCMC) method. Our method only needs to perform noisy gradient descent updates to learn the exact posterior distribution of the Q function, which makes our approach easy to deploy in deep RL. We provide a rigorous theoretical analysis for the proposed method and demonstrate that, in the linear Markov decision process (linear MDP) setting, it has a regret bound of
2024-01-16
ICLR.cc/2024/Conference (poster)
doi.org
openreview.net
Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning
Tianyu Li
Doina Precup
Guillaume Rabusseau
2024-01-01
Mach. Learn. (publié)
doi.org
arxiv.org
Policy Gradient Methods in the Presence of Symmetries and State Abstractions
Prakash Panangaden
Sahand Rezaei-Shoshtari
Rosie Zhao
David Meger
Doina Precup
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
R'emi Munos
Michal Valko
Daniele Calandriello
M. G. Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Learning domain-invariant classifiers for infant cry sounds
Charles Onu
Hemanth K. Sheetha
Arsenii Gorin
Doina Precup
2023-11-30
ArXiv (prépublication)
arxiv.org
arxiv.org
MUDiff: Unified Diffusion for Complete Molecule Generation
Chenqing Hua
Sitao Luan
Minkai Xu
Zhitao Ying
Rex Ying
Jie Fu
Stefano Ermon
Doina Precup
2023-11-18
logconference.io/LOG/2023/Conference (poster)
doi.org
openreview.net
DGFN: Double Generative Flow Networks
Elaine Lau
Nikhil Murali Vemgal
Doina Precup
Emmanuel Bengio
2023-10-27
NeurIPS.cc/2023/Workshop/GenBio (poster)
doi.org
openreview.net
Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search
Abbas Mehrabian
Ankit Anand
Hyunjik Kim
Nicolas Sonnerat
Matej Balog
Gheorghe Comanici
Tudor Berariu
Andrew Lee
Anian Ruoss
Anna Bulanova
Daniel Toyama
Sam Blackwell
Bernardino Romera Paredes
Petar Veličković
Laurent Orseau
Joonkyung Lee
Anurag Murty Naredla
Doina Precup
Adam Zsolt Wagner
2023-10-27
NeurIPS.cc/2023/Workshop/MATH-AI (poster)
doi.org
openreview.net