Portrait de Doina Precup

Doina Precup

Membre académique principal
precupdo@mila.quebec
Chaire en IA Canada-CIFAR
Professeure agrégée, McGill University, École d'informatique
Chef d'équipe de recherche, Google DeepMind
Sujets de recherche
Apprentissage automatique médical
Apprentissage par renforcement
Modèles probabilistes
Modélisation moléculaire
Raisonnement

Biographie

Doina Precup enseigne à l'Université McGill tout en menant des recherches fondamentales sur l'apprentissage par renforcement, notamment les applications de l'IA dans des domaines ayant des répercussions sociales, tels que les soins de santé. Elle s'intéresse à la prise de décision automatique dans des situations d'incertitude élevée.

Elle est membre de l'Institut canadien de recherches avancées (CIFAR) et de l'Association pour l'avancement de l'intelligence artificielle (AAAI), et dirige le bureau montréalais de DeepMind.

Ses spécialités sont les suivantes : intelligence artificielle, apprentissage machine, apprentissage par renforcement, raisonnement et planification sous incertitude, applications.

Étudiants actuels

Samin Yeasar Arnob
Doctorat - McGill
Site web
Google Scholar
Sumana Basu
Doctorat - McGill
Co-superviseur⋅e :
Adriana Romero Soriano
sumana.basu@mail.mcgill.ca
Site web
Github
Google Scholar
Veronica Chelu
Collaborateur·rice alumni - McGill
Github
Lynn Cherif
Maîtrise recherche - McGill
Co-superviseur⋅e :
Khimya Khetarpal
Site web
Github
Google Scholar
Raymond Chua
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
Site web
Google Scholar
Wesley Chung
Doctorat - McGill
Superviseur⋅e principal⋅e :
David Meger
Jonathan Colaço Carr
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Prakash Panangaden
Élodie Coté-Gauthier
Collaborateur·rice de recherche - McGill
Github
Franco Del Balso
Stagiaire de recherche - UdeM
Github
Jesse Farebrother
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Site web
Github
Google Scholar
Tugce Gurbuz
Doctorat - McGill
Superviseur⋅e principal⋅e :
Eilif Benjamin Muller
Site web
Github
Google Scholar
Howard Huang
Doctorat - McGill
Haque Ishfaq
Collaborateur·rice alumni - McGill
Site web
Github
Google Scholar
Mohammad Sami Nur Islam Islam
Maîtrise recherche - McGill
Github
Google Scholar
Arushi Jain
Doctorat - McGill
Site web
Github
Google Scholar
Flemming Kondrup
Postdoctorat - McGill
Google Scholar
Elaine Lau
Maîtrise recherche - McGill
Jonathan Lebensold
Collaborateur·rice alumni - McGill
Site web
Github
Google Scholar
Jeremy Lucas
Baccalauréat - McGill
Ray Luo
Doctorat - McGill
Superviseur⋅e principal⋅e :
Xujie Si
Site web
Github
Google Scholar
G McCracken
Doctorat - McGill
Google Scholar
Nazanin Mohammadi Sepahvand
Collaborateur·rice alumni - McGill
Github
Shahrad Mohammadzadeh
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Reihaneh Rabbany
Site web
Github
Google Scholar
Gabriela Moisescu-Pareja
Collaborateur·rice de recherche - McGill
Co-superviseur⋅e :
Irina Rish
Padideh Nouri
Doctorat - UdeM
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar
Gandharv Patil
Doctorat - McGill
Co-superviseur⋅e :
Pablo Samuel Castro
Site web
Github
Google Scholar
Nate Rahn
Doctorat - McGill
Superviseur⋅e principal⋅e :
Marc Gendron-Bellemare
Sahand Rezaei-Shoshtari
Doctorat - McGill
Co-superviseur⋅e :
David Meger
Site web
Github
Google Scholar
Manoosh Samiei
Doctorat - McGill
Co-superviseur⋅e :
Paul Masset
Site web
Github
Google Scholar
Mandana Samiei
Doctorat - McGill
Co-superviseur⋅e :
Blake Richards
samiemandana@gmail.com
Site web
Github
Google Scholar
Shishir Sharma
Doctorat - McGill
Site web
Github
Google Scholar
Nishanth Anand Vemgal
Doctorat - McGill
Site web
Github
Google Scholar
Priyesh Vijayan
Doctorat - McGill
Co-superviseur⋅e :
Samira Ebrahimi Kahou
Site web
Github
Google Scholar
Zihan Wang
Doctorat - McGill
Site web
Google Scholar
Guangyuan Wang
Stagiaire de recherche - McGill
Github
Steve Wen
Maîtrise recherche - McGill
Co-superviseur⋅e :
Gregory Dudek
Github
Zijing Wu
Doctorat - McGill
Co-superviseur⋅e :
Paul Masset
Site web
Github
Google Scholar
Shuyuan Zhang
Doctorat - McGill
Github
Harry Zhao
Collaborateur·rice alumni - McGill
Co-superviseur⋅e :
Yoshua Bengio
Site web
Github
Google Scholar

Publications

Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo
Haque Ishfaq
Qingfeng Lan
Pan Xu
A. Rupam Mahmood
Doina Precup
Animashree Anandkumar
Kamyar Azizzadenesheli
We present a scalable and effective exploration strategy based on Thompson sampling for reinforcement learning (RL). One of the key shortcom… (voir plus)ings of existing Thompson sampling algorithms is the need to perform a Gaussian approximation of the posterior distribution, which is not a good surrogate in most practical settings. We instead directly sample the Q function from its posterior distribution, by using Langevin Monte Carlo, an efficient type of Markov Chain Monte Carlo (MCMC) method. Our method only needs to perform noisy gradient descent updates to learn the exact posterior distribution of the Q function, which makes our approach easy to deploy in deep RL. We provide a rigorous theoretical analysis for the proposed method and demonstrate that, in the linear Markov decision process (linear MDP) setting, it has a regret bound of
2024-01-16
ICLR.cc/2024/Conference (poster)
doi.org
openreview.net
An Attentive Approach for Building Partial Reasoning Agents from Pixels
Safa Alver
Doina Precup
We study the problem of building reasoning agents that are able to generalize in an effective manner. Towards this goal, we propose an end-t… (voir plus)o-end approach for building model-based reinforcement learning agents that dynamically focus their reasoning to the relevant aspects of the environment: after automatically identifying the distinct aspects of the environment, these agents dynamically filter out the relevant ones and then pass them to their simulator to perform partial reasoning. Unlike existing approaches, our approach works with pixel-based inputs and it allows for interpreting the focal points of the agent. Our quantitative analyses show that the proposed approach allows for effective generalization in high-dimensional domains with raw observational inputs. We also perform ablation analyses to validate our design choices. Finally, we demonstrate through qualitative analyses that our approach actually allows for building agents that focus their reasoning on the relevant aspects of the environment.
2024-01-01
Trans. Mach. Learn. Res. (publié)
openreview.net
openreview.net
Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning
Tianyu Li
Doina Precup
Guillaume Rabusseau
2024-01-01
Mach. Learn. (publié)
doi.org
arxiv.org
Policy Gradient Methods in the Presence of Symmetries and State Abstractions
Prakash Panangaden
Sahand Rezaei-Shoshtari
Rosie Zhao
David Meger
Doina Precup
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
R'emi Munos
Michal Valko
Daniele Calandriello
M. G. Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Nash Learning from Human Feedback
Remi Munos
Michal Valko
Daniele Calandriello
Mohammad Gheshlaghi Azar
Mark Rowland
Zhaohan Daniel Guo
Yunhao Tang
Matthieu Geist
Thomas Mesnard
Andrea Michi
Marco Selvi
Sertan Girgin
Nikola Momchev
Olivier Bachem
Daniel J Mankowitz
Doina Precup
Bilal Piot
Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human pref… (voir plus)erences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.
2023-12-01
ArXiv (prépublication)
doi.org
arxiv.org
Learning domain-invariant classifiers for infant cry sounds
Charles Onu
Hemanth K. Sheetha
Arsenii Gorin
Doina Precup
2023-11-30
ArXiv (prépublication)
arxiv.org
arxiv.org
MUDiff: Unified Diffusion for Complete Molecule Generation
Chenqing Hua
Sitao Luan
Minkai Xu
Zhitao Ying
Rex Ying
Jie Fu
Stefano Ermon
Doina Precup
2023-11-18
logconference.io/LOG/2023/Conference (poster)
doi.org
openreview.net