Portrait of Doina Precup

Doina Precup

Core Academic Member
precupdo@mila.quebec
Canada CIFAR AI Chair
Associate Professor, McGill University, School of Computer Science
Research Team Leader, Google DeepMind
Research Topics
Medical Machine Learning
Molecular Modeling
Probabilistic Models
Reasoning
Reinforcement Learning

Biography

Doina Precup combines teaching at McGill University with fundamental research on reinforcement learning, in particular AI applications in areas of significant social impact, such as health care. She is interested in machine decision-making in situations where uncertainty is high.

In addition to heading the Montreal office of Google DeepMind, Precup is a Senior Fellow of the Canadian Institute for Advanced Research and a Fellow of the Association for the Advancement of Artificial Intelligence.

Her areas of speciality are artificial intelligence, machine learning, reinforcement learning, reasoning and planning under uncertainty, and applications.

Current Students

Safa Alver
PhD - McGill University
alversaf@mila.quebec
Samin Yeasar Arnob
PhD - McGill University
Website
Google Scholar
Sumana Basu
PhD - McGill University
Co-supervisor :
Adriana Romero Soriano
Website
Github
Google Scholar
Veronica Chelu
PhD - McGill University
Github
Lynn Cherif
Master's Research - McGill University
Co-supervisor :
Khimya Khetarpal
Website
Google Scholar
Raymond Chua
PhD - McGill University
Co-supervisor :
Blake Richards
Website
Google Scholar
Wesley Chung
PhD - McGill University
Principal supervisor :
David Meger
Jonathan Colaço Carr
Master's Research - McGill University
Principal supervisor :
Prakash Panangaden
Élodie Coté-Gauthier
Research Intern - McGill University
Franco Del Balso
Research Intern - Université de Montréal
Github
Jesse Farebrother
PhD - McGill University
Principal supervisor :
Marc Gendron-Bellemare
Website
Github
Google Scholar
Tugce Gurbuz
PhD - McGill University
Principal supervisor :
Eilif Benjamin Muller
Website
Github
Google Scholar
Howard Huang
PhD - McGill University
Haque Ishfaq
PhD - McGill University
Website
Github
Google Scholar
Mohammad Sami Nur Islam Islam
Master's Research - McGill University
Github
Google Scholar
Arushi Jain
PhD - McGill University
Website
Github
Google Scholar
Martin Klissarov
PhD - McGill University
Website
Github
Google Scholar
Flemming Kondrup
Postdoctorate - McGill University
Google Scholar
Elaine Lau
Master's Research - McGill University
Jonathan Lebensold
Collaborating Alumni - McGill University
Website
Github
Google Scholar
Ray Luo
PhD - McGill University
Principal supervisor :
Xujie Si
Website
Github
Google Scholar
G McCracken
PhD - McGill University
Google Scholar
Nazanin Mohammadi Sepahvand
PhD - McGill University
Github
Shahrad Mohammadzadeh
Master's Research - McGill University
Principal supervisor :
Reihaneh Rabbany
Website
Github
Google Scholar
Gabriela Moisescu-Pareja
Master's Research - McGill University
Padideh Nouri
PhD - Université de Montréal
Co-supervisor :
Yoshua Bengio
Website
Github
Google Scholar
Charles Onu
PhD - McGill University
Website
Github
Google Scholar
Gandharv Patil
PhD - McGill University
Co-supervisor :
Pablo Samuel Castro
Website
Github
Google Scholar
Nate Rahn
PhD - McGill University
Principal supervisor :
Marc Gendron-Bellemare
Sahand Rezaei-Shoshtari
PhD - McGill University
Co-supervisor :
David Meger
Website
Github
Google Scholar
Manoosh Samiei
PhD - McGill University
Co-supervisor :
Paul Masset
Website
Github
Google Scholar
Mandana Samiei
PhD - McGill University
Co-supervisor :
Blake Richards
samiemandana@gmail.com
Website
Github
Google Scholar
Shishir Sharma
PhD - McGill University
Website
Github
Google Scholar
Nishanth Anand Vemgal
PhD - McGill University
Website
Github
Google Scholar
David Anthony Venuto
PhD - McGill University
Priyesh Vijayan
PhD - McGill University
Co-supervisor :
Samira Ebrahimi Kahou
Website
Github
Google Scholar
Guangyuan Wang
Research Intern - McGill University
Github
Steve Wen
Master's Research - McGill University
Co-supervisor :
Gregory Dudek
Github
Zijing Wu
PhD - McGill University
Co-supervisor :
Paul Masset
Website
Github
Google Scholar
Shuyuan Zhang
PhD - McGill University
Github
Harry Zhao
PhD - McGill University
Co-supervisor :
Yoshua Bengio
Website
Github
Google Scholar

Publications

Plasticity as the Mirror of Empowerment
David Abel
Michael Bowling
Andre Barreto
Will Dabney
Shi Dong
Steven Hansen
Anna Harutyunyan
Khimya Khetarpal
Clare Lyle
Razvan Pascanu
Georgios Piliouras
Doina Precup
Jonathan Richens
Mark Rowland
Tom Schaul
Satinder Singh
2025-05-15
ArXiv (preprint)
arxiv.org
arxiv.org
Plasticity as the Mirror of Empowerment
David Abel
Michael Bowling
Andre Barreto
Will Dabney
Shi Dong
Steven Hansen
Anna Harutyunyan
Khimya Khetarpal
Clare Lyle
Razvan Pascanu
Georgios Piliouras
Doina Precup
Jonathan Richens
Mark Rowland
Tom Schaul
Satinder Singh
2025-05-15
ArXiv (preprint)
arxiv.org
arxiv.org
Language Agents Mirror Human Causal Reasoning Biases. How Can We Help Them Think Like Scientists?
Anthony GX-Chen
Dongyan Lin
Mandana Samiei
Doina Precup
Blake Richards
Rob Fergus
Kenneth Marino
2025-05-14
ArXiv (preprint)
arxiv.org
arxiv.org
Understanding the Effectiveness of Learning Behavioral Metrics in Deep Reinforcement Learning
Ziyan Luo
Tianwei Ni
Pierre-Luc Bacon
Doina Precup
Xujie Si
A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space, and embed … (see more)these learned distances in the representation space. While promising for robustness to task-irrelevant noise shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep RL, we evaluate five recent approaches. We unify them under isometric embedding, identify key design choices, and benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 250+ configurations with diverse noise settings. Beyond final returns, we introduce the denoising factor to quantify the encoder’s ability to filter distractions. To further isolate the effect of metric learning, we propose an isolated metric estimation setting, where the encoder is influenced solely by the metric loss. Our results show that metric learning improves return and denoising only marginally, as its benefits fade when key design choices, such as layer normalization and self-prediction loss, are incorporated into the baseline. We also find that commonly used benchmarks (e.g., grayscale videos, varying state-based Gaussian noise dimensions) add little difficulty, while Gaussian noise with random projection and pixel-based Gaussian noise remain challenging even for the best methods. Finally, we release an open-source, modular codebase to improve reproducibility and support future research on metric learning in deep RL.
2025-05-09
rl-conference.cc/RLC/2025/Conference (published)
openreview.net
openreview.net
Understanding the Effectiveness of Learning Behavioral Metrics in Deep Reinforcement Learning
Ziyan Luo
Tianwei Ni
Pierre-Luc Bacon
Doina Precup
Xujie Si
A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space, and embed … (see more)these learned distances in the representation space. While promising for robustness to task-irrelevant noise shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep RL, we evaluate five recent approaches. We unify them under isometric embedding, identify key design choices, and benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 250+ configurations with diverse noise settings. Beyond final returns, we introduce the denoising factor to quantify the encoder’s ability to filter distractions. To further isolate the effect of metric learning, we propose an isolated metric estimation setting, where the encoder is influenced solely by the metric loss. Our results show that metric learning improves return and denoising only marginally, as its benefits fade when key design choices, such as layer normalization and self-prediction loss, are incorporated into the baseline. We also find that commonly used benchmarks (e.g., grayscale videos, varying state-based Gaussian noise dimensions) add little difficulty, while Gaussian noise with random projection and pixel-based Gaussian noise remain challenging even for the best methods. Finally, we release an open-source, modular codebase to improve reproducibility and support future research on metric learning in deep RL.
2025-05-09
rl-conference.cc/RLC/2025/Conference (accepted)
openreview.net
openreview.net
Generative AI: Hype, Hope, and Responsible Use in Science and Everyday Life
Doina Precup
2025-05-01
Biological Psychiatry (published)
doi.org
Capturing Individual Human Preferences with Reward Features
Andre Barreto
Vincent Dumoulin
Yiran Mao
Nicolas Perez-Nieves
Bobak Shahriari
Yann Dauphin
Doina Precup
Hugo Larochelle
Reinforcement learning from human feedback usually models preferences using a reward model that does not distinguish between people. We argu… (see more)e that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We propose a method to specialise a reward model to a person or group of people. Our approach builds on the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models comparing the proposed architecture with a non-adaptive reward model and also adaptive counterparts, including models that do in-context personalisation. Depending on how much disagreement there is in the training data, our model either significantly outperforms the baselines or matches their performance with a simpler architecture and more stable training.
2025-03-21
ArXiv (preprint)
doi.org
arxiv.org
Capturing Individual Human Preferences with Reward Features
Andr'e Barreto
Vincent Dumoulin
Yiran Mao
Nicolas Perez-Nieves
Bobak Shahriari
Yann Dauphin
Doina Precup
Hugo Larochelle
2025-03-21
ArXiv (preprint)
arxiv.org
arxiv.org
Cracking the Code of Action: A Generative Approach to Affordances for Reinforcement Learning
Lynn Cherif
Flemming Kondrup
David Venuto
Ankit Anand
Doina Precup
Khimya Khetarpal
Agents that can autonomously navigate the web through a graphical user interface (GUI) using a unified action space (e.g., mouse and keyboar… (see more)d actions) can require very large amounts of domain-specific expert demonstrations to achieve good performance. Low sample efficiency is often exacerbated in sparse-reward and large-action-space environments, such as a web GUI, where only a few actions are relevant in any given situation. In this work, we consider the low-data regime, with limited or no access to expert behavior. To enable sample-efficient learning, we explore the effect of constraining the action space through intent-based affordances -- i.e., considering in any situation only the subset of actions that achieve a desired outcome. We propose **Code as Generative Affordances**
2025-03-05
ICLR.cc/2025/Workshop/DL4C (published)
openreview.net
openreview.net
Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning
Lynn Cherif
Flemming Kondrup
David Venuto
Ankit Anand
Doina Precup
Khimya Khetarpal
Agents that can autonomously navigate the web through a graphical user interface (GUI) using a unified action space (e.g., mouse and keyboar… (see more)d actions) can require very large amounts of domain-specific expert demonstrations to achieve good performance. Low sample efficiency is often exacerbated in sparse-reward and large-action-space environments, such as a web GUI, where only a few actions are relevant in any given situation. In this work, we consider the low-data regime, with limited or no access to expert behavior. To enable sample-efficient learning, we explore the effect of constraining the action space through *intent-based affordances* -- i.e., considering in any situation only the subset of actions that achieve a desired outcome. We propose **Code as Generative Affordances (
2025-03-05
ICLR.cc/2025/Workshop/DL4C (published)
doi.org
openreview.net
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Samin Yeasar Arnob
Zhan Su
Minseon Kim
Oleksiy Ostapenko
Doina Precup
Lucas Caccia
Alessandro Sordoni
Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly a… (see more)dapted on the fly for specific downstream tasks, without requiring additional fine-tuning. Typically, LoRA (Low-Rank Adaptation) serves as the foundational building block of such parameter-efficient modular architectures, leveraging low-rank weight structures to reduce the number of trainable parameters. In this paper, we study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures. First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature and surprisingly outperforms both LoRA and full fine-tuning in our setting. Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks, thus scaling beyond what is usually studied in the literature. Our findings demonstrate that sparse adapters yield superior in-distribution performance post-merging compared to LoRA or full model merging. Achieving strong held-out performance remains a challenge for all methods considered.
2025-03-05
ICLR.cc/2025/Workshop/MCDC (accepted)
openreview.net
openreview.net
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Samin Yeasar Arnob
Zhan Su
Minseon Kim
Oleksiy Ostapenko
Doina Precup
Lucas Caccia
Alessandro Sordoni
Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly a… (see more)dapted on the fly for specific downstream tasks, without requiring additional fine-tuning. Typically, LoRA (Low-Rank Adaptation) serves as the foundational building block of such parameter-efficient modular architectures, leveraging low-rank weight structures to reduce the number of trainable parameters. In this paper, we study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures. First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature and surprisingly outperforms both LoRA and full fine-tuning in our setting. Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks, thus scaling beyond what is usually studied in the literature. Our findings demonstrate that sparse adapters yield superior in-distribution performance post-merging compared to LoRA or full model merging. Achieving strong held-out performance remains a challenge for all methods considered.
2025-03-05
ICLR.cc/2025/Workshop/MCDC (accepted)
openreview.net
openreview.net