Portrait de Doina Precup

Doina Precup

Membre académique principal
Chaire en IA Canada-CIFAR
Professeure agrégée, McGill University
Chef d'équipe de recherche, Google DeepMind
Sujets de recherche
Apprentissage automatique médical
Apprentissage par renforcement
Modèles probabilistes
Modélisation moléculaire
Raisonnement

Biographie

Doina Precup enseigne à l'Université McGill tout en menant des recherches fondamentales sur l'apprentissage par renforcement, notamment les applications de l'IA dans des domaines ayant des répercussions sociales, tels que les soins de santé. Elle s'intéresse à la prise de décision automatique dans des situations d'incertitude élevée.

Elle est membre de l'Institut canadien de recherches avancées (CIFAR) et de l'Association pour l'avancement de l'intelligence artificielle (AAAI), et dirige le bureau montréalais de DeepMind.

Ses spécialités sont les suivantes : intelligence artificielle, apprentissage machine, apprentissage par renforcement, raisonnement et planification sous incertitude, applications.

Étudiants actuels

Collaborateur·rice alumni - McGill
Co-superviseur⋅e :
Collaborateur·rice alumni - McGill
Doctorat - McGill
Superviseur⋅e principal⋅e :
Maîtrise recherche - McGill
Superviseur⋅e principal⋅e :
Maîtrise recherche - McGill
Co-superviseur⋅e :
Maîtrise recherche - McGill
Doctorat - McGill
Superviseur⋅e principal⋅e :
Maîtrise recherche - McGill
Doctorat - Polytechnique
Superviseur⋅e principal⋅e :
Visiteur de recherche indépendant - McGill
Co-superviseur⋅e :
Collaborateur·rice alumni - McGill
Doctorat - McGill
Superviseur⋅e principal⋅e :
Maîtrise recherche - McGill
Collaborateur·rice de recherche - McGill
Co-superviseur⋅e :
Doctorat - McGill
Superviseur⋅e principal⋅e :
Stagiaire de recherche - McGill
Maîtrise recherche - McGill
Co-superviseur⋅e :
Stagiaire de recherche - McGill
Doctorat - McGill
Superviseur⋅e principal⋅e :
Collaborateur·rice alumni - McGill
Co-superviseur⋅e :

Publications

Rotation-Preserving Supervised Fine-Tuning
Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that thi… (voir plus)s degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-
Addressing Sparse-Rewards in RL with Scalable Hierarchical Novel Eigen Options
Temporally extended exploration via graph Laplacian-based options is a promising approach to sparse-reward reinforcement learning (RL), but … (voir plus)existing methods either do not explicitly target novelty or fail to scale to pixel-based domains under function approximation. Novel Exploration via Orthogonality (NEO) addresses the first issue by constructing options that navigate from highly visited regions toward less visited ones, yet prior results were limited to settings where exact eigenvectors can be computed. We present a scalable extension of NEO to pixel-based domains, built on three contributions. First, we use a novelty-weighted continuous Laplacian graph-drawing objective, which enables RL with continuous observations. Second, we embed the resulting eigen-potential options within a hierarchical reinforcement learning framework, enabling coherent temporally extended behavior. Third, we observe that learned eigen-potential rewards are directional but locally unreliable under online approximation; we therefore augment each option reward with a novelty bonus, a novel design idea that proves essential for stabilizing option learning while preserving novelty-directed exploration. Together, these contributions yield stronger and more persistent exploration, enabling longer option rollouts and better access to hard-to-reach novel states. Empirically, our method significantly outperforms both the prior scalable Laplacian-option baseline and a direct extension of NEO on sparse-reward benchmarks under a fixed budget. On Montezuma's Revenge, our best variant achieves approximately 1.8x higher return than both baselines. On Venture, both baselines yield returns near zero, whereas our method achieves a return of 1135. Across seven hard ProcGen games, our method achieves approximately 3.5x and 5.6x higher aggregate normalized return than the two baselines, respectively.
The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks
Shuo Wen
Beixi Du
Junming Shi
Chloe Si
DexSuite: An All-in-One Framework for Dexterous Manipulation Across Arms, Hands, and Tasks
Hugo He
Junming Shi
Ricardo Chahine
Shuo Wen
Mariana Sosa Guzmán
Rayan Houssaini
Adam Landa
Théophile Soulié
Charlotte Morissette
Annie Tims
Martino Russi
Jonathan Lussier
Julen Urain … (voir 2 de plus)
Human-level dexterity remains a central open problem in robotics, yet few simulation frameworks cover the full workflow of dexterous manipul… (voir plus)ation, from setting up a robot to collecting dexterous demonstrations and evaluating policies. We present DexSuite, an all-in-one simulation framework for dexterous manipulation built on the Genesis simulator. DexSuite provides 30 tasks and assembles its 10 arms and 12 end-effectors, from parallel-jaw grippers to dexterous hands, into more than 100 single-arm robots and thousands of bimanual pairs. It also ships a standardized environment API, integrated teleoperation, and simulated tactile sensing. Moreover, we define a benchmark of 16 task-embodiment pairs spanning diverse contact complexities and horizons, and release a dataset of 4,700 successful episodes in simulation from human teleoperation, scripted experts, and learned policies. We release DexSuite as open source and hope others will build on it. Videos and datasets are available at https://dexsuiteorg.github.io
Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data
Sami Nur Islam
Johan Obando-Ceron
Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the of… (voir plus)fline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
Learning Where to Constrain: Meta-Learned Trust Allocation for Mathematical Reasoning
Long mathematical solutions expose a weakness in uniform policy-update constraints. Early token changes affect a long suffix, while late cha… (voir plus)nges affect fewer future states. Prefix divergence-based methods like CPPO encode this asymmetry with a fixed position schedule and prefix budget. We introduce Meta-TA, a bilevel method for learning both choices from post-update performance on disjoint prompts. A direct one-step design fails when behavior and current policies match because the constraint starts inactive. We prove the resulting hypergradient equals zero. A detached warm-up creates controlled policy drift. A soft-gated step then supplies the meta-gradient. On Qwen3-8B-Base, five frozen-retraining seeds raise mean AIME24/25/26 Avg@16 from 29.01 for CPPO to 31.95 for Meta-TA with closely matched realized KL. At a 16k response cap, Meta-TA lowers normalized reasoning effective horizon from 0.475 to 0.430 and importance-ratio variance from 0.0534 to 0.0475. These results link adaptive trust allocation to higher accuracy and more localized updates in long-horizon mathematical reasoning.
Exploration-Preserving Policy Optimization
Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain acc… (voir plus)essible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
A reinforcement-learning agent moving between tasks in a fixed environment faces a mismatch: the reward changes, but the geometry does not. … (voir plus)Methods that learn a representation and a value function jointly relearn both at every task switch, which is slow and prone to forgetting. We take the identification of grid cells with eigenvectors of the successor representation as a design principle, and build the feature basis from the low-frequency eigenvectors of the symmetrised random-walk graph Laplacian, defined over state-action pairs rather than states so that the features encode what an action does. Because this basis diagonalises the transition operator, the successor-feature transform is exactly diagonal with closed-form entries fixed by the Laplacian eigenvalues and the discount. No temporal-difference learning is needed for it, and neither a successor-feature network nor a replay buffer is required: the only learned quantity is a linear reward map, re-estimated per task. On two goal locations in a two-room maze sharing an identical transition structure, the resulting agent holds near-ceiling return across every task switch, where a successor-feature baseline that learns its features collapses at each change. The cost is a slightly lower ceiling imposed by the linear parameterisation. The basis here is computed from the true transition graph, so this arm is an oracle upper bound rather than a method competing on equal information; learning the basis online is in progress.
From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs
The integration of graphs with Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) has received increasing attention, as graphs nat… (voir plus)urally encode task hierarchies for effective subgoal sampling. However, existing methods often overlook intrinsic connectivity information, failing to fully leverage the underlying topology for efficient learning. Most graph-based GCHRL methods use the graph as a stochastic sampling tool rather than as an environmental model that encodes connectivity and state-accessibility information. This limitation is particularly acute in quasimetric environments, where the inherent asymmetry of state transitions poses a fundamental challenge to stable policy learning and robust path planning. In this paper, we address these problems by introducing a state connectivity model designed to predict pairwise state connectivity strength in asymmetric environments. We transform these connectivity strengths into scalar auxiliary dense rewards, providing continuous guidance across multiple hierarchical levels. We demonstrate that our proposed framework, Graph-Guided Quasimetric Dense Reward (G2QDR), can theoretically be integrated into any existing GCHRL architecture, and the state connectivity model is efficiently implemented via a neural network trained on a directed state graph generated during exploration. Empirical results across a wide range of sparse reward environments indicate that, in general, G2QDR can enhance the performance of baseline GCHRL approaches with acceptable computational overhead.
SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training
Anita Marie Slominska
Rennie Bimman
Elizabeth Di Flumeri
Amanda Mayappo-Neeposh
Conall Francoeur
Tamara Ellen Carver
Xiao-Wen Chang
Esin Darici Haritaoglu
Ismail Haritaoglu
Akshatha Arodi
Naomi Goloff
Pediatric serious illness communication (SIC) is critically important, yet scalable communication training for clinicians remains limited. C… (voir plus)ompared with other dialogue simulation settings, pediatric SIC poses additional challenges, including multi-party interactions, response to parental distress and strong dependence on feedback dynamics. Existing LLM-based simulators optimize generic dialogue quality rather than curriculum-contingent behavior required for effective SIC training. In collaboration with educators and pediatric clinicians, we introduce the first benchmark suite and simulation framework tailored to pediatric SIC training. Our benchmarks, PitfallBench and DialogueBench, evaluate simulators both at the turn-level and across full dialogues. We further propose SIC-Agents, a self-improving framework that generates a clinician-editable skill document to guide simulator behavior. Our experiments show that SIC-Agents outperforms static expert prompting. To support future research, we release our benchmarks for parent simulation in pediatric SIC at https://github.com/Beikewzh/sic-benchmarks
Analytic Planning under Uncertainty with Moment Closure
Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagat… (voir plus)ing full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable. Consequently, modern deep reinforcement learning has largely retreated to either stochastic sampling, which introduces significant target variance, or deterministic point estimates that ignore predictive covariance entirely. We investigate whether distribution-aware planning is possible without these constraints. Using a quadratic action-value parameterization, we first reduce the Bellman backup to an expectation over the state-value function alone; the key idea is then a compatibility principle between the predictive transition distribution and the value function class, under which this expectation is analytic in the distribution's moments. We instantiate this principle with a Gaussian transition model paired with a radial-basis value function, yielding a closed-form backup that propagates both predictive mean and covariance. Empirically, our approach reduces target variance and yields well-calibrated predictive uncertainty under stochastic observations in continuous control, providing a principled framework for planning with learned distribution models.
Toward a general understanding of neural representations learned by deep neural networks on group multiplications
We study the neural representations learned by deep neural networks trained on alternating group multiplication, discovering that they are S… (voir plus)chreier coset graphs, a generalization of Cayley graphs. Previous works inspected neural representations learned from cyclic group multiplication and identified them as Cayley graphs. Since cyclic group multiplication is Abelian, all subgroups are normal, and Schreier coset graphs reduce to Cayley graphs in this case. This contribution makes a step toward finding a general theory of what neural representations are in networks learning group multiplications.