Publications

Capturing Individual Human Preferences with Reward Features
Andre Barreto
Vincent Dumoulin
Yiran Mao
Nicolas Perez-Nieves
Mark Rowland
Bobak Shahriari
Yann Dauphin
Doina Precup
Hugo Larochelle
Reinforcement learning from human feedback usually models preferences using a reward model that does not distinguish between people. We argu… (see more)e that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We propose a method to specialise a reward model to a person or group of people. Our approach builds on the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models comparing the proposed architecture with a non-adaptive reward model and also adaptive counterparts, including models that do in-context personalisation. Depending on how much disagreement there is in the training data, our model either significantly outperforms the baselines or matches their performance with a simpler architecture and more stable training.
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
Detecting High-Stakes Interactions with Activation Probes
Alex McKenzie
Urja Pawar
Phil Blandfort
William Bankes
David M. Krueger
Ekdeep Singh Lubana
Dmitrii Krasheninnikov
Monitoring is an important aspect of safely deploying Large Language Models (LLMs). This paper examines activation probes for detecting "hig… (see more)h-stakes" interactions -- where the text indicates that the interaction might lead to significant harm -- as a critical, yet underexplored, target for such monitoring. We evaluate several probe architectures trained on synthetic data, and find them to exhibit robust generalization to diverse, out-of-distribution, real-world data. Probes' performance is comparable to that of prompted or finetuned medium-sized LLM monitors, while offering computational savings of six orders-of-magnitude. Our experiments also highlight the potential of building resource-aware hierarchical monitoring systems, where probes serve as an efficient initial filter and flag cases for more expensive downstream analysis. We release our novel synthetic dataset and codebase to encourage further study.
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
Distributional Training Data Attribution: What do Influence Functions Sample?
Bruno Mlodozeniec
Isaac Reid
Samuel Power
David M. Krueger
Murat A Erdogdu
Richard E. Turner
Roger Baker Grosse
2025-09-17
NeurIPS.cc/2025/Conference (spotlight)
openreview.net
openreview.net
FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency
Yifei Su
Ning Liu
Dong Chen
Zhen Zhao
Kun Wu
Meng Li
Zhiyuan Xu
Zhengping Che
Jian Tang
2025-09-17
NeurIPS.cc/2025/Conference (poster)
openreview.net
openreview.net
From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization
Shoaib Ahmed Siddiqui
Adrian Weller
David M. Krueger
Gintare Karolina Dziugaite
Michael Curtis Mozer
Eleni Triantafillou
Recent unlearning methods for LLMs are vulnerable to relearning attacks: knowledge believed-to-be-unlearned re-emerges by fine-tuning on a s… (see more)mall set of (even seemingly-unrelated) examples. We study this phenomenon in a controlled setting for example-level unlearning in vision classifiers. We make the surprising discovery that forget-set accuracy can recover from around 50% post-unlearning to nearly 100% with fine-tuning on just the retain set -- i.e., zero examples of the forget set. We observe this effect across a wide variety of unlearning methods, whereas for a model retrained from scratch excluding the forget set (gold standard), the accuracy remains at 50%. We observe that resistance to relearning attacks can be predicted by weight-space properties, specifically,
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
GreenHyperSpectra: A multi-source hyperspectral dataset for global vegetation trait prediction
Eya Cherif
Arthur Ouaknine
Luke A. Brown
Phuong D. Dao
Kyle R. Kovach
Bing Lu
Daniel Mederer
Hannes Feilhauer
Teja Kattenborn
David Rolnick
Plant traits such as leaf carbon content and leaf mass are essential variables in the study of biodiversity and climate change. However, con… (see more)ventional field sampling cannot feasibly cover trait variation at ecologically meaningful spatial scales. Machine learning represents a valuable solution for plant trait prediction across ecosystems, leveraging hyperspectral data from remote sensing. Nevertheless, trait prediction from hyperspectral data is challenged by label scarcity and substantial domain shifts (\eg across sensors, ecological distributions), requiring robust cross-domain methods. Here, we present GreenHyperSpectra, a pretraining dataset encompassing real-world cross-sensor and cross-ecosystem samples designed to benchmark trait prediction with semi- and self-supervised methods. We adopt an evaluation framework encompassing in-distribution and out-of-distribution scenarios. We successfully leverage GreenHyperSpectra to pretrain label-efficient multi-output regression models that outperform the state-of-the-art supervised baseline. Our empirical analyses demonstrate substantial improvements in learning spectral representations for trait prediction, establishing a comprehensive methodological framework to catalyze research at the intersection of representation learning and plant functional traits assessment. All code and data are available at: https://github.com/echerif18/HyspectraSSL.
2025-09-17
NeurIPS.cc/2025/Datasets_and_Benchmarks_Track (poster)
doi.org
openreview.net
Learning to Solve Complex Problems via Dataset Decomposition
Wanru Zhao
Lucas Caccia
Zhengyan Shi
Minseon Kim
Weijia Xu
Xingdi Yuan
Alessandro Sordoni
Marc-Alexandre Côté
2025-09-17
NeurIPS.cc/2025/Conference (poster)
openreview.net
openreview.net
Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Ghada Sokar
Pablo Samuel Castro
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
POCO: Scalable Neural Forecasting through Population Conditioning
Yu Duan
Hamza Tahir Chaudhry
Misha B. Ahrens
Christopher D. Harvey
Matthew G. Perich
Karl Deisseroth
Kanaka Rajan
Predicting future neural activity is a core challenge in modeling brain dynamics, with applications ranging from scientific investigation to… (see more) closed-loop neurotechnology. While recent models of population activity emphasize interpretability and behavioral decoding, neural forecasting-particularly across multi-session, spontaneous recordings-remains underexplored. We introduce POCO, a unified forecasting model that combines a lightweight univariate forecaster with a population-level encoder to capture both neuron-specific and brain-wide dynamics. Trained across five calcium imaging datasets spanning zebrafish, mice, and C. elegans, POCO achieves state-of-the-art accuracy at cellular resolution in spontaneous behaviors. After pre-training, POCO rapidly adapts to new recordings with minimal fine-tuning. Notably, POCO's learned unit embeddings recover biologically meaningful structure-such as brain region clustering-without any anatomical labels. Our comprehensive analysis reveals several key factors influencing performance, including context length, session diversity, and preprocessing. Together, these results position POCO as a scalable and adaptable approach for cross-session neural forecasting and offer actionable insights for future model design. By enabling accurate, generalizable forecasting models of neural dynamics across individuals and species, POCO lays the groundwork for adaptive neurotechnologies and large-scale efforts for neural foundation models. Code is available at https://github.com/yuvenduan/POCO.
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
Risks of AI scientists: prioritizing safeguarding over autonomy
Xiangru Tang
Qiao Jin
Kunlun Zhu
Tongxin Yuan
Yichi Zhang
Wangchunshu Zhou
Meng Qu
Yilun Zhao
Jian Tang
Zhuosheng Zhang
Arman Cohan
Dov Greenbaum
Zhiyong Lu
Mark Gerstein
AI scientists powered by large language models have demonstrated substantial promise in autonomously conducting experiments and facilitating… (see more) scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that require careful consideration for safety. However, there has been limited comprehensive exploration of these vulnerabilities. This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse, and emphasizing the need for safety measures. We begin by providing an overview of the potential risks inherent to AI scientists, taking into account user intent, the specific scientific domain, and their potential impact on the external environment. Then, we explore the underlying causes of these vulnerabilities and provide a scoping review of the limited existing works. Based on our analysis, we propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback (agent regulation) to mitigate these identified risks. Furthermore, we highlight the limitations and challenges associated with safeguarding AI scientists and advocate for the development of improved models, robust benchmarks, and comprehensive regulations.
2025-09-17
Nature Communications (published)
doi.org
SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
Wanxin Tian
Shijie Zhang
Kevin Zhang
Xiaowei Chi
Chun-Kai Fan
Junyu Lu
Yulin Luo
Qiang Zhou
Yiming Zhao
Ning Liu
Siyu Lin
Zhiyuan Qin
Xiaozhu Ju
Shanghang Zhang
Jian Tang
2025-09-17
NeurIPS.cc/2025/Conference (poster)
openreview.net
openreview.net
State Entropy Regularization for Robust Reinforcement Learning
Yonatan Ashlag
Uri Koren
Mirco Mutti
Esther Derman
Pierre-Luc Bacon
Shie Mannor
State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its the… (see more)oretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.
2025-09-17
NeurIPS.cc/2025/Conference (oral)
doi.org
openreview.net