Publications

Towards Sustainable Investment Policies Informed by Opponent Shaping
Juan Agustin Duque
Razvan Ciuca
Ayoub Echchahed
Hugo Larochelle
Aaron Courville
Addressing climate change requires global coordination, yet rational economic actors often prioritize immediate gains over collective welfar… (see more)e, resulting in social dilemmas. InvestESG is a recently proposed multi-agent simulation that captures the dynamic interplay between investors and companies under climate risk. We provide a formal characterization of the conditions under which InvestESG exhibits an intertemporal social dilemma, deriving theoretical thresholds at which individual incentives diverge from collective welfare. Building on this, we apply Advantage Alignment, a scalable opponent shaping algorithm shown to be effective in general-sum games, to influence agent learning in InvestESG. We offer theoretical insights into why Advantage Alignment systematically favors socially beneficial equilibria by biasing learning dynamics toward cooperative outcomes. Our results demonstrate that strategically shaping the learning processes of economic agents can result in better outcomes that could inform policy mechanisms to better align market incentives with long-term sustainability goals.
2025-06-23
rl-conference.cc/RLC/2025/Workshop/CoCoMARL (poster)
openreview.net
openreview.net
Rethinking Full Finetuning from Pretraining Checkpoints in Active Learning for African Languages
Bonaventure F. P. Dossou
Ines Arous
Jackie Cheung
2025-06-22
aclweb.org/ACL/2025/SRW (poster)
openreview.net
openreview.net
Revisiting Laplacian Representations for Value Function Approximation in Deep RL
Priyesh Vijayan
Padideh Nouri
Rishav
Sarath Chandar
Yash Chandak
Mathieu Reymond
Samira Ebrahimi Kahou
Doina Precup
Proto-value functions (PVFs) introduced Laplacian embeddings as an effective feature basis for value-function approximation; however, their … (see more)utility remained limited to small, fully known state spaces. Recent work has scaled Laplacian embeddings to high-dimensional inputs, using them for reward shaping and option discovery in goal-directed tasks, yet only as auxiliary signals, rather than directly using them as features for value functions. In this paper, we learn Laplacian eigenvectors online and employ them as features for Q-learning in 23 Atari games. We empirically demonstrate that these online–learned embeddings substantially improve model-free RL in large, high-dimensional domains. We demonstrate that enriching state representations with action embeddings yields additional gains under both behavior-policy and uniform-random policies. Additionally, we introduce the Fusion architecture, which augments the representation with useful inductive bias at the embedding level. To assess the usefulness of each embedding used in the Fusion architecture, we use Shapley values analysis.
2025-06-22
rl-conference.cc/RLC/2025/Workshop/IBRL (published)
openreview.net
openreview.net
Training PPO-Clip with Parallelized Data Generation: A Case of Fixed-Point Convergence
Homayoun Honari
Roger Creus Castanyer
Pablo Samuel Castro
Glen Berseth
In recent years, with the increase in the compute power of GPUs, parallelized data collection has become the dominant approach for training … (see more)reinforcement learning (RL) agents. Proximal Policy Optimization (PPO) is one of the widely-used on-policy methods for training RL agents. In this paper, we focus on the training behavior of PPO-Clip with the increase in the number of parallel environments. In particular, we show that as we increase the amount of data used to train PPO-Clip, the optimized policy would converge to a fixed distribution. We use the results to study the behavior of PPO-Clip in two case studies: the effect of change in the minibatch size and the effect of increase in the number of parallel environments versus the increase in the rollout lengths. The experiments show that settings with high-return PPO runs result in slower convergence to the fixed-distribution and higher consecutive KL divergence changes. Our results aim to offer a better understanding for the prediction of the performance of PPO with the scaling of the parallel environments.
2025-06-22
rl-conference.cc/RLC/2025/Workshop/IBRL (published)
openreview.net
openreview.net
Alveolar epithelial cell plasticity and injury memory in human pulmonary fibrosis
Taylor S Adams
Jonas C Schupp
Agshin Balayev
Johad Khoury
Aurelien Justet
Fadi Nikola
Laurens J De Sadeleer
De Sadeleer J Laurens
Juan Cala Garcia
Marta Zapata-Ortega
Panayiotis V Benos
Benos V Panayiotis
P.V. Benos
John E McDonough
Farida Ahangari
Melanie Königshoff
Jun Ding
Robert J Homer
Ivan O Rosas
Xiting Yan … (see 3 more)
Bart M Vanaudenaerde
Wim A Wuyts
Naftali Kaminski
2025-06-21
bioRxiv (preprint)
doi.org
Neuromorphic hierarchical modular reservoirs
Filip Milisav
Andrea I Luppi
Laura E Suárez
Guillaume Lajoie
Bratislav Mišić
2025-06-21
bioRxiv (preprint)
doi.org
Behavioral Suite Analysis of Self-Supervised Learning in Atari
Somjit Nath
Rishav
Gopeshh Subbaraj
Derek Nowrouzezahrai
Samira Ebrahimi Kahou
2025-06-20
rl-conference.cc/RLC/2025/Workshop/RLVG (accepted)
openreview.net
openreview.net
A deep generative model for deciphering cellular dynamics and in silico drug discovery in complex diseases.
Yumin Zheng
Jonas C Schupp
Taylor S Adams
Geremy Clair
Aurelien Justet
Farida Ahangari
Xiting Yan
Paul Hansen
Marianne Carlon
Emanuela Cortesi
Marie Vermant
Robin Vos
De Sadeleer J Laurens
Ivan O Rosas
Ricardo Pineda
John Sembrat
Melanie Königshoff
John E McDonough
Bart M. Vanaudenaerde
Wim A Wuyts … (see 2 more)
Naftali Kaminski
Jun Ding
2025-06-20
Nature Biomedical Engineering (published)
doi.org
Human-AI Alignment of Learning Trajectories in Video Games: a continual RL benchmark proposal
Yann Harel
Lune Bellec
François Paugam
Hugo Delhaye
Audrey Durand
We propose a design for a continual reinforcement learning (CRL) benchmark called GHAIA, centered on human-AI alignment of learning trajecto… (see more)ries in structured video game environments. Using \textit{Super Mario Bros.} as a case study, gameplay is decomposed into short, annotated scenes organized into diverse task sequences based on gameplay patterns and difficulty. Evaluation protocols measure both plasticity and stability, with flexible revisit and pacing schedules. A key innovation is the inclusion of high-resolution human gameplay data collected under controlled conditions, enabling direct comparison of human and agent learning. In addition to adapting classical CRL metrics like forgetting and backward transfer, we introduce semantic transfer metrics capturing learning over groups of scenes sharing similar game patterns. We demonstrate the feasibility of our approach on human and agent data, and discuss key aspects of the first release for community input.
2025-06-20
rl-conference.cc/RLC/2025/Workshop/RLVG (accepted)
openreview.net
openreview.net
Human-AI Alignment of Learning Trajectories in Video Games: a continual RL benchmark proposal
Yann Harel
Lune Bellec
François Paugam
Hugo Delhaye
Audrey Durand
We propose a design for a continual reinforcement learning (CRL) benchmark called GHAIA, centered on human-AI alignment of learning trajecto… (see more)ries in structured video game environments. Using \textit{Super Mario Bros.} as a case study, gameplay is decomposed into short, annotated scenes organized into diverse task sequences based on gameplay patterns and difficulty. Evaluation protocols measure both plasticity and stability, with flexible revisit and pacing schedules. A key innovation is the inclusion of high-resolution human gameplay data collected under controlled conditions, enabling direct comparison of human and agent learning. In addition to adapting classical CRL metrics like forgetting and backward transfer, we introduce semantic transfer metrics capturing learning over groups of scenes sharing similar game patterns. We demonstrate the feasibility of our approach on human and agent data, and discuss key aspects of the first release for community input.
2025-06-20
rl-conference.cc/RLC/2025/Workshop/RLVG (published)
openreview.net
openreview.net
Robust Reinforcement Learning for Discrete Compositional Generation via General Soft Operators
Marco Jiralerspong
Esther Derman
Danilo Vucetic
Nikolay Malkin
Bilun Sun
Tianyu Zhang
Pierre-Luc Bacon
Gauthier Gidel
A major bottleneck in scientific discovery involves narrowing a large combinatorial set of objects, such as proteins or molecules, to a smal… (see more)l set of promising candidates. While this process largely relies on expert knowledge, recent methods leverage reinforcement learning (RL) to enhance this filtering. They achieve this by estimating proxy reward functions from available datasets and using regularization to generate more diverse candidates. These reward functions are inherently uncertain, raising a particularly salient challenge for scientific discovery. In this work, we show that existing methods, often framed as sampling proportional to a reward function, are inadequate and yield suboptimal candidates, especially in large search spaces. To remedy this issue, we take a robust RL approach and introduce a unified operator that seeks robustness to the uncertainty of the proxy reward function. This general operator targets peakier sampling distributions while encompassing known soft RL operators. It also leads us to a novel algorithm that identifies higher-quality, diverse candidates in both synthetic and real-world tasks. Ultimately, our work offers a new, flexible perspective on discrete compositional generation tasks. Code: https://github.com/marcojira/tgm.
2025-06-20
ArXiv (preprint)
arxiv.org
arxiv.org
Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
Samin Yeasar Arnob
Scott Fujimoto
Doina Precup
In this paper, we investigate the use of small datasets in the context of offline reinforcement learning (RL). While many common offline RL … (see more)benchmarks employ datasets with over a million data points, many offline RL applications rely on considerably smaller datasets. We show that offline RL algorithms can overfit on small datasets, resulting in poor performance. To address this challenge, we introduce"Sparse-Reg": a regularization technique based on sparsity to mitigate overfitting in offline reinforcement learning, enabling effective learning in limited data settings and outperforming state-of-the-art baselines in continuous control.
2025-06-20
ArXiv (preprint)
arxiv.org
arxiv.org