Dane Malenfant

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Emiliano Penaloza

Dheeraj Vattikonda

Roger Creus Castanyer

Siddarth Venkatraman

Abhay Puri

Jonathan Light

Matthew J. Sargent

Augustine N. Mavor-Parker

M. Caccia

Lucas Caccia

Glen Berseth

Esmeralda S. Whitammer

Minseon Kim

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been… (voir plus) the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

2026-09-27

arXiv (prépublication)

Moral Hazard in Multi-Agent Language Models

Cooperation can fail when socially valuable effort is costly, hard to observe, and benefits mainly someone else. Building on Holmstrom's mod… (voir plus)el of moral hazard in teams, the Dialogue Moral Hazard Game instantiates this hidden-action structure as a textual environment for language agents. An agent chooses between keeping an immediate local reward and paying a query cost to reveal a hidden safety fact that helps another agent's downstream decision. We evaluate fourteen open-weight and four frontier models using measures of information acquisition, communication, downstream use, and team success. In matched 3,015-decision-per-model experiments, GPT-5.6 Sol, Claude Opus 4.8, and Nemotron-3 Ultra track the derived private-share boundary across nine query costs, with mean absolute errors of 0.013, 0.030, and 0.024. Muse Spark 1.1 responds directionally, whereas Fable 5 remains query-saturated. SFT, RLOO, SFT+RLOO, and GEPA produce heterogeneous mechanism changes. GEPA raises Muse team success from 22.2% to 100.0% while reducing query use from 51.1% to 0.3%. Frozen-prompt interventions show that this success depends on a learned rank-label mapping rather than direct revelation: changing the mapping reduces team success from 100.0% to 12.5% and then 0.0%. We introduce CREDIT (Counterfactual Replay for Evidence-Driven Information Transfer), a mechanism-aligned multi-agent prompt-optimization algorithm that uses matched hidden-state twins and total-action replay to reward robust causal contribution rather than query frequency. Across five models and multiple seeds, CREDIT preserves query-mediated behavior while revealing model-specific acquisition and downstream-use bottlenecks. Optimization can reach the same aggregate outcome through direct revelation or a learned effective information structure, motivating mechanism-level evaluation and optimization rather than team success alone.

2026-07-26

arXiv (prépublication)

Compositional Flow Matching with Factored Velocity Fields

Avery Hee-Woon Ryoo

Matthew G Perich

Guillaume Lajoie

Conditional generative models can have difficulty generating attribute combinations absent from training, even when each individual factor i… (voir plus)s densely covered, otherwise known as a failure to compositionally generalize. We propose a factored conditional flow matching architecture that uses a shared base velocity augmented by per-factor heads, summed at the bottleneck. We show that on the Shapes3D and MPI3D-real datasets, the factored architecture matches or beats a parameter-matched monolithic baseline under three structured zero-shot holdout strengths over a two-attribute lattice, notably lowering heldout FID by

2026-05-29

SPIGM @ International Conference on Machine Learning (poster)

openreview.net

Reinforcing the World's Edge: A Continual Learning Problem in the Multi-Agent-World Boundary

Ordinary decentralized multi-agent reinforcement learning presents each focal agent with a continual learning problem: peer updates change i… (voir plus)ts induced rewards and dynamics even when the joint Markov game is stationary. We connect the lifetime of success-conditioned reusable structure to peer learning and policy reuse. An invariant core represents maximal abstract patterns shared by a high fraction of successful trajectories; survival refers to a fixed pattern's coverage, not an unchanged maximal frontier. An established sharp conditioning bound limits coverage loss to

2026-03-05

arXiv (prépublication)

The challenge of hidden gifts in multi-agent reinforcement learning

Blake Aaron Richards

Cooperation between people is not always obvious. Sometimes we benefit from actions that others have taken even when we are unaware that the… (voir plus)y took those actions. For example, if your neighbor chooses not to take a parking spot in front of your house when you are not there, you can benefit, even without being aware that they took this action. These “hidden gifts” represent an interesting challenge for multi-agent reinforcement learning (MARL), since assigning credit to your own actions correctly when the beneficial actions of others are hidden is non-trivial. Here, we study the impact of hidden gifts with a very simple MARL task. In this task, agents in a grid-world environment have individual doors to unlock in order to obtain individual rewards. As well, if all the agents unlock their door the group receives a larger collective reward. However, there is only one key for all of the doors, such that the collective reward can only be obtained when the agents drop the key for others after they use it. Notably, there is nothing to indicate to an agent that the other agents have dropped the key, thus the act of dropping the key for others is a “hidden gift”. We show that several different state-of-the-art RL algorithms, including MARL algorithms, fail to learn how to obtain the collective reward in this simple task. Interestingly, we find that independent model-free policy gradient agents can solve the task when we provide them with information about their action history, but MARL agents still cannot solve the task with action history. Finally, we derive a correction term for these independent agents, inspired by learning aware approaches, which reduces the variance in learning and helps them to converge to collective success more reliably. These results show how credit assignment in multi-agent settings can be particularly challenging in the presence of “hidden gifts”, and demonstrate that learning awareness can benefit these settings

2025-07-16

EWRL/2025/Workshop (poster)

openreview.net

Contrastive Retrospection: honing in on critical steps for rapid learning and generalization in RL

Chen Sun

Wannan Yang

Thomas Jiralerspong

Benjamin Alsbury-Nealy

Yoshua Bengio

Blake Richards

In real life, success is often contingent upon multiple critical steps that are distant in time from each other and from the final reward. T… (voir plus)hese critical steps are challenging to identify with traditional reinforcement learning (RL) methods that rely on the Bellman equation for credit assignment. Here, we present a new RL algorithm that uses offline contrastive learning to hone in on these critical steps. This algorithm, which we call Contrastive Retrospection (ConSpec), can be added to any existing RL algorithm. ConSpec learns a set of prototypes for the critical steps in a task by a novel contrastive loss and delivers an intrinsic reward when the current state matches one of the prototypes. The prototypes in ConSpec provide two key benefits for credit assignment: (i) They enable rapid identification of all the critical steps. (ii) They do so in a readily interpretable manner, enabling out-of-distribution generalization when sensory features are altered. Distinct from other contemporary RL approaches to credit assignment, ConSpec takes advantage of the fact that it is easier to retrospectively identify the small set of steps that success is contingent upon (and ignoring other states) than it is to prospectively predict reward at every taken step. ConSpec greatly improves learning in a diverse set of RL tasks. The code is available at the link: https://github.com/sunchipsster1/ConSpec

2023-12-11

Advances in Neural Information Processing Systems (Accept (poster))