Marc-Alexandre Côté

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Dane Malenfant

Dheeraj Vattikonda

Roger Creus Castanyer

Siddarth Venkatraman

Abhay Puri

Jonathan Light

Matthew J. Sargent

Augustine N. Mavor-Parker

M. Caccia

Lucas Caccia

Glen Berseth

Esmeralda S. Whitammer

Minseon Kim

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been… (see more) the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

2026-09-27

arXiv (preprint)

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Dane Malenfant

Dheeraj Vattikonda

Roger Creus Castanyer

Siddarth Venkatraman

Abhay Puri

Jonathan Light

Matthew J. Sargent

Augustine N. Mavor-Parker

M. Caccia

Lucas Caccia

Glen Berseth

Esmeralda S. Whitammer

Minseon Kim

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been… (see more) the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

2026-09-27

arXiv (preprint)

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

Jonathan Light

Christopher Zhang Cui

Jeonghye Kim

Roger Creus Castanyer

Zhengyan Shi

Xingdi Yuan

Minseon Kim

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding… (see more) to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.

2026-09-27

arXiv (preprint)

FrogNano: Training a 4B Coding Agent via Online Task Synthesis

Minseon Kim

Zhengyan Shi

Christopher Cui

Roger Creus Castanyer

Maryam Hashemzadeh

Isadora White

Jonathan Light

Jeonghye Kim

Matheus Pereira

Darya Moldavskaya

Chinmay Singh

Fabio Vera

Baolin Peng

Xingdi Yuan

We present FrogNano, a 4B coding agent designed to tackle software engineering (SWE) tasks efficiently and effectively, even under resource-… (see more)constrained environments. It is post-trained exclusively via RL on around 1,500 SWE environments with synthetic tasks. A key ingredient for improving performance is an online task synthesis pipeline that creates tasks calibrated to the frontier of learnability for the current checkpoint. This report provides evidence that competitive small coding agents can be trained with synthetic tasks alone, without traditional distillation from larger models, and that generating tasks at the learnability frontier of the current agent is important. We report details on the training methodology, evaluations across diverse environments, and in-depth analyses, serving as a foundation for our ongoing exploration of lightweight yet capable coding agents that can run on minimal hardware.

2026-09-06

arXiv (preprint)

Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

Minseon Kim

2025-12-31

Neural Information Processing Systems (Accept (poster))

openreview.net

Gistify: Codebase-Level Understanding via Runtime Execution

Hyunji Lee

Minseon Kim

Chinmay Singh

Matheus Pereira

Atharv Sonwane

Isadora White

Elias Stengel-Eskin

Mohit Bansal

Zhengxiang Shi

Eric Yuan

Lucas Caccia

As coding agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is ce… (see more)ntral. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.

2025-12-31

International Conference on Learning Representations (Accept (Poster))

openreview.net

Learning to Extract Context for Context-Aware LLM Inference

Minseon Kim

Lucas Caccia

Zhengyan Shi

Matheus Pereira

Xingdi Yuan

2025-12-11

ArXiv (preprint)

Gistify! Codebase-Level Understanding via Runtime Execution

Hyunji Lee

Minseon Kim

Chinmay Singh

Matheus Pereira

Atharv Sonwane

Isadora White

Elias Stengel-Eskin

Mohit Bansal

Zhengyan Shi

Xingdi Yuan

Lucas Caccia

As coding agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is ce… (see more)ntral. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.

2025-10-29

ArXiv (preprint)