Portrait de Matthew Riemer n'est pas disponible

Matthew Riemer

Doctorat - UdeM
Superviseur⋅e principal⋅e
Irina Rish
Co-supervisor
Irina Rish
Sujets de recherche
Apprentissage par renforcement

Publications

Emergent Reasoning via Recursive Latent Reinforcement Pretraining
Gopeshh Subbaraj
Istabrak Abbes
Artem Zholus
Matthew Riemer
Irina Rish
Sarath Chandar
Large language models (LLMs) often rely on explicit chain-of-thought (CoT) traces to solve multi-step reasoning problems, but these traces i… (voir plus)ncrease inference cost, expose brittle prompt dependence, and complicate training objectives. We study an alternative: \emph{latent deliberation} implemented as a small recurrent refinement module that performs multiple internal ``thinking`` steps while keeping the external sequence length fixed. We introduce \textbf{Recursive Latent Reinforcement Pretraining (RLRP)}, a training recipe that augments a base causal LLM with a shared latent head executed for
2026-03-04
LLM_Reasoning @ International Conference on Learning Representations (publié)
openreview.net
openreview.net
Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisions
Matthew Riemer
Tommaso Tosato
Maximilian Puelma Touzel
Amin Memarian
Guillaume Dumas
Glen Berseth
Irina Rish
This position paper argues that AI agents with chain-of-thought reasoning capabilities are predisposed to exhibit collusive behavior and sho… (voir plus)uld be required to obtain behavioral certification before making decisions that affect economic markets. This is because integrating these agents into society could collapse the legal evidentiary distinction between competition and collusion among independent firms without eroding the economic harm distinction. Experiments with DeepSeek-R1 agents in the Bertrand oligopoly pricing domain reveal a tendency towards tacit collusion that persists even when humans prompt the agents not to collude. We further show that the chain-of- thought of these agents can be steered toward either extremely collusive or highly competitive behavior in a way that is not semantically detectable by another LLM analyzing the reasoning traces. As a result, deploying reasoning agents for market decisions leads to collusive economic outcomes without any evidence of conspiracy or intent. Thus, certification based on observed behavior in representative situations is necessary to prevent collusion. We provide preliminary evidence that such agents can be steered in a generalizable way toward efficient competitive equilibria. However, developing a comprehensive behavioral certification will be required before these models can be deployed in real-world markets while ensuring their stability and efficiency.
2025-12-31
International Conference on Machine Learning (Accept (regular))
openreview.net
openreview.net
Handling Delay in Real-Time Reinforcement Learning
Ivan Anokin
Rishav Rishav
Matthew Riemer
Stephen Chung
Irina Rish
Samira Ebrahimi Kahou
Real-time reinforcement learning (RL) introduces several challenges. First, policies are constrained to a fixed number of actions per second… (voir plus) due to hardware limitations. Second, the environment may change while the network is still computing an action, leading to observational delay. The first issue can partly be addressed with pipelining, leading to higher throughput and potentially better policies. However, the second issue remains: if each neuron operates in parallel with an execution time of
2025-01-21
International Conference on Learning Representations (poster)
doi.org
openreview.net
Balancing Context Length and Mixing Times for Reinforcement Learning at Scale
Sarath Chandar
Khimya Khetarpal
Janarthanan Rajendran
Matthew Riemer
Due to the recent remarkable advances in artificial intelligence, researchers have begun to consider challenging learning problems such as … (voir plus)learning to generalize behavior from large offline datasets or learning online in non-Markovian environments. Meanwhile, recent advances in both of these areas have increasingly relied on conditioning policies on large context lengths. A natural question is if there is a limit to the performance benefits of increasing the context length if the computation needed is available. In this work, we establish a novel theoretical result that links the context length of a policy to the time needed to reliably evaluate its performance (i.e., its mixing time) in large scale partially observable reinforcement learning environments that exhibit latent sub-task structure. This analysis underscores a key tradeoff: when we extend the context length, our policy can more effectively model non-Markovian dependencies, but this comes at the cost of potentially slower policy evaluation and as a result slower downstream learning. Moreover, our empirical results highlight the relevance of this analysis when leveraging Transformer based neural networks. This perspective will become increasingly pertinent as the field scales towards larger and more realistic environments, opening up a number of potential future directions for improving the way we design learning agents.
2024-09-24
NeurIPS.cc/2024/Conference (poster)
doi.org
openreview.net
Continual Learning In Environments With Polynomial Mixing Times
Matthew Riemer
Sharath Chandra Raparthy
Ignacio Cases
Gopeshh Subbaraj
Maximilian Puelma Touzel
Irina Rish
The mixing time of the Markov chain induced by a policy limits performance in real-world continual learning scenarios. Yet, the effect of mi… (voir plus)xing times on learning in continual reinforcement learning (RL) remains underexplored. In this paper, we characterize problems that are of long-term interest to the development of continual RL, which we call scalable MDPs, through the lens of mixing times. In particular, we theoretically establish that scalable MDPs have mixing times that scale polynomially with the size of the problem. We go on to demonstrate that polynomial mixing times present significant difficulties for existing approaches, which suffer from myopic bias and stale bootstrapped estimates. To validate our theory, we study the empirical scaling behavior of mixing times with respect to the number of tasks and task duration for high performing policies deployed across multiple Atari games. Our analysis demonstrates both that polynomial mixing times do emerge in practice and how their existence may lead to unstable learning behavior like catastrophic forgetting in continual learning settings.
2021-12-31
Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (publié)
doi.org
openreview.net
Sequoia: A Software Framework to Unify Continual Learning Research
Fabrice Normandin
Florian Golemo
Oleksiy Ostapenko
Pau Rodríguez
Matthew Riemer
J. Hurtado
Lucas Cecchi
Khimya Khetarpal
Dominic Zhao
Ryan Lindeborg
Timothee LESORT
David Vázquez
Laurent Charlin
Irina Rish
Massimo Caccia
The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non… (voir plus)-stationary environments. In practice, a plethora of evaluation procedures (settings) and algorithmic solutions (methods) exist, each with their own potentially disjoint set of assumptions. This variety makes measuring progress in CL difficult. We propose a taxonomy of settings, where each setting is described as a set of assumptions. A tree-shaped hierarchy emerges from this view, where more general settings become the parents of those with more restrictive assumptions. This makes it possible to use inheritance to share and reuse research, as developing a method for a given setting also makes it directly applicable onto any of its children. We instantiate this idea as a publicly available software framework called Sequoia, which features a wide variety of settings from both the Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL) domains. Sequoia also includes a growing suite of methods which are easy to extend and customize, in addition to more specialized methods from external libraries. We hope that this new paradigm and its first implementation can help unify and accelerate research in CL. You can help us grow the tree by visiting www.github.com/lebrice/Sequoia.
2021-08-01
ArXiv (prépublication)
doi.org
openreview.net