Publications

At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization
Praneet Suresh
Jack Stanley
Sonia Joseph
Luca Scimeca
Danilo Bzdok
Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. … (see more)Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-of-distribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of fallacious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to robustify LLMs. Expanding the very notion of OOD from input data to a model’s private computational processes—a new transformer diagnostic at inference time—is a critical step toward making AI systems safe for deployment across science, business, and government.
2025-12-31
International Conference on Machine Learning (Accept (regular))
openreview.net
openreview.net
The Expressive Limits of Diagonal SSMs for State-Tracking
Mehran Shakerinava
Behnoush Khavari
Siamak Ravanbakhsh
Sarath Chandar
State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling task… (see more)s while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited. In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) State-Space Models (SSMs) on sequential state-tracking tasks for abstract groups. It is easy to show that a single DCD SSM layer with a universal decoder can track any Abelian group at finite precision by decomposing it into a product of cyclic groups. We show that this is tight by proving that such a model cannot track any non-Abelian group at finite precision. We further establish the expressivity of multi-layer DCD SSMs. We show that a
2025-12-31
International Conference on Learning Representations (Accept (Poster))
doi.org
openreview.net
The Future of the AI Summit Series: One Year Later
Lucia Velasco
Yoshua Bengio
Robert Trager
Joslyn Barnhart
Julia Morse
Roxana Radu
Ranjit Lall
Stephen Casper
Charles Martinet
2025-12-31
SSRN Electronic Journal (accepted)
doi.org
The Geometry and Topology of Circuits: the Manifolds of Modular Addition
Gabriela Moisescu-Pareja
Gavin McCracken
Harley Wiltzer
Vincent Létourneau
Colin Daniels
Doina Precup
Jonathan Love
The Clock and Pizza interpretations, associated with architectures differing in either uniform or learnable attention, were introduced to ar… (see more)gue that different architectural designs can yield distinct circuits for modular addition. In this work, we show that this is not the case, and that both the uniform and trainable attention architectures implement the same algorithm via topologically and geometrically equivalent representations. Our methodology goes beyond the interpretation of individual neurons and weights. Instead, we identify all of the neurons corresponding to each learned representation and then study the collective group of neurons as one entity. This method reveals that each learned representation is a manifold that we can study utilizing tools from topology. Based on this insight, we can statistically analyze the learned representations across hundreds of circuits to demonstrate the similarity between learned modular addition circuits that arise naturally from common deep learning paradigms.
2025-12-31
International Conference on Learning Representations (Accept (Poster))
openreview.net
openreview.net
The Impact of Pediatric Surgery Global Travel Fellowships: A Study by the Canadian Association of Paediatric Surgeons Global Partnership Committee.
Sacha Williams
Natasha Bejjani
Elena Guadagno
Robert Baird
Shahrzad Joharifard
Melanie Morris
Robin Petroze
Dan Poenaru
Sherif Emil
2025-12-31
Journal of Pediatric Surgery (published)
doi.org
The Intricate Dance of Prompt Complexity, Quality, Diversity and Consistency in T2I Models
Zhang Xiaofeng
Aaron Courville
Michal Drozdzal
Adriana Romero-Soriano
Text-to-image (T2I) models offer great potential for creating virtually limitless synthetic data, a valuable resource compared to fixed and … (see more)finite real datasets. Previous works evaluate the utility of synthetic data from T2I models on three key desiderata: quality, diversity, and consistency. While prompt engineering is the primary means of interacting with T2I models, the systematic impact of prompt complexity on these critical utility axes remains underexplored. In this paper, we first conduct synthetic experiments to motivate the difficulty of generalization w.r.t. prompt complexity and explain the observed difficulty with theoretical derivations. Then, we introduce a new evaluation framework that can compare the utility of real data and synthetic data, and present a comprehensive analysis of how prompt complexity influences the utility of synthetic data generated by commonly used T2I models. We conduct our study across diverse datasets, including CC12M, ImageNet-1k, and DCI, and evaluate different inference-time intervention methods. Our synthetic experiments show that generalizing to more general conditions is harder than the other way round, since the former needs an estimated likelihood that is not learned by diffusion models. Our large-scale empirical experiments reveal that increasing prompt complexity results in lower conditional diversity and prompt consistency, while reducing the synthetic-to-real distribution shift, which aligns with the synthetic experiments. Moreover, current inference-time interventions can augment the diversity of the generations at the expense of moving outside the support of real data. Among those interventions, prompt expansion, by deliberately using a pre-trained language model as a likelihood estimator, consistently achieves the highest performance in both image diversity and aesthetics, even higher than that of real data. Combining advanced guidance interventions with prompt expansion results in the most appealing utility trade-offs of synthetic data.
2025-12-31
International Conference on Learning Representations (Accept (Poster))
openreview.net
openreview.net
The Markovian Thinker
Milad Aghajohari
Kamran Chitsaz
Amirhossein Kazemnejad
Sarath Chandar
Alessandro Sordoni
Aaron Courville
Siva Reddy
Reasoning LLMs suffer from quadratic compute growth as their context length increases, making reinforcement learning with verifiable rewards… (see more) (RLVR) and test-time scaling prohibitively expensive. Prior work has tried to lighten the computational burden by shortening reasoning traces through pruning, summarization, or multi-stage training, but these methods remain bound to quadratic costs. We introduce Delethink, a thinking algorithm that realizes the Markovian Thinking Paradigm. Instead of producing one long monolithic reasoning trace, Delethink thinks in a sequence of chunks, the Delethink trace. Each chunk continues reasoning by referring only to a fixed number of prior tokens, which functions as a Markovian state sufficient for progressing reasoning, while deleting the rest. This preserves continuity without carrying the quadratic baggage. As a result, compute scales linearly and peak memory remains constant. In experiments, we show that Delethink can be applied directly to off-the-shelf reasoning models ranging from
2025-12-31
International Conference on Learning Representations (Accept (Poster))
openreview.net
openreview.net
On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning
Sacha Morin
Moonsub Byeon
Alexia Jolicoeur-Martineau
Sébastien Lachapelle
Semi-supervised imitation learning (SSIL) consists in learning a policy from a small dataset of action-labeled trajectories and a much large… (see more)r dataset of action-free trajectories. Some SSIL methods learn an inverse dynamics model (IDM) to predict the action from the current state and the next state. An IDM can act as a policy when paired with a video model (VM-IDM) or as a label generator to perform behavior cloning on action-free data (IDM labeling). In this work, we first show that VM-IDM and IDM labeling learn the same policy in a limit case, which we call the IDM-based policy. We then argue that the previously observed advantage of IDM-based policies over behavior cloning is due to the superior sample efficiency of IDM learning, which we attribute to two causes: (i) the ground-truth IDM tends to be contained in a lower complexity hypothesis class relative to the expert policy, and (ii) the ground-truth IDM is often less stochastic than the expert policy. We argue these claims based on insights from statistical learning theory and novel experiments, including a study of IDM-based policies using recent architectures for unified video-action prediction (UVA). Motivated by these insights, we finally propose an improved version of the existing LAPO algorithm for latent action policy learning. We experiment on the ProcGen, Push-T and Libero benchmarks.
2025-12-31
International Conference on Machine Learning (Accept (regular))
doi.org
openreview.net
On the sensitivity of SAR C- and L-band dual-polarized data for detection of early deforestation in the tropics
Africa I. Flores-Anderson
Jeffrey A. Cardille
Josef Kellndorfer
Franz J. Meyer
Pontus Olofsson
2025-12-31
Remote Sensing of Environment (published)
doi.org
The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning
Donghang Wu
Tianyu Zhang
Yuxin Li
Hexin Liu
Chen Chen
Eng Siong Chng
Yoshua Bengio
During conversational interactions, humans subconsciously engage in concurrent thinking while listening to a speaker. Although this internal… (see more) cognitive processing may not always manifest as explicit linguistic structures, it is instrumental in formulating high-quality responses. Inspired by this cognitive phenomenon, we propose a novel **F**ull-duplex **LA**tent and **I**nternal **R**easoning method named FLAIR that conducts *latent* thinking simultaneously with speech perception. Unlike conventional "thinking" mechanisms in NLP, which require post-hoc generation, our approach aligns seamlessly with spoken dialogue systems: during the user’s speaking phase, it recursively feeds the latent embedding output from the previous step into the next step, enabling continuous reasoning that strictly adheres to causality without introducing additional latency. To enable this latent reasoning, we design an Evidence Lower Bound-based objective that supports efficient supervised finetuning via teacher forcing, circumventing the need for explicit reasoning annotations. Experiments demonstrate the effectiveness of this think-while-listening design, which achieves competitive results on a range of speech benchmarks. Furthermore, FLAIR robustly handles conversational dynamics and attains competitive performance on full-duplex interaction metrics.
2025-12-31
International Conference on Machine Learning (Accept (regular))
doi.org
openreview.net
The Surprising Difficulty of Search in Model-Based Reinforcement Learning
Wei-Di Chang
Mikael Henaff
Brandon Amos
Gregory Dudek
Scott Fujimoto
This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compound… (see more)ing errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a plug-and-play replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating distribution shift matters more than improving model or value function accuracy. Building on this insight, we identify key techniques for enabling effective search, achieving state-of-the-art performance across multiple popular benchmark domains.
2025-12-31
arXiv (preprint)
doi.org
arxiv.org
On The Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Tongtian Zhu
Tianyu Zhang
Mingze Wang
Zhanpeng Zhou
Can Wang
Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited … (see more)peer-to-peer communication. In this paper, we study how communication should be scheduled over time to improve global generalization, including determining when and how frequently devices synchronize. Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global generalization. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significant improve the generalization performance of decentralized learning under serve high data heterogeneity. Our theoretical contributions, which explains these phenomena, are first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD. Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides promising results that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research. The code will be made publicly available.
2025-12-31
International Conference on Learning Representations (Accept (Oral))
openreview.net
openreview.net