Irina Rish

rishika.bhagwatkar@mila.quebec

Biographie

Irina Rish est professeure titulaire à l'Université de Montréal (UdeM), où elle dirige le Laboratoire d'IA autonome. Membre du corps professoral de Mila – Institut québécois d’intelligence artificielle, elle est titulaire d'une chaire d'excellence en recherche du Canada (CERC) et d'une chaire en IA Canada-CIFAR. Irina dirige le projet INCITE du ministère américain de l'Environnement au sujet des modèles de fondation évolutifs sur les superordinateurs Summit et Frontier à l'Oak Ridge Leadership Computing Facility (OLCF). Elle est cofondatrice et directrice scientifique de Nolano.ai.

Ses recherches actuelles portent sur les lois de mise à l'échelle neuronale et les comportements émergents (capacités et alignement) dans les modèles de fondation, ainsi que sur l'apprentissage continu, la généralisation hors distribution et la robustesse. Avant de se joindre à l'UdeM en 2019, Irina était chercheuse au Centre de recherche IBM Thomas J. Watson, où elle a travaillé sur divers projets à l'intersection des neurosciences et de l'IA, et dirigé le défi NeuroAI. Elle a reçu plusieurs prix IBM : ceux de l’excellence et de l’innovation exceptionnelle (2018), celui de la réalisation technique exceptionnelle (2017), et celui de l’accomplissement en recherche (2009). Elle détient 64 brevets et a écrit plus de 120 articles de recherche, plusieurs chapitres de livres, trois livres publiés et une monographie sur la modélisation éparse.

Étudiants actuels

Ivan Anokhin

Doctorat - UdeM

Co-superviseur⋅e :

Samira Ebrahimi Kahou

ivan.anokhin@mila.quebec

Doctorat - UdeM

Arjun Ashok

Doctorat - UdeM

Co-superviseur⋅e :

Alexandre Drouin

arjun.ashok@mila.quebec

Maîtrise recherche - UdeM

Doctorat - UdeM

ethan.caballero@mila.quebec

Pietro Cagnasso

Doctorat - Concordia

Superviseur⋅e principal⋅e :

pietro.cagnasso@mila.quebec

Mohammad Javad Darvishi Bayazi

Amin Darabi

Doctorat - UdeM

amin.darabi@mila.quebec

Collaborateur·rice de recherche - UdeM

mj.darvishi92@gmail.com

nadhir.hassen@mila.quebec

Wagner Drew

Maîtrise recherche - Concordia

Superviseur⋅e principal⋅e :

Mirco Ravanelli

drew.wagner@mila.quebec

Doctorat - UdeM

Visiteur de recherche indépendant - -

parviz.haggi@gmail.com

Nadhir Hassen

Collaborateur·rice de recherche - UdeM

Niklas Herbster

Stagiaire de recherche - UdeM

Co-superviseur⋅e :

Gauthier Gidel

niklas.herbster@gmail.com

prateek.humane@mila.quebec

Mai-Thi Ho

Maîtrise recherche - UdeM

mai-thi.ho@mila.quebec

Prateek Humane

Doctorat

arnav-kumar.jain@mila.quebec

Nizar Islah

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Eilif B. Muller

nizar.islah@mila.quebec

Doctorat - UdeM

Doctorat - UdeM

ayush.kaushal@mila.quebec

Arian Khorasani

Collaborateur·rice de recherche - UdeM

arian.khorasani@mila.quebec

Neeraj Kumar

Collaborateur·rice alumni - UdeM

neeraj.kumar@mila.quebec

Gwen Legate

Doctorat - Concordia

Superviseur⋅e principal⋅e :

gwendolyne.legate@mila.quebec

David Lemay

Maîtrise recherche - UdeM

david.lemay@mila.quebec

amin.mansouri@mila.quebec

Amin Mansouri

Collaborateur·rice alumni - UdeM

Collaborateur·rice de recherche

Doctorat - UdeM

Collaborateur·rice de recherche - UdeM

diganta.misra@mila.quebec

motahareh.pourrahimi@mila.quebec

Gabriela Moisescu-Pareja

Collaborateur·rice de recherche - McGill

Superviseur⋅e principal⋅e :

Doina Precup

moisescg@mila.quebec

Timothy Nest

Doctorat - UdeM

Co-superviseur⋅e :

Eilif B. Muller

timothy.nest@mila.quebec

Jonathan Pilault

Collaborateur·rice de recherche - Polytechnique

Doctorat - McGill

Superviseur⋅e principal⋅e :

Pouya Bashivan

Mahta Ramezanian

Maîtrise recherche - UdeM

Co-superviseur⋅e :

Guillaume Dumas

mahta.ramezanian@mila.quebec

Matthew Riemer

Doctorat - UdeM

matthew.riemer@mila.quebec

Alexis Roger

Doctorat - McGill

Superviseur⋅e principal⋅e :

Blake Richards

alexis.roger@mila.quebec

munish.sathish-kumar@mila.quebec

Munish Sathish Kumar

Collaborateur·rice de recherche

Zibo Shang

Doctorat - UdeM

zibo.shang@mila.quebec

Vaibhav Singh

Doctorat - Concordia

Superviseur⋅e principal⋅e :

vaibhav.singh@mila.quebec

Gopeshh Subbaraj

Doctorat - UdeM

gopeshh.subbaraj@mila.quebec

Doctorat - UdeM

Co-superviseur⋅e :

benjamin.therien@mila.quebec

Collaborateur·rice de recherche - UdeM

tejas.vaidhya@mila.quebec

andrew.williams@mila.quebec

Sihui Wei

Baccalauréat - McGill

sihui.wei@mila.quebec

Andrew Williams

Doctorat - UdeM

Co-superviseur⋅e :

Nicolas Chapados

daria.yasafova@mila.quebec

Daria Yasafova

Maîtrise recherche - UdeM

He Zhu

Doctorat - McGill

he.zhu@mila.quebec

Publications

What Does Layer-Importance Reveal About Transformers and State-Space Models?

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the anal… (voir plus)ytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. \emph{Necessity} captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. \emph{Plasticity} captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to

2026-09-14

arXiv (prépublication)

Temporal Recurrence Favors Fewer Layers

Ivan Anokhin

Johan Obando-Ceron

Sebastian Risi

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced ear… (voir plus)lier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

2026-09-10

arXiv (prépublication)

What Survives KV-Cache Compression for Reasoning? A Dual-Channel View of Structural and Associative Memory

Md Rifat Arefin

Long reasoning traces make Transformer key–value (KV) caches grow linearly with generation length. Existing compression methods adopt diff… (voir plus)erent memory topologies, but it is unclear how those choices affect the information that survives eviction. We observe a task–fidelity mismatch: dense summaries achieve low KV reconstruction error yet lose information required for structured reasoning. We explain this through a dual-channel model of compressed memory consisting of a temporally organized scaffold and sparse associative bindings, suggesting that structure and association should be stored separately. We instantiate this idea as TT–Delta, which combines a Tensor-Train structural state with a Delta-rule associative state. On a frozen Qwen2.5-0.5B task model, TT–Delta compresses an evicted prefix by 25.3× (46,680 persistent scalars) and improves exact-answer generation on sequential linear-system reasoning from 0.302±0.113 to 0.839±0.171. In contrast, multi-head Delta is strongest on associative recall and branching graph search. These results suggest that no single compressed-memory topology is universally optimal; the appropriate topology depends on the structure of the reasoning state.

2026-08-06

Efficient Reasoning @ Conference on Language Modeling (publié)

Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs

Arjun Ashok

Andrew R. Williams

Vincent Zhihao Zheng

Nicolas Chapados

Étienne Marcotte

Valentina Zantedeschi

Alexandre Drouin

Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual fo… (voir plus)rm. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at https://github.com/ashok-arjun/beyond-naive-prompting.

2026-07-27

Transactions on Machine Learning Research (accepté)

Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

Parviz Haggi-Mani

Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the t… (voir plus)rained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point shift formula and obtain four testable predictions for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum. Testing these on synthetic Markov chain sequences with controlled correlation length, we find: (1) For large chains(long correlation), attention is strongly relevant: it closes a residual loss gap the MLP cannot bridge and drives a phase transition in representation space, with effective rank jumping above input dimensionality at layer 1 and stabilizing at a high-dimensional plateau. (2) For short chains(short correlation), attention is irrelevant: the Transformer converges to the same loss and fixed-point geometry as the MLP, though it contracts perturbations faster. (3) The transition is dominated by the first-layer head (L0H0), which accounts for more than 4 times the representational shift of any subsequent head, consistent with the prediction that the relevant operator acts before the MLP begins integrating out positional variation. (4) Perturbation decay experiments reveal a regime reversal: in the long correlation regime the Transformer selectively preserves slow Markov modes (5.4 times the dynamic range in decay length vs. 1.3 times for the MLP); in the short correlation regime it suppresses all modes faster than the MLP, with no spectral selectivity. Together, these results show that the relevance of attention is not a property of the architecture but of the spectral structure of the data-generating process, and that a first-order RG perturbation framework provides a predictive account of that difference.

2026-07-15

arXiv (prépublication)

When Geometry Aligns: Dihedral Hidden-State Transformations in UNet, ViT, and DiT Architectures

Mojtaba Faramarzi

Alex Lamb

Diffusion architectures now encompass convolutional UNets as well as transformer-based designs such as Diffusion Transformers (DiTs), inspir… (voir plus)ed by Vision Transformers (ViTs), yet the effects of structured geometric perturbations within these architectures remain poorly understood. We study this question through a unified framework that applies reflection-based elements of the dihedral group to intermediate hidden states as controlled internal interventions, contrasting geometrically consistent and inconsistent variants. Using activation-level diagnostics, including Self-Consistency Shift (SCS), Activation Mass Scatter (AMS), and Drift, we analyze feature stability and geometric drift. We find that consistent transformations improve stability, while inconsistent ones induce predictable, architecture-specific failures. In the main Stable Diffusion 2.1 U-Net study, we evaluate seven intervention modes over three seeds and complement the internal diagnostics with image-level FID, KID, CLIP score, and LPIPS diversity. Taken together with supporting ViT and controlled DiT analyses, these results establish geometric consistency as a key principle for stable hidden-state interventions in spatially structured vision and diffusion models.

2026-07-02

arXiv (prépublication)

Formal Epistemic Structure Leaves Geometric Traces in LLMs: Evidence from S5 Multi-Agent Logic and RoBERTa

David John Lemay

Emil Sayilov

We ask whether the formal structure of S5 multi-agent epistemic logic leaves recoverable geometric traces in a fine-tuned language model. Va… (voir plus)n Benthem's product topology for S5 predicts that the state space of

2026-06-26

LogiSymb @ International Joint Conference on Artificial Intelligence (poster)

Feature Geometry of Language Models Transfer Across Modalities to Time Series

Zhenghan Tai

Vasilii Feofanov

Language models transfer to time-series forecasting, but it is unclear whether this reflects reusable internal structure or rapid relearning… (voir plus) under a familiar architecture. We study this transfer directly by comparing pretrained and randomly initialized versions of the same model on a forecasting objective whose inputs have little semantic overlap with text but still require autoregressive sequential structure. Across Qwen3-0.6B finetuning experiments, language initialization gives coherent per-example gradients from the first update, while random initialization first passes through a low-alignment warmup phase. Effective-rank and hidden-state analyses show that finetuning selectively reshapes an existing representation geometry rather than constructing the simpler temporal geometry found by models trained from scratch. Cross-domain sparse features and causal ablations then expose candidate transferred primitives, including a Layer~1 head--MLP circuit whose ablation selectively increases loss on periodic forecasting and repetitive language passages. These results support an account of cross-modal transfer in which autoregressive pretraining creates temporal feature geometry that can be selected and specialized outside language.

2026-06-10

ICML.cc/2026/Workshop/Mech_Interp (poster)

Forecasting Emerges from Auto-Regressive Pretraining: Latent Predictive Structure in Language Models

Zhenghan Tai

Vasilii Feofanov

Predicting how a sequence will continue is a basic problem for intelligent systems. We show that large language models contain usable foreca… (voir plus)sting structure before any explicit time-series supervision. A single linear readout from frozen Qwen3-0.6B hidden states maps ordinary text sequences to numerical trajectories that resemble real time series, and those trajectories can be used for straightforward forecasts. The distribution over output tokens also gives coherent, non-crossing probabilistic forecasts in a single forward pass. After time-series specialization, pretrained models show aligned gradients and improve immediately, whereas randomly initialized models spend early training in a destructive-interference regime. These findings suggest that auto-regressive pretraining already shapes representations around temporal continuation; and finetuning adapts that structure to numerical forecasting rather than creating it from scratch.

2026-06-10

ICML.cc/2026/Workshop/Forecast (présentation orale)

Representing Time Series as Structured Programs for LLM Reasoning

Jaeho Kim

Changhun Oh

Seokhyun Lee

Changhee Lee

Large language models (LLMs) have demonstrated strong reasoning and instruction-following capabilities, making them potentially powerful too… (voir plus)ls for time-series analysis. However, time series lie outside their native textual modality, raising a fundamental question: how should time series be represented so that LLMs can reason about them effectively? Existing work typically serializes raw numerical sequences or fine-tunes pre-trained LLMs on time-series data. These approaches place the burden of extracting temporal structure directly on the LLM, creating a modality mismatch that often degrades performance on long sequences and introduces substantial computational overhead. In this work, we introduce Time-Series-to-Structured-Program representation (T2SP), a deterministic, training-free method that represents a time series as a structured symbolic program. T2SP decomposes time series into trends, periods, and salient events, expressing them in a program-friendly format aligned with the textual and code-like modalities on which LLMs are natively trained. By shifting temporal-structure extraction from the model to the representation itself, T2SP enables off-the-shelf LLMs to leverage their existing reasoning capabilities for time-series understanding. We evaluate T2SP on three reasoning tasks -- editing, captioning, and question answering -- where it consistently improves performance, reduces reasoning time, and lowers failure rates compared with raw-string representations. Our results demonstrate that T2SP provides an effective interface between time series and LLMs.

2026-06-09

arXiv (prépublication)

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Parviz Haggi-Mani

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but… (voir plus) existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified. We study the simplest architecture for which the analogy is tractable: a pure MLP residual stack trained on masked token prediction over synthetic Markov chain sequences with known spectral properties. We report three findings. (i) The effective rank of the residual stream decreases monotonically with depth after training, consistent with progressive integration of irrelevant degrees of freedom. (ii) This rank collapse is selective: it occurs for chains with short correlation length approximately 1 but is absent for chains with long correlation length approximately 7, measured at the position level to control for mean-pooling artifacts. The network preserves exactly the degrees of freedom relevant to the prediction task, the content of the RG relevance criterion. (iii) Inter-layer kernel drift is concentrated at one or two specific transitions, with the remainder of the network near a fixed point, consistent with a discrete fixed-point plateau. Together these findings constitute the first quantitative, position-level evidence that MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

2026-06-08

arXiv (prépublication)

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional a… (voir plus)ttempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods (

2026-06-02

arXiv (prépublication)