Irina Rish

Biography

Irina Rish is a full professor at the Université de Montréal (UdeM), where she leads the Autonomous AI Lab, and a core academic member of Mila – Quebec Artificial Intelligence Institute.

In addition to holding a Canada Excellence Research Chair (CERC) and a CIFAR Chair, she leads the U.S. Department of Energy’s INCITE project on Scalable Foundation Models on Summit & Frontier supercomputers at the Oak Ridge Leadership Computing Facility. She co-founded and serves as CSO of Nolano.ai.

Rish’s current research interests include neural scaling laws and emergent behaviors (capabilities and alignment) in foundation models, as well as continual learning, out-of-distribution generalization and robustness.

Before joining UdeM in 2019, she was a research scientist at the IBM T.J. Watson Research Center, where she worked on various projects at the intersection of neuroscience and AI, and led the Neuro-AI challenge. She was awarded the IBM Eminence & Excellence Award and IBM Outstanding Innovation Award (2018), IBM Outstanding Technical Achievement Award (2017) and IBM Research Accomplishment Award (2009).

She holds 64 patents and has published 120 research papers, several book chapters, three edited books and a monograph on sparse modeling.

Current Students

Ivan Anokhin

PhD - Université de Montréal

Co-supervisor :

Samira Ebrahimi Kahou

ivan.anokhin@mila.quebec

rishika.bhagwatkar@mila.quebec

Rifat Arefin

PhD - Université de Montréal

Arjun Ashok

PhD - Université de Montréal

Co-supervisor :

Alexandre Drouin

arjun.ashok@mila.quebec

Master's Research - Université de Montréal

PhD - Université de Montréal

ethan.caballero@mila.quebec

Pietro Cagnasso

PhD - Concordia University

Principal supervisor :

pietro.cagnasso@mila.quebec

Mohammad Javad Darvishi Bayazi

Amin Darabi

PhD - Université de Montréal

amin.darabi@mila.quebec

Collaborating researcher - Université de Montréal

mj.darvishi92@gmail.com

Wagner Drew

Master's Research - Concordia University

Principal supervisor :

Mirco Ravanelli

drew.wagner@mila.quebec

kshitij.gupta@mila.quebec

Mojtaba Faramarzi

PhD - Université de Montréal

faramarm@mila.quebec

Kshitij Gupta

Collaborating researcher

Co-supervisor :

Sarath Chandar

Parviz Haggi Mani

Independent visiting researcher - -

parviz.haggi@gmail.com

Nadhir Hassen

Collaborating researcher - Université de Montréal

nadhir.hassen@mila.quebec

Niklas Herbster

Research Intern - Université de Montréal

Co-supervisor :

Gauthier Gidel

niklas.herbster@gmail.com

prateek.humane@mila.quebec

Prateek Humane

PhD

arnav-kumar.jain@mila.quebec

Nizar Islah

PhD - Université de Montréal

Principal supervisor :

Eilif B. Muller

nizar.islah@mila.quebec

PhD - Université de Montréal

PhD - Université de Montréal

ayush.kaushal@mila.quebec

Arian Khorasani

Collaborating researcher - Université de Montréal

arian.khorasani@mila.quebec

Neeraj Kumar

Collaborating Alumni - Université de Montréal

neeraj.kumar@mila.quebec

Gwen Legate

PhD - Concordia University

Principal supervisor :

gwendolyne.legate@mila.quebec

David Lemay

Master's Research - Université de Montréal

david.lemay@mila.quebec

amin.mansouri@mila.quebec

Amin Mansouri

Collaborating Alumni - Université de Montréal

Collaborating researcher

memariaa@mila.quebec

Andrei Mircea

PhD - Université de Montréal

Collaborating researcher - Université de Montréal

diganta.misra@mila.quebec

motahareh.pourrahimi@mila.quebec

Timothy Nest

PhD - Université de Montréal

Co-supervisor :

Eilif B. Muller

timothy.nest@mila.quebec

Jonathan Pilault

Collaborating researcher - Polytechnique Montréal

PhD - McGill University

Principal supervisor :

Pouya Bashivan

Mahta Ramezanian

Master's Research - Université de Montréal

Co-supervisor :

Guillaume Dumas

mahta.ramezanian@mila.quebec

Matthew Riemer

PhD - Université de Montréal

matthew.riemer@mila.quebec

Alexis Roger

PhD - McGill University

Principal supervisor :

Blake Richards

alexis.roger@mila.quebec

munish.sathish-kumar@mila.quebec

Munish Sathish Kumar

Collaborating researcher

Zibo Shang

PhD - Université de Montréal

zibo.shang@mila.quebec

Vaibhav Singh

PhD - Concordia University

Principal supervisor :

vaibhav.singh@mila.quebec

Gopeshh Subbaraj

PhD - Université de Montréal

gopeshh.subbaraj@mila.quebec

PhD - Université de Montréal

Co-supervisor :

benjamin.therien@mila.quebec

Collaborating researcher - Université de Montréal

tejas.vaidhya@mila.quebec

andrew.williams@mila.quebec

Sihui Wei

Undergraduate - McGill University

sihui.wei@mila.quebec

Andrew Williams

PhD - Université de Montréal

Co-supervisor :

Nicolas Chapados

daria.yasafova@mila.quebec

Daria Yasafova

Master's Research - Université de Montréal

He Zhu

PhD - McGill University

he.zhu@mila.quebec

Publications

What Survives KV-Cache Compression for Reasoning? A Dual-Channel View of Structural and Associative Memory

Md Rifat Arefin

Long reasoning traces make Transformer key–value (KV) caches grow linearly with generation length. Existing compression methods adopt diff… (see more)erent memory topologies, but it is unclear how those choices affect the information that survives eviction. We observe a task–fidelity mismatch: dense summaries achieve low KV reconstruction error yet lose information required for structured reasoning. We explain this through a dual-channel model of compressed memory consisting of a temporally organized scaffold and sparse associative bindings, suggesting that structure and association should be stored separately. We instantiate this idea as TT–Delta, which combines a Tensor-Train structural state with a Delta-rule associative state. On a frozen Qwen2.5-0.5B task model, TT–Delta compresses an evicted prefix by 25.3× (46,680 persistent scalars) and improves exact-answer generation on sequential linear-system reasoning from 0.302±0.113 to 0.839±0.171. In contrast, multi-head Delta is strongest on associative recall and branching graph search. These results suggest that no single compressed-memory topology is universally optimal; the appropriate topology depends on the structure of the reasoning state.

2026-08-06

Efficient Reasoning @ Conference on Language Modeling (published)

Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs

Arjun Ashok

Andrew Robert Williams

Vincent Zhihao Zheng

Nicolas Chapados

Étienne Marcotte

Valentina Zantedeschi

Alexandre Drouin

Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual fo… (see more)rm. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at https://github.com/ashok-arjun/beyond-naive-prompting.

2026-07-27

Transactions on Machine Learning Research (accepted)

Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

Parviz Haggi-Mani

Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the t… (see more)rained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point shift formula and obtain four testable predictions for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum. Testing these on synthetic Markov chain sequences with controlled correlation length, we find: (1) For large chains(long correlation), attention is strongly relevant: it closes a residual loss gap the MLP cannot bridge and drives a phase transition in representation space, with effective rank jumping above input dimensionality at layer 1 and stabilizing at a high-dimensional plateau. (2) For short chains(short correlation), attention is irrelevant: the Transformer converges to the same loss and fixed-point geometry as the MLP, though it contracts perturbations faster. (3) The transition is dominated by the first-layer head (L0H0), which accounts for more than 4 times the representational shift of any subsequent head, consistent with the prediction that the relevant operator acts before the MLP begins integrating out positional variation. (4) Perturbation decay experiments reveal a regime reversal: in the long correlation regime the Transformer selectively preserves slow Markov modes (5.4 times the dynamic range in decay length vs. 1.3 times for the MLP); in the short correlation regime it suppresses all modes faster than the MLP, with no spectral selectivity. Together, these results show that the relevance of attention is not a property of the architecture but of the spectral structure of the data-generating process, and that a first-order RG perturbation framework provides a predictive account of that difference.

2026-07-15

arXiv (preprint)

When Geometry Aligns: Dihedral Hidden-State Transformations in UNet, ViT, and DiT Architectures

Mojtaba Faramarzi

Alex Lamb

Diffusion architectures now encompass convolutional UNets as well as transformer-based designs such as Diffusion Transformers (DiTs), inspir… (see more)ed by Vision Transformers (ViTs), yet the effects of structured geometric perturbations within these architectures remain poorly understood. We study this question through a unified framework that applies reflection-based elements of the dihedral group to intermediate hidden states as controlled internal interventions, contrasting geometrically consistent and inconsistent variants. Using activation-level diagnostics, including Self-Consistency Shift (SCS), Activation Mass Scatter (AMS), and Drift, we analyze feature stability and geometric drift. We find that consistent transformations improve stability, while inconsistent ones induce predictable, architecture-specific failures. In the main Stable Diffusion 2.1 U-Net study, we evaluate seven intervention modes over three seeds and complement the internal diagnostics with image-level FID, KID, CLIP score, and LPIPS diversity. Taken together with supporting ViT and controlled DiT analyses, these results establish geometric consistency as a key principle for stable hidden-state interventions in spatially structured vision and diffusion models.

2026-07-02

arXiv (preprint)

Formal Epistemic Structure Leaves Geometric Traces in LLMs: Evidence from S5 Multi-Agent Logic and RoBERTa

David John Lemay

Emil Sayilov

We ask whether the formal structure of S5 multi-agent epistemic logic leaves recoverable geometric traces in a fine-tuned language model. Va… (see more)n Benthem's product topology for S5 predicts that the state space of

2026-06-26

LogiSymb @ International Joint Conference on Artificial Intelligence (poster)

Feature Geometry of Language Models Transfer Across Modalities to Time Series

Zhenghan Tai

Vasilii Feofanov

Language models transfer to time-series forecasting, but it is unclear whether this reflects reusable internal structure or rapid relearning… (see more) under a familiar architecture. We study this transfer directly by comparing pretrained and randomly initialized versions of the same model on a forecasting objective whose inputs have little semantic overlap with text but still require autoregressive sequential structure. Across Qwen3-0.6B finetuning experiments, language initialization gives coherent per-example gradients from the first update, while random initialization first passes through a low-alignment warmup phase. Effective-rank and hidden-state analyses show that finetuning selectively reshapes an existing representation geometry rather than constructing the simpler temporal geometry found by models trained from scratch. Cross-domain sparse features and causal ablations then expose candidate transferred primitives, including a Layer~1 head--MLP circuit whose ablation selectively increases loss on periodic forecasting and repetitive language passages. These results support an account of cross-modal transfer in which autoregressive pretraining creates temporal feature geometry that can be selected and specialized outside language.

2026-06-10

ICML.cc/2026/Workshop/Mech_Interp (poster)

Forecasting Emerges from Auto-Regressive Pretraining: Latent Predictive Structure in Language Models

Zhenghan Tai

Vasilii Feofanov

Predicting how a sequence will continue is a basic problem for intelligent systems. We show that large language models contain usable foreca… (see more)sting structure before any explicit time-series supervision. A single linear readout from frozen Qwen3-0.6B hidden states maps ordinary text sequences to numerical trajectories that resemble real time series, and those trajectories can be used for straightforward forecasts. The distribution over output tokens also gives coherent, non-crossing probabilistic forecasts in a single forward pass. After time-series specialization, pretrained models show aligned gradients and improve immediately, whereas randomly initialized models spend early training in a destructive-interference regime. These findings suggest that auto-regressive pretraining already shapes representations around temporal continuation; and finetuning adapts that structure to numerical forecasting rather than creating it from scratch.

2026-06-10

ICML.cc/2026/Workshop/Forecast (oral)

Representing Time Series as Structured Programs for LLM Reasoning

Jaeho Kim

Changhun Oh

Seokhyun Lee

Changhee Lee

Large language models (LLMs) have demonstrated strong reasoning and instruction-following capabilities, making them potentially powerful too… (see more)ls for time-series analysis. However, time series lie outside their native textual modality, raising a fundamental question: how should time series be represented so that LLMs can reason about them effectively? Existing work typically serializes raw numerical sequences or fine-tunes pre-trained LLMs on time-series data. These approaches place the burden of extracting temporal structure directly on the LLM, creating a modality mismatch that often degrades performance on long sequences and introduces substantial computational overhead. In this work, we introduce Time-Series-to-Structured-Program representation (T2SP), a deterministic, training-free method that represents a time series as a structured symbolic program. T2SP decomposes time series into trends, periods, and salient events, expressing them in a program-friendly format aligned with the textual and code-like modalities on which LLMs are natively trained. By shifting temporal-structure extraction from the model to the representation itself, T2SP enables off-the-shelf LLMs to leverage their existing reasoning capabilities for time-series understanding. We evaluate T2SP on three reasoning tasks -- editing, captioning, and question answering -- where it consistently improves performance, reduces reasoning time, and lowers failure rates compared with raw-string representations. Our results demonstrate that T2SP provides an effective interface between time series and LLMs.

2026-06-09

arXiv (preprint)

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Parviz Haggi-Mani

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but… (see more) existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified. We study the simplest architecture for which the analogy is tractable: a pure MLP residual stack trained on masked token prediction over synthetic Markov chain sequences with known spectral properties. We report three findings. (i) The effective rank of the residual stream decreases monotonically with depth after training, consistent with progressive integration of irrelevant degrees of freedom. (ii) This rank collapse is selective: it occurs for chains with short correlation length approximately 1 but is absent for chains with long correlation length approximately 7, measured at the position level to control for mean-pooling artifacts. The network preserves exactly the degrees of freedom relevant to the prediction task, the content of the RG relevance criterion. (iii) Inter-layer kernel drift is concentrated at one or two specific transitions, with the remainder of the network near a fixed point, consistent with a discrete fixed-point plateau. Together these findings constitute the first quantitative, position-level evidence that MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

2026-06-08

arXiv (preprint)

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional a… (see more)ttempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods (

2026-06-02

arXiv (preprint)

Reusable Low-Rank Subspaces Explain Why Cross-Modal Transfer Adapts with Tiny Updates

Zhenghan Tai

Vasilii Feofanov

Parameter-efficient finetuning methods such as LoRA routinely adapt massive pretrained transformers to new tasks using only tiny low-rank up… (see more)dates, but the representational geometry that makes this possible remains unclear. We use cross-modal transfer from a language-pretrained transformer to time-series forecasting as a controlled probe of low-rank adaptation, asking why so few directions are sufficient. Across adaptation regimes, LoRA recovers most of the transfer benefit of full finetuning; effective-rank analyses show that pretrained representations already concentrate on a low-rank subspace that finetuning \emph{redistributes} rather than rebuilds; and a single linear projection over frozen hidden states aligns with realistic time-series trajectories without paired supervision. Randomly initialized models, by contrast, first construct a compressed representation through a uniform layer-wise collapse before they can specialize. These results support a view of cross-modal adaptation as low-rank \emph{direction selection} within reusable pretrained subspaces.

2026-05-28

CoLoRAI @ International Conference on Machine Learning (poster)

Language Pretraining Gives Structured Forecasters a Sequential Prior

Zhenghan Tai

Vasilii Feofanov

Structured-data foundation models now target transfer across time series, tabular data, text, and other sequential domains, but it remains u… (see more)nclear what language pretraining contributes to forecasting beyond a generic transformer architecture. We study this question by adapting Qwen3-0.6B to probabilistic time-series forecasting on GiftEval and comparing language-pretrained models with identical randomly initialized models under full and parameter-efficient finetuning. Language pretraining gives a large early optimization advantage, especially in limited-adaptation regimes: LoRA attention updates recover most of the effective transfer benefit of full finetuning. Frozen-state probes, retrieval forecasts, gradient coherence, and effective-rank dynamics indicate that language models already contain reusable sequential structure before time-series supervision. These results frame language-to-time-series transfer as efficient low-rank specialization of a pretrained sequential inductive bias, with implications for future structured-data foundation models.

2026-05-24

FMSD @ International Conference on Machine Learning (poster)