Irina Rish

rishika.bhagwatkar@mila.quebec

Biographie

Irina Rish est professeure titulaire à l'Université de Montréal (UdeM), où elle dirige le Laboratoire d'IA autonome. Membre du corps professoral de Mila – Institut québécois d’intelligence artificielle, elle est titulaire d'une chaire d'excellence en recherche du Canada (CERC) et d'une chaire en IA Canada-CIFAR. Irina dirige le projet INCITE du ministère américain de l'Environnement au sujet des modèles de fondation évolutifs sur les superordinateurs Summit et Frontier à l'Oak Ridge Leadership Computing Facility (OLCF). Elle est cofondatrice et directrice scientifique de Nolano.ai.

Ses recherches actuelles portent sur les lois de mise à l'échelle neuronale et les comportements émergents (capacités et alignement) dans les modèles de fondation, ainsi que sur l'apprentissage continu, la généralisation hors distribution et la robustesse. Avant de se joindre à l'UdeM en 2019, Irina était chercheuse au Centre de recherche IBM Thomas J. Watson, où elle a travaillé sur divers projets à l'intersection des neurosciences et de l'IA, et dirigé le défi NeuroAI. Elle a reçu plusieurs prix IBM : ceux de l’excellence et de l’innovation exceptionnelle (2018), celui de la réalisation technique exceptionnelle (2017), et celui de l’accomplissement en recherche (2009). Elle détient 64 brevets et a écrit plus de 120 articles de recherche, plusieurs chapitres de livres, trois livres publiés et une monographie sur la modélisation éparse.

Étudiants actuels

Ivan Anokhin

Doctorat - UdeM

Co-superviseur⋅e :

Samira Ebrahimi Kahou

ivan.anokhin@mila.quebec

Doctorat - UdeM

Arjun Ashok

Doctorat - UdeM

Co-superviseur⋅e :

Alexandre Drouin

arjun.ashok@mila.quebec

Maîtrise recherche - UdeM

Doctorat - UdeM

ethan.caballero@mila.quebec

Pietro Cagnasso

Doctorat - Concordia

Superviseur⋅e principal⋅e :

pietro.cagnasso@mila.quebec

Mohammad Javad Darvishi Bayazi

Amin Darabi

Doctorat - UdeM

amin.darabi@mila.quebec

Collaborateur·rice de recherche - UdeM

mj.darvishi92@gmail.com

kshitij.gupta@mila.quebec

Wagner Drew

Maîtrise recherche - Concordia

Superviseur⋅e principal⋅e :

Mirco Ravanelli

drew.wagner@mila.quebec

Doctorat - UdeM

Collaborateur·rice de recherche

Co-superviseur⋅e :

Sarath Chandar

Parviz Haggi Mani

Visiteur de recherche indépendant - -

parviz.haggi@gmail.com

Nadhir Hassen

Collaborateur·rice de recherche - UdeM

nadhir.hassen@mila.quebec

Niklas Herbster

Stagiaire de recherche - UdeM

Co-superviseur⋅e :

Gauthier Gidel

niklas.herbster@gmail.com

prateek.humane@mila.quebec

Prateek Humane

Doctorat

arnav-kumar.jain@mila.quebec

Nizar Islah

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Eilif B. Muller

nizar.islah@mila.quebec

Doctorat - UdeM

Doctorat - UdeM

ayush.kaushal@mila.quebec

Arian Khorasani

Collaborateur·rice de recherche - UdeM

arian.khorasani@mila.quebec

Neeraj Kumar

Collaborateur·rice alumni - UdeM

neeraj.kumar@mila.quebec

Gwen Legate

Doctorat - Concordia

Superviseur⋅e principal⋅e :

gwendolyne.legate@mila.quebec

David Lemay

Maîtrise recherche - UdeM

david.lemay@mila.quebec

amin.mansouri@mila.quebec

Amin Mansouri

Collaborateur·rice alumni - UdeM

Collaborateur·rice de recherche

Doctorat - UdeM

Collaborateur·rice de recherche - UdeM

diganta.misra@mila.quebec

motahareh.pourrahimi@mila.quebec

Timothy Nest

Doctorat - UdeM

Co-superviseur⋅e :

Eilif B. Muller

timothy.nest@mila.quebec

Jonathan Pilault

Collaborateur·rice de recherche - Polytechnique

Doctorat - McGill

Superviseur⋅e principal⋅e :

Pouya Bashivan

Mahta Ramezanian

Maîtrise recherche - UdeM

Co-superviseur⋅e :

Guillaume Dumas

mahta.ramezanian@mila.quebec

Matthew Riemer

Doctorat - UdeM

matthew.riemer@mila.quebec

Alexis Roger

Doctorat - McGill

Superviseur⋅e principal⋅e :

Blake Richards

alexis.roger@mila.quebec

munish.sathish-kumar@mila.quebec

Munish Sathish Kumar

Collaborateur·rice de recherche

Zibo Shang

Doctorat - UdeM

zibo.shang@mila.quebec

Vaibhav Singh

Doctorat - Concordia

Superviseur⋅e principal⋅e :

vaibhav.singh@mila.quebec

Gopeshh Subbaraj

Doctorat - UdeM

gopeshh.subbaraj@mila.quebec

Doctorat - UdeM

Co-superviseur⋅e :

benjamin.therien@mila.quebec

Collaborateur·rice de recherche - UdeM

tejas.vaidhya@mila.quebec

andrew.williams@mila.quebec

Sihui Wei

Baccalauréat - McGill

sihui.wei@mila.quebec

Andrew Williams

Doctorat - UdeM

Co-superviseur⋅e :

Nicolas Chapados

daria.yasafova@mila.quebec

Daria Yasafova

Maîtrise recherche - UdeM

He Zhu

Doctorat - McGill

he.zhu@mila.quebec

Publications

What Survives KV-Cache Compression for Reasoning? A Dual-Channel View of Structural and Associative Memory

Md Rifat Arefin

Long reasoning traces make Transformer key–value (KV) caches grow linearly with generation length. Existing compression methods adopt diff… (voir plus)erent memory topologies, but it is unclear how those choices affect the information that survives eviction. We observe a task–fidelity mismatch: dense summaries achieve low KV reconstruction error yet lose information required for structured reasoning. We explain this through a dual-channel model of compressed memory consisting of a temporally organized scaffold and sparse associative bindings, suggesting that structure and association should be stored separately. We instantiate this idea as TT–Delta, which combines a Tensor-Train structural state with a Delta-rule associative state. On a frozen Qwen2.5-0.5B task model, TT–Delta compresses an evicted prefix by 25.3× (46,680 persistent scalars) and improves exact-answer generation on sequential linear-system reasoning from 0.302±0.113 to 0.839±0.171. In contrast, multi-head Delta is strongest on associative recall and branching graph search. These results suggest that no single compressed-memory topology is universally optimal; the appropriate topology depends on the structure of the reasoning state.

2026-08-06

Efficient Reasoning @ Conference on Language Modeling (publié)

Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs

Arjun Ashok

Andrew Robert Williams

Vincent Zhihao Zheng

Nicolas Chapados

Étienne Marcotte

Valentina Zantedeschi

Alexandre Drouin

Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual fo… (voir plus)rm. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at https://github.com/ashok-arjun/beyond-naive-prompting.

2026-07-27

Transactions on Machine Learning Research (accepté)

Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

Parviz Haggi-Mani

Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the t… (voir plus)rained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point shift formula and obtain four testable predictions for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum. Testing these on synthetic Markov chain sequences with controlled correlation length, we find: (1) For large chains(long correlation), attention is strongly relevant: it closes a residual loss gap the MLP cannot bridge and drives a phase transition in representation space, with effective rank jumping above input dimensionality at layer 1 and stabilizing at a high-dimensional plateau. (2) For short chains(short correlation), attention is irrelevant: the Transformer converges to the same loss and fixed-point geometry as the MLP, though it contracts perturbations faster. (3) The transition is dominated by the first-layer head (L0H0), which accounts for more than 4 times the representational shift of any subsequent head, consistent with the prediction that the relevant operator acts before the MLP begins integrating out positional variation. (4) Perturbation decay experiments reveal a regime reversal: in the long correlation regime the Transformer selectively preserves slow Markov modes (5.4 times the dynamic range in decay length vs. 1.3 times for the MLP); in the short correlation regime it suppresses all modes faster than the MLP, with no spectral selectivity. Together, these results show that the relevance of attention is not a property of the architecture but of the spectral structure of the data-generating process, and that a first-order RG perturbation framework provides a predictive account of that difference.

2026-07-15

arXiv (prépublication)

When Geometry Aligns: Dihedral Hidden-State Transformations in UNet, ViT, and DiT Architectures

Mojtaba Faramarzi

Alex Lamb

Diffusion architectures now encompass convolutional UNets as well as transformer-based designs such as Diffusion Transformers (DiTs), inspir… (voir plus)ed by Vision Transformers (ViTs), yet the effects of structured geometric perturbations within these architectures remain poorly understood. We study this question through a unified framework that applies reflection-based elements of the dihedral group to intermediate hidden states as controlled internal interventions, contrasting geometrically consistent and inconsistent variants. Using activation-level diagnostics, including Self-Consistency Shift (SCS), Activation Mass Scatter (AMS), and Drift, we analyze feature stability and geometric drift. We find that consistent transformations improve stability, while inconsistent ones induce predictable, architecture-specific failures. In the main Stable Diffusion 2.1 U-Net study, we evaluate seven intervention modes over three seeds and complement the internal diagnostics with image-level FID, KID, CLIP score, and LPIPS diversity. Taken together with supporting ViT and controlled DiT analyses, these results establish geometric consistency as a key principle for stable hidden-state interventions in spatially structured vision and diffusion models.

2026-07-02

arXiv (prépublication)

Formal Epistemic Structure Leaves Geometric Traces in LLMs: Evidence from S5 Multi-Agent Logic and RoBERTa

David John Lemay

Emil Sayilov

We ask whether the formal structure of S5 multi-agent epistemic logic leaves recoverable geometric traces in a fine-tuned language model. Va… (voir plus)n Benthem's product topology for S5 predicts that the state space of

2026-06-26

LogiSymb @ International Joint Conference on Artificial Intelligence (poster)

Feature Geometry of Language Models Transfer Across Modalities to Time Series

Zhenghan Tai

Vasilii Feofanov

Language models transfer to time-series forecasting, but it is unclear whether this reflects reusable internal structure or rapid relearning… (voir plus) under a familiar architecture. We study this transfer directly by comparing pretrained and randomly initialized versions of the same model on a forecasting objective whose inputs have little semantic overlap with text but still require autoregressive sequential structure. Across Qwen3-0.6B finetuning experiments, language initialization gives coherent per-example gradients from the first update, while random initialization first passes through a low-alignment warmup phase. Effective-rank and hidden-state analyses show that finetuning selectively reshapes an existing representation geometry rather than constructing the simpler temporal geometry found by models trained from scratch. Cross-domain sparse features and causal ablations then expose candidate transferred primitives, including a Layer~1 head--MLP circuit whose ablation selectively increases loss on periodic forecasting and repetitive language passages. These results support an account of cross-modal transfer in which autoregressive pretraining creates temporal feature geometry that can be selected and specialized outside language.

2026-06-10

ICML.cc/2026/Workshop/Mech_Interp (poster)

Forecasting Emerges from Auto-Regressive Pretraining: Latent Predictive Structure in Language Models

Zhenghan Tai

Vasilii Feofanov

Predicting how a sequence will continue is a basic problem for intelligent systems. We show that large language models contain usable foreca… (voir plus)sting structure before any explicit time-series supervision. A single linear readout from frozen Qwen3-0.6B hidden states maps ordinary text sequences to numerical trajectories that resemble real time series, and those trajectories can be used for straightforward forecasts. The distribution over output tokens also gives coherent, non-crossing probabilistic forecasts in a single forward pass. After time-series specialization, pretrained models show aligned gradients and improve immediately, whereas randomly initialized models spend early training in a destructive-interference regime. These findings suggest that auto-regressive pretraining already shapes representations around temporal continuation; and finetuning adapts that structure to numerical forecasting rather than creating it from scratch.

2026-06-10

ICML.cc/2026/Workshop/Forecast (présentation orale)

Representing Time Series as Structured Programs for LLM Reasoning

Jaeho Kim

Changhun Oh

Seokhyun Lee

Changhee Lee

Large language models (LLMs) have demonstrated strong reasoning and instruction-following capabilities, making them potentially powerful too… (voir plus)ls for time-series analysis. However, time series lie outside their native textual modality, raising a fundamental question: how should time series be represented so that LLMs can reason about them effectively? Existing work typically serializes raw numerical sequences or fine-tunes pre-trained LLMs on time-series data. These approaches place the burden of extracting temporal structure directly on the LLM, creating a modality mismatch that often degrades performance on long sequences and introduces substantial computational overhead. In this work, we introduce Time-Series-to-Structured-Program representation (T2SP), a deterministic, training-free method that represents a time series as a structured symbolic program. T2SP decomposes time series into trends, periods, and salient events, expressing them in a program-friendly format aligned with the textual and code-like modalities on which LLMs are natively trained. By shifting temporal-structure extraction from the model to the representation itself, T2SP enables off-the-shelf LLMs to leverage their existing reasoning capabilities for time-series understanding. We evaluate T2SP on three reasoning tasks -- editing, captioning, and question answering -- where it consistently improves performance, reduces reasoning time, and lowers failure rates compared with raw-string representations. Our results demonstrate that T2SP provides an effective interface between time series and LLMs.

2026-06-09

arXiv (prépublication)

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Parviz Haggi-Mani

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but… (voir plus) existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified. We study the simplest architecture for which the analogy is tractable: a pure MLP residual stack trained on masked token prediction over synthetic Markov chain sequences with known spectral properties. We report three findings. (i) The effective rank of the residual stream decreases monotonically with depth after training, consistent with progressive integration of irrelevant degrees of freedom. (ii) This rank collapse is selective: it occurs for chains with short correlation length approximately 1 but is absent for chains with long correlation length approximately 7, measured at the position level to control for mean-pooling artifacts. The network preserves exactly the degrees of freedom relevant to the prediction task, the content of the RG relevance criterion. (iii) Inter-layer kernel drift is concentrated at one or two specific transitions, with the remainder of the network near a fixed point, consistent with a discrete fixed-point plateau. Together these findings constitute the first quantitative, position-level evidence that MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

2026-06-08

arXiv (prépublication)

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional a… (voir plus)ttempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods (

2026-06-02

arXiv (prépublication)

Reusable Low-Rank Subspaces Explain Why Cross-Modal Transfer Adapts with Tiny Updates

Zhenghan Tai

Vasilii Feofanov

Parameter-efficient finetuning methods such as LoRA routinely adapt massive pretrained transformers to new tasks using only tiny low-rank up… (voir plus)dates, but the representational geometry that makes this possible remains unclear. We use cross-modal transfer from a language-pretrained transformer to time-series forecasting as a controlled probe of low-rank adaptation, asking why so few directions are sufficient. Across adaptation regimes, LoRA recovers most of the transfer benefit of full finetuning; effective-rank analyses show that pretrained representations already concentrate on a low-rank subspace that finetuning \emph{redistributes} rather than rebuilds; and a single linear projection over frozen hidden states aligns with realistic time-series trajectories without paired supervision. Randomly initialized models, by contrast, first construct a compressed representation through a uniform layer-wise collapse before they can specialize. These results support a view of cross-modal adaptation as low-rank \emph{direction selection} within reusable pretrained subspaces.

2026-05-28

CoLoRAI @ International Conference on Machine Learning (poster)

Language Pretraining Gives Structured Forecasters a Sequential Prior

Zhenghan Tai

Vasilii Feofanov

Structured-data foundation models now target transfer across time series, tabular data, text, and other sequential domains, but it remains u… (voir plus)nclear what language pretraining contributes to forecasting beyond a generic transformer architecture. We study this question by adapting Qwen3-0.6B to probabilistic time-series forecasting on GiftEval and comparing language-pretrained models with identical randomly initialized models under full and parameter-efficient finetuning. Language pretraining gives a large early optimization advantage, especially in limited-adaptation regimes: LoRA attention updates recover most of the effective transfer benefit of full finetuning. Frozen-state probes, retrieval forecasts, gradient coherence, and effective-rank dynamics indicate that language models already contain reusable sequential structure before time-series supervision. These results frame language-to-time-series transfer as efficient low-rank specialization of a pretrained sequential inductive bias, with implications for future structured-data foundation models.

2026-05-24

FMSD @ International Conference on Machine Learning (poster)