Guillaume Rabusseau

Biographie

Depuis septembre 2018, je suis professeur adjoint à Mila – Institut québécois d’intelligence artificielle et au Département d'informatique et de recherche opérationnelle (DIRO) de l'Université de Montréal (UdeM). Je suis titulaire d’une chaire de recherche en IA Canada-CIFAR depuis mars 2019. Avant de me joindre à l’UdeM, j’ai été chercheur postdoctoral au laboratoire de raisonnement et d'apprentissage de l'Université McGill, où j'ai travaillé avec Prakash Panangaden, Joelle Pineau et Doina Precup.

J'ai obtenu mon doctorat en 2016 à l’Université d’Aix-Marseille (AMU), où j'ai travaillé dans l'équipe Qarma (apprentissage automatique et multimédia), sous la supervision de François Denis et Hachem Kadri. Auparavant, j'ai obtenu une maîtrise en informatique fondamentale de l'AMU et une licence en informatique de la même université en formation à distance.

Je m'intéresse aux méthodes de tenseurs pour l'apprentissage automatique et à la conception d'algorithmes d'apprentissage pour les données structurées par l’utilisation de l'algèbre linéaire et multilinéaire (par exemple, les méthodes spectrales).

Étudiants actuels

Kareem Ahmed

Postdoctorat - McGill

Co-superviseur⋅e :

Siva Reddy

kareem.yousrii@gmail.com

Jun Dai

Postdoctorat - UdeM

jun.dai@mila.quebec

alireza.dizaji@mila.quebec

Alireza Dizaji

Collaborateur·rice alumni - UdeM

marawan.gamal@mila.quebec

Marawan Gamal

Doctorat - UdeM

Stagiaire de recherche - UdeM

emett.haddad@mila.quebec

farzaneh.heidari@mila.quebec

Farzana Heidari

Doctorat - UdeM

Co-superviseur⋅e :

Jian Tang

Collaborateur·rice de recherche - McGill

Superviseur⋅e principal⋅e :

Doctorat - UdeM

Sitao Luan

Postdoctorat - McGill

Co-superviseur⋅e :

Larry Ma

Stagiaire de recherche - Polytechnique

maning.hust@gmail.com

Ewan Murphy

Collaborateur·rice de recherche - INRIA

ewan.murphy@mila.quebec

charlotte.noxon@mila.quebec

Charlotte Noxon

Maîtrise recherche - UdeM

michael.rizvi-martel@mila.quebec

Michael Rizvi-Martel

Doctorat - UdeM

Co-superviseur⋅e :

Adam M. Oberman

Anantha Krishnan Sunilkumar

Doctorat - UdeM

anantha.krishnan.sunilkumar@mila.quebec

pascal.tikeng@mila.quebec

Pascal Tikeng Notsawo

Doctorat - UdeM

Co-superviseur⋅e :

Guillaume Dumas

Collaborateur·rice de recherche - UdeM

Co-superviseur⋅e :

Reihaneh Rabbany

aaron.tjandra@mila.quebec

Beheshteh Toloueirakhshan

Doctorat - UdeM

rakhshab@mila.quebec

jingjing.zheng@mila.quebec

Jingjing Zheng

Stagiaire de recherche - UdeM

Publications

Tensor Network Datamodels for Training Data Attribution

Farzaneh Heidari

Md Rifat Arefin

Louis Jaburi

Datamodels predict how a model's behavior changes when different parts of its training data are included or removed. However, linear datamod… (voir plus)els assign fixed contributions to training units and cannot represent non-additive joint effects. We introduce Tensor Network Datamodels (TNDMs), which use low-rank tensor networks to model dependencies among training units while keeping the representation compact. TNDMs learn a map from subsets of the training data to the outcome of training on those subsets, allowing them to represent interactions among several training units. We prove that complementary and redundant training groups can induce high-degree subset functions with compact low-rank representations. A single fitted TNDM can then be used for several training data questions, including predicting the effect of removing data, measuring the importance of individual training units, identifying interactions between training units, and selecting useful subsets of the training data. We evaluate TNDMs on controlled tasks designed to contain strong interactions, as well as standard fine-tuning settings for classification and reasoning. On a controlled reasoning task, TNDMs outperform every baseline while observing only half as many training subsets, and TNDM-TT reduces prediction NMSE by

2026-10-16

arXiv (prépublication)

Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching

Beheshteh T. Rakhshan

Sahar Rajabi

Maziar Sargordi

Shikai Fang

Sirisha Rambhatla

Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curv… (voir plus)ature, whereas full-curvature methods such as SOAP can accelerate convergence at prohibitive memory costs. We introduce Clean, a memory-efficient and full-curvature optimizer designed to resolve this bottleneck. Clean leverages the randomized Nystrom method to accurately approximate the left and right preconditioners in SOAP, and to reduce the optimizer's memory complexity from quadratic to linear in terms of model dimensions. We subsequently reintegrate the off-subspace components to capture curvature information beyond the low-rank approximation, preserving rich curvature at minimal memory cost. We further propose Q-Clean, a low-precision variant that aggressively compresses optimizer states. Q-Clean reduces optimizer memory consumption by \textbf{over 50\%} compared to Muon when pre-training a LLaMA-1.3B architecture, all while maintaining strong and competitive predictive performance. Notably, Clean operates with a smaller optimizer-state footprint than standard AdamW while reaching AdamW's final performance \textbf{26\% faster} in wall-clock time. Furthermore, our methods uniquely enable the pre-training of a 13B-parameter model on a single 80GB GPU, providing a scalable, efficient, and accessible approach to large-scale model optimization.

2026-10-02

arXiv (prépublication)

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

Farzaneh Heidari

Amin Memarian

Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about t… (voir plus)heir behavior in deployment. This assumption can fail, should models infer that they are being evaluated and condition their response on such context. This hypothesis, termed ``evaluation awareness'', has been observed in frontier and open-weight language models alike. We provide a systematic study of this phenomenon, by probing for it across six language models (from four families and three sizes) and three metrics. More precisely, we examine whether (i) being under evaluation is linearly represented within the models'activations space, (ii) it is verbalized in their output tokens (as scored by an LLM-as-judge), and (iii) steering causally affects their behavior. For the open-checkpoint Olmo models, we further test these measures at every training stage. In doing so, we report that evaluation awareness is linearly decodable from the residual streams of every model (best AUROC

2026-08-21

arXiv (prépublication)

Generative Learning for Quantum Measurement Design

Jun Dai

Olivier Nahman-Lévesque

Hong-Ye Hu

Cunlu Zhou

Extracting quantum information from a quantum state is a fundamental task of quantum computation, often requiring the estimation of many non… (voir plus)-commuting observables under a finite measurement budget. For both near-term and early fault-tolerant settings, the measurement protocol must balance statistical efficiency against implementation resources such as circuit depth, connectivity, and entangling-gate count. Many existing strategies focus on two extremes: hardware-friendly product measurements with high sampling cost, and fully commuting measurements with deep circuits. Here we recast resource-constrained measurement design as a generative learning problem. We introduce FlowMeas, which uses a generative flow network to directly sample finite ensembles of shallow Clifford measurement circuits subject to a prescribed shot budget and hardware constraints. At zero entangling depth, FlowMeas learns qubit-wise commuting measurement schedules and already matches or improves leading product-measurement methods on nearly all molecular benchmarks. Allowing one or two entangling gate layers yields further reductions in energy estimation error of up to

2026-08-10

arXiv (prépublication)

Merging Adapted Models via Data-Free Covariance Estimation

Marawan Gamal

Derek Tam

Pascal Junior Tikeng Notsawo

Colin Raffel

Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While s… (voir plus)ome merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from

2026-08-03

CBW @ Conference on Language Modeling (poster)

From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP

Michael Rizvi-Martel

Satwik Bhattamishra

Michael Hahn

A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs). … (voir plus)There is much work analyzing the expressivity of attention-based models. By proposing handcrafted weights or using computational complexity arguments, a large amount of past theoretical works have sought to characterize which tasks are and which are not in the hypothesis class of Transformer models. However, little work investigates the learnability of such solutions. In this work, we make progress towards this goal. Inspired by recent loss landscape analysis work, we propose preliminary sample complexity bounds for learning C-RASP constructions with Transformers.

2026-07-12

arXiv (prépublication)

The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models

Michael Rizvi-Martel

Marius Mosbach

Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating i… (voir plus)n continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether language models actually leverage superposition when reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural language data biases models to commit to a token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought reasoning, and identify the conditions under which it collapses.

2026-07-07

Conference on Language Modeling (accepté)

Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

Tatsuhiro Nakamori

Laura Gomezjurado Gonzalez

Ganesh Talluri

Ansh Tiwari

Hideyuki Kawashima

Ioannis Mitliagkas

Hiroki Naganuma

Low-rank gradient compression reduces communication in distributed training by representing updates with rank-…

2026-05-28

CoLoRAI @ International Conference on Machine Learning (poster)

Hydra: Towards Transferable Multi-Task Learning on Temporal Graphs

Kiarash Shamsi

Tran Gia Bao Ngo

Baris Coskunuzer

Michael M. Bronstein

Cuneyt Gurcan Akcora

Real-world evolving networks are naturally modeled as temporal graphs (TGs), where capturing temporal dynamics is essential for predicting f… (voir plus)uture graph properties that support downstream decision-making. Existing temporal graph methods have been developed primarily for single-task prediction, and little is known about their generalization across tasks or transfer to unseen networks. This leaves the challenge of multi-task graph property prediction in TGs largely open. We address this challenge by introducing Hydra, a novel architecture that integrates local connectivity features from temporal GNNs with a spectral learning module that captures global connectivity patterns. This design enables joint learning of local and global information under a multi-task objective. In multi-task classification, Hydra achieves an 8.9% relative gain in AUC over the strongest competitor. In multi-task regression, Hydra achieves competitive results in all three tasks, while obtaining the best results in two tasks with a 8.2% relative gain in MAE compared to the strongest baseline. Moreover, Hydra delivers these gains with a 22× reduction in training time compared to temporal transfer models. These results provide the first systematic evidence that multi-task transferable learning on temporal graphs is effective. By delivering consistent top-ranked performance, Hydra highlights multi-task training on temporal graphs as a promising direction toward adaptable foundation models for temporal graphs.

2026-05-20

GFM @ International Conference on Machine Learning (poster)

Tensor Cookbook: Mastering Tensors through Diagrams

Beheshteh T. Rakhshan

High-dimensional data arise naturally in many areas of science and engineering, including machine learning, signal processing, computational… (voir plus) physics, and statistics. Such data are often represented as tensors, multi-dimensional generalizations of matrices. While tensors provide a natural representation for multi-modal structure, their direct manipulation quickly becomes challenging as the order grows: the number of parameters increases exponentially, and algebraic expressions involving many indices become difficult to interpret and implement. Tensor networks (TNs) provide an effective framework for addressing these challenges. Originally introduced by Penrose and developed extensively in quantum physics, the graphical language of tensor networks encodes contractions as edges in a graph, reducing notational overhead and revealing structural properties obscured by index notation. Despite the central role of high-dimensional tensors in modern machine learning and numerical analysis, tensor network diagrams remain underutilized outside quantum computing, partly due to the lack of a self-contained mathematical reference accessible to a broad technical audience. This manuscript provides a self-contained guide to tensor networks and their use in tensor algebra. We present the main operations on tensors, contractions, products, and reshaping through, graphical notation, and show how classical tensor decompositions and related computations are naturally expressed in this framework. We also illustrate how tensor networks simplify the derivation of gradients and the manipulation of high-dimensional probability distributions. Throughout, we show that the diagrammatic approach yields genuinely shorter and more transparent proofs of classical identities, rank bounds, and gradient formulas that would otherwise require laborious index manipulation.

2026-05-14

arXiv (prépublication)

T-GRAB: A Synthetic Diagnostic Benchmark for Learning on Temporal Graphs

Alireza Dizaji

Benedict Aaron Tjandra

Mehrab Hamidi

Shenyang Huang

Dynamic graph learning methods have recently emerged as powerful tools for modelling relational data evolving through time. However, despite… (voir plus) extensive benchmarking efforts, it remains unclear whether current Temporal Graph Neural Networks (TGNNs) effectively capture core temporal patterns such as periodicity, cause-and-effect, and long-range dependencies. In this work, we introduce the Temporal Graph Reasoning Benchmark (T-GRAB), a comprehensive set of synthetic tasks designed to systematically probe the capabilities of TGNNs to reason across time. T-GRAB provides controlled, interpretable tasks that isolate key temporal skills: counting/memorizing periodic repetitions, inferring delayed causal effects, and capturing long-range dependencies over both spatial and temporal dimensions. We evaluate 11 temporal graph learning methods on these tasks, revealing fundamental shortcomings in their ability to generalize temporal patterns. Our findings offer actionable insights into the limitations of current models, highlight challenges hidden by traditional real-world benchmarks, and motivate the development of architectures with stronger temporal reasoning abilities. The code for T-GRAB can be found at: https://github.com/alirezadizaji/T-GRAB.

2026-04-19

ACM SIGKDD Conference on Knowledge Discovery and Data Mining (publié)

Model Merging via Data-Free Covariance Estimation

Marawan Gamal Abdel Hameed

Derek Tam

Pascal Jr Tikeng Notsawo

Colin Raffel

2026-03-31

arXiv (prépublication)