Portrait de Maude Lizaire

Maude Lizaire

lizairem@mila.quebec
Doctorat - UdeM
Superviseur⋅e principal⋅e
Guillaume Rabusseau
Sujets de recherche
Apprentissage spectral
Factorisation tensorielle
Méthodes de réduction de la dimensionnalité
Méthodes formelles
Réseaux de neurones récurrents
Théorie de l'apprentissage automatique
Google Scholar

Publications

On the Role of Depth in the Expressivity of RNNs
Maude Lizaire
Michael Rizvi-Martel
Éric Dupuis
Guillaume Rabusseau
The benefits of depth in feedforward neural networks (FNNs) are well known: composing multiple layers of linear transformations with nonline… (voir plus)ar activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs’ memory capacity efficiently with respect to parameters, enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We extend our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.
2026-02-02
International Conference on Artificial Intelligence and Statistics (spotlight)
doi.org
openreview.net
A Tensor Decomposition Perspective on Second-order RNNS
Maude Lizaire
Michael Rizvi-Martel
Marawan Gamal Abdel Hameed
Guillaume Rabusseau
Second-order Recurrent Neural Networks (2RNNs) extend RNNs by leveraging second-order interactions for sequence modelling. These models are … (voir plus)provably more expressive than their first-order counterparts and have connections to well-studied models from formal language theory. However, their large parameter tensor makes computations intractable. To circumvent this issue, one approach known as MIRNN consists in limiting the type of interactions used by the model. Another is to leverage tensor decomposition to diminish the parameter count. In this work, we study the model resulting from parameterizing 2RNNs using the CP decomposition, which we call CPRNN. Intuitively, the rank of the decomposition should reduce expressivity. We analyze how rank and hidden size affect model capacity and show the relationships between RNNs, 2RNNs, MIRNNs, and CPRNNs based on these parameters. We support these results empirically with experiments on the Penn Treebank dataset which demonstrate that, with a fixed parameter budget, CPRNNs outperforms RNNs, 2RNNs, and MIRNNs with the right choice of rank and hidden size.
2024-07-22
International Conference on Machine Learning (Accept (Spotlight))
doi.org
proceedings.mlr.press
Simulating Weighted Automata over Sequences and Trees with Transformers
Michael Rizvi
Maude Lizaire
Clara Lacroce
Guillaume Rabusseau
Transformers are ubiquitous models in the natural language processing (NLP) community and have shown impressive empirical successes in the p… (voir plus)ast few years. However, little is understood about how they reason and the limits of their computational capabilities. These models do not process data sequentially, and yet outperform sequential neural models such as RNNs. Recent work has shown that these models can compactly simulate the sequential reasoning abilities of deterministic finite automata (DFAs). This leads to the following question: can transformers simulate the reasoning of more complex finite state machines? In this work, we show that transformers can simulate weighted finite automata (WFAs), a class of models which subsumes DFAs, as well as weighted tree automata (WTA), a generalization of weighted automata to tree structured inputs. We prove these claims formally and provide upper bounds on the sizes of the transformer models needed as a function of the number of states the target automata. Empirically, we perform synthetic experiments showing that transformers are able to learn these compact solutions via standard gradient-based training.
2024-05-01
International Conference on Artificial Intelligence and Statistics (Accept (Poster))
doi.org
proceedings.mlr.press