Portrait of Ryan Vogt is unavailable

Ryan Vogt

ryan.vogt@mila.quebec
Postdoctorate - Université de Montréal
Supervisor
Guillaume Lajoie
Co-supervisor
Guillaume Lajoie
Research Topics
Computational Neuroscience
Data Science
Dynamical Systems
NeuroAI
Neuroscience

Publications

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Ryan Vogt
Maximilian Puelma Touzel
Eli Shlizerman
Guillaume Lajoie
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization i… (see more)s sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics because forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of nonlinear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this understudied, but theoretically grounded tool for understanding training stability in RNNs.
2022-03-01
Frontiers in Applied Mathematics and Statistics (published)
doi.org
arxiv.org