Portrait of Kirill Neklyudov

Kirill Neklyudov

Core Academic Member
kirill.neklyudov@mila.quebec
Assistant Professor, Université de Montréal, Mathematics and Statistics
Research Topics
Deep Learning
Dynamical Systems
Generative Models
Molecular Modeling
Probabilistic Models
Website
Google Scholar
X

Current Students

Majdi Hassan
Master's Research - Université de Montréal
Principal supervisor :
Dominique Beaini
Website
Github
Google Scholar
Tobias Höppe
Independent visiting researcher - Helmholtz Zentrum München
Github
Google Scholar
Jessie Richter-Powell
Independent visiting researcher - Université de Montréal
Google Scholar
Charlie Tan
Independent visiting researcher - University of Oxford

Publications

Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC
Wu Lin
Felix Dangel
Runa Eschenhagen
Kirill Neklyudov
Agustinus Kristiadi
Richard E. Turner
Alireza Makhzani
Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their precondition… (see more)ing Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.
2024-01-01
ICML (published)
proceedings.mlr.press
arxiv.org