Portrait de Felix Dangel

Felix Dangel

Membre académique associé
felix.dangel@mila.quebec
Concordia University
Sujets de recherche
Apprentissage profond
IA pour la science
Optimisation
Paysages de perte
Symétrie
Site web
Google Scholar
X

Étudiants actuels

Will Gevao
Collaborateur·rice de recherche
william.gevao@mila.quebec
Ishikawa Satoki
Visiteur de recherche indépendant - School of Computing, Institute of Science Tokyo
Co-superviseur⋅e :
Kirill Neklyudov
robosatoki@outlook.jp

Publications

Exploiting weight-space symmetries for approximating curvature
Artem Artemev
Rui Xia
Benjamin M. Boyd
Youjing Yu
Felix Dangel
Guillaume Hennequin
Alberto Bernacchia
Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of moder… (voir plus)n deep networks. Surprisingly, no previous work has exploited the curvature constraints that arise from well known weight-space symmetries in loss landscapes. By analytically averaging over group actions that leave the loss invariant, we construct structured Hessian approximations from single gradients that can be tractably estimated, stored, and inverted. The choice of user-specified symmetry group directly governs the trade-off between approximation accuracy and computational cost. Moreover, our framework provides a unifying theoretical lens for viewing existing methods; in particular, a specific choice of symmetry group recovers Shampoo/Muon-like curvature estimates. We validate our method on a range of network architectures, and deploy it to second-order optimization benchmarks, including a small language model. Our curvature estimation framework might find applications in other machine learning problems such as uncertainty estimation, continual learning, compression/pruning, training data attribution, and more.
2026-05-29
arXiv (prépublication)
doi.org
arxiv.org
Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage
Alan Milligan
Zikun Xu
Simon Lacoste-Julien
Felix Dangel
Wu Lin
Shampoo-based methods, such as KL-Shampoo and SOAP, have demonstrated strong performance in training neural networks and rely on QR decompos… (voir plus)ition. Because existing QR implementations require single-precision (FP32) arithmetic and remain computationally expensive, these methods become time- and memory-intensive when their preconditioning matrices are large. Moreover, using BFloat16 (BFP16) storage to reduce memory usage can degrade the performance of Shampoo-based methods. We propose a reparametrization of the preconditioner that supports BFP16 storage and forms a complete basis by combining updated basis vectors with unchanged ones. By updating only part of the basis through QR decomposition in a subspace, our approach reduces computational overhead while mitigating the performance degradation caused by BFP16 storage. Our approach applies broadly to Shampoo-based methods that employ QR decomposition, including KL-Shampoo, SOAP, and KL-SOAP. In particular, it improves the performance of SOAP and KL-SOAP under BFP16 storage, enabling KL-SOAP to match or exceed KL-Shampoo. Overall, our approach makes Shampoo-based methods more memory- and time-efficient.
2026-05-24
arXiv (prépublication)
doi.org
openreview.net
Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
Angelo Porrello
Pietro Buzzega
Felix Dangel
Thomas Sommariva
Riccardo Salami
Lorenzo Bonicelli
Simone Calderara
Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task … (voir plus)interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.
2026-02-18
Open MIND (prépublication)
doi.org
arxiv.org
Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC
Wu Lin
Felix Dangel
Runa Eschenhagen
Kirill Neklyudov
Agustinus Kristiadi
Richard E. Turner
Alireza Makhzani
Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their precondition… (voir plus)ing Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.
2023-12-31
ICML (publié)
proceedings.mlr.press
proceedings.mlr.press