Portrait de Gül Sena Altıntaş n'est pas disponible

Gül Sena Altıntaş

Alumni
Site web
Google Scholar

Publications

The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
Devin Kwok
Gül Sena Altıntaş
Colin Raffel
David Rolnick
Neural network training is inherently sensitive to initialization and the randomness induced by stochastic gradient descent. However, it is … (voir plus)unclear to what extent such effects lead to meaningfully different networks, either in terms of the models' weights or the underlying functions that were learned. In this work, we show that during the initial "chaotic" phase of training, even extremely small perturbations reliably causes otherwise identical training trajectories to diverge-an effect that diminishes rapidly over training time. We quantify this divergence through (i)
2024-12-31
ICML (publié)
doi.org
proceedings.mlr.press
The Butterfly Effect: Tiny Perturbations Cause Neural Network Training to Diverge
Gül Sena Altıntaş
Devin Kwok
David Rolnick
Neural network training begins with a chaotic phase in which the network is sensitive to small perturbations, such as those caused by stocha… (voir plus)stic gradient descent (SGD). This sensitivity can cause identically initialized networks to diverge both in parameter space and functional similarity. However, the exact degree to which networks are sensitive to perturbation, and the sensitivity of networks as they transition out of the chaotic phase, is unclear. To address this uncertainty, we apply a controlled perturbation at a single point in training time and measure its effect on otherwise identical training trajectories. We find that both the
2024-06-15
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net