Portrait de Aristide Baratin

Aristide Baratin

aristidebaratin@hotmail.com
Visiteur de recherche indépendant - Samsung SAIT
Superviseur⋅e principal⋅e
Simon Lacoste-Julien
Sujets de recherche
Apprentissage profond
Site web
Google Scholar
GitHub

Publications

Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Thomas George
Guillaume Lajoie
Aristide Baratin
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called `laz… (voir plus)y' training regime in which the network can be well approximated by its linearization around initialization. Here we investigate the comparative effect of the lazy (linear) and feature learning (non-linear) regimes on subgroups of examples based on their difficulty. Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones. In other words, the non-linear dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, including c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a new understanding of how deep networks prioritize resources across example difficulty.
2021-12-31
Trans. Mach. Learn. Res. (publié)
doi.org
openreview.net
Implicit Regularization via Neural Feature Alignment
Aristide Baratin
Thomas George
César Laurent
R Devon Hjelm
Guillaume Lajoie
Pascal Vincent
Simon Lacoste-Julien
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induc… (voir plus)ed by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.
2019-12-31
arXiv (prépublication)
doi.org
proceedings.mlr.press
On the Spectral Bias of Neural Networks
Nasim Rahaman
Aristide Baratin
Devansh Arpit
Felix Draxler
Min Lin
Fred A. Hamprecht
Yoshua Bengio
Aaron Courville
Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with …
2018-12-31
International Conference on Machine Learning (publié)
doi.org
proceedings.mlr.press
Mutual Information Neural Estimation
Mohamed Ishmael Belghazi
Aristide Baratin
Sai Rajeswar
Sherjil Ozair
Yoshua Bengio
Aaron Courville
R Devon Hjelm
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent … (voir plus)over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.
2018-07-02
International Conference on Machine Learning (inconnu)
doi.org
proceedings.mlr.press
On the Spectral Bias of Deep Neural Networks
Nasim Rahaman
Devansh Arpit
Aristide Baratin
Felix Draxler
Min Lin
Fred Hamprecht
Yoshua Bengio
Aaron Courville
It is well known that over-parametrized deep neural networks (DNNs) are an overly expressive class of functions that can memorize even rando… (voir plus)m data with
2018-06-21
arXiv.org (prépublication)
dblp.uni-trier.de
MINE: Mutual Information Neural Estimation
Ishmael Belghazi
Sai Rajeswar
Aristide Baratin
R Devon Hjelm
Aaron Courville
This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size. MINE… (voir plus) is back-propable and we prove that it is strongly consistent. We illustrate a handful of applications in which MINE is succesfully applied to enhance the property of generative models in both unsupervised and supervised settings. We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems. Our results demonstrate substantial added flexibility and improvement in these settings.
2018-05-03
ICLR.cc/2018/Conference (inconnu)
openreview.net
openreview.net
A3T: Adversarially Augmented Adversarial Training
Akram Erraqabi
Aristide Baratin
Yoshua Bengio
Simon Lacoste-Julien
Recent research showed that deep neural networks are highly sensitive to so-called adversarial perturbations, which are tiny perturbations o… (voir plus)f the input data purposely designed to fool a machine learning classifier. Most classification models, including deep learning models, are highly vulnerable to adversarial attacks. In this work, we investigate a procedure to improve adversarial robustness of deep neural networks through enforcing representation invariance. The idea is to train the classifier jointly with a discriminator attached to one of its hidden layer and trained to filter the adversarial noise. We perform preliminary experiments to test the viability of the approach and to compare it to other standard adversarial training methods.
2018-01-11
ArXiv (prépublication)
arxiv.org
arxiv.org