Elvis Dohmatob

Associate Academic Member

Associate Professor, Concordia University, Department of Computer Science and Software Engineering

Meta Facebook AI Research (FAIR)

Research Topics

Adversarial Robustness

Algorithmic Fairness

Machine Learning Theory

Optimization

Website

Google Scholar

Publications

Model Collapse Demystified: The Case of Regression

Elvis Dohmatob

Yunzhen Feng

Julia Kempe

In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereb… (see more)y as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e the model collapses. In this work, we study this phenomenon in the setting of high-dimensional regression and obtain analytic formulae which quantitatively outline this phenomenon in a broad range of regimes. In the special case of polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.

2024-02-01

arXiv (published)

doi.org

arxiv.org

Precise Accuracy / Robustness Tradeoffs in Regression: Case of General Norms

Elvis Dohmatob

Meyer Scetbon

2024-01-01

International Conference on Machine Learning (published)

proceedings.mlr.press

Dark control: The default mode network as a reinforcement learning agent

Elvis Dohmatob