Portrait de Mark-Anthony Moisescu-Pareja n'est pas disponible

Mark-Anthony Moisescu-Pareja

mark-anthony.moisescu-pareja@mila.quebec
Maîtrise recherche - Concordia
Superviseur⋅e principal⋅e
Elvis Dohmatob
Sujets de recherche
Optimisation
Théorie de l'apprentissage automatique
Google Scholar

Publications

New Multiple Descent Phenomena Induced by Heterogeneous Data
Mark-Anthony Moisescu-Pareja
Elvis Dohmatob
Modern machine learning models are often trained on large datasets assembled from many different sources, whose statistical properties can d… (voir plus)iffer substantially. Understanding how this heterogeneity affects generalization performance is therefore important for explaining how model performance changes as model size increases. In this work, we study this question in linear random-feature models and show that mixtures of datasets with different covariance structures can give rise to a range of multiple-descent phenomena. We derive exact asymptotic expressions for the test error of each data source in the high-dimensional regime, and show how heterogeneity in covariance scale and spectral structure can introduce additional, group-dependent interpolation thresholds as model size increases. These phenomena arise even with noiseless labels and linear activations, showing that neither label noise nor activation nonlinearity is necessary for multiple-descent. Our results therefore provide a different perspective from classical explanations of multiple descent. More broadly, our results have potential implications for algorithmic fairness by showing that model scaling can affect different data groups unevenly, amplifying disparities in their generalization errors. We validate our theoretical results on synthetic Gaussian mixtures and real-world datasets, including MNIST and CIFAR-10.
2026-10-26
arXiv (prépublication)
doi.org
arxiv.org