Portrait de Julian Enoch n'est pas disponible

Julian Enoch

julian.enoch@mila.quebec
Doctorat - Concordia
Superviseur⋅e principal⋅e
Elvis Dohmatob
Sujets de recherche
IA digne de confiance
Interprétabilité
Robustesse antagoniste
Théorie de l'apprentissage automatique

Publications

Exact Scaling Laws for a Random-Features Bigram Model
Julian Enoch
Elvis Dohmatob
Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even i… (voir plus)n simple settings. In this work, we study capacity-limited scaling in a bottlenecked bigram model with power-law (Zipf) token statistics, viewed as an analytically tractable toy model of a large language model. Fixed random embeddings serve as a proxy for representational capacity and induce an explicit bottleneck of width, allowing us to derive scaling laws for prediction regret as a function of model width, vocabulary size, and the Zipf exponent. Our results complement recent theoretical accounts based on optimization dynamics by isolating scaling due purely to representational constraints. In the appropriate regime, the theory recovers the functional form of classical neural scaling laws and suggests that heavy-tailed language statistics may underlie both optimization- and capacity-driven scaling. Experiments natural language datasets (WikiText2, OpenWebText) closely match the predicted curves.
2026-10-12
arXiv (prépublication)
doi.org
arxiv.org