Portrait of Julian Enoch is unavailable

Julian Enoch

julian.enoch@mila.quebec
PhD - Concordia University
Supervisor
Elvis Dohmatob
Research Topics
Adversarial Robustness
Interpretability
Machine Learning Theory
Trustworthy AI

Publications

Exact Scaling Laws for a Random-Features Bigram Model
Julian Enoch
Elvis Dohmatob
Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even i… (see more)n simple settings. In this work, we study capacity-limited scaling in a bottlenecked bigram model with power-law (Zipf) token statistics, viewed as an analytically tractable toy model of a large language model. Fixed random embeddings serve as a proxy for representational capacity and induce an explicit bottleneck of width, allowing us to derive scaling laws for prediction regret as a function of model width, vocabulary size, and the Zipf exponent. Our results complement recent theoretical accounts based on optimization dynamics by isolating scaling due purely to representational constraints. In the appropriate regime, the theory recovers the functional form of classical neural scaling laws and suggests that heavy-tailed language statistics may underlie both optimization- and capacity-driven scaling. Experiments natural language datasets (WikiText2, OpenWebText) closely match the predicted curves.
2026-10-12
arXiv (preprint)
doi.org
arxiv.org