Portrait of Ivan Titov

Ivan Titov

Collaborating researcher
Supervisor
Siva Reddy
Co-supervisor
Yoshua Bengio
Research Topics
Deep Learning
Natural Language Processing
Website
Google Scholar

Publications

Operationalizing the Superficial Alignment Hypothesis via Task Complexity
Tomás Vergara Browne
Darshan Patil
Ivan Titov
Siva Reddy
Tiago Pimentel
Marius Mosbach
The superficial alignment hypothesis (SAH) posits that large language models learn most of their knowledge during pre-training, and that pos… (see more)t-training merely surfaces this knowledge. The SAH, however, lacks a precise definition, which has led to (i) different and seemingly orthogonal arguments supporting it, and (ii) important critiques to it. We propose a new metric called **Task Complexity**: the length of the shortest program that achieves a target performance on a task. In this framework, the SAH claims that pre-trained models drastically reduce the task complexity of achieving high performance on many tasks. Our definition unifies prior arguments supporting the SAH, interpreting them as different strategies to find such short programs. Experimentally, we estimate task complexities of mathematical reasoning, machine translation, and instruction following tasks and show that their respective task complexities can be remarkably low when conditioned on a pre-trained model. Further, we find that pre-training enables access to strong performances on our tasks, but it can require programs of gigabytes of length to access them. Post-training, on the other hand, collapses the complexity of reaching this same performance by several orders of magnitude. Overall, our results highlight that task adaptation can require remarkably little information—often just a few kilobytes.
2025-12-31
International Conference on Machine Learning (Accept (regular))
openreview.net
openreview.net