Portrait de Fengyuan Liu

Fengyuan Liu

Maîtrise recherche - McGill
Superviseur⋅e principal⋅e
Siva Reddy
Co-supervisor
Dzmitry Bahdanau
Sujets de recherche
Agent basé sur un LLM
Alignement de l'IA
Apprentissage continu
Apprentissage décentralisé
Apprentissage fédéré
Apprentissage multimodal
Apprentissage par transfert
Apprentissage profond
Apprentissage sur variétés
Compression de modèles
Grands modèles de langage (LLM)
IAG (Intelligence Artificielle Générale)
Optimisation
Optimisation décentralisée
Traitement du langage naturel
XAI (IA explicable)
Site web
GitHub

Publications

BRIDGE: Predicting Human Task Completion Time From Model Performance
Fengyuan Liu
Jay Gala
Dzmitry Bahdanau
Siva Reddy
Hugo Larochelle
Mila - Québec
AI Institute
McGill University
Polytechnique Montréal
Periodic Labs
Servicenow Research
Canada Cifar
AI Chair
Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task diffic… (voir plus)ulty. Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks. In this work, we propose BRIDGE, a unified psychometric framework that learns the latent difficulty scale from model responses and anchors it to human task completion time. Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple benchmarks. We demonstrate that latent task difficulty varies linearly with the logarithm of human completion time, allowing human task completion time to be inferred for new benchmarks from model performance alone. Leveraging this alignment, we forecast frontier model capabilities in terms of human task length and independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months.
2026-02-05
ArXiv (prépublication)
doi.org
arxiv.org