Shawn Whitfield

shawn.whitfield@mila.quebec

Scientifique en recherche appliquée, Recherche appliquée en apprentissage automatique

Alumni

Google Scholar

Publications

High-resolution dissection of concept acquisition in different families of protein language models

Shawn Whitfield

Tom Marty

Robert M. Vernon

Christopher J. Langmead

Dhanya Sridhar

Quentin Fournier

Protein language models have been increasingly successful on tasks ranging from fitness prediction to functional design, yet what biological… (voir plus) knowledge they acquire and where it is encoded within their internal representations remain underexplored. Through a high-resolution layer-by-layer interpretability analysis of 8 models from the ESM2 and AMPLIFY families on 22 concepts from human proteome annotations, we found that these models encode concepts of increasing levels of complexity along their depth: basic physicochemical properties and linear motifs are best captured by early-layer embeddings, secondary structure from subsequent layers, and domain-level semantics from middle layers. Principal component projections of these embeddings showed that they separate biologically meaningful protein groupings, and molecular-biology-inspired interventions demonstrated that pLM embeddings can discriminate phosphomimic-active from inactive mutants. Perhaps surprisingly, we observed that pretraining data and compute had a greater impact on the linear emergence of biological concepts than scaling up parameters. By revealing where biological knowledge is captured in pLMs and which choices shape its emergence, our work offers insights to develop more robust, biologically grounded protein language models.

2026-07-22

bioRxiv (accepté)

doi.org

Virtual Cells: Predict, Explain, Discover

Emmanuel Noutahi

Jason Hartford

Prudencio Tossou

Shawn Whitfield

Ali Denton

Cas Wognum

Kristina Ulicna

Michael Craig