Julia Kreutzer

Associate Industry Member

Research Topics

Natural Language Processing

Biography

Julia Kreutzer is a Senior Research Scientist at Cohere Labs, where she conducts fundamental research for LLMs. Previously, she worked at Google Translate in Montreal, and completed her PhD at Heidelberg University.

She is broadly interested in making NLP research more language-inclusive and accessible, and advancing evaluation methodologies to more rigorously measure progress.

Publications

JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

Erlis Lushtaku

Bora Kargi

Ali Elganzory

Fabio Ferreira

Alejandro Salamanca

Julia Kreutzer

David Salinas

LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks … (see more)ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices--the benchmark, the judge model, the prompt, the inference backend--affect the conclusions we draw about model quality. We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, llama.cpp, or OpenRouter can serve as both candidate and judge. Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.

2026-06-30

arXiv (preprint)

doi.org

arxiv.org

The Culture Funnel: You Can't Align What isn't in the Data

Ananya Sahu

Mehrnaz Mofakhami

Daniel D'souza

Thomas Euyang

Julia Kreutzer

Marzieh Fadaee

Current cultural alignment approaches focus on inference-time interventions, assuming models already contain sufficient cultural knowledge. … (see more)We argue modern LLM pipelines suffer from a cultural data funnel. Using a multidimensional tagging framework across pretraining, fine-tuning, alignment, and reasoning datasets, we show explicit cultural signals decline sharply during post-training, while geographically concentrated, task-specialized data dominates. Multilinguality enhances geographic diversity of cultural knowledge but does not ensure balanced representation. Our tags improve downstream cultural benchmark performance, demonstrating that advances require shifting focus in training data pipelines. To facilitate future research, we release our culturally tagged dataset with 5.6M samples at https://huggingface.co/datasets/CohereLabs/CultureMarkers.

2026-06-10

arXiv (preprint)

doi.org

arxiv.org

Tiny Aya: Bridging Scale and Multilingual Depth

Alejandro Salamanca

Diana Abagyan

Daniel D'souza

Ammar Khairi

David Mora

Saurabh Dash

Viraat Aryabumi

Sara Rajaee

Mehrnaz Mofakhami

Ananya Sahu

Thomas Euyang

Brittawnya Prince

Madeline Smith

Hangyu Lin

Acyr Locatelli

Sara Hooker

Tom Kocmi

Aidan Gomez

Ivan Zhang

Phil Blunsom … (see 6 more)

Nick Frosst

Joelle Pineau

Beyza Ermis

Ahmet Üstün

Julia Kreutzer

Marzieh Fadaee

Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraini… (see more)ng, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.

2026-03-11

arXiv (preprint)

doi.org

arxiv.org

Modelling Latent Translations for Cross-Lingual Transfer

Edoardo Ponti

Julia Kreutzer

Ivan Vulić

Siva Reddy

2021-07-22

ArXiv (preprint)

arxiv.org