Julia Kreutzer

Membre industriel associé

Sujets de recherche

Traitement du langage naturel

Biographie

Julia Kreutzer est chercheuse principale (Senior Research Scientist) chez Cohere Labs, où elle mène des recherches fondamentales sur les grands modèles de langage (LLM). Auparavant, elle a travaillé pour Google Translate à Montréal et a obtenu son doctorat à l’Université de Heidelberg.

Ses intérêts de recherche portent principalement sur la promotion d’une recherche en traitement automatique du langage naturel (TALN) plus inclusive sur le plan linguistique et plus accessible, ainsi que sur l’amélioration des méthodologies d’évaluation afin de mesurer les progrès de manière plus rigoureuse.

Publications

JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

Erlis Lushtaku

Bora Kargi

Ali Elganzory

Fabio Ferreira

Alejandro Salamanca

Julia Kreutzer

David Salinas

LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks … (voir plus)ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices--the benchmark, the judge model, the prompt, the inference backend--affect the conclusions we draw about model quality. We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, llama.cpp, or OpenRouter can serve as both candidate and judge. Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.

2026-06-30

arXiv (prépublication)

doi.org

arxiv.org

The Culture Funnel: You Can't Align What isn't in the Data

Ananya Sahu

Mehrnaz Mofakhami

Daniel D'souza

Thomas Euyang

Julia Kreutzer

Marzieh Fadaee

Current cultural alignment approaches focus on inference-time interventions, assuming models already contain sufficient cultural knowledge. … (voir plus)We argue modern LLM pipelines suffer from a cultural data funnel. Using a multidimensional tagging framework across pretraining, fine-tuning, alignment, and reasoning datasets, we show explicit cultural signals decline sharply during post-training, while geographically concentrated, task-specialized data dominates. Multilinguality enhances geographic diversity of cultural knowledge but does not ensure balanced representation. Our tags improve downstream cultural benchmark performance, demonstrating that advances require shifting focus in training data pipelines. To facilitate future research, we release our culturally tagged dataset with 5.6M samples at https://huggingface.co/datasets/CohereLabs/CultureMarkers.

2026-06-10

arXiv (prépublication)

doi.org

arxiv.org

Tiny Aya: Bridging Scale and Multilingual Depth

Alejandro Salamanca

Diana Abagyan

Daniel D'souza

Ammar Khairi

David Mora

Saurabh Dash

Viraat Aryabumi

Sara Rajaee

Mehrnaz Mofakhami

Ananya Sahu

Thomas Euyang

Brittawnya Prince

Madeline Smith

Hangyu Lin

Acyr Locatelli

Sara Hooker

Tom Kocmi

Aidan Gomez

Ivan Zhang

Phil Blunsom … (voir 6 de plus)

Nick Frosst

Joelle Pineau

Beyza Ermis

Ahmet Üstün

Julia Kreutzer

Marzieh Fadaee

Tiny Aya redefines what a small multilingual language model can achieve. Trained on 70 languages and refined through region-aware posttraini… (voir plus)ng, it delivers state-of-the-art in translation quality, strong multilingual understanding, and high-quality target-language generation, all with just 3.35B parameters. The release includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models targeting languages from Africa, South Asia, Europe, Asia-Pacific, and West Asia. This report details the training strategy, data composition, and comprehensive evaluation framework behind Tiny Aya, and presents an alternative scaling path for multilingual AI: one centered on efficiency, balanced performance across languages, and practical deployment.

2026-03-11

arXiv (prépublication)

doi.org

arxiv.org

Modelling Latent Translations for Cross-Lingual Transfer

Edoardo Ponti

Julia Kreutzer

Ivan Vulić

Siva Reddy

2021-07-22

ArXiv (prépublication)

arxiv.org