Desmond Elliott

LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

Transforming a large language model (LLM) into a Vision-Language Model (VLM) can be achieved by mapping the visual tokens from a vision enco… (see more)der into the embedding space of an LLM. Intriguingly, this mapping can be as simple as a shallow MLP transformation. To understand why LLMs can so readily process visual tokens, we need interpretability methods that reveal what is encoded in the visual token representations at every layer of LLM processing. In this work, we introduce LatentLens, a novel approach for mapping latent representations to descriptions in natural language. LatentLens works by encoding a large text corpus and storing contextualized token representations for each token in that corpus. Visual token representations are then compared to their contextualized textual representations, with the top-k nearest neighbor representations providing descriptions of the visual token. We evaluate this method on 10 different VLMs, showing that commonly used methods, such as LogitLens, substantially underestimate the interpretability of visual tokens. With LatentLens instead, the majority of visual tokens are interpretable across all studied models and all layers. Qualitatively, we show that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens. More broadly, our findings contribute new evidence on the alignment between vision and language representations, opening up new directions for analyzing latent representations.

2026-04-29

International Conference on Machine Learning (accepted)

doi.org

openreview.net

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

Israfel Salazar

Manuel Fernández Burda

Shayekh Bin Islam

Arshia Soltani Moakhar

Shivalika Singh

Fabian Farestam

Angelika Romanou

Danylo Boiko

Dipika Khullar

Mike Zhang

Dominik Krzemiński

Jekaterina Novikova

Luísa Shimabucoro

Joseph Marvin Imperial

Rishabh Maheshwary

Sharad Duwal

Alfonso Amayuelas

Swati Rajwal

Jebish Purbey

Ahmed Ruby … (see 25 more)

Nicholas Popović

Marek Šuppa

Azmine Toushik Wasi

Ram Mohan Rao Kadiyala

Olga Tsymboi

Maksim Kostritsya

Bardia Soltani Moakhar

Gabriel da Costa Merlin

Otávio Ferracioli Coletti

Maral Jabbari Shiviari

MohammadAmin Farahani Fard

Silvia Fernandez

María Grandury

Dmitry Abulkhanov

Drishti Sharma

Andre Guarnier De Mitri

Leticia Bossatto Marchezi

Setayesh Heydari

Johan Obando-Ceron

Nazar Kohut

Beyza Ermis

Desmond Elliott

Enzo Ferrante

Sara Hooker

Marzieh Fadaee

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multiling… (see more)ual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.

2025-12-31

International Conference on Learning Representations (Accept (Poster))

openreview.net

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

Emanuele Bugliarello

Fangyu Liu

Jonas Pfeiffer

Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of… (see more) a multilingual benchmark, however, vision-and-language research has mostly focused on English language tasks. To fill this gap, we introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together - by both aggregating pre-existing datasets and creating new ones - visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups. Based on the evaluation of the available state-of-the-art models, we find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks. Moreover, downstream performance is partially explained by the amount of available unlabelled textual data for pretraining, and only weakly by the typological distance of target-source languages. We hope to encourage future research efforts in this area by releasing the benchmark to the community.

2022-07-18

ICML (Accept for Short Presentation)

doi.org

proceedings.mlr.press

Visually Grounded Reasoning across Languages and Cultures

Fangyu Liu

Emanuele Bugliarello

Edoardo Ponti

Siva Reddy

Nigel Collier

Desmond Elliott

The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and … (see more)images of ImageNet. While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias. Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. In particular, we let the selection of both concepts and images be entirely driven by native speakers, rather than scraping them automatically. Specifically, we focus on a typologically diverse set of languages, namely, Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish. On top of the concepts and images obtained through this new protocol, we create a multilingual dataset for Multicultural Reasoning over Vision and Language (MaRVL) by eliciting statements from native speaker annotators about pairs of images. The task consists of discriminating whether each grounded statement is true or false. We establish a series of baselines using state-of-the-art models and find that their cross-lingual transfer performance lags dramatically behind supervised performance in English. These results invite us to reassess the robustness and accuracy of current state-of-the-art models beyond a narrow domain, but also open up new exciting challenges for the development of truly multilingual and multicultural systems.

2021-10-31

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (published)

doi.org

openreview.net

Mila's Community of Practice: Agentic AI

Venture Sprint

AI Advantage: Productivity in Public Service

Desmond Elliott

Publications

Mila's Community of Practice: Agentic AI

Venture Sprint

AI Advantage: Productivity in Public Service

Popular keywords:

Desmond Elliott

Publications