Portrait de Matteo Boglioni

Matteo Boglioni

matteo.boglioni@mila.quebec
Collaborateur·rice de recherche - McGill
Superviseur⋅e principal⋅e
Siva Reddy
Site web
Google Scholar

Publications

LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Matteo Boglioni
Thibault Rousset
Siva Reddy
Marius Mosbach
Verna Dankers
LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc r… (voir plus)emoval methods. Unlearning has emerged as a promising solution, with state-of-the-art(SOTA) methods often following a localize-first, unlearn-second paradigm that targets specific model parameters. However, existing benchmarks evaluate unlearning solely at the output level, leaving open the question of whether unlearning truly erases knowledge from a model's parameters or merely obfuscates it, a concern reinforced by the success of resurfacing attacks. To bridge this gap, we introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage. We use LACUNA to benchmark current SOTA unlearning methods and find that, despite strong output-level performance, existing methods are highly imprecise and susceptible to resurfacing attacks. We further show that when localization is successful, even a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks, highlighting the importance of precise unlearning. We release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.
2026-07-01
arXiv (prépublication)
doi.org
arxiv.org
WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents
Imene Kerboua
Fatemeh Pesaran zadeh
Xing Han Lu
Weijian Qi
Alexander Miller
Junyi Song
Yunjia Tian
Dongjin Kang
Seyeon Choi
Marzia Nouri
Ewen Gueguen
Matteo Boglioni
Fengyuan Liu
Zeyi Liao
Mengqi Yuan
Yue Li
Alexandre Lacoste
Alexandre Drouin
Spandana Gella
Huan Sun … (voir 2 de plus)
Gunhee Kim
Siva Reddy
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous … (voir plus)applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by Gemini 3.1 Pro, which attains 37.0 % success under a 50-step budget, while open-source models' performance does not exceed 27.7% success. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release.
2026-05-22
AIWILD @ International Conference on Machine Learning (publié)
openreview.net
openreview.net