Gauthier Gidel

A Generative Approach to LLM Harmfulness Detection with Red Flag Tokens

Sophie Xhonneux

David Dobre

Mehrnaz Mofakhami

Leo Schwinn

Most safety training methods for large-language models (LLMs) based on fine-tuning rely on dramatically changing the output distribution of … (voir plus)the model when faced with a harmful request, shifting it from an unsafe answer to a refusal to respond. These methods inherently compromise model capabilities and might make auto-regressive models vulnerable to attacks that make likely an initial token of affirmative response. To avoid that, we propose to expand the model's vocabulary with a special token we call a *red flag token* (

2025-03-05

ICLR.cc/2025/Workshop/BuildingTrust (accepté)

openreview.net

LLM-Safety Evaluations Lack Robustness

Tim Beyer

Sophie Xhonneux

Simon Geisler

Gauthier Gidel

Leo Schwinn

Stephan Günnemann

In this paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of… (voir plus) noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field's ability to generate easily comparable results and make measurable progress.

2025-03-04

ArXiv (prépublication)

arxiv.org

A generative approach to LLM harmfulness detection with special red flag tokens

Sophie Xhonneux

David Dobre

Mehrnaz Mofakhami

Leo Schwinn

Gauthier Gidel

Most safety training methods for large language models (LLMs) based on fine-tuning rely on dramatically changing the output distribution of … (voir plus)the model when faced with a harmful request, shifting it from an unsafe answer to a refusal to respond. These methods inherently compromise model capabilities and might make auto-regressive models vulnerable to attacks that make likely an initial token of affirmative response. To avoid that, we propose to expand the model's vocabulary with a special token we call red flag token () and propose to fine-tune the model to generate this token at any time harmful content is generated or about to be generated. This novel safety training method effectively augments LLMs into generative classifiers of harmfulness at all times during the conversation. This method offers several advantages: it enables the model to explicitly learn the concept of harmfulness while marginally affecting the generated distribution, thus maintaining the model's utility. It also evaluates each generated answer rather than just the input prompt and provides a stronger defence against sampling-based attacks. In addition, it simplifies the evaluation of the model's robustness and reduces correlated failures when combined with a classifier. We further show an increased robustness to long contexts, and supervised fine-tuning attacks.

2025-02-22

ArXiv (prépublication)

arxiv.org

Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives

Leo Schwinn

Yan Scholten

Tom Wollschlager

Sophie Xhonneux

Stephen Casper

Stephan Günnemann

Gauthier Gidel

Misaligned research objectives have considerably hindered progress in adversarial robustness research over the past decade. For instance, an… (voir plus) extensive focus on optimizing target metrics, while neglecting rigorous standardized evaluation, has led researchers to pursue ad-hoc heuristic defenses that were seemingly effective. Yet, most of these were exposed as flawed by subsequent evaluations, ultimately contributing little measurable progress to the field. In this position paper, we illustrate that current research on the robustness of large language models (LLMs) risks repeating past patterns with potentially worsened real-world implications. To address this, we argue that realigned objectives are necessary for meaningful progress in adversarial alignment. To this end, we build on established cybersecurity taxonomy to formally define differences between past and emerging threat models that apply to LLMs. Using this framework, we illustrate that progress requires disentangling adversarial alignment into addressable sub-problems and returning to core academic principles, such as measureability, reproducibility, and comparability. Although the field presents significant challenges, the fresh start on adversarial robustness offers the unique opportunity to build on past experience while avoiding previous mistakes.

2025-02-17

ArXiv (prépublication)

arxiv.org

Advantage Alignment Algorithms

Juan Agustin Duque

Milad Aghajohari

Tim Cooijmans

razvan ciuca

Tianyu Zhang

Gauthier Gidel

Aaron Courville

2025-01-22

ICLR.cc/2025/Conference (présentation orale)

openreview.net

Learning diverse attacks on large language models for robust red-teaming and safety tuning

Seanie Lee

Minsu Kim

Lynn Cherif

David Dobre

Juho Lee

Sung Ju Hwang

Kenji Kawaguchi

Gauthier Gidel

Yoshua Bengio

Nikolay Malkin

Moksh J. Jain

Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of lar… (voir plus)ge language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.

2025-01-22

ICLR.cc/2025/Conference (poster)

doi.org

openreview.net

Performative Prediction on Games and Mechanism Design

António Góis

Mehrnaz Mofakhami

Fernando P. Santos

Simon Lacoste-Julien

Gauthier Gidel

2025-01-22

aistats.org/AISTATS/2025/Conference (poster)

doi.org

openreview.net

Solving Hidden Monotone Variational Inequalities with Surrogate Losses

Ryan D'Orazio

Danilo Vucetic

Zichu Liu

Junhyung Lyle Kim

Ioannis Mitliagkas

Gauthier Gidel

Deep learning has proven to be effective in a wide variety of loss minimization problems. However, many applications of interest, like minim… (voir plus)izing projected Bellman error and min-max optimization, cannot be modelled as minimizing a scalar loss function but instead correspond to solving a variational inequality (VI) problem. This difference in setting has caused many practical challenges as naive gradient-based approaches from supervised learning tend to diverge and cycle in the VI case. In this work, we propose a principled surrogate-based approach compatible with deep learning to solve VIs. We show that our surrogate-based approach has three main benefits: (1) under assumptions that are realistic in practice (when hidden monotone structure is present, interpolation, and sufficient optimization of the surrogates), it guarantees convergence, (2) it provides a unifying perspective of existing methods, and (3) is amenable to existing deep learning optimizers like ADAM. Experimentally, we demonstrate our surrogate-based approach is effective in min-max optimization and minimizing projected Bellman error. Furthermore, in the deep reinforcement learning case, we propose a novel variant of TD(0) which is more compute and sample efficient.

2025-01-22

ICLR.cc/2025/Conference (poster)

doi.org

openreview.net

Tight Lower Bounds and Improved Convergence in Performative Prediction

Pedram J. Khorsandi

Rushil Gupta

Mehrnaz Mofakhami

Simon Lacoste-Julien

Gauthier Gidel

Performative prediction is a framework accounting for the shift in the data distribution induced by the prediction of a model deployed in th… (voir plus)e real world. Ensuring rapid convergence to a stable solution where the data distribution remains the same after the model deployment is crucial, especially in evolving environments. This paper extends the Repeated Risk Minimization (RRM) framework by utilizing historical datasets from previous retraining snapshots, yielding a class of algorithms that we call Affine Risk Minimizers and enabling convergence to a performatively stable point for a broader class of problems. We introduce a new upper bound for methods that use only the final iteration of the dataset and prove for the first time the tightness of both this new bound and the previous existing bounds within the same regime. We also prove that utilizing historical datasets can surpass the lower bound for last iterate RRM, and empirically observe faster convergence to the stable point on various performative prediction benchmarks. We offer at the same time the first lower bound analysis for RRM within the class of Affine Risk Minimizers, quantifying the potential improvements in convergence speed that could be achieved with other variants in our framework.

2024-12-04

ArXiv (prépublication)

arxiv.org

Tight Lower Bounds and Improved Convergence in Performative Prediction

Pedram J. Khorsandi

Rushil Gupta

Mehrnaz Mofakhami

Simon Lacoste-Julien

Gauthier Gidel

Performative prediction is a framework accounting for the shift in the data distribution induced by the prediction of a model deployed in th… (voir plus)e real world. Ensuring rapid convergence to a stable solution where the data distribution remains the same after the model deployment is crucial, especially in evolving environments. This paper extends the Repeated Risk Minimization (RRM) framework by utilizing historical datasets from previous retraining snapshots, yielding a class of algorithms that we call Affine Risk Minimizers and enabling convergence to a performatively stable point for a broader class of problems. We introduce a new upper bound for methods that use only the final iteration of the dataset and prove for the first time the tightness of both this new bound and the previous existing bounds within the same regime. We also prove that utilizing historical datasets can surpass the lower bound for last iterate RRM, and empirically observe faster convergence to the stable point on various performative prediction benchmarks. We offer at the same time the first lower bound analysis for RRM within the class of Affine Risk Minimizers, quantifying the potential improvements in convergence speed that could be achieved with other variants in our framework.

2024-12-04

ArXiv (prépublication)

doi.org

arxiv.org

Solving Hidden Monotone Variational Inequalities with Surrogate Losses

Ryan D'Orazio

Danilo Vucetic

Zichu Liu

Junhyung Lyle Kim

Ioannis Mitliagkas

Gauthier Gidel

Deep learning has proven to be effective in a wide variety of loss minimization problems. However, many applications of interest, like minim… (voir plus)izing projected Bellman error and min-max optimization, cannot be modelled as minimizing a scalar loss function but instead correspond to solving a variational inequality (VI) problem. This difference in setting has caused many practical challenges as naive gradient-based approaches from supervised learning tend to diverge and cycle in the VI case. In this work, we propose a principled surrogate-based approach compatible with deep learning to solve VIs. We show that our surrogate-based approach has three main benefits: (1) under assumptions that are realistic in practice (when hidden monotone structure is present, interpolation, and sufficient optimization of the surrogates), it guarantees convergence, (2) it provides a unifying perspective of existing methods, and (3) is amenable to existing deep learning optimizers like ADAM. Experimentally, we demonstrate our surrogate-based approach is effective in min-max optimization and minimizing projected Bellman error. Furthermore, in the deep reinforcement learning case, we propose a novel variant of TD(0) which is more compute and sample efficient.

2024-11-07

ArXiv (prépublication)

doi.org

arxiv.org

General Causal Imputation via Synthetic Interventions

Marco Jiralerspong

Thomas Jiralerspong

Vedant Shah

Dhanya Sridhar

Gauthier Gidel

Given two sets of elements (such as cell types and drug compounds), researchers typically only have access to a limited subset of their inte… (voir plus)ractions. The task of causal imputation involves using this subset to predict unobserved interactions. Squires et al. (2022) have proposed two estimators for this task based on the synthetic interventions (SI) estimator: SI-A (for actions) and SI-C (for contexts). We extend their work and introduce a novel causal imputation estimator, generalized synthetic interventions (GSI). We prove the identifiability of this estimator for data generated from a more complex latent factor model. On synthetic and real data we show empirically that it recovers or outperforms their estimators.

2024-10-30

NeurIPS.cc/2024/Workshop/CRL (poster)

doi.org

openreview.net

À la hauteur du moment

Mila Techaide 2025

Développement du groupe d'experts de l'ONU sur l'IA

Perspectives sur l’IA pour les responsables des politiques

À la hauteur du moment

Mila Techaide 2025

Gauthier Gidel

Biographie

Étudiants actuels

Billets de blogue

Publications

À la hauteur du moment

Mila Techaide 2025

Développement du groupe d'experts de l'ONU sur l'IA

Perspectives sur l’IA pour les responsables des politiques

À la hauteur du moment

Mila Techaide 2025

Mots-clés populaires:

Gauthier Gidel

Biographie

Étudiants actuels

Billets de blogue

Publications