Portrait de Khaoula Chehbouni

Khaoula Chehbouni

kchehbouni@gmail.com
Doctorat - McGill
Superviseur⋅e principal⋅e
Golnoosh Farnadi
Co-supervisor
Jackie Cheung
Sujets de recherche
Équité
Équité algorithmique
Éthique de l'IA
IA responsable
Modèles génératifs
Sécurité de l'IA
Traitement du langage naturel
Vie privée
Site web
Google Scholar
GitHub

Publications

LLJ Cards: Best practices for the Use of LLMs as Judges
Khaoula Chehbouni
Melina Medjdoub
Florian Carichon
Golnoosh Farnadi
Jackie Chi Kit Cheung
In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJ… (voir plus)s), these systems have been widely adopted by researchers and practitioners across a broad range of measurement tasks, driven by their strong performance, scalability, and cost-effectiveness relative to human judgment. However, a growing body of work has shown that the use of LLJs raise concerns about their validity and reliability as evaluators. Existing efforts to address these challenges have largely focused on developing bias-mitigation techniques and refining prompting strategies. While these approaches represent an important step forward, they primarily offer technical fixes and leave a more fundamental challenge unaddressed: the lack of standardized, transparent, and reproducible evaluation practices. In this paper, we introduce LLJ Cards, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations. While LLJs offer a promising path toward scalable evaluation, their effective use requires grounding in rigorous evaluation principles to ensure validity, reliability, and reproducibility. LLJ Cards addresses this need by providing a structured framework for applying these principles in the design and reporting of automated evaluations.
2026-09-20
arXiv (prépublication)
doi.org
arxiv.org
From Infrastructure to Interface, the AI Value Chain Drives LLM Homogenization
Khaoula Chehbouni
Cléa Chataigner
Prakhar Ganesh
Pablo Piantanida
Jackie CK Cheung
Golnoosh Farnadi
2025-12-31
Neural Information Processing Systems (Accept)
openreview.net
openreview.net
Neither Valid Nor Reliable? Investigating the Use of LLMs as Judges
Khaoula Chehbouni
Mohammed Haddou
Jackie Chi Kit Cheung
Golnoosh Farnadi
2025-12-02
Neural Information Processing Systems (Accept)
doi.org
openreview.net
The effect of the typicality of song lyrics on song popularity: A natural language processing analysis of the British top singles chart
Khaoula Chehbouni
Florian Carichon
Adrien Simonnot-Lanciaux
Gilles Caporossi
Danilo C. Dantas
This study investigates the relationship between the lexical typicality of song lyrics and song popularity in the UK Official Singles Chart … (voir plus)from 1999 to 2013. Drawing on natural language processing (NLP) techniques, we develop a multidimensional measure of lyrics’ typicality that captures lexical repetition, complexity, thematic content, and emotional tone, addressing methodological limitations in previous research that relied heavily on static word dictionaries. By analysing 1,457 songs that reached the top five chart positions, we demonstrate that lyrics’ typicality significantly predicts the duration a song remains in the top five but does not affect the peak chart position or the trajectory of popularity (skewness and kurtosis). Our findings suggest that while typical lyrics may contribute to a song’s longevity within a given canon, they do not necessarily guarantee chart-topping success. This research contributes a replicable, dictionary-free methodology for assessing lyrics’ typicality and offers insights into the nuanced role of lyrics in shaping musical preferences.
2025-11-23
Psychology of Music (publié)
doi.org
Fairness in Federated Learning: Fairness for Whom?
Afaf Taïk
Khaoula Chehbouni
Golnoosh Farnadi
Fairness in federated learning has emerged as a rapidly growing area of research, with numerous works proposing formal definitions and algor… (voir plus)ithmic interventions. Yet, despite this technical progress, fairness in FL is often defined and evaluated in ways that abstract away from the sociotechnical contexts in which these systems are deployed. In this paper, we argue that existing approaches tend to optimize narrow system level metrics, such as performance parity or contribution-based rewards, while overlooking how harms arise throughout the FL lifecycle and how they impact diverse stakeholders. We support this claim through a critical analysis of the literature, based on a systematic annotation of papers for their fairness definitions, design decisions, evaluation practices, and motivating use cases. Our analysis reveals five recurring pitfalls: 1) fairness framed solely through the lens of server client architecture, 2) a mismatch between simulations and motivating use-cases and contexts, 3) definitions that conflate protecting the system with protecting its users, 4) interventions that target isolated stages of the lifecycle while neglecting upstream and downstream effects, 5) and a lack of multi-stakeholder alignment where multiple fairness definitions can be relevant at once. Building on these insights, we propose a harm centered framework that links fairness definitions to concrete risks and stakeholder vulnerabilities. We conclude with recommendations for more holistic, context-aware, and accountable fairness research in FL.
2025-10-14
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (publié)
doi.org
arxiv.org
Enhancing Privacy in the Early Detection of Sexual Predators Through Federated Learning and Differential Privacy
Khaoula Chehbouni
Martine De Cock
Gilles Caporossi
Afaf Taïk
Reihaneh Rabbany
Golnoosh Farnadi
The increased screen time and isolation caused by the COVID-19 pandemic have led to a significant surge in cases of online grooming, which i… (voir plus)s the use of strategies by predators to lure children into sexual exploitation. Previous efforts to detect grooming in industry and academia have involved accessing and monitoring private conversations through centrally-trained models or sending private conversations to a global server. In this work, we implement a privacy-preserving pipeline for the early detection of sexual predators. We leverage federated learning and differential privacy in order to create safer online spaces for children while respecting their privacy. We investigate various privacy-preserving implementations and discuss their benefits and shortcomings. Our extensive evaluation using real-world data proves that privacy and utility can coexist with only a slight reduction in utility.
2025-04-10
AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset
Khaoula Chehbouni
Jonathan Colaço Carr
Yash More
Jackie CK Cheung
Golnoosh Farnadi
In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards o… (voir plus)utputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.
2024-11-11
ArXiv (prépublication)
doi.org
arxiv.org
From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
Khaoula Chehbouni
Megha Roshan
Emmanuel Ma
Futian Andrew Wei
Afaf Taïk
Jackie CK Cheung
Golnoosh Farnadi
2023-12-31
Association for Computational Linguistics (publié)
doi.org
arxiv.org
Early Detection of Sexual Predators with Federated Learning
Khaoula Chehbouni
Gilles Caporossi
Reihaneh Rabbany
Martine De Cock
Golnoosh Farnadi
The rise in screen time and the isolation brought by the different containment measures implemented during the COVID-19 pandemic have led to… (voir plus) an alarming increase in cases of online grooming. Online grooming is defined as all the strategies used by predators to lure children into sexual exploitation. Previous attempts made in industry and academia on the detection of grooming rely on accessing and monitoring users’ private conversations through the training of a model centrally or by sending personal conversations to a global server. We introduce a first, privacy-preserving, cross-device, federated learning framework for the early detection of sexual predators, which aims to ensure a safe online environment for children while respecting their privacy.
2022-10-19
NeurIPS.cc/2022/Workshop/Federated_Learning (poster)
openreview.net
openreview.net