Jonathan Colaço Carr

Maîtrise recherche - McGill

Superviseur⋅e principal⋅e

Prakash Panangaden

Co-supervisor

Doina Precup

Sujets de recherche

Apprentissage par renforcement

Théorie de l'apprentissage automatique

Traitement du langage naturel

Publications

Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

Yash More

In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards o… (voir plus)utputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

2024-11-12

ArXiv (prépublication)

doi.org

arxiv.org

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies

Jonathan Colaço Carr

Prakash Panangaden

Doina Precup

2024-04-18

Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (publié)

doi.org

arxiv.org