Jackie Cheung

Ines Arous

Collaborateur·rice alumni - McGill

Doctorat - McGill

Collaborateur·rice de recherche

Nadhem Benhadjali

Collaborateur·rice de recherche

Meng (Caden) Cao

Collaborateur·rice alumni - McGill

Doctorat - McGill

Doctorat - McGill

Superviseur⋅e principal⋅e :

Maîtrise recherche - McGill

Site web

Nelson Filipe Costa

Collaborateur·rice de recherche - Concordia University

Maxime Darrin

Doctorat - McGill

Co-superviseur⋅e :

Doctorat - McGill

Aylin Erman

Doctorat - McGill

Co-superviseur⋅e :

Dan Poenaru

Ori Ernst

Postdoctorat - McGill

Maîtrise recherche - McGill

Site web

Google Scholar

Zichao Li

Doctorat - McGill

Superviseur⋅e principal⋅e :

Siva Reddy

Caleb Moses

Doctorat - McGill

Ian Porada

Doctorat - McGill

Doctorat - McGill

Baccalauréat - McGill

Cesare Spinoso-Di Piano

Doctorat - McGill

Sihui Wei

Baccalauréat - McGill

Xiyuan Zou

Maîtrise recherche - McGill

Publications

Systematic Rectification of Language Models via Dead-end Analysis

Meng Cao

Mehdi Fatemi

Samira Shabanian

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to re… (voir plus)duce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. This is crucial as many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach significantly improves the generated discourse compared to the base LLMs and other techniques in terms of both the overall language and detoxification performance.

2023-02-01

ICLR.cc/2023/Conference (poster)

openreview.net

Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness

The potential of using a large language model (LLM) as a knowledge base (KB) has sparked significant interest. To maintain the knowledge acq… (voir plus)uired by LLMs, we need to ensure that the editing of learned facts respects internal logical constraints, which are known as dependency of knowledge. Existing work on editing LLMs has partially addressed the issue of dependency, when the editing of a fact should apply to its lexical variations without disrupting irrelevant ones. However, they neglect the dependency between a fact and its logical implications. We propose an evaluation protocol with an accompanying question-answering dataset, StandUp, that provides a comprehensive assessment of the editing process considering the above notions of dependency. Our protocol involves setting up a controlled environment in which we edit facts and monitor their impact on LLMs, along with their implications based on If-Then rules. Extensive experiments on StandUp show that existing knowledge editing methods are sensitive to the surface form of knowledge, and that they have limited performance in inferring the implications of edited facts.

2023-01-01

EMNLP (Findings) (publié)

openreview.net

How Useful Are Educational Questions Generated by Large Language Models?

Sabina Elkins

Ekaterina Kochmar

Iulian V. Serban

2023-01-01

AIED (Posters/Late Breaking Results/...) (publié)

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

Yu Lu Liu

Meng Cao

Su Lin Blodgett

Alexandra Olteanu

Adam Trischler

2023-01-01

EMNLP (Findings) (publié)

openreview.net

The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources

Akshatha Arodi

Martin Pömsl

Kaheer Suleman

Adam Trischler

Alexandra Olteanu

Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models. These models often make in… (voir plus)ferences using information from multiple sources. An important class of such inferences are those that require both background knowledge, presumably contained in a model’s pretrained parameters, and instance-specific information that is supplied at inference time. However, the integration and reasoning abilities of NLU models in the presence of multiple knowledge sources have been largely understudied. In this work, we propose a test suite of coreference resolution subtasks that require reasoning over multiple facts. These subtasks differ in terms of which knowledge sources contain the relevant facts. We also introduce subtasks where knowledge is present only at inference time using fictional knowledge. We evaluate state-of-the-art coreference resolution models on our dataset. Our results indicate that several models struggle to reason on-the-fly over knowledge observed both at pretrain time and at inference time. However, with task-specific training, a subset of models demonstrates the ability to integrate certain knowledge types from multiple sources. Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.

2023-01-01

ACL (1) (publié)

A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques

Malik H. Altakrori

Thomas Scialom

Benjamin Fung

2022-12-01

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (publié)

Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge

Ian Porada

Alessandro Sordoni

2022-07-01

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (publié)

MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification

Yu Lu Liu

Rachel Bawden

Thomas Scaliom

Benoı̂t Sagot

2022-05-24

ArXiv (prépublication)

Characterizing Idioms: Conventionality and Contingency

Michaela Socolof

Michael Wagner

Timothy O'Donnell

Idioms are unlike most phrases in two important ways. First, words in an idiom have non-canonical meanings. Second, the non-canonical meanin… (voir plus)gs of words in an idiom are contingent on the presence of other words in the idiom. Linguistic theories differ on whether these properties depend on one another, as well as whether special theoretical machinery is needed to accommodate idioms. We define two measures that correspond to the properties above, and we show that idioms fall at the expected intersection of the two dimensions, but that the dimensions themselves are not correlated. Our results suggest that introducing special machinery to handle idioms may not be warranted.

2022-05-01

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (publié)

Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization

Meng Cao

Yue Dong

2022-05-01

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (publié)

Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment

Prakhar Sharma

2022-05-01

Findings of the Association for Computational Linguistics: ACL 2022 (publié)

Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment

Prakhar Sharma

Most research on question answering focuses on the pre-deployment stage; i.e., building an accurate model for deployment.In this paper, we a… (voir plus)sk the question: Can we improve QA systems further post-deployment based on user interactions? We focus on two kinds of improvements: 1) improving the QA system’s performance itself, and 2) providing the model with the ability to explain the correctness or incorrectness of an answer.We collect a retrieval-based QA dataset, FeedbackQA, which contains interactive feedback from users. We collect this dataset by deploying a base QA system to crowdworkers who then engage with the system and provide feedback on the quality of its answers.The feedback contains both structured ratings and unstructured natural language explanations.We train a neural model with this feedback data that can generate explanations and re-score answer candidates. We show that feedback data not only improves the accuracy of the deployed QA system but also other stronger non-deployed systems. The generated explanations also help users make informed decisions about the correctness of answers.

2022-04-06

ArXiv (preprint)