Jean-François Godbout

Kellin Pelrine

2024-01-13

ArXiv (prépublication)

Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigation

Tyler Vergho

Jean-François Godbout

Kellin Pelrine

Recent large language models (LLMs) have been shown to be effective for misinformation detection. However, the choice of LLMs for experiment… (voir plus)s varies widely, leading to uncertain conclusions. In particular, GPT-4 is known to be strong in this domain, but it is closed source, potentially expensive, and can show instability between different versions. Meanwhile, alternative LLMs have given mixed results. In this work, we show that Zephyr-7b presents a consistently viable alternative, overcoming key limitations of commonly used approaches like Llama-2 and GPT-3.5. This provides the research community with a solid open-source option and shows open-source models are gradually catching up on this task. We then highlight how GPT-3.5 exhibits unstable performance, such that this very widely used model could provide misleading results in misinformation detection. Finally, we validate new tools including approaches to structured output and the latest version of GPT-4 (Turbo), showing they do not compromise performance, thus unlocking them for future research and potentially enabling more complex pipelines for misinformation mitigation.

2024-01-12

ArXiv (prépublication)

Uncertainty Resolution in Misinformation Detection

Yury Orlovskiy

Camille Thibault

Anne Imouza

Jean-François Godbout

Kellin Pelrine

2024-01-02

ArXiv (prépublication)

Quantifying learning-style adaptation in effectiveness of LLM teaching

Ruben Weijers

Gabrielle Fidelis de Castilho

Jean-François Godbout

Kellin Pelrine

This preliminary study aims to investigate whether AI, when prompted based on individual learning styles, can effectively improve comprehens… (voir plus)ion and learning experiences in educational settings. It involves tailoring LLMs baseline prompts and comparing the results of a control group receiving standard content and an experimental group receiving learning style-tailored content. Preliminary results suggest that GPT-4 can generate responses aligned with various learning styles, indicating the potential for enhanced engagement and comprehension. However, these results also reveal challenges, including the model’s tendency for sycophantic behavior and variability in responses. Our findings suggest that a more sophisticated prompt engineering approach is required for integrating AI into education (AIEd) to improve educational outcomes.

2024-01-01

PERSONALIZE (publié)

www.semanticscholar.org

Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4

Kellin Pelrine

Anne Imouza

Meilina Reksoprodjo

Camille Thibault

Caleb Gupta

Joel Christoph

Jean-François Godbout

Misinformation poses a critical societal challenge, and current approaches have yet to produce an effective solution. We propose focusing on… (voir plus) generalization, uncertainty, and how to leverage recent large language models, in order to create more practical tools to evaluate information veracity in contexts where perfect classification is impossible. We first demonstrate that GPT-4 can outperform prior methods in multiple settings and languages. Next, we explore generalization, revealing that GPT-4 and RoBERTa-large exhibit differences in failure modes. Third, we propose techniques to handle uncertainty that can detect impossible examples and strongly improve outcomes. We also discuss results on other language models, temperature, prompting, versioning, explainability, and web retrieval, each one providing practical insights and directions for future research. Finally, we publish the LIAR-New dataset with novel paired English and French misinformation data and Possibility labels that indicate if there is sufficient context for veracity evaluation. Overall, this research lays the groundwork for future tools that can drive real-world progress to combat misinformation.

2023-10-07

EMNLP/2023/Conference (accepté)

openreview.net

Party Prediction for Twitter

Kellin Pelrine

Anne Imouza

Zachary Yang

Jacob-Junqi Tian

Sacha Lévy

Gabrielle Desrosiers-Brisebois

Aarash Feizi

C'ecile Amadoro

André Blais

Jean-François Godbout

2023-08-25

ArXiv (prépublication)

Open, Closed, or Small Language Models for Text Classification?

Hao Yu

Zachary Yang

Kellin Pelrine

Jean-François Godbout

Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, … (voir plus)including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements

2023-08-19

ArXiv (prépublication)

Online Partisan Polarization of COVID-19

Zachary Yang

Anne Imouza

Kellin Pelrine

Sacha Lévy

Jiewen Liu

Gabrielle Desrosiers-Brisebois

Jean-François Godbout

André Blais

In today’s age of (mis)information, many people utilize various social media platforms in an attempt to shape public opinion on several im… (voir plus)portant issues, including elections and the COVID-19 pandemic. These two topics have recently become intertwined given the importance of complying with public health measures related to COVID-19 and politicians’ management of the pandemic. Motivated by this, we study the partisan polarization of COVID-19 discussions on social media. We propose and utilize a novel measure of partisan polarization to analyze more than 380 million posts from Twitter and Parler around the 2020 US presidential election. We find strong correlation between peaks in polarization and polarizing events, such as the January 6th Capitol Hill riot. We further classify each post into key COVID-19 issues of lockdown, masks, vaccines, as well as miscellaneous, to investigate both the volume and polarization on these topics and how they vary through time. Parler includes more negative discussions around lockdown and masks, as expected, but not much around vaccines. We also observe more balanced discussions on Twitter and a general disconnect between the discussions on Parler and Twitter.

2021-12-01

2021 International Conference on Data Mining Workshops (ICDMW) (publié)