Florence Laflamme

A Guide to Misinformation Detection Data and Evaluation

Camille Thibault

Jacob-Junqi Tian

Gabrielle Péloquin-Skulski

Taylor Lynn Curtis

James Zhou

Florence Laflamme

Yuxiang Guan

Reihaneh Rabbany

Jean-François Godbout

Kellin Pelrine

Misinformation is a complex societal issue, and mitigating solutions are difficult to create due to data deficiencies. To address this probl… (see more)em, we have curated the largest collection of (mis)information datasets in the literature, totaling 75. From these, we evaluated the quality of all of the 36 datasets that consist of statements or claims, as well as the 9 datasets that consists of data in purely paragraph form. We assess these datasets to identify those with solid foundations for empirical work and those with flaws that could result in misleading and non-generalizable results, such as insufficient label quality, spurious correlations. We further provide state-of-the-art baselines on all these datasets, but show that regardless of label quality, categorical labels may no longer give an accurate evaluation of detection model performance. We discuss alternatives to mitigate this problem. Overall, this guide aims to provide a roadmap for obtaining higher quality data and conducting more effective evaluations, ultimately improving research in misinformation detection. All datasets and other artifacts are available at [anonymized].

2025-08-03

Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (published)

doi.org

openreview.net

A Guide to Misinformation Detection Data and Evaluation

Camille Thibault

Jacob-Junqi Tian

Gabrielle Péloquin-Skulski

Taylor Lynn Curtis

James Zhou

Florence Laflamme

Luke Yuxiang Guan