Dorsaf Sallami

dorsaf.sallami@mila.quebec

Independent visiting researcher - McGill University

Supervisor

Reihaneh Rabbany

Website

Google Scholar

Publications

Ambiguity-Aware Evaluation for Automated Fact-Checking

Sukanya Krishna

Dorsaf Sallami

Jean-François Godbout

Reihaneh Rabbany

Maimuna S. Majumder

Finale Doshi-Velez

Automated fact-checking systems are evaluated almost entirely by whether their predicted verdict matches a single gold label, an approach th… (see more)at assumes every claim has one stable interpretation and that the available evidence is always sufficient to resolve it. In practice, claims are frequently underspecified in time, place, or meaning, and retrieved evidence is often incomplete or contradictory, so this approach cannot distinguish a system that reasoned well on a genuinely ambiguous input from one that reasoned poorly but matched the label anyway. We introduce a framework that separates ambiguity in the claim and evidence from failures in the system's own reasoning and grounding, and conditions accuracy and response appropriateness on this ambiguity profile. Applied to a retrieval-based fact-checking pipeline across five benchmarks, plus a human pilot study, we find that ambiguity is pervasive and dominated by missing timeframes, that most correct verdicts on several benchmarks are backed by reasoning that would not hold up under scrutiny, and that systems grow more confident, not less, when their evidence conflicts.

2026-09-29

WiML @ Neural Information Processing Systems (published)

openreview.net