Andreea Musulan

Deepfakes, synthetic media created using advanced AI techniques, have intensified the spread of misinformation, particularly in politically … (see more)sensitive contexts. Existing deepfake detection datasets are often limited, relying on outdated generation methods, low realism, or single-face imagery, restricting the effectiveness for general synthetic image detection. By analyzing social media posts, we identify multiple modalities through which deepfakes propagate misinformation. Furthermore, our human perception study demonstrates that recently developed proprietary models produce synthetic images increasingly indistinguishable from real ones, complicating accurate identification by the general public. Consequently, we present a comprehensive, politically-focused dataset specifically crafted for benchmarking detection against modern generative models. This dataset contains three million real images paired with descriptive captions, which are used for generating 963k corresponding high-quality synthetic images from a mix of proprietary and open-source models. Recognizing the continual evolution of generative techniques, we introduce an innovative crowdsourced adversarial platform, where participants are incentivized to generate and submit challenging synthetic images. This ongoing community-driven initiative ensures that deepfake detection methods remain robust and adaptive, proactively safeguarding public discourse from sophisticated misinformation threats.

2025-09-11

ArXiv (preprint)

Rethinking Anti-Misinformation AI

Vidya Sujaya

Kellin Pelrine

This paper takes a position on how anti-misinformation AI works should be developed for the online misinformation context. We observe that t… (see more)he current literature is dominated by works that produce more information for users to process and that this function faces various challenges in bringing meaningful effects to reality. We use anti-misinformation insights from other domains to suggest a redirection of the existing line of work and identify an under-explored opportunity AI can facilitate exploring.

2025-03-05

ICLR.cc/2025/Workshop/HAIC (published)

Rethinking Anti-Misinformation AI

Vidya Sujaya

Kellin Pelrine

This paper takes a position on how anti-misinformation AI works should be developed for the online misinformation context. We observe that t… (see more)he current literature is dominated by works that produce more information for users to process and that this function faces various challenges in bringing meaningful effects to reality. We use anti-misinformation insights from other domains to suggest a redirection of the existing line of work and identify an under-explored opportunity AI can facilitate exploring.

2025-03-05

ICLR.cc/2025/Workshop/HAIC (published)

Online Influence Campaigns: Strategies and Vulnerabilities

Veronica Xia

Ethan Kosak-Hine

Tom Gibbs

Vidya Sujaya

Kellin Pelrine

U. Montr'eal

Ivado

M. University

In order to combat the creation and spread of harmful content online, this paper defines and contextualizes the concept of inauthentic, soci… (see more)etal-scale manipulation by malicious actors. We review the literature on societally harmful content and how it proliferates to analyze the manipulation strategies used by such actors and the vulnerabilities they target. We also provide an overview of three case studies of extensive manipulation campaigns to emphasize the severity of the problem. We then address the role that Artificial Intelligence plays in the development and dissemination of harmful content, and how its evolution presents new threats to societal cohesion for countries across the globe. Our survey aims to increase our understanding of not just particular aspects of these threats, but also the strategies underlying their deployment, so we can effectively prepare for the evolving cybersecurity landscape.

2024-12-18

ArXiv (preprint)

doi.org

A Simulation System Towards Solving Societal-Scale Manipulation

Maximilian Puelma Touzel

Austin Welch

Gayatri Krishnakumar

Dan Zhao

Hao Yu

Ethan Kosak-Hine

Tom Gibbs

Busra Tugce Gurbuz

Kellin Pelrine

The rise of AI-driven manipulation poses significant risks to societal trust and democratic processes. Yet, studying these effects in real-w… (see more)orld settings at scale is ethically and logistically impractical, highlighting a need for simulation tools that can model these dynamics in controlled settings to enable experimentation with possible defenses. We present a simulation environment designed to address this. We elaborate upon the Concordia framework that simulates offline, `real life' activity by adding online interactions to the simulation through social media with the integration of a Mastodon server. We improve simulation efficiency and information flow, and add a set of measurement tools, particularly longitudinal surveys. We demonstrate the simulator with a tailored example in which we track agents' political positions and show how partisan manipulation of agents can affect election results.

2024-10-17

ArXiv (preprint)

doi.org

A Simulation System Towards Solving Societal-Scale Manipulation

Maximilian Puelma Touzel

Austin Welch

Gayatri K

Dan Zhao

Hao Yu

Ethan Kosak-Hine

Tom Gibbs

Busra Tugce Gurbuz

Kellin Pelrine

The rise of AI-driven manipulation poses significant risks to societal trust and democratic processes. Yet, studying these effects in real-w… (see more)orld settings at scale is ethically and logistically impractical, highlighting a need for simulation tools that can model these dynamics in controlled settings to enable experimentation with possible defenses. We present a simulation environment designed to address this. We elaborate upon the Concordia framework that simulates offline, `real life' activity by adding online interactions to the simulation through social media with the integration of a Mastodon server. We improve simulation efficiency and information flow, and add a set of measurement tools, particularly longitudinal surveys. We demonstrate the simulator with a tailored example in which we track agents' political positions and show how partisan manipulation of agents can affect election results.

2024-10-17

ArXiv (preprint)

doi.org

Simulation System Towards Solving Societal-Scale Manipulation

Maximilian Puelma Touzel

Austin Welch

Gayatri K

Dan Zhao

Hao Yu

Tom Gibbs

Ethan Kosak-Hine

Busra Tugce Gurbuz

Kellin Pelrine

The rise of AI-driven manipulation poses significant risks to societal trust and democratic processes. Yet, studying these effects in real-w… (see more)orld settings at scale is ethically and logistically impractical, highlighting a need for simulation tools that can model these dynamics in controlled settings to enable experimentation with possible defenses. We present a simulation environment designed to address this. We elaborate upon the Concordia framework that simulates offline, `real life' activity by adding online interactions to the simulation through social media with the integration of a Mastodon server. Through a variety of means we then improve simulation efficiency and information flow, and add a set of measurement tools, particularly longitudinal surveys of the agents' political positions. We demonstrate the simulator with a tailored example of how partisan manipulation of agents can affect election results.

2024-10-12

NeurIPS.cc/2024/Workshop/SafeGenAi (poster)

Simulation System Towards Solving Societal-Scale Manipulation

Maximilian Puelma Touzel

Austin Welch

Gayatri K

Dan Zhao

Hao Yu

Tom Gibbs

Ethan Kosak-Hine

Busra Tugce Gurbuz

Kellin Pelrine

The rise of AI-driven manipulation poses significant risks to societal trust and democratic processes. Yet, studying these effects in real-w… (see more)orld settings at scale is ethically and logistically impractical, highlighting a need for simulation tools that can model these dynamics in controlled settings to enable experimentation with possible defenses. We present a simulation environment designed to address this. We elaborate upon the Concordia framework that simulates offline, `real life' activity by adding online interactions to the simulation through social media with the integration of a Mastodon server. Through a variety of means we then improve simulation efficiency and information flow, and add a set of measurement tools, particularly longitudinal surveys of the agents' political positions. We demonstrate the simulator with a tailored example of how partisan manipulation of agents can affect election results.

2024-10-12

NeurIPS.cc/2024/Workshop/SafeGenAi (poster)