Portrait de Tommaso Tosato

Tommaso Tosato

Postdoctorat - UdeM
Superviseur⋅e principal⋅e
Guillaume Dumas
Sujets de recherche
NeuroIA
Traitement du langage naturel
Site web
Google Scholar
GitHub

Publications

Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
Tommaso Tosato
Saskia Helbling
Yorguin-Jose Mantilla-Ramos
Mahmood Hegazy
Alberto Tosato
D. Lemay
Irina Rish
Guillaume Dumas
2025-08-06
ArXiv (prépublication)
doi.org
arxiv.org
Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
Tommaso Tosato
Saskia Helbling
Yorguin-Jose Mantilla-Ramos
Mahmood Hegazy
Alberto Tosato
D. Lemay
Irina Rish
Guillaume Dumas
Large language models require consistent behavioral patterns for safe deployment, yet their personality-like traits remain poorly understood… (voir plus). We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25+ open-source models (1B-671B parameters) across 500,000+ responses. Using traditional (BFI-44, SD3) and novel LLM-adapted personality instruments, we systematically vary question order, paraphrasing, personas, and reasoning modes. Our findings challenge fundamental deployment assumptions: (1) Even 400B+ models exhibit substantial response variability (SD>0.4); (2) Minor prompt reordering alone shifts personality measurements by up to 20%; (3) Interventions expected to stabilize behavior, such as chain-of-thought reasoning, detailed personas instruction, inclusion of conversation history, can paradoxically increase variability; (4) LLM-adapted instruments show equal instability to human-centric versions, confirming architectural rather than translational limitations. This persistent instability across scales and mitigation strategies suggests current LLMs lack the foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that personality-based alignment strategies may be fundamentally inadequate.
2025-08-06
ArXiv (prépublication)
doi.org
arxiv.org
Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
Tommaso Tosato
Saskia Helbling
Yorguin-Jose Mantilla-Ramos
Mahmood Hegazy
Alberto Tosato
D. Lemay
Irina Rish
Guillaume Dumas
Large language models require consistent behavioral patterns for safe deployment, yet their personality-like traits remain poorly understood… (voir plus). We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25+ open-source models (1B-671B parameters) across 500,000+ responses. Using traditional (BFI-44, SD3) and novel LLM-adapted personality instruments, we systematically vary question order, paraphrasing, personas, and reasoning modes. Our findings challenge fundamental deployment assumptions: (1) Even 400B+ models exhibit substantial response variability (SD>0.4); (2) Minor prompt reordering alone shifts personality measurements by up to 20%; (3) Interventions expected to stabilize behavior, such as chain-of-thought reasoning, detailed personas instruction, inclusion of conversation history, can paradoxically increase variability; (4) LLM-adapted instruments show equal instability to human-centric versions, confirming architectural rather than translational limitations. This persistent instability across scales and mitigation strategies suggests current LLMs lack the foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that personality-based alignment strategies may be fundamentally inadequate.
2025-08-06
ArXiv (prépublication)
arxiv.org
arxiv.org