Publications

Multilingual Hallucination Gaps in Large Language Models
Cl'ea Chataigner
Afaf Taïk
Golnoosh Farnadi
Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that … (see more)resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading or false information that appears highly credible. In this study, we explore the phenomenon of hallucinations across multiple languages in freeform text generation, focusing on what we call multilingual hallucination gaps. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. To quantify such hallucinations, we used the FactScore metric and extended its framework to a multilingual setting. We conducted experiments using LLMs from the LLaMA, Qwen, and Aya families, generating biographies in 19 languages and comparing the results to Wikipedia pages. Our results reveal variations in hallucination rates, especially between high and low resource languages, raising important questions about LLM multilingual performance and the challenges in evaluating hallucinations in multilingual freeform text generation.
2024-10-23
ArXiv (preprint)
arxiv.org
arxiv.org
Overcoming State and Action Space Disparities in Multi-Domain, Multi-Task Reinforcement Learning
Reginald McLean
Kai Yuan
Isaac Woungang
Nariman Farsad
Pablo Samuel Castro
Current multi-task reinforcement learning (MTRL) methods have the ability to perform a large number of tasks with a single policy. However w… (see more)hen attempting to interact with a new domain, the MTRL agent would need to be re-trained due to differences in domain dynamics and structure. Because of these limitations, we are forced to train multiple policies even though tasks may have shared dynamics, leading to needing more samples and is thus sample inefficient. In this work, we explore the ability of MTRL agents to learn in various domains with various dynamics by simultaneously learning in multiple domains, without the need to fine-tune extra policies. In doing so we find that a MTRL agent trained in multiple domains induces an increase in sample efficiency of up to 70\% while maintaining the overall success rate of the MTRL agent.
2024-10-23
corl.org/2024/Workshop/MAPoDeL (published)
openreview.net
openreview.net
Stick-breaking Attention
Shawn Tan
Yikang Shen
Songlin Yang
Aaron Courville
Rameswar Panda
2024-10-23
ArXiv (preprint)
arxiv.org
arxiv.org
FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation
Rohan Sukumaran
Aarash Feizi
Adriana Romero-Sorian
Golnoosh Farnadi
2024-10-22
ArXiv (preprint)
arxiv.org
arxiv.org
Fine-Tuning Web Agents: It Works, But It's Trickier Than You Think
Massimo Caccia
Megh Thakkar
Léo Boisvert
Thibault Le Sellier de Chezelles
Alexandre Piché
Nicolas Chapados
Alexandre Drouin
Maxime Gasse
Alexandre Lacoste
Recent advancements in large language models (LLMs) have sparked interest in developing autonomous web agents capable of performing digital … (see more)tasks through web interfaces in a human-like manner. However, even the strongest closed-source models often struggle to achieve robust results on several benchmarks, while a notable performance gap exists between them and open-source counterparts. This study investigates the potential of fine-tuning to enhance the performance of a smaller, lower-performing but cost-efficient LLM by leveraging successful traces from stronger LLMs, referred to as experts. We outline a comprehensive pipeline for data collection, filtering, and supervised fine-tuning and explore various behavior cloning parameters. Our experiments provide key insights into the challenges of fine-tuning LLMs into web agents on benchmarks like MiniWoB and WorkArena. Notably, we find that the fine-tuned agents' ability to predict expert trajectories does not consistently lead to improved downstream task performance. This raises issues such as off-policy bias and the loss of reasoning abilities during fine-tuning. We discuss potential solutions to these challenges and make both the codebase and a dataset of 140M tokens open-source for the community to build upon.
2024-10-22
NeurIPS.cc/2024/Workshop/OWA (poster)
openreview.net
openreview.net
Graph Knowledge Distillation to Mixture of Experts
Pavel Rumiantsev
Mark Coates
2024-10-22
TMLR (accepted)
doi.org
openreview.net
Health satisfaction outcome from integrated autonomous mobile clinics
Yuzhang Huang
Shaoshan Liu
Zhongying Pan
Carl Wu
Herng-Chia Chiu
Xue (Steve) Liu
Leiyu Shi
2024-10-22
Scientific Reports (published)
doi.org
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Jerry Huang
Prasanna Parthasarathi
Mehdi Rezagholizadeh
Boxing Chen
Sarath Chandar
The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some … (see more)of this is also owed to the risks and costs associated with their use. On one front is their tendency to \textit{hallucinate} false or misleading information, limiting their reliability. On another is the increasing focus on the computational limitations associated with traditional self-attention based LLMs, which has brought about new alternatives, in particular recurrent models, meant to overcome them. Yet it remains uncommon to consider these two concerns simultaneously. Do changes in architecture exacerbate/alleviate existing concerns about hallucinations? Do they affect how and where they occur? Through an extensive evaluation, we study how these architecture-based inductive biases affect the propensity to hallucinate. While hallucination remains a general phenomenon not limited to specific architectures, the situations in which they occur and the ease with which specific types of hallucinations can be induced can significantly differ based on the model architecture. These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.
2024-10-22
ArXiv (preprint)
arxiv.org
arxiv.org
Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases
Cristian Meo
Akihiro Nakano
Mircea Licua
Aniket Rajiv Didolkar
Masahiro Suzuki
Anirudh Goyal
Mengmi Zhang
Justin Dauwels
Yutaka Matsuo
Yoshua Bengio
2024-10-21
ArXiv (preprint)
arxiv.org
arxiv.org
Generating Tabular Data Using Heterogeneous Sequential Feature Forest Flow Matching
Ange-Cl'ement Akazan
Alexia Jolicoeur-Martineau
Ioannis Mitliagkas
2024-10-20
ArXiv (preprint)
arxiv.org
arxiv.org
Action abstractions for amortized sampling
Oussama Boussif
Lena Nehale Ezzine
Joseph D. Viviano
Michał Koziarski
Moksh J. Jain
Nikolay Malkin
Emmanuel Bengio
Rim Assouel
Yoshua Bengio
2024-10-19
ArXiv (preprint)
arxiv.org
arxiv.org
Beyond Causal Discovery for Astronomy: Learning Meaningful Representations with Independent Component Analysis
Zehao Jin
Mario Pasquato
Benjamin L. Davis
A. Macciò
Yashar Hezaveh
2024-10-18
ArXiv (preprint)
arxiv.org
arxiv.org