Publications

Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?

Salah Zaiem

Youcef Kemiche

Titouan Parcollet

Slim Essid

Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on… (see more) speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.

2023-08-20

INTERSPEECH 2023 (published)

doi.org

arxiv.org

Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi

Hadi Nekoei

Xutong Zhao

Janarthanan Rajendran

Miao Liu

Sarath Chandar

Cooperative Multi-agent Reinforcement Learning (MARL) algorithms with Zero-Shot Coordination (ZSC) have gained significant attention in rece… (see more)nt years. ZSC refers to the ability of agents to coordinate zero-shot (without additional interaction experience) with independently trained agents. While ZSC is crucial for cooperative MARL agents, it might not be possible for complex tasks and changing environments. Agents also need to adapt and improve their performance with minimal interaction with other agents. In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners. To investigate this issue, we formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods. In particular, we created a diverse set of pre-trained agents and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance. After evaluating several SOTA algorithms using our framework, our experiments reveal that naive Independent Q-Learning (IQL) agents in most cases adapt as quickly as the SOTA ZSC algorithm Off-Belief Learning (OBL). This finding raises an interesting research question: How to design MARL algorithms with high ZSC performance and capability of fast adaptation to unseen partners. As a first step, we studied the role of different hyper-parameters and design choices on the adaptability of current MARL algorithms. Our experiments show that two categories of hyper-parameters controlling the training data diversity and optimization process have a significant impact on the adaptability of Hanabi agents.

2023-08-20

ArXiv (preprint)

doi.org

arxiv.org

MARCO: A Memory-Augmented Reinforcement Framework for Combinatorial Optimization

Andoni I. Garmendia

Quentin Cappart

Josu Ceberio

Alexander Mendiburu

2023-08-19

Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence (published)

doi.org

arxiv.org

Open, Closed, or Small Language Models for Text Classification?

Hao Yu

Zachary Yang

Kellin Pelrine

Jean-François Godbout

Reihaneh Rabbany

Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, … (see more)including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements

2023-08-19

ArXiv (preprint)

doi.org

arxiv.org

Pontomedullary junction as a reference for spinal cord cross-sectional area: validation across neck positions

Sandrine Bédard

Maxime Bouthillier

Julien Cohen-Adad

2023-08-19

Scientific Reports (published)

doi.org

GTM-decon: guided-topic modeling of single-cell transcriptomes enables sub-cell-type and disease-subtype deconvolution of bulk transcriptomes

Lakshmipuram Seshadri Swapna

Michael Huang

Yue Li

2023-08-18

Genome Biology (published)

doi.org

YORC: Yoruba Reading Comprehension dataset

Aremu Anuoluwapo

Jesujoba Oluwadara Alabi

David Ifeoluwa Adelani

In this paper, we create YORC: a new multi-choice Yoruba Reading Comprehension dataset that is based on Yoruba high-school reading comprehen… (see more)sion examination. We provide baseline results by performing cross-lingual transfer using existing English RACE dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.

2023-08-18

ArXiv (preprint)

doi.org

arxiv.org

Age-related bias and artificial intelligence: a scoping review

Charlene H Chu

Simon Donato-Woodger

Shehroz S Khan

Rune Nyrup

Kathleen Leslie

Alexandra Lyn

Tianyu Shi