Mirco Ravanelli

heidari.allan@mila.quebec

Biography

Mirco Ravanelli is an assistant professor at Concordia University, adjunct professor at Université de Montréal and associate member of Mila – Quebec Artificial Intelligence Institute.

Ravanelli is an expert in deep learning and conversational AI, publishing over sixty papers in these fields. His contributions were honoured with a 2022 Amazon Research Award.

His research focuses primarily on novel deep learning algorithms, including self-supervised, continual, multimodal, cooperative and energy-efficient learning.

Formerly a postdoctoral fellow at Mila under Yoshua Bengio, he founded and now leads SpeechBrain, one of the most extensively used open-source toolkits in the field of speech processing and conversational AI.

Current Students

Allan Allan

Undergraduate - Concordia University

severin.baroudi@mila.quebec

Séverin Baroudi

Collaborating researcher - Toulon University

gaspard.botte@mila.quebec

Gaspard Botté

Research Intern - Concordia University

briac.cordelle@mila.quebec

Cordelle Briac

Master's Research - Concordia University University

Leo Brodeur

Master's Research - Concordia University

leobrod44@gmail.com

Matthieu Cervera

Collaborating researcher

Principal supervisor :

matthieu.cervera@mila.quebec

Victor Cruz

Master's Research - Concordia University

victor.cruz@mila.quebec

Luca Della Libera

PhD - Concordia University

Co-supervisor :

luca.dellalibera@mila.quebec

Wagner Drew

Master's Research - Concordia University

Co-supervisor :

Irina Rish

drew.wagner@mila.quebec

Gianfranco Dumoulin Bertucci

Master's Research - Concordia University

gianfranco.bertucci@mila.quebec

nadine.el-mufti@mila.quebec

Nadine El-Mufti

Master's Research - Concordia University

Maab Elrashid Ahmed Mohamed

PhD - Concordia University

Co-supervisor :

maab-elrashid.ahmed-mohamed@mila.quebec

francesco.bonzi@mila.quebec

Bonzi Francesco

PhD - Concordia University

salman.hussainali@mila.quebec

Salman Sami Hussain Ali

Master's Research - Concordia University University

sidra.intikhab@mila.quebec

Sidra Intikhab

Research Intern - Concordia University

Lovenya Jain

Collaborating researcher - Université Laval

Principal supervisor :

f20211732@pilani.bits-pilani.ac.in

Jihoon Jeong

PhD - Université Laval

Principal supervisor :

jihoon.jeong@mila.quebec

Haoyu Li

Professional Master's - Concordia University Univesity

lihaoyuwilson@gmail.com

Eleonora Mancini

Collaborating Alumni - Université de Montréal

Principal supervisor :

eleonora.mancini@mila.quebec

ricard.marxer@mila.quebec

Ricard Marxer

Collaborating researcher - University of Toulon

Principal supervisor :

Pablo Piantanida

pierfrancesco.melucci@mila.quebec

Pierfrancesco Melucci

Research Intern - Concordia University

PhD - Université de Montréal

hussein-mohamu.jama@mila.quebec

PhD - Concordia University

pooneh.mousavi@mila.quebec

PhD - Concordia University

Co-supervisor :

Laurent Charlin

firat.oncel@mila.quebec

artem.ploujnikov@mila.quebec

Peter Peter

Postdoctorate - McGill University

PhD - Université de Montréal

Raphael Switzer

Research Intern - Concordia University

raphael.switzer@mila.quebec

benjamin.l.van.niekerk@gmail.com

Benjamin Van Niekerk

Postdoctorate - Concordia University

FocalCodec: Giving LLMs Ears and a Voice at Ultra-Low Bitrates

Blog Posts

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

January 23, 2026

Luca Della Libera

Francesco Paissan

Cem Subakan

Mirco Ravanelli

Read the article

June 13, 2024

SpeechBrain 1.0: Making Conversational AI Accessible to Everyone

Mirco Ravanelli

Read the article

April 28, 2021

Introducing SpeechBrain: A General-Purpose PyTorch Speech Processing Toolkit

Mirco Ravanelli

Loren Lugosch

Read the article

Publications

Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets

Dominique Beaini

Shenyang Huang

Joao Alex Cunha

Zhiyi Li

Gabriela Moisescu-Pareja

Oleksandr Dymov

Samuel Maddrell-Mander

Callum McLean

Jama Hussein Mohamud

Michael Craig

Cristian Gabellini

Kerstin Klaser

Josef Dean

Cas Wognum … (see 14 more)

Maciej Sypetkowski

Hadrien Mary

Therence Bois

Andrew Fitzgibbon

Błażej Banaszewski

Chad Martin

Dominic Masters

Recently, pre-trained foundation models have shown significant advancements in multiple fields. However, the lack of datasets with labeled f… (see more)eatures and codebases has hindered the development of a supervised foundation model for molecular tasks. Here, we have carefully curated seven datasets specifically tailored for node- and graph-level prediction tasks to facilitate supervised learning on molecules. Moreover, to support the development of multi-task learning on our proposed datasets, we created the Graphium graph machine learning library. Our dataset collection encompasses two distinct categories. Firstly, the TOYMIX category modifies three small existing datasets with additional data for multi-task learning. Secondly, the LARGEMIX category includes four large-scale datasets with 344M graph-level data points and 409M node-level data points from ∼5M unique molecules. Finally, the ultra-large dataset contains 2,210M graph-level data points and 2,031M node-level data points coming from 86M molecules. Hence our datasets represent an order of magnitude increase in data volume compared to other 2D-GNN datasets. In addition, recognizing that molecule-related tasks often span multiple levels, we have designed our library to explicitly support multi-tasking, offering a diverse range of multi-level representations, i.e., representations at the graph, node, edge, and node-pair level. We equipped the library with an extensive collection of models and features to cover different levels of molecule analysis. By combining our curated datasets with this versatile library, we aim to accelerate the development of molecule foundation models. Datasets and code are available at https://github.com/datamol-io/graphium.

2024-05-06

International Conference on Learning Representations (Accept (poster))

openreview.net

Focal Modulation Networks for Interpretable Sound Classification

Luca Della Libera

The increasing success of deep neural networks has raised concerns about their inherent black-box nature, posing challenges related to inter… (see more)pretability and trust. While there has been extensive exploration of interpretation techniques in vision and language, interpretability in the audio domain has received limited attention, primarily focusing on post-hoc explanations. This paper addresses the problem of interpretability by-design in the audio domain by utilizing the recently proposed attention-free focal modulation networks (FocalNets). We apply FocalNets to the task of environmental sound classification for the first time and evaluate their interpretability properties on the popular ESC-50 dataset. Our method outperforms a similarly sized vision transformer both in terms of accuracy and interpretability. Furthermore, it is competitive against PIQ, a method specifically designed for post-hoc interpretation in the audio domain.

2024-04-13

2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) (published)

Resource-Efficient Separation Transformer

Luca Della Libera

Samuele Cornell

Frédéric Lepoutre

François Grondin

Transformers have recently achieved state-of-the-art performance in speech separation. These models, however, are computationally demanding … (see more)and require a lot of learnable parameters. This paper explores Transformer-based speech separation with a reduced computational cost. Our main contribution is the development of the Resource-Efficient Separation Transformer (RE-SepFormer), a self-attention-based architecture that reduces the computational burden in two ways. First, it uses non-overlapping blocks in the latent space. Second, it operates on compact latent summaries calculated from each chunk. The RE-SepFormer reaches a competitive performance on the popular WSJ0-2Mix and WHAM! datasets in both causal and non-causal settings. Remarkably, it scales significantly better than the previous Transformer-based architectures in terms of memory and inference time, making it more suitable for processing long mixtures.

2024-04-13

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (published)

Open-Source Conversational AI with SpeechBrain 1.0

Titouan Parcollet

Adel Moumen

Sylvain de Langen

Yingzhi Wang

Zeyu Zhao

Shucong Zhang

Georgios Karakasidis

Sung-Lin Yeh

Pierre Champion

Aku Rouhe

Rudolf Braun … (see 13 more)

Florian Mai

Juan Zuluaga-Gomez

Seyed Mahed Mousavi

Andreas Nautsch

Ha Nguyen

Xuechen Liu

Sangeet Sagar

Jarod Duret

Salima Mdhaffar

Gaëlle Laperrière

Mickael Rouvier

Renato De Mori

Yannick Estève

SpeechBrain is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech rec… (see more)ognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete "recipes" of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

2023-12-31

Journal of Machine Learning Research (published)

Phoneme Discretized Saliency Maps for Explainable Detection of AI-Generated Voice

Shubham Gupta

Pascal Germain

In this paper, we propose Phoneme Discretized Saliency Maps (PDSM), a discretization algorithm for saliency maps that takes advantage of pho… (see more)neme boundaries for explainable detection of AI-generated voice. We experimentally show with two different Text-to-Speech systems (i.e., Tacotron2 and Fastspeech2) that the proposed algorithm produces saliency maps that result in more faithful explanations compared to standard posthoc explanation methods. Moreover, by associating the saliency maps to the phoneme representations, this methodology generates explanations that tend to be more understandable than standard saliency maps on magnitude spectrograms.

2023-12-31

Conference of the International Speech Communication Association (published)

Speech Model Pre-training for End-to-End Spoken Language Understanding

Loren Lugosch

Patrick Ignoto

Vikrant Singh Tomar

Yoshua Bengio

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map spe… (see more)ech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

2023-12-31

TIB (published)

RescueSpeech: A German Corpus for Speech Recognition in Search and Rescue Domain

Sangeet Sagar

Bernd Kiefer

Ivana Kruijff-Korbayová

Josef van Genabith

Despite the recent advancements in speech recognition, there are still difficulties in accurately transcribing conversational and emotional … (see more)speech in noisy and reverberant acoustic environments. This poses a particular challenge in the search and rescue (SAR) domain, where transcribing conversations among rescue team members is crucial to support real-time decision-making. The scarcity of speech data and associated background noise in SAR scenarios make it difficult to deploy robust speech recognition systems. To address this issue, we have created and made publicly available a German speech dataset called RescueSpeech. This dataset includes real speech recordings from simulated rescue exercises. Additionally, we have released competitive training recipes and pre-trained models. Our study highlights that the performance attained by state-of-the-art methods in this challenging scenario is still far from reaching an acceptable level.

2023-12-15

2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (published)

Speech Emotion Diarization: Which Emotion Appears When?

Yingzhi Wang

Alaa Nfissi

Alya Yacoubi

Speech Emotion Recognition (SER) typically relies on utterance-level solutions. However, emotions conveyed through speech should be consider… (see more)ed as discrete speech events with definite temporal boundaries, rather than attributes of the entire utterance. To reflect the fine-grained nature of speech emotions, we propose a new task: Speech Emotion Diarization (SED). Just as Speaker Diarization answers the question of "Who speaks when?", Speech Emotion Diarization answers the question of "Which emotion appears when?". To facilitate the evaluation of the performance and establish a common benchmark for researchers, we introduce the Zaion Emotion Dataset (ZED), an openly accessible speech emotion dataset that includes non-acted emotions recorded in real-life conditions, along with manually-annotated boundaries of emotion segments within the utterance. We provide competitive baselines and open-source the code and the pre-trained models.

2023-12-15

2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (published)

Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets (Ultra Large Dataset)

Dominique Beaini

Shenyang Huang

Joao Alex Cunha

Zhiyi Li

Gabriela Moisescu-Pareja

Oleksandr Dymov

Samuel Maddrell-Mander

Callum McLean

Ali Parviz

Luis T. Díaz Müller

Jama Hussein Mohamud

Michael Craig

Cristian Gabellini

Jian Tang … (see 8 more)

Christopher G. Morris

Guy Wolf

Prudencio Tossou

Hadrien Mary

Błażej Banaszewski

Chad Martin

Dominic Masters

2023-09-21

Zenodo (unknown)

Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?

Salah Zaiem

Youcef Kemiche

Titouan Parcollet

Slim Essid

Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on… (see more) speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.

2023-08-19

INTERSPEECH 2023 (published)

Simulated Annealing in Early Layers Leads to Better Generalization

Amir M. Sarfi

Zahra Karimpour

Muawiz Chaudhary

Nasir M. Khalid

Sudhir Mudur

Eugene Belilovsky

Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer… (see more) periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance.

2023-06-19

IEEE/CVF Conference on Computer Vision and Pattern Recognition (Accept)

Fine-Tuning Strategies for Faster Inference Using Speech Self-Supervised Models: A Comparative Study

Salah Zaiem

Robin Algayres

Titouan Parcollet

Slim Essid

Self-supervised learning (SSL) has allowed substantial progress in Automatic Speech Recognition (ASR) performance in low-resource settings. … (see more)In this context, it has been demonstrated that larger self-supervised feature extractors are crucial for achieving lower downstream ASR error rates. Thus, better performance might be sanctioned with longer inferences. This article explores different approaches that may be deployed during the fine-tuning to reduce the computations needed in the SSL encoder, leading to faster inferences. We adapt a number of existing techniques to common ASR settings and benchmark them, displaying performance drops and gains in inference times. Interestingly, we found that given enough downstream data, a simple downsampling of the input sequences outperforms the other methods with both low performance drops and high computational savings, reducing computations by 61.3% with an WER increase of only 0.81. Finally, we analyze the robustness of the comparison to changes in dataset conditions, revealing sensitivity to dataset size.

2023-06-03

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) (published)