Mirco Ravanelli

heidari.allan@mila.quebec

Biographie

Je suis chercheur en IA, enseignant et chef de file de l'open source axé sur la prochaine génération de machines intelligentes.

Je suis également professeur agrégé à l'Université Concordia, professeur adjoint à l'Université de Montréal et membre associé de Mila – Institut québécois d'intelligence artificielle. Mes travaux de recherche portent sur l'apprentissage profond appliqué à l'IA conversationnelle, les modèles fondateurs multimodaux, l'IA explicable et l'apprentissage automatique efficient.

Mes travaux ont donné lieu à plus de 100 publications scientifiques et ont été couronnés par plusieurs prix internationaux, dont l'Amazon Research Award. J'ai eu le privilège d'effectuer mon stage postdoctoral à Mila sous la direction de Yoshua Bengio.

Je suis le fondateur et responsable de SpeechBrain, l'une des boîtes à outils open source les plus adoptées au monde pour la parole et l'IA conversationnelle, utilisée par les chercheurs et les ingénieurs tant dans le milieu académique qu'industriel. SpeechBrain est devenu un véritable effort communautaire mondial, contribuant à démocratiser l'accès aux technologies d'IA de pointe.

Au-delà de la recherche, je me passionne pour l'enseignement et le mentorat. Je suis convaincu que le progrès de l'IA ne repose pas uniquement sur les percées scientifiques, mais exige aussi de former la prochaine génération de chercheurs et d'ingénieurs à développer une IA responsable et au service de la société.

Domaines de recherche : IA conversationnelle • LLM multimodaux • Jetons audio (Audio Tokens) • Apprentissage de représentations • IA explicable • Apprentissage profond efficient • IA open source

Étudiants actuels

Allan Allan

Baccalauréat - Concordia

gaspard.botte@mila.quebec

Gaspard Botté

Stagiaire de recherche - Concordia

briac.cordelle@mila.quebec

Cordelle Briac

Maîtrise recherche - Concordia University

Leo Brodeur

Maîtrise recherche - Concordia

leobrod44@gmail.com

Matthieu Cervera

Collaborateur·rice de recherche

Superviseur⋅e principal⋅e :

matthieu.cervera@mila.quebec

Victor Cruz

Maîtrise recherche - Concordia

victor.cruz@mila.quebec

Luca Della Libera

Doctorat - Concordia

Co-superviseur⋅e :

luca.dellalibera@mila.quebec

Wagner Drew

Maîtrise recherche - Concordia

Co-superviseur⋅e :

Irina Rish

drew.wagner@mila.quebec

Gianfranco Dumoulin Bertucci

Maîtrise recherche - Concordia

gianfranco.bertucci@mila.quebec

nadine.el-mufti@mila.quebec

Nadine El-Mufti

Maîtrise recherche - Concordia

Maab Elrashid Ahmed Mohamed

Doctorat - Concordia

Co-superviseur⋅e :

maab-elrashid.ahmed-mohamed@mila.quebec

francesco.bonzi@mila.quebec

Bonzi Francesco

Doctorat - Concordia

salman.hussainali@mila.quebec

Salman Sami Hussain Ali

Maîtrise recherche - Concordia University

sidra.intikhab@mila.quebec

Sidra Intikhab

Stagiaire de recherche - Concordia

Lovenya Jain

Collaborateur·rice de recherche - Université Laval

Superviseur⋅e principal⋅e :

f20211732@pilani.bits-pilani.ac.in

Jihoon Jeong

Doctorat - Université Laval

Superviseur⋅e principal⋅e :

jihoon.jeong@mila.quebec

Haoyu Li

Maîtrise professionnelle - Concordia Univesity

lihaoyuwilson@gmail.com

Eleonora Mancini

Collaborateur·rice alumni - UdeM

Superviseur⋅e principal⋅e :

eleonora.mancini@mila.quebec

hussein-mohamu.jama@mila.quebec

Jama Mohamud

Doctorat - UdeM

Doctorat - Concordia

pooneh.mousavi@mila.quebec

Doctorat - Concordia

Co-superviseur⋅e :

Laurent Charlin

firat.oncel@mila.quebec

artem.ploujnikov@mila.quebec

Peter Peter

Postdoctorat - McGill

Doctorat - UdeM

Raphael Switzer

Stagiaire de recherche - Concordia

raphael.switzer@mila.quebec

benjamin.l.van.niekerk@gmail.com

Benjamin Van Niekerk

Postdoctorat - Concordia

FocalCodec : donner l’ouïe et la parole aux LLM à débit ultra-faible

Billets de blogue

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

23 janvier 2026

par

Luca Della Libera

Francesco Paissan

Cem Subakan

Mirco Ravanelli

Lire l'article

13 juin 2024

SpeechBrain 1.0 : rendre l’IA conversationnelle accessible à tout le monde

par

Mirco Ravanelli

Lire l'article

Introducing SpeechBrain: A general-purpose PyTorch speech processing toolkit

28 avril 2021

Voici SpeechBrain : Une boîte à outils polyvalente de traitement de la parole basée sur PyTorch

par

Mirco Ravanelli

Loren Lugosch

Lire l'article

Publications

Listenable Maps for Audio Classifiers

Francesco Paissan

Despite the impressive performance of deep learning models across diverse tasks, their complexity poses challenges for interpretation. This … (voir plus)challenge is particularly evident for audio signals, where conveying interpretations becomes inherently difficult. To address this issue, we introduce Listenable Maps for Audio Classifiers (L-MAC), a posthoc interpretation method that generates faithful and listenable interpretations. L-MAC utilizes a decoder on top of a pretrained classifier to generate binary masks that highlight relevant portions of the input audio. We train the decoder with a loss function that maximizes the confidence of the classifier decision on the masked-in portion of the audio while minimizing the probability of model output for the masked-out portion. Quantitative evaluations on both in-domain and out-of-domain data demonstrate that L-MAC consistently produces more faithful interpretations than several gradient and masking-based methodologies. Furthermore, a user study confirms that, on average, users prefer the interpretations generated by the proposed technique.

2024-07-22

International Conference on Machine Learning (Accept (Oral))

proceedings.mlr.press

Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets

Dominique Beaini

Shenyang Huang

Joao Alex Cunha

Zhiyi Li

Gabriela Moisescu-Pareja

Oleksandr Dymov

Samuel Maddrell-Mander

Callum McLean

Jama Hussein Mohamud

Michael Craig

Cristian Gabellini

Kerstin Klaser

Josef Dean

Cas Wognum … (voir 14 de plus)

Maciej Sypetkowski

Hadrien Mary

Therence Bois

Andrew Fitzgibbon

Błażej Banaszewski

Chad Martin

Dominic Masters

Recently, pre-trained foundation models have shown significant advancements in multiple fields. However, the lack of datasets with labeled f… (voir plus)eatures and codebases has hindered the development of a supervised foundation model for molecular tasks. Here, we have carefully curated seven datasets specifically tailored for node- and graph-level prediction tasks to facilitate supervised learning on molecules. Moreover, to support the development of multi-task learning on our proposed datasets, we created the Graphium graph machine learning library. Our dataset collection encompasses two distinct categories. Firstly, the TOYMIX category modifies three small existing datasets with additional data for multi-task learning. Secondly, the LARGEMIX category includes four large-scale datasets with 344M graph-level data points and 409M node-level data points from ∼5M unique molecules. Finally, the ultra-large dataset contains 2,210M graph-level data points and 2,031M node-level data points coming from 86M molecules. Hence our datasets represent an order of magnitude increase in data volume compared to other 2D-GNN datasets. In addition, recognizing that molecule-related tasks often span multiple levels, we have designed our library to explicitly support multi-tasking, offering a diverse range of multi-level representations, i.e., representations at the graph, node, edge, and node-pair level. We equipped the library with an extensive collection of models and features to cover different levels of molecule analysis. By combining our curated datasets with this versatile library, we aim to accelerate the development of molecule foundation models. Datasets and code are available at https://github.com/datamol-io/graphium.

2024-05-06

International Conference on Learning Representations (Accept (poster))

openreview.net

Focal Modulation Networks for Interpretable Sound Classification

Luca Della Libera

The increasing success of deep neural networks has raised concerns about their inherent black-box nature, posing challenges related to inter… (voir plus)pretability and trust. While there has been extensive exploration of interpretation techniques in vision and language, interpretability in the audio domain has received limited attention, primarily focusing on post-hoc explanations. This paper addresses the problem of interpretability by-design in the audio domain by utilizing the recently proposed attention-free focal modulation networks (FocalNets). We apply FocalNets to the task of environmental sound classification for the first time and evaluate their interpretability properties on the popular ESC-50 dataset. Our method outperforms a similarly sized vision transformer both in terms of accuracy and interpretability. Furthermore, it is competitive against PIQ, a method specifically designed for post-hoc interpretation in the audio domain.

2024-04-13

2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) (publié)

Resource-Efficient Separation Transformer

Luca Della Libera

Samuele Cornell

Frédéric Lepoutre

François Grondin

Transformers have recently achieved state-of-the-art performance in speech separation. These models, however, are computationally demanding … (voir plus)and require a lot of learnable parameters. This paper explores Transformer-based speech separation with a reduced computational cost. Our main contribution is the development of the Resource-Efficient Separation Transformer (RE-SepFormer), a self-attention-based architecture that reduces the computational burden in two ways. First, it uses non-overlapping blocks in the latent space. Second, it operates on compact latent summaries calculated from each chunk. The RE-SepFormer reaches a competitive performance on the popular WSJ0-2Mix and WHAM! datasets in both causal and non-causal settings. Remarkably, it scales significantly better than the previous Transformer-based architectures in terms of memory and inference time, making it more suitable for processing long mixtures.

2024-04-13

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (publié)

Open-Source Conversational AI with SpeechBrain 1.0

Titouan Parcollet

Adel Moumen

Sylvain de Langen

Yingzhi Wang

Zeyu Zhao

Shucong Zhang

Georgios Karakasidis

Sung-Lin Yeh

Pierre Champion

Aku Rouhe

Rudolf Braun … (voir 13 de plus)

Florian Mai

Juan Zuluaga-Gomez

Seyed Mahed Mousavi

Andreas Nautsch

Ha Nguyen

Xuechen Liu

Sangeet Sagar

Jarod Duret

Salima Mdhaffar

Gaëlle Laperrière

Mickael Rouvier

Renato De Mori

Yannick Estève

SpeechBrain is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech rec… (voir plus)ognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete "recipes" of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

2023-12-31

Journal of Machine Learning Research (publié)

Phoneme Discretized Saliency Maps for Explainable Detection of AI-Generated Voice

Shubham Gupta

Pascal Germain

In this paper, we propose Phoneme Discretized Saliency Maps (PDSM), a discretization algorithm for saliency maps that takes advantage of pho… (voir plus)neme boundaries for explainable detection of AI-generated voice. We experimentally show with two different Text-to-Speech systems (i.e., Tacotron2 and Fastspeech2) that the proposed algorithm produces saliency maps that result in more faithful explanations compared to standard posthoc explanation methods. Moreover, by associating the saliency maps to the phoneme representations, this methodology generates explanations that tend to be more understandable than standard saliency maps on magnitude spectrograms.

2023-12-31

Conference of the International Speech Communication Association (publié)

Speech Model Pre-training for End-to-End Spoken Language Understanding

Loren Lugosch

Patrick Ignoto

Vikrant Singh Tomar

Yoshua Bengio

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map spe… (voir plus)ech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

2023-12-31

TIB (publié)

RescueSpeech: A German Corpus for Speech Recognition in Search and Rescue Domain

Sangeet Sagar

Bernd Kiefer

Ivana Kruijff-Korbayová

Josef van Genabith

Despite the recent advancements in speech recognition, there are still difficulties in accurately transcribing conversational and emotional … (voir plus)speech in noisy and reverberant acoustic environments. This poses a particular challenge in the search and rescue (SAR) domain, where transcribing conversations among rescue team members is crucial to support real-time decision-making. The scarcity of speech data and associated background noise in SAR scenarios make it difficult to deploy robust speech recognition systems. To address this issue, we have created and made publicly available a German speech dataset called RescueSpeech. This dataset includes real speech recordings from simulated rescue exercises. Additionally, we have released competitive training recipes and pre-trained models. Our study highlights that the performance attained by state-of-the-art methods in this challenging scenario is still far from reaching an acceptable level.

2023-12-15

2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (publié)

Speech Emotion Diarization: Which Emotion Appears When?

Yingzhi Wang

Alaa Nfissi

Alya Yacoubi

Speech Emotion Recognition (SER) typically relies on utterance-level solutions. However, emotions conveyed through speech should be consider… (voir plus)ed as discrete speech events with definite temporal boundaries, rather than attributes of the entire utterance. To reflect the fine-grained nature of speech emotions, we propose a new task: Speech Emotion Diarization (SED). Just as Speaker Diarization answers the question of "Who speaks when?", Speech Emotion Diarization answers the question of "Which emotion appears when?". To facilitate the evaluation of the performance and establish a common benchmark for researchers, we introduce the Zaion Emotion Dataset (ZED), an openly accessible speech emotion dataset that includes non-acted emotions recorded in real-life conditions, along with manually-annotated boundaries of emotion segments within the utterance. We provide competitive baselines and open-source the code and the pre-trained models.

2023-12-15

2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (publié)

Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets (Ultra Large Dataset)

Dominique Beaini

Shenyang Huang

Joao Alex Cunha

Zhiyi Li

Gabriela Moisescu-Pareja

Oleksandr Dymov

Samuel Maddrell-Mander

Callum McLean

Ali Parviz

Luis T. Díaz Müller

Jama Hussein Mohamud

Michael Craig

Cristian Gabellini

Jian Tang … (voir 8 de plus)

Christopher G. Morris

Guy Wolf

Prudencio Tossou

Hadrien Mary

Błażej Banaszewski

Chad Martin

Dominic Masters

2023-09-21

Zenodo (inconnu)

Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?

Salah Zaiem

Youcef Kemiche

Titouan Parcollet

Slim Essid

Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on… (voir plus) speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.

2023-08-19

INTERSPEECH 2023 (publié)

Simulated Annealing in Early Layers Leads to Better Generalization

Amir M. Sarfi

Zahra Karimpour

Muawiz Chaudhary

Nasir M. Khalid

Sudhir Mudur

Eugene Belilovsky

Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer… (voir plus) periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance.

2023-06-19

IEEE/CVF Conference on Computer Vision and Pattern Recognition (Accept)