Mirco Ravanelli

heidari.allan@mila.quebec

Biographie

Je suis chercheur en IA, enseignant et chef de file de l'open source axé sur la prochaine génération de machines intelligentes.

Je suis également professeur agrégé à l'Université Concordia, professeur adjoint à l'Université de Montréal et membre associé de Mila – Institut québécois d'intelligence artificielle. Mes travaux de recherche portent sur l'apprentissage profond appliqué à l'IA conversationnelle, les modèles fondateurs multimodaux, l'IA explicable et l'apprentissage automatique efficient.

Mes travaux ont donné lieu à plus de 100 publications scientifiques et ont été couronnés par plusieurs prix internationaux, dont l'Amazon Research Award. J'ai eu le privilège d'effectuer mon stage postdoctoral à Mila sous la direction de Yoshua Bengio.

Je suis le fondateur et responsable de SpeechBrain, l'une des boîtes à outils open source les plus adoptées au monde pour la parole et l'IA conversationnelle, utilisée par les chercheurs et les ingénieurs tant dans le milieu académique qu'industriel. SpeechBrain est devenu un véritable effort communautaire mondial, contribuant à démocratiser l'accès aux technologies d'IA de pointe.

Au-delà de la recherche, je me passionne pour l'enseignement et le mentorat. Je suis convaincu que le progrès de l'IA ne repose pas uniquement sur les percées scientifiques, mais exige aussi de former la prochaine génération de chercheurs et d'ingénieurs à développer une IA responsable et au service de la société.

Domaines de recherche : IA conversationnelle • LLM multimodaux • Jetons audio (Audio Tokens) • Apprentissage de représentations • IA explicable • Apprentissage profond efficient • IA open source

Étudiants actuels

Allan Allan

Baccalauréat - Concordia

gaspard.botte@mila.quebec

Gaspard Botté

Collaborateur·rice de recherche - Concordia

briac.cordelle@mila.quebec

Cordelle Briac

Maîtrise recherche - Concordia University

Leo Brodeur

Maîtrise recherche - Concordia

leobrod44@gmail.com

Matthieu Cervera

Collaborateur·rice de recherche

Superviseur⋅e principal⋅e :

matthieu.cervera@mila.quebec

Victor Cru

Maîtrise recherche - Concordia

victor.cruz@mila.quebec

Luca Della Libera

Doctorat - Concordia

Co-superviseur⋅e :

luca.dellalibera@mila.quebec

Wagner Drew

Maîtrise recherche - Concordia

Co-superviseur⋅e :

Irina Rish

drew.wagner@mila.quebec

Gianfranco Dumoulin Bertucci

Maîtrise recherche - Concordia

gianfranco.bertucci@mila.quebec

nadine.el-mufti@mila.quebec

Nadine El-Mufti

Maîtrise recherche - Concordia

Maab Elrashid Ahmed Mohamed

Doctorat - Concordia

Co-superviseur⋅e :

maab-elrashid.ahmed-mohamed@mila.quebec

francesco.bonzi@mila.quebec

Bonzi Francesco

Doctorat - Concordia

salman.hussainali@mila.quebec

Salman Sami Hussain Ali

Maîtrise recherche - Concordia University

sidra.intikhab@mila.quebec

Sidra Intikhab

Stagiaire de recherche - Concordia

Lovenya Jain

Collaborateur·rice de recherche - Université Laval

Superviseur⋅e principal⋅e :

f20211732@pilani.bits-pilani.ac.in

Jihoon Jeong

Doctorat - Université Laval

Superviseur⋅e principal⋅e :

jihoon.jeong@mila.quebec

Haoyu Li

Maîtrise professionnelle - Concordia Univesity

lihaoyuwilson@gmail.com

Eleonora Mancini

Collaborateur·rice alumni - UdeM

Superviseur⋅e principal⋅e :

eleonora.mancini@mila.quebec

hussein-mohamu.jama@mila.quebec

Jama Mohamud

Doctorat - UdeM

Doctorat - Concordia

pooneh.mousavi@mila.quebec

Doctorat - Concordia

Co-superviseur⋅e :

Laurent Charlin

firat.oncel@mila.quebec

artem.ploujnikov@mila.quebec

Peter Peter

Postdoctorat - McGill

Doctorat - UdeM

Raphael Switzer

Stagiaire de recherche - Concordia

raphael.switzer@mila.quebec

benjamin.l.van.niekerk@gmail.com

Benjamin Van Niekerk

Postdoctorat - Concordia

FocalCodec : donner l’ouïe et la parole aux LLM à débit ultra-faible

Billets de blogue

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

23 janvier 2026

par

Luca Della Libera

Francesco Paissan

Cem Subakan

Mirco Ravanelli

Lire l'article

13 juin 2024

SpeechBrain 1.0 : rendre l’IA conversationnelle accessible à tout le monde

par

Mirco Ravanelli

Lire l'article

Introducing SpeechBrain: A general-purpose PyTorch speech processing toolkit

28 avril 2021

Voici SpeechBrain : Une boîte à outils polyvalente de traitement de la parole basée sur PyTorch

par

Mirco Ravanelli

Loren Lugosch

Lire l'article

Publications

Adaptive Order Policies for Masked Diffusion

Jama Hussein Mohamud

Mohsin Hasan

Yoshua Bengio

Masked diffusion models have seen great success in capturing data distributions over discrete sequences in domains such as text and proteins… (voir plus). These models generate data by iteratively unmasking tokens starting from a fully masked sequence, with the unmasking order typically chosen at random or using a heuristic based on denoiser probabilities. In this work, we propose a scheme for learning the unmasking order using an additional lightweight policy network on top of a diffusion model. Our proposed loss reweights terms in the masked diffusion loss according to policy probabilities, and results in a policy that prefers positions where the denoiser is more likely to be correct. We study this loss in two settings: (i) training solely the policy while using a frozen pre-trained denoiser, and (ii) training the policy and denoiser jointly with the weighted loss to allow for mutual adaptation. We demonstrate that our approach outperforms common heuristics on problems that are sensitive to token ordering, such as combinatorial problems, proteins as well as various coding and language tasks.

2026-12-30

OpenReview (publié)

openreview.net

DASB - Discrete Audio and Speech Benchmark

Jarod Duret

Darius Petermann

Anastasia Kuznetsova

Duret, Jarod

Kuznetsova, Anastasia

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling mult… (voir plus)imodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

2026-12-30

OpenReview (publié)

openreview.net

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chai… (voir plus)ns remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

2026-12-30

OpenReview (publié)

openreview.net

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling mean… (voir plus)s that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

2026-10-06

arXiv (prépublication)

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

Séverin Baroudi

Yanis Labrak

Pierfrancesco Melucci

Sergio Burdisso

Petr Motlicek

Hervé Bredin

Ricard Marxer

Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application… (voir plus) to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.

2026-10-02

arXiv (prépublication)

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

Amir Ivry

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We… (voir plus) propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.

2026-09-29

arXiv (prépublication)

Random Recursive Models

Jama Hussein Mohamud

Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. Ho… (voir plus)wever, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of

2026-09-29

arXiv (prépublication)

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

Gaspard Botté

Séverin Baroudi

Samir Sadok

Francesco Paissan

Thomas Hueber

Xavier Alameda-Pineda

Ricard Marxer

Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely … (voir plus)on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.

2026-09-28

arXiv (prépublication)

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

Fırat Öncel

Salman Hussain Ali

Çağatay Yıldız

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Co… (voir plus)ntinued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams. This is a poor fit for time-incremental updates on web-scale crawls, where successive snapshots share substantial URL overlap by design. We study time-incremental CPT in this realistic regime: continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families (OLMo2, Llama-3.1/3.2, Gemma-3-1B) and four parameter scales (1B-3B-7B-8B). We organize our findings around four practical questions. (i) Is knowledge acquired? Yes, but heterogeneously, and without catastrophic forgetting: five of six models also improve on pre-cutoff factual recall, and the gains track pretraining saturation (driven primarily by token budget per parameter). (ii) What does it cost? Almost nothing: the macro-average across a thirteen-task suite stays within 0.01 of the base for every model. (iii) What is the recipe? Data quality dominates quantity (a curated 6B-token slice matches a broader 40B one); the optima for knowledge acquisition and general capability are separated by roughly an order of magnitude in learning rate; and LoRA at sufficient rank matches full CPT. (iv) Does it survive deployment? CPT gains transfer through SFT, while DPO's effect is family-dependent. Together, these results paint a more optimistic picture of time-incremental CPT than the prior continual learning literature suggests.

2026-09-19

arXiv (prépublication)

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Luca Della Libera

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, … (voir plus)reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

2026-09-09

arXiv (prépublication)

Enhancing Audio Reasoning via Semantic Summary Prediction

Francesco Bonzi

Large Audio Language Models (LALMs) perform well on complex question answering but often show a reasoning gap, where explicit Chain-of-Thoug… (voir plus)ht (CoT) reduces accuracy compared to direct answers. We hypothesize that long reasoning sequences shift attention away from the audio input. To address this, we propose SPARE (Semantic Prediction for Audio REasoning), which introduces a register token aligned with the final conclusion using a cosine similarity loss with a Sentence-BERT embedding. This conditions the model's latent space with the target semantic goal before reasoning begins. Experiments on MMAU and MMAR with SALMONN show improved zero-shot reasoning and stronger early attention to audio without additional inference cost.

2026-08-06

arXiv (prépublication)

Controllable and Content-Based Recommendations

Traditional recommendation systems rely on latent (dense) representations, making them difficult to interpret and control. We propose the Co… (voir plus)ntrollable and Content-Based Recommendations (CCBR) framework, which builds its recommendations from textual user profile representations. CCBR plugs into collaborative filtering models and introduces controllability via text bottlenecks. We show that CCBR enables text-based and multimodal interventions, allowing users to steer the model towards the directions they prefer. Different from existing controllable recommendation systems, CCBR infers the text summaries directly from item contents (images, audio or video). Across image-, audio-, and video-based datasets, we demonstrate that the proposed framework obtains competitive model performance with standard (latent-representation) models while providing controllable model summaries via text. The model also outperforms TEARS, a recent baseline for controllable recommendation systems. Through systematic interventions, we demonstrate the efficacy of the user steering mechanism.

2026-07-22

arXiv (prépublication)