Mirco Ravanelli

heidari.allan@mila.quebec

Biography

I am an AI researcher, educator, and open-source leader working on the next generation of intelligent machines. I am also an Associate Professor at Concordia University, Adjunct Professor at Université de Montréal, and Associate Member of Mila – Quebec AI Institute. My research focuses on deep learning for Conversational AI, multimodal foundation models, interpretable AI, and efficient machine learning.

My work has resulted in over 100 scientific publications and has been recognized through international awards, including the Amazon Research Award. I had the privilege of completing my postdoctoral training at Mila under the supervision of Yoshua Bengio.

I am the founder and leader of SpeechBrain, one of the world's most widely adopted open-source toolkits for speech and conversational AI, used by researchers and engineers across academia and industry. SpeechBrain has become a global community effort, helping democratize access to cutting-edge AI technologies.

Beyond research, I am passionate about education and mentorship. I believe that advancing AI requires not only scientific breakthroughs but also training the next generation of researchers and engineers to develop AI responsibly and for the benefit of society.

Research interests: Conversational AI • Multimodal LLMs • Audio Tokens • Representation Learning • Interpretable AI • Efficient Deep Learning • Open-Source AI

Current Students

Allan Allan

Undergraduate - Concordia University

gaspard.botte@mila.quebec

Gaspard Botté

Research Intern - Concordia University

briac.cordelle@mila.quebec

Cordelle Briac

Master's Research - Concordia University University

Leo Brodeur

Master's Research - Concordia University

leobrod44@gmail.com

Matthieu Cervera

Collaborating researcher

Principal supervisor :

matthieu.cervera@mila.quebec

Victor Cruz

Master's Research - Concordia University

victor.cruz@mila.quebec

Luca Della Libera

PhD - Concordia University

Co-supervisor :

luca.dellalibera@mila.quebec

Wagner Drew

Master's Research - Concordia University

Co-supervisor :

Irina Rish

drew.wagner@mila.quebec

Gianfranco Dumoulin Bertucci

Master's Research - Concordia University

gianfranco.bertucci@mila.quebec

nadine.el-mufti@mila.quebec

Nadine El-Mufti

Master's Research - Concordia University

Maab Elrashid Ahmed Mohamed

PhD - Concordia University

Co-supervisor :

maab-elrashid.ahmed-mohamed@mila.quebec

francesco.bonzi@mila.quebec

Bonzi Francesco

PhD - Concordia University

salman.hussainali@mila.quebec

Salman Sami Hussain Ali

Master's Research - Concordia University University

sidra.intikhab@mila.quebec

Sidra Intikhab

Research Intern - Concordia University

Lovenya Jain

Collaborating researcher - Université Laval

Principal supervisor :

f20211732@pilani.bits-pilani.ac.in

Jihoon Jeong

PhD - Université Laval

Principal supervisor :

jihoon.jeong@mila.quebec

Haoyu Li

Professional Master's - Concordia University Univesity

lihaoyuwilson@gmail.com

Eleonora Mancini

Collaborating Alumni - Université de Montréal

Principal supervisor :

eleonora.mancini@mila.quebec

hussein-mohamu.jama@mila.quebec

Jama Mohamud

PhD - Université de Montréal

PhD - Concordia University

pooneh.mousavi@mila.quebec

PhD - Concordia University

Co-supervisor :

Laurent Charlin

firat.oncel@mila.quebec

artem.ploujnikov@mila.quebec

Peter Peter

Postdoctorate - McGill University

PhD - Université de Montréal

Raphael Switzer

Research Intern - Concordia University

raphael.switzer@mila.quebec

benjamin.l.van.niekerk@gmail.com

Benjamin Van Niekerk

Postdoctorate - Concordia University

FocalCodec: Giving LLMs Ears and a Voice at Ultra-Low Bitrates

Blog Posts

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

January 23, 2026

Luca Della Libera

Francesco Paissan

Cem Subakan

Mirco Ravanelli

Read the article

June 13, 2024

SpeechBrain 1.0: Making Conversational AI Accessible to Everyone

Mirco Ravanelli

Read the article

April 28, 2021

Introducing SpeechBrain: A General-Purpose PyTorch Speech Processing Toolkit

Mirco Ravanelli

Loren Lugosch

Read the article

Publications

Adaptive Order Policies for Masked Diffusion

Jama Hussein Mohamud

Mohsin Hasan

Yoshua Bengio

Masked diffusion models have seen great success in capturing data distributions over discrete sequences in domains such as text and proteins… (see more). These models generate data by iteratively unmasking tokens starting from a fully masked sequence, with the unmasking order typically chosen at random or using a heuristic based on denoiser probabilities. In this work, we propose a scheme for learning the unmasking order using an additional lightweight policy network on top of a diffusion model. Our proposed loss reweights terms in the masked diffusion loss according to policy probabilities, and results in a policy that prefers positions where the denoiser is more likely to be correct. We study this loss in two settings: (i) training solely the policy while using a frozen pre-trained denoiser, and (ii) training the policy and denoiser jointly with the weighted loss to allow for mutual adaptation. We demonstrate that our approach outperforms common heuristics on problems that are sensitive to token ordering, such as combinatorial problems, proteins as well as various coding and language tasks.

2026-12-30

OpenReview (published)

openreview.net

DASB - Discrete Audio and Speech Benchmark

Jarod Duret

Darius Petermann

Anastasia Kuznetsova

Duret, Jarod

Kuznetsova, Anastasia

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling mult… (see more)imodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

2026-12-30

OpenReview (published)

openreview.net

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chai… (see more)ns remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

2026-12-30

OpenReview (published)

openreview.net

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling mean… (see more)s that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

2026-10-06

arXiv (preprint)

Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition

Séverin Baroudi

Yanis Labrak

Pierfrancesco Melucci

Sergio Burdisso

Petr Motlicek

Hervé Bredin

Ricard Marxer

Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application… (see more) to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.

2026-10-02

arXiv (preprint)

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

Amir Ivry

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We… (see more) propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.

2026-09-29

arXiv (preprint)

Random Recursive Models

Jama Hussein Mohamud

Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. Ho… (see more)wever, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of

2026-09-29

arXiv (preprint)

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

Gaspard Botté

Séverin Baroudi

Samir Sadok

Francesco Paissan

Thomas Hueber

Xavier Alameda-Pineda

Ricard Marxer

Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely … (see more)on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.

2026-09-28

arXiv (preprint)

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

Fırat Öncel

Salman Hussain Ali

Çağatay Yıldız

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Co… (see more)ntinued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams. This is a poor fit for time-incremental updates on web-scale crawls, where successive snapshots share substantial URL overlap by design. We study time-incremental CPT in this realistic regime: continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families (OLMo2, Llama-3.1/3.2, Gemma-3-1B) and four parameter scales (1B-3B-7B-8B). We organize our findings around four practical questions. (i) Is knowledge acquired? Yes, but heterogeneously, and without catastrophic forgetting: five of six models also improve on pre-cutoff factual recall, and the gains track pretraining saturation (driven primarily by token budget per parameter). (ii) What does it cost? Almost nothing: the macro-average across a thirteen-task suite stays within 0.01 of the base for every model. (iii) What is the recipe? Data quality dominates quantity (a curated 6B-token slice matches a broader 40B one); the optima for knowledge acquisition and general capability are separated by roughly an order of magnitude in learning rate; and LoRA at sufficient rank matches full CPT. (iv) Does it survive deployment? CPT gains transfer through SFT, while DPO's effect is family-dependent. Together, these results paint a more optimistic picture of time-incremental CPT than the prior continual learning literature suggests.

2026-09-19

arXiv (preprint)

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Luca Della Libera

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, … (see more)reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

2026-09-09

arXiv (preprint)

Enhancing Audio Reasoning via Semantic Summary Prediction

Francesco Bonzi

Large Audio Language Models (LALMs) perform well on complex question answering but often show a reasoning gap, where explicit Chain-of-Thoug… (see more)ht (CoT) reduces accuracy compared to direct answers. We hypothesize that long reasoning sequences shift attention away from the audio input. To address this, we propose SPARE (Semantic Prediction for Audio REasoning), which introduces a register token aligned with the final conclusion using a cosine similarity loss with a Sentence-BERT embedding. This conditions the model's latent space with the target semantic goal before reasoning begins. Experiments on MMAU and MMAR with SALMONN show improved zero-shot reasoning and stronger early attention to audio without additional inference cost.

2026-08-06

arXiv (preprint)

Controllable and Content-Based Recommendations

Traditional recommendation systems rely on latent (dense) representations, making them difficult to interpret and control. We propose the Co… (see more)ntrollable and Content-Based Recommendations (CCBR) framework, which builds its recommendations from textual user profile representations. CCBR plugs into collaborative filtering models and introduces controllability via text bottlenecks. We show that CCBR enables text-based and multimodal interventions, allowing users to steer the model towards the directions they prefer. Different from existing controllable recommendation systems, CCBR infers the text summaries directly from item contents (images, audio or video). Across image-, audio-, and video-based datasets, we demonstrate that the proposed framework obtains competitive model performance with standard (latent-representation) models while providing controllable model summaries via text. The model also outperforms TEARS, a recent baseline for controllable recommendation systems. Through systematic interventions, we demonstrate the efficacy of the user steering mechanism.

2026-07-22

arXiv (preprint)