Mirco Ravanelli

heidari.allan@mila.quebec

Biography

I am an AI researcher, educator, and open-source leader working on the next generation of intelligent machines. I am also an Associate Professor at Concordia University, Adjunct Professor at Université de Montréal, and Associate Member of Mila – Quebec AI Institute. My research focuses on deep learning for Conversational AI, multimodal foundation models, interpretable AI, and efficient machine learning.

My work has resulted in over 100 scientific publications and has been recognized through international awards, including the Amazon Research Award. I had the privilege of completing my postdoctoral training at Mila under the supervision of Yoshua Bengio.

I am the founder and leader of SpeechBrain, one of the world's most widely adopted open-source toolkits for speech and conversational AI, used by researchers and engineers across academia and industry. SpeechBrain has become a global community effort, helping democratize access to cutting-edge AI technologies.

Beyond research, I am passionate about education and mentorship. I believe that advancing AI requires not only scientific breakthroughs but also training the next generation of researchers and engineers to develop AI responsibly and for the benefit of society.

Research interests: Conversational AI • Multimodal LLMs • Audio Tokens • Representation Learning • Interpretable AI • Efficient Deep Learning • Open-Source AI

Current Students

Allan Allan

Undergraduate - Concordia University

gaspard.botte@mila.quebec

Gaspard Botté

Research Intern - Concordia University

briac.cordelle@mila.quebec

Cordelle Briac

Master's Research - Concordia University University

Leo Brodeur

Master's Research - Concordia University

leobrod44@gmail.com

Matthieu Cervera

Collaborating researcher

Principal supervisor :

matthieu.cervera@mila.quebec

Victor Cruz

Master's Research - Concordia University

victor.cruz@mila.quebec

PhD - Concordia University

Co-supervisor :

luca.dellalibera@mila.quebec

Wagner Drew

Master's Research - Concordia University

Co-supervisor :

Irina Rish

drew.wagner@mila.quebec

Gianfranco Dumoulin Bertucci

Master's Research - Concordia University

gianfranco.bertucci@mila.quebec

nadine.el-mufti@mila.quebec

Nadine El-Mufti

Master's Research - Concordia University

Maab Elrashid Ahmed Mohamed

PhD - Concordia University

Co-supervisor :

maab-elrashid.ahmed-mohamed@mila.quebec

francesco.bonzi@mila.quebec

Bonzi Francesco

PhD - Concordia University

salman.hussainali@mila.quebec

Salman Sami Hussain Ali

Master's Research - Concordia University University

sidra.intikhab@mila.quebec

Sidra Intikhab

Research Intern - Concordia University

Lovenya Jain

Collaborating researcher - Université Laval

Principal supervisor :

f20211732@pilani.bits-pilani.ac.in

Jihoon Jeong

PhD - Université Laval

Principal supervisor :

jihoon.jeong@mila.quebec

Haoyu Li

Professional Master's - Concordia University Univesity

lihaoyuwilson@gmail.com

Eleonora Mancini

Collaborating Alumni - Université de Montréal

Principal supervisor :

eleonora.mancini@mila.quebec

hussein-mohamu.jama@mila.quebec

Jama Mohamud

PhD - Université de Montréal

PhD - Concordia University

pooneh.mousavi@mila.quebec

PhD - Concordia University

Co-supervisor :

Laurent Charlin

firat.oncel@mila.quebec

artem.ploujnikov@mila.quebec

Peter Peter

Postdoctorate - McGill University

PhD - Université de Montréal

Raphael Switzer

Research Intern - Concordia University

raphael.switzer@mila.quebec

benjamin.l.van.niekerk@gmail.com

Benjamin Van Niekerk

Postdoctorate - Concordia University

FocalCodec: Giving LLMs Ears and a Voice at Ultra-Low Bitrates

Blog Posts

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

January 23, 2026

Luca Della Libera

Francesco Paissan

Cem Subakan

Mirco Ravanelli

Read the article

June 13, 2024

SpeechBrain 1.0: Making Conversational AI Accessible to Everyone

Mirco Ravanelli

Read the article

April 28, 2021

Introducing SpeechBrain: A General-Purpose PyTorch Speech Processing Toolkit

Mirco Ravanelli

Loren Lugosch

Read the article

Publications

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, … (see more)reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

2026-09-09

arXiv (preprint)

Controllable and Content-Based Recommendations

Traditional recommendation systems rely on latent (dense) representations, making them difficult to interpret and control. We propose the Co… (see more)ntrollable and Content-Based Recommendations (CCBR) framework, which builds its recommendations from textual user profile representations. CCBR plugs into collaborative filtering models and introduces controllability via text bottlenecks. We show that CCBR enables text-based and multimodal interventions, allowing users to steer the model towards the directions they prefer. Different from existing controllable recommendation systems, CCBR infers the text summaries directly from item contents (images, audio or video). Across image-, audio-, and video-based datasets, we demonstrate that the proposed framework obtains competitive model performance with standard (latent-representation) models while providing controllable model summaries via text. The model also outperforms TEARS, a recent baseline for controllable recommendation systems. Through systematic interventions, we demonstrate the efficacy of the user steering mechanism.

2026-07-22

arXiv (preprint)

HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

Artem Ploujnikov

Francesco Verdini

Samir Sadok

Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilitie… (see more)s into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps.

2026-06-25

arXiv (preprint)

MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio

Salman Hussain Ali

Umberto Cappellazzo

Fine-tuning Transformer-based foundation models has become the dominant strategy for domain adaptation in audio and speech processing. To re… (see more)duce the computational and memory costs of this process, parameter-efficient transfer learning (PETL) methods have been widely explored. Meanwhile, Mamba, a recent state-space model, has emerged as a promising alternative to Transformers for sequence modeling. In this work, we present MambAdapter, a parameter-efficient transfer learning approach that integrates Mamba into low-rank bottleneck adapters. Our design combines parameter sharing across adapters with the injection of a lightweight Mamba module, enabling more effective modeling of audio features. We demonstrate that MambAdapter matches or outperforms strong PETL baselines on four audio classification tasks and five speech recognition languages, even when operating under reduced parameter budgets.

2026-06-13

arXiv (preprint)

Exploring Token-Space Manipulation in Latent Audio Tokenizers

Neural audio codecs provide compact discrete representations for speech generation and manipulation. However, most codecs organize tokens as… (see more) frame-level sequences, making it difficult to study or intervene on global factors of variation. In this work, we propose the Latent Audio Tokenizer for Token-space Editing (LATTE) that appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding. This design produces a compact, non-temporally aligned bottleneck in which each token can aggregate global information across the full utterance. We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings while enabling simple token-space interventions. In particular, we find that swapping selected latent token positions between utterances can modify global attributes, such as speaker identity and background noise, and we evaluate these interventions on voice conversion and denoising tasks. Our results suggest that compact latent audio tokenizers can support controllable audio manipulation without supervision in task-specific editing models.

2026-05-10

arXiv (preprint)

DASB - Discrete Audio and Speech Benchmark

Pooneh Mousavi

Jarod Duret

Darius Petermann

Anastasia Kuznetsova

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling mult… (see more)imodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

2026-04-12

Transactions on Machine Learning Research (accepted)

openreview.net

Self-Routing: Parameter-Free Expert Routing from Hidden States

Jama Hussein Mohamud

Drew Wagner

Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a lear… (see more)ned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary for MoE routing. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on language modeling across different expert counts and model scales, and on ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17\% higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.

2026-03-31

arXiv (preprint)

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chai… (see more)ns remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

2026-03-18

arXiv (preprint)

Cough acoustic analysis using artificial intelligence for COVID-19 detection: A comparative study of patient cohorts from Lima, Peru and Montreal, Canada

A. Zimmer

Vijay Ravi

Patricia Espinoza-Lopez

George P. Kafentzis

Samira Abbasgholizadeh Rahimi

Madhukar Pai

Cesar Ugarte-Gil

Serge Lapierre

2026-03-17

Annals of Epidemiology (published)

LL-SDR: Low-Latency Speech enhancement through Discrete Representations

Jingyi Li

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autore… (see more)gressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.

2026-03-09

arXiv (preprint)

WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation

Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to sp… (see more)eech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference. Demo samples are available at https://lucadellalib.github.io/wavslm-web/.

2026-03-04

arXiv (preprint)

Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization

Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete to… (see more)kens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.

2026-01-29

Open MIND (preprint)