Portrait of Cem Subakan

Cem Subakan

Associate Academic Member

cem.subakan@mila.quebec

Assistant Professor, Université Laval, Department of Computer Science and Software Engineering

Affiliate Assistant Professor, Concordia University, Gina Cody School of Engineering and Computer Science

Research Topics

Multimodal Learning

Biography

Cem Subakan is an assistant professor in the Computer Science and Software Engineering Department at Université Laval, and an affiliate assistant professor in the Computer Science and Software Engineering Department at Concordia University. He is also an associate academic member of Mila – Quebec Artificial Intelligence Institute. After receiving his PhD in computer science from the University of Illinois at Urbana-Champaign (UIUC), Subakan did a postdoc at Mila. He serves as a reviewer for many conferences including NeurIPS, ICML, ICLR, ICASSP and MLSP, as well as for journals, such as IEEE Signal Processing Letters and IEEE Transactions on Audio, Speech, and Language Processing. His principal research interest is machine learning for speech and audio. More specifically, he works on deep learning for source separation and speech enhancement under realistic conditions, neural network interpretability, continual learning and multi-modal learning.

Subakan was awarded the Best Student Paper Award at the 2017 IEEE Machine Learning for Signal Processing Conference, and also obtained a Sabura Muroga Fellowship from UIUC’s Department of Computer Science. He is a core contributor to the SpeechBrain project, leading the speech separation component.

Current Students

Matthieu Cervera

Collaborating researcher

Co-supervisor :

Mirco Ravanelli

matthieu.cervera@mila.quebec

Luca Della Libera

PhD - Concordia University

Principal supervisor :

Mirco Ravanelli

luca.dellalibera@mila.quebec

Maab Elrashid Ahmed Mohamed

PhD - Concordia University

Principal supervisor :

Mirco Ravanelli

maab-elrashid.ahmed-mohamed@mila.quebec

PhD - Université Laval

Co-supervisor :

Laurent Charlin

shubham.gupta@mila.quebec

Collaborating researcher - Université Laval

Co-supervisor :

Mirco Ravanelli

f20211732@pilani.bits-pilani.ac.in

PhD - Université Laval

Co-supervisor :

Mirco Ravanelli

jihoon.jeong@mila.quebec

Aravind Krishnan

Collaborating Alumni - Saarland University

aravind.krishnan@mila.quebec

Eleonora Mancini

Collaborating Alumni - Université de Montréal

Co-supervisor :

Mirco Ravanelli

eleonora.mancini@mila.quebec

Francesco Paissan

PhD - Université Laval

Co-supervisor :

Aishwarya Agrawal

francescopaissan@gmail.com

Blog Posts

Visual of FocalCodec,nouvelle méthode pour compresser la parole sans sacrifier la qualité, en vue d'obtenir des LLM multimodaux plus efficaces.

January 23, 2026

FocalCodec: Giving LLMs Ears and a Voice at Ultra-Low Bitrates

by

Luca Della Libera

Francesco Paissan

Mirco Ravanelli

Read the article

Publications

DASB - Discrete Audio and Speech Benchmark

Jarod Duret

Darius Petermann

Artem Ploujnikov

Luca Della Libera

Anastasia Kuznetsova

Mirco Ravanelli

Duret, Jarod

Kuznetsova, Anastasia

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling mult… (see more)imodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

2026-12-30

OpenReview (published)

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Mirco Ravanelli

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chai… (see more)ns remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

2026-12-30

OpenReview (published)

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Mirco Ravanelli

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling mean… (see more)s that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

2026-10-06

arXiv (preprint)

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

Amir Ivry

Mirco Ravanelli

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We… (see more) propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.

2026-09-29

arXiv (preprint)

Hierarchical Retrieval at Scale: Bridging Interpretability and Efficiency

Tianyi Chen

Perouz Taslakian

Valentina Zantedeschi

Information retrieval is a core component of many intelligent systems as it enables conditioning of outputs on new and large-scale datasets.… (see more) While effective, the standard practice of encoding data into high-dimensional representations for similarity search entails large memory and compute footprints, and also makes it hard to inspect the inner workings of the system. Hierarchical retrieval methods offer an interpretable alternative by organizing data at multiple granular levels, yet do not match the efficiency and performance of flat retrieval approaches. In this paper, we propose Retreever, a tree-based method that makes hierarchical retrieval viable at scale by directly optimizing its structure for retrieval performance while naturally providing transparency through meaningful semantic groupings. Our method offers the flexibility to balance cost and utility by indexing data using representations from any tree level. We show that Retreever delivers strong coarse (intermediate levels) and fine representations (terminal level), while achieving the highest retrieval accuracy at the lowest latency among hierarchical methods. These results demonstrate that this family of techniques is viable in practical applications.

2026-09-28

International Conference on Machine Learning (published)

proceedings.mlr.press

proceedings.mlr.press

SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression

Benjamin Léger

Kazem Meidani

Christian Gagné

Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is centr… (see more)al to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical embeddings, reliably distinguish the effects of local edits that the original alignment cannot, and transfer to external SR corpora.

2026-09-24

arXiv (preprint)

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

Salman Hussain Ali

Mirco Ravanelli

Çağatay Yıldız

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Co… (see more)ntinued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams. This is a poor fit for time-incremental updates on web-scale crawls, where successive snapshots share substantial URL overlap by design. We study time-incremental CPT in this realistic regime: continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families (OLMo2, Llama-3.1/3.2, Gemma-3-1B) and four parameter scales (1B-3B-7B-8B). We organize our findings around four practical questions. (i) Is knowledge acquired? Yes, but heterogeneously, and without catastrophic forgetting: five of six models also improve on pre-cutoff factual recall, and the gains track pretraining saturation (driven primarily by token budget per parameter). (ii) What does it cost? Almost nothing: the macro-average across a thirteen-task suite stays within 0.01 of the base for every model. (iii) What is the recipe? Data quality dominates quantity (a curated 6B-token slice matches a broader 40B one); the optima for knowledge acquisition and general capability are separated by roughly an order of magnitude in learning rate; and LoRA at sufficient rank matches full CPT. (iv) Does it survive deployment? CPT gains transfer through SFT, while DPO's effect is family-dependent. Together, these results paint a more optimistic picture of time-incremental CPT than the prior continual learning literature suggests.

2026-09-19

arXiv (preprint)

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Luca Della Libera

Mirco Ravanelli

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, … (see more)reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

2026-09-09

arXiv (preprint)

Enhancing Audio Reasoning via Semantic Summary Prediction

Francesco Bonzi

Mirco Ravanelli

Large Audio Language Models (LALMs) perform well on complex question answering but often show a reasoning gap, where explicit Chain-of-Thoug… (see more)ht (CoT) reduces accuracy compared to direct answers. We hypothesize that long reasoning sequences shift attention away from the audio input. To address this, we propose SPARE (Semantic Prediction for Audio REasoning), which introduces a register token aligned with the final conclusion using a cosine similarity loss with a Sentence-BERT embedding. This conditions the model's latent space with the target semantic goal before reasoning begins. Experiments on MMAU and MMAR with SALMONN show improved zero-shot reasoning and stronger early attention to audio without additional inference cost.

2026-08-06

arXiv (preprint)

Controllable and Content-Based Recommendations

Emiliano Penaloza

Mirco Ravanelli

Laurent Charlin

Traditional recommendation systems rely on latent (dense) representations, making them difficult to interpret and control. We propose the Co… (see more)ntrollable and Content-Based Recommendations (CCBR) framework, which builds its recommendations from textual user profile representations. CCBR plugs into collaborative filtering models and introduces controllability via text bottlenecks. We show that CCBR enables text-based and multimodal interventions, allowing users to steer the model towards the directions they prefer. Different from existing controllable recommendation systems, CCBR infers the text summaries directly from item contents (images, audio or video). Across image-, audio-, and video-based datasets, we demonstrate that the proposed framework obtains competitive model performance with standard (latent-representation) models while providing controllable model summaries via text. The model also outperforms TEARS, a recent baseline for controllable recommendation systems. Through systematic interventions, we demonstrate the efficacy of the user steering mechanism.

2026-07-22

arXiv (preprint)

PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems

Nazanin Mohammadi Sepahvand

Abhinav Kumar

Pierre-Andre Noel

Perouz Taslakian

Eugene Bagdasarian

Valentina Zantedeschi

As LLM agents evolve from single-user assistants into shared organizational infrastructure, new privacy risks emerge: inappropriate informat… (see more)ion may not only be exposed through outputs for external recipients, but also internally across users through inter-agent messages, shared memory and agents. These data spillage risks are not captured by existing privacy benchmarks grounded in contextual integrity (CI) as they focus primarily on either single-user settings or interactions between independently owned agents. We introducePiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks with dual CI annotations: whether an information is appropriate for the task, and which users may legitimately access it. This enables direct measurement of cross-user spillage across agentic system components and interfaces, such as outputs, inter-agent communication, and memory. PiSAsis system-agnostic and supports evaluation across different agent topologies and memory regimes. We find that, although system design improves CI compliance, results are bottlenecked by incorrect LLM judgment calls: even state-of-the-art models fail to reliably filter inappropriate content or restrict transmission to authorized users. Our findings underscore the need for privacy-preserving strategies, beyond those studied in this work.

2026-07-05

arXiv (preprint)

Exploring Token-Space Manipulation in Latent Audio Tokenizers

Francesco Paissan

Luca Della Libera

Mirco Ravanelli

Neural audio codecs provide compact discrete representations for speech generation and manipulation. However, most codecs organize tokens as… (see more) frame-level sequences, making it difficult to study or intervene on global factors of variation. In this work, we propose the Latent Audio Tokenizer for Token-space Editing (LATTE) that appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding. This design produces a compact, non-temporally aligned bottleneck in which each token can aggregate global information across the full utterance. We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings while enabling simple token-space interventions. In particular, we find that swapping selected latent token positions between utterances can modify global attributes, such as speaker identity and background noise, and we evaluate these interventions on voice conversion and denoising tasks. Our results suggest that compact latent audio tokenizers can support controllable audio manipulation without supervision in task-specific editing models.

2026-05-10

arXiv (preprint)