Portrait de Francesco Verdini n'est pas disponible

Francesco Verdini

Alumni

Publications

HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
Artem Ploujnikov
Francesco Verdini
Samir Sadok
Mirco Ravanelli
Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilitie… (voir plus)s into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps.
2026-06-25
arXiv (prépublication)
doi.org
arxiv.org