Dragos Secrieru

Collaborating Alumni - Université de Montréal

Supervisor

Yoshua Bengio

Research Topics

Deep Learning

Generative Models

Publications

Sliding Window Recurrences for Sequence Models

Dragos Secrieru

Garyk Brixi

Yoshua Bengio

Taiji Suzuki

Michael Poli

Stefano Massaroli

Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical deco… (see more)mposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.

2025-11-30

arXiv (published)

doi.org

arxiv.org

Delta-AI: Local objectives for amortized inference in sparse graphical models

Jean-Pierre R. Falet

Hae Beom Lee

Chen Sun