Dragos Secrieru

Collaborateur·rice alumni - UdeM

Superviseur⋅e principal⋅e

Yoshua Bengio

Sujets de recherche

Apprentissage profond

Modèles génératifs

Publications

Sliding Window Recurrences for Sequence Models

Dragos Secrieru

Garyk Brixi

Yoshua Bengio

Taiji Suzuki

Michael Poli

Stefano Massaroli

Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical deco… (voir plus)mposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.

2025-11-30

arXiv (publié)

doi.org

arxiv.org

Delta-AI: Local Objectives for Amortized Inference in Sparse Graphical Models

Hae Beom Lee

Chen Sun