Portrait de Dragos Secrieru n'est pas disponible

Dragos Secrieru

Alumni

Publications

Sliding Window Recurrences for Sequence Models
Dragos Secrieru
Garyk Brixi
Yoshua Bengio
Taiji Suzuki
Michael Poli
Stefano Massaroli
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical deco… (voir plus)mposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.
2025-12-14
arXiv (prépublication)
doi.org
arxiv.org
Delta-AI: Local Objectives for Amortized Inference in Sparse Graphical Models
Jean-Pierre Falet
Hae Beom Lee
Nikolay Malkin
Chen Sun
Dragos Secrieru
Dinghuai Zhang
Guillaume Lajoie
Yoshua Bengio
We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call …
2024-05-06
International Conference on Learning Representations (Accept (poster))
doi.org
openreview.net