Portrait de Shawn Tan n'est pas disponible

Shawn Tan

tanjings@mila.quebec
Doctorat - Université de Montréal
Superviseur⋅e principal⋅e
Aaron Courville
Site web
Github
Google Scholar

Publications

Scattered Mixture-of-Experts Implementation
Shawn Tan
Yikang Shen
Rameswar Panda
Aaron Courville
We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and o… (voir plus)vercoming some of the limitations to improve inference and training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input. We introduce ParallelLinear, the main component we use to build our implementation and the various kernels used to speed up the operation. We benchmark our implementation against Megablocks, and show that it enables a higher throughput and lower memory footprint. We also show how ParallelLinear enables extension of the Mixture-of-Experts concept by demonstrating with an implementation of Mixture of Attention.
2024-03-13
ArXiv (prépublication)
doi.org
arxiv.org
Sparse Universal Transformer
Shawn Tan
Yikang Shen
Zhenfang Chen
Aaron Courville
Chuang Gan
The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers and is Turing-complete under certain… (voir plus) assumptions. Empirical evidence also shows that UTs have better compositional generalization than Vanilla Transformers (VTs) in formal language tasks. The parameter-sharing also affords it better parameter efficiency than VTs. Despite its many advantages, most state-of-the-art NLP systems use VTs as their backbone model instead of UTs. This is mainly because scaling UT parameters is more compute and memory intensive than scaling up a VT. This paper proposes the Sparse Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE) to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. Experiments show that SUT combines the best of both worlds, achieving strong generalization results on formal language tasks (Logical inference and CFQ) and impressive parameter and computation efficiency on standard natural language benchmarks like WMT'14.
2023-10-07
EMNLP/2023/Conference (accepté)
doi.org
openreview.net
Unsupervised Dependency Graph Network
Yikang Shen
Shawn Tan
Alessandro Sordoni
Peng Li
Jie Zhou
Aaron Courville
2022-05-01
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (publié)
doi.org
Unsupervised Dependency Graph Network
Yikang Shen
Shawn Tan
Alessandro Sordoni
Peng Li
Jie Zhou
Aaron Courville
Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, s… (voir plus)ome self-attention heads correspond well to individual dependency types. Inspired by these developments, we propose a new competitive mechanism that encourages these attention heads to model different dependency relations. We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task. Experiment results show that UDGN achieves very strong unsupervised dependency parsing performance without gold POS tags and any other external information. The competitive gated heads show a strong correlation with human-annotated dependency types. Furthermore, the UDGN can also achieve competitive performance on masked language modeling and sentence textual similarity tasks.
2022-01-01
Annual Meeting of the Association for Computational Linguistics (published)
doi.org
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
Yikang Shen
Shawn Tan
Alessandro Sordoni
Siva Reddy
Aaron Courville
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization proble… (voir plus)ms and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing, and syntactic generalization tests while using fewer parameters than other models.
2021-06-01
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (publié)
doi.org
arxiv.org
Explicitly Modeling Syntax in Language Model improves Generalization
Yikang Shen
Shawn Tan
Alessandro Sordoni
Siva Reddy
Aaron Courville
Syntax is fundamental to our thinking about language. Although neural networks are very successful in many tasks, they do not explicitly mod… (voir plus)el syntactic structure. Failing to capture the structure of inputs could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with a one-step look-ahead parser and maintains the conditional probability setting of the standard language model. Experiments show that SOM can achieve strong results in language modeling and syntactic generalization tests, while using fewer parameters then other models.
2020-10-21
arXiv.org (prépublication)
dblp.uni-trier.de