Publications

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
Yikang Shen
Shawn Tan
Alessandro Sordoni
Siva Reddy
Aaron Courville
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization proble… (see more)ms and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing, and syntactic generalization tests while using fewer parameters than other models.
2021-06-01
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (published)
doi.org
arxiv.org
Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management
Zhengxu Hou
Bang Liu
Ruihui Zhao
Zijing Ou
Yafei Liu
Xi Chen
Yefeng Zheng
For task-oriented dialog systems, training a Reinforcement Learning (RL) based Dialog Management module suffers from low sample efficiency a… (see more)nd slow convergence speed due to the sparse rewards in RL. To solve this problem, many strategies have been proposed to give proper rewards when training RL, but their rewards lack interpretability and cannot accurately estimate the distribution of state-action pairs in real dialogs. In this paper, we propose a multi-level reward modeling approach that factorizes a reward into a three-level hierarchy: domain, act, and slot. Based on inverse adversarial reinforcement learning, our designed reward model can provide more accurate and explainable reward signals for state-action pairs. Extensive evaluations show that our approach can be applied to a wide range of reinforcement learning-based dialog systems and significantly improves both the performance and the speed of convergence.
2021-06-01
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (published)
doi.org
arxiv.org
Modeling Event Plausibility with Consistent Conceptual Abstraction
Ian Porada
Kaheer Suleman
Adam Trischler
Jackie Cheung
2021-06-01
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (published)
doi.org
arxiv.org
Understanding by Understanding Not: Modeling Negation in Language Models
Arian Hosseini
Siva Reddy
Dzmitry Bahdanau
(Rex) Devon Hjelm
Alessandro Sordoni
Aaron Courville
Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language mode… (see more)ls often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.
2021-06-01
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (published)
doi.org
arxiv.org
On the benefits of representation regularization in invariance based domain generalization
Changjian Shui
Boyu Wang
Christian Gagné
2021-05-30
ArXiv (preprint)
doi.org
arxiv.org
Gotta Go Fast When Generating Data with Score-Based Models
Alexia Jolicoeur-Martineau
Ke Li
Rémi Piché-Taillefer
Tal Kachman
Ioannis Mitliagkas
Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These ap… (see more)proaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly - they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.
2021-05-28
ArXiv (preprint)
openreview.net
openreview.net
Noised Consistency Training for Text Summarization
J. Liu
Qianren Mao
Bang Liu
Hao Peng
Hongdong Zhu
Jianxin Li
Neural abstractive summarization methods often require large quantities of labeled training data. However, labeling large amounts of summari… (see more)zation data is often prohibitive due to time, financial, and expertise constraints, which has limited the usefulness of summarization systems to practical applications. In this paper, we argue that this limitation can be overcome by a semi-supervised approach: consistency training which is to leverage large amounts of unlabeled data to improve the performance of supervised learning over a small corpus. The consistency regularization semi-supervised learning can regularize model predictions to be invariant to small noise applied to input articles. By adding noised unlabeled corpus to help regularize consistency training, this framework obtains comparative performance without using the full dataset. In particular, we have verified that leveraging large amounts of unlabeled data decently improves the performance of supervised learning over an insufficient labeled dataset.
2021-05-28
ArXiv (preprint)
arxiv.org
arxiv.org
Learning Brain Dynamics With Coupled Low-Dimensional Nonlinear Oscillators and Deep Recurrent Networks
Germán Abrevaya
Guillaume Dumas
Aleksandr Y. Aravkin
Peng Zheng
Jean-Christophe Gagnon-Audet
James Kozloski
Pablo Polosecki
Guillaume Lajoie
David Cox
Silvina Ponce Dawson
Guillermo Cecchi
Irina Rish
Many natural systems, especially biological ones, exhibit complex multivariate nonlinear dynamical behaviors that can be hard to capture by … (see more)linear autoregressive models. On the other hand, generic nonlinear models such as deep recurrent neural networks often require large amounts of training data, not always available in domains such as brain imaging; also, they often lack interpretability. Domain knowledge about the types of dynamics typically observed in such systems, such as a certain type of dynamical systems models, could complement purely data-driven techniques by providing a good prior. In this work, we consider a class of ordinary differential equation (ODE) models known as van der Pol (VDP) oscil lators and evaluate their ability to capture a low-dimensional representation of neural activity measured by different brain imaging modalities, such as calcium imaging (CaI) and fMRI, in different living organisms: larval zebrafish, rat, and human. We develop a novel and efficient approach to the nontrivial problem of parameters estimation for a network of coupled dynamical systems from multivariate data and demonstrate that the resulting VDP models are both accurate and interpretable, as VDP's coupling matrix reveals anatomically meaningful excitatory and inhibitory interactions across different brain subsystems. VDP outperforms linear autoregressive models (VAR) in terms of both the data fit accuracy and the quality of insight provided by the coupling matrices and often tends to generalize better to unseen data when predicting future brain activity, being comparable to and sometimes better than the recurrent neural networks (LSTMs). Finally, we demonstrate that our (generative) VDP model can also serve as a data-augmentation tool leading to marked improvements in predictive accuracy of recurrent neural networks. Thus, our work contributes to both basic and applied dimensions of neuroimaging: gaining scientific insights and improving brain-based predictive models, an area of potentially high practical importance in clinical diagnosis and neurotechnology.
2021-05-26
Neural Computation (published)
doi.org
Inferring global-scale temporal latent topics from news reports to predict public health interventions for COVID-19
Zhi Wen
Guido Powell
Imane Chafi
David Buckeridge
Y. K. Li
2021-05-24
Patterns (published)
doi.org
Artificial intelligence in nursing: Priorities and opportunities from an international invitational think‐tank of the Nursing and Artificial Intelligence Leadership Collaborative
Charlene Esteban Ronquillo
Laura‐Maria Peltonen
Lisiane Pruinelli
Charlene H Chu
Suzanne Bakken
Ana Beduschi
Kenrick Cato
Nicholas Hardiker
Alain Junger
Martin Michalowski
Rune Nyrup
Samira Abbasgholizadeh-Rahimi
Donald Nigel Reed
Tapio Salakoski
Sanna Salanterä
Nancy Walton
Patrick Weber
Thomas Wiegand
Maxim Topaz
2021-05-18
Journal of Advanced Nursing (published)
doi.org
Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems
Matt Grenander
Robert Belfer
Ekaterina Kochmar
Iulian V. Serban
Franccois St-Hilaire
Jackie Cheung
We explore creating automated, personalized feedback in an intelligent tutoring system (ITS). Our goal is to pinpoint correct and incorrect … (see more)concepts in student answers in order to achieve better student learning gains. Although automatic methods for providing personalized feedback exist, they do not explicitly inform students about which concepts in their answers are correct or incorrect. Our approach involves decomposing students answers using neural discourse segmentation and classification techniques. This decomposition yields a relational graph over all discourse units covered by the reference solutions and student answers. We use this inferred relational graph structure and a neural classifier to match student answers with reference solutions and generate personalized feedback. Although the process is completely automated and data-driven, the personalized feedback generated is highly contextual, domain-aware and effectively targets each student's misconceptions and knowledge gaps. We test our method in a dialogue-based ITS and demonstrate that our approach results in high-quality feedback and significantly improved student learning gains.
2021-05-18
Proceedings of the AAAI Conference on Artificial Intelligence (published)
doi.org
arxiv.org
DIBS: Diversity inducing Information Bottleneck in Model Ensembles
Samarth Sinha
Homanga Bharadhwaj
Anirudh Goyal
Hugo Larochelle
Animesh Garg
Florian Shkurti
2021-05-18
Proceedings of the AAAI Conference on Artificial Intelligence (published)
doi.org
arxiv.org