Publications

StereoSet: Measuring stereotypical bias in pretrained language models

Moin Nadeem

Anna Bethke

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athlet… (see more)ic. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real-world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing literature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data 2) current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. We contrast both stereotypical bias and language modeling ability of popular models like BERT, GPT-2, RoBERTa, and XLnet. We show that these models exhibit strong stereotypical biases. Our data and code are available at https://stereoset.mit.edu.

2021-07-31

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (published)

doi.org

arxiv.org

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

Yikang Shen

Yi Tay

Che Zheng

Dara Bahri

Donald Metzler

Aaron Courville

There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and t… (see more)he constituency grammar that models the assembly of one or several corresponded words. While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable manner, through a novel dependency-constrained self-attention mechanism. Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.

2021-07-31

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (published)

doi.org

openreview.net

Supervised multi-specialist topic model with applications on large-scale electronic health record data

Ziyang Song

Xavier Sumba Toral

Yixin Xu

Aihua Liu

Liming Guo

Guido Powell

Aman Verma

David Buckeridge

Ariane Marelli

Yue Li

Motivation: Electronic health record (EHR) data provides a new venue to elucidate disease comorbidities and latent phenotypes for precision … (see more)medicine. To fully exploit its potential, a realistic data generative process of the EHR data needs to be modelled. We present MixEHR-S to jointly infer specialist-disease topics from the EHR data. As the key contribution, we model the specialist assignments and ICD-coded diagnoses as the latent topics based on patient's underlying disease topic mixture in a novel unified supervised hierarchical Bayesian topic model. For efficient inference, we developed a closed-form collapsed variational inference algorithm to learn the model distributions of MixEHR-S. We applied MixEHR-S to two independent large-scale EHR databases in Quebec with three targeted applications: (1) Congenital Heart Disease (CHD) diagnostic prediction among 154,775 patients; (2) Chronic obstructive pulmonary disease (COPD) diagnostic prediction among 73,791 patients; (3) future insulin treatment prediction among 78,712 patients diagnosed with diabetes as a mean to assess the disease exacerbation. In all three applications, MixEHR-S conferred clinically meaningful latent topics among the most predictive latent topics and achieved superior target prediction accuracy compared to the existing methods, providing opportunities for prioritizing high-risk patients for healthcare services. MixEHR-S source code and scripts of the experiments are freely available at https://github.com/li-lab-mcgill/mixehrS

2021-07-31

Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (published)

doi.org

arxiv.org

A Survey of Data Augmentation Approaches for NLP

Steven Y. Feng

Varun Gangal

Jason Wei

Sarath Chandar

Soroush Vosoughi

Teruko Mitamura

Eduard Hovy

Data augmentation has recently seen increased interest in NLP due to more work in low-resource domains, new tasks, and the popularity of lar… (see more)ge-scale neural networks that require large amounts of training data. Despite this recent upsurge, this area is still relatively underexplored, perhaps due to the challenges posed by the discrete nature of language data. In this paper, we present a comprehensive and unifying survey of data augmentation for NLP by summarizing the literature in a structured manner. We first introduce and motivate data augmentation for NLP, and then discuss major methodologically representative approaches. Next, we highlight techniques that are used for popular NLP applications and tasks. We conclude by outlining current challenges and directions for future research. Overall, our paper aims to clarify the landscape of existing literature in data augmentation for NLP and motivate additional work in this area. We also present a GitHub repository with a paper list that will be continuously updated at https://github.com/styfeng/DataAug4NLP

2021-07-31

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (published)

doi.org

arxiv.org

Temporal Profiles of Social Attention Are Different Across Development in Autistic and Neurotypical People.

Teresa Del Bianco

Luke Mason

Tony Charman

Julianne Tillman

Eva Loth

Hannah Hayward

F. Shic

Jan K. Buitelaar

Mark Johnson

Emily J. H. Jones

Jumana Ahmad

Sara Ambrosino

Tobias Banaschewski

Simon Baron-Cohen

Sarah Baumeister

Christian Beckmann

Sven Bölte

Thomas Bourgeron

Carsten Bours

M. Brammer … (see 46 more)

Daniel Brandeis

Claudia Brogna

Yvette de Bruijn

Ineke Cornelissen

Daisy Crawley

Flavio Dell’Acqua

Guillaume Dumas

Sarah Durston

Christine Ecker

Jessica Faulkner

Vincent Frouin

Pilar Garcés

David Goyard

Lindsay Ham

Joerg F. Hipp

Rosemary Holt

Meng-Chuan Lai

Xavier Liogier D’ardhuy

Michael V. Lombardo

David J. Lythgoe

René Mandl

Andre Marquand

Maarten Mennes

Andreas Meyer-Lindenberg

Carolin Moessnang

Nico Mueller

Declan Murphy

Beth Oakley

Larry O’Dwyer

Marianne Oldehinkel

Bob Oranje

Gahan Pandina

Antonio Persico

Barbara Ruggeri

Amber N. V. Ruigrok

Jessica Sabet

Roberto Sacco

Antonia San José Cáceres

Emily Simonoff

Will Spooren

Roberto Toro

Heike Tost

Jack Waldman

Steve C. R. Williams

Caroline Wooldridge

Marcel P. Zwiers

2021-07-31

Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (published)

doi.org

Why do sleep disorders belong to mental disorder classifications? A network analysis of the "Sleep-Wake Disorders" section of the DSM-5.

Christophe Gauld

Régis Lopez

Charles Morin

Julien Maquet

Aileen McGonigal

Pierre A. GEOFFROY

Eric Fakra

Pierre Philip

Guillaume Dumas

Jean‐Arthur Micoulaud‐Franchi

2021-07-31

Journal of Psychiatric Research (published)

doi.org

Atlas-Based Quantification of DTI Measures in a Typically Developing Pediatric Spinal Cord

Shiva Shahrampour

Benjamin De Leener

Mahdi Alizadeh

D. Middleton

Laura Krisa

Adam E. Flanders

S. Faro

Julien Cohen-Adad

F. Mohamed

2021-07-28

American Journal of Neuroradiology (published)

doi.org

A Unified Few-Shot Classification Benchmark to Compare Transfer and Meta Learning Approaches

Vincent Dumoulin

Neil Houlsby

Utku Evci

Xiaohua Zhai

Ross Goroshin

Sylvain Gelly

Hugo Larochelle

Meta and transfer learning are two successful families of approaches to few-shot 1 learning. Despite highly related goals, state-of-the-art … (see more)advances in each family are 2 measured largely in isolation of each other. As a result of diverging evaluation 3 norms, a direct or thorough comparison of different approaches is challenging. 4 To bridge this gap, we introduce a few-shot classiﬁcation evaluation protocol 5 named VTAB+MD with the explicit goal of facilitating sharing of insights from 6 each community. We demonstrate its accessibility in practice by performing a 7 cross-family study of the best transfer and meta learners which report on both a 8 large-scale meta-learning benchmark (Meta-Dataset, MD), and a transfer learning 9 benchmark (Visual Task Adaptation Benchmark, VTAB). We ﬁnd that, on average, 10 large-scale transfer methods (Big Transfer, BiT) outperform competing approaches 11 on MD, even when trained only on ImageNet. In contrast, meta-learning approaches 12 struggle to compete on VTAB when trained and validated on MD. However, BiT 13 is not without limitations, and pushing for scale does not improve performance 14 on highly out-of-distribution MD tasks. We hope that this work contributes to 15 accelerating progress on few-shot learning research. 16

2021-07-28

NeurIPS.cc/2021/Track/Datasets_and_Benchmarks/Round1 (accepted)

openreview.net

Models of Computational Profiles to Study the Likelihood of DNN Metamorphic Test Cases

Ettore Merlo

Mira Marhaba

Foutse Khomh

Houssem Ben Braiek

Giuliano Antoniol

Neural network test cases are meant to exercise different reasoning paths in an architecture and used to validate the prediction outcomes. I… (see more)n this paper, we introduce "computational profiles" as vectors of neuron activation levels. We investigate the distribution of computational profile likelihood of metamorphic test cases with respect to the likelihood distributions of training, test and error control cases. We estimate the non-parametric probability densities of neuron activation levels for each distinct output class. Probabilities are inferred using training cases only, without any additional knowledge about metamorphic test cases. Experiments are performed by training a network on the MNIST Fashion library of images and comparing prediction likelihoods with those obtained from error control-data and from metamorphic test cases. Experimental results show that the distributions of computational profile likelihood for training and test cases are somehow similar, while the distribution of the random-noise control-data is always remarkably lower than the observed one for the training and testing sets. In contrast, metamorphic test cases show a prediction likelihood that lies in an extended range with respect to training, tests, and random noise. Moreover, the presented approach allows the independent assessment of different training classes and experiments to show that some of the classes are more sensitive to misclassifying metamorphic test cases than other classes. In conclusion, metamorphic test cases represent very aggressive tests for neural network architectures. Furthermore, since metamorphic test cases force a network to misclassify those inputs whose likelihood is similar to that of training cases, they could also be considered as adversarial attacks that evade defenses based on computational profile likelihood evaluation.

2021-07-27

arXiv (Cornell University) (preprint)

doi.org

arxiv.org

Automated Data-Driven Generation of Personalized Pedagogical Interventions in Intelligent Tutoring Systems

Ekaterina Kochmar

Dung D. Vu

Robert Belfer

Varun Gupta

Iulian V. Serban

Joelle Pineau

2021-07-26