Ian Porada

Investigating Failures to Generalize for Coreference Resolution Models

Kaheer Suleman

Adam Trischler

Coreference resolution models are often evaluated on multiple datasets. Datasets vary, however, in how coreference is realized -- i.e., how … (voir plus)the theoretical concept of coreference is operationalized in the dataset -- due to factors such as the choice of corpora and annotation guidelines. We investigate the extent to which errors of current coreference resolution models are associated with existing differences in operationalization across datasets (OntoNotes, PreCo, and Winogrande). Specifically, we distinguish between and break down model performance into categories corresponding to several types of coreference, including coreferring generic mentions, compound modifiers, and copula predicates, among others. This break down helps us investigate how state-of-the-art models might vary in their ability to generalize across different coreference types. In our experiments, for example, models trained on OntoNotes perform poorly on generic mentions and copula predicates in PreCo. Our findings help calibrate expectations of current coreference resolution models; and, future work can explicitly account for those types of coreference that are empirically associated with poor generalization when developing models.

2024-08-01

Findings of the Association for Computational Linguistics ACL 2024 (publié)

doi.org

arxiv.org

Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge

Ian Porada

Alessandro Sordoni

Jackie Cheung

2022-07-01

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (publié)

doi.org

arxiv.org

Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge

Ian Porada

Alessandro Sordoni

Jackie Cheung

Transformer models pre-trained with a masked-language-modeling objective (e.g., BERT) encode commonsense knowledge as evidenced by behaviora… (voir plus)l probes; however, the extent to which this knowledge is acquired by systematic inference over the semantics of the pre-training corpora is an open question. To answer this question, we selectively inject verbalized knowledge into the pre-training minibatches of BERT and evaluate how well the model generalizes to supported inferences after pre-training on the injected knowledge. We find generalization does not improve over the course of pre-training BERT from scratch, suggesting that commonsense knowledge is acquired from surface-level, co-occurrence patterns rather than induced, systematic reasoning.

2021-12-16

ArXiv (preprint)

doi.org

arxiv.org

Modeling Event Plausibility with Consistent Conceptual Abstraction

Ian Porada

Kaheer Suleman

Adam Trischler

Jackie Cheung

2021-06-01

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (publié)

doi.org

arxiv.org

ADEPT: An Adjective-Dependent Plausibility Task

Ali Emami

Ian Porada

Alexandra Olteanu

Kaheer Suleman

Adam Trischler

Jackie Cheung

2021-01-01

Annual Meeting of the Association for Computational Linguistics (publié)

doi.org

META-Learning State-based Eligibility Traces for More Sample-Efficient Policy Evaluation

Mingde Zhao

Sitao Luan

Ian Porada

Xiao-Wen Chang

Doina Precup

Temporal-Difference (TD) learning is a standard and very successful reinforcement learning approach, at the core of both algorithms that lea… (voir plus)rn the value of a given policy, as well as algorithms which learn how to improve policies. TD-learning with eligibility traces provides a way to boost sample efficiency by temporal credit assignment, i.e. deciding which portion of a reward should be assigned to predecessor states that occurred at different previous times, controlled by a parameter

2020-01-01

AAMAS (publié)

dblp.uni-trier.de

Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text

Ian Porada

Kaheer Suleman

Jackie Cheung

2019-11-01

Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (publié)

doi.org

arxiv.org