Portrait of Timothy O'Donnell

Timothy O'Donnell

Core Academic Member
odonnelt@mila.quebec
Canada CIFAR AI Chair
Assistant Professor, McGill University, Department of Linguistics
Website

Biography

I am an assistant professor in the Department of Linguistics at McGill University.

In my research, I develop mathematical models of language generalization, learning and processing. My research draws on experimental methods from psychology, formal modelling techniques from natural language processing, theoretical tools from linguistics, and problems in all three of these areas.

Current Students

Publications

Systematic Generalization by Finetuning? Analyzing Pretrained Language Models Using Constituency Tests
Aishik Chakraborty
Jackie CK Cheung
Timothy O'Donnell
Constituents are groups of words that behave as a syntactic unit. Many linguistic phenomena (e.g., question formation, diathesis alternation… (see more)s) require the manipulation and rearrangement of constituents in a sentence. In this paper, we investigate how different finetuning setups affect the ability of pretrained sequence-to-sequence language models such as BART and T5 to replicate constituency tests — transformations that involve manipulating constituents in a sentence. We design multiple evaluation settings by varying the combinations of constituency tests and sentence types that a model is exposed to during finetuning. We show that models can replicate a linguistic transformation on a specific type of sentence that they saw during finetuning, but performance degrades substantially in other settings, showing a lack of systematic generalization. These results suggest that models often learn to manipulate sentences at a surface level unrelated to the constituent-level syntactic structure, for example by copying the first word of a sentence. These results may partially explain the brittleness of pretrained language models in downstream tasks.
2023-12-01
Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP (published)
doi.org
The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing
Jacob Louis Hoover
Morgan Sonderegger
Steven T. Piantadosi
Timothy O'Donnell
Abstract Words that are more surprising given context take longer to process. However, no incremental parsing algorithm has been shown to di… (see more)rectly predict this phenomenon. In this work, we focus on a class of algorithms whose runtime does naturally scale in surprisal—those that involve repeatedly sampling from the prior. Our first contribution is to show that simple examples of such algorithms predict runtime to increase superlinearly with surprisal, and also predict variance in runtime to increase. These two predictions stand in contrast with literature on surprisal theory (Hale, 2001; Levy, 2008a) which assumes that the expected processing cost increases linearly with surprisal, and makes no prediction about variance. In the second part of this paper, we conduct an empirical study of the relationship between surprisal and reading time, using a collection of modern language models to estimate surprisal. We find that with better language models, reading time increases superlinearly in surprisal, and also that variance increases. These results are consistent with the predictions of sampling-based algorithms.
2023-06-01
Open Mind (published)
doi.org
The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation
Kushal Arora
Timothy O'Donnell
Doina Precup
Jason Aaron Edward Weston
Jackie C.K.Cheung
State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story gen… (see more)eration, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of vocabulary diversity, and self-repetition or copying from the context. In this paper, we postulate that ``human-like'' generations usually lie in a narrow and nearly flat entropy band, and violation of these entropy bounds correlates with degenerate behavior. Our experiments show that this stable narrow entropy zone exists across models, tasks, and domains and confirm the hypothesis that violations of this zone correlate with degeneration. We then use this insight to propose an entropy-aware decoding algorithm that respects these entropy bounds resulting in less degenerate, more contextual, and"human-like"language generation in open-ended text generation settings.
2023-02-14
ArXiv (preprint)
doi.org
arxiv.org
Simplicity and learning to distinguish arguments from modifiers
Leon Bergen
E. Gibson
Timothy O'Donnell
2022-12-28
Journal of Language Modelling (published)
doi.org
Characterizing Idioms: Conventionality and Contingency
Michaela Socolof
Jackie Cheung
Michael Wagner
Timothy O'Donnell
Idioms are unlike most phrases in two important ways. First, words in an idiom have non-canonical meanings. Second, the non-canonical meanin… (see more)gs of words in an idiom are contingent on the presence of other words in the idiom. Linguistic theories differ on whether these properties depend on one another, as well as whether special theoretical machinery is needed to accommodate idioms. We define two measures that correspond to the properties above, and we show that idioms fall at the expected intersection of the two dimensions, but that the dimensions themselves are not correlated. Our results suggest that introducing special machinery to handle idioms may not be warranted.
2022-05-01
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (published)
doi.org
arxiv.org
Compositional Generalization in Dependency Parsing
Emily D. Goodwin
Siva Reddy
Timothy O'Donnell
Dzmitry Bahdanau
2022-05-01
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (published)
doi.org
arxiv.org
Compositional Generalization in Dependency Parsing
Emily D. Goodwin
Siva Reddy
Timothy O'Donnell
Dzmitry Bahdanau
Compositionality— the ability to combine familiar units like words into novel phrases and sentences— has been the focus of intense inter… (see more)est in artificial intelligence in recent years. To test compositional generalization in semantic parsing, Keysers et al. (2020) introduced Compositional Freebase Queries (CFQ). This dataset maximizes the similarity between the test and train distributions over primitive units, like words, while maximizing the compound divergence: the dissimilarity between test and train distributions over larger structures, like phrases. Dependency parsing, however, lacks a compositional generalization benchmark. In this work, we introduce a gold-standard set of dependency parses for CFQ, and use this to analyze the behaviour of a state-of-the art dependency parser (Qi et al., 2020) on the CFQ dataset. We find that increasing compound divergence degrades dependency parsing performance, although not as dramatically as semantic parsing performance. Additionally, we find the performance of the dependency parser does not uniformly degrade relative to compound divergence, and the parser performs differently on different splits with the same compound divergence. We explore a number of hypotheses for what causes the non-uniform degradation in dependency parsing performance, and identify a number of syntactic structures that drive the dependency parser’s lower performance on the most challenging splits.
2021-10-13
ArXiv (preprint)
doi.org
arxiv.org
Characterizing Idioms: Conventionality and Contingency
Michaela Socolof
Jackie Cheung
Michael Wagner
Timothy O'Donnell
Idioms are unlike most phrases in two important ways. First, words in an idiom have non-canonical meanings. Second, the non-canonical meanin… (see more)gs of words in an idiom are contingent on the presence of other words in the idiom. Linguistic theories differ on whether these properties depend on one another, as well as whether special theoretical machinery is needed to accommodate idioms. We define two measures that correspond to the properties above, and we show that idioms fall at the expected intersection of the two dimensions, but that the dimensions themselves are not correlated. Our results suggest that introducing special machinery to handle idioms may not be warranted.
2021-04-17
ArXiv (preprint)
doi.org
arxiv.org