Publications

The Curious Case of Absolute Position Embeddings
Koustuv Sinha
Amirhossein Kazemnejad
Siva Reddy
Joelle Pineau
Dieuwke Hupkes
Adina Williams
Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is repr… (see more)esented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.
2022-11-30
Findings of the Association for Computational Linguistics: EMNLP 2022 (published)
doi.org
arxiv.org
The Emergence of Argument Structure in Artificial Languages
Tom Bosc
Pascal Vincent
Computational approaches to the study of language emergence can help us understand how natural languages are shaped by cognitive and sociocu… (see more)ltural factors. Previous work focused on tasks where agents refer to a single entity. In contrast, we study how agents predicate, that is, how they express that some relation holds between several entities. We introduce a setup where agents talk about a variable number of entities that can be partially observed by the listener. In the presence of a least-effort pressure, they tend to discuss only entities that are not observed by the listener. Thus we can obtain artificial phrases that denote a single entity, as well as artificial sentences that denote several entities. In natural languages, if we ignore the verb, phrases are usually concatenated, either in a specific order or by adding case markers to form sentences. Our setup allows us to quantify how much this holds in emergent languages using a metric we call concatenability. We also measure transitivity, which quantifies the importance of word order. We demonstrate the usefulness of this new setup and metrics for studying factors that influence argument structure. We compare agents having access to input representations structured into pre-segmented objects with properties, versus unstructured representations. Our results indicate that the awareness of object structure yields a more natural sentence organization.
2022-11-30
Transactions of the Association for Computational Linguistics (published)
doi.org
Using incorpoRATE to examine clinician willingness to engage in shared decision making: A study of Family Medicine residents
Roland Grad
Amrita Sandhu
Michael Ferrante
Vinita D’Souza
Lily Puterman-salzman
Samira Abbasgholizadeh Rahimi
Gabrielle Stevens
Glyn Elwyn
2022-11-30
Patient Education and Counseling (published)
doi.org
Bayesian Dynamic Causal Discovery
Alexander Tong
Lazar Atanackovic
Jason Hartford
Yoshua Bengio
Learning the causal structure of observable variables is a central focus for scientific discovery. Bayesian causal discovery methods tackle … (see more)this problem by learning a posterior over the set of admissible graphs that are equally likely given our priors and observations. Existing methods primarily consider observations from static systems and assume the underlying causal structure takes the form of a directed acyclic graph (DAG). In settings with dynamic feedback mechanisms that regulate the trajectories of individual variables, this acyclicity assumption fails unless we account for time. We treat causal discovery in the unrolled causal graph as a problem of sparse identification of a dynamical system. This imposes a natural temporal causal order between variables and captures cyclic feedback loops through time. Under this lens, we propose a new framework for Bayesian causal discovery for dynamical systems and present a novel generative flow network architecture (Dyn-GFN) tailored for this task. Dyn-GFN imposes an edge-wise sparse prior to sequentially build a k -sparse causal graph. Through evaluation on temporal data, our results show that the posterior learned with Dyn-GFN yields improved Bayes coverage of admissible causal structures relative to state of the art Bayesian causal discovery methods.
2022-11-29
NeurIPS.cc/2022/Workshop/CDS (poster)
openreview.net
openreview.net
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
Nasir Mohammad Khalid
Tianhao Xie
Eugene Belilovsky
Tiberiu Popa
We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms… (see more) the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.
2022-11-29
SIGGRAPH Asia 2022 Conference Papers (published)
doi.org
arxiv.org
Isometric Energies for Recovering Injectivity in Constrained Mapping
Xingyi Du
Danny M. Kaufman
Qingnan Zhou
Shahar Kovalsky
Yajie Yan
Noam Aigerman
Tao Ju
2022-11-29
SIGGRAPH Asia 2022 Conference Papers (published)
doi.org
Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests
Christopher Beckham
Martin Weiss
Florian Golemo
Sina Honari
Derek Nowrouzezahrai
Christopher Pal
Different types of mental rotation tests have been used extensively in psychology to understand human visual reasoning and perception. Under… (see more)standing what an object or visual scene would look like from another viewpoint is a challenging problem that is made even harder if it must be performed from a single image. We explore a controlled setting whereby questions are posed about the properties of a scene if that scene was observed from another viewpoint. To do this we have created a new version of the CLEVR dataset that we call CLEVR Mental Rotation Tests (CLEVR-MRT). Using CLEVR-MRT we examine standard methods, show how they fall short, then explore novel neural architectures that involve inferring volumetric representations of a scene. These volumes can be manipulated via camera-conditioned transformations to answer the question. We examine the efficacy of different model variants through rigorous ablations and demonstrate the efficacy of volumetric representations.
2022-11-29
Pattern Recognition (unknown)
doi.org
openreview.net
α-ReQ: Assessing representation quality by measuring eigenspectrum decay
Kumar Krishna Agrawal
Arnab Kumar Mondal
Arna Ghosh
Blake A. Richards
2022-11-28
Conference on Neural Information Processing Systems (Accept)
openreview.net
openreview.net
Attention-based Neural Cellular Automata
Mattie Tesfaldet
Derek Nowrouzezahrai
Christopher Pal
Recent extensions of Cellular Automata (CA) have incorporated key ideas from modern deep learning, dramatically extending their capabilities… (see more) and catalyzing a new family of Neural Cellular Automata (NCA) techniques. Inspired by Transformer-based architectures, our work presents a new class of
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
Yuhan Helena Liu
Arna Ghosh
Blake A. Richards
Eric Shea-Brown
Guillaume Lajoie
To unveil how the brain learns, ongoing work seeks biologically-plausible approximations of gradient descent algorithms for training recurre… (see more)nt neural networks (RNNs). Yet, beyond task accuracy, it is unclear if such learning rules converge to solutions that exhibit different levels of generalization than their nonbiologically-plausible counterparts. Leveraging results from deep learning theory based on loss landscape curvature, we ask: how do biologically-plausible gradient approximations affect generalization? We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely. Next, we verify that such generalization performance is correlated significantly with loss landscape curvature, and we show that biologically-plausible learning rules tend to approach high-curvature regions in synaptic weight space. Using tools from dynamical systems, we derive theoretical arguments and present a theorem explaining this phenomenon. This predicts our numerical results, and explains why biologically-plausible rules lead to worse and more variable generalization properties. Finally, we suggest potential remedies that could be used by the brain to mitigate this effect. To our knowledge, our analysis is the first to identify the reason for this generalization gap between artificial and biologically-plausible learning rules, which can help guide future investigations into how the brain learns solutions that generalize.
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Beyond Mahalanobis-Based Scores for Textual OOD Detection
Pierre Colombo
Eduardo D. C. Gomes
Guillaume Staerman
Nathan Noiry
Pablo Piantanida
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Can Hybrid Geometric Scattering Networks Help Solve the Maximum Clique Problem?
Yimeng Min
Frederik Wenkel
Michael A. Perlmutter
Guy Wolf
We propose a geometric scattering-based graph neural network (GNN) for approximating solutions of the NP-hard maximum clique (MC) problem. W… (see more)e construct a loss function with two terms, one which encourages the network to find highly connected nodes and the other which acts as a surrogate for the constraint that the nodes form a clique. We then use this loss to train an efficient GNN architecture that outputs a vector representing the probability for each node to be part of the MC and apply a rule-based decoder to make our final prediction. The incorporation of the scattering transform alleviates the so-called oversmoothing problem that is often encountered in GNNs and would degrade the performance of our proposed setup. Our empirical results demonstrate that our method outperforms representative GNN baselines in terms of solution accuracy and inference speed as well as conventional solvers like Gurobi with limited time budgets. Furthermore, our scattering model is very parameter efficient with only
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net