Publications

Driving into the Loop: Mapping Automation Bias and Liability Issues for Advanced Driver Assistance Systems

Katie Szilagyi

Jason Millar

AJung Moon

Shalaleh Rismani

2023-10-06

Digital Society (published)

doi.org

Efficient Classification of Long Documents via State-Space Models

Peng Lu

Suyuchen Wang

Mehdi Rezagholizadeh

Bang Liu

Ivan Kobyzev

2023-10-06

EMNLP/2023/Conference (accepted)

openreview.net

Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness

Zichao Li

Ines Arous

Siva Reddy

Jackie C.K. Cheung

The potential of using a large language model (LLM) as a knowledge base (KB) has sparked significant interest. To maintain the knowledge acq… (see more)uired by LLMs, we need to ensure that the editing of learned facts respects internal logical constraints, which are known as dependency of knowledge. Existing work on editing LLMs has partially addressed the issue of dependency, when the editing of a fact should apply to its lexical variations without disrupting irrelevant ones. However, they neglect the dependency between a fact and its logical implications. We propose an evaluation protocol with an accompanying question-answering dataset, StandUp, that provides a comprehensive assessment of the editing process considering the above notions of dependency. Our protocol involves setting up a controlled environment in which we edit facts and monitor their impact on LLMs, along with their implications based on If-Then rules. Extensive experiments on StandUp show that existing knowledge editing methods are sensitive to the surface form of knowledge, and that they have limited performance in inferring the implications of edited facts.

2023-10-06

Conference on Empirical Methods in Natural Language Processing (accepted)

doi.org

openreview.net

HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science

Yu Song

Santiago Miret

Huan Zhang

Bang Liu

2023-10-06

EMNLP/2023/Conference (accepted)

openreview.net

Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions

Jad Kabbara

Jackie CK Cheung

2023-10-06

EMNLP/2023/Conference (accepted)

openreview.net

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Arkil Patel

Satwik Bhattamishra

Siva Reddy

Dzmitry Bahdanau

2023-10-06

Conference on Empirical Methods in Natural Language Processing (accepted)

doi.org

openreview.net

MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization

Yuyan Chen

Zhihao Wen

Ge Fan

Zhengyu Chen

Wei Wu

Dayiheng Liu

Zhixu Li

Bang Liu

Yanghua Xiao

2023-10-06

EMNLP/2023/Conference (accepted)

openreview.net

MOQAGPT: Zero-Shot Multi-modal Open-domain Question Answering with Large Language Models

Le Zhang

Yihong Wu

Fengran Mo

Jian-Yun Nie

Aishwarya Agrawal

Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, t… (see more)ables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework. Using a divide-and-conquer strategy that bypasses intricate multi-modality ranking, our framework can accommodate new modalities and seamlessly transition to new models for the task. Built upon LLMs, MoqaGPT retrieves and extracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our methodology boosts performance on the MMCoQA dataset, improving F1 by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MoqaGPT surpasses the zero-shot baseline, improving F1 by 9.5 points and EM by 10.1 points, and significantly closes the gap with supervised methods. Our codebase is available at https://github.com/lezhang7/MOQAGPT.

2023-10-06

EMNLP/2023/Conference (accepted)

doi.org

openreview.net

RainProof: An Umbrella to Shield Text Generator from Out-Of-Distribution Data

Maxime Darrin

Pablo Piantanida

Pierre Colombo

Implementing effective control mechanisms to ensure the proper functioning and security of deployed NLP models, from translation to chatbots… (see more), is essential. A key ingredient to ensure safe system behaviour is Out-Of-Distribution (OOD) detection, which aims to detect whether an input sample is statistically far from the training distribution. Although OOD detection is a widely covered topic in classification tasks, most methods rely on hidden features output by the encoder. In this work, we focus on leveraging soft-probabilities in a black-box framework, i.e. we can access the soft-predictions but not the internal states of the model. Our contributions include: (i) RAINPROOF a Relative informAItioN Projection OOD detection framework; and (ii) a more operational evaluation setting for OOD detection. Surprisingly, we find that OOD detection is not necessarily aligned with task-specific measures. The OOD detector may filter out samples well processed by the model and keep samples that are not, leading to weaker performance. Our results show that RAINPROOF provides OOD detection methods more aligned with task-specific performance metrics than traditional OOD detectors.

2023-10-06

EMNLP/2023/Conference (accepted)

openreview.net

Sparse Universal Transformer

Shawn Tan

Yikang Shen

Zhenfang Chen

Aaron Courville

Chuang Gan

The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers and is Turing-complete under certain… (see more) assumptions. Empirical evidence also shows that UTs have better compositional generalization than Vanilla Transformers (VTs) in formal language tasks. The parameter-sharing also affords it better parameter efficiency than VTs. Despite its many advantages, most state-of-the-art NLP systems use VTs as their backbone model instead of UTs. This is mainly because scaling UT parameters is more compute and memory intensive than scaling up a VT. This paper proposes the Sparse Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE) to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. Experiments show that SUT combines the best of both worlds, achieving strong generalization results on formal language tasks (Logical inference and CFQ) and impressive parameter and computation efficiency on standard natural language benchmarks like WMT'14.

2023-10-06

EMNLP/2023/Conference (accepted)

doi.org

openreview.net

The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning

Tian Jin

Nolan Clement

Xin Dong

Vaishnavh Nagarajan

Michael Carbin

Jonathan Ragan-kelley

Gintare Karolina Dziugaite

How does scaling the number of parameters in large language models (LLMs) affect their core capabilities? We study two natural scaling techn… (see more)iques -- weight pruning and simply training a smaller or larger model, which we refer to as dense scaling -- and their effects on two core capabilities of LLMs: (a) recalling facts presented during pre-training and (b) processing information presented in-context during inference. By curating a suite of tasks that help disentangle these two capabilities, we find a striking difference in how these two abilities evolve due to scaling. Reducing the model size by more than 30\% (via either scaling approach) significantly decreases the ability to recall facts seen in pre-training. Yet, a 60--70\% reduction largely preserves the various ways the model can process in-context information, ranging from retrieving answers from a long context to learning parameterized functions from in-context exemplars. The fact that both dense scaling and weight pruning exhibit this behavior suggests that scaling model size has an inherently disparate effect on fact recall and in-context learning.

2023-10-06

ArXiv (preprint)

doi.org

arxiv.org

Using In-Context Learning to Improve Dialogue Safety

Devamanyu Hazarika

Di Jin

Yang Liu