Publications

Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

Providing better language tools for low-resource and endangered languages is imperative for equitable growth. Recent progress with massively… (see more) multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages. However, this transfer is not universal, with many languages not currently understood by multilingual approaches. It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, the vast majority of languages not having the resources available to simply evaluate performances on. In this work, we attempt to clarify which languages do and do not currently benefit from such transfer. To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language. We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.

2022-11-30

Findings of the Association for Computational Linguistics: EMNLP 2022 (published)

doi.org

arxiv.org

Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining

To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for makin… (see more)g a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using relative area-between-curves (RACU), which allows for easy comparison across papers, models, and tasks. We evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.

2022-11-30

Findings of the Association for Computational Linguistics: EMNLP 2022 (published)

doi.org

arxiv.org

Findings of the WMT’22 Shared Task on Large-Scale Machine Translation Evaluation for African Languages

David Ifeoluwa Adelani

Md Mahfuz Ibn Alam

Antonios Anastasopoulos

Akshita Bhagia

Marta R. Costa-jussa

Jesse Dodge

Fahim Faisal

Christian Federmann

Natalia N. Fedorova

Francisco S. Guzm'an

Sergey Koshelev

Jean Maillard

Vukosi Marivate

Jonathan Mbuya

Alexandre Mourachko

Safiyyah Saleem

Holger Schwenk

Guillaume Wenzek

We present the results of the WMT’22 SharedTask on Large-Scale Machine Translation Evaluation for African Languages. The shared taskinclud… (see more)ed both a data and a systems track, alongwith additional innovations, such as a focus onAfrican languages and extensive human evaluation of submitted systems. We received 14system submissions from 8 teams, as well as6 data track contributions. We report a largeprogress in the quality of translation for Africanlanguages since the last iteration of this sharedtask: there is an increase of about 7.5 BLEUpoints across 72 language pairs, and the average BLEU scores went from 15.09 to 22.60.

2022-11-30

Proceedings of the Seventh Conference on Machine Translation (WMT) (published)

doi.org

Implementing automation in deep brain stimulation: has the time come?

Marco Bonizzato

Alfonso Fasano

2022-11-30

The Lancet Digital Health (published)

doi.org

Improving Passage Retrieval with Zero-Shot Question Generation

Devendra Singh Sachan

Mike Lewis

Mandar Joshi

Armen Aghajanyan

Wen-tau Yih

Joelle Pineau

Luke Zettlemoyer

We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retr… (see more)ieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage. This approach can be applied on top of any retrieval method (e.g. neural or keyword-based), does not require any domain- or task-specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval models by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state-of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.

2022-11-30

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (published)

doi.org

arxiv.org

K27M in canonical and noncanonical H3 variants occurs in distinct oligodendroglial cell lineages in brain midline gliomas

Selin Jessa

Abdulshakour Mohammadnia

Ashot S. Harutyunyan

Maud Hulswit

Srinidhi Varadharajan

Hussein Lakkis

Nisha Kabir

Zahedeh Bashardanesh

Steven Hébert

Damien Faury

Maria C. Vladoiu

Samantha Worme

Marie Coutelier

Brian Krug

Augusto Faria Andrade

Manav Pathania

Andrea Bajic

Alexander G. Weil

Benjamin Ellezam

Jeffrey Atkinson … (see 14 more)

Roy W. R. Dudley

Jean-Pierre Farmer

Sebastien Perreault

Benjamin A. Garcia

Valérie Larouche

Mathieu Blanchette

Livia Garzia

Aparna Bhaduri

Keith L. Ligon

Pratiti Bandopadhayay

Michael D. Taylor

Stephen C. Mack

Nada Jabado

Claudia L. Kleinman

Canonical (H3.1/H3.2) and noncanonical (H3.3) histone 3 K27M-mutant gliomas have unique spatiotemporal distributions, partner alterations, a… (see more)nd molecular profiles. The contribution of the cell-of-origin to these differences has been challenging to uncouple from the oncogenic reprogramming induced by the mutation. Here, we perform an integrated analysis of 116 tumors, including single-cell transcriptome and chromatin accessibility, 3D chromatin architecture and epigenomic profiles, and show that K27M-mutant gliomas faithfully maintain chromatin configuration at developmental genes consistent with anatomically distinct oligodendrocyte-precursor-like cells (OPC). H3.3K27M thalamic gliomas map to prosomere 2-derived lineages. In turn, H3.1K27M ACVR1-mutant pontine gliomas uniformly mirror early ventral NKX6-1+/SHH-dependent brainstem OPCs, while H3.3K27M gliomas frequently resemble dorsal PAX3+/BMP-dependent progenitors. Our data suggest a context-specific vulnerability in H3.1K27M-mutant SHH-dependent ventral OPCs, which rely on acquisition of ACVR1 mutations to drive aberrant BMP signaling required for oncogenesis. The unifying action of K27M mutations is to restrict H3K27me3 at PRC2 landing sites, while other epigenetic changes are mainly contingent on the cell-of-origin chromatin state and cycling rate.

2022-11-30

Nature Genetics (unknown)

doi.org

Learning with Rejection for Abstractive Text Summarization

Meng Cao

Yue Dong

Jingyi He

Jackie CK Cheung

2022-11-30

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (published)

doi.org

arxiv.org

Manifold Interpolating Optimal-Transport Flows for Trajectory Inference

Guillaume Huguet

D.S. Magruder

Alexander Tong

Oluwadamilola Fasina

Manik Kuchroo

Guy Wolf

Smita Krishnaswamy

We present a method called Manifold Interpolating Optimal-Transport Flow (MIOFlow) that learns stochastic, continuous population dynamics fr… (see more)om static snapshot samples taken at sporadic timepoints. MIOFlow combines dynamic models, manifold learning, and optimal transport by training neural ordinary differential equations (Neural ODE) to interpolate between static population snapshots as penalized by optimal transport with manifold ground distance. Further, we ensure that the flow follows the geometry by operating in the latent space of an autoencoder that we call a geodesic autoencoder (GAE). In GAE the latent space distance between points is regularized to match a novel multiscale geodesic distance on the data manifold that we define. We show that this method is superior to normalizing flows, Schrödinger bridges and other generative models that are designed to flow from noise to data in terms of interpolating between populations. Theoretically, we link these trajectories with dynamic optimal transport. We evaluate our method on simulated data with bifurcations and merges, as well as scRNA-seq data from embryoid body differentiation, and acute myeloid leukemia treatment.

2022-11-30

PubMed (unknown)

doi.org

openreview.net

A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques

Malik H. Altakrori

Thomas Scialom

Benjamin C. M. Fung

Jackie CK Cheung

2022-11-30

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (published)

doi.org

QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance

Xiaoqiang Wang

Bang Liu

Siliang Tang

Lingfei Wu

Existing metrics for assessing question generation not only require costly human reference but also fail to take into account the input cont… (see more)ext of generation, rendering the lack of deep understanding of the relevance between the generated questions and input contexts. As a result, they may wrongly penalize a legitimate and reasonable candidate question when it (1) involves complicated reasoning with the context or (2) can be grounded by multiple evidences in the context.In this paper, we propose QRelScore, a context-aware Relevance evaluation metric for Question Generation.Based on off-the-shelf language models such as BERT and GPT2, QRelScore employs both word-level hierarchical matching and sentence-level prompt-based generation to cope with the complicated reasoning and diverse generation from multiple evidences, respectively.Compared with existing metrics, our experiments demonstrate that QRelScore is able to achieve a higher correlation with human judgments while being much more robust to adversarial samples.

2022-11-30

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (published)

doi.org

arxiv.org

Structure-Aware Reinforcement Learning for Node-Overload Protection in Mobile Edge Computing

Anirudha Jitani

Aditya Mahajan

Zhongwen Zhu

Hatem Abou-Zeid

Emmanuel Thepie Fapi

Hakimeh Purmehdi

Mobile Edge Computing (MEC) involves placing computational capability and applications at the edge of the network, providing benefits such a… (see more)s reduced latency, reduced network congestion, and improved performance of applications. The performance and reliability of MEC degrades significantly when the edge server(s) in the cluster are overloaded. In this work, an adaptive admission control policy to prevent edge node from getting overloaded is presented. This approach is based on a recently-proposed low complexity RL (Reinforcement Learning) algorithm called SALMUT (Structure-Aware Learning for Multiple Thresholds), which exploits the structure of the optimal admission control policy in multi-class queues for an average-cost setting. We extend the framework to work for node overload-protection problem in a discounted-cost setting. The proposed solution is validated using several scenarios mimicking real-world deployments in two different settings — computer simulations and a docker testbed. Our empirical evaluations show that the total discounted cost incurred by SALMUT is similar to state-of-the-art deep RL algorithms such as PPO (Proximal Policy Optimization) and A2C (Advantage Actor Critic) but requires an order of magnitude less time to train, outputs easily interpretable policy, and can be deployed in an online manner.

2022-11-30

IEEE Transactions on Cognitive Communications and Networking (published)

doi.org

arxiv.org

On the Compositional Generalization Gap of In-Context Learning

Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abiliti… (see more)es. Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any fine-tuning (also known as in-context learning). In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. In the ID settings, the demonstrations are from the same split (test or train) that the model is being evaluated on, and in the OOD settings, they are from the other split. We look at how the relative generalization gap of in-context learning evolves as models are scaled up. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.

2022-11-30

Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (published)

doi.org

arxiv.org

Venture Scientist Bootcamp

AI Policy Compass

Supervision Requests

Publications

Venture Scientist Bootcamp

AI Policy Compass

Supervision Requests

Popular keywords:

Publications