Publications

A Variational Perspective on Diffusion-Based Generative Models and Score Matching
Chin-wei Huang
Jae Hyun Lim
Aaron Courville
Discrete-time diffusion-based generative models and score matching methods have shown promising results in modeling high-dimensional image d… (see more)ata. Recently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, i.e. the gradient of the log-density of the perturbed data. They propose to plug the learned score function into an inverse formula to define a generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive a variational framework for likelihood estimation, which includes continuous-time normalizing flows as a special case, and can be seen as an infinitely deep variational autoencoder. Under this framework, we show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE proposed by Song et al. (2021), bridging the theoretical gap.
2021-11-08
NeurIPS.cc/2021/Conference (spotlight)
openreview.net
openreview.net
Problèmes associés au déploiement des modèles fondés sur l’apprentissage machine en santé
Joseph Paul Cohen
Tianshi Cao
Joseph Viviano
Chin-wei Huang
Michael Fralick
Marzyeh Ghassemi
Muhammad Mamdani
Russell Greiner
Yoshua Bengio
2021-11-06
Canadian Medical Association Journal (published)
doi.org
Learned Image Compression for Machine Perception
Felipe Codevilla
Jean Gabriel Simard
Ross Goroshin
Chris Pal
Recent work has shown that learned image compression strategies can outperform standard hand-crafted compression algorithms that have been d… (see more)eveloped over decades of intensive research on the rate-distortion trade-off. With growing applications of computer vision, high quality image reconstruction from a compressible representation is often a secondary objective. Compression that ensures high accuracy on computer vision tasks such as image segmentation, classification, and detection therefore has the potential for significant impact across a wide variety of settings. In this work, we develop a framework that produces a compression format suitable for both human perception and machine perception. We show that representations can be learned that simultaneously optimize for compression and performance on core vision tasks. Our approach allows models to be trained directly from compressed representations, and this approach yields increased performance on new tasks and in low-shot learning settings. We present results that improve upon segmentation and detection performance compared to standard high quality JPGs, but with representations that are four to ten times smaller in terms of bits per pixel. Further, unlike naive compression methods, at a level ten times smaller than standard JEPGs, segmentation and detection models trained from our format suffer only minor degradation in performance.
2021-11-02
ArXiv (preprint)
doi.org
arxiv.org
Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval
Devang Kulshreshtha
Robert Belfer
Iulian V. Serban
Siva Reddy
In this work, we introduce back-training, an alternative to self-training for unsupervised domain adaptation (UDA). While self-training gene… (see more)rates synthetic training data where natural inputs are aligned with noisy outputs, back-training results in natural outputs aligned with noisy inputs. This significantly reduces the gap between target domain and synthetic data distribution, and reduces model overfitting to source domain. We run UDA experiments on question generation and passage retrieval from the Natural Questions domain to machine learning and biomedical domains. We find that back-training vastly outperforms self-training by a mean improvement of 7.8 BLEU-4 points on generation, and 17.6% top-20 retrieval accuracy across both domains. We further propose consistency filters to remove low-quality synthetic data before training. We also release a new domain-adaptation dataset - MLQuestions containing 35K unaligned questions, 50K unaligned passages, and 3K aligned question-passage pairs.
2021-10-31
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (published)
doi.org
arxiv.org
#EEGManyLabs: Investigating the replicability of influential EEG experiments
Yuri G Pavlov
Nika Adamian
Stefan Appelhoff
Mahnaz Arvaneh
Christopher Benwell
Christian Beste
Amy Bland
Daniel E. Bradford
Florian Bublatzky
Niko A. Busch
Peter E. Clayson
Damian Cruse
Artur Czeszumski
Anna Dreber
Guillaume Dumas
Benedikt Valerian Ehinger
Giorgio Ganis
Xun He
José Antonio Hinojosa
Christoph Huber-Huber … (see 39 more)
Michael Inzlicht
Bradley N Jack
Magnus Johannesson
Rhiannon Jones
Evgenii Kalenkovich
Laura Kaltwasser
Hamid Karimi-Rouzbahani
Andreas Keil
Peter König
Layla Kouara
Louisa Kulke
Cecile D. Ladouceur
Nicolas Langer
Heinrich R. Liesefeld
David Luque
Annmarie MacNamara
Liad Mudrik
Muthuraman Muthuraman
Lauren Browning Neal
Gustav Nilsonne
Guiomar Niso
Sebastian Ocklenburg
Robert Oostenveld
Cyril R. Pernet
Gilles Pourtois
Manuela Ruzzoli
Sarah Sass
Alexandre Schaefer
Magdalena Senderecka
Joel S. Snyder
Christian K. Tamnes
E Tognoli
Marieke K. van Vugt
Edelyn Verona
Robin Vloeberghs
Dominik Welke
Jan Wessel
Ilya Zakharov
Faisal Mushtaq
2021-10-31
Cortex (published)
doi.org
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
Koustuv Sinha
Robin Jia
Dieuwke Hupkes
Joelle Pineau
Adina Williams
Douwe Kiela
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to repres… (see more)ent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks—including tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.
2021-10-31
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (published)
doi.org
arxiv.org
Opioid prescribing among new users for non-cancer pain in the USA, Canada, UK, and Taiwan: A population-based cohort study
Meghna Jani
Nadyne Girard
David W. Bates
David L Buckeridge
Therese Sheppard
Jack Li
Usman Iqbal
Shelly Vik
Colin Weaver
Judy Seidel
William G. Dixon
Robyn Tamblyn
Background The opioid epidemic in North America has been driven by an increase in the use and potency of prescription opioids, with ensuing … (see more)excessive opioid-related deaths. Internationally, there are lower rates of opioid-related mortality, possibly because of differences in prescribing and health system policies. Our aim was to compare opioid prescribing rates in patients without cancer, across 5 centers in 4 countries. In addition, we evaluated differences in the type, strength, and starting dose of medication and whether these characteristics changed over time. Methods and findings We conducted a retrospective multicenter cohort study of adults who are new users of opioids without prior cancer. Electronic health records and administrative health records from Boston (United States), Quebec and Alberta (Canada), United Kingdom, and Taiwan were used to identify patients between 2006 and 2015. Standard dosages in morphine milligram equivalents (MMEs) were calculated according to The Centers for Disease Control and Prevention. Age- and sex-standardized opioid prescribing rates were calculated for each jurisdiction. Of the 2,542,890 patients included, 44,690 were from Boston (US), 1,420,136 Alberta, 26,871 Quebec (Canada), 1,012,939 UK, and 38,254 Taiwan. The highest standardized opioid prescribing rates in 2014 were observed in Alberta at 66/1,000 persons compared to 52, 51, and 18/1,000 in the UK, US, and Quebec, respectively. The median MME/day (IQR) at initiation was highest in Boston at 38 (20 to 45); followed by Quebec, 27 (18 to 43); Alberta, 23 (9 to 38); UK, 12 (7 to 20); and Taiwan, 8 (4 to 11). Oxycodone was the first prescribed opioid in 65% of patients in the US cohort compared to 14% in Quebec, 4% in Alberta, 0.1% in the UK, and none in Taiwan. One of the limitations was that data were not available from all centers for the entirety of the 10-year period. Conclusions In this study, we observed substantial differences in opioid prescribing practices for non-cancer pain between jurisdictions. The preference to start patients on higher MME/day and more potent opioids in North America may be a contributing cause to the opioid epidemic.
2021-10-31
PLoS Medicine (published)
doi.org
Post-Editing Extractive Summaries by Definiteness Prediction
Jad Kabbara
Jackie Chi Kit Cheung
Extractive summarization has been the mainstay of automatic summarization for decades. Despite all the progress, extractive summarizers stil… (see more)l suffer from shortcomings including coreference issues arising from extracting sentences away from their original context in the source document. This affects the coherence and readability of extractive summaries. In this work, we propose a lightweight post-editing step for extractive summaries that centers around a single linguistic decision: the definiteness of noun phrases. We conduct human evaluation studies that show that human expert judges substantially prefer the output of our proposed system over the original summaries. Moreover, based on an automatic evaluation study, we provide evidence for our system’s ability to generate linguistic decisions that lead to improved extractive summaries. We also draw insights about how the automatic system is exploiting some local cues related to the writing style of the main article texts or summary texts to make the decisions, rather than reasoning about the contexts pragmatically.
2021-10-31
Findings of the Association for Computational Linguistics: EMNLP 2021 (published)
doi.org
Refining BERT Embeddings for Document Hashing via Mutual Information Maximization
Zijing Ou
Qinliang Su
Jianxing Yu
Ruihui Zhao
Yefeng Zheng
Bang Liu
Existing unsupervised document hashing methods are mostly established on generative models. Due to the difficulties of capturing long depend… (see more)ency structures, these methods rarely model the raw documents directly, but instead to model the features extracted from them (e.g. bag-of-words (BOW), TFIDF). In this paper, we propose to learn hash codes from BERT embeddings after observing their tremendous successes on downstream tasks. As a first try, we modify existing generative hashing models to accommodate the BERT embeddings. However, little improvement is observed over the codes learned from the old BOW or TFIDF features. We attribute this to the reconstruction requirement in the generative hashing, which will enforce irrelevant information that is abundant in the BERT embeddings also compressed into the codes. To remedy this issue, a new unsupervised hashing paradigm is further proposed based on the mutual information (MI) maximization principle. Specifically, the method first constructs appropriate global and local codes from the documents and then seeks to maximize their mutual information. Experimental results on three benchmark datasets demonstrate that the proposed method is able to generate hash codes that outperform existing ones learned from BOW features by a substantial margin.
2021-10-31
Findings of the Association for Computational Linguistics: EMNLP 2021 (published)
doi.org
arxiv.org
Sometimes We Want Ungrammatical Translations
Prasanna Parthasarathi
Koustuv Sinha
Joelle Pineau
Adina Williams
Rapid progress in Neural Machine Translation (NMT) systems over the last few years has focused primarily on improving translation quality, a… (see more)nd as a secondary focus, improving robustness to perturbations (e.g. spelling). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness or faithfulness, by focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure their effects on the target side. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which prop-erty is best suited for their use case.
2021-10-31
Empirical Methods in Natural Language Processing (unknown)
doi.org
Textual Time Travel: A Temporally Informed Approach to Theory of Mind
Akshatha Arodi
Jackie CK Cheung
Natural language processing systems such as dialogue agents should be able to reason about other people’s beliefs, intentions and desires.… (see more) This capability, called theory of mind (ToM), is crucial, as it allows a model to predict and interpret the needs of users based on their mental states. A recent line of research evaluates the ToM capability of existing memoryaugmented neural models through questionanswering. These models perform poorly on false belief tasks where beliefs differ from reality, especially when the dataset contains distracting sentences. In this paper, we propose a new temporally informed approach for improving the ToM capability of memory-augmented neural models. Our model incorporates priors about the entities’ minds and tracks their mental states as they evolve over time through an extended passage. It then responds to queries through textual time travel—i.e., by accessing the stored memory of an earlier time step. We evaluate our model on ToM datasets and find that this approach improves performance, particularly by correcting the predicted mental states to match the false belief.
2021-10-31
Findings of the Association for Computational Linguistics: EMNLP 2021 (published)
doi.org
The meaning of significant mean group differences for biomarker discovery
Eva Loth
Jumana Ahmad
Chris Chatham
Beatriz López
Ben Carter
Daisy Crawley
Bethany Oakley
Hannah Hayward
Jennifer Cooke
Antonia San José Cáceres
Danilo Bzdok
Emily Jones
Tony Charman
Christian Beckmann
Thomas Bourgeron
Roberto Toro
Jan Buitelaar
Declan Murphy
Guillaume Dumas
Over the past decade, biomarker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterog… (see more)eneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between “cases” and “controls,” which tends to ignore within-group variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research—autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen’s d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the “on average” when summarising their findings in their abstracts (“autistic people have deficits in X”), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.
2021-10-31
PLoS Computational Biology (published)
doi.org