Publications

Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction

Zuobai Zhang

Minghao Xu

Aurelie Lozano

Vijil Chenthamarakshan

Payel Das

Pre-training methods on proteins are recently gaining interest, leveraging either protein sequences or structures, while modeling their join… (see more)t energy landscape is largely unexplored. In this work, inspired by the success of denoising diffusion models, we propose the DiffPreT approach to pre-train a protein encoder by sequence-structure multimodal diffusion modeling. DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the multimodal diffusion trajectory, which acquires the joint distribution of sequences and structures. Considering the essential protein conformational variations, we enhance DiffPreT by a physics-inspired method called Siamese Diffusion Trajectory Prediction ( SiamDiff ) to capture the correlation between different conformers of a protein. SiamDiff attains this goal by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers. We study the effectiveness of DiffPreT and SiamDiff on both atom-and residue-level structure-based protein understanding tasks. Experimental results show that the performance of DiffPreT is consistently competitive on all tasks, and SiamDiff achieves new state-of-the-art performance, considering the mean ranks on all tasks. The source code will be released upon acceptance.

2023-01-01

arXiv.org (preprint)

doi.org

Preference-Based Offline Evaluation

C. Clarke

Fernando Diaz

Negar Arabzadeh

A core step in production model research and development involves the offline evaluation of a system before production deployment. Tradition… (see more)al offline evaluation of search, recommender, and other systems involves gathering item relevance labels from human editors. These labels can then be used to assess system performance using offline evaluation metrics. Unfortunately, this approach does not work when evaluating highly effective ranking systems, such as those emerging from the advances in machine learning. Recent work demonstrates that moving away from pointwise item and metric evaluation can be a more effective approach to the offline evaluation of systems. This tutorial, intended for both researchers and practitioners, reviews early work in preference-based evaluation and covers recent developments in detail.

2023-01-01

WSDM (published)

doi.org

Price Forecasting in the Ontario Electricity Market via TriConvGRU Hybrid Model: Univariate vs. Multivariate Frameworks

Behdad Ehsani

Pierre-Olivier Pineau

Laurent Charlin

2023-01-01

SSRN Electronic Journal (published)

doi.org

Privacy-Aware Compression for Federated Learning Through Numerical Mechanism Design

Chuan Guo

Kamalika Chaudhuri

Pierre Stock

Michael Rabbat

In private federated learning (FL), a server aggregates differentially private updates from a large number of clients in order to train a ma… (see more)chine learning model. The main challenge in this setting is balancing privacy with both classification accuracy of the learnt model as well as the number of bits communicated between the clients and server. Prior work has achieved a good trade-off by designing a privacy-aware compression mechanism, called the minimum variance unbiased (MVU) mechanism, that numerically solves an optimization problem to determine the parameters of the mechanism. This paper builds upon it by introducing a new interpolation procedure in the numerical design process that allows for a far more efficient privacy analysis. The result is the new Interpolated MVU mechanism that is more scalable, has a better privacy-utility trade-off, and provides SOTA results on communication-efficient private FL on a variety of datasets.

2023-01-01

ICML (published)

openreview.net

PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation

Gaurav Sahu

Olga Vechtomova

Dzmitry Bahdanau

Issam Hadj Laradji

Data augmentation is a widely used technique to address the problem of text classification when there is a limited amount of training data. … (see more)Recent work often tackles this problem using large language models (LLMs) like GPT3 that can generate new examples given already available ones. In this work, we propose a method to generate more helpful augmented data by utilizing the LLM's abilities to follow instructions and perform few-shot classifications. Our specific PromptMix method consists of two steps: 1) generate challenging text augmentations near class boundaries; however, generating borderline examples increases the risk of false positives in the dataset, so we 2) relabel the text augmentations using a prompting-based LLM classifier to enhance the correctness of labels in the generated data. We evaluate the proposed method in challenging 2-shot and zero-shot settings on four text classification datasets: Banking77, TREC6, Subjectivity (SUBJ), and Twitter Complaints. Our experiments show that generating and, crucially, relabeling borderline examples facilitates the transfer of knowledge of a massive LLM like GPT3.5-turbo into smaller and cheaper classifiers like DistilBERT

2023-01-01

EMNLP (published)

doi.org

openreview.net

Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning

Nader Asadi

MohammadReza Davari

Sudhir Mudur

Rahaf Aljundi

Eugene Belilovsky

In Continual learning (CL) balancing effective adaptation while combating catastrophic forgetting is a central challenge. Many of the recent… (see more) best-performing methods utilize various forms of prior task data, e.g. a replay buffer, to tackle the catastrophic forgetting problem. Having access to previous task data can be restrictive in many real-world scenarios, for example when task data is sensitive or proprietary. To overcome the necessity of using previous tasks' data, in this work, we start with strong representation learning methods that have been shown to be less prone to forgetting. We propose a holistic approach to jointly learn the representation and class prototypes while maintaining the relevance of old class prototypes and their embedded similarities. Specifically, samples are mapped to an embedding space where the representations are learned using a supervised contrastive loss. Class prototypes are evolved continually in the same latent space, enabling learning and prediction at any point. To continually adapt the prototypes without keeping any prior task data, we propose a novel distillation loss that constrains class prototypes to maintain relative similarities as compared to new task data. This method yields state-of-the-art performance in the task-incremental setting, outperforming methods relying on large amounts of data, and provides strong performance in the class-incremental setting without using any stored data points.

2023-01-01

ICML (published)

doi.org

openreview.net

Publisher Correction: Advancing ethics review practices in AI research

Madhulika Srikumar

Rebecca Finlay

Grace M. Abuhamad

Carolyn Ashurst

Rosie Campbell

Emily Campbell-Ratcliffe

Hudson Hongo

Sara Rene Jordan

Joseph Lindley

Aviv Ovadya

Joelle Pineau

2023-01-01

Nature Machine Intelligence (published)

doi.org

A rapid review for developing a co-design framework for a pediatric surgical communication application

Michelle Cwintal

Hamed Ranjbar

Parsa Bandamiri

Elena Guadagno

Esli Osmanlliu

Dan Poenaru

2023-01-01

Journal of Pediatric Surgery (published)

doi.org

Recall as a Measure of Ranking Robustness

Fernando Diaz

Bhaskar Mitra

2023-01-01

arXiv.org (preprint)

doi.org

Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learning

Ali Rahimi-Kalahroudi

Janarthanan Rajendran

Ida Momennejad

Harm van Seijen

Sarath Chandar Anbil Parthipan

2023-01-01

CoLLAs (published)

proceedings.mlr.press

arxiv.org

A reproducible benchmark of resting-state fMRI denoising strategies using fMRIPrep and Nilearn

Hao-Ting Wang

Steven L. Meisler

Hanad Sharmarke

Natasha Clarke

Nicolas Gensollen

Christopher J Markiewicz

Fraçois Paugam

Bertrand Thirion

Pierre (Louis) Bellec

Reducing contributions from non-neuronal sources is a crucial step in functional magnetic resonance imaging (fMRI) analyses. Many viable str… (see more)ategies for denoising fMRI are used in the literature, and practitioners rely on denoising benchmarks for guidance in the selection of an appropriate choice for their study. However, fMRI denoising software is an ever-evolving field, and the benchmarks can quickly become obsolete as the techniques or implementations change. In this work, we present a fully reproducible denoising benchmark featuring a range of denoising strategies and evaluation metrics, built primarily on the fMRIPrep and Nilearn software packages. We apply this reproducible benchmark to investigate the robustness of the conclusions across two different datasets and two versions of fMRIPrep. The majority of benchmark results were consistent with prior literature. Scrubbing, a technique which excludes time points with excessive motion, combined with global signal regression, is generally effective at noise removal. Scrubbing however disrupts the continuous sampling of brain images and is incompatible with some statistical analyses, e.g. auto-regressive modeling. In this case, a simple strategy using motion parameters, average activity in select brain compartments, and global signal regression should be preferred. Importantly, we found that certain denoising strategies behave inconsistently across datasets and/or versions of fMRIPrep, or had a different behavior than in previously published benchmarks, especially ICA-AROMA. These results demonstrate that a reproducible denoising benchmark can effectively assess the robustness of conclusions across multiple datasets and software versions. Technologies such as BIDS-App, the Jupyter Book and Neurolibre provided the infrastructure to publish the metadata and report figures. Readers can reproduce the report figures beyond the ones reported in the published manuscript. With the denoising benchmark, we hope to provide useful guidelines for the community, and that our software infrastructure will facilitate continued development as the state-of-the-art advances.

2023-01-01

(published)

www.semanticscholar.org

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

Yu Lu Liu

Meng Cao

Su Lin Blodgett

Jackie Cheung

Alexandra Olteanu

Adam Trischler

2023-01-01

EMNLP (Findings) (published)