Publications

Protective effectiveness of previous SARS-CoV-2 infection and hybrid immunity against the omicron variant and severe disease: a systematic review and meta-regression

Niklas Bobrovitz

Harriet Ware

Xiaomeng Ma

Zihan Li

Reza Hosseini

Christian Cao

Anabel Selemon

Mairead Whelan

Zahra Premji

Hanane Issa

Brianna Cheng

Laith J Abu Raddad

David L Buckeridge

Maria D Van Kerkhove

Vanessa Piechotta

Melissa M Higdon

Annelies Wilder-Smith

Isabel Bergeri

Daniel R Feikin

Rahul K. Arora … (see 2 more)

Minal K Patel

Lorenzo Subissi

2023-04-30

The Lancet Infectious Diseases (published)

doi.org

Protein Representation Learning by Geometric Structure Pretraining

Zuobai Zhang

Minghao Xu

Arian Jamasb

Vijil Chenthamarakshan

Aurelie Lozano

Payel Das

Jian Tang

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Ex… (see more)isting approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less pretraining data. Our implementation is available at https://github.com/DeepGraphLearning/GearNet.

2023-04-30

International Conference on Learning Representations (Accept: poster)

doi.org

openreview.net

Protein Sequence and Structure Co-design with Equivariant Translation

Chence Shi

Chuanrui Wang

Jiarui Lu

Bozitao Zhong

Jian Tang

Proteins are macromolecules that perform essential functions in all living organisms. Designing novel proteins with specific structures and … (see more)desired functions has been a long-standing challenge in the field of bioengineering. Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models, both of which suffer from high inference costs. In this paper, we propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state from random initialization, based on context features given a priori. Our model consists of a trigonometry-aware encoder that reasons geometrical constraints and interactions from context features, and a roto-translation equivariant decoder that translates protein sequence and structure interdependently. Notably, all protein amino acids are updated in one shot in each translation step, which significantly accelerates the inference process. Experimental results across multiple tasks show that our model outperforms previous state-of-the-art baselines by a large margin, and is able to design proteins of high fidelity as regards both sequence and structure, with running time orders of magnitude less than sampling-based methods.

2023-04-30

International Conference on Learning Representations (Accept: poster)

doi.org

openreview.net

Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks

Joshua Greaves

Bellemare Marc-Emmanuel

Auxiliary tasks improve the representations learned by deep reinforcement learning agents. Analytically, their effect is reasonably well und… (see more)erstood; in practice, however, their primary use remains in support of a main learning objective, rather than as a method for learning representations. This is perhaps surprising given that many auxiliary tasks are defined procedurally, and hence can be treated as an essentially infinite source of information about the environment. Based on this observation, we study the effectiveness of auxiliary tasks for learning rich representations, focusing on the setting where the number of tasks and the size of the agent's network are simultaneously increased. For this purpose, we derive a new family of auxiliary tasks based on the successor measure. These tasks are easy to implement and have appealing theoretical properties. Combined with a suitable off-policy learning rule, the result is a representation learning algorithm that can be understood as extending Mahadevan & Maggioni (2007)'s proto-value functions to deep reinforcement learning -- accordingly, we call the resulting object proto-value networks. Through a series of experiments on the Arcade Learning Environment, we demonstrate that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms, using only linear approximation and a small number (~4M) of interactions with the environment's reward function.

2023-04-30

International Conference on Learning Representations (Accept: poster)

doi.org

openreview.net

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Aishwarya Agrawal

Ivana Kaji'c

Emanuele Bugliarello

Elnaz Davoodi

Anita Gergely

Phil Blunsom

Aida Nematzadeh

2023-04-30

Association for Computational Linguistics (published)

doi.org

arxiv.org

Reliability of CKA as a Similarity Measure in Deep Learning

MohammadReza Davari

Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different w… (see more)ays. The Centered Kernel Alignment (CKA) similarity metric, particularly its linear variant, has recently become a popular approach and has been widely used to compare representations of a network's different layers, of architecturally similar networks trained differently, or of models with different architectures trained on the same data. A wide variety of conclusions about similarity and dissimilarity of these various representations have been made using CKA. In this work we present analysis that formally characterizes CKA sensitivity to a large class of simple transformations, which can naturally occur in the context of modern machine learning. This provides a concrete explanation of CKA sensitivity to outliers, which has been observed in past works, and to transformations that preserve the linear separability of the data, an important generalization attribute. We empirically investigate several weaknesses of the CKA similarity metric, demonstrating situations in which it gives unexpected or counter-intuitive results. Finally we study approaches for modifying representations to maintain functional behaviour while changing the CKA value. Our results illustrate that, in many cases, the CKA value can be easily manipulated without substantial changes to the functional behaviour of the models, and call for caution when leveraging activation alignment metrics.

2023-04-30

International Conference on Learning Representations (Accept: poster)

doi.org

openreview.net

Robust and Controllable Object-Centric Learning through Energy-based Models

Ruixiang Zhang

Tong Che

Boris Ivanovic

Renhao Wang

Marco Pavone

Yoshua Bengio

Liam Paull

Humans are remarkably good at understanding and reasoning about complex visual scenes. The capability to decompose low-level observations in… (see more)to discrete objects allows us to build a grounded abstract representation and identify the compositional structure of the world. Accordingly, it is a crucial step for machine learning models to be capable of inferring objects and their properties from visual scenes without explicit supervision. However, existing works on object-centric representation learning either rely on tailor-made neural network modules or strong probabilistic assumptions in the underlying generative and inference processes. In this work, we present \ours, a conceptually simple and general approach to learning object-centric representations through an energy-based model. By forming a permutation-invariant energy function using vanilla attention blocks readily available in Transformers, we can infer object-centric latent variables via gradient-based MCMC methods where permutation equivariance is automatically guaranteed. We show that \ours can be easily integrated into existing architectures and can effectively extract high-quality object-centric representations, leading to better segmentation accuracy and competitive downstream task performance. Further, empirical evaluations show that \ours's learned representations are robust against distribution shift. Finally, we demonstrate the effectiveness of \ours in systematic compositional generalization, by re-composing learned energy functions for novel scene generation and manipulation.

2023-04-30

International Conference on Learning Representations (Accept: poster)

doi.org

openreview.net

Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier

Bellemare Marc-Emmanuel

Aaron Courville

2023-04-30

International Conference on Learning Representations (Accept: notable-top-5%)

openreview.net

Simplicial Embeddings in Self-Supervised Learning and Downstream Classification

Samuel Lavoie

Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into …

2023-04-30

International Conference on Learning Representations (Accept: notable-top-25%)

doi.org

openreview.net

Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning

Dianbo Liu

Tianmin Shu

Michael Mozer

Nicolas Heess

Yoshua Bengio

In cooperative multi-agent reinforcement learning, a team of agents works together to achieve a common goal. Different environments or tasks… (see more) may require varying degrees of coordination among agents in order to achieve the goal in an optimal way. The nature of coordination will depend on the properties of the environment -- its spatial layout, distribution of obstacles, dynamics, etc. We term this variation of properties within an environment as heterogeneity. Existing literature has not sufficiently addressed the fact that different environments may have different levels of heterogeneity. We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment. Further, we propose a Centralized Training Decentralized Execution learning approach called Stateful Active Facilitator (SAF) that enables agents to work efficiently in high-coordination and high-heterogeneity environments through a differentiable and shared knowledge source used during training and dynamic selection from a shared pool of policies. We evaluate SAF and compare its performance against baselines IPPO and MAPPO on HECOGrid. Our results show that SAF consistently outperforms the baselines across different tasks and different heterogeneity and coordination levels. We release the code for HECOGrid as well as all our experiments.

2023-04-30

ICLR (Accept: poster)

doi.org

openreview.net

Systematic Rectification of Language Models via Dead-End Analysis

Meng Cao

Mehdi Fatemi

Jackie Chi Kit Cheung

Samira Shabanian

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to re… (see more)duce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. We believe this is important since many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach generates notably better results compared to the base LLMs and other techniques in terms of the overall language and detoxification performance.

2023-04-30

ICLR (Accept: poster)

doi.org

openreview.net

TCR/Chimeric Antigen Receptor (CAR) Cross-Antagonism to Fine-Tune CAR-T cell Immunotherapy

Grégoire Altan-Bonnet

Taisuke Kondo

François X. P. Bourassa

Sooraj R. Achar

Justyn DuSold

Pablo Cespedes

Madison Wahlsten

Guillaume Gaud

Paul E Love

Mike Dustin

Paul François

Naomi Taylor

Chimeric antigen receptor (CAR) T cells, are created by extracting T cells from a cancer patient, engineering them to express a CAR targetin… (see more)g a tumor specific molecule, then reintroducing them back into the patient. A patient’s T cells contain their own endogenous T cell receptors (TCRs) however, which could potentially interact with the exogenous CAR inserted into the cell. In this study, we examine how TCR and CAR signals interact upon CAR-T activation. We show that weak TCR stimulation can reduce (antagonize) or increase overall CAR-T response, both in vitro and in vivo, across multiple tumor models, in both mouse and human T cells. We further show that the behavior of these TCR/CAR interactions can be manipulated by changing various characteristics of the TCR, CAR, and associated ligands. While this behavior is complex, we show that it can be described by a single mathematical model based on the adaptive kinetic proofreading scheme of ligand discrimination. We conclude by presenting potential applications for cancer immunotherapy. Intramural Research Program of the National Cancer Institute

2023-04-30

Journal of Immunology (published)

doi.org

Mila Ventures Launchpad

AI Policy Compass

Virtual Information Sessions

Publications

Mila Ventures Launchpad

AI Policy Compass

Virtual Information Sessions

Popular keywords:

Publications