Jian Tang

chuanrui.wang@mila.quebec

Chuanru Wang

Maîtrise recherche - Université de Montréal

farzaneh.heidari@mila.quebec

Farzaneh Heidari

Doctorat - Université de Montréal

Superviseur⋅e principal⋅e :

Guillaume Rabusseau

Huiyu Cai

Doctorat - Université de Montréal

huiyu.cai@mila.quebec

Jianan Zhao

Doctorat - Université de Montréal

jianan.zhao@mila.quebec

Jerry Lu

Doctorat - Université de Montréal

jiarui.lu@mila.quebec

mikhail.galkin@mila.quebec

Meng Qu

Doctorat - Université de Montréal

qumeng@mila.quebec

Michael Galkin

Collaborateur·rice de recherche

sophie.xhonneux@mila.quebec

Minghao Xu

Collaborateur·rice de recherche

minghao.xu@mila.quebec

Stagiaire de recherche - HEC Montréal

nicole.duane@mila.quebec

Sophie Xhonneux

Doctorat - Université de Montréal

Co-superviseur⋅e :

Gauthier Gidel

Xinyu Yuan

Doctorat - Université de Montréal

xinyu.yuan@mila.quebec

yangtian.zhang@mila.quebec

Yangtian Zhang

Collaborateur·rice de recherche

Zewen Chi

Stagiaire de recherche - Beijing Institute of Technology

zewen.chi@mila.quebec

Zhaocheng Zhu

Doctorat - Université de Montréal

Zhihao Zhan

Doctorat - Université de Montréal

zhihao.zhan@mila.quebec

Zuobai Zhang

Doctorat - Université de Montréal

zuobai.zhang@mila.quebec

Publications

A Systematic Study of Joint Representation Learning on Protein Sequences and Structures

Zuobai Zhang

Chuanrui Wang

Minghao Xu

Vijil Chenthamarakshan

Aurelie Lozano

Payel Das

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein functions. Recent sequenc… (voir plus)e representation learning methods based on Protein Language Models (PLMs) excel in sequence-based tasks, but their direct adaptation to tasks involving protein structures remains a challenge. In contrast, structure-based methods leverage 3D structural information with graph neural networks and geometric pre-training methods show potential in function prediction tasks, but still suffers from the limited number of available structures. To bridge this gap, our study undertakes a comprehensive exploration of joint protein representation learning by integrating a state-of-the-art PLM (ESM-2) with distinct structure encoders (GVP, GearNet, CDConv). We introduce three representation fusion strategies and explore different pre-training techniques. Our method achieves significant improvements over existing sequence- and structure-based methods, setting new state-of-the-art for function annotation. This study underscores several important design choices for fusing protein sequence and structure information. Our implementation is available at https://github.com/DeepGraphLearning/ESM-GearNet.

2023-03-11

ArXiv (prépublication)

arxiv.org

Enhancing Protein Language Model with Structure-based Encoder and Pre-training

Zuobai Zhang

Minghao Xu

Aurelie Lozano

Vijil Chenthamarakshan

Payel Das

Protein language models (PLMs) pre-trained on large-scale protein sequence corpora have achieved impressive performance on various downstrea… (voir plus)m protein understanding tasks. Despite the ability to implicitly capture inter-residue contact information, transformer-based PLMs cannot encode protein structures explicitly for better structure-aware protein representations. Besides, the power of pre-training on available protein structures has not been explored for improving these PLMs, though structures are important to determine functions. To tackle these limitations, in this work, we enhance the PLM with structure-based encoder and pre-training. We first explore feasible model architectures to combine the advantages of a state-of-the-art PLM (i.e., ESM-1b) and a state-of-the-art protein structure encoder (i.e., GearNet). We empirically verify the ESM-GearNet that connects two encoders in a series way as the most effective combination model. To further improve the effectiveness of ESM-GearNet, we pre-train it on massive unlabeled protein structures with contrastive learning, which aligns representations of co-occurring subsequences so as to capture their biological correlation. Extensive experiments on EC and GO protein function prediction benchmarks demonstrate the superiority of ESM-GearNet over previous PLMs and structure encoders, and clear performance gains are further achieved by structure-based pre-training upon ESM-GearNet. The source code will be made public upon acceptance.

2023-03-06

ICLR.cc/2023/Workshop/MLDD (poster)

Enhancing Protein Language Model with Structure-based Encoder and Pre-training

Zuobai Zhang

Minghao Xu

Aurelie Lozano

Vijil Chenthamarakshan

Payel Das

2023-03-06

ICLR.cc/2023/Workshop/MLDD (poster)

EurNet: Efficient Multi-Range Relational Modeling of Protein Structure

Minghao Xu

Yuanfan Guo

Yi Xu

Xinlei Chen

Yuandong Tian

Modeling the 3D structures of proteins is critical for obtaining effective protein structure representations, which further boosts protein f… (voir plus)unction understanding. Existing protein structure encoders mainly focus on modeling short-range interactions within protein structures, while they neglect modeling the interactions at multiple length scales that are actually complete interactive patterns in protein structures. To attain complete interaction modeling with efficient computation, we introduce the EurNet for Efficient multi-range relational modeling. In EurNet, we represent the protein structure as a multi-relational residue-level graph with different types of edges for modeling short-range, medium-range and long-range interactions. To efficiently process these different interactive relations, we propose a novel modeling layer, called Gated Relational Message Passing (GRMP), as the basic building block of EurNet. GRMP can capture multiple interactive relations in protein structures with little extra computational cost. We verify the state-of-the-art performance of EurNet on EC and GO protein function prediction benchmarks, and the proposed GRMP layer is proved to achieve better efficiency-performance trade-off than the widely-used relational graph convolution.

2023-03-06

ICLR.cc/2023/Workshop/MLDD (poster)

A Text-guided Protein Design Framework

Shengchao Liu

Yutao Zhu

Jiarui Lu

Zhao Xu

Weili Nie

Anthony James Gitter

Chaowei Xiao

Hongyu Guo

Animashree Anandkumar

2023-02-09

ArXiv (prépublication)

arxiv.org

E3Bind: An End-to-End Equivariant Network for Protein-Ligand Docking

Yang Zhang

Huiyu Cai

Chence Shi

Bozitao Zhong

In silico prediction of the ligand binding pose to a given protein target is a crucial but challenging task in drug discovery. This work foc… (voir plus)uses on blind flexible self-docking, where we aim to predict the positions, orientations and conformations of docked molecules. Traditional physics-based methods usually suffer from inaccurate scoring functions and high inference cost. Recently, data-driven methods based on deep learning techniques are attracting growing interest thanks to their efficiency during inference and promising performance. These methods usually either adopt a two-stage approach by first predicting the distances between proteins and ligands and then generating the final coordinates based on the predicted distances, or directly predicting the global roto-translation of ligands. In this paper, we take a different route. Inspired by the resounding success of AlphaFold2 for protein structure prediction, we propose E3Bind, an end-to-end equivariant network that iteratively updates the ligand pose. E3Bind models the protein-ligand interaction through careful consideration of the geometric constraints in docking and the local context of the binding site. Experiments on standard benchmark datasets demonstrate the superior performance of our end-to-end trainable model compared to traditional and recently-proposed deep learning methods.

2023-02-01

ICLR.cc/2023/Conference (poster)

EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Data

Minghao Xu

Yuanfan Guo

Yi Xu

Xinlei Chen

Yuandong Tian

Modeling spatial relationship in the data remains critical across many different tasks, such as image classification, semantic segmentation … (voir plus)and protein structure understanding. Previous works often use a unified solution like relative positional encoding. However, there exists different kinds of spatial relations, including short-range, medium-range and long-range relations, and modeling them separately can better capture the focus of different tasks on the multi-range relations (e.g., short-range relations can be important in instance segmentation, while long-range relations should be upweighted for semantic segmentation). In this work, we introduce the EurNet for Efficient multi-range relational modeling. EurNet constructs the multi-relational graph, where each type of edge corresponds to short-, medium- or long-range spatial interactions. In the constructed graph, EurNet adopts a novel modeling layer, called gated relational message passing (GRMP), to propagate multi-relational information across the data. GRMP captures multiple relations within the data with little extra computational cost. We study EurNets in two important domains for image and protein structure modeling. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation verify the gains of EurNet over the previous SoTA FocalNet. On the EC and GO protein function prediction benchmarks, EurNet consistently surpasses the previous SoTA GearNet. Our results demonstrate the strength of EurNets on modeling spatial multi-relational data from various domains.

2023-02-01

ICLR.cc/2023/Conference (rejected)

GraphVF: Controllable Protein-Specific 3D Molecule Generation with Variational Flow

Fang Sun

Zhihao Zhan

Hongyu Guo

Ming Zhang

Designing molecules that bind to specific target proteins is a fundamental task in drug discovery. Recent generative models leveraging geome… (voir plus)trical constraints imposed by proteins and molecules have shown great potential in generating protein-specific 3D molecules. Nevertheless, these existing methods fail to generate 3D molecules with 2D skeletal curtailments, which encode pharmacophoric patterns essential to drug potency. To cope with this challenge, we propose GraphVF, which seamlessly integrates geometrical and skeletal restraints into a variational flow framework, where the former is captured through a flow transformation and the latter is encoded by an amortized factorized Gaussian. We empirically verify that our method achieves state-of-the-art performance on protein-specific 3D molecule generation in terms of binding affinity and some other drug properties. In particular, it represents the first controllable geometry-aware, protein-specific molecule generation method, which enables creating 3D molecules with specified chemical sub-structures or drug properties.

2023-02-01

ICLR.cc/2023/Conference (rejected)

Learning on Large-scale Text-attributed Graphs via Variational Inference

Jianan Zhao

Meng Qu

Chaozhuo Li

Hao Yan

Qian Liu

Rui Li

Xing Xie

This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description. An ideal solution for s… (voir plus)uch a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs). However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by training large language models and GNNs together. In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational Expectation-Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such a procedure allows training the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach.

2023-02-01

ICLR.cc/2023/Conference (notable)

Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching

Shengchao Liu

Hongyu Guo

Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labele… (voir plus)d molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy landscape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.

2023-02-01

ICLR.cc/2023/Conference (poster)

Pre-training Protein Structure Encoder via Siamese Diffusion Trajectory Prediction

Zuobai Zhang

Minghao Xu

Aurelie Lozano

Vijil Chenthamarakshan

Payel Das

Due to the determining role of protein structures on diverse protein functions, pre-training representations of proteins on massive unlabele… (voir plus)d protein structures has attracted rising research interests. Among recent efforts on this direction, mutual information (MI) maximization based methods have gained the superiority on various downstream benchmark tasks. The core of these methods is to design correlated views that share common information about a protein. Previous view designs focus on capturing structural motif co-occurrence on the same protein structure, while they cannot capture detailed atom/residue interactions. To address this limitation, we propose the Siamese Diffusion Trajectory Prediction (SiamDiff) method. SiamDiff builds a view as the trajectory that gradually approaches protein native structure from scratch, which facilitates the modeling of atom/residue interactions underlying the protein structural dynamics. Specifically, we employ the multimodal diffusion process as a faithful simulation of the structure-sequence co-diffusion trajectory, where rich patterns of protein structural changes are embedded. On such basis, we design a principled theoretical framework to maximize the MI between correlated multimodal diffusion trajectories. We study the effectiveness of SiamDiff on both residue-level and atom-level structures. On the EC and ATOM3D benchmarks, we extensively compare our method with previous protein structure pre-training approaches. The experimental results verify the consistently superior or competitive performance of SiamDiff on all benchmark tasks compared to existing baselines. The source code will be made public upon acceptance.

2023-02-01

ICLR.cc/2023/Conference (rejected)

Protein Representation Learning by Geometric Structure Pretraining

Zuobai Zhang

Minghao Xu

Arian Rokkum Jamasb

Vijil Chenthamarakshan

Aurelie Lozano

Payel Das

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Ex… (voir plus)isting approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less pretraining data. Our implementation is available at https://github.com/DeepGraphLearning/GearNet.

2023-02-01

ICLR.cc/2023/Conference (poster)