Publications

Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching
Shengchao Liu
Hongyu Guo
Jian Tang
Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labele… (voir plus)d molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy landscape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
Alireza Mousavi-Hosseini
Sejun Park
Manuela Girotti
Ioannis Mitliagkas
Murat A. Erdogdu
We study the problem of training a two-layer neural network (NN) of arbitrary width using stochastic gradient descent (SGD) where the input …
2023-01-31
ICLR.cc/2023/Conference (notable)
doi.org
openreview.net
Predictive Inference with Feature Conformal Prediction
Jiaye Teng
Chuan Wen
Dinghuai Zhang
Yoshua Bengio
Yang Gao
Yang Yuan
Conformal prediction is a distribution-free technique for establishing valid prediction intervals. Although conventionally people conduct co… (voir plus)nformal prediction in the output space, this is not the only possibility. In this paper, we propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces by leveraging the inductive bias of deep representation learning. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods. Apart from experiments on existing predictive inference benchmarks, we also demonstrate the state-of-the-art performance of the proposed methods on large-scale tasks such as ImageNet classification and Cityscapes image segmentation.The code is available at https://github.com/AlvinWen428/FeatureCP.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Protein Representation Learning by Geometric Structure Pretraining
Zuobai Zhang
Minghao Xu
Arian Jamasb
Vijil Chenthamarakshan
Aurelie Lozano
Payel Das
Jian Tang
Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Ex… (voir plus)isting approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less pretraining data. Our implementation is available at https://github.com/DeepGraphLearning/GearNet.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Protein Sequence and Structure Co-design with Equivariant Translation
Chence Shi
Chuanrui Wang
Jiarui Lu
Bozitao Zhong
Jian Tang
Proteins are macromolecules that perform essential functions in all living organisms. Designing novel proteins with specific structures and … (voir plus)desired functions has been a long-standing challenge in the field of bioengineering. Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models, both of which suffer from high inference costs. In this paper, we propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state from random initialization, based on context features given a priori. Our model consists of a trigonometry-aware encoder that reasons geometrical constraints and interactions from context features, and a roto-translation equivariant decoder that translates protein sequence and structure interdependently. Notably, all protein amino acids are updated in one shot in each translation step, which significantly accelerates the inference process. Experimental results across multiple tasks show that our model outperforms previous state-of-the-art baselines by a large margin, and is able to design proteins of high fidelity as regards both sequence and structure, with running time orders of magnitude less than sampling-based methods.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks
Jesse Farebrother
Joshua Greaves
Rishabh Agarwal
Charline Le Lan
Ross Goroshin
Pablo Samuel Castro
Bellemare Marc-Emmanuel
Auxiliary tasks improve the representations learned by deep reinforcement learning agents. Analytically, their effect is reasonably well und… (voir plus)erstood; in practice, however, their primary use remains in support of a main learning objective, rather than as a method for learning representations. This is perhaps surprising given that many auxiliary tasks are defined procedurally, and hence can be treated as an essentially infinite source of information about the environment. Based on this observation, we study the effectiveness of auxiliary tasks for learning rich representations, focusing on the setting where the number of tasks and the size of the agent's network are simultaneously increased. For this purpose, we derive a new family of auxiliary tasks based on the successor measure. These tasks are easy to implement and have appealing theoretical properties. Combined with a suitable off-policy learning rule, the result is a representation learning algorithm that can be understood as extending Mahadevan & Maggioni (2007)'s proto-value functions to deep reinforcement learning -- accordingly, we call the resulting object proto-value networks. Through a series of experiments on the Arcade Learning Environment, we demonstrate that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms, using only linear approximation and a small number (~4M) of interactions with the environment's reward function.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Reliability of CKA as a Similarity Measure in Deep Learning
MohammadReza Davari
Stefan Horoi
Amine Natik
Guillaume Lajoie
Guy Wolf
Eugene Belilovsky
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different w… (voir plus)ays. The Centered Kernel Alignment (CKA) similarity metric, particularly its linear variant, has recently become a popular approach and has been widely used to compare representations of a network's different layers, of architecturally similar networks trained differently, or of models with different architectures trained on the same data. A wide variety of conclusions about similarity and dissimilarity of these various representations have been made using CKA. In this work we present analysis that formally characterizes CKA sensitivity to a large class of simple transformations, which can naturally occur in the context of modern machine learning. This provides a concrete explanation of CKA sensitivity to outliers, which has been observed in past works, and to transformations that preserve the linear separability of the data, an important generalization attribute. We empirically investigate several weaknesses of the CKA similarity metric, demonstrating situations in which it gives unexpected or counter-intuitive results. Finally we study approaches for modifying representations to maintain functional behaviour while changing the CKA value. Our results illustrate that, in many cases, the CKA value can be easily manipulated without substantial changes to the functional behaviour of the models, and call for caution when leveraging activation alignment metrics.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Revisiting Populations in multi-agent Communication
Paul Michel
Mathieu Rita
Kory Mathewson
Olivier Tieleman
Angeliki Lazaridou
Despite evidence from cognitive sciences that larger groups of speakers tend to develop more structured languages in human communication, sc… (voir plus)aling up to populations has failed to yield significant benefits in emergent multi-agent communication. In this paper we advocate for an alternate population-level training paradigm for referential games based on the idea of "partitioning" the agents into sender-receiver pairs and limiting co-adaptation across pairs. We show that this results in optimizing a different objective at the population level, where agents maximize (1) their respective "internal" communication accuracy and (2) some measure of alignment between agents. In experiments, we find that this leads to the emergence of languages that are significantly more compositional. Moreover, when agents are trained in populations that are not fully connected (ie. not all agent pairs interact at training time), this approach reduces multi-linguality and improves zero-shot communication with new agents (ie. agents are able to communicate successfully with other agents outside their training partners).
2023-01-31
ICLR.cc/2023/Conference (poster)
openreview.net
openreview.net
Robust and Controllable Object-Centric Learning through Energy-based Models
Ruixiang Zhang
Tong Che
Boris Ivanovic
Renhao Wang
Marco Pavone
Yoshua Bengio
Liam Paull
Humans are remarkably good at understanding and reasoning about complex visual scenes. The capability to decompose low-level observations in… (voir plus)to discrete objects allows us to build a grounded abstract representation and identify the compositional structure of the world. Accordingly, it is a crucial step for machine learning models to be capable of inferring objects and their properties from visual scenes without explicit supervision. However, existing works on object-centric representation learning either rely on tailor-made neural network modules or strong probabilistic assumptions in the underlying generative and inference processes. In this work, we present \ours, a conceptually simple and general approach to learning object-centric representations through an energy-based model. By forming a permutation-invariant energy function using vanilla attention blocks readily available in Transformers, we can infer object-centric latent variables via gradient-based MCMC methods where permutation equivariance is automatically guaranteed. We show that \ours can be easily integrated into existing architectures and can effectively extract high-quality object-centric representations, leading to better segmentation accuracy and competitive downstream task performance. Further, empirical evaluations show that \ours's learned representations are robust against distribution shift. Finally, we demonstrate the effectiveness of \ours in systematic compositional generalization, by re-composing learned energy functions for novel scene generation and manipulation.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
Pierluca D'Oro
Max Schwarzer
Evgenii Nikishin
Pierre-Luc Bacon
Bellemare Marc-Emmanuel
Aaron Courville
2023-01-31
ICLR.cc/2023/Conference (notable)
openreview.net
openreview.net
Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Samuel Lavoie
Christos Tsirigotis
Max Schwarzer
Kenji Kawaguchi
Ankit Vani
Michael Noukhovitch
Aaron Courville
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into …
2023-01-31
ICLR.cc/2023/Conference (notable)
doi.org
openreview.net
Social isolation is linked to classical risk factors of Alzheimer's disease-related dementias
Kimia Shafighi
Sylvia Villeneuve
Pedro Rosa Neto
AmanPreet Badhwar
Judes Poirier
Vaibhav Sharma
Yasser Iturria Medina
Patricia P. Silveira
Laurette Dubé
David Glahn
Danilo Bzdok
Alzheimer’s disease and related dementias is a major public health burden–compounding over upcoming years due to longevity. Recently, cl… (voir plus)inical evidence hinted at the experience of social isolation in expediting dementia onset. In 502,506 UK Biobank participants and 30,097 participants from the Canadian Longitudinal Study of Aging, we revisited traditional risk factors for developing dementia in the context of loneliness and lacking social support. Across these measures of subjective and objective social deprivation, we have identified strong links between individuals’ social capital and various indicators of Alzheimer’s disease and related dementias risk, which replicated across both population cohorts. The quality and quantity of daily social encounters had deep connections with key aetiopathological factors, which represent 1) personal habits and lifestyle factors, 2) physical health, 3) mental health, and 4) societal and external factors. Our population-scale assessment suggest that social lifestyle determinants are linked to most neurodegeneration risk factors, highlighting them as promising targets for preventive clinical action.
2023-01-31
PLOS ONE (publié)
doi.org