Dominique Beaini

2023-07-30

TMLR (accepted)

Repurposing Density Functional Theory to Suit Deep Learning

Alexander Mathiasen

Hatem Helal

Paul Balanca

Kerstin Klaser

Josef Dean

Carlo Luschi

Andrew William Fitzgibbon

Dominic Masters

Density Functional Theory (DFT) accurately predicts the properties of molecules given their atom types and positions, and often serves as gr… (see more)ound truth for molecular property prediction tasks. Neural Networks (NN) are popular tools for such tasks and are trained on DFT datasets, with the aim to approximate DFT at a fraction of the computational cost. Research in other areas of machine learning has shown that generalisation performance of NNs tends to improve with increased dataset size, however, the computational cost of DFT limits the size of DFT datasets. We present PySCFIPU, a DFT library that allows us to iterate on both dataset generation and NN training. We create QM10X, a dataset with 100M conformers, in 13 hours, on which we subsequently train SchNet in 12 hours. We show that the predictions of SchNet improve solely by increasing training data without incorporating further inductive biases.

2023-07-28

ICML.cc/2023/Workshop/SynS_and_ML (published)

Task-Agnostic Graph Neural Network Evaluation via Adversarial Collaboration

Xiangyu Zhao

Hannes Stärk

Pietro Lió

Yiren Zhao

2023-03-06

ICLR.cc/2023/Workshop/MLDD (poster)

Generating QM1B with PySCF$_{\text{IPU}}$

Alexander Mathiasen

Hatem Helal

Kerstin Klaser

Paul Balanca

Josef Dean

Carlo Luschi

Andrew William Fitzgibbon

Dominic Masters

Generating QM1B with PySCFIPU

Alexander Mathiasen

Hatem Helal

Kerstin Klaser

Paul Balanca

Josef Dean

Carlo Luschi

Andrew William Fitzgibbon

Dominic Masters

2023-01-01

NeurIPS (published)

arxiv.org

GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Prediction

Dominic Masters

Josef Dean

Kerstin Klaser

Zhiyi Li

Samuel Maddrell-Mander

Adam Sanders

Hatem Helal

Deniz Beker

Ladislav Rampášek

2022-11-18

ArXiv (preprint)

arxiv.org

3D Infomax improves GNNs for Molecular Property Prediction

Hannes Stärk

Gabriele Corso

Prudencio Tossou

Christian Dallago

Stephan Günnemann

Pietro Lió

Molecular property prediction is one of the fastest-growing applications of deep learning with critical real-world impacts. Including 3D mol… (see more)ecular structure as input to learned models improves their predictions for many molecular properties. However, this information is infeasible to compute at the scale required by most real-world applications. We propose pre-training a model to understand the geometry of molecules given only their 2D molecular graph. Using methods from self-supervised learning, we maximize the mutual information between a 3D summary vector and the representations of a Graph Neural Network (GNN) such that they contain latent 3D information. During fine-tuning on molecules with unknown geometry, the GNN still generates implicit 3D information and can use it to inform downstream tasks. We show that 3D pre-training provides significant improvements for a wide range of molecular properties, such as a 22% average MAE reduction on eight quantum mechanical properties. Crucially, the learned representations can be effectively transferred between datasets with vastly different molecules.

2022-06-28

Proceedings of the 39th International Conference on Machine Learning (published)

proceedings.mlr.press

Long Range Graph Benchmark

Vijay Prakash Dwivedi

Ladislav Rampášek

Mikhail Galkin

Ali Parviz

Guy Wolf

Anh Tuan Luu

Graph Neural Networks (GNNs) that are based on the message passing (MP) paradigm generally exchange information between 1-hop neighbors to b… (see more)uild node representations at each layer. In principle, such networks are not able to capture long-range interactions (LRI) that may be desired or necessary for learning a given task on graphs. Recently, there has been an increasing interest in development of Transformer-based methods for graphs that can consider full node connectivity beyond the original sparse structure, thus enabling the modeling of LRI. However, MP-GNNs that simply rely on 1-hop message passing often fare better in several existing graph benchmarks when combined with positional feature representations, among other innovations, hence limiting the perceived utility and ranking of Transformer-like architectures. Here, we present the Long Range Graph Benchmark (LRGB) with 5 graph learning datasets: PascalVOC-SP, COCO-SP, PCQM-Contact, Peptides-func and Peptides-struct that arguably require LRI reasoning to achieve strong performance in a given task. We benchmark both baseline GNNs and Graph Transformer networks to verify that the models which capture long-range dependencies perform significantly better on these tasks. Therefore, these datasets are suitable for benchmarking and exploration of MP-GNNs and Graph Transformer architectures that are intended to capture LRI.

Recipe for a General, Powerful, Scalable Graph Transformer

Ladislav Rampášek

Mikhail Galkin

Vijay Prakash Dwivedi

Anh Tuan Luu

Guy Wolf

We propose a recipe on how to build a general, powerful, scalable (GPS) graph Transformer with linear complexity and state-of-the-art result… (see more)s on a diverse set of benchmarks. Graph Transformers (GTs) have gained popularity in the field of graph representation learning with a variety of recent publications but they lack a common foundation about what constitutes a good positional or structural encoding, and what differentiates them. In this paper, we summarize the different types of encodings with a clearer definition and categorize them as being

Directional Graph Networks

Saro Passaro

Vincent Létourneau

William L. Hamilton

Gabriele Corso

Pietro Lió

The lack of anisotropic kernels in graph neural networks (GNNs) strongly limits their expressiveness, contributing to well-known issues such… (see more) as over-smoothing. To overcome this limitation, we propose the first globally consistent anisotropic kernels for GNNs, allowing for graph convolutions that are defined according to topologicaly-derived directional flows. First, by defining a vector field in the graph, we develop a method of applying directional derivatives and smoothing by projecting node-specific messages into the field. Then, we propose the use of the Laplacian eigenvectors as such vector field. We show that the method generalizes CNNs on an

2021-07-01

Proceedings of the 38th International Conference on Machine Learning (published)

proceedings.mlr.press

Rethinking Graph Transformers with Spectral Attention

Devin Kreuzer

William L. Hamilton

Vincent Létourneau

Prudencio Tossou

In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data str… (see more)uctures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the

Improving Convolutional Neural Networks Via Conservative Field Regularisation and Integration