Publications

Graph Inductive Biases in Transformers without Message Passing
Liheng Ma
Chen Lin
Derek Lim
Adriana Romero Soriano
Puneet K. Dokania
Mark Coates
Philip Torr
Ser-Nam Lim
Transformers for graph data are increasingly widely studied and successful in numerous learning tasks. Graph inductive biases are crucial fo… (see more)r Graph Transformers, and previous works incorporate them using message-passing modules and/or positional encodings. However, Graph Transformers that use message-passing inherit known issues of message-passing, and differ significantly from Transformers used in other domains, thus making transfer of research advances more difficult. On the other hand, Graph Transformers without message-passing often perform poorly on smaller datasets, where inductive biases are more crucial. To bridge this gap, we propose the Graph Inductive bias Transformer (GRIT) -- a new Graph Transformer that incorporates graph inductive biases without using message passing. GRIT is based on several architectural changes that are each theoretically and empirically justified, including: learned relative positional encodings initialized with random walk probabilities, a flexible attention mechanism that updates node and node-pair representations, and injection of degree information in each layer. We prove that GRIT is expressive -- it can express shortest path distances and various graph propagation matrices. GRIT achieves state-of-the-art empirical performance across a variety of graph datasets, thus showing the power that Graph Transformers without message-passing can deliver.
2023-01-01
ICML (published)
doi.org
openreview.net
Graphically Structured Diffusion Models
Christian Dietrich Weilbach
William Harvey
Frank Wood
We introduce a framework for automatically defining and learning deep generative models with problem-specific structure. We tackle problem d… (see more)omains that are more traditionally solved by algorithms such as sorting, constraint satisfaction for Sudoku, and matrix factorization. Concretely, we train diffusion models with an architecture tailored to the problem specification. This problem specification should contain a graphical model describing relationships between variables, and often benefits from explicit representation of subcomputations. Permutation invariances can also be exploited. Across a diverse set of experiments we improve the scaling relationship between problem dimension and our model's performance, in terms of both training time and final accuracy.
2023-01-01
ICML (published)
doi.org
openreview.net
A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining
Shengchao Liu
weitao Du
Zhi-Ming Ma
Hongyu Guo
Jian Tang
Molecule pretraining has quickly become the go-to schema to boost the performance of AI-based drug discovery. Naturally, molecules can be re… (see more)presented as 2D topological graphs or 3D geometric point clouds. Although most existing pertaining methods focus on merely the single modality, recent research has shown that maximizing the mutual information (MI) between such two modalities enhances the molecule representation ability. Meanwhile, existing molecule multi-modal pretraining approaches approximate MI based on the representation space encoded from the topology and geometry, thus resulting in the loss of critical structural information of molecules. To address this issue, we propose MoleculeSDE. MoleculeSDE leverages group symmetric (e.g., SE(3)-equivariant and reflection-antisymmetric) stochastic differential equation models to generate the 3D geometries from 2D topologies, and vice versa, directly in the input space. It not only obtains tighter MI bound but also enables prosperous downstream tasks than the previous work. By comparing with 17 pretraining baselines, we empirically verify that MoleculeSDE can learn an expressive representation with state-of-the-art performance on 26 out of 32 downstream tasks.
2023-01-01
ICML (published)
doi.org
openreview.net
Guessing Random Additive Noise Decoding
Syed Mohsin Abbas
Marwan Jalaleddine
Warren Gross
2023-01-01
(published)
doi.org
GUILGET: GUI Layout GEneration with Transformer
Andrey Sobolevsky
Guillaume-Alexandre Bilodeau
Jinghui Cheng
Jin Guo
2023-01-01
Canadian AI (published)
doi.org
arxiv.org
A Heat Diffusion Perspective on Geodesic Preserving Dimensionality Reduction
Guillaume Huguet
Alexander Tong
Edward De Brouwer
Yanlei Zhang
Guy Wolf
Ian Adelstein
Smita Krishnaswamy
openreview.net
Hierarchical Distributed Energy Management Framework for Multiple Greenhouses Considering Demand Response
Ehsan Rezaei
Hanane Dagdougui
Kianoosh Ojand
Greenhouses are a key component of modernised agriculture, aiming for producing high-quality crops and plants. Furthermore, a network of gre… (see more)enhouses has enormous potential as part of demand response programs. Saving energy during off-peak time, reducing power consumption and delaying the start time of subsystems during on-peak time are some strategies that can be used to limit power exchanged with the main grid. In this work, a hierarchical distributed alternating direction method of multipliers-based model predictive control framework is proposed that has two main objectives: 1) providing appropriate conditions for greenhouses' crops and plants to grow, and 2) limiting the total power exchanged with the main grid. At each time step in the framework, an aggregator coordinates the greenhouses to reach a consensus and limit the total electric power exchanged while managing shared resources, e.g., reservoir water. The proposed framework's performance is investigated through a case study.
2023-01-01
IEEE Transactions on Sustainable Energy (published)
doi.org
How can intelligent systems revolutionise healthcare?
Narges Armanfard
2023-01-01
Futurum Careers (published)
doi.org
How Useful Are Educational Questions Generated by Large Language Models?
Sabina Elkins
Ekaterina Kochmar
Jackie Cheung
Iulian V. Serban
2023-01-01
AIED (Posters/Late Breaking Results/...) (published)
doi.org
arxiv.org
Human-Centered Responsible Artificial Intelligence: Current & Future Trends
Mohammad Tahaei
Marios Constantinides
Daniele Quercia
Sean Kennedy
Michael Muller
Simone Stumpf
Q. Vera Liao
Ricardo Baeza-Yates
Lora Aroyo
Jess Holbrook
Ewa Luger
Michael Madaio
Ilana Golbin Blumenfeld
Maria De-Arteaga
Jessica Vitak
Alexandra Olteanu
2023-01-01
CHI Extended Abstracts (published)
doi.org
arxiv.org
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Eric Nguyen
Michael Poli
Marjan Faizi
Armin W Thomas
Callum Birch-Sykes
Michael Wornow
Aman Patel
Clayton M. Rabideau
Stefano Massaroli
Yoshua Bengio
Stefano Ermon
Stephen Baccus
Christopher Re
openreview.net
Identification of Substitutable Context-Free Languages over Infinite Alphabets from Positive Data
Yutaro Numaya
Diptarama Hendrian
Ryo Yoshinaka
Ayumi Shinohara
François Coste
Faissal Ouardi
Guillaume Rabusseau
This paper is concerned with the identification in the limit from positive data of sub-stitutable context-free languages cfl s) over infinit… (see more)e alphabets. Clark and Eyraud (2007) showed that substitutable cfl s over finite alphabets are learnable in this learning paradigm. We show that substitutable cfl s generated by grammars whose production rules may have predicates that represent sets of potentially infinitely many terminal symbols in a compact manner are learnable if the terminal symbol sets represented by those predicates are learnable, under a certain condition. This can be seen as a result parallel to Argyros and D’Antoni’s work (2018) that amplifies the query learnability of predicate classes to that of symbolic automata classes. Our result is the first that shows such amplification is possible for identifying some cfl s in the limit from positive data.
2023-01-01
ICGI (published)
dblp.uni-trier.de