Publications

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen

Michael Poli

Marjan Faizi

Armin W Thomas

Callum Birch-Sykes

Michael Wornow

Aman Patel

Clayton M. Rabideau

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen

Michael Poli

Marjan Faizi

Armin W Thomas

Callum Birch-Sykes

Michael Wornow

Aman Patel

Clayton M. Rabideau

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen

Michael Poli

Marjan Faizi

Armin W Thomas

Callum Birch-Sykes

Michael Wornow

Aman Patel

Clayton M. Rabideau

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen

Michael Poli

Marjan Faizi

Armin W Thomas

Callum Birch-Sykes

Michael Wornow

Aman Patel

Clayton M. Rabideau

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Eric Nguyen

Michael Poli

Marjan Faizi

Armin W Thomas

Callum Birch-Sykes

Michael Wornow

Aman Patel

Clayton M. Rabideau

Stefano Ermon

Stephen Baccus

Christopher Re

Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language mode… (voir plus)ls, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on toke

2023-06-27

ArXiv (prépublication)

Laurence Perreault-Levasseur

Pixelated Reconstruction of Foreground Density and Background Surface Brightness in Gravitational Lensing Systems Using Recurrent Inference Machines

Alexandre Adam

Yashar Hezaveh

Max Welling

Modeling strong gravitational lenses in order to quantify distortions in the images of background sources and to reconstruct the mass densit… (voir plus)y in foreground lenses has been a difficult computational challenge. As the quality of gravitational lens images increases, the task of fully exploiting the information they contain becomes computationally and algorithmically more difficult. In this work, we use a neural network based on the recurrent inference machine to reconstruct simultaneously an undistorted image of the background source and the lens mass density distribution as pixelated maps. The method iteratively reconstructs the model parameters (the image of the source and a pixelated density map) by learning the process of optimizing the likelihood given the data using the physical model (a ray-tracing simulation), regularized by a prior implicitly learned by the neural network through its training data. When compared to more traditional parametric models, the proposed method is significantly more expressive and can reconstruct complex mass distributions, which we demonstrate by using realistic lensing galaxies taken from the IllustrisTNG cosmological hydrodynamic simulation.

2023-06-27

The Astrophysical Journal (publié)

Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization

Dianbo Liu

Alex Lamb

Xu Ji

Pascal Notsawo

Michael Curtis Mozer

Kenji Kawaguchi

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

Hypernetworks for Zero-shot Transfer in Reinforcement Learning

Sahand Rezaei-Shoshtari

Charlotte Morissette

Francois Hogan

Gregory Dudek

David Meger

In this paper, hypernetworks are trained to generate behaviors across a range of unseen task conditions, via a novel TD-based training objec… (voir plus)tive and data from a set of near-optimal RL solutions for training tasks. This work relates to meta RL, contextual RL, and transfer learning, with a particular focus on zero-shot performance at test time, enabled by knowledge of the task parameters (also known as context). Our technical approach is based upon viewing each RL algorithm as a mapping from the MDP specifics to the near-optimal value function and policy and seek to approximate it with a hypernetwork that can generate near-optimal value functions and policies, given the parameters of the MDP. We show that, under certain conditions, this mapping can be considered as a supervised learning problem. We empirically evaluate the effectiveness of our method for zero-shot transfer to new reward and transition dynamics on a series of continuous control tasks from DeepMind Control Suite. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

Latent Space Evolution under Incremental Learning with Concept Drift (Student Abstract)

Charles Bourbeau

Audrey Durand

This work investigates the evolution of latent space when deep learning models are trained incrementally in non-stationary environments that… (voir plus) stem from concept drift. We propose a methodology for visualizing the incurred change in latent representations. We further show that classes not targeted by concept drift can be negatively affected, suggesting that the observation of all classes during learning may regularize the latent space.

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

Signed Laplacian Graph Neural Networks

Yu Li

Meng Qu

Jian Tang

Yi Chang

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

The Effect of diversity in Meta-Learning

Ramnath Kumar

Tristan Deleu

Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task … (voir plus)distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (publié)