Tristan Deleu

The Effect of diversity in Meta-Learning

Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task … (see more)distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

2023-06-26

Proceedings of the AAAI Conference on Artificial Intelligence (published)

doi.org

openreview.net

BatchGFN: Generative Flow Networks for Batch Active Learning

Shreshth A Malik

Salem Lahlou

Andrew Jesson

Moksh J. Jain

Nikolay Malkin

Tristan Deleu

Yoshua Bengio

Yarin Gal

We introduce BatchGFN—a novel approach for pool-based active learning that uses generative flow networks to sample sets of data points pro… (see more)portional to a batch reward. With an appropriate reward function to quantify the utility of acquiring a batch, such as the joint mutual information between the batch and the model parameters, BatchGFN is able to construct highly informative batches for active learning in a principled way. We show our approach enables sampling near-optimal utility batches at inference time with a single forward pass per point in the batch in toy regression problems. This alleviates the computational complexity of batch-aware algorithms and removes the need for greedy approximations to find maximizers for the batch reward. We also present early results for amortizing training across acquisition steps, which will enable scaling to real-world tasks.

2023-06-19

ICML.cc/2023/Workshop/SPIGM (poster)

doi.org

openreview.net

Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation

2023-06-19

ICML.cc/2023/Workshop/SPIGM (poster)

doi.org

openreview.net

GFlowNets and variational inference

Edward J Hu

Katie E Everett

Dinghuai Zhang

Yoshua Bengio

This paper builds bridges between two families of probabilistic algorithms: (hierarchical) variational inference (VI), which is typically us… (see more)ed to model distributions over continuous spaces, and generative flow networks (GFlowNets), which have been used for distributions over discrete structures such as graphs. We demonstrate that, in certain cases, VI algorithms are equivalent to special cases of GFlowNets in the sense of equality of expected gradients of their learning objectives. We then point out the differences between the two families and show how these differences emerge experimentally. Notably, GFlowNets, which borrow ideas from reinforcement learning, are more amenable than VI to off-policy training without the cost of high gradient variance induced by importance sampling. We argue that this property of GFlowNets can provide advantages for capturing diversity in multimodal target distributions.

2023-02-01

ICLR.cc/2023/Conference (poster)

doi.org

openreview.net

GFlowNets for AI-Driven Scientific Discovery

Moksh J. Jain

Tristan Deleu

Jason Hartford

Cheng-Hao Liu

Alex Hernandez-Garcia

Yoshua Bengio

Tackling the most pressing problems for humanity, such as the climate crisis and the threat of global pandemics, requires accelerating the p… (see more)ace of scientific discovery. While science has traditionally relied...

2023-02-01

ArXiv (preprint)

doi.org

arxiv.org

A theory of continuous generative flow networks

Alex Hernandez-Garcia

Lena Nehale Ezzine

Yoshua Bengio

Nikolay Malkin

Generative flow networks (GFlowNets) are amortized variational inference algorithms that are trained to sample from unnormalized target dist… (see more)ributions over compositional objects. A key limitation of GFlowNets until this time has been that they are restricted to discrete spaces. We present a theory for generalized GFlowNets, which encompasses both existing discrete GFlowNets and ones with continuous or hybrid state spaces, and perform experiments with two goals in mind. First, we illustrate critical points of the theory and the importance of various assumptions. Second, we empirically demonstrate how observations about discrete GFlowNets transfer to the continuous case and show strong results compared to non-GFlowNet baselines on several previously studied tasks. This work greatly widens the perspectives for the application of GFlowNets in probabilistic inference and various modeling settings.

2023-01-30

ArXiv (preprint)

GFlowNet Foundations

Edward J Hu

Mo Tiwari

GFlowNet Foundations

Edward J. Hu

Mo Tiwari

Emmanuel Bengio

GFlowNets for AI-Driven Scientific Discovery

Moksh J. Jain

Tristan Deleu

Jason Hartford

Cheng-Hao Liu

Alex Hernandez-Garcia

Yoshua Bengio

Tackling the most pressing problems for humanity, such as the climate crisis and the threat of global pandemics, requires accelerating the p… (see more)ace of scientific discovery. While science has traditionally relied...

2023-01-01

Digital Discovery (published)

doi.org

arxiv.org

Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding … (see more)is limited. In this work, we provide evidence that disentangled representations coupled with sparse base-predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM base-predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.

2023-01-01

ICML (published)

openreview.net

Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding … (see more)is limited. In this work, we provide evidence that disentangled representations coupled with sparse task-specific predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.

2023-01-01

ICML (published)

proceedings.mlr.press

openreview.net

A theory of continuous generative flow networks

Alex Hernandez-Garcia

Lena Nehale Ezzine