Dinghuai Zhang

Latent State Marginalization as a Low-cost Approach for Improving Exploration

Aaron Courville

Qinqing Zheng

Amy Zhang

Ricky T. Q. Chen

2023-02-01

ICLR.cc/2023/Conference (poster)

P REDICTIVE I NFERENCE WITH F EATURE C ONFORMAL P REDICTION

Jiaye Teng

Chuan Wen

Yan Gao

Yang Yuan

2023-02-01

ICLR.cc/2023/Conference (poster)

A theory of continuous generative flow networks

Alex Hernandez-Garcia

Lena Nehale Ezzine

Generative flow networks (GFlowNets) are amortized variational inference algorithms that are trained to sample from unnormalized target dist… (voir plus)ributions over compositional objects. A key limitation of GFlowNets until this time has been that they are restricted to discrete spaces. We present a theory for generalized GFlowNets, which encompasses both existing discrete GFlowNets and ones with continuous or hybrid state spaces, and perform experiments with two goals in mind. First, we illustrate critical points of the theory and the importance of various assumptions. Second, we empirically demonstrate how observations about discrete GFlowNets transfer to the continuous case and show strong results compared to non-GFlowNet baselines on several previously studied tasks. This work greatly widens the perspectives for the application of GFlowNets in probabilistic inference and various modeling settings.

2023-01-30

ArXiv (prépublication)

arxiv.org

GFlowOut: Dropout with Generative Flow Networks

Dianbo Liu

Moksh J. Jain

Bonaventure F. P. Dossou

Qianli Shen

Anirudh Goyal

Xu Ji

Kenji Kawaguchi

2023-01-01

ICML (publié)

GFlowOut: Dropout with Generative Flow Networks

Dianbo Liu

Moksh J. Jain

Bonaventure F. P. Dossou

Qianli Shen

Anirudh Goyal

Xu Ji

Kenji Kawaguchi

2023-01-01

ICML (publié)

Stochastic Generative Flow Networks

Ling Pan

Moksh J. Jain

Longbo Huang

2023-01-01

UAI (publié)

Stochastic Generative Flow Networks

Ling Pan

Moksh J. Jain

Longbo Huang

Generative Flow Networks (or GFlowNets for short) are a family of probabilistic agents that learn to sample complex combinatorial structures… (voir plus) through the lens of ``inference as control''. They have shown great potential in generating high-quality and diverse candidates from a given energy landscape. However, existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with stochastic dynamics, which can limit their applicability. To overcome this challenge, this paper introduces Stochastic GFlowNets, a new algorithm that extends GFlowNets to stochastic environments. By decomposing state transitions into two steps, Stochastic GFlowNets isolate environmental stochasticity and learn a dynamics model to capture it. Extensive experimental results demonstrate that Stochastic GFlowNets offer significant advantages over standard GFlowNets as well as MCMC- and RL-based approaches, on a variety of standard benchmarks with stochastic dynamics.

2023-01-01

UAI (publié)

A theory of continuous generative flow networks

Alex Hernandez-Garcia

Lena Nehale Ezzine

Generative flow networks (GFlowNets) are amortized variational inference algorithms that are trained to sample from unnormalized target dist… (voir plus)ributions over compositional objects. A key limitation of GFlowNets until this time has been that they are restricted to discrete spaces. We present a theory for generalized GFlowNets, which encompasses both existing discrete GFlowNets and ones with continuous or hybrid state spaces, and perform experiments with two goals in mind. First, we illustrate critical points of the theory and the importance of various assumptions. Second, we empirically demonstrate how observations about discrete GFlowNets transfer to the continuous case and show strong results compared to non-GFlowNet baselines on several previously studied tasks. This work greatly widens the perspectives for the application of GFlowNets in probabilistic inference and various modeling settings.

2023-01-01

ICML (publié)

A theory of continuous generative flow networks

Alex Hernandez-Garcia

Lena Nehale Ezzine

2023-01-01

ICML (publié)

Latent State Marginalization as a Low-cost Approach for Improving Exploration

Aaron Courville

Qinqing Zheng

Amy Zhang

Ricky T. Q. Chen

While the maximum entropy (MaxEnt) reinforcement learning (RL) framework -- often touted for its exploration and robustness capabilities -- … (voir plus)is usually motivated from a probabilistic perspective, the use of deep probabilistic models has not gained much traction in practice due to their inherent complexity. In this work, we propose the adoption of latent variable policies within the MaxEnt framework, which we show can provably approximate any policy distribution, and additionally, naturally emerges under the use of world models with a latent belief state. We discuss why latent variable policies are difficult to train, how naive approaches can fail, then subsequently introduce a series of improvements centered around low-cost marginalization of the latent state, allowing us to make full use of the latent state at minimal additional cost. We instantiate our method under the actor-critic framework, marginalizing both the actor and critic. The resulting algorithm, referred to as Stochastic Marginal Actor-Critic (SMAC), is simple yet effective. We experimentally validate our method on continuous control tasks, showing that effective marginalization can lead to better exploration and more robust training. Our implementation is open sourced at https://github.com/zdhNarsil/Stochastic-Marginal-Actor-Critic.

2022-10-03

ArXiv (prépublication)

arxiv.org

Unifying Generative Models with GFlowNets

Ricky T. Q. Chen

There are many frameworks for deep generative modeling, each often presented with their own specific training algorithms and inference metho… (voir plus)ds. Here, we demonstrate the connections between existing deep generative models and the recently introduced GFlowNet framework, a probabilistic inference machine which treats sampling as a decision-making process. This analysis sheds light on their overlapping traits and provides a unifying viewpoint through the lens of learning with Markovian trajectories. Our framework provides a means for unifying training and inference algorithms, and provides a route to shine a unifying light over many generative models. Beyond this, we provide a practical and experimentally verified recipe for improving generative modeling with insights from the GFlowNet perspective.

2022-09-06

ArXiv (prépublication)