Portrait of Xu Ji is unavailable

Xu Ji

Alumni

Publications

Sources of Richness and Ineffability for Phenomenally Conscious States
Xu Ji
Eric Elmoznino
George Deane
Axel Constant
Guillaume Dumas
Guillaume Lajoie
Jonathan Simon
Yoshua Bengio
Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully descr… (see more)ibe or recall. The problem of ineffability, in particular, is a longstanding issue in philosophy that partly motivates the explanatory gap: the belief that consciousness cannot be reduced to underlying physical processes. Here, we provide an information theoretic dynamical systems perspective on the richness and ineffability of consciousness. In our framework, the richness of conscious experience corresponds to the amount of information in a conscious state and ineffability corresponds to the amount of information lost at different stages of processing. We describe how attractor dynamics in working memory would induce impoverished recollections of our original experiences, how the discrete symbolic nature of language is insufficient for describing the rich and high-dimensional structure of experiences, and how similarity in the cognitive function of two individuals relates to improved communicability of their experiences to each other. While our model may not settle all questions relating to the explanatory gap, it makes progress toward a fully physicalist explanation of the richness and ineffability of conscious experience: two important aspects that seem to be part of what makes qualitative character so puzzling.
2023-12-31
Neuroscience of Consciousness (published)
doi.org
arxiv.org
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization for Heterogeneous Representational Coarseness
Dianbo Liu
Alex Lamb
Xu Ji
Pascal Junior Tikeng Notsawo
Michael Mozer
Yoshua Bengio
Kenji Kawaguchi
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit. It ha… (see more)s been theoretically and empirically shown that discretization of representations leads to improved generalization, including in reinforcement learning where discretization can be used to bottleneck multi-agent communication to promote agent specialization and robustness. The discretization tightness of most VQ-based methods is defined by the number of discrete codes in the representation vector and the codebook size, which are fixed as hyperparameters. In this work, we propose learning to dynamically select discretization tightness conditioned on inputs, based on the hypothesis that data naturally contains variations in complexity that call for different levels of representational coarseness which is observed in many heterogeneous data sets. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks with heterogeneity in representations.
2023-06-25
Proceedings of the AAAI Conference on Artificial Intelligence (published)
doi.org
arxiv.org
GFlowNets and Variational Inference
Nikolay Malkin
Salem Lahlou
Tristan Deleu
Xu Ji
Edward Hu
Katie Everett
Dinghuai Zhang
Yoshua Bengio
This paper builds bridges between two families of probabilistic algorithms: (hierarchical) variational inference (VI), which is typically us… (see more)ed to model distributions over continuous spaces, and generative flow networks (GFlowNets), which have been used for distributions over discrete structures such as graphs. We demonstrate that, in certain cases, VI algorithms are equivalent to special cases of GFlowNets in the sense of equality of expected gradients of their learning objectives. We then point out the differences between the two families and show how these differences emerge experimentally. Notably, GFlowNets, which borrow ideas from reinforcement learning, are more amenable than VI to off-policy training without the cost of high gradient variance induced by importance sampling. We argue that this property of GFlowNets can provide advantages for capturing diversity in multimodal target distributions.
2023-01-31
ICLR.cc/2023/Conference (poster)
doi.org
openreview.net
GFlowOut: Dropout with Generative Flow Networks
Dianbo Liu
Moksh Jain
Bonaventure F. P. Dossou
Qianli Shen
Salem Lahlou
Anirudh Goyal
Nikolay Malkin
Chris C. Emezue
Dinghuai Zhang
Nadhir Hassen
Xu Ji
Kenji Kawaguchi
Yoshua Bengio
Bayesian Inference offers principled tools to tackle many critical problems with modern neural networks such as poor calibration and general… (see more)ization, and data inefficiency. However, scaling Bayesian inference to large architectures is challenging and requires restrictive approximations. Monte Carlo Dropout has been widely used as a relatively cheap way for approximate Inference and to estimate uncertainty with deep neural networks. Traditionally, the dropout mask is sampled independently from a fixed distribution. Recent works show that the dropout mask can be viewed as a latent variable, which can be inferred with variational inference. These methods face two important challenges: (a) the posterior distribution over masks can be highly multi-modal which can be difficult to approximate with standard variational inference and (b) it is not trivial to fully utilize sample-dependent information and correlation among dropout masks to improve posterior estimation. In this work, we propose GFlowOut to address these issues. GFlowOut leverages the recently proposed probabilistic framework of Generative Flow Networks (GFlowNets) to learn the posterior distribution over dropout masks. We empirically demonstrate that GFlowOut results in predictive distributions that generalize better to out-of-distribution data, and provide uncertainty estimates which lead to better performance in downstream tasks.
2022-12-31
ICML (published)
doi.org
proceedings.mlr.press
Test Sample Accuracy Scales with Training Sample Density in Neural Networks
Xu Ji
Razvan Pascanu
Devon Hjelm
Andrea Vedaldi
Balaji Lakshminarayanan
Yoshua Bengio
Intuitively, one would expect accuracy of a trained neural network's prediction on test samples to correlate with how densely the samples ar… (see more)e surrounded by seen training samples in representation space. We find that a bound on empirical training error smoothed across linear activation regions scales inversely with training sample density in representation space. Empirically, we verify this bound is a strong predictor of the inaccuracy of the network's prediction on test samples. For unseen test sets, including those with out-of-distribution samples, ranking test samples by their local region's error bound and discarding samples with the highest bounds raises prediction accuracy by up to 20% in absolute terms for image classification datasets, on average over thresholds.
2020-12-31
arXiv.org (preprint)
doi.org
openreview.net