Publications

Cell Morphology-Guided Small Molecule Generation with GFlowNets
Stephen Zhewen Lu
Ziqing Lu
Ehsan Hajiramezanali
Tommaso Biancalani
Yoshua Bengio
Gabriele Scalia
Michał Koziarski
2024-06-16
ICML.cc/2024/Workshop/AI4Science (poster)
openreview.net
openreview.net
Expressivity of Neural Networks with Fixed Weights and Learned Biases
Ezekiel Williams
Avery Hee-Woon Ryoo
Thomas Jiralerspong
Alexandre Payeur
Matt Perich
Luca Mazzucato
Guillaume Lajoie
2024-06-16
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Gradient descent induces alignment between weights and the pre-activation tangents for deep non-linear networks
Daniel Beaglehole
Ioannis Mitliagkas
Atish Agarwala
Understanding the mechanisms through which neural networks extract statistics from input-label pairs is one of the most important unsolved p… (voir plus)roblems in supervised learning. Prior works have identified that the gram matrices of the weights in trained neural networks of general architectures are proportional to the average gradient outer product of the model, in a statement known as the Neural Feature Ansatz (NFA). However, the reason these quantities become correlated during training is poorly understood. In this work, we clarify the nature of this correlation and explain its emergence at early training times. We identify that the NFA is equivalent to alignment between the left singular structure of the weight matrices and the newly defined pre-activation tangent kernel. We identify a centering of the NFA that isolates this alignment and is robust to initialization scale. We show that, through this centering, the speed of NFA development can be predicted analytically in terms of simple statistics of the inputs and labels.
2024-06-16
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Gradient Dissent in Language Model Training and Saturation
Andrei Mircea
Ekaterina Lobacheva
Irina Rish
We seek to shed light on language model (LM) saturation from the perspective of learning dynamics. To this end, we define a decomposition o… (voir plus)f the cross-entropy gradient, which forms a shared low-dimensional basis for analyzing the training dynamics of models across scales. Intuitively, this decomposition consists of attractive and repulsive components that increase the logit of the correct class and decrease the logits of incorrect classes, respectively. Our analysis in this subspace reveals a phenomenon we term \textit{gradient dissent}, characterized by gradient components becoming systematically opposed such that loss cannot be improved along one component without being degraded along the other. Notably, we find that complete opposition, which we term \textit{total dissent}, reliably occurs in tandem with the saturation of smaller LMs. Based on these results, we hypothesize that gradient dissent can provide a useful foundation for better understanding and mitigating saturation.
2024-06-16
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Inpainting Galaxy Counts onto N-Body Simulations over Multiple Cosmologies and Astrophysics
Antoine Bourdin
Ronan Legin
Matthew Ho
Alexandre Adam
Yashar Hezaveh
Laurence Perreault-Levasseur
2024-06-16
ICML.cc/2024/Workshop/AI4Science (poster)
openreview.net
openreview.net
Linear Weight Interpolation Leads to Transient Performance Gains
Gaurav Iyer
Gintare Karolina Dziugaite
David Rolnick
2024-06-16
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Local lateral connectivity is sufficient for replicating cortex-like topographical organization in deep neural networks
Xinyu Qian
Amirozhan Dehghani
Asa Borzabadi Farahani
Pouya Bashivan
Across the primate cortex, neurons that perform similar functions tend to be spatially grouped together. In high-level visual cortex, this w… (voir plus)idely observed biological rule manifests itself as a modular organization of neuronal clusters, each tuned to a specific object category. The tendency toward short connections is one of the most widely accepted views of why such an organization exists in the brains of many animals. Yet, how such a feat is implemented at the neural level remains unclear. Here, using artificial deep neural networks as test beds, we demonstrate that a topographical organization similar to that in the primary, intermediate, and high-level human visual cortex emerges when units in these models are laterally connected and their weight parameters are tuned by top-down credit assignment. Importantly, the emergence of the modular organization in the absence of explicit topography-inducing learning rules and objectives questions their necessity and suggests that local lateral connectivity alone may be sufficient for the formation of the topographic organization across the cortex.
2024-06-16
ICML.cc/2024/Workshop/AI4Science (poster)
doi.org
openreview.net
A Survey on Fairness Without Demographics
Patrik Joslin Kenfack
Samira Ebrahimi Kahou
Éts Montréal
Ulrich Aivodji
The issue of bias in Machine Learning (ML) models is a significant challenge for the machine learning community. Real-world biases can be em… (voir plus)bedded in the data used to train models, and prior studies have shown that ML models can learn and even amplify these biases. This can result in unfair treatment of individuals based on their inherent characteristics or sensitive attributes such as gender, race, or age. Ensuring fairness is crucial with the increasing use of ML models in high-stakes scenarios and has gained significant attention from researchers in recent years. However, the challenge of ensuring fairness becomes much greater when the assumption of full access to sensitive attributes does not hold. The settings where the hypothesis does not hold include cases where (1) only limited or noisy demographic information is available or (2) demographic information is entirely unobserved due to privacy restrictions. This survey reviews recent research efforts to enforce fairness when sensitive attributes are missing. We propose a taxonomy of existing works and, more importantly, highlight current challenges and future research directions to stimulate research in ML fairness in the setting of missing sensitive attributes.
2024-06-16
TMLR (accepté)
openreview.net
openreview.net
The Butterfly Effect: Tiny Perturbations Cause Neural Network Training to Diverge
Gül Sena Altıntaş
Devin Kwok
David Rolnick
Neural network training begins with a chaotic phase in which the network is sensitive to small perturbations, such as those caused by stocha… (voir plus)stic gradient descent (SGD). This sensitivity can cause identically initialized networks to diverge both in parameter space and functional similarity. However, the exact degree to which networks are sensitive to perturbation, and the sensitivity of networks as they transition out of the chaotic phase, is unclear. To address this uncertainty, we apply a controlled perturbation at a single point in training time and measure its effect on otherwise identical training trajectories. We find that both the
2024-06-16
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Variable Star Light Curves in Koopman Space
Mario Pasquato
Gaia Carenini
Nicolas Mekhaël
Vittorio F. Braga
Piero Trevisan
Giuseppe Bono
Yashar Hezaveh
2024-06-16
ICML.cc/2024/Workshop/AI4Science (spotlight)
openreview.net
openreview.net
How Should We Extract Discrete Audio Tokens from Self-Supervised Models?
Pooneh Mousavi
Jarod Duret
Salah Zaiem
Luca Della Libera
Artem Ploujnikov
Cem (Yusuf) Subakan
Mirco Ravanelli
2024-06-15
ArXiv (prépublication)
doi.org
arxiv.org
Consistency-diversity-realism Pareto fronts of conditional image generative models
Pietro Astolfi
Marlene Careil
Melissa Hall
Oscar Mañas
Matthew Muckley
Jakob Verbeek
Adriana Romero Soriano
Michal Drozdzal
Building world models that accurately and comprehensively represent the real world is the utmost aspiration for conditional image generative… (voir plus) models as it would enable their use as world simulators. For these models to be successful world models, they should not only excel at image quality and prompt-image consistency but also ensure high representation diversity. However, current research in generative models mostly focuses on creative applications that are predominantly concerned with human preferences of image quality and aesthetics. We note that generative models have inference time mechanisms - or knobs - that allow the control of generation consistency, quality, and diversity. In this paper, we use state-of-the-art text-to-image and image-and-text-to-image models and their knobs to draw consistency-diversity-realism Pareto fronts that provide a holistic view on consistency-diversity-realism multi-objective. Our experiments suggest that realism and consistency can both be improved simultaneously; however there exists a clear tradeoff between realism/consistency and diversity. By looking at Pareto optimal points, we note that earlier models are better at representation diversity and worse in consistency/realism, and more recent models excel in consistency/realism while decreasing significantly the representation diversity. By computing Pareto fronts on a geodiverse dataset, we find that the first version of latent diffusion models tends to perform better than more recent models in all axes of evaluation, and there exist pronounced consistency-diversity-realism disparities between geographical regions. Overall, our analysis clearly shows that there is no best model and the choice of model should be determined by the downstream application. With this analysis, we invite the research community to consider Pareto fronts as an analytical tool to measure progress towards world models.
2024-06-14
ArXiv (prépublication)
doi.org
arxiv.org