Publications

Learning Generative Population Models From Multiple Clinical Datasets Via Probabilistic Programming
João Loula
Katherine M. Collins
Ulrich Schaechtle
Joshua B. Tenenbaum
Adrian Weller
Feras Saad
Timothy J. O'Donnell
Vikash Mansinghka
Accurate, efficient generative models of clinical populations could accelerate clinical research and improve patient outcomes. For example, … (see more)such models could infer probable treatment outcomes for different subpopulations, generate high-fidelity synthetic data that can be shared across organizational boundaries, and discover new relationships among clinical variables. Using Bayesian structure learning, we show that it is possible to learn probabilistic program models of clinical populations by combining data from multiple, sparsely overlapping clinical datasets. Through experiments with multiple clinical trials and real-world evidence from census health surveys, we show that our model generates higher quality synthetic data than neural network baselines, supports more accurate inferences across datasets than traditional statistical methods, and can be queried more efficiently than both, opening up new avenues for accessible and efficient AI assistance in clinical research.
2024-06-16
ICML.cc/2024/Workshop/AccMLBio (poster)
openreview.net
openreview.net
Lost in Translation: The Algorithmic Gap Between LMs and the Brain
Tosato Tommaso
Tikeng Notsawo Pascal Junior
Helbling Saskia
Irina Rish
Guillaume Dumas
Language Models (LMs) have achieved impressive performance on various linguistic tasks, but their relationship to human language processing … (see more)in the brain remains unclear. This paper examines the gaps and overlaps between LMs and the brain at different levels of analysis, emphasizing the importance of looking beyond input-output behavior to examine and compare the internal processes of these systems. We discuss how insights from neuroscience, such as sparsity, modularity, internal states, and interactive learning, can inform the development of more biologically plausible language models. Furthermore, we explore the role of scaling laws in bridging the gap between LMs and human cognition, highlighting the need for efficiency constraints analogous to those in biological systems. By developing LMs that more closely mimic brain function, we aim to advance both artificial intelligence and our understanding of human cognition.
2024-06-16
ICML.cc/2024/Workshop/LLMs_and_Cognition (poster)
openreview.net
openreview.net
Neural Ratio Estimators Meet Distributional Shift and Mode Misspecification: A Cautionary Tale from Strong Gravitational Lensing
Andreas Filipp
Yashar Hezaveh
Laurence Perreault-Levasseur
In recent years, there has been increasing interest in the field of astrophysics in applying Neural Ratio Estimators (NREs) to large-scale i… (see more)nference problems where both amortization and marginalization over a large number of nuisance parameters are needed. Here, in order to assess the true potential of this method to produce unbiased inference on real data, we investigate the robustness of NREs to distribution shifts and model misspecification in the specific scientific application of the measurement of dark matter population-level parameters using strong gravitational lensing. We investigate the behaviour of a trained NRE for test data presenting distributional shifts inside the bounds of training, as well as out of distribution, both in the linear and non-linear parameters of this problem. While our results show that NREs perform when tested perfectly in distribution, we find that they exhibit significant biases and drawbacks when confronted with slight deviations from the examples seen in the training distribution. This indicates the necessity for caution when applying NREs to real astrophysical data, where underlying distributions are not perfectly known and models do not perfectly reconstruct the true underlying distributions.
2024-06-16
ICML.cc/2024/Workshop/SPIGM (poster)
openreview.net
openreview.net
On The Local Geometry of Deep Generative Manifolds
Ahmed Imtiaz Humayun
Ibtihel Amara
Candice Schumann
Golnoosh Farnadi
Negar Rostamzadeh
Mohammad Havaei
In this paper, we study theoretically inspired local geometric descriptors of the data manifolds approximated by pre-trained generative mode… (see more)ls. The descriptors – local scaling (ψ), local rank (ν), and local complexity (δ) — characterize the uncertainty, dimensionality, and smoothness on the learned manifold, using only the network weights and architecture. We investigate and emphasize their critical role in understanding generative models. Our analysis reveals that the local geometry is intricately linked to the quality and diversity of generated outputs. Additionally, we see that the geometric properties are distinct for out-of-distribution (OOD) inputs as well as for prompts memorized by Stable Diffusion, showing the possible application of our proposed descriptors for downstream detection and assessment of pre-trained generative models.
2024-06-16
ICML.cc/2024/Workshop/GRaM (published)
openreview.net
openreview.net
Expressivity of Neural Networks with Fixed Weights and Learned Biases
Ezekiel Williams
Avery Hee-Woon Ryoo
Thomas Jiralerspong
Alexandre Payeur
Matthew G Perich
Luca Mazzucato
Guillaume Lajoie
2024-06-15
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Gradient descent induces alignment between weights and the pre-activation tangents for deep non-linear networks
Daniel Beaglehole
Ioannis Mitliagkas
Atish Agarwala
Understanding the mechanisms through which neural networks extract statistics from input-label pairs is one of the most important unsolved p… (see more)roblems in supervised learning. Prior works have identified that the gram matrices of the weights in trained neural networks of general architectures are proportional to the average gradient outer product of the model, in a statement known as the Neural Feature Ansatz (NFA). However, the reason these quantities become correlated during training is poorly understood. In this work, we clarify the nature of this correlation and explain its emergence at early training times. We identify that the NFA is equivalent to alignment between the left singular structure of the weight matrices and the newly defined pre-activation tangent kernel. We identify a centering of the NFA that isolates this alignment and is robust to initialization scale. We show that, through this centering, the speed of NFA development can be predicted analytically in terms of simple statistics of the inputs and labels.
2024-06-15
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Gradient Dissent in Language Model Training and Saturation
Andrei Mircea
Ekaterina Lobacheva
Irina Rish
We seek to shed light on language model (LM) saturation from the perspective of learning dynamics. To this end, we define a decomposition o… (see more)f the cross-entropy gradient, which forms a shared low-dimensional basis for analyzing the training dynamics of models across scales. Intuitively, this decomposition consists of attractive and repulsive components that increase the logit of the correct class and decrease the logits of incorrect classes, respectively. Our analysis in this subspace reveals a phenomenon we term \textit{gradient dissent}, characterized by gradient components becoming systematically opposed such that loss cannot be improved along one component without being degraded along the other. Notably, we find that complete opposition, which we term \textit{total dissent}, reliably occurs in tandem with the saturation of smaller LMs. Based on these results, we hypothesize that gradient dissent can provide a useful foundation for better understanding and mitigating saturation.
2024-06-15
ICML.cc/2024/Workshop/HiLD (poster)
openreview.net
openreview.net
Inpainting Galaxy Counts onto N-Body Simulations over Multiple Cosmologies and Astrophysics
Antoine Bourdin
Ronan Legin
Matthew Ho
Alexandre Adam
Yashar Hezaveh
Laurence Perreault-Levasseur
2024-06-15
ICML.cc/2024/Workshop/AI4Science (poster)
openreview.net
openreview.net
Local lateral connectivity is sufficient for replicating cortex-like topographical organization in deep neural networks
Xinyu Qian
Amir Ozhan Dehghani
Asa Borzabadi Farahani
Pouya Bashivan
Across the primate cortex, neurons that perform similar functions tend to be spatially grouped together. This biological principle extends t… (see more)o many other species as well, reflecting a common way of organizing sensory processing across diverse forms of life. In the visual cortex, this biological principle manifests itself as a modular organization of neuronal clusters, each tuned to a specific visual property. The tendency toward short connections is widely believed to explain the existence of such an organization in the brains of many animals. However, the neural mechanisms underlying this phenomenon remain unclear. Here, we use artificial deep neural network models to demonstrate that a topographical organization akin to that in the primary, intermediate, and high-level human visual cortex emerges when units in these models are locally laterally connected and their weight parameters are tuned by top-down credit assignment. The emergence of modular organization without explicit topography-inducing learning rules or objective functions challenges their necessity and suggests that local lateral connectivity alone may suffice for the formation of topographic organization across the cortex. Furthermore, the incorporation of lateral connections in deep convolutional networks enhances their robustness to subtle alterations in visual inputs, such as those designed to deceive the model (i.e. adversarial examples), indicating an additional role for these connections in learning robust representations.
2024-06-15
ICML.cc/2024/Workshop/AI4Science (poster)
doi.org
openreview.net
Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
Oren Kraus
Kian Kenyon-Dean
Saber Saberian
Maryam Fallah
Peter McLean
Jess Leung
Vasudev Sharma
Ayla Khan
Jia Balakrishnan
Safiye Celik
Dominique Beaini
Maciej Sypetkowski
Chi Vicky Cheng
Kristen Morse
Maureen Makes
Ben Mabey
Berton Earnshaw
2024-06-15
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (published)
doi.org
arxiv.org
A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
Jack Urbanek
Florian Bordes
Pietro Astolfi
Mary Williamson
Vasu Sharma
Adriana Romero
2024-06-15
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (published)
doi.org
arxiv.org
A Survey on Fairness Without Demographics
Patrik Joslin Kenfack
Éts Montréal
S Ebrahimi Kahou
Ulrich Matchi Aïvodji
The issue of bias in Machine Learning (ML) models is a significant challenge for the machine learning community. Real-world biases can be em… (see more)bedded in the data used to train models, and prior studies have shown that ML models can learn and even amplify these biases. This can result in unfair treatment of individuals based on their inherent characteristics or sensitive attributes such as gender, race, or age. Ensuring fairness is crucial with the increasing use of ML models in high-stakes scenarios and has gained significant attention from researchers in recent years. However, the challenge of ensuring fairness becomes much greater when the assumption of full access to sensitive attributes does not hold. The settings where the hypothesis does not hold include cases where (1) only limited or noisy demographic information is available or (2) demographic information is entirely unobserved due to privacy restrictions. This survey reviews recent research efforts to enforce fairness when sensitive attributes are missing. We propose a taxonomy of existing works and, more importantly, highlight current challenges and future research directions to stimulate research in ML fairness in the setting of missing sensitive attributes.
2024-06-15
TMLR (accepted)
openreview.net
openreview.net