Publications

Categorical Generative Model Evaluation via Synthetic Distribution Coarsening
Florence Regol
Mark Coates
As we expect to see a rapid integration of generative models in our day to day lives, the development of rigorous methods of evaluation and … (see more)analysis for generative models has never been more pressing. Multiple works have highlighted the shortcomings of widely used metrics and exposed how they fail to behave as expected in some settings. So far, the response has been to use a variety of metrics that target different desirable and interpretable properties such as fidelity, diversity, and authenticity, to obtain a clearer picture of a generative model’s capabilities. These methods mainly focus on ordinal data and they all suffer from the same unavoidable issues stemming from estimating quantities of high-dimensional data from a limited number of samples. We propose to take an alternative approach and to return to the synthetic data setting where the ground truth is explicit and known. We focus on nominal categorical data and introduce an evaluation method that can scale to the high-dimensional settings often encountered in practice. Our method involves successively binning the large space to obtain smaller probability spaces and coarser distributions where meaningful statistical estimates can be obtained. This allows us to provide probabilistic guarantees and sample complexities and we illustrate how our method can be applied to distinguish between the capabilities of several state-of-the-art categorical models.
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
proceedings.mlr.press
Conditions on Preference Relations that Guarantee the Existence of Optimal Policies
Jonathan Colaco Carr
Prakash Panangaden
Doina Precup
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
arxiv.org
Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias
Yu Yang
Eric Gan
Gintare Karolina Dziugaite
Baharan Mirzasoleiman
Neural networks trained with (stochastic) gradient descent have an inductive bias towards learning simpler solutions. This makes them highly… (see more) prone to learning spurious correlations in the training data, that may not hold at test time. In this work, we provide the first theoretical analysis of the effect of simplicity bias on learning spurious correlations. Notably, we show that examples with spurious features are provably separable based on the model's output early in training. We further illustrate that if spurious features have a small enough noise-to-signal ratio, the network's output on the majority of examples is almost exclusively determined by the spurious features, leading to poor worst-group test accuracy. Finally, we propose SPARE, which identifies spurious correlations early in training and utilizes importance sampling to alleviate their effect. Empirically, we demonstrate that SPARE outperforms state-of-the-art methods by up to 21.1% in worst-group accuracy, while being up to 12x faster. We also show that SPARE is a highly effective but lightweight method to discover spurious correlations.
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
openreview.net
Introducing v0.5 of the AI Safety Benchmark from MLCommons
Bertie Vidgen
Adarsh Agrawal
Ahmed M. Ahmed
Victor Akinwande
Namir Al-nuaimi
Najla Alfaraj
Elie Alhajjar
Lora Aroyo
Trupti Bavalatti
Borhane Blili-Hamelin
K. Bollacker
Rishi Bomassani
Marisa Ferrara Boston
Sim'eon Campos
Kal Chakra
Canyu Chen
Cody Coleman
Zacharie Delpierre Coudert
Leon Strømberg Derczynski
Debojyoti Dutta … (see 77 more)
Ian Eisenberg
James R. Ezick
Heather Frase
Brian Fuller
Ram Gandikota
Agasthya Gangavarapu
Ananya Gangavarapu
James Gealy
Rajat Ghosh
James Goel
Usman Gohar
Sujata Goswami
Scott A. Hale
Wiebke Hutiri
Joseph Marvin Imperial
Surgan Jandial
Nicholas C. Judd
Felix Juefei-Xu
Foutse Khomh
Bhavya Kailkhura
Hannah Rose Kirk
Kevin Klyman
Chris Knotz
Michael Kuchnik
Shachi H. Kumar
Chris Lengerich
Bo Li
Zeyi Liao
Eileen Peters Long
Victor Lu
Yifan Mai
Priyanka Mary Mammen
Kelvin Manyeki
Sean McGregor
Virendra Mehta
Shafee Mohammed
Emanuel Moss
Lama Nachman
Dinesh Jinenhally Naganna
Amin Nikanjam
Besmira Nushi
Luis Oala
Iftach Orr
Alicia Parrish
Çigdem Patlak
William Pietri
Forough Poursabzi-Sangdeh
Eleonora Presani
Fabrizio Puletti
Paul Rottger
Saurav Sahay
Tim Santos
Nino Scherrer
Alice Schoenauer Sebag
Patrick Schramowski
Abolfazl Shahbazi
Vin Sharma
Xudong Shen
Vamsi Sistla
Leonard Tang
Davide Testuggine
Vithursan Thangarasa
Elizabeth A Watkins
Rebecca Weiss
Christoper A. Welty
Tyler Wilbers
Adina Williams
Carole-Jean Wu
Poonam Yadav
Xianjun Yang
Yi Zeng
Wenhui Zhang
Fedor Zhdanov
Jiacheng Zhu
Percy Liang
Peter Mattson
Joaquin Vanschoren
2024-04-18
ArXiv (preprint)
doi.org
arxiv.org
On learning history-based policies for controlling Markov decision processes
Gandharv Patil
Aditya Mahajan
Doina Precup
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
openreview.net
Length independent PAC-Bayes bounds for Simple RNNs
Volodimir Mitarchuk
Clara Lacroce
Rémi Eyraud
Rémi Emonet
Amaury Habrard
Guillaume Rabusseau
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
proceedings.mlr.press
Multi-phase black-hole feedback and a bright [CII] halo in a Lo-BAL quasar at $z\sim6.6$
Manuela Bischetti
Hyunseop Choi
Fabrizio Fiore
Chiara Feruglio
Stefano Carniani
Valentina D'Odorico
Eduardo Banados
Huanqing Chen
Roberto Decarli
Simona Gallerani
Julie Hlavacek-larrondo
Samuel Lai
K. Leighly
Chiara Mazzucchelli
Laurence Perreault-Levasseur
Roberta Tripodi
Fabian Walter
Feige Wang
Jinyi Yang
Maria Vittoria Zanchettin … (see 1 more)
Yongda Zhu
2024-04-18
ArXiv (preprint)
doi.org
arxiv.org
Multi-resolution Time-Series Transformer for Long-term Forecasting
Yitian Zhang
Liheng Ma
Soumyasundar Pal
Yingxue Zhang
Mark Coates
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
arxiv.org
Simulating weighted automata over sequences and trees with transformers
Michael Rizvi-Martel
Maude Lizaire
Clara Lacroce
Guillaume Rabusseau
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
arxiv.org
Tackling the XAI Disagreement Problem with Regional Explanations
gabriel laberge
Yann Batiste Pequignot
Mario Marchand
Foutse Khomh
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
proceedings.mlr.press
On the Privacy of Selection Mechanisms with Gaussian Noise
Jonathan Lebensold
Doina Precup
Borja Balle
Report Noisy Max and Above Threshold are two classical differentially private (DP) selection mechanisms. Their output is obtained by adding … (see more)noise to a sequence of low-sensitivity queries and reporting the identity of the query whose (noisy) answer satisfies a certain condition. Pure DP guarantees for these mechanisms are easy to obtain when Laplace noise is added to the queries. On the other hand, when instantiated using Gaussian noise, standard analyses only yield approximate DP guarantees despite the fact that the outputs of these mechanisms lie in a discrete space. In this work, we revisit the analysis of Report Noisy Max and Above Threshold with Gaussian noise and show that, under the additional assumption that the underlying queries are bounded, it is possible to provide pure ex-ante DP bounds for Report Noisy Max and pure ex-post DP bounds for Above Threshold. The resulting bounds are tight and depend on closed-form expressions that can be numerically evaluated using standard methods. Empirically we find these lead to tighter privacy accounting in the high privacy, low data regime. Further, we propose a simple privacy filter for composing pure ex-post DP guarantees, and use it to derive a fully adaptive Gaussian Sparse Vector Technique mechanism. Finally, we provide experiments on mobility and energy consumption datasets demonstrating that our Sparse Vector Technique is practically competitive with previous approaches and requires less hyper-parameter tuning.
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
arxiv.org
Weight-Sharing Regularization
Mehran Shakerinava
Motahareh Sohrabi
Siamak Ravanbakhsh
Simon Lacoste-Julien
2024-04-18
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)
doi.org
arxiv.org