Publications

Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
Md. Rabiul Awal
Le Zhang
Aishwarya Agrawal
In this paper, we explore effective prompting techniques to enhance zero- and few-shot Visual Question Answering (VQA) performance in contem… (voir plus)porary Vision-Language Models (VLMs). Central to our investigation is the role of question templates in guiding VLMs to generate accurate answers. We identify that specific templates significantly influence VQA outcomes, underscoring the need for strategic template selection. Another pivotal aspect of our study is augmenting VLMs with image captions, providing them with additional visual cues alongside direct image features in VQA tasks. Surprisingly, this augmentation significantly improves the VLMs' performance in many cases, even though VLMs"see"the image directly! We explore chain-of-thought (CoT) reasoning and find that while standard CoT reasoning causes drops in performance, advanced methods like self-consistency can help recover it. Furthermore, we find that text-only few-shot examples enhance VLMs' alignment with the task format, particularly benefiting models prone to verbose zero-shot answers. Lastly, to mitigate the challenges associated with evaluating free-form open-ended VQA responses using string-matching based VQA metrics, we introduce a straightforward LLM-guided pre-processing technique to adapt the model responses to the expected ground-truth answer distribution. In summary, our research sheds light on the intricacies of prompting strategies in VLMs for VQA, emphasizing the synergistic use of captions, templates, and pre-processing to enhance model efficacy.
2023-06-16
ArXiv (prépublication)
doi.org
arxiv.org
Preventing Dimensional Collapse in Contrastive Local Learning with Subsampling
Louis Fournier
Adeetya Patel
Michael Eickenberg
Edouard Oyallon
Eugene Belilovsky
2023-06-16
ICML.cc/2023/Workshop/LLW (publié)
openreview.net
openreview.net
Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
Le Zhang
Rabiul Awal
Aishwarya Agrawal
Vision-Language Models (VLMs), such as CLIP, exhibit strong image-text comprehension abilities, facilitating advances in several downstream … (voir plus)tasks such as zero-shot image classification, image-text retrieval, and text-to-image generation. However, the compositional reasoning abilities of existing VLMs remains subpar. The root of this limitation lies in the inadequate alignment between the images and captions in the pretraining datasets. Additionally, the current contrastive learning objective fails to focus on fine-grained grounding components like relations, actions, and attributes, resulting in"bag-of-words"representations. We introduce a simple and effective method to improve compositional reasoning in VLMs. Our method better leverages available datasets by refining and expanding the standard image-text contrastive learning framework. Our approach does not require specific annotations and does not incur extra parameters. When integrated with CLIP, our technique yields notable improvement over state-of-the-art baselines across five vision-language compositional benchmarks. We open-source our code at https://github.com/lezhang7/Enhance-FineGrained.
2023-06-15
ArXiv (prépublication)
arxiv.org
arxiv.org
GEANT4-DNA simulation of temperature-dependent and pH-dependent yields of chemical radiolytic species
Jingyi Bian
Juan Duran
Wook-Geun Shin
Jose Ramos-Méndez
Jack C Sankey
Lilian Childress
Jan Seuntjens
Shirin A. Enger
2023-06-15
Physics in Medicine & Biology (publié)
doi.org
A solution algorithm for chance-constrained problems with integer second-stage recourse decisions
Andrea Lodi
Enrico Malaguti
Michele Monaci
Giacomo Nannicini
Paolo
Paronuzzi
2023-06-15
Mathematical programming (publié)
doi.org
A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learning
Adel Nabli
Eugene Belilovsky
Edouard Oyallon
2023-06-14
ArXiv (prépublication)
doi.org
arxiv.org
Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision
Fernando Diaz
Across a variety of ranking tasks, researchers use reciprocal rank to measure the effectiveness for users interested in exactly one relevant… (voir plus) item. Despite its widespread use, evidence suggests that reciprocal rank is brittle when discriminating between systems. This brittleness, in turn, is compounded in modern evaluation settings where current, high-precision systems may be difficult to distinguish. We address the lack of sensitivity of reciprocal rank by introducing and connecting it to the concept of best-case retrieval, an evaluation method focusing on assessing the quality of a ranking for the most satisfied possible user across possible recall requirements. This perspective allows us to generalize reciprocal rank and define a new preference-based evaluation we call lexicographic precision or lexiprecision. By mathematical construction, we ensure that lexiprecision preserves differences detected by reciprocal rank, while empirically improving sensitivity and robustness across a broad set of retrieval and recommendation tasks.
2023-06-13
ArXiv (prépublication)
doi.org
arxiv.org
Who Controlled the Evidence? Question Answering for Disclosure Information Extraction
Hardy Hardy
Derek Ruths
Nicholas B King
Conflict of interest (COI) disclosure statements provide rich information to support trans-parency and reduce bias in research. We introduce… (voir plus) a novel task to identify relationships between sponsoring entities and the research studies they sponsor from the disclosure statement. This task is challenging due to the complexity of recognizing all potential relationship patterns and the hierarchical nature of identifying entities first and then extracting their relationships to the study. To overcome these challenges, in this paper, we also constructed a new annotated dataset and proposed a Question Answering-based method to recognize entities and extract relationships. Our method has demonstrated robustness in handling diverse relationship patterns, and it remains effective even when trained on a low-resource dataset.
2023-06-13
Proceedings of the Conference on Health, Inference, and Learning (publié)
proceedings.mlr.press
Benchmarking Neural Network Training Algorithms
George Edward Dahl
Frank Schneider
Zachary Nado
Naman Agarwal
Chandramouli Shama Sastry
Philipp Hennig
Sourabh Medapati
Runa Eschenhagen
Priya Kasimbeg
Daniel Suo
Juhan Bae
Justin M. Gilmer
A. L. Peirson
Bilal Muhammad Khan
Rohan Anil
Michael Rabbat
Shankar Krishnan
Daniel Snider
Ehsan Amid
Kongtao Chen … (voir 5 de plus)
Chris J. Maddison
R. Vasudev
Michal Badura
Ankush Garg
Peter Mattson
2023-06-12
ArXiv (prépublication)
doi.org
arxiv.org
Harms from Increasingly Agentic Algorithmic Systems
Alan Chan
Rebecca Salganik
Alva Markelius
Chris Pang
Nitarshan Rajkumar
Dmitrii Krasheninnikov
Lauro Langosco
Zhonghao He
Yawen Duan
Micah Carroll
Michelle Lin
Alex Mayhew
Katherine Collins
Maryam Molamohammadi
John Burden
Wanru Zhao
Shalaleh Rismani
Konstantinos Voudouris
Umang Bhatt
Adrian Weller … (voir 2 de plus)
David Scott Krueger
Tegan Maharaj
Research in Fairness, Accountability, Transparency, and Ethics (FATE)1 has established many sources and forms of algorithmic harm, in domain… (voir plus)s as diverse as health care, finance, policing, and recommendations. Much work remains to be done to mitigate the serious harms of these systems, particularly those disproportionately affecting marginalized communities. Despite these ongoing harms, new systems are being developed and deployed, typically without strong regulatory barriers, threatening the perpetuation of the same harms and the creation of novel ones. In response, the FATE community has emphasized the importance of anticipating harms, rather than just responding to them. Anticipation of harms is especially important given the rapid pace of developments in machine learning (ML). Our work focuses on the anticipation of harms from increasingly agentic systems. Rather than providing a definition of agency as a binary property, we identify 4 key characteristics which, particularly in combination, tend to increase the agency of a given algorithmic system: underspecification, directness of impact, goal-directedness, and long-term planning. We also discuss important harms which arise from increasing agency – notably, these include systemic and/or long-range impacts, often on marginalized or unconsidered stakeholders. We emphasize that recognizing agency of algorithmic systems does not absolve or shift the human responsibility for algorithmic harms. Rather, we use the term agency to highlight the increasingly evident fact that ML systems are not fully under human control. Our work explores increasingly agentic algorithmic systems in three parts. First, we explain the notion of an increase in agency for algorithmic systems in the context of diverse perspectives on agency across disciplines. Second, we argue for the need to anticipate harms from increasingly agentic systems. Third, we discuss important harms from increasingly agentic systems and ways forward for addressing them. We conclude by reflecting on implications of our work for anticipating algorithmic harms from emerging systems.
2023-06-12
2023 ACM Conference on Fairness, Accountability, and Transparency (publié)
doi.org
arxiv.org
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Tiago Salvador
Kilian FATRAS
Ioannis Mitliagkas
Adam M. Oberman
Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In the case of an extreme l… (voir plus)abel shift scenario between the source and target domains, where we have extra source classes not present in the target domain, the UDA problem becomes a harder problem called Partial Domain Adaptation (PDA). While different methods have been developed to solve the PDA problem, most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. These strategies violate the main assumption in PDA: only unlabeled target domain samples are available. In addition, there are also experimental inconsistencies between developed methods - different architectures, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods under different model selection strategies and a consistent evaluation protocol. We evaluate 6 state-of-the-art PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
2023-06-11
TMLR (accepté)
doi.org
openreview.net
Conditions for indexability of restless bandits and an algorithm to compute whittle index – CORRIGENDUM
Nima Akbarzadeh
Aditya Mahajan
2023-06-09
Advances in Applied Probability (publié)
doi.org