Publications

INViTE: INterpret and Control Vision-Language Models with Text Explanations
Haozhe Chen
Junfeng Yang
Carl Vondrick
Chengzhi Mao
Columbia University
M. University
Large-scale pre-trained vision foundation models, such as CLIP, have become de facto backbones for various vision tasks. However, due to the… (see more)ir black-box nature, understanding the underlying rules behind these models’ predictions and controlling model behaviors have remained open challenges. We present INViTE: a framework for INterpreting Vision Transformer’s latent tokens with Text Explanations. Given a latent token, INViTE retains its semantic information to the final layer using transformer’s local operations and retrieves the closest text for explanation. INViTE enables understanding of model visual reasoning procedure without needing additional model training or data collection. Based on the obtained interpretations, INViTE allows for model editing that controls model reasoning behaviors and improves model robustness against biases and spurious correlations. Our code is available at https://github.com/tonychenxyz/vit-interpret.
2024-01-01
International Conference on Learning Representations (published)
openreview.net
openreview.net
ÌròyìnSpeech: A multi-purpose Yorùbá Speech Corpus
Tolúlope' Ògúnremí
Kọ́lá Túbọ̀sún
Aremu Anuoluwapo
Iroro Orife
David Ifeoluwa Adelani
2024-01-01
LREC/COLING (published)
doi.org
arxiv.org
Do Large Language Models Know How Much They Know?
Gabriele Prato
Jerry Huang
Prasanna Parthasarathi
Shagun Sodhani
Sarath Chandar
Large Language Models (LLMs) have emerged as highly capable systems and are increasingly being integrated into various uses. Nevertheless, t… (see more)he rapid advancement in their deployment trails a comprehensive understanding of their internal mechanisms, as well as a delineation of their capabilities and limitations. A desired characteristic of an intelligent system is its ability to recognize the scope of its own knowledge. To investigate whether LLMs embody this attribute, we develop a benchmark that challenges these models to enumerate all information they possess on specific topics. This benchmark assesses whether the models recall excessive, insufficient, or the precise amount of required information, thereby indicating their awareness of how much they know about the given topic. Our findings reveal that the emergence of this property varies across different architectures and manifests at diverse rates. However, with sufficient scaling, all tested models are ultimately capable of performing this task. The insights gained from this research advance our understanding of LLMs, shedding light on their operational capabilities and contributing to the ongoing exploration of their intricate dynamics.
2024-01-01
Conference on Empirical Methods in Natural Language Processing (published)
www.semanticscholar.org
Do Large Language Models Know How Much They Know?
Gabriele Prato
Jerry Huang
Prasanna Parthasarathi
Shagun Sodhani
Sarath Chandar
2024-01-01
EMNLP (published)
doi.org
Learnable Filters for Geometric Scattering Modules
Alexander Tong
Frederik Wenkel
Dhananjay Bhaskar
Kincaid MacDonald
Jackson Grady
Michael Perlmutter
Smita Krishnaswamy
Guy Wolf
2024-01-01
IEEE Transactions on Signal Processing (published)
doi.org
arxiv.org
Learning Conditional Policies for Crystal Design Using Offline Reinforcement Learning
Prashant Govindarajan
Santiago Miret
Jarrid Rector-Brooks
Mariano Phielipp
Janarthanan Rajendran
Sarath Chandar
Navigating through the exponentially large chemical space to search for desirable materials is an extremely challenging task in material dis… (see more)covery. Recent developments in generative and geometric deep learning have shown...
2024-01-01
Digital Discovery (published)
doi.org
openreview.net
Learning Lagrangian Multipliers for the Travelling Salesman Problem
Augustin Parjadis
Quentin Cappart
Bistra Dilkina
Aaron M. Ferber
Louis-Martin Rousseau
2024-01-01
CP (published)
doi.org
arxiv.org
Learning Precedences for Scheduling Problems with Graph Neural Networks
Hélène Verhaeghe
Quentin Cappart
Gilles Pesant
Claude-Guy Quimper
2024-01-01
CP (published)
doi.org
Learning to repeatedly solve routing problems
Mouad Morabit
Guy Desaulniers
Andrea Lodi
In the last years, there has been a great interest in machine‐learning‐based heuristics for solving NP‐hard combinatorial optimization… (see more) problems. The developed methods have shown potential on many optimization problems. In this paper, we present a learned heuristic for the reoptimization of a problem after a minor change in its data. We focus on the case of the capacited vehicle routing problem with static clients (i.e., same client locations) and changed demands. Given the edges of an original solution, the goal is to predict and fix the ones that have a high chance of remaining in an optimal solution after a change of client demands. This partial prediction of the solution reduces the complexity of the problem and speeds up its resolution, while yielding a good quality solution. The proposed approach resulted in solutions with an optimality gap ranging from 0% to 1.7% on different benchmark instances within a reasonable computing time.
2024-01-01
Networks (published)
doi.org
arxiv.org
Learning Tabu Search Algorithms: A Scheduling Application
Nazgol Niroumandrad
Nadia Lahrichi
Andrea Lodi
. Metaheuristics are widely recognized as efficient approaches for many combinatorial problems. Studies to improve the performance of metahe… (see more)uristics have increasingly relied on the use of various methods either combining different metaheuristics or methods originating outside of the metaheuristic field. This paper presents a learning algorithm to improve tabu search by reducing its search space and the evaluation effort. We study the performance of a learning tabu search algorithm using classification methods in an attempt to select moves through the search space more wisely. The experimental results demonstrate the benefit of using a learning mechanism under deterministic and stochastic conditions.
2024-01-01
Comput. Oper. Res. (published)
doi.org
Learning the Game: Decoding the Differences between Novice and Expert Players in a Citizen Science Game with Millions of Players
Eddie Cai
Roman Sarrazin-Gendron
Renata Mutalova
Parham Ghasemloo Gheidari
Alexander Butyaev
Gabriel Richard
Sébastien Caisse
Rob Knight
Mathieu Blanchette
Attila Szantner
Jérôme Waldispühl
2024-01-01
FDG (published)
doi.org
On learning Whittle index policy for restless bandits with scalable regret
Nima Akbarzadeh
Aditya Mahajan
Reinforcement learning is an attractive approach to learn good resource allocation and scheduling policies based on data when the system mod… (see more)el is unknown. However, the cumulative regret of most RL algorithms scales as ˜ O(S
2024-01-01
IEEE Transactions on Control of Network Systems (published)
doi.org
arxiv.org