Publications

Group Membership Bias

Ali Vardasbi

Maarten de Rijke

Mostafa Dehghani

2024-07-11

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)

doi.org

arxiv.org

Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping

Lucas Lehnert

Sainbayar Sukhbaatar

DiJia Su

Paul McVay

Qinqing Zheng

Michael Rabbat

Yuandong Tian

While Transformers have enabled tremendous progress in various application settings, such architectures still lag behind traditional symboli… (see more)c planners for solving complex decision making tasks. In this work, we demonstrate how to train Transformers to solve complex planning tasks. This is accomplished by training an encoder-decoder Transformer model to predict the _search dynamics_ of the

2024-07-10

colmweb.org/COLM/2024/Conference (accepted)

doi.org

openreview.net

Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs

Sylvain Kouemo Ngassom

Arghavan Moradi Dakhel

Florian Tambon

Foutse Khomh

2024-07-10

Proceedings of the 1st ACM International Conference on AI-Powered Software (published)

doi.org

arxiv.org

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

Qian Yang

Weixiang Yan

Aishwarya Agrawal

Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate… (see more) and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence generation, often suffer from overconfidence. Other methods use self-consistency comparison but are affected by confirmation biases. To alleviate these, we propose \textbf{De}compose and \textbf{C}ompare \textbf{C}onsistency (\texttt{DeCC}) for reliability measurement. By comparing the consistency between the direct answer generated using the VLM's internal reasoning process, and the indirect answers obtained by decomposing the question into sub-questions and reasoning over the sub-answers produced by the VLM, \texttt{DeCC} measures the reliability of VLM's direct answer. Experiments across six vision-language tasks with three VLMs show \texttt{DeCC}'s reliability estimation achieves better correlation with task accuracy compared to the existing methods.

2024-07-10

ArXiv (preprint)

doi.org

arxiv.org

Guiding Language Model Reasoning with Planning Tokens

Xinyi Wang

Lucas Caccia

Oleksiy Ostapenko

Xingdi Yuan

William Yang Wang

Alessandro Sordoni

Large language models (LLMs) have recently attracted considerable interest for their ability to perform complex reasoning tasks, such as cha… (see more)in-of-thought (CoT) reasoning. However, most of the existing approaches to enhance this ability rely heavily on data-driven methods, while neglecting the structural aspects of the model's reasoning capacity. To encourage a more structural generation of CoT steps, we propose a hierarchical generation scheme: we let the LM generate a planning token at the start of each reasoning step, intuitively serving as a high-level plan of the current step, and add their embeddings to the model parameters. Our approach requires a negligible increase in trainable parameters (0.001%) and can be applied through either full fine-tuning or a more parameter-efficient scheme. We demonstrate our method's effectiveness by applying it to three different LLMs, showing notable accuracy improvements across three math word problem datasets and one multihop QA dataset with respect to standard fine-tuning baselines.

2024-07-10

colmweb.org/COLM/2024/Conference (accepted)

openreview.net

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Parishad BehnamGhader

Vaibhav Adlakha

Marius Mosbach

Dzmitry Bahdanau

Nicolas Chapados

Siva Reddy

2024-07-10

colmweb.org/COLM/2024/Conference (accepted)

doi.org

openreview.net

Redesigning Information Markets in the Era of Language Models

Martin Weiss

Nasim Rahaman

Manuel Wüthrich

Yoshua Bengio

Li Erran Li

Bernhard Schölkopf

Chris Pal

2024-07-10

colmweb.org/COLM/2024/Conference (accepted)

openreview.net

Scattered Mixture-of-Experts Implementation

Shawn Tan

Yikang Shen

Rameswar Panda

Aaron Courville

ScatterMoE is an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon techniques in existing implementations, … (see more)and overcoming some of the current limitations to improve batched inference, training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input. We also fuse expert linear transforms and reordering operations with ParallelLinear, a module that can be used to extend the concept of SMoEs. We benchmark our implementation against Megablocks, and show that it enables a higher throughput and lower memory footprint. We also show how ParallelLinear enables extension of the Mixture-of-Experts concept by demonstrating with an implementation of Mixture-of-Attention.

2024-07-10

colmweb.org/COLM/2024/Conference (accepted)

doi.org

openreview.net

Seeking Interpretability and Explainability in Binary Activated Neural Networks

Benjamin Leblanc

Pascal Germain

2024-07-10

Communications in Computer and Information Science (published)

doi.org

arxiv.org

Should We Attend More or Less? Modulating Attention for Fairness