Siva Reddy

aristides.milios@mila.quebec

Arkil Patel

PhD - McGill University

Principal supervisor :

arkil.patel@mila.quebec

PhD - McGill University

benno.krojer@mila.quebec

gaurav.kamath@mila.quebec

Gaurav Kamath

PhD - McGill University

Karolina Ewa Stańczak

Postdoctorate - McGill University

karolina.stanczak@mila.quebec

PhD - McGill University

laurestine.bradford@mila.quebec

Marius Mosbach

Postdoctorate - McGill University

marius.mosbach@mila.quebec

nicholas.meade@mila.quebec

Nicholas Meade

PhD - McGill University

Github

parishad.behnamghader@mila.quebec

Parishad BehnamGhader

Master's Research - McGill University

Collaborating researcher

spandana.gella@mila.quebec

vaibhav.adlakha@mila.quebec

Vaibhav Adlakha

PhD - McGill University

Xing Han Lu

PhD - McGill University

PhD - None

zdenek.kasner@mila.quebec

Zichao Li

PhD - McGill University

Co-supervisor :

Jackie Cheung

zichao.li@mila.quebec

Publications

Faithfulness Measurable Masked Language Models

Andreas Madsen

Sarath Chandar Anbil Parthipan

2024-05-01

ICML.cc/2024/Conference (spotlight)

openreview.net

Universal Adversarial Triggers Are Not Universal

Nicholas Meade

Arkil Patel

2024-04-24

ArXiv (preprint)

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Parishad BehnamGhader

Vaibhav Adlakha

Marius Mosbach

Nicolas Chapados

2024-04-09

ArXiv (preprint)

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Parishad BehnamGhader

Vaibhav Adlakha

Marius Mosbach

Nicolas Chapados

Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is… (see more) only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3 popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data. Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

2024-04-09

ArXiv (preprint)

Scope Ambiguities in Large Language Models

Gaurav Kamath

Sebastian Schuster

Sowmya Vajjala

2024-04-05

ArXiv (preprint)

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve… (see more) real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WebLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx.

2024-03-11

ICLR.cc/2024/Workshop/LLMAgents (poster)

openreview.net

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve… (see more) real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

2024-03-11

ICLR.cc/2024/Workshop/LLMAgents (poster)

openreview.net

A Compositional Typed Semantics for Universal Dependencies

Laurestine Bradford

Timothy John O'donnell

2024-03-02

ArXiv (preprint)

When does word order matter and when doesn't it?

Xuanda Chen

Timothy John O'donnell

Language models (LMs) may appear insensitive to word order changes in natural language understanding (NLU) tasks. In this paper, we propose … (see more)that linguistic redundancy can explain this phenomenon, whereby word order and other linguistic cues such as case markers provide overlapping and thus redundant information. Our hypothesis is that models exhibit insensitivity to word order when the order provides redundant information, and the degree of insensitivity varies across tasks. We quantify how informative word order is using mutual information (MI) between unscrambled and scrambled sentences. Our results show the effect that the less informative word order is, the more consistent the model's predictions are between unscrambled and scrambled sentences. We also find that the effect varies across tasks: for some tasks, like SST-2, LMs' prediction is almost always consistent with the original one even if the Pointwise-MI (PMI) changes, while for others, like RTE, the consistency is near random when the PMI gets lower, i.e., word order is really important.

2024-02-29

ArXiv (preprint)

The Leukemoid Reaction in Severe Alcoholic Hepatitis: A Case Report

Sachin Agrawal

Sunil Kumar

Sourya Acharya

2024-02-11

Cureus (published)