Siva Reddy

Biographie

Siva Reddy est professeur adjoint en informatique et linguistique à l’Université McGill. Ses travaux portent sur les algorithmes qui permettent aux ordinateurs de comprendre et de traiter les langues humaines. Il a fait ses études postdoctorales avec le Stanford NLP Group. Son expertise inclut la construction de symboliques linguistiques et induites et de modèles d’apprentissage profond pour le langage.

Étudiants actuels

Vaibhav Adlakha

Doctorat - McGill

Parishad BehnamGhader

Maîtrise recherche - McGill

Doctorat - McGill

Collaborateur·rice de recherche

Gaurav Kamath

Doctorat - McGill

Aditi Khandelwal

Doctorat - McGill

Superviseur⋅e principal⋅e :

Doctorat - McGill

Co-superviseur⋅e :

Timothy O'Donnell

Aravind Krishnan

Collaborateur·rice alumni - UNIVERSITÄT DES SAARLANDES

Doctorat - McGill

Zichao Li

Doctorat - McGill

Co-superviseur⋅e :

Jackie Cheung

Xing Han Lu

Doctorat - McGill

Stagiaire de recherche - McGill

Doctorat - McGill

Postdoctorat - McGill

Oh Oh

Collaborateur·rice de recherche

Arkil Patel

Doctorat - McGill

Superviseur⋅e principal⋅e :

Collaborateur·rice de recherche

Karolina Ewa Stańczak

Collaborateur·rice alumni - McGill

Comment expliquer l’IA et s’assurer que cette explication est vraie? Les modèles mesurables de fidélité vous indiquent comment y parvenir

Ada Tur

Stagiaire de recherche - McGill

Collaborateur·rice alumni - McGill

Billets de blogue

1 octobre 2024

par

Andrea Madsen

Siva Reddy

Sarath Chandar

Lire l'article

Publications

A Compositional Typed Semantics for Universal Dependencies

Laurestine Bradford

Timothy John O'donnell

2024-03-02

ArXiv (prépublication)

When does word order matter and when doesn't it?

Xuanda Chen

Timothy John O'donnell

Language models (LMs) may appear insensitive to word order changes in natural language understanding (NLU) tasks. In this paper, we propose … (voir plus)that linguistic redundancy can explain this phenomenon, whereby word order and other linguistic cues such as case markers provide overlapping and thus redundant information. Our hypothesis is that models exhibit insensitivity to word order when the order provides redundant information, and the degree of insensitivity varies across tasks. We quantify how informative word order is using mutual information (MI) between unscrambled and scrambled sentences. Our results show the effect that the less informative word order is, the more consistent the model's predictions are between unscrambled and scrambled sentences. We also find that the effect varies across tasks: for some tasks, like SST-2, LMs' prediction is almost always consistent with the original one even if the Pointwise-MI (PMI) changes, while for others, like RTE, the consistency is near random when the PMI gets lower, i.e., word order is really important.

2024-02-29

ArXiv (prépublication)

The Leukemoid Reaction in Severe Alcoholic Hepatitis: A Case Report

Sachin Agrawal

Sunil Kumar

Sourya Acharya

2024-02-11

Cureus (publié)

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve… (voir plus) real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

2024-02-08

ArXiv (prépublication)

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

2024-02-08

ArXiv (prépublication)

Data science opportunities of large language models for neuroscience and biomedicine

Andrew Thieme

Oleksiy Levkovskyy

Paul Wren

Thomas Ray

2024-02-01

Neuron (publié)

Data science opportunities of large language models for neuroscience and biomedicine

Andrew Thieme

Oleksiy Levkovskyy

Paul Wren

Thomas Ray

2024-02-01

Neuron (publié)

Data science opportunities of large language models for neuroscience and biomedicine

Andrew Thieme

Oleksiy Levkovskyy

Paul Wren

Thomas Ray

2024-02-01

Neuron (publié)

Data science opportunities of large language models for neuroscience and biomedicine

Andrew Thieme

Oleksiy Levkovskyy

Paul Wren

Thomas Ray

2024-02-01

Neuron (publié)

Benchmarking Vision Language Models for Cultural Understanding

Shravan Nayak

Kanishk Jain

Rabiul Awal

Sjoerd van Steenkiste

Lisa Anne Hendricks

Karolina Stanczak

Aishwarya Agrawal

Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of vi… (voir plus)sual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly lower performance for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.

2024-01-01

EMNLP (publié)

Evaluating In-Context Learning of Libraries for Code Generation

Arkil Patel

Dzmitry Bahdanau

Pradeep Dasigi

Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising ar… (voir plus)ea is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.

2024-01-01

North American Chapter of the Association for Computational Linguistics (publié)

Scope Ambiguities in Large Language Models

Gaurav Kamath

Sebastian Schuster

Sowmya Vajjala

2024-01-01

Trans. Assoc. Comput. Linguistics (publié)