Siva Reddy

vaibhav.adlakha@mila.quebec

Biography

Siva Reddy is an assistant professor at the School of Computer Science and in the Department of Linguistics at McGill University. He completed a postdoc with the Stanford NLP Group in September 2019.

Reddy’s research goal is to enable machines with natural language understanding abilities in order to facilitate applications like question answering and conversational systems. His expertise includes building symbolic (linguistic and induced) and deep learning models for language.

Current Students

Vaibhav Adlakha

PhD - McGill University

Kareem Ahmed

Postdoctorate - McGill University

Principal supervisor :

Guillaume Rabusseau

kareem.yousrii@gmail.com

Parishad BehnamGhader

PhD - McGill University

parishad.behnamghader@mila.quebec

PhD - McGill University

mehar.bhatia@mila.quebec

verna.dankers@mila.quebec

Verna Dankers

Postdoctorate - McGill University

Independent visiting researcher

desmond.elliott@mila.quebec

spandana.gella@mila.quebec

Jay Gala

Master's Research - McGill University

Co-supervisor :

Collaborating researcher

gaurav.kamath@mila.quebec

Gaurav Kamath

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

Aditi Khandelwal

PhD - McGill University

Principal supervisor :

Golnoosh Farnadi

aditi.khandelwal@mila.quebec

Austin Kraft

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

austin.kraft@mila.quebec

Benno Krojer

PhD - McGill University

benno.krojer@mila.quebec

Zichao Li

PhD - McGill University

Co-supervisor :

Jackie Cheung

zichao.li@mila.quebec

Fengyuan Liu

Master's Research - McGill University

Co-supervisor :

Dzmitry Bahdanau

fengyuan.liu@mila.quebec

sreenath.madathil@mila.quebec

Sreenath Madathil

Master's Research - McGill University

nicholas.meade@mila.quebec

Nicholas Meade

PhD - McGill University

Postdoctorate - McGill University

marius.mosbach@mila.quebec

Marzia Nouri

Master's Research - McGill University

marzia.nouri@mila.quebec

Arkil Patel

PhD - McGill University

Principal supervisor :

Dzmitry Bahdanau

arkil.patel@mila.quebec

erfan.raoofian@mila.quebec

Neel Rajani

Collaborating researcher - University of Edinburgh

neel.rajani@mila.quebec

Collaborating researcher - N/A

Ben Saine

Collaborating researcher - McGill University

ben.saine@mila.quebec

Kusha Sareen

PhD - McGill University

kusha.sareen@mila.quebec

dongchan.shin@mila.quebec

Dongchan Shin

Collaborating Alumni

tomasvergara.browne@mila.quebec

Ada Tur

Research Intern - McGill University

PhD - McGill University

Collaborating researcher - McGill University

donghao.zeng@mila.quebec

How Do We Explain AI and Ensure the Explanation Is True? Faithfulness Measurable Models Tell You How

Blog Posts

October 1, 2024

Andrea Madsen

Siva Reddy

Sarath Chandar

Read the article

Publications

Using In-Context Learning to Improve Dialogue Safety

Devamanyu Hazarika

Di Jin

Yang Liu

Dilek Hakkani-Tur

2023-10-06

EMNLP/2023/Conference (accepted)

ROSA: Random Orthogonal Subspace Adaptation

2023-06-19

ICML.cc/2023/Workshop/ES-FoMO (poster)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava

Abhinav Rastogi

Abhishek Rao

Abu Awal Md Shoeb

Abubakar Abid

Adam Fisch

Adam R. Brown

Adam Santoro

Aditya Gupta

Adrià Garriga-Alonso

Agnieszka Kluska

Aitor Lewkowycz

Akshat Agarwal

Alethea Power

Alex Ray

Alex Warstadt

Alexander W. Kocurek

Ali Safaya

Ali Tazarv

Alice Xiang … (see 432 more)

Alicia Parrish

Allen Nie

Aman Hussain

Amanda Askell

Amanda Dsouza

Ambrose Slone

Ameet Rahane

Anantharaman S. Iyer

Anders Johan Andreassen

Andrea Madotto

Andrea Santilli

Andreas Stuhlmüller

Andrew M. Dai

Andrew La

Andrew Lampinen

Andy Zou

Angela Jiang

Angelica Chen

Anh Vuong

Animesh Gupta

Anna Gottardi

Antonio Norelli

Anu Venkatesh

Arash Gholamidavoodi

Arfa Tabassum

Arul Menezes

Arun Kirubarajan

Asher Mullokandov

Ashish Sabharwal

Austin Herrick

Avia Efrat

Aykut Erdem

Ayla Karakaş

B. Ryan Roberts

Bao Sheng Loe

Barret Zoph

Bartłomiej Bojanowski

Batuhan Özyurt

Behnam Hedayatnia

Behnam Neyshabur

Benjamin Inden

Benno Stein

Berk Ekmekci

Bill Yuchen Lin

Blake Howald

Bryan Orinion

Cameron Diao

Cameron Dour

Catherine Stinson

Cedrick Argueta

Cesar Ferri

Chandan Singh

Charles Rathkopf

Chenlin Meng

Chitta Baral

Chiyu Wu

Chris Callison-Burch

Christopher Waites

Christian Voigt

Christopher D Manning

Christopher Potts

Cindy Ramirez

Clara E. Rivera

Clemencia Siro

Colin Raffel

Courtney Ashcraft

Cristina Garbacea

Damien Sileo

Dan Garrette

Dan Hendrycks

Dan Kilman

Dan Roth

C. Daniel Freeman

Daniel Khashabi

Daniel Levy

Daniel Moseguí González

Danielle Perszyk

Danny Hernandez

Danqi Chen

Daphne Ippolito

Dar Gilboa

David Dohan

David Drakard

David Jurgens

Debajyoti Datta

Deep Ganguli

Denis Emelin

Denis Kleyko

Deniz Yuret

Derek Chen

Derek Tam

Dieuwke Hupkes

Diganta Misra

Dilyar Buzan

Dimitri Coelho Mollo

Diyi Yang

Dong-Ho Lee

Dylan Schrader

Ekaterina Shutova

Ekin Dogus Cubuk

Elad Segal

Eleanor Hagerman

Elizabeth Barnes

Elizabeth Donoway

Ellie Pavlick

Emanuele Rodolà

Emma Lam

Eric Chu

Eric Tang

Erkut Erdem

Ernie Chang

Ethan A Chi

Ethan Dyer

Ethan Jerzak

Ethan Kim

Eunice Engefu Manyasi

Evgenii Zheltonozhskii

Fanyue Xia

Fatemeh Siar

Fernando Martínez-Plumed

Francesca Happé

François Chollet

Frieda Rong

Gaurav Mishra

Genta Indra Winata

Gerard de Melo

Germán Kruszewski

Giambattista Parascandolo

Giorgio Mariani

Gloria Xinyue Wang

Gonzalo Jaimovitch-Lopez

Gregor Betz

Guy Gur-Ari

Hana Galijasevic

Hannah Kim

Hannah Rashkin

Hannaneh Hajishirzi

Harsh Mehta

Hayden Bogar

Henry Shevlin

Henry Francis Anthony Shevlin

Hinrich Schuetze

Hiromu Yakura

Hongming Zhang

Hugh Mee Wong

Ian Ng

Isaac Noble

Jaap Jumelet

Jack Geissinger

Jackson Kernion

Jacob Hilton

Jaehoon Lee

Jaime Fernández Fisac

James B Simon

James Koppel

James Zheng

James Zou

Jan Kocon

Jana Thompson

Janelle Wingfield

Jared Kaplan

Jarema Radom

Jascha Sohl-Dickstein

Jason Phang

Jason Wei

Jason Yosinski

Jekaterina Novikova

Jelle Bosscher

Jennifer Marsh

Jeremy Kim

Jeroen Taal

Jesse Engel

Jesujoba Oluwadara Alabi

Jiacheng Xu

Jiaming Song

Jillian Tang

Joan Waweru

John Burden

John Miller

John U. Balis

Jonathan Batchelder

Jonathan Berant

Jörg Frohberg

Jos Rozen

Jose Hernandez-Orallo

Joseph Boudeman

Joseph Guerr

Joseph Jones

Joshua B. Tenenbaum

Joshua S. Rule

Joyce Chua

Kamil Kanclerz

Karen Livescu

Karl Krauth

Karthik Gopalakrishnan

Katerina Ignatyeva

Katja Markert

Kaustubh Dhole

Kevin Gimpel

Kevin Omondi

Kory Mathewson

Kristen Chiafullo

Ksenia Shkaruta

Kumar Shridhar

Kyle McDonell

Kyle Richardson

Laria Reynolds

Leo Gao

Ling Zhang

Liam Dugan

Lianhui Qin

Lidia Contreras-Ochando

Louis-Philippe Morency

Luca Moschella

Lucas Lam

Lucy Noble

Ludwig Schmidt

Luheng He

Luis Oliveros-Colón

Luke Metz

Lütfi Kerem Senel

Maarten Bosma

Maarten Sap

Maartje Ter Hoeve

Maheen Farooqi

Manaal Faruqui

Mantas Mazeika

Marco Baturan

Marco Marelli

Marco Maru

Maria Jose Ramirez-Quintana

Marie Tolkiehn

Mario Giulianelli

Martha Lewis

Martin Potthast

Matthew L Leavitt

Matthias Hagen

Mátyás Schubert

Medina Orduna Baitemirova

Melody Arnaud

Melvin McElrath

Michael Andrew Yee

Michael Cohen

Michael Gu

Michael Ivanitskiy

Michael Starritt

Michael Strube

Michał Swędrowski

Michele Bevilacqua

Michihiro Yasunaga

Mihir Kale

Mike Cain

Mimee Xu

Mirac Suzgun

Mitch Walker

Mo Tiwari

Mohit Bansal

Moin Aminnaseri

Mor Geva

Mozhdeh Gheini

Mukund Varma T

Nanyun Peng

Nathan Andrew Chi

Nayeon Lee

Neta Gur-Ari Krakover

Nicholas Cameron

Nicholas Roberts

Nick Doiron

Nicole Martinez

Nikita Nangia

Niklas Deckers

Niklas Muennighoff

Nitish Shirish Keskar

Niveditha S. Iyer

Noah Constant

Noah Fiedel

Nuan Wen

Oliver Zhang

Omar Agha

Omar Elbaghdadi

Omer Levy

Owain Evans

Pablo Antonio Moreno Casares

Parth Doshi

Pascale Fung

Paul Pu Liang

Paul Vicol

Pegah Alipoormolabashi

Peiyuan Liao

Percy Liang

Peter W Chang

Peter Eckersley

Phu Mon Htut

Pinyu Hwang

Pi-Bei Hwang

Piotr Miłkowski

Piyush Patil

Pouya Pezeshkpour

Priti Oli

Qiaozhu Mei

Qing Lyu

Qinlang Chen

Rabin Banjade

Rachel Etta Rudolph

Raefer Gabriel

Rahel Habacker

Ramon Risco

Raphaël Millière

Rhythm Garg

Richard Barnes

Rif A. Saurous

Riku Arakawa

Robbe Raymaekers

Robert Frank

Rohan Sikand

Roman Novak

Roman Sitelew

Ronan Le Bras

Rosanne Liu

Rowan Jacobs

Rui Zhang

Russ Salakhutdinov

Ryan Andrew Chi

Seungjae Ryan Lee

Ryan Stovall

Ryan Teehan

Rylan Yang

Sahib Singh

Saif Mohammad

Sajant Anand

Sam Dillavou

Sam Shleifer

Sam Wiseman

Samuel Gruetter

Samuel R. Bowman

Samuel Stern Schoenholz

Sanghyun Han

Sanjeev Kwatra

Sarah A. Rous

Sarik Ghazarian

Sayan Ghosh

Sean Casey

Sebastian Bischoff

Sebastian Gehrmann

Sebastian Schuster

Sepideh Sadeghi

Shadi Hamdan

Sharon Zhou

Shashank Srivastava

Sherry Shi

Shikhar Singh

Shima Asaadi

Shixiang Shane Gu

Shubh Pachchigar

Shubham Toshniwal

Shyam Upadhyay

Shyamolima Shammie Debnath

Siamak Shakeri

Simon Thormeyer

Simone Melzi

Sneha Priscilla Makini

Soo-Hwan Lee

Spencer Torene

Sriharsha Hatwar

Stanislas Dehaene

Stefan Divic

Stefano Ermon

Stella Biderman

Stephanie Lin

Stephen Prasad

Steven Piantadosi

Stuart Shieber

Summer Misherghi

Svetlana Kiritchenko

Swaroop Mishra

Tal Linzen

Tal Schuster

Tao Li

Tao Yu

Tariq Ali

Tatsunori Hashimoto

Te-Lin Wu

Théo Desbordes

Theodore Rothschild

Thomas Phan

Tianle Wang

Tiberius Nkinyili

Timo Schick

Timofei Kornev

Titus Tunduny

Tobias Gerstenberg

Trenton Chang

Trishala Neeraj

Tushar Khot

Tyler Shultz

Uri Shaham

Vedant Misra

Vera Demberg

Victoria Nyamai

Vikas Raunak

Vinay Venkatesh Ramasesh

vinay uday prabhu

Vishakh Padmakumar

Vivek Srikumar

William Fedus

William Saunders

William Zhang

Wout Vossen

Xiang Ren

Xiaoyu Tong

Xinran Zhao

Xinyi Wu

Xudong Shen

Yadollah Yaghoobzadeh

Yair Lakretz

Yangqiu Song

Yasaman Bahri

Yejin Choi

Yichi Yang

Sophie Hao

Yiding Hao

Yifu Chen

Yonatan Belinkov

Yufang Hou

Yuntao Bai

Zachary Seid

Zhuoye Zhao

Zijian Wang

Zijie J. Wang

Zirui Wang

Ziyi Wu

2023-05-10

TMLR (accepted)

The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents

Xing Han Lu

Harm de Vries

We introduce the StatCan Dialogue Dataset consisting of 19,379 conversation turns between agents working at Statistics Canada and online use… (see more)rs looking for published data tables. The conversations stem from genuine intents, are held in English or French, and lead to agents retrieving one of over 5000 complex data tables. Based on this dataset, we propose two tasks: (1) automatic retrieval of relevant tables based on a on-going conversation, and (2) automatic generation of appropriate agent responses at each turn. We investigate the difficulty of each task by establishing strong baselines. Our experiments on a temporal data split reveal that all models struggle to generalize to future conversations, as we observe a significant drop in performance across both tasks when we move from the validation to the test set. In addition, we find that response generation models struggle to decide when to return a table. Considering that the tasks pose significant challenges to existing models, we encourage the community to develop models for our task, which can be directly used to help knowledge workers find relevant tables for live chat users.

2023-04-30

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (published)

Combining Parameter-efficient Modules for Task-level Generalisation

Edoardo M. Ponti

Alessandro Sordoni

Yoshua Bengio

A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to ne… (see more)w tasks. In this work, we assume that each task is associated with a subset of latent skills from an (arbitrary size) inventory. In turn, each skill corresponds to a parameter-efficient (sparse / low-rank) model adapter. By jointly learning adapters and a routing function that allocates skills to each task, the full network is instantiated as the average of the parameters of active skills. We propose several inductive biases that encourage re-usage and composition of the skills, including variable-size skill allocation and a dual-speed learning rate. We evaluate our latent-skill model in two main settings: 1) multitask reinforcement learning for instruction following on 8 levels of the BabyAI platform; and 2) few-shot fine-tuning of language models on 160 NLP tasks of the CrossFit benchmark. We find that the modular design of our network enhances sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to a series of baselines. These include models where parameters are fully shared, task-specific, conditionally generated (HyperFormer), or sparse mixture-of-experts (TaskMoE).

2022-12-31

Conference of the European Chapter of the Association for Computational Linguistics (published)

In-Context Learning for Text Classification with Many Labels

Aristides Milios

Dzmitry Bahdanau

2022-12-31

GenBench @ Conference on Empirical Methods in Natural Language Processing (published)

FaithDial: A Faithful Benchmark for Information-Seeking Dialogue

Nouha Dziri

Ehsan Kamalloo

Sivan Milton

Osmar Zaiane

Mo Yu

Edoardo M. Ponti

The goal of information-seeking dialogue is to respond to seeker queries with natural language utterances that are grounded on knowledge sou… (see more)rces. However, dialogue systems often produce unsupported utterances, a phenomenon known as hallucination. To mitigate this behavior, we adopt a data-centric solution and create FaithDial, a new benchmark for hallucination-free dialogues, by editing hallucinated responses in the Wizard of Wikipedia (WoW) benchmark. We observe that FaithDial is more faithful than WoW while also maintaining engaging conversations. We show that FaithDial can serve as training signal for: i) a hallucination critic, which discriminates whether an utterance is faithful or not, and boosts the performance by 12.8 F1 score on the BEGIN benchmark compared to existing datasets for dialogue coherence; ii) high-quality dialogue generation. We benchmark a series of state-of-the-art models and propose an auxiliary contrastive objective that achieves the highest level of faithfulness and abstractiveness based on several automated metrics. Further, we find that the benefits of FaithDial generalize to zero-shot transfer on other datasets, such as CMU-Dog and TopicalChat. Finally, human evaluation reveals that responses generated by models trained on FaithDial are perceived as more interpretable, cooperative, and engaging.

2022-12-22

Transactions of the Association for Computational Linguistics (published)

Post-hoc Interpretability for Neural NLP: A Survey

Andreas Madsen

A. Chandar

2022-12-22

ACM Computing Surveys (published)

Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining

To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for makin… (see more)g a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using relative area-between-curves (RACU), which allows for easy comparison across papers, models, and tasks. We evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.

2022-11-30

Findings of the Association for Computational Linguistics: EMNLP 2022 (published)

The Curious Case of Absolute Position Embeddings

Koustuv Sinha

Amirhossein Kazemnejad

Joelle Pineau

Dieuwke Hupkes

Adina Williams

Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is repr… (see more)esented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.

2022-11-30

Findings of the Association for Computational Linguistics: EMNLP 2022 (published)

Does Entity Abstraction Help Generative Transformers Reason?

Nicolas Gontier

Christopher Pal

We study the utility of incorporating entity type abstractions into pre-trained Transformers and test these methods on four NLP tasks requir… (see more)ing different forms of logical reasoning: (1) compositional language understanding with text-based relational reasoning (CLUTRR), (2) abductive reasoning (ProofWriter), (3) multi-hop question answering (HotpotQA), and (4) conversational question answering (CoQA). We propose and empirically explore three ways to add such abstraction: (i) as additional input embeddings, (ii) as a separate sequence to encode, and (iii) as an auxiliary prediction task for the model. Overall, our analysis demonstrates that models with abstract entity knowledge performs better than without it. The best abstraction aware models achieved an overall accuracy of 88.8% and 91.8% compared to the baseline model achieving 62.9% and 89.8% on CLUTRR and ProofWriter respectively. However, for HotpotQA and CoQA, we find that F1 scores improve by only 0.5% on average. Our results suggest that the benefit of explicit abstraction is significant in formally defined logical reasoning settings requiring many reasoning hops, but point to the notion that it is less beneficial for NLP tasks having less formal logical structure.

2022-11-19

TMLR (accepted)

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

Emanuele Bugliarello

Fangyu Liu

Jonas Pfeiffer

Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of… (see more) a multilingual benchmark, however, vision-and-language research has mostly focused on English language tasks. To fill this gap, we introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together - by both aggregating pre-existing datasets and creating new ones - visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups. Based on the evaluation of the available state-of-the-art models, we find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks. Moreover, downstream performance is partially explained by the amount of available unlabelled textual data for pretraining, and only weakly by the typological distance of target-source languages. We hope to encourage future research efforts in this area by releasing the benchmark to the community.

2022-07-18

ICML (Accept for Short Presentation)