Siva Reddy

Biography

Siva Reddy is an assistant professor at the School of Computer Science and in the Department of Linguistics at McGill University. He completed a postdoc with the Stanford NLP Group in September 2019.

Reddy’s research goal is to enable machines with natural language understanding abilities in order to facilitate applications like question answering and conversational systems. His expertise includes building symbolic (linguistic and induced) and deep learning models for language.

Current Students

Vaibhav Adlakha

PhD - McGill University

Parishad BehnamGhader

Master's Research - McGill University

PhD - McGill University

Matteo Boglioni

Collaborating researcher - McGill University

Jerry Chen

Research Intern - McGill University

Postdoctorate - McGill University

Independent visiting researcher

Co-supervisor :

Yoshua Bengio

Jay Gala

Master's Research - McGill University

Co-supervisor :

Collaborating researcher

Gaurav Kamath

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

Aditi Khandelwal

PhD - McGill University

Principal supervisor :

Golnoosh Farnadi

Austin Kraft

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

Benno Krojer

PhD - McGill University

Zichao Li

PhD - McGill University

Co-supervisor :

Jackie Cheung

Fengyuan Liu

Master's Research - McGill University

Co-supervisor :

Xing Han Lu

PhD - McGill University

Master's Research - McGill University

Nicholas Meade

PhD - McGill University

Postdoctorate - McGill University

Marzia Nouri

Master's Research - McGill University

Arkil Patel

PhD - McGill University

Principal supervisor :

Collaborating researcher - N/A

Ben Saine

Collaborating researcher - McGill University

Collaborating Alumni

Ada Tur

Research Intern - McGill University

PhD - McGill University

Collaborating researcher - McGill University

How Do We Explain AI and Ensure the Explanation Is True? Faithfulness Measurable Models Tell You How

Blog Posts

October 1, 2024

Andrea Madsen

Siva Reddy

Sarath Chandar

Read the article

Publications

Scope Ambiguities in Large Language Models

Gaurav Kamath

Sebastian Schuster

Sowmya Vajjala

Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguiti… (see more)es. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models -- GPT-2, GPT-3/3.5, Llama 2 and GPT-4 -- treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).

2023-12-31

Trans. Assoc. Comput. Linguistics (published)

arxiv.org

Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model

Parishad BehnamGhader

Santiago Miret

Augmenting pretrained language models with retrievers to select the supporting documents has shown promise in effectively solving common NLP… (see more) problems, including language modeling and question answering, in an interpretable way. In this paper, we first study the strengths and weaknesses of different retriever-augmented language models (REALM,

2023-11-30

Findings of the Association for Computational Linguistics: EMNLP 2023 (published)

Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness

Zichao Li

Ines Arous

Jackie C.K. Cheung

The potential of using a large language model (LLM) as a knowledge base (KB) has sparked significant interest. To maintain the knowledge acq… (see more)uired by LLMs, we need to ensure that the editing of learned facts respects internal logical constraints, which are known as dependency of knowledge. Existing work on editing LLMs has partially addressed the issue of dependency, when the editing of a fact should apply to its lexical variations without disrupting irrelevant ones. However, they neglect the dependency between a fact and its logical implications. We propose an evaluation protocol with an accompanying question-answering dataset, StandUp, that provides a comprehensive assessment of the editing process considering the above notions of dependency. Our protocol involves setting up a controlled environment in which we edit facts and monitor their impact on LLMs, along with their implications based on If-Then rules. Extensive experiments on StandUp show that existing knowledge editing methods are sensitive to the surface form of knowledge, and that they have limited performance in inferring the implications of edited facts.

2023-10-06

Conference on Empirical Methods in Natural Language Processing (accepted)

Using In-Context Learning to Improve Dialogue Safety

Devamanyu Hazarika

Di Jin

Yang Liu

Dilek Hakkani-Tur

2023-10-06

EMNLP/2023/Conference (accepted)

Are Diffusion Models Vision-And-Language Reasoners?

Christopher Pal

Text-conditioned image generation models have recently shown immense qualitative success using denoising diffusion processes. However, unlik… (see more)e discriminative vision-and-language models, it is a non-trivial task to subject these diffusion-based generative models to automatic fine-grained quantitative evaluation of high-level phenomena such as compositionality. Towards this goal, we perform two innovations. First, we transform diffusion-based models (in our case, Stable Diffusion) for any image-text matching (ITM) task using a novel method called DiffusionITM. Second, we introduce the Generative-Discriminative Evaluation Benchmark (GDBench) benchmark with 7 complex vision-and-language tasks, bias evaluation and detailed analysis. We find that Stable Diffusion + DiffusionITM is competitive on many tasks and outperforms CLIP on compositional tasks like like CLEVR and Winoground. We further boost its compositional performance with a transfer setup by fine-tuning on MS-COCO while retaining generative capabilities. We also measure the stereotypical bias in diffusion models, and find that Stable Diffusion 2.1 is, for the most part, less biased than Stable Diffusion 1.5. Overall, our results point in an exciting direction bringing discriminative and generative model evaluation closer. We will release code and benchmark setup soon.

2023-09-20

NeurIPS.cc/2023/Conference (poster)

The Impact of Positional Encoding on Length Generalization in Transformers

Amirhossein Kazemnejad

Inkit Padhi

Karthikeyan Natesan Ramamurthy

Payel Das

Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the developmen… (see more)t of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.

2023-09-20

NeurIPS.cc/2023/Conference (poster)

In-Context Learning for Text Classification with Many Labels

Aristides Milios

2023-09-18

ArXiv (preprint)

arxiv.org

ROSA: Random Orthogonal Subspace Adaptation

2023-06-19

ICML.cc/2023/Workshop/ES-FoMO (poster)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava

Abhinav Rastogi

Abhishek Rao

Abu Awal Md Shoeb

Abubakar Abid

Adam Fisch

Adam R. Brown

Adam Santoro

Aditya Gupta

Adrià Garriga-Alonso

Agnieszka Kluska

Aitor Lewkowycz

Akshat Agarwal

Alethea Power

Alex Ray

Alex Warstadt

Alexander W. Kocurek

Ali Safaya

Ali Tazarv

Alice Xiang … (see 432 more)

Alicia Parrish

Allen Nie

Aman Hussain

Amanda Askell

Amanda Dsouza

Ambrose Slone

Ameet Rahane

Anantharaman S. Iyer

Anders Johan Andreassen

Andrea Madotto

Andrea Santilli

Andreas Stuhlmüller

Andrew M. Dai

Andrew La

Andrew Lampinen

Andy Zou

Angela Jiang

Angelica Chen

Anh Vuong

Animesh Gupta

Anna Gottardi

Antonio Norelli

Anu Venkatesh

Arash Gholamidavoodi

Arfa Tabassum

Arul Menezes

Arun Kirubarajan

Asher Mullokandov

Ashish Sabharwal

Austin Herrick

Avia Efrat

Aykut Erdem

Ayla Karakaş

B. Ryan Roberts

Bao Sheng Loe

Barret Zoph

Bartłomiej Bojanowski

Batuhan Özyurt

Behnam Hedayatnia

Behnam Neyshabur

Benjamin Inden

Benno Stein

Berk Ekmekci

Bill Yuchen Lin

Blake Howald

Bryan Orinion

Cameron Diao

Cameron Dour

Catherine Stinson

Cedrick Argueta

Cesar Ferri

Chandan Singh

Charles Rathkopf

Chenlin Meng

Chitta Baral

Chiyu Wu

Chris Callison-Burch

Christopher Waites

Christian Voigt

Christopher D Manning

Christopher Potts

Cindy Ramirez

Clara E. Rivera

Clemencia Siro

Colin Raffel

Courtney Ashcraft

Cristina Garbacea

Damien Sileo

Dan Garrette

Dan Hendrycks

Dan Kilman

Dan Roth

C. Daniel Freeman

Daniel Khashabi

Daniel Levy

Daniel Moseguí González

Danielle Perszyk

Danny Hernandez

Danqi Chen

Daphne Ippolito

Dar Gilboa

David Dohan

David Drakard

David Jurgens

Debajyoti Datta

Deep Ganguli

Denis Emelin

Denis Kleyko

Deniz Yuret

Derek Chen

Derek Tam

Dieuwke Hupkes

Diganta Misra

Dilyar Buzan

Dimitri Coelho Mollo

Diyi Yang

Dong-Ho Lee

Dylan Schrader

Ekaterina Shutova

Ekin Dogus Cubuk

Elad Segal

Eleanor Hagerman

Elizabeth Barnes

Elizabeth Donoway

Ellie Pavlick

Emanuele Rodolà

Emma Lam

Eric Chu

Eric Tang

Erkut Erdem

Ernie Chang

Ethan A Chi

Ethan Dyer

Ethan Jerzak

Ethan Kim

Eunice Engefu Manyasi

Evgenii Zheltonozhskii

Fanyue Xia

Fatemeh Siar

Fernando Martínez-Plumed

Francesca Happé

François Chollet

Frieda Rong

Gaurav Mishra

Genta Indra Winata

Gerard de Melo

Germán Kruszewski

Giambattista Parascandolo

Giorgio Mariani

Gloria Xinyue Wang

Gonzalo Jaimovitch-Lopez

Gregor Betz

Guy Gur-Ari

Hana Galijasevic

Hannah Kim

Hannah Rashkin

Hannaneh Hajishirzi

Harsh Mehta

Hayden Bogar

Henry Shevlin

Henry Francis Anthony Shevlin

Hinrich Schuetze

Hiromu Yakura

Hongming Zhang

Hugh Mee Wong

Ian Ng

Isaac Noble

Jaap Jumelet

Jack Geissinger

Jackson Kernion

Jacob Hilton

Jaehoon Lee

Jaime Fernández Fisac

James B Simon

James Koppel

James Zheng

James Zou

Jan Kocon

Jana Thompson

Janelle Wingfield

Jared Kaplan

Jarema Radom

Jascha Sohl-Dickstein

Jason Phang

Jason Wei

Jason Yosinski

Jekaterina Novikova

Jelle Bosscher

Jennifer Marsh

Jeremy Kim

Jeroen Taal

Jesse Engel

Jesujoba Oluwadara Alabi

Jiacheng Xu

Jiaming Song

Jillian Tang

Joan Waweru

John Burden

John Miller

John U. Balis

Jonathan Batchelder

Jonathan Berant

Jörg Frohberg

Jos Rozen

Jose Hernandez-Orallo

Joseph Boudeman

Joseph Guerr

Joseph Jones

Joshua B. Tenenbaum

Joshua S. Rule

Joyce Chua

Kamil Kanclerz

Karen Livescu

Karl Krauth

Karthik Gopalakrishnan

Katerina Ignatyeva

Katja Markert

Kaustubh Dhole

Kevin Gimpel

Kevin Omondi

Kory Mathewson

Kristen Chiafullo

Ksenia Shkaruta

Kumar Shridhar

Kyle McDonell

Kyle Richardson

Laria Reynolds

Leo Gao

Ling Zhang

Liam Dugan

Lianhui Qin

Lidia Contreras-Ochando

Louis-Philippe Morency

Luca Moschella

Lucas Lam

Lucy Noble

Ludwig Schmidt

Luheng He

Luis Oliveros-Colón

Luke Metz

Lütfi Kerem Senel

Maarten Bosma

Maarten Sap

Maartje Ter Hoeve

Maheen Farooqi

Manaal Faruqui

Mantas Mazeika

Marco Baturan

Marco Marelli

Marco Maru

Maria Jose Ramirez-Quintana

Marie Tolkiehn

Mario Giulianelli

Martha Lewis

Martin Potthast

Matthew L Leavitt

Matthias Hagen

Mátyás Schubert

Medina Orduna Baitemirova

Melody Arnaud

Melvin McElrath

Michael Andrew Yee

Michael Cohen

Michael Gu

Michael Ivanitskiy

Michael Starritt

Michael Strube

Michał Swędrowski

Michele Bevilacqua

Michihiro Yasunaga

Mihir Kale

Mike Cain

Mimee Xu

Mirac Suzgun

Mitch Walker

Mo Tiwari

Mohit Bansal

Moin Aminnaseri

Mor Geva

Mozhdeh Gheini

Mukund Varma T

Nanyun Peng

Nathan Andrew Chi

Nayeon Lee

Neta Gur-Ari Krakover

Nicholas Cameron

Nicholas Roberts

Nick Doiron

Nicole Martinez

Nikita Nangia

Niklas Deckers

Niklas Muennighoff

Nitish Shirish Keskar

Niveditha S. Iyer

Noah Constant

Noah Fiedel

Nuan Wen

Oliver Zhang

Omar Agha

Omar Elbaghdadi

Omer Levy

Owain Evans

Pablo Antonio Moreno Casares

Parth Doshi

Pascale Fung

Paul Pu Liang

Paul Vicol

Pegah Alipoormolabashi

Peiyuan Liao

Percy Liang

Peter W Chang

Peter Eckersley

Phu Mon Htut

Pinyu Hwang

Pi-Bei Hwang

Piotr Miłkowski

Piyush Patil

Pouya Pezeshkpour

Priti Oli

Qiaozhu Mei

Qing Lyu

Qinlang Chen

Rabin Banjade

Rachel Etta Rudolph

Raefer Gabriel

Rahel Habacker

Ramon Risco

Raphaël Millière

Rhythm Garg

Richard Barnes

Rif A. Saurous

Riku Arakawa

Robbe Raymaekers

Robert Frank

Rohan Sikand

Roman Novak

Roman Sitelew

Ronan Le Bras

Rosanne Liu

Rowan Jacobs

Rui Zhang

Russ Salakhutdinov

Ryan Andrew Chi

Seungjae Ryan Lee

Ryan Stovall

Ryan Teehan

Rylan Yang

Sahib Singh

Saif Mohammad

Sajant Anand

Sam Dillavou

Sam Shleifer

Sam Wiseman

Samuel Gruetter

Samuel R. Bowman

Samuel Stern Schoenholz

Sanghyun Han

Sanjeev Kwatra

Sarah A. Rous

Sarik Ghazarian

Sayan Ghosh

Sean Casey

Sebastian Bischoff

Sebastian Gehrmann

Sebastian Schuster

Sepideh Sadeghi

Shadi Hamdan

Sharon Zhou

Shashank Srivastava

Sherry Shi

Shikhar Singh

Shima Asaadi

Shixiang Shane Gu

Shubh Pachchigar

Shubham Toshniwal

Shyam Upadhyay

Shyamolima Shammie Debnath

Siamak Shakeri

Simon Thormeyer

Simone Melzi

Sneha Priscilla Makini

Soo-Hwan Lee

Spencer Torene

Sriharsha Hatwar

Stanislas Dehaene

Stefan Divic

Stefano Ermon

Stella Biderman

Stephanie Lin

Stephen Prasad

Steven Piantadosi

Stuart Shieber

Summer Misherghi

Svetlana Kiritchenko

Swaroop Mishra

Tal Linzen

Tal Schuster

Tao Li

Tao Yu

Tariq Ali

Tatsunori Hashimoto

Te-Lin Wu

Théo Desbordes

Theodore Rothschild

Thomas Phan

Tianle Wang

Tiberius Nkinyili

Timo Schick

Timofei Kornev

Titus Tunduny

Tobias Gerstenberg

Trenton Chang

Trishala Neeraj

Tushar Khot

Tyler Shultz

Uri Shaham

Vedant Misra

Vera Demberg

Victoria Nyamai

Vikas Raunak

Vinay Venkatesh Ramasesh

vinay uday prabhu

Vishakh Padmakumar

Vivek Srikumar

William Fedus

William Saunders

William Zhang

Wout Vossen

Xiang Ren

Xiaoyu Tong

Xinran Zhao

Xinyi Wu

Xudong Shen

Yadollah Yaghoobzadeh

Yair Lakretz

Yangqiu Song

Yasaman Bahri

Yejin Choi

Yichi Yang

Sophie Hao

Yiding Hao

Yifu Chen

Yonatan Belinkov

Yufang Hou

Yuntao Bai

Zachary Seid

Zhuoye Zhao

Zijian Wang

Zijie J. Wang

Zirui Wang

Ziyi Wu

2023-05-10

TMLR (accepted)

Combining Parameter-efficient Modules for Task-level Generalisation

Edoardo M. Ponti

Alessandro Sordoni

Yoshua Bengio

A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to ne… (see more)w tasks. In this work, we assume that each task is associated with a subset of latent skills from an (arbitrary size) inventory. In turn, each skill corresponds to a parameter-efficient (sparse / low-rank) model adapter. By jointly learning adapters and a routing function that allocates skills to each task, the full network is instantiated as the average of the parameters of active skills. We propose several inductive biases that encourage re-usage and composition of the skills, including variable-size skill allocation and a dual-speed learning rate. We evaluate our latent-skill model in two main settings: 1) multitask reinforcement learning for instruction following on 8 levels of the BabyAI platform; and 2) few-shot fine-tuning of language models on 160 NLP tasks of the CrossFit benchmark. We find that the modular design of our network enhances sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to a series of baselines. These include models where parameters are fully shared, task-specific, conditionally generated (HyperFormer), or sparse mixture-of-experts (TaskMoE).

2022-12-31

Conference of the European Chapter of the Association for Computational Linguistics (published)

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Arkil Patel

Satwik Bhattamishra

2022-12-31

EMNLP (published)

StarCoder: may the source be with you!

Raymond Li

Loubna Ben allal

Yangtian Zi

Niklas Muennighoff

Denis Kocetkov

Chenghao Mou

Marc Marone

Christopher Akiki

Jia LI

Jenny Chim

Qian Liu

Evgenii Zheltonozhskii

Terry Yue Zhuo

Thomas Wang

Olivier Dehaene

Mishig Davaadorj

Joel Lamy-Poirier

Joao Monteiro

Oleh Shliazhko

Nicolas Gontier … (see 47 more)

Nicholas Meade

Armel Zebaze

Ming-Ho Yee

Logesh Kumar Umapathi

Jian Zhu

Ben Lipkin

Muhtasham Oblokulov

Zhiruo Wang

Rudra Murthy

Jason T Stillerman

Siva Sankalp Patel

Dmitry Abulkhanov

Marco Zocca

Manan Dey

Zhihan Zhang

N. Fahmy

Urvashi Bhattacharyya

Wenhao Yu

Swayam Singh

Sasha Luccioni

Paulo Villegas

M. Kunakov

Fedor Zhdanov

Manuel Romero

Tony Lee

Nadav Timor

Jennifer Ding

Claire S Schlesinger

Hailey Schoelkopf

Jan Ebert

Tri Dao

Mayank Mishra

Alex Gu

Jennifer Robinson

Carolyn Jane Anderson

Brendan Dolan-Gavitt

Danish Contractor

Daniel Fried

Yacine Jernite

Carlos Muñoz Ferrandis

Sean Hughes

Thomas Wolf

Arjun Guha

Leandro Von Werra

Harm de Vries

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs)… (see more), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

2022-12-31

Trans. Mach. Learn. Res. (published)