Siva Reddy

vaibhav.adlakha@mila.quebec

Biography

Siva Reddy is an assistant professor at the School of Computer Science and in the Department of Linguistics at McGill University. He completed a postdoc with the Stanford NLP Group in September 2019.

Reddy’s research goal is to enable machines with natural language understanding abilities in order to facilitate applications like question answering and conversational systems. His expertise includes building symbolic (linguistic and induced) and deep learning models for language.

Current Students

Vaibhav Adlakha

PhD - McGill University

Kareem Ahmed

Postdoctorate - McGill University

Principal supervisor :

Guillaume Rabusseau

kareem.yousrii@gmail.com

Parishad BehnamGhader

PhD - McGill University

parishad.behnamghader@mila.quebec

PhD - McGill University

mehar.bhatia@mila.quebec

verna.dankers@mila.quebec

Verna Dankers

Postdoctorate - McGill University

Independent visiting researcher

desmond.elliott@mila.quebec

spandana.gella@mila.quebec

Jay Gala

Master's Research - McGill University

Co-supervisor :

Collaborating researcher

gaurav.kamath@mila.quebec

Gaurav Kamath

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

Aditi Khandelwal

PhD - McGill University

Principal supervisor :

Golnoosh Farnadi

aditi.khandelwal@mila.quebec

Austin Kraft

PhD - McGill University

Co-supervisor :

Timothy O'Donnell

austin.kraft@mila.quebec

Benno Krojer

PhD - McGill University

benno.krojer@mila.quebec

Zichao Li

PhD - McGill University

Co-supervisor :

Jackie Cheung

zichao.li@mila.quebec

Fengyuan Liu

Master's Research - McGill University

Co-supervisor :

Dzmitry Bahdanau

fengyuan.liu@mila.quebec

sreenath.madathil@mila.quebec

Sreenath Madathil

Master's Research - McGill University

nicholas.meade@mila.quebec

Nicholas Meade

PhD - McGill University

Postdoctorate - McGill University

marius.mosbach@mila.quebec

Marzia Nouri

Master's Research - McGill University

marzia.nouri@mila.quebec

Arkil Patel

PhD - McGill University

Principal supervisor :

Dzmitry Bahdanau

arkil.patel@mila.quebec

erfan.raoofian@mila.quebec

Neel Rajani

Collaborating researcher - University of Edinburgh

neel.rajani@mila.quebec

Collaborating researcher - N/A

Ben Saine

Collaborating researcher - McGill University

ben.saine@mila.quebec

Kusha Sareen

PhD - McGill University

kusha.sareen@mila.quebec

dongchan.shin@mila.quebec

Dongchan Shin

Collaborating Alumni

tomasvergara.browne@mila.quebec

Ada Tur

Research Intern - McGill University

PhD - McGill University

Collaborating researcher - McGill University

donghao.zeng@mila.quebec

How Do We Explain AI and Ensure the Explanation Is True? Faithfulness Measurable Models Tell You How

Blog Posts

October 1, 2024

Andrea Madsen

Siva Reddy

Sarath Chandar

Read the article

Publications

BRIDGE: Predicting Human Task Completion Time From Model Performance

Nilaksh

Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task diffic… (see more)ulty. Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks. In this work, we propose BRIDGE, a unified psychometric framework that learns the latent difficulty scale from model responses and anchors it to human task completion time. Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple benchmarks. We demonstrate that latent task difficulty varies linearly with the logarithm of human completion time, allowing human task completion time to be inferred for new benchmarks from model performance alone. Leveraging this alignment, we forecast frontier model capabilities in terms of human task length and independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months.

2025-12-31

International Conference on Machine Learning (Accept (regular))

Hierarchical Retrieval at Scale: Bridging Transparency and Efficiency

Tianyi Chen

Valentina Zantedeschi

Information retrieval is a core component of many intelligent systems as it enables conditioning of outputs on new and large-scale datasets.… (see more) While effective, the standard practice of encoding data into high-dimensional representations for similarity search entails large memory and compute footprints, and also makes it hard to inspect the inner workings of the system. Hierarchical retrieval methods offer an interpretable alternative by organizing data at multiple granular levels, yet do not match the efficiency and performance of flat retrieval approaches. In this paper, we propose ReTreever, a tree-based method that makes hierarchical retrieval viable at scale by directly optimizing its structure for retrieval performance while naturally providing transparency through meaningful semantic groupings. Our method offers the flexibility to balance cost and utility by indexing data using representations from any tree level. We show that ReTreever delivers strong coarse (intermediate levels) and fine representations (terminal level), while achieving the highest retrieval accuracy at the lowest latency among hierarchical methods. These results demonstrate that this family of techniques is viable in practical applications.

2025-12-31

International Conference on Machine Learning (Accept (regular))

Operationalizing the Superficial Alignment Hypothesis via Task Complexity

Tiago Pimentel

The superficial alignment hypothesis (SAH) posits that large language models learn most of their knowledge during pre-training, and that pos… (see more)t-training merely surfaces this knowledge. The SAH, however, lacks a precise definition, which has led to (i) different and seemingly orthogonal arguments supporting it, and (ii) important critiques to it. We propose a new metric called **Task Complexity**: the length of the shortest program that achieves a target performance on a task. In this framework, the SAH claims that pre-trained models drastically reduce the task complexity of achieving high performance on many tasks. Our definition unifies prior arguments supporting the SAH, interpreting them as different strategies to find such short programs. Experimentally, we estimate task complexities of mathematical reasoning, machine translation, and instruction following tasks and show that their respective task complexities can be remarkably low when conditioned on a pre-trained model. Further, we find that pre-training enables access to strong performances on our tasks, but it can require programs of gigabytes of length to access them. Post-training, on the other hand, collapses the complexity of reaching this same performance by several orders of magnitude. Overall, our results highlight that task adaptation can require remarkably little information—often just a few kilobytes.

2025-12-31

International Conference on Machine Learning (Accept (regular))

The Markovian Thinker

Milad Aghajohari

Kamran Chitsaz

Amirhossein Kazemnejad

Reasoning LLMs suffer from quadratic compute growth as their context length increases, making reinforcement learning with verifiable rewards… (see more) (RLVR) and test-time scaling prohibitively expensive. Prior work has tried to lighten the computational burden by shortening reasoning traces through pruning, summarization, or multi-stage training, but these methods remain bound to quadratic costs. We introduce Delethink, a thinking algorithm that realizes the Markovian Thinking Paradigm. Instead of producing one long monolithic reasoning trace, Delethink thinks in a sequence of chunks, the Delethink trace. Each chunk continues reasoning by referring only to a fixed number of prior tokens, which functions as a Markovian state sufficient for progressing reasoning, while deleting the rest. This preserves continuity without carrying the quadratic baggage. As a result, compute scales linearly and peak memory remains constant. In experiments, we show that Delethink can be applied directly to off-the-shelf reasoning models ranging from

2025-12-31

International Conference on Learning Representations (Accept (Poster))

The Promise of RL for Autoregressive Image Editing

Saba Ahmadi

Rabiul Awal

Ankur Sikarwar

Amirhossein Kazemnejad

Ge Ya Luo

Juan A. Rodriguez

Sai Rajeswar

Christopher Pal

Benno Krojer

Aishwarya Agrawal

While image generation techniques are now capable of producing high-quality images that respect prompts which span multiple sentences, the t… (see more)ask of text-guided image editing remains a challenge. Even edit requests that consist of only a few words often fail to be executed correctly. We explore three strategies to enhance performance on a wide range of image editing tasks: supervised fine-tuning (SFT), reinforcement learning (RL), and Chain-of-Thought (CoT) reasoning. In order to study all these components in one consistent framework, we adopt an autoregressive multimodal model that processes textual and visual tokens in a unified manner. We find RL combined with a large multi-modal LLM verifier to be the most effective of these strategies. As a result, we release EARL: Editing with Autoregression and RL, a strong RL-based image editing model that performs competitively on a diverse range of edits compared to strong baselines, despite using much less training data. Thus, EARL pushes the frontier of autoregressive multimodal models on image editing. We release our code, training data, and trained models at https://github.com/mair-lab/EARL.

2025-12-02

Neural Information Processing Systems (Accept (poster))

Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models

Eva Portelance

Timothy J. O’Donnell

Semantic and syntactic bootstrapping posit that children use their prior knowledge of one linguistic domain, say syntactic relations, to hel… (see more)p later acquire another, such as the meanings of new words. Empirical results supporting both theories may tempt us to believe that these are different learning strategies, where one may precede the other. Here, we argue that they are instead both contingent on a more general learning strategy for language acquisition: joint learning. Using a series of neural visually-grounded grammar induction models, we demonstrate that both syntactic and semantic bootstrapping effects are strongest when syntax and semantics are learnt simultaneously. Joint learning results in better grammar induction, realistic lexical category learning, and better interpretations of novel sentence and verb meanings. Joint learning makes language acquisition easier for learners by mutually constraining the hypotheses spaces for both syntax and semantics. Studying the dynamics of joint inference over many input sources and modalities represents an important new direction for language modeling and learning research in both cognitive sciences and AI, as it may help us explain how language can be acquired in more constrained learning settings.

2025-11-30

OpenReview (published)

REARANK: Reasoning Re-ranking Agent via Reinforcement Learning

Le Zhang

Bo Wang

Xipeng Qiu

Aishwarya Agrawal

We present REARANK, a large language model (LLM)-based listwise reasoning reranking agent. REARANK explicitly reasons before reranking, sign… (see more)ificantly improving both performance and interpretability. Leveraging reinforcement learning and data augmentation, REARANK achieves substantial improvements over baseline models across popular information retrieval benchmarks, notably requiring only 179 annotated samples. Built on top of Qwen2.5-7B, our REARANK-7B demonstrates performance comparable to GPT-4 on both in-domain and out-of-domain benchmarks and even surpasses GPT-4 on reasoning-intensive BRIGHT benchmarks. These results underscore the effectiveness of our approach and highlight how reinforcement learning can enhance LLM reasoning capabilities in reranking.

2025-10-31

Conference on Empirical Methods in Natural Language Processing (published)

arxiv.org

WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation

Christopher Pal

Juan A. Rodriguez

Sai Rajeswar

We present WebMMU, a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing inv… (see more)olving HTML/CSS/JavaScript, and (3) mockup-to-code generation. Unlike prior benchmarks that treat these tasks separately, WebMMU unifies them using expert-annotated, real-world web data to assess models'abilities in complex multi-step reasoning, precise element grounding, and functional UI comprehension and coding. Our evaluation shows that while multimodal large language models (MLLMs) perform well on basic information extraction, they struggle with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content. These findings reveal key limitations in current MLLMs and underscore the need for improved multimodal and cross-lingual reasoning to build future web agents capable of automating diverse web development tasks.

2025-10-31

Conference on Empirical Methods in Natural Language Processing (published)

SAFEARENA: Evaluating the Safety of Autonomous Web Agents

Ada Defne Tur

Esin Durmus

Karolina Stanczak

LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for ma… (see more)licious purposes, such as posting misinformation in an online forum or selling illicit substances on a website. To evaluate these risks, we propose SafeArena, the first benchmark to focus on the deliberate misuse of web agents. SafeArena comprises 250 safe and 250 harmful tasks across four websites. We classify the harmful tasks into five harm categories -- misinformation, illegal activity, harassment, cybercrime, and social bias, designed to assess realistic misuses of web agents. We evaluate leading LLM-based web agents, including GPT-4o, Claude-3.5 Sonnet, Qwen-2-VL 72B, and Llama-3.2 90B, on our benchmark. To systematically assess their susceptibility to harmful tasks, we introduce the Agent Risk Assessment framework that categorizes agent behavior across four risk levels. We find agents are surprisingly compliant with malicious requests, with GPT-4o and Qwen-2 completing 34.7% and 27.3% of harmful requests, respectively. Our findings highlight the urgent need for safety alignment procedures for web agents. Our benchmark is available here: https://safearena.github.io

2025-10-05

Proceedings of the 42nd International Conference on Machine Learning (published)

proceedings.mlr.press

How to Get Your LLM to Generate Challenging Problems for Evaluation

Arkil Patel

Dzmitry Bahdanau

The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional hum… (see more)an annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems, particularly for tasks such as long-context reasoning. Moreover, the rapid saturation of existing human-curated benchmarks by LLMs further necessitates the need to develop scalable and automatically renewable evaluation methodologies. In this work, we introduce **CHASE**, a unified framework to synthetically generate challenging problems using LLMs without human involvement. For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover since we want to generate synthetic data for evaluation, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: document-based question answering, repository-level code completion, and math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60\% accuracy, thereby demonstrating the effectiveness of our framework at generating hard problems. Our experiments further reveal that the Gemini models significantly outperform other LLMs at long-context reasoning, and that the performance of all LLMs drastically drops by as much as 70\% when we scale up the context size to 50k tokens.

2025-09-22

NeurIPS.cc/2025/Workshop/LLM_Evaluation (poster)

Towards Democratizing LLMs: Investigating Multilingual Mixture-of-Experts Models

2025-09-21

NeurIPS.cc/2025/Workshop/WiML (published)

Semantic change in adults is not primarily a generational phenomenon

Gaurav Kamath

Michelle Yang

Morgan Sonderegger

Dallas Card

A central question in the study of language change is whether or not such change is generational. If a language changes over time generation… (see more)-by-generation, the process looks as follows: New generations of speakers introduce innovations, while older speakers conserve their usage patterns, and the language changes as new generations replace older ones. At the opposite extreme, language change could be a zeitgeist phenomenon, in which changes are universally adopted by speakers simultaneously, regardless of age or generational cohort. This paper asks this question in the context of word meaning change. We analyze meaning change in over 100 words across more than 7.9 million U.S. congressional speeches, to observe whether, when a word sense rises or falls in prominence, adult speakers from different generations uniformly adopt it, or those from older generations conserve their prior usage. Using language model-based word sense induction methods, we identify different senses of each word, and then model the prevalence of each of these word senses as a function of time and speaker age. We find that most words show a small but statistically significant effect of speaker age; across almost 140 y of Congress, older speakers typically take longer than younger speakers to follow changes in word usage, but nevertheless do so within a few years. Our findings indicate that despite minor age-based differences, word meaning change among mature speakers is likely not a generational process, but rather a zeitgeist process, in which older adult speakers can readily adopt new word usage patterns.

2025-07-27

Proceedings of the National Academy of Sciences (published)