Siva Reddy

Biography

Siva Reddy is an assistant professor at the School of Computer Science and in the Department of Linguistics at McGill University. He completed a postdoc with the Stanford NLP Group in September 2019.

Reddy’s research goal is to enable machines with natural language understanding abilities in order to facilitate applications like question answering and conversational systems. His expertise includes building symbolic (linguistic and induced) and deep learning models for language.

Current Students

Vaibhav Adlakha

PhD - McGill University

Parishad BehnamGhader

Master's Research - McGill University

PhD - McGill University

Collaborating researcher

Gaurav Kamath

PhD - McGill University

Amirhossein Kazemnejad

Master's Research - McGill University

Aditi Khandelwal

PhD - McGill University

Principal supervisor :

PhD - McGill University

Aravind Krishnan

Collaborating Alumni - UNIVERSITÄT DES SAARLANDES

Benno Krojer

PhD - McGill University

andreas.madsen@mila.quebec

Zichao Li

PhD - McGill University

Co-supervisor :

Jackie Cheung

Xing Han Lu

PhD - McGill University

PhD - Polytechnique Montréal

Principal supervisor :

Sarath Chandar

Research Intern - McGill University

PhD - McGill University

Postdoctorate - McGill University

Arkil Patel

PhD - McGill University

Principal supervisor :

Collaborating researcher

Nooshin Soheililangroodi

Research Intern - McGill University

Karolina Ewa Stańczak

Collaborating Alumni - McGill University

How Do We Explain AI and Ensure the Explanation Is True? Faithfulness Measurable Models Tell You How

Ada Tur Tur

Research Intern - McGill University

Ivan Vulić

Collaborating researcher - Cambridge University

Alejandra Zambrano

Collaborating Alumni - McGill University

Blog Posts

October 1, 2024

Andrea Madsen

Siva Reddy

Sarath Chandar

Read the article

Publications

Exploiting Instruction-Following Retrievers for Malicious Information Retrieval

Parishad BehnamGhader

Nicholas Meade

2025-03-11

ArXiv (preprint)

The BrowserGym Ecosystem for Web Agent Research

Thibault Le Sellier de Chezelles

Maxime Gasse

Alexandre Lacoste

Alexandre Drouin

Massimo Caccia

Léo Boisvert

Megh Thakkar

Tom Marty

Rim Assouel

Sahar Omidi Shayegan

Lawrence Keunho Jang

Xing Han Lu

Ori Yoran

Dehan Kong

Frank F. Xu

Quentin Cappart

Graham Neubig

Russ Salakhutdinov

Nicolas Chapados

The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging a… (see more)utomation and Large Language Models (LLMs) for web interaction tasks. Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. BrowserGym aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. Combined with AgentLab, a complementary framework that aids in agent creation, testing, and analysis, BrowserGym offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. This standardized approach seeks to reduce the time and complexity of developing web agents, supporting more reliable comparisons and facilitating in-depth analysis of agent behaviors, and could result in more adaptable, capable agents, ultimately accelerating innovation in LLM-driven automation. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across all benchmarks currently available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.

2025-03-08

TMLR (accepted)

openreview.net

SafeArena: Evaluating the Safety of Autonomous Web Agents

Ada Defne Tur

Nicholas Meade

Xing Han Lu

Alejandra Zambrano

Arkil Patel

Esin Durmus

Spandana Gella

Karolina Sta'nczak

2025-03-06

ArXiv (preprint)

WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation

Rabiul Awal

Mahsa Massoud

Zichao Li

Aarash Feizi

Suyuchen Wang

Chris Pal

Aishwarya Agrawal

David Vazquez

Juan A. Rodriguez

Perouz Taslakian

Spandana Gella

Sai Rajeswar

Understanding diverse web data and automating web development presents an exciting challenge for agentic AI. While existing benchmarks addre… (see more)ss isolated web-based tasks—such as website-based Visual Question Answering (VQA) and UI-to-code generation—they lack a unified evaluation suite for assessing web agents that interact with and reason about web environments. We introduce WebMMU, a large-scale benchmark for evaluating AI-driven web agents across multilingual website VQA, HTML/CSS/JavaScript code editing, and sketch-to-code generation. WebMMU provides a comprehensive evaluation suite with real-world website data, multi-step reasoning tasks, and functional UI understanding. Benchmarking state-of-the-art multimodal models on WebMMU reveals significant limitations in web-based reasoning, layout understanding, and structured code generation, particularly in preserving UI hierarchy, handling multilingual content, and producing robust, functional code. While most existing models are optimized for English-only settings, WebMMU highlights the challenges of cross-lingual adaptation in real-world web development. These findings expose critical gaps in current models’ ability to understand website structures, execute user instructions, and generate high-quality web code, underscoring the need for more advanced multimodal reasoning in AI-driven web understanding and development.

2025-03-05

ICLR.cc/2025/Workshop/DL4C (published)

openreview.net

Large language models deconstruct the clinical intuition behind diagnosing autism

Jack Stanley

Emmett Rabot

Eugene Belilovsky

L. Mottron

Danilo Bzdok

2025-03-01

Cell (published)

SafeArena: Evaluating the Safety of Autonomous Web Agents

Ada Defne Tur

Nicholas Meade

Xing Han Lu

Alejandra Zambrano

Arkil Patel

Esin Durmus

Spandana Gella

Karolina Sta'nczak

2025-03-01

arXiv (published)

Societal Alignment Frameworks Can Improve LLM Alignment

Karolina Sta'nczak

Nicholas Meade

Mehar Bhatia

Hattie Zhou

Konstantin Bottinger

Jeremy Barnes

Jason Stanley

Jessica Montgomery

Richard Zemel

Nicolas Papernot

Nicolas Chapados

Denis Therien

Timothy P. Lillicrap

Ana Marasovi'c

Sylvie Delacroix

Gillian K. Hadfield

Recent progress in large language models (LLMs) has focused on producing responses that meet human expectations and align with shared values… (see more) - a process coined alignment. However, aligning LLMs remains challenging due to the inherent disconnect between the complexity of human values and the narrow nature of the technological approaches designed to address them. Current alignment methods often lead to misspecified objectives, reflecting the broader issue of incomplete contracts, the impracticality of specifying a contract between a model developer, and the model that accounts for every scenario in LLM alignment. In this paper, we argue that improving LLM alignment requires incorporating insights from societal alignment frameworks, including social, economic, and contractual alignment, and discuss potential solutions drawn from these domains. Given the role of uncertainty within societal alignment frameworks, we then investigate how it manifests in LLM alignment. We end our discussion by offering an alternative view on LLM alignment, framing the underspecified nature of its objectives as an opportunity rather than perfect their specification. Beyond technical improvements in LLM alignment, we discuss the need for participatory alignment interface designs.

2025-03-01

arXiv (published)

Societal Alignment Frameworks Can Improve LLM Alignment

Karolina Sta'nczak

Nicholas Meade

Mehar Bhatia

Hattie Zhou

Konstantin Bottinger

Jeremy Barnes

Jason Stanley

Jessica Montgomery

Richard Zemel

Nicolas Papernot

Nicolas Chapados

Denis Therien

Timothy P. Lillicrap

Ana Marasovi'c

Sylvie Delacroix

Gillian K. Hadfield

2025-02-27

ArXiv (preprint)

How to Get Your LLM to Generate Challenging Problems for Evaluation

Arkil Patel

Dzmitry Bahdanau

The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional hum… (see more)an annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems, particularly for tasks such as long-context reasoning. Moreover, the rapid saturation of existing human-curated benchmarks by LLMs further necessitates the need to develop scalable and automatically renewable evaluation methodologies. In this work, we introduce **CHASE**, a unified framework to synthetically generate challenging problems using LLMs without human involvement. For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover since we want to generate synthetic data for evaluation, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: document-based question answering, repository-level code completion, and math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60\% accuracy, thereby demonstrating the effectiveness of our framework at generating hard problems. Our experiments further reveal that the Gemini models significantly outperform other LLMs at long-context reasoning, and that the performance of all LLMs drastically drops by as much as 70\% when we scale up the context size to 50k tokens.

2025-02-20

ArXiv (preprint)

openreview.net

Warmup Generations: A Task-Agnostic Approach for Guiding Sequence-to-Sequence Learning with Unsupervised Initial State Generation

Senyu Li

Zipeng Sun

Jiayi Wang

Xue (Steve) Liu

Pontus Stenetorp

David Ifeoluwa Adelani

2025-02-17

ArXiv (preprint)

ReTreever: Tree-based Coarse-to-Fine Representations for Retrieval

Shubham Gupta

Zichao Li

Tianyi Chen

Cem Subakan

Perouz Taslakian

Valentina Zantedeschi

2025-02-11

ArXiv (preprint)

ReTreever: Tree-based Coarse-to-Fine Representations for Retrieval

Shubham Gupta

Zichao Li

Tianyi Chen

Cem Subakan

Perouz Taslakian

Valentina Zantedeschi

Document retrieval is a core component of question-answering systems, as it enables conditioning answer generation on new and large-scale co… (see more)rpora. While effective, the standard practice of encoding documents into high-dimensional embeddings for similarity search entails large memory and compute footprints, and also makes it hard to inspect the inner workings of the system. In this paper, we propose a tree-based method for organizing and representing reference documents at various granular levels, which offers the flexibility to balance cost and utility, and eases the inspection of the corpus content and retrieval operations. Our method, called ReTreever, jointly learns a routing function per internal node of a binary tree such that query and reference documents are assigned to similar tree branches, hence directly optimizing for retrieval performance. Our evaluations show that ReTreever generally preserves full representation accuracy. Its hierarchical structure further provides strong coarse representations and enhances transparency by indirectly learning meaningful semantic groupings. Among hierarchical retrieval methods, ReTreever achieves the best retrieval accuracy at the lowest latency, proving that this family of techniques can be viable in practical applications.

2025-02-11

ArXiv (preprint)