Portrait of Sai Rajeswar Mudumba

Sai Rajeswar Mudumba

Associate Industry Member
Adjunct Professor, Université de Montréal
Senior Research Scientist, ServiceNow
Research Topics
Generative Models
Multimodal Learning
Reinforcement Learning
Representation Learning
Website
Google Scholar
LinkedIn
X

Biography

Sai Rajeswar is a Staff Research Scientist at ServiceNow, an Adjunct professor at Université de Montréal and an Academic Industrial Member at Mila - Quebec Artificial Intelligence Institute. His work over the last eight years spans generative models, reinforcement learning and multimodal AI. Lately, has been focusing on building multimodal systems that serve as the foundation for generalist AI agents, systems that integrate perception and action while incorporating feedback from the environment. Broadly, his work aims to integrate perception and action to improve real-world applicability, always with an eye towards responsible impact on society at large.

Publications

InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation
Gaurav Sahu
Abhay Puri
Juan Rodriguez
Amirhossein Abaskohi
Mohammad Chegini
Alexandre Drouin
Perouz Taslakian
Valentina Zantedeschi
Alexandre Lacoste
David Vázquez
Nicolas Chapados
Christopher Pal
Sai Rajeswar Mudumba
Issam Hadj Laradji
Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We intro… (see more)duce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics and can be accessed here: https://github.com/ServiceNow/insight-bench.
2025-01-21
ICLR.cc/2025/Conference (poster)
doi.org
openreview.net