Portrait de Sai Rajeswar Mudumba

Sai Rajeswar Mudumba

Membre industriel associé
Professeur associé, Université de Montréal
Chercheur scientifique principal, ServiceNow
Sujets de recherche
Apprentissage de représentations
Apprentissage multimodal
Apprentissage par renforcement
Modèles génératifs
Site web
Google Scholar
LinkedIn
X

Biographie

Sai Rajeswar est chercheur principal chez ServiceNow, professeur associé à l'Université de Montréal et membre industriel associé à Mila - Institut québécois d'intelligence artificielle. Au cours des huit dernières années, ses travaux ont porté sur les modèles génératifs, l’apprentissage par renforcement et l’IA multimodale. Plus récemment, il s’est concentré sur la création de systèmes multimodaux servant de fondement à des agents d’IA généralistes — des systèmes intégrant perception et action tout en incorporant des retours de l’environnement. De manière générale, ses recherches visent à intégrer perception et action afin d’améliorer l’applicabilité dans le monde réel, tout en accordant une attention particulière à l’impact responsable sur la société dans son ensemble.

Publications

InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation
Gaurav Sahu
Abhay Puri
Juan Rodriguez
Amirhossein Abaskohi
Mohammad Chegini
Alexandre Drouin
Perouz Taslakian
Valentina Zantedeschi
Alexandre Lacoste
David Vázquez
Nicolas Chapados
Christopher Pal
Sai Rajeswar Mudumba
Issam Hadj Laradji
Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We intro… (voir plus)duce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics and can be accessed here: https://github.com/ServiceNow/insight-bench.
2025-01-21
ICLR.cc/2025/Conference (poster)
doi.org
openreview.net