Portrait of Prateek Humane

Prateek Humane

Master's Research
Supervisor
Irina Rish
Research Topics
Multimodal Learning
Website
Google Scholar
GitHub

Publications

CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
Alexis Roger
Prateek Humane
Daniel Z Kaplan
Kshitij Gupta
Qirui Sun
George Adamopoulos
Jonathan Siu Chi Lim
Quentin Gregory Anthony
Edwin Fennell
Irina Rish
The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and ben… (see more)chmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research.
2025-01-16
ArXiv (preprint)
arxiv.org
arxiv.org
Robin: a Suite of Multi-Scale Vision-Language Models and the CHIRP Evaluation Benchmark
Alexis Roger
Prateek Humane
Daniel Z Kaplan
Kshitij Gupta
Qirui Sun
George Adamopoulos
Jonathan Siu Chi Lim
Quentin Gregory Anthony
Edwin Fennell
Irina Rish
The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and ben… (see more)chmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research.
2025-01-16
ArXiv (preprint)
doi.org
arxiv.org