Portrait of Lu Li is unavailable

Lu Li

PhD - Université de Montréal
Supervisor
Pierre-Luc Bacon
Research Topics
Deep Learning
Reinforcement Learning
Google Scholar
GitHub

Publications

VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured … (see more)texts using pixel-level hints within images through complex reasoning. This task stems from the observation that text embedded in images intrinsically differs from common visual elements and text due to the need to align the modalities of vision, text, and text embedded in images. While many works incorporate text into images for visual question answering, they mostly rely on OCR or masked language modeling, reducing the task to text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny, exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct VCR-WIKI for VCR using Wikipedia images with captions, including 2.11M English and 346K Chinese training entities, plus 5K validation and 5K test entities in both languages, each in easy and hard configurations. We also make a hidden test set, VCR-HIDDEN, to avoid potential overfitting on VCR-WIKI. Our results reveal that current vision-language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-WIKI and the data construction code to facilitate future research.
2025-01-22
ICLR.cc/2025/Conference (poster)
openreview.net
openreview.net
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
Lu Li
Tianyu Zhang
Zhiqi Bu
Suyuchen Wang
Huan He
Jie Fu
Yonghui Wu
Jiang Bian
Yong Chen
Yoshua Bengio
Model merging has emerged as an effective approach to combine multiple single-task models into a multitask model. This process typically inv… (see more)olves computing a weighted average of the model parameters without any additional training. Existing model-merging methods focus on enhancing average task accuracy. However, interference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a set of solutions with various trade-offs can be more informative, helping practitioners make decisions based on diverse preferences. In this paper, we introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from a pre-selected set of scaling coefficients. Experimental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.
2024-06-11
ArXiv (preprint)
doi.org
arxiv.org
VCR: Visual Caption Restoration
Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured … (see more)texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants. Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-Wiki and the data construction code to facilitate future research.
2024-06-10
ArXiv (preprint)
doi.org
arxiv.org
VCR: Visual Caption Restoration
Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured … (see more)texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants. Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-Wiki and the data construction code to facilitate future research.
2024-06-10
ArXiv (preprint)
doi.org
arxiv.org
VCR: Visual Caption Restoration
Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured … (see more)texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants. Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-Wiki and the data construction code to facilitate future research.
2024-06-10
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
arxiv.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid
Nikolay Savinov
Denis Teplyashin
Dmitry Lepikhin
Timothy P. Lillicrap
Jean-Baptiste Alayrac
Radu Soricut
Angeliki Lazaridou
Orhan Firat
Julian Schrittwieser
Ioannis Antonoglou
Rohan Anil
Sebastian Borgeaud
Andrew M. Dai
Katie Millican
Ethan Dyer
Mia Glaese
Thibault Sottiaux
Benjamin Lee
Fabio Viola … (see 479 more)
Malcolm Reynolds
Yuanzhong Xu
James L. Molloy
Jilin Chen
Michael Acheson Isard
Paul R. Barham
Tom Hennigan
Ross McIlroy
Melvin Johnson
J. Schalkwyk
Eli Collins
Eliza Rutherford
Erica Moreira
Kareem W. Ayoub
Megha Goel
Clemens Meyer
Gregory Thornton
Zhen Yang
Henryk Michalewski
Zaheer Abbas
Nathan Schucher
Ankesh Anand
Richard Ives
James Keeling
Karel Lenc
Salem Haykal
Siamak Shakeri
Pranav Shyam
Aakanksha Chowdhery
Roman Ring
Stephen Spencer
Eren Sezener
Luke Vilnis
Oscar Chang
Nobuyuki Morioka
George Tucker
Ce Zheng
Oliver Woodman
Nithya Attaluri
Tomas Kocisky
Evgenii Eltyshev
Xi Chen
Timothy Chung
Vittorio Selo
Siddhartha Brahma
Petko Georgiev
Ambrose Slone
Zhenkai Zhu
James Lottes
Siyuan Qiao
Ben Caine
Sebastian Riedel
Alex Tomala
Martin J. Chadwick
J Christopher Love
Peter Choy
Sid Mittal
Neil Houlsby
Yunhao Tang
Matthew Lamm
Libin Bai
Qiao Zhang
Luheng He
Yong Cheng
Peter Conway Humphreys
Yujia Li
Sergey Brin
Albin Cassirer
Ying-Qi Miao
Lukáš Žilka
Taylor Tobin
Kelvin Xu
Lev Proleev
Daniel Sohn
Alberto Magni
Lisa Anne Hendricks
Isabel Gao
Santiago Ontan'on
Oskar Bunyan
Nathan Byrd
Abhanshu Sharma
Biao Zhang
Mario Pinto
Rishika Sinha
Harsh Mehta
Dawei Jia
Sergi Caelles
Albert Webson
Alex Morris
Becca Roelofs
Yifan Ding
Robin Strudel
Xuehan Xiong
Marvin Ritter
Mostafa Dehghani
Rahma Chaabouni
Abhijit Karmarkar
Guangda Lai
Fabian Mentzer
Bibo Xu
YaGuang Li
Yujing Zhang
T. Paine
Alex Goldin
Behnam Neyshabur
Kate Baumli
Anselm C. Levskaya
Michael Laskin
Wenhao Jia
Jack W. Rae
Kefan Xiao
Antoine He
Skye Giordano
Laksh-man Yagati
Jean-Baptiste Lespiau
Paul Natsev
Sanjay Ganapathy
Fangyu Liu
Danilo Martins
Nanxin Chen
Yunhan Xu
Megan Barnes
Rhys May
Arpi Vezer
Junhyuk Oh
Ken Franko
Sophie Bridgers
Ruizhe Zhao
Boxi Wu
Basil Mustafa
Sean Sechrist
Emilio Parisotto
Thanumalayan Sankaranarayana Pillai
Chris Larkin
Chenjie Gu
Christina Sorokin
M. Krikun
Alexey Guseynov
Jessica Landon
Romina Datta
Alexander Pritzel
Phoebe Thacker
Fan Yang
Kevin Hui
A.E. Hauth
Chih-Kuan Yeh
David Barker
Justin Mao-jones
Sophia Austin
Hannah Rachel Sheahan
Parker Schuh
James Svensson
Rohan Jain
Vinay Venkatesh Ramasesh
Anton Briukhov
Da-Woon Chung
Tamara von Glehn
Christina Butterfield
Priya Jhakra
Matt Wiethoff
Justin Frye
Jordan Grimstad
Beer Changpinyo
Charline Le Lan
Anna Bortsova
Yonghui Wu
Paul Voigtlaender
Tara N. Sainath
Charlotte Smith
Will Hawkins
Kris Cao
James Besley
Srivatsan Srinivasan
Mark Omernick
Colin Gaffney
Gabriela Surita
Ryan Burnell
Bogdan Damoc
Junwhan Ahn
Andrew Brock
Mantas Pajarskas
Anastasia Petrushkina
Seb Noury
Lorenzo Blanco
Kevin Swersky
Arun Ahuja
Thi Avrahami
Vedant Misra
Raoul de Liedekerke
Mariko Iinuma
Alex Polozov
Sarah York
George van den Driessche
Paul Michel
Justin Chiu
Rory Blevins
Zach Gleicher
Adria Recasens
Alban Rrustemi
Elena Gribovskaya
Aurko Roy
Wiktor Gworek
S'ebastien M. R. Arnold
Lisa Lee
James Lee-Thorp
Marcello Maggioni
Enrique Piqueras
Kartikeya Badola
Sharad Mandyam Vikram
Lucas Gonzalez
Anirudh Baddepudi
Evan Senter
Jacob Devlin
James Qin
Michael Azzam
Maja Trebacz
M. Polacek
Kashyap Krishnakumar
Shuo-yiin Chang
Matthew Tung
Ivo Penchev
Rishabh Joshi
Kate Olszewska
Carrie Muir
Mateo Wirth
Ale Jakse Hartman
Joshua Newlan
Sheleem Kashem
Vijay Bolina
Elahe Dabir
Joost Van Amersfoort
Zafarali Ahmed
James Cobon-Kerr
Aishwarya B Kamath
Arnar Mar Hrafnkelsson
Le Hou
Ian Mackinnon
Alexandre Fréchette
Eric Noland
Xiance Si
Emanuel Taropa
Dong Li
Phil Crone
Anmol Gulati
S'ebastien Cevey
Jonas Adler
Ada Ma
David Silver
Simon Tokumine
Richard Powell
Stephan Lee
Samer Hassan
Diana Mincu
Antoine Yang
Nir Levine
Jenny Brennan
Mingqiu Wang
Sarah Hodkinson
Jeffrey Zhao
Josh Lipschultz
Aedan Pope
Michael B. Chang
Cheng Li
Laurent El Shafey
Michela Paganini
Sholto Douglas
Bernd Bohnet
Fabio Pardo
Seth Odoom
Mihaela Rosca
Cicero Nogueira dos Santos
Kedar Soparkar
Arthur Guez
Tom Hudson
Steven Hansen
Chulayuth Asawaroengchai
Ravichandra Addanki
Tianhe Yu
Wojciech Stokowiec
Mina Khan
Justin Gilmer
Jaehoon Lee
Carrie Grimes Bostock
Keran Rong
Jonathan Caton
Pedram Pejman
Filip Pavetic
Geoff Brown
Vivek Sharma
Mario Luvci'c
Rajkumar Samuel
Josip Djolonga
Amol Mandhane
Lars Lowe Sjosund
Elena Buchatskaya
Elspeth White
Natalie Clay
Jiepu Jiang
Hyeontaek Lim
Ross Hemsley
Jane Labanowski
Nicola De Cao
David Steiner
Sayed Hadi Hashemi
Jacob Austin
Anita Gergely
Tim Blyth
Joe Stanton
Kaushik Shivakumar
Aditya Siddhant
Anders Johan Andreassen
Carlos L. Araya
Nikhil Sethi
Rakesh Shivanna
Steven Hand
Ankur Bapna
A. Khodaei
Antoine Miech
Garrett Tanzer
Andy Swing
Shantanu Thakoor
Zhufeng Pan
Zachary Nado
Stephanie Winkler
Dian Yu
Mohammad Saleh
Lorenzo Maggiore
Iain Barr
Minh Giang
Thais Kagohara
Ivo Danihelka
Amit Marathe
Vladimir Feinberg
Mohamed Elhawaty
Nimesh Ghelani
Dan Horgan
Helen Miller
Lexi Walker
Richard Tanburn
Mukarram Tariq
Disha Shrivastava
Fei Xia
Chung-Cheng Chiu
Zoe Ashwood
Khuslen Baatarsukh
Sina Samangooei
Fred Alcober
Axel Stjerngren
Paul Komarek
Katerina Tsihlas
Anudhyan Boral
Ramona Comanescu
Jeremy Chen
Ruibo Liu
Dawn Bloxwich
Charlie Chen
Yanhua Sun
Fangxiaoyu Feng
Matthew Mauger
Xerxes Dotiwalla
Vincent Hellendoorn
Michael Sharman
Ivy Zheng
Krishna S Haridasan
Gabriel Barth-Maron
Craig Swanson
Dominika Rogozi'nska
Alek Andreev
Paul Kishan Rubenstein
Ruoxin Sang
Dan Hurt
Gamaleldin Elsayed
Renshen Wang
Dave Lacey
Anastasija Ili'c
Yao Zhao
Woohyun Han
Lora Aroyo
Chimezie Iwuanyanwu
Vitaly Nikolaev
Balaji Lakshminarayanan
Sadegh Jazayeri
Raphael Lopez Kaufman
Mani Varadarajan
Chetan Tekur
Doug Fritz
Misha Khalman
David Reitter
Kingshuk Dasgupta
Shourya Sarcar
Tina Ornduff
Javier Snaider
Fantine Huot
Johnson Jia
Rupert Kemp
Nejc Trdin
Anitha Vijayakumar
Lucy Kim
Christof Angermueller
Li Lao
Tianqi Liu
Haibin Zhang
David Engel
Somer Greene
Anais White
Jessica Austin
Lilly Taylor
Shereen Ashraf
Dangyi Liu
Maria Georgaki
Irene Cai
Yana Kulizhskaya
Sonam Goenka
Brennan Saeta
Kiran N. Vodrahalli
Christian Frank
D. Cesare
Brona Robenek
Harry Richardson
Mahmoud Alnahlawi
Christopher Yew
Priya Ponnapalli
Marco Tagliasacchi
Alex Korchemniy
Yelin Kim
Dinghua Li
Bill Rosgen
Kyle Levin
Jeremy Wiesner
Praseem Banzal
Praveen Srinivasan
Hongkun Yu
cCauglar Unlu
David Reid
Zora Tung
Daniel Finchelstein
Ravin Kumar
Andre Elisseeff
Jin Huang
Ming Zhang
Rui Zhu
Ricardo Aguilar
Mai Gim'enez
Jiawei Xia
Olivier Dousse
W. Gierke
S. Yeganeh
Damion Yates
Komal Jalan
Lu Li
Eri Latorre-Chimoto
Duc Dung Nguyen
Ken Durden
Praveen Kallakuri
Yaxin Liu
Matthew Johnson
Tomy Tsai
Alice Talbert
Jasmine Liu
Alexander Neitz
Chen Elkind
Marco Selvi
Mimi Jasarevic
Livio Baldini Soares
Albert Cui
Pidong Wang
Alek Wenjiao Wang
Xinyu Ye
Krystal Kallarackal
Lucia Loher
Hoi Lam
Josef Broder
D. Holtmann-Rice
Nina Martin
Bramandia Ramadhana
Daniel Toyama
Mrinal Shukla
Sujoy Basu
Abhi Mohan
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (see more) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
2024-03-08
ArXiv (preprint)
doi.org
arxiv.org