Ankesh Anand

Federico Lebron

Alban Rrustemi

Natalie Clay

Phil Crone

Tomas Kocisky

Jeffrey Zhao

Bartek Perz

Dian Yu

Heidi Howard

Adam E. Bloniarz

Jack W. Rae

Han Lu

Laurent Sifre

Marcello Maggioni

Fred Alcober

Dan Garrette

Megan Barnes

Shantanu Thakoor

Jacob Austin

Gabriel Barth-Maron

William Wong

Rishabh Joshi

Rahma Chaabouni

Deeni Fatiha

Arun Ahuja

Ruibo Liu

Yunxuan Li

Sarah Cogan

Jeremy Chen

Chao Jia

Chenjie Gu

Qiao Zhang

Ale Jordan Grimstad

Jakse Hartman

Martin Chad-wick

Gaurav Singh Tomar

Xavier Garcia

Evan Senter

Emanuel Taropa

Thanumalayan Sankaranarayana Pillai

Jacob Devlin

Michael Laskin

Diego de

Las Casas

Dasha Valter

Connie Tao

Lorenzo Blanco

Adrià Puigdomènech Badia

David Reitter

Mianna Chen

Jenny Brennan

Clara E. Rivera

Sergey Brin

Shariq N Iqbal

Gabriela Surita

Jane Labanowski

Abhishek Rao

Stephanie Winkler

Emilio Parisotto

Yiming Gu

Kate Olszewska

Yujing Zhang

Ravi Ad-danki

Antoine Miech

Annie Louis

Laurent El

Denis Teplyashin

Geoff Brown

Elliot Catt

Nithya Attaluri

Jan Balaguer

Jackie Xiang

Pidong Wang

Zoe C. Ashwood

Anton Briukhov

Albert Webson

Sanjay Ganapathy

Smit Sanghavi

Ajay Kannan

Ming-Wei Chang

Axel Stjerngren

Josip Djolonga

Yuting Sun

Ankur Bapna

Matthew Aitchison

Pedram Pejman

Henryk Michalewski

Tianhe Yu

Cindy Wang

J Christopher Love

Junwhan Ahn

Dawn Bloxwich

Kehang Han

Peter Conway Humphreys

Thibault Sellam

James Bradbury

Varun Godbole

Sina Samangooei

Bogdan Damoc

Alex Kaskasoli

S'ebastien M. R. Arnold

Vijay Vasudevan

Shubham Agrawal

Jason Riesa

Dmitry Lepikhin

Richard Tan-burn

Srivatsan Srinivasan

Hyeontaek Lim

Sarah Hodkinson

Pranav Shyam

Johan Ferret

Steven Hand

Ankush Garg

T. Paine

Jian Li

Yu-jia Li

Minh Giang

Zaheer Abbas

Sarah York

Machel Reid

Elizabeth Cole

Aakanksha Chowdhery

Dipanjan Das

Dominika Rogozi´nska

Vitaly Nikolaev

Pablo G. Sprechmann

Zachary Nado

Lukáš Žilka

Flavien Prost

Luheng He

Marianne Monteiro

Gaurav Mishra

Christoper A. Welty

Joshua Newlan

Dawei Jia

Miltiadis Allamanis

Clara Huiyi Hu

Raoul de Liedekerke

Justin Gilmer

Carl Saroufim

Shruti Rijhwani

Shaobo Hou

Disha Shrivastava

Anirudh Baddepudi

Alex Goldin

Adnan Ozturel

Albin Cassirer

Yunhan Xu

Daniel Sohn

Deven-dra Sachan

Reinald Kim Amplayo

Craig Swan-son

Dessie Petrova

Shashi Narayan

Arthur Guez

Siddhartha Brahma

Jessica Landon

Miteyan Patel

Ruizhe Zhao

Kevin Villela

Luyu Wang

Wenhao Jia

Matthew Rahtz

Mai Giménez

Legg Yeung

Hanzhao Lin

James Keeling

Petko Georgiev

Diana Mincu

Boxi Wu

Salem Haykal

Rachel Sapu-tro

Kiran N. Vodrahalli

James Qin

Zeynep Cankara

Abhanshu Sharma

Nicholas Fernando

Will Hawkins

Behnam Neyshabur

Solomon Kim

Adrian Hut-ter

Priyanka Agrawal

Alex Castro-Ros

George van den Driessche

Tao Wang

Fan Yang

Shuo yiin

Paul Chang

Ross Komarek

Mario McIlroy

Luˇci´c Guodong

Wael Zhang

Michael Farhan

Sharman Paul

Paul Natsev

Yong Michel

Yamini Cheng

Siyuan Bansal

Kris Qiao

Siamak Cao

Shakeri Christina

Justin Butterfield

Paul Kishan Chung

Shivani Rubenstein

Arthur Agrawal

Kedar Mensch

Karel Soparkar

Timothy Lenc

Aedan Chung

Pope Loren

Jackie Maggiore

Priya Kay

Shibo Jhakra

Joshua Wang

Mary Maynez

Taylor Phuong

Tobin Andrea

Maja Tacchetti

Kevin Trebacz

Robinson Yash

Sebas-tian Katariya

Paige Riedel

Ke-fan Bailey

Nimesh Xiao

Lora Ghelani

Ambrose Aroyo

Neil Slone

Xuehan Houlsby

Zhen Xiong

Yang Elena

Jonas Gribovskaya

Mateo Adler

Lisa Wirth

M. Lee

Jay Kagohara

So-phie Pavagadhi

Anna Bridgers

Sanjay Bortsova

Ghemawat Zafarali

Tianqi Ahmed

Richard Liu

Vijay Powell

Mariko Bolina

Polina Iinuma

James Zablotskaia

Da-Woon Besley

Timothy Chung

Ramona Dozat

Xi-ance Comanescu

Jeremy Si

Guolong Greer

Su Martin

Raphaël Lopez Polacek

Simon Kaufman

Hex-iang Tokumine

Elena Hu

Yingjie Buchatskaya

Mo-hamed Miao

Aditya Elhawaty

Nenad Siddhant

Jinwei Tomasev

Christina Xing

Helen Greer

Miller Shereen

Aurko Ashraf

Zizhao Roy

Ada Zhang

Mathew Angelos

Milos Filos

Rory Besta

Ted Blevins

Kli-5927 Chih-Kuan

Soravit Yeh

Jiaqi Changpinyo

Oscar Mu

Mantas Chang

Carrie Pajarskas

Muir Vered

Charline Cohen

Krishna Le Lan

Haridasan Amit

Steven Marathe

Sholto Hansen

Ra-jkumar Douglas

Mingqiu Samuel

Sophia Wang

Austin Chang

Jiepu Lan

Justin Jiang

Jaime Alonso Chiu

Lars Lowe Lorenzo

Sébastien Sjösund

Cevey Zach

Thi Gleicher

Anudhyan Avrahami

Boral Hansa

Vittorio Srinivasan

Rhys Selo

Kon-stantinos May

Léonard Aisopos

Hussenot

Livio Baldini

Kate Soares

Michael B Baumli

Adrià Chang

Ben Caine

Alexander Pritzel

Filip Pavetic

Fabio Pardo

Anita Gergely

Justin Frye

Dan Horgan

Nora Kassner

Subhrajit Roy

Víctor Ethan Dyer

Alex Tomala

Yunhao Tang

Dalia El Badawy

Elspeth White

Basil Mustafa

Oran Lang

Abhishek Jindal

Sharad Mandyam Vikram

Zhitao Gong

Sergi Caelles

Ross Hemsley

Gregory Thornton

Fangxiaoyu Feng

Wojciech Stokowiec

Ce Zheng

Phoebe Thacker

Ça˘glar Ünlü

Zhishuai Zhang

Mohammad Saleh

James Svensson

Maxwell Bileschi

Piyush Patil

Roman Ring

Katerina Tsihlas

Arpi Vezer

Marco Selvi

Toby Shevlane

Mikel Ro-driguez

Tom Kwiatkowski

Samira Daruki

Keran Rong

Allan Dafoe

Nicholas Fitzgerald

Keren Gu-Lemberg

Mina Khan

Lisa Anne Hendricks

Marie Pellat

Vladimir Feinberg

James Cobon-Kerr

Tara N. Sainath

Maribeth Rauh

Sayed Hadi

Richard Hashemi

Yana Ives

YaGuang Hasson

Eric Li

Yuan Noland

Nathan Cao

Le Byrd

Qingze Hou

Thibault Wang

Michela Sottiaux

Paganini Jean-Baptiste

Alexandre Lespiau

Samer Moufarek

Kaushik Hassan

Joost Shivakumar

Amol van Amers-foort

Pratik Mandhane

Anirudh Joshi

Matthew Goyal

Andrew Tung

Hannah Brock

Vedant Shea-han

Cheng Misra

Nemanja Li

Raki´cevi´c Mostafa

Fangyu Dehghani

Sid Liu

Junhyuk Mittal

Seb Oh

Eren Noury

Fantine Sezener

Matthew Huot

Nicola Lamm

Charlie De Cao

Gamaleldin Chen

Ed Elsayed

Mahdis Chi

Ian Mahdieh

Nan Tenney

Ivan Hua

Patrick Petrychenko

Dylan Kane

Rishub Scand-inaro

Jonathan Jain

Romina Uesato

Datta Adam

Oskar Sadovsky

Dominik Bunyan

Rabiej Shimu

,

2024-11-01

Findings of the Association for Computational Linguistics: EMNLP 2024 (publié)

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Stephanie C.Y. Chan

Luis Rosias

Biao Zhang

Zaheer Abbas

Azade Nova

John D Co-Reyes

Eric Chu

Feryal Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples – the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated outputs. To mitigate this limitation, we explore two new settings: (1) "Reinforced ICL" that uses model-generated chain-of-thought rationales in place of human rationales, and (2) "Unsupervised ICL" where we remove rationales from the prompt altogether, and prompts the model only with domain-specific inputs. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. We demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to supervised fine-tuning. Finally, we reveal the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-09-25

NeurIPS.cc/2024/Conference (spotlight)

openreview.net

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Luis Rosias

Stephanie C.Y. Chan

Zaheer Abbas

Biao Zhang

Azade Nova

John D. Co-Reyes

Eric Chu

Feryal M. P. Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases and can learn high-dimensional functions with numerical inputs. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-06-18

ICML.cc/2024/Workshop/ICL (poster)

openreview.net

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Stephanie C.Y. Chan

Zaheer Abbas

Azade Nova

John D Co-Reyes

Eric Chu

Feryal Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-04-17

ArXiv (prépublication)

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Stephanie C.Y. Chan

Zaheer Abbas

Azade Nova

John D Co-Reyes

Eric Chu

Feryal Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-04-17

ArXiv (prépublication)

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Stephanie C.Y. Chan

Biao Zhang

Zaheer Abbas

Azade Nova

John D Co-Reyes

Eric Chu

Feryal Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-04-17

ArXiv (prépublication)

Many-Shot In-Context Learning

Avi Singh

Lei M Zhang

Bernd Bohnet

Stephanie C.Y. Chan

Zaheer Abbas

Azade Nova

John D Co-Reyes

Eric Chu

Feryal Behbahani

Aleksandra Faust

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, w… (voir plus)ithout any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

2024-04-17

ArXiv (prépublication)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

Xi Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Albert Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Laksh-man Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Albert Cui

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (voir plus) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

2024-03-08

ArXiv (prépublication)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

Xi Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Albert Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Laksh-man Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Albert Cui

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (voir plus) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

2024-03-08

ArXiv (prépublication)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

Xi Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Albert Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Laksh-man Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Albert Cui

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (voir plus) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

2024-03-08

ArXiv (prépublication)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Fabio Viola … (voir 478 de plus)

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

X. T. Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Alex Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Laksh-man Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Lu Liu

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (voir plus) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

2024-03-08

ArXiv (prépublication)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

Xi Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Albert Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Laksh-man Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Albert Cui

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models… (voir plus) capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

2024-03-08

ArXiv (prépublication)