Publications

Stochastic gradient descent-based inference for dynamic network models with attractors

Hancong Pan

Xiaojing Zhu

Cantay Caliskan

Dino P. Christenson

Konstantinos Spiliopoulos

Dylan Walker

Eric Kolaczyk

2024-03-11

ArXiv (preprint)

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

Siva Reddy

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve… (see more) real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

2024-03-11

ICLR.cc/2024/Workshop/LLMAgents (poster)

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lu

Zdeněk Kasner

Siva Reddy

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve… (see more) real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WebLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx.

2024-03-11

ICLR.cc/2024/Workshop/LLMAgents (poster)

WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks?

Alexandre Drouin

Maxime Gasse

Massimo Caccia

Issam Hadj Laradji

Manuel Del Verme

Tom Marty

Léo Boisvert

Megh Thakkar

Quentin Cappart

David Vazquez

Nicolas Chapados

Alexandre Lacoste

We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuri… (see more)ng the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 29 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field.

2024-03-11

ICLR.cc/2024/Workshop/LLMAgents (poster)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Machel Reid

Nikolay Savinov

Denis Teplyashin

Dmitry Lepikhin

Timothy P. Lillicrap

Jean-Baptiste Alayrac

Radu Soricut

Angeliki Lazaridou

Orhan Firat

Julian Schrittwieser

Ioannis Antonoglou

Rohan Anil

Sebastian Borgeaud

Andrew M. Dai

Katie Millican

Ethan Dyer

Mia Glaese

Thibault Sottiaux

Benjamin Lee

Fabio Viola … (see 480 more)

Malcolm Reynolds

Yuanzhong Xu

James L. Molloy

Jilin Chen

Michael Acheson Isard

Paul R. Barham

Tom Hennigan

Ross McIlroy

Melvin Johnson

J. Schalkwyk

Eli Collins

Eliza Rutherford

Erica Moreira

Kareem W. Ayoub

Megha Goel

Clemens Meyer

Gregory Thornton

Zhen Yang

Henryk Michalewski

Zaheer Abbas

Nathan Schucher

Ankesh Anand

Richard Ives

James Keeling

Karel Lenc

Salem Haykal

Siamak Shakeri

Pranav Shyam

Aakanksha Chowdhery

Roman Ring

Stephen Spencer

Eren Sezener

Luke Vilnis

Oscar Chang

Nobuyuki Morioka

George Tucker

Ce Zheng

Oliver Woodman

Nithya Attaluri

Tomas Kocisky

Evgenii Eltyshev

Xi Chen

Timothy Chung

Vittorio Selo

Siddhartha Brahma

Petko Georgiev

Ambrose Slone

Zhenkai Zhu

James Lottes

Siyuan Qiao

Ben Caine

Sebastian Riedel

Alex Tomala

Martin J. Chadwick

J Christopher Love

Peter Choy

Sid Mittal

Neil Houlsby

Yunhao Tang

Matthew Lamm

Libin Bai

Qiao Zhang

Luheng He

Yong Cheng

Peter Conway Humphreys

Yujia Li

Sergey Brin

Albin Cassirer

Ying-Qi Miao

Lukáš Žilka

Taylor Tobin

Kelvin Xu

Lev Proleev

Daniel Sohn

Alberto Magni

Lisa Anne Hendricks

Isabel Gao

Santiago Ontan'on

Oskar Bunyan

Nathan Byrd

Abhanshu Sharma

Biao Zhang

Mario Pinto

Rishika Sinha

Harsh Mehta

Dawei Jia

Sergi Caelles

Albert Webson

Alex Morris

Becca Roelofs

Yifan Ding

Robin Strudel

Xuehan Xiong

Marvin Ritter

Mostafa Dehghani

Rahma Chaabouni

Abhijit Karmarkar

Guangda Lai

Fabian Mentzer

Bibo Xu

YaGuang Li

Yujing Zhang

T. Paine

Alex Goldin

Behnam Neyshabur

Kate Baumli

Anselm C. Levskaya

Michael Laskin

Wenhao Jia

Jack W. Rae

Kefan Xiao

Antoine He

Skye Giordano

Lakshman N. Yagati

Jean-Baptiste Lespiau

Paul Natsev

Sanjay Ganapathy

Fangyu Liu

Danilo Martins

Nanxin Chen

Yunhan Xu

Megan Barnes

Rhys May

Arpi Vezer

Junhyuk Oh

Ken Franko

Sophie Bridgers

Ruizhe Zhao

Boxi Wu

Basil Mustafa

Sean Sechrist

Emilio Parisotto

Thanumalayan Sankaranarayana Pillai

Chris Larkin

Chenjie Gu

Christina Sorokin

M. Krikun

Alexey Guseynov

Jessica Landon

Romina Datta

Alexander Pritzel

Phoebe Thacker

Fan Yang

Kevin Hui

A.E. Hauth

Chih-Kuan Yeh

David Barker

Justin Mao-jones

Sophia Austin

Hannah Rachel Sheahan

Parker Schuh

James Svensson

Rohan Jain

Vinay Venkatesh Ramasesh

Anton Briukhov

Da-Woon Chung

Tamara von Glehn

Christina Butterfield

Priya Jhakra

Matt Wiethoff

Justin Frye

Jordan Grimstad

Beer Changpinyo

Charline Le Lan

Anna Bortsova

Yonghui Wu

Paul Voigtlaender

Tara N. Sainath

Charlotte Smith

Will Hawkins

Kris Cao

James Besley

Srivatsan Srinivasan

Mark Omernick

Colin Gaffney

Gabriela Surita

Ryan Burnell

Bogdan Damoc

Junwhan Ahn

Andrew Brock

Mantas Pajarskas

Anastasia Petrushkina

Seb Noury

Lorenzo Blanco

Kevin Swersky

Arun Ahuja

Thi Avrahami

Vedant Misra

Raoul de Liedekerke

Mariko Iinuma

Alex Polozov

Sarah York

George van den Driessche

Paul Michel

Justin Chiu

Rory Blevins

Zach Gleicher

Adria Recasens

Alban Rrustemi

Elena Gribovskaya

Aurko Roy

Wiktor Gworek

S'ebastien M. R. Arnold

Lisa Lee

James Lee-Thorp

Marcello Maggioni

Enrique Piqueras

Kartikeya Badola

Sharad Mandyam Vikram

Lucas Gonzalez

Anirudh Baddepudi

Evan Senter

Jacob Devlin

James Qin

Michael Azzam

Maja Trebacz

M. Polacek

Kashyap Krishnakumar

Shuo-yiin Chang

Matthew Tung

Ivo Penchev

Rishabh Joshi

Kate Olszewska

Carrie Muir

Mateo Wirth

Ale Jakse Hartman

Joshua Newlan

Sheleem Kashem

Vijay Bolina

Elahe Dabir

Joost Van Amersfoort

Zafarali Ahmed

James Cobon-Kerr

Aishwarya B Kamath

Arnar Mar Hrafnkelsson

Le Hou

Ian Mackinnon

Alexandre Fréchette

Eric Noland

Xiance Si

Emanuel Taropa

Dong Li

Phil Crone

Anmol Gulati

S'ebastien Cevey

Jonas Adler

Ada Ma

David Silver

Simon Tokumine

Richard Powell

Stephan Lee

Samer Hassan

Diana Mincu

Antoine Yang

Nir Levine

Jenny Brennan

Mingqiu Wang

Sarah Hodkinson

Jeffrey Zhao

Josh Lipschultz

Aedan Pope

Michael B. Chang

Cheng Li

Laurent El Shafey

Michela Paganini

Sholto Douglas

Bernd Bohnet

Fabio Pardo

Seth Odoom

Mihaela Rosca

Cicero Nogueira dos Santos

Kedar Soparkar

Arthur Guez

Tom Hudson

Steven Hansen

Chulayuth Asawaroengchai

Ravichandra Addanki

Tianhe Yu

Wojciech Stokowiec

Mina Khan

Justin Gilmer

Jaehoon Lee

Carrie Grimes Bostock

Keran Rong

Jonathan Caton

Pedram Pejman

Filip Pavetic

Geoff Brown

Vivek Sharma

Mario Luvci'c

Rajkumar Samuel

Josip Djolonga

Amol Mandhane

Lars Lowe Sjosund

Elena Buchatskaya

Elspeth White

Natalie Clay

Jiepu Jiang

Hyeontaek Lim

Ross Hemsley

Jane Labanowski

Nicola De Cao

David Steiner

Sayed Hadi Hashemi

Jacob Austin

Anita Gergely

Tim Blyth

Joe Stanton

Kaushik Shivakumar

Aditya Siddhant

Anders Johan Andreassen

Carlos L. Araya

Nikhil Sethi

Rakesh Shivanna

Steven Hand

Ankur Bapna

A. Khodaei

Antoine Miech

Garrett Tanzer

Andy Swing

Shantanu Thakoor

Zhufeng Pan

Zachary Nado

Stephanie Winkler

Dian Yu

Mohammad Saleh

Lorenzo Maggiore

Iain Barr

Minh Giang

Thais Kagohara

Ivo Danihelka

Amit Marathe

Vladimir Feinberg

Mohamed Elhawaty

Nimesh Ghelani

Dan Horgan

Helen Miller

Lexi Walker

Richard Tanburn

Mukarram Tariq

Disha Shrivastava

Fei Xia

Chung-Cheng Chiu

Zoe C. Ashwood

Khuslen Baatarsukh

Sina Samangooei

Fred Alcober

Axel Stjerngren

Paul Komarek

Katerina Tsihlas

Anudhyan Boral

Ramona Comanescu

Jeremy Chen

Ruibo Liu

Dawn Bloxwich

Charlie Chen

Yanhua Sun

Fangxiaoyu Feng

Matthew Mauger

Xerxes Dotiwalla

Vincent Hellendoorn

Michael Sharman

Ivy Zheng

Krishna S Haridasan

Gabriel Barth-Maron

Craig Swanson

Dominika Rogozi'nska

Alek Andreev

Paul Kishan Rubenstein

Ruoxin Sang

Dan Hurt

Gamaleldin Elsayed

Renshen Wang

Dave Lacey

Anastasija Ili'c

Yao Zhao

Woohyun Han

Lora Aroyo

Chimezie Iwuanyanwu

Vitaly Nikolaev

Balaji Lakshminarayanan

Sadegh Jazayeri

Raphael Lopez Kaufman

Mani Varadarajan

Chetan Tekur

Doug Fritz

Misha Khalman

David Reitter

Kingshuk Dasgupta

Shourya Sarcar

Tina Ornduff

Javier Snaider

Fantine Huot

Johnson Jia

Rupert Kemp

Nejc Trdin

Anitha Vijayakumar

Lucy Kim

Christof Angermueller

Li Lao

Tianqi Liu

Haibin Zhang

David Engel

Somer Greene

Anais White

Jessica Austin

Lilly Taylor

Shereen Ashraf

Dangyi Liu

Maria Georgaki

Irene Cai

Yana Kulizhskaya

Sonam Goenka

Brennan Saeta

Kiran N. Vodrahalli

Christian Frank

D. Cesare

Brona Robenek

Harry Richardson

Mahmoud Alnahlawi

Christopher Yew

Priya Ponnapalli

Marco Tagliasacchi

Alex Korchemniy

Yelin Kim

Dinghua Li

Bill Rosgen

Kyle Levin

Jeremy Wiesner

Praseem Banzal

Praveen Srinivasan

Hongkun Yu

cCauglar Unlu

David Reid

Zora Tung

Daniel Finchelstein

Ravin Kumar

Andre Elisseeff

Jin Huang

Ming Zhang

Rui Zhu

Ricardo Aguilar

Mai Gim'enez

Jiawei Xia

Olivier Dousse

W. Gierke

S. Yeganeh

Damion Yates

Komal Jalan

Lu Li

Eri Latorre-Chimoto

Duc Dung Nguyen

Ken Durden

Praveen Kallakuri

Yaxin Liu

Matthew Johnson

Tomy Tsai

Alice Talbert

Jasmine Liu

Alexander Neitz

Chen Elkind

Marco Selvi

Mimi Jasarevic

Livio Baldini Soares

Albert Cui

Pidong Wang

Alek Wenjiao Wang

Xinyu Ye

Krystal Kallarackal

Lucia Loher

Hoi Lam

Josef Broder

D. Holtmann-Rice

Nina Martin

Bramandia Ramadhana

Daniel Toyama

Mrinal Shukla

Sujoy Basu

Abhi Mohan

Nicholas Fernando

2024-03-08

ArXiv (preprint)

Generative Models for Decision Making

Bogdan Mazoure

Lisa Lee

Roberta Raileanu

Yilun Du

Walter Talbott

Katherine Metcalf

(Rex) Devon Hjelm

Alexander T Toshev

Generative Artificial Intelligence (AI) has made significant advancements in recent years, particularly with the development of large langua… (see more)ge and diffusion models. These generative models have demonstrated impressive capabilities in various tasks, such as text generation and image and audio synthesis. Concurrently, Reinforcement Learning (RL) has made significant strides in solving complex sequential decision-making problems with the help of external knowledge sources . However, there remains untapped potential in combining generative models with RL algorithms to tackle real-world challenges, particularly to improve sample efficiency of tabula rasa training by introducing priors from related domains such as visual question-answering, image captioning and image generation. This workshop aims to bring together researchers and practitioners from the fields of generative AI and reinforcement learning to explore the latest advances, methodologies, and applications. By fostering collaborations between these two domains, we intend to unlock new opportunities for addressing complex problems that lie at the intersection of both fields.

2024-03-08

ICLR.cc/2024/Workshop_Proposals (published)

Global AI Cultures

Rida Qadri

Fernando Diaz

Arjun Subramonian

Sunipa Dev

Georgina Emma Born

Mary L. Gray

Jessica Quaye

Rachel Bergmann

2024-03-08

ICLR.cc/2024/Workshop_Proposals (published)

Integrating Generative and Experimental Platforms or Biomolecular Design

Cheng-Hao Liu

Jarrid Rector-Brooks

Jason Yim

Soojung Yang

Sidney Lisanza

Francesca-Zhoufan Li

Pranam Chatterjee

Tommi Jaakkola

Regina Barzilay

David Baker

Frances H. Arnold

Yoshua Bengio

2024-03-08

ICLR.cc/2024/Workshop_Proposals (published)

Tackling Climate Change with Machine Learning: Fostering the Maturity of ML Applications for Climate Change

Shiva Madadkhani

Olivia Mendivil Ramos

Millie Chapman

Jesse Dunietz

Arthur Ouaknine

David Rolnick

Yoshua Bengio

2024-03-08

ICLR.cc/2024/Workshop_Proposals (published)

Machine learning and information theory concepts towards an AI Mathematician

Yoshua Bengio

Nikolay Malkin

The current state-of-the-art in artificial intelligence is impressive, especially in terms of mastery of language, but not so much in terms … (see more)of mathematical reasoning. What could be missing? Can we learn something useful about that gap from how the brains of mathematicians go about their craft? This essay builds on the idea that current deep learning mostly succeeds at system 1 abilities -- which correspond to our intuition and habitual behaviors -- but still lacks something important regarding system 2 abilities -- which include reasoning and robust uncertainty estimation. It takes an information-theoretical posture to ask questions about what constitutes an interesting mathematical statement, which could guide future work in crafting an AI mathematician. The focus is not on proving a given theorem but on discovering new and interesting conjectures. The central hypothesis is that a desirable body of theorems better summarizes the set of all provable statements, for example by having a small description length while at the same time being close (in terms of number of derivation steps) to many provable statements.

2024-03-07

ArXiv (preprint)

Personalized Negative Reservoir for Incremental Learning in Recommender Systems

Antonios Valkanas

Yuening Wang

Yingxue Zhang

Mark Coates

2024-03-06

ArXiv (preprint)