Publications

TACTiS: Transformer-Attentional Copulas for Time Series

Étienne Marcotte

The estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. However, t… (voir plus)he practical utility of such estimates is limited by how accurately they quantify predictive uncertainty. In this work, we address the problem of estimating the joint predictive distribution of high-dimensional multivariate time series. We propose a versatile method, based on the transformer architecture, that estimates joint distributions using an attention-based decoder that provably learns to mimic the properties of non-parametric copulas. The resulting model has several desirable properties: it can scale to hundreds of time series, supports both forecasting and interpolation, can handle unaligned and non-uniformly sampled data, and can seamlessly adapt to missing data during training. We demonstrate these properties empirically and show that our model produces state-of-the-art predictions on multiple real-world datasets.

2022-01-01

ICML (publié)

proceedings.mlr.press

arxiv.org

Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline

Massimo Caccia

Jonas Mueller

Taesup Kim

Laurent Charlin

Rasool Fakoor

We study task-agnostic continual reinforcement learning (TACRL) in which standard RL challenges are compounded with partial observability st… (voir plus)emming from task agnosticism, as well as additional difﬁculties of continual learning (CL), i.e., learning on a non-stationary sequence of tasks. Here we compare TACRL methods with their soft upper bounds prescribed by previous literature: multi-task learning (MTL) methods which do not have to deal with non-stationary data distributions, as well as task-aware methods, which are allowed to operate under full observability . We consider a previously unexplored and straightforward baseline for TACRL, replay-based recurrent RL (3RL), in which we augment an RL algorithm with recurrent mechanisms to address partial observability and experience replay mechanisms to address catastrophic forgetting in CL. Studying empirical performance in a sequence of RL tasks, we ﬁnd surprising occurrences of 3RL matching and overcoming the MTL and task-aware soft upper bounds. We lay out hypotheses that could explain this inﬂection point of continual and task-agnostic learning research. Our hypotheses are empirically tested in continuous control tasks via a large-scale study of the popular multi-task and continual learning benchmark Meta-World. By analyzing different training statistics including gradient conﬂict, we ﬁnd evidence that 3RL’s outperformance stems from its ability to quickly infer how new tasks relate with the previous ones, enabling forward transfer.

2022-01-01

arXiv.org (prépublication)

doi.org

On the benefits of representation regularization in invariance based domain generalization

Changjian Shui

Boyu Wang

Christian Gagné

2022-01-01

Machine Learning (publié)

doi.org

arxiv.org

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon

Lucile Saulnier

Thomas Wang

Christopher Akiki

Albert Villanova del Moral

Teven Le Scao

Leandro Von Werra

Chenghao Mou

Eduardo González Ponferrada

Huu Nguyen

Jörg Frohberg

Mario Šaško

Quentin Lhoest

Angelina McMillan-Major

Gérard Dupont

Stella Biderman

Anna Rogers

Loubna Ben allal

Francesco De Toni

Giada Pistilli … (voir 34 de plus)

Olivier Nguyen

Somaieh Nikpoor

Maraim Masoud

Pierre Colombo

Javier de la Rosa

Paulo Villegas

Tristan Thrush

Shayne Longpre

Sebastian Nagel

Leon Weber

Manuel Romero Muñoz

Jian Zhu

Daniel Van Strien

Zaid Alyafeai

Khalid Almubarak

Vu Minh Chien

Itziar Gonzalez-Dios

Aitor Soroa

Kyle Lo

Manan Dey

Pedro Ortiz Suarez

Aaron Gokaslan

Shamik Bose

David Ifeoluwa Adelani

Long Phan

Hieu Tran

Ian Yu

Suhas Pai

Jenny Chim

Violette Lepercq

Suzana Ilic

Margaret Mitchell

Sasha Luccioni

Yacine Jernite

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multili… (voir plus)ngual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

openreview.net

On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging

Chris Junchi Li

Yaodong Yu

Nicolas Loizou

Gauthier Gidel

Yi Ma

Nicolas Le Roux

Michael I. Jordan

We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) meth… (voir plus)od with constant step size, and presenting variations of the method that yield favorable convergence. In sharp contrasts with the basic SEG method whose last iterate only contracts to a fixed neighborhood of the Nash equilibrium, SEG augmented with iteration averaging provably converges to the Nash equilibrium under the same standard settings, and such a rate is further improved by incorporating a scheduled restarting procedure. In the interpolation setting where noise vanishes at the Nash equilibrium, we achieve an optimal convergence rate up to tight constants. We present numerical experiments that validate our theoretical findings and demonstrate the effectiveness of the SEG method when equipped with iteration averaging and restarting.

2022-01-01

AISTATS (publié)

arxiv.org

The Curious Case of Absolute Position Embeddings

Koustuv Sinha

Amirhossein Kazemnejad

Siva Reddy

Joelle Pineau

Dieuwke Hupkes

Adina Williams

2022-01-01

EMNLP (Findings) (publié)

doi.org

arxiv.org

On the Performance Implications of Deploying IoT Apps as FaaS

M. Aly

Foutse Khomh

Soumaya Yacout

2022-01-01

Communications in Computer and Information Science (published)

doi.org

The Secret to Better AI and Better Software (Is Requirements Engineering)

Nelly Bencomo

Jin Guo

Rachel. Harrison

Hans-Martin Heyn

Tim J Menzies

Recently, practitioners and researchers met to discuss the role of requirements, and AI and SE. We offer here notes on that fascinating disc… (voir plus)ussion. Also, have you considered writing for this column? This “SE for AI” column publishes commentaries on the growing field of SE for AI. Submissions are welcomed and encouraged (1,000–2,400 words, each figure and table counts as 250 words, try to use fewer than 12 references, and keep the discussion practitioner focused). Please submit your ideas to me at timm@ieee.org.—Tim Menzies

2022-01-01

IEEE Software (published)

doi.org

The Secret to Better AI and Better Software (Is Requirements Engineering)

Nelly Bencomo

Jin Guo

Rachel Harrison

Hans-Martin Heyn

Tim Menzies

Recently, practitioners and researchers met to discuss the role of requirements, and AI and SE. We offer here notes on that fascinating disc… (voir plus)ussion. Also, have you considered writing for this column? This “SE for AI” column publishes commentaries on the growing field of SE for AI. Submissions are welcomed and encouraged (1,000–2,400 words, each figure and table counts as 250 words, try to use fewer than 12 references, and keep the discussion practitioner focused). Please submit your ideas to me at timm@ieee.org.—Tim Menzies

2022-01-01

IEEE Software (publié)

doi.org

Towards Better Evaluation for Dynamic Link Prediction

Farimah Poursafaei

Andy Huang

Shenyang Huang

Kellin Pelrine

Reihaneh Rabbany

Despite the prevalence of recent success in learning from static graphs, learning from time-evolving graphs remains an open challenge. In th… (voir plus)is work, we design new, more stringent evaluation procedures for link prediction specific to dynamic graphs, which reflect real-world considerations, to better compare the strengths and weaknesses of methods. First, we create two visualization techniques to understand the reoccurring patterns of edges over time and show that many edges reoccur at later time steps. Based on this observation, we propose a pure memorization-based baseline called EdgeBank. EdgeBank achieves surprisingly strong performance across multiple settings which highlights that the negative edges used in the current evaluation are easy. To sample more challenging negative edges, we introduce two novel negative sampling strategies that improve robustness and better match real-world applications. Lastly, we introduce six new dynamic graph datasets from a diverse set of domains missing from current benchmarks, providing new challenges and opportunities for future research. Our code repository is accessible at https://github.com/fpour/DGB.git.

openreview.net

Towards Painless Policy Optimization for Constrained MDPs

Arushi Jain

Sharan Vaswani

Reza Babanezhad Harikandeh

Csaba Szepesvari

Doina Precup

We study policy optimization in an infinite horizon, …

2022-01-01

UAI (publié)

doi.org

openreview.net