Publications

On the Expressivity of Markov Reward (Extended Abstract)

David Abel

Will Dabney

Anna Harutyunyan

Mark K. Ho

Michael L. Littman

Doina Precup

Satinder Singh

2022-07-22

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (published)

doi.org

Flaky Performances when Pre-Training on Relational Databases with a Plan for Future Characterization Efforts

Shengchao Liu

David Vázquez

Jian Tang

Pierre-Andre Noel

We explore the downstream task performances for graph neural network (GNN) self-supervised learning (SSL) methods trained on subgraphs extra… (see more)cted from relational databases (RDBs). Intu-itively, this joint use of SSL and GNNs allows us to leverage more of the available data, which could translate to better results. However, while we observe positive transfer in some cases, others showed systematic performance degradation, including some spectacular ones. We hypothesize a mechanism that could explain this behaviour and draft the plan for future work testing it by characterizing how much relevant information different strategies can (theoretically and/or empirically) extract from (synthetic and/or real) RDBs.

2022-07-21

ICML.cc/2022/Workshop/Pre-Training (unknown)

openreview.net

Revisiting Hotels-50K and Hotel-ID

Aarash Feizi

Arantxa Casanova

Adriana Romero

Reihaneh Rabbany

In this paper, we propose revisited versions for two recent hotel recognition datasets: Hotels-50K and Hotel-ID. The revisited versions prov… (see more)ide evaluation setups with different levels of difﬁculty to better align with the intended real-world application, i.e. countering human trafﬁcking. Real-world scenarios involve hotels and locations that are not captured in the current data sets, therefore it is important to consider evaluation settings where classes are truly unseen. We test this setup using multiple state-of-the-art image retrieval models and show that as expected, the models’ performances decrease as the evaluation gets closer to the real-world unseen settings. The rankings of the best performing models also change across the different evaluation settings, which further motivates using the proposed revisited datasets.

2022-07-19

ArXiv (preprint)

doi.org

arxiv.org

On the Generalization and Adaption Performance of Causal Models

Nan Rosemary Ke

2022-07-19

ICML.cc/2022/Workshop/SCIS (poster)

doi.org

openreview.net

The Machine Learning for Combinatorial Optimization Competition (ML4CO): Results and Insights

Maxime Gasse

Quentin Cappart

Jonas Charfreitag

Laurent Charlin

Didier Chételat

Antonia Chmiela

Justin Dumouchelle

Ambros Gleixner

Aleksandr M. Kazachkov

Elias Khalil

Pawel Lichocki

Andrea Lodi

Miles Lubin

Chris J. Maddison

Christopher Morris

Dimitri J. Papageorgiou

Augustin Parjadis

Sebastian Pokutta

Antoine Prouvost

Lara Scavuzzo … (see 21 more)

Giulia Zarpellon

Linxin Yang

Sha Lai

Akang Wang

Xiaodong Luo

Xiang Zhou

Haohan Huang

Shengcheng Shao

Yuanming Zhu

Dong Zhang

Tao Quan

Zixuan Cao

Yang Xu

Zhewei Huang

Shuchang Zhou

Chen Binbin

He Minggui

Hao Hao

Zhang Zhiyu

An Zhiwu

Mao Kun

Combinatorial optimization is a well-established area in operations research and computer science. Until recently, its methods have focused … (see more)on solving problem instances in isolation, ignoring that they often stem from related data distributions in practice. However, recent years have seen a surge of interest in using machine learning as a new approach for solving combinatorial problems, either directly as solvers or by enhancing exact solvers. Based on this context, the ML4CO aims at improving state-of-the-art combinatorial optimization solvers by replacing key heuristic components. The competition featured three challenging tasks: finding the best feasible solution, producing the tightest optimality certificate, and giving an appropriate solver configuration. Three realistic datasets were considered: balanced item placement, workload apportionment, and maritime inventory routing. This last dataset was kept anonymous for the contestants.

2022-07-19

Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track (published)

doi.org

proceedings.mlr.press

AnyMorph: Learning Transferable Polices By Inferring Agent Morphology

Brandon Trabucco

Mariano Phielipp

Glen Berseth

The prototypical approach to reinforcement learning involves training policies tailored to a particular agent from scratch for every new mor… (see more)phology. Recent work aims to eliminate the re-training of policies by investigating whether a morphology-agnostic policy, trained on a diverse set of agents with similar task objectives, can be transferred to new agents with unseen morphologies without re-training. This is a challenging problem that required previous approaches to use hand-designed descriptions of the new agent's morphology. Instead of hand-designing this description, we propose a data-driven method that learns a representation of morphology directly from the reinforcement learning objective. Ours is the first reinforcement learning algorithm that can train a policy to generalize to new agent morphologies without requiring a description of the agent's morphology in advance. We evaluate our approach on the standard benchmark for agent-agnostic control, and improve over the current state of the art in zero-shot generalization to new agents. Importantly, our method attains good performance without an explicit description of morphology.

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

doi.org

proceedings.mlr.press

Biological Sequence Design with GFlowNets

Moksh Jain

Emmanuel Bengio

Alex-Hernandez Garcia

Jarrid Rector-Brooks

Bonaventure F. P. Dossou

Chanakya Ekbote

Michael Kilgour

Payel Das

Design of de novo biological sequences with desired properties, like protein and DNA sequences, often involves an active loop with several r… (see more)ounds of molecule ideation and expensive wet-lab evaluations. These experiments can consist of multiple stages, with increasing levels of precision and cost of evaluation, where candidates are filtered. This makes the diversity of proposed candidates a key consideration in the ideation phase. In this work, we propose an active learning algorithm leveraging epistemic uncertainty estimation and the recently proposed GFlowNets as a generator of diverse candidate solutions, with the objective to obtain a diverse batch of useful (as defined by some utility function, for example, the predicted anti-microbial activity of a peptide) and informative candidates after each round. We also propose a scheme to incorporate existing labeled datasets of candidates, in addition to a reward function, to speed up learning in GFlowNets. We present empirical results on several biological sequence design tasks, and we find that our method generates more diverse and novel batches with high scoring candidates compared to existing approaches.

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

doi.org

proceedings.mlr.press

Building Robust Ensembles via Margin Boosting

Dinghuai Zhang

Hongyang Zhang

Aaron Courville

Yoshua Bengio

Pradeep Ravikumar

Arun Sai Suggala

In the context of adversarial robustness, a single model does not usually have enough power to defend against all possible adversarial attac… (see more)ks, and as a result, has sub-optimal robustness. Consequently, an emerging line of work has focused on learning an ensemble of neural networks to defend against adversarial attacks. In this work, we take a principled approach towards building robust ensembles. We view this problem from the perspective of margin-boosting and develop an algorithm for learning an ensemble with maximum margin. Through extensive empirical evaluation on benchmark datasets, we show that our algorithm not only outperforms existing ensembling techniques, but also large models trained in an end-to-end fashion. An important byproduct of our work is a margin-maximizing cross-entropy (MCE) loss, which is a better alternative to the standard cross-entropy (CE) loss. Empirically, we show that replacing the CE loss in state-of-the-art adversarial training techniques with our MCE loss leads to significant performance improvement.

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

doi.org

proceedings.mlr.press

Characterizing User Behaviors in Open-Source Software User Forums: An Empirical Study

Jazlyn Hellman

Jiahao Chen

Md. Sami Uddin

Jinghui Cheng

Jin L.C. Guo

User forums of Open Source Software (OSS) enable end-users to collaboratively discuss problems concerning the OSS applications. Despite deca… (see more)des of research on OSS, we know very little about how end-users engage with OSS communities on these forums, in particular, the challenges that hinder their continuous and meaningful participation in the OSS community. Many previous works are developer-centric and overlook the importance of end-user forums. As a result, end-users' expectations are seldom reflected in OSS development. To better understand user behaviors in OSS user forums, we carried out an empirical study analyzing about 1.3 million posts from user forums of four popular OSS applications: Zotero, Audacity, VLC, and RStudio. Through analyzing the contribution patterns of three common user types (end-users, developers, and organizers), we observed that end-users not only initiated most of the threads (above 96% of threads in three projects, 86% in the other), but also acted as the significant contributors for responding to other users' posts, even though they tended to lack confidence in their activities as indicated by psycho-linguistic analyses. Moreover, we found end-users more open, reflecting a more positive emotion in communication than organizers and developers in the forums. Our work contributes new knowledge about end-users' activities and behaviors in OSS user forums that the vital OSS stakeholders can leverage to improve end-user engagement in the OSS development process.

2022-07-18

Proceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering (published)

doi.org

arxiv.org

Direct Behavior Specification via Constrained Reinforcement Learning

Christopher Pal

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most oft… (see more)en, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

doi.org

proceedings.mlr.press

Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning

Harley Wiltzer

David Meger

Bellemare Marc-Emmanuel

Continuous-time reinforcement learning offers an appealing formalism for describing control problems in which the passage of time is not nat… (see more)urally divided into discrete increments. Here we consider the problem of predicting the distribution of returns obtained by an agent interacting in a continuous-time, stochastic environment. Accurate return predictions have proven useful for determining optimal policies for risk-sensitive control, learning state representations, multiagent coordination, and more. We begin by establishing the distributional analogue of the Hamilton-Jacobi-Bellman (HJB) equation for Itô diffusions and the broader class of Feller-Dynkin processes. We then specialize this equation to the setting in which the return distribution is approximated by

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

doi.org

proceedings.mlr.press

EqR: Equivariant Representations for Data-Efficient Reinforcement Learning

Arnab Kumar Mondal

Vineet Jain

Kaleem Siddiqi

Siamak Ravanbakhsh

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

proceedings.mlr.press