Publications

Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning
Bogdan Mazoure
Paul Mineiro
Pavithra Srinath
Reza Sharifi Sedeh
Doina Precup
Adith Swaminathan
We study session-based recommendation scenarios where we want to recommend items to users during sequential interactions to improve their lo… (voir plus)ng-term utility. Optimizing a long-term metric is challenging because the learning signal (whether the recommendations achieved their desired goals) is delayed and confounded by other user interactions with the system. Targeting immediately measurable proxies such as clicks can lead to suboptimal recommendations due to misalignment with the long-term metric. We develop a new reinforcement learning algorithm called Short Horizon Policy Improvement (SHPI) that approximates policy-induced drift in user behavior across sessions. SHPI is a straightforward modification of episodic RL algorithms for session-based recommendation, that additionally gives an appropriate termination bonus in each session. Empirical results on four recommendation tasks show that SHPI can outperform state-of-the-art recommendation techniques like matrix factorization with offline proxy signals, bandits with myopic online proxies, and RL baselines with limited amounts of user interaction.
2021-05-31
ArXiv (prépublication)
arxiv.org
arxiv.org
Modeling Event Plausibility with Consistent Conceptual Abstraction
Ian Porada
Kaheer Suleman
Adam Trischler
Jackie CK Cheung
2021-05-31
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (publié)
doi.org
arxiv.org
Data-Efficient Reinforcement Learning with Self-Predictive Representations
Max Schwarzer
Ankesh Anand
Rishab Goel
R Devon Hjelm
Aaron Courville
Philip Bachman
While deep reinforcement learning excels at solving tasks where large amounts of data can be collected through virtually unlimited interacti… (voir plus)on with the environment, learning from limited interaction remains a key challenge. We posit that an agent can learn more efficiently if we augment reward maximization with self-supervised objectives based on structure in its visual input and sequential interaction with the environment. Our method, Self-Predictive Representations(SPR), trains an agent to predict its own latent state representations multiple steps into the future. We compute target representations for future states using an encoder which is an exponential moving average of the agent's parameters and we make predictions using a learned transition model. On its own, this future prediction objective outperforms prior methods for sample-efficient deep RL from pixels. We further improve performance by adding data augmentation to the future prediction loss, which forces the agent's representations to be consistent across multiple views of an observation. Our full self-supervised objective, which combines future prediction and data augmentation, achieves a median human-normalized score of 0.415 on Atari in a setting limited to 100k steps of environment interaction, which represents a 55% relative improvement over the previous state-of-the-art. Notably, even in this limited data regime, SPR exceeds expert human scores on 7 out of 26 games. The code associated with this work is available at https://github.com/mila-iqia/spr
2021-05-30
Université de Montréal (publié)
doi.org
openreview.net
Gotta Go Fast When Generating Data with Score-Based Models
Alexia Jolicoeur-Martineau
Fei Li
Rémi Piché-Taillefer
Tal Kachman
Ioannis Mitliagkas
Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These ap… (voir plus)proaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly - they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.
2021-05-27
ArXiv (prépublication)
openreview.net
openreview.net
Noised Consistency Training for Text Summarization
J. Y. Liu
Qianren Mao
Bang Liu
Hao Peng
Hongdong Zhu
Jian-Xin Li
Neural abstractive summarization methods often require large quantities of labeled training data. However, labeling large amounts of summari… (voir plus)zation data is often prohibitive due to time, financial, and expertise constraints, which has limited the usefulness of summarization systems to practical applications. In this paper, we argue that this limitation can be overcome by a semi-supervised approach: consistency training which is to leverage large amounts of unlabeled data to improve the performance of supervised learning over a small corpus. The consistency regularization semi-supervised learning can regularize model predictions to be invariant to small noise applied to input articles. By adding noised unlabeled corpus to help regularize consistency training, this framework obtains comparative performance without using the full dataset. In particular, we have verified that leveraging large amounts of unlabeled data decently improves the performance of supervised learning over an insufficient labeled dataset.
2021-05-27
ArXiv (prépublication)
arxiv.org
arxiv.org
AndroidEnv: A Reinforcement Learning Platform for Android
Daniel Toyama
Philippe Hamel
Anita Gergely
Gheorghe Comanici
Amelia Glaese
Zafarali Ahmed
Tyler Jackson
Shibl Mourad
Doina Precup
We introduce AndroidEnv, an open-source platform for Reinforcement Learning (RL) research built on top of the Android ecosystem. AndroidEnv … (voir plus)allows RL agents to interact with a wide variety of apps and services commonly used by humans through a universal touchscreen interface. Since agents train on a realistic simulation of an Android device, they have the potential to be deployed on real devices. In this report, we give an overview of the environment, highlighting the significant features it provides for research, and we present an empirical evaluation of some popular reinforcement learning agents on a set of tasks built on this platform.
2021-05-26
ArXiv (prépublication)
arxiv.org
arxiv.org
Robotic Object Manipulation with Full-Trajectory GAN-Based Imitation Learning
Haoxu Wang
David Meger
This paper develops a novel generative imitation learning system capable of capturing the distribution of expert demonstrations in trajector… (voir plus)y space, which allows longer temporal context within complex motion sequences to be captured. While auto-regressive models that model time-steps sequentially can in principle be recursively applied to capture long sequences, there are known issues with learning such models reliably. In contrast, our model represents full trajectories a first-class entities, which has required us to adapt the typical generative adversarial learning architecture. We pair a full-trajectory discriminator with an imitation-inspired generative trajectory model and train these two in adversarial fashion. Our results show that our method matches the performance of existing approaches for simple tasks, in simulation and on real robot deployments. We produce state-of-the-art accuracy in replicating motions that contain long-term dependencies such as pouring.
2021-05-25
2021 18th Conference on Robots and Vision (CRV) (publié)
doi.org
Publisher Correction: The default network of the human brain is associated with perceived social isolation
R. Nathan Spreng
Emile Dimas
Laetitia Mwilambwe-Tshilobo
Alain Dagher
Philipp Koellinger
Gideon Nave
Anthony Ong
Julius M. Kernbach
Thomas V. Wiecki
Tian Ge
Yue Li
Avram J. Holmes
B. T. Thomas Yeo
Gary R. Turner
Robin I. M. Dunbar
Danilo Bzdok
2021-05-20
Nature Communications (publié)
doi.org
Periodic Freight Demand Estimation for Large-scale Tactical Planning
Greta Laage
Emma Frejinger
Gilles Savard
Freight carriers rely on tactical planning to design their service network to satisfy demand in a cost-effective way. For computational trac… (voir plus)tability, deterministic and cyclic Service Network Design (SND) formulations are used to solve large-scale problems. A central input is the periodic demand, that is, the demand expected to repeat in every period in the planning horizon. In practice, demand is predicted by a time series forecasting model and the periodic demand is the average of those forecasts. This is, however, only one of many possible mappings. The problem consisting in selecting this mapping has hitherto been overlooked in the literature. We propose to use the structure of the downstream decision-making problem to select a good mapping. For this purpose, we introduce a multilevel mathematical programming formulation that explicitly links the time series forecasts to the SND problem of interest. The solution is a periodic demand estimate that minimizes costs over the tactical planning horizon. We report results in an extensive empirical study of a large-scale application from the Canadian National Railway Company. They clearly show the importance of the periodic demand estimation problem. Indeed, the planning costs exhibit an important variation over different periodic demand estimates and using an estimate different from the mean forecast can lead to substantial cost reductions. Moreover, the costs associated with the periodic demand estimates based on forecasts were comparable to, or even better than those obtained using the mean of actual demand.
2021-05-18
ArXiv (prépublication)
arxiv.org
arxiv.org
Artificial intelligence in nursing: Priorities and opportunities from an international invitational think-tank of the Nursing and Artificial Intelligence Leadership Collaborative
Charlene Esteban Ronquillo
Laura-Maria Peltonen
Lisiane Pruinelli
Charlene H. Chu
Suzanne Bakken
Ana Beduschi
Kenrick Cato
Nicholas Hardiker
Alain Junger
Martin Michalowski
Rune Nyrup
Samira Rahimi
Donald Nigel Reed
Tapio Salakoski
Sanna Salanterä
Nancy Walton
Patrick Weber
Thomas Wiegand
Maxim Topaz
To develop a consensus paper on the central points of an international invitational think‐tank on nursing and artificial intelligence (AI)… (voir plus). We established the Nursing and Artificial Intelligence Leadership (NAIL) Collaborative, comprising interdisciplinary experts in AI development, biomedical ethics, AI in primary care, AI legal aspects, philosophy of AI in health, nursing practice, implementation science, leaders in health informatics practice and international health informatics groups, a representative of patients and the public, and the Chair of the ITU/WHO Focus Group on Artificial Intelligence for Health. The NAIL Collaborative convened at a 3‐day invitational think tank in autumn 2019. Activities included a pre‐event survey, expert presentations and working sessions to identify priority areas for action, opportunities and recommendations to address these. In this paper, we summarize the key discussion points and notes from the aforementioned activities. Nursing's limited current engagement with discourses on AI and health posts a risk that the profession is not part of the conversations that have potentially significant impacts on nursing practice. There are numerous gaps and a timely need for the nursing profession to be among the leaders and drivers of conversations around AI in health systems. We outline crucial gaps where focused effort is required for nursing to take a leadership role in shaping AI use in health systems. Three priorities were identified that need to be addressed in the near future: (a) Nurses must understand the relationship between the data they collect and AI technologies they use; (b) Nurses need to be meaningfully involved in all stages of AI: from development to implementation; and (c) There is a substantial untapped and an unexplored potential for nursing to contribute to the development of AI technologies for global health and humanitarian efforts.
2021-05-17
Journal of Advanced Nursing (inconnu)
doi.org
Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems
Matt Grenander
Robert Belfer
Ekaterina Kochmar
Iulian V. Serban
Franccois St-Hilaire
Jackie CK Cheung
We explore creating automated, personalized feedback in an intelligent tutoring system (ITS). Our goal is to pinpoint correct and incorrect … (voir plus)concepts in student answers in order to achieve better student learning gains. Although automatic methods for providing personalized feedback exist, they do not explicitly inform students about which concepts in their answers are correct or incorrect. Our approach involves decomposing students answers using neural discourse segmentation and classification techniques. This decomposition yields a relational graph over all discourse units covered by the reference solutions and student answers. We use this inferred relational graph structure and a neural classifier to match student answers with reference solutions and generate personalized feedback. Although the process is completely automated and data-driven, the personalized feedback generated is highly contextual, domain-aware and effectively targets each student's misconceptions and knowledge gaps. We test our method in a dialogue-based ITS and demonstrate that our approach results in high-quality feedback and significantly improved student learning gains.
2021-05-17
Proceedings of the AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org
Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
Tong Che
Xiaofeng Liu
Site Li
Yubin Ge
Ruixiang Zhang
Caiming Xiong
Yoshua Bengio
AI Safety is a major concern in many deep learning applications such as autonomous driving. Given a trained deep learning model, an importan… (voir plus)t natural problem is how to reliably verify the model's prediction. In this paper, we propose a novel framework --- deep verifier networks (DVN) to detect unreliable inputs or predictions of deep discriminative models, using separately trained deep generative models. Our proposed model is based on conditional variational auto-encoders with disentanglement constraints to separate the label information from the latent representation. We give both intuitive and theoretical justifications for the model. Our verifier network is trained independently with the prediction model, which eliminates the need of retraining the verifier network for a new model. We test the verifier network on both out-of-distribution detection and adversarial example detection problems, as well as anomaly detection problems in structured prediction tasks such as image caption generation. We achieve state-of-the-art results in all of these problems.
2021-05-17
Proceedings of the AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org