Joelle Pineau

Romain Laroche

We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the … (voir plus)scenario where: (i) we have a dataset collected under a known baseline policy, (ii) multiple reward signals are received from the environment inducing as many objectives to optimize. We present an SPI formulation for this RL setting that takes into account the preferences of the algorithm’s user for handling the trade-offs for different reward signals while ensuring that the new policy performs at least as well as the baseline policy along each individual objective. We build on traditional SPI algorithms and propose a novel method based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche et al., 2019) that provides high probability guarantees on the performance of the agent in the true environment. We show the effectiveness of our method on a synthetic grid-world safety task as well as in a real-world critical care context to learn a policy for the administration of IV fluids and vasopressors to treat sepsis.

2021-11-08

NeurIPS.cc/2021/Conference (poster)

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

Koustuv Sinha

Robin Jia

Dieuwke Hupkes

Adina Williams

Douwe Kiela

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to repres… (voir plus)ent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks—including tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.

2021-10-31

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (publié)

arxiv.org

Sometimes We Want Ungrammatical Translations

Prasanna Parthasarathi

Koustuv Sinha

Adina Williams

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has focused primarily on improving translation quality, a… (voir plus)nd as a secondary focus, improving robustness to perturbations (e.g. spelling). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness or faithfulness, by focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure their effects on the target side. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which prop-erty is best suited for their use case.

2021-10-31

Empirical Methods in Natural Language Processing (inconnu)

Circulating proteins to predict adverse COVID-19 outcomes

Chen-Yang Su

Sirui Zhou

Edgar Gonzalez-Kozlova

Guillaume Butler-Laporte

Elsa Brunet-Ratnasingham

Tomoko Nakanishi

Wonseok Jeon

David Morrison

Laetitia Laurent

Jonathan Afilalo

Marc Afilalo

Danielle Henry

Yiheng Chen

Julia Carrasco-Zanini

Yossi Farjoun

Maik Pietzner

Nofar Kimchi

Zaman Afrasiabi

Nardin Rezk

Meriem Bouab … (voir 47 de plus)

Louis Petitjean

Charlotte Guzman

Xiaoqing Xue

Chris Tselios

Branka Vulesevic

Olumide Adeleye

Tala Abdullah

Noor Almamlouk

Yara Moussa

Chantal DeLuca

Naomi Duggan

Erwin Schurr

Nathalie Brassard

Madeleine Durand

Diane Marie Del Valle

Ryan Thompson

Mario A. Cedillo

Eric Schadt

Kai Nie

Nicole W Simons

Konstantinos Mouskas

Nicolas Zaki

Manishkumar Patel

Hui Xie

Jocelyn Harris

Robert Marvin

Esther Cheng

Kevin Tuballes

Kimberly Argueta

Ieisha Scott

Celia M T Greenwood

Clare Paterson

Michael A. Hinterberg

Claudia Langenberg

Vincenzo Forgetta

Vincent Mooser

Thomas Marron

Noam Beckmann

Ephraim Kenigsberg

Seunghee Kim-schulze

Alexander W. Charney

Sacha Gnjatic

Daniel E. Kaufmann

Miriam Merad

J Brent Richards

Predicting COVID-19 severity is difficult, and the biological pathways involved are not fully understood. To approach this problem, we measu… (voir plus)red 4,701 circulating human protein abundances in two independent cohorts totaling 986 individuals. We then trained prediction models including protein abundances and clinical risk factors to predict adverse COVID-19 outcomes in 417 subjects and tested these models in a separate cohort of 569 individuals. For severe COVID-19, a baseline model including age and sex provided an area under the receiver operator curve (AUC) of 65% in the test cohort. Selecting 92 proteins from the 4,701 unique protein abundances improved the AUC to 88% in the training cohort, which remained relatively stable in the testing cohort at 86%, suggesting good generalizability. Proteins selected from different adverse COVID-19 outcomes were enriched for cytokine and cytokine receptors, but more than half of the enriched pathways were not immune-related. Taken together, these findings suggest that circulating proteins measured at early stages of disease progression are reasonably accurate predictors of adverse COVID-19 outcomes. Further research is needed to understand how to incorporate protein measurement into clinical care.

2021-10-04

medRxiv (prépublication)

Automated Data-Driven Generation of Personalized Pedagogical Interventions in Intelligent Tutoring Systems

Ekaterina Kochmar

Dung D. Vu

Robert Belfer

Varun Gupta

Iulian V. Serban

2021-07-26

International Journal of Artificial Intelligence in Education (publié)

A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss

Prasanna Parthasarathi

Mohamed Abdelsalam

Sarath Chandar

Neural models trained for next utterance generation in dialogue task learn to mimic the n-gram sequences in the training set with training o… (voir plus)bjectives like negative log-likelihood (NLL) or cross-entropy. Such commonly used training objectives do not foster generating alternate responses to a context. But, the effects of minimizing an alternate training objective that fosters a model to generate alternate response and score it on semantic similarity has not been well studied. We hypothesize that a language generation model can improve on its diversity by learning to generate alternate text during training and minimizing a semantic loss as an auxiliary objective. We explore this idea on two different sized data sets on the task of next utterance generation in goal oriented dialogues. We make two observations (1) minimizing a semantic objective improved diversity in responses in the smaller data set (Frames) but only as-good-as minimizing the NLL in the larger data set (MultiWoZ) (2) large language model embeddings can be more useful as a semantic loss objective than as initialization for token embeddings.

2021-06-30

Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (publié)

arxiv.org

Learning Robust State Abstractions for Hidden-Parameter Block MDPS

2021-05-02

International Conference on Learning Representations (Poster)

TDprop: Does Adaptive Optimization With Jacobi Preconditioning Help Temporal Difference Learning?

We investigate whether Jacobi preconditioning, accounting for the bootstrap term in temporal difference (TD) learning, can help boost perfor… (voir plus)mance of adaptive optimizers. Our method, TDprop, computes a per-parameter learning rate based on the diagonal preconditioning of the TD update rule. We show how this can be used in both n-step returns and TD(λ). Our theoretical findings demonstrate that including this additional preconditioning information is comparable to normal semi-gradient TD if the optimal learning rate is found for both via a hyperparameter search. This matches our experimental results. In Deep RL experiments using Expected SARSA, TDprop meets or exceeds the performance of Adam in all tested games under near-optimal learning rates, but a well-tuned SGD can yield similar performance in most settings. Our findings suggest that Jacobi preconditioning may improve upon Adam in Deep RL, but despite incorporating additional information from the TD bootstrap term, may not always be better than SGD. Moreover, they suggest that more theoretical investigations are needed to understand adaptive optimizers under optimal hyperparameter regimes in TD learning: simpler methods may, surprisingly, be theoretically comparable after a hyperparameter search.

2021-05-02

Adaptive Agents and Multi-Agent Systems (publié)

Model-Invariant State Abstractions for Model-Based Reinforcement Learning

Manan Tomar

Amy Zhang

Roberto Calandra

Matthew E. Taylor

Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of task… (voir plus)s increases, so does the sample inefficiency of learning accurate dynamics models. However, many complex tasks also exhibit sparsity in the dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called \textit{model-invariance}. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately learn a model-invariant representation for complex domains and validate our approach by showing improved modelling performance over standard maximum likelihood approaches on challenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL setting we show strong performance gains with respect to sample efficiency across a host of other continuous control tasks.

2021-02-18

ArXiv (prépublication)

Correcting Momentum in Temporal Difference Learning

Emmanuel Bengio

Doina Precup

A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, r… (voir plus)eapplying them at each iteration. We argue that, unlike in supervised learning, momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale: not only does the gradient of the loss change due to parameter updates, the loss itself changes due to bootstrapping. We first show that this phenomenon exists, and then propose a first-order correction term to momentum. We show that this correction term improves sample efficiency in policy evaluation by correcting target value drift. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.

2020-12-31

arXiv (prépublication)