Publications

Don't Freeze Your Embedding: Lessons from Policy Finetuning in Environment Transfer
Victoria Dean
Daniel Toyama
Doina Precup
A common occurrence in reinforcement learning (RL) research is making use of a pretrained vision stack that converts image observations to l… (see more)atent vectors. Using a visual embedding in this way leaves open questions, though: should the vision stack be updated with the policy? In this work, we evaluate the effectiveness of such decisions in RL transfer settings. We introduce policy update formulations for use after pretraining in a different environment and analyze the performance of such formulations. Through this evaluation, we also detail emergent metrics of benchmark suites and present results on Atari and AndroidEnv.
2022-04-26
ICLR.cc/2022/Workshop/GPL (poster)
openreview.net
openreview.net
Multi-tract multi-symptom relationships in pediatric concussion
Guido I. Guberman
Sonja Stojanovski
Eman Nishat
Alain Ptito
Danilo Bzdok
Anne Wheeler
Maxime Descoteaux
The heterogeneity of white matter damage and symptoms in concussion has been identified as a major obstacle to therapeutic innovation. In co… (see more)ntrast, most diffusion MRI (dMRI) studies on concussion have traditionally relied on group-comparison approaches that average out heterogeneity. To leverage, rather than average out, concussion heterogeneity, we combined dMRI and multivariate statistics to characterize multi-tract multi-symptom relationships. Using cross-sectional data from 306 previously concussed children aged 9–10 from the Adolescent Brain Cognitive Development Study, we built connectomes weighted by classical and emerging diffusion measures. These measures were combined into two informative indices, the first representing microstructural complexity, the second representing axonal density. We deployed pattern-learning algorithms to jointly decompose these connectivity features and 19 symptom measures. Early multi-tract multi-symptom pairs explained the most covariance and represented broad symptom categories, such as a general problems pair, or a pair representing all cognitive symptoms, and implicated more distributed networks of white matter tracts. Further pairs represented more specific symptom combinations, such as a pair representing attention problems exclusively, and were associated with more localized white matter abnormalities. Symptom representation was not systematically related to tract representation across pairs. Sleep problems were implicated across most pairs, but were related to different connections across these pairs. Expression of multi-tract features was not driven by sociodemographic and injury-related variables, as well as by clinical subgroups defined by the presence of ADHD. Analyses performed on a replication dataset showed consistent results. Using a double-multivariate approach, we identified clinically-informative, cross-demographic multi-tract multi-symptom relationships. These results suggest that rather than clear one-to-one symptom-connectivity disturbances, concussions may be characterized by subtypes of symptom/connectivity relationships. The symptom/connectivity relationships identified in multi-tract multi-symptom pairs were not apparent in single-tract/single-symptom analyses. Future studies aiming to better understand connectivity/symptom relationships should take into account multi-tract multi-symptom heterogeneity. Financial support for this work came from a Vanier Canada Graduate Scholarship from the Canadian Institutes of Health Research (G.I.G.), an Ontario Graduate Scholarship (S.S.), a Restracomp Research Fellowship provided by the Hospital for Sick Children (S.S.), an Institutional Research Chair in Neuroinformatics (M.D.), as well as a Natural Sciences and Engineering Research Council CREATE grant (M.D.).
2022-04-26
eLife (published)
doi.org
A Probabilistic Perspective on Reinforcement Learning via Supervised Learning
Alexandre Piché
Rafael Pardinas
David Vázquez
Christopher Pal
2022-04-26
ICLR.cc/2022/Workshop/GPL (poster)
openreview.net
openreview.net
Accepted Tutorials at The Web Conference 2022
Riccardo Tommasini
Senjuti Basu Roy
Xuan Wang
Hongwei Wang
Heng Ji
Jiawei Han
Preslav Nakov
Giovanni Da San Martino
Firoj Alam
Markus Schedl
Elisabeth Lex
Akash Bharadwaj
Graham Cormode
Milan Dojchinovski
Jan Forberg
Johannes Frey
Pieter Bonte
Marco Balduini
Matteo Belcao
Emanuele Della Valle … (see 53 more)
Junliang Yu
Hongzhi Yin
Tong Chen
Haochen Liu
Yiqi Wang
Wenqi Fan
Xiaorui Liu
Jamell Dacon
Lingjuan Lye
Jiliang Tang
Aristides Gionis
Stefan Neumann
Bruno Ordozgoiti
Simon Razniewski
Hiba Arnaout
Shrestha Ghosh
Fabian Suchanek
Lingfei Wu
Yu Chen
Yunyao Li
Bang Liu
Filip Ilievski
Daniel Garijo
Hans Chalupsky
Pedro Szekely
Ilias Kanellos
Dimitris Sacharidis
Thanasis Vergoulis
Nurendra Choudhary
Nikhil Rao
Karthik Subbian
Srinivasan Sengamedu
Chandan K. Reddy
Friedhelm Victor
Bernhard Haslhofer
George Katsogiannis- Meimarakis
Georgia Koutrika
Shengmin Jin
Danai Koutra
Reza Zafarani
Yulia Tsvetkov
Vidhisha Balachandran
Sachin Kumar
Xiangyu Zhao
Bo Chen
Huifeng Guo
Yejing Wang
Ruiming Tang
Yang Zhang
Wenjie Wang
Peng Wu
Fuli Feng
Xiangnan He
This paper summarizes the content of the 20 tutorials that have been given at The Web Conference 2022: 85% of these tutorials are lecture st… (see more)yle, and 15% of these are hands on.
2022-04-24
The Web Conference (published)
doi.org
Chunked Autoregressive GAN for Conditional Waveform Synthesis
Max Morrison
Rithesh Kumar
Kundan Kumar
Prem Seetharaman
Aaron Courville
Yoshua Bengio
Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such as text, mel-spectrograms, or MIDI. Th… (see more)ese systems employ deep generative models that model the waveform via either sequential (autoregressive) or parallel (non-autoregressive) sampling. Generative adversarial networks (GANs) have become a common choice for non-autoregressive waveform synthesis. However, state-of-the-art GAN-based models produce artifacts when performing mel-spectrogram inversion. In this paper, we demonstrate that these artifacts correspond with an inability for the generator to learn accurate pitch and periodicity. We show that simple pitch and periodicity conditioning is insufficient for reducing this error relative to using autoregression. We discuss the inductive bias that autoregression provides for learning the relationship between instantaneous frequency and phase, and show that this inductive bias holds even when autoregressively sampling large chunks of the waveform during each forward pass. Relative to prior state-of-the-art GAN-based models, our proposed model, Chunked Autoregressive GAN (CARGAN) reduces pitch error by 40-60%, reduces training time by 58%, maintains a fast generation speed suitable for real-time or interactive applications, and maintains or improves subjective quality.
2022-04-24
International Conference on Learning Representations (Accept (Poster))
doi.org
openreview.net
Compositional Attention: Disentangling Search and Retrieval
Sarthak Mittal
Sharath Chandra Raparthy
Irina Rish
Yoshua Bengio
Guillaume Lajoie
Multi-head, key-value attention is the backbone of the widely successful Transformer model and its variants. This attention mechanism uses m… (see more)ultiple parallel key-value attention blocks (called heads), each performing two fundamental computations: (1) search - selection of a relevant entity from a set via query-key interactions, and (2) retrieval - extraction of relevant features from the selected entity via a value matrix. Importantly, standard attention heads learn a rigid mapping between search and retrieval. In this work, we first highlight how this static nature of the pairing can potentially: (a) lead to learning of redundant parameters in certain tasks, and (b) hinder generalization. To alleviate this problem, we propose a novel attention mechanism, called Compositional Attention, that replaces the standard head structure. The proposed mechanism disentangles search and retrieval and composes them in a dynamic, flexible and context-dependent manner through an additional soft competition stage between the query-key combination and value pairing. Through a series of numerical experiments, we show that it outperforms standard multi-head attention on a variety of tasks, including some out-of-distribution settings. Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed. Our proposed mechanism generalizes multi-head attention, allows independent scaling of search and retrieval, and can easily be implemented in lieu of standard attention heads in any network architecture.
2022-04-24
International Conference on Learning Representations (Accept (Spotlight))
doi.org
openreview.net
Meta-matching as a simple framework to translate phenotypic predictive models from big to small data
Tong He
Lijun An
Pansheng Chen
Jianzhong Chen
Jiashi Feng
Danilo Bzdok
Avram J. Holmes
Simon B. Eickhoff
B. T. Thomas Yeo
We propose a simple framework—meta-matching—to translate predictive models from large-scale datasets to new unseen non-brain-imaging phe… (see more)notypes in small-scale studies. The key consideration is that a unique phenotype from a boutique study likely correlates with (but is not the same as) related phenotypes in some large-scale dataset. Meta-matching exploits these correlations to boost prediction in the boutique study. We apply meta-matching to predict non-brain-imaging phenotypes from resting-state functional connectivity. Using the UK Biobank (N = 36,848) and Human Connectome Project (HCP) (N = 1,019) datasets, we demonstrate that meta-matching can greatly boost the prediction of new phenotypes in small independent datasets in many scenarios. For example, translating a UK Biobank model to 100 HCP participants yields an eight-fold improvement in variance explained with an average absolute gain of 4.0% (minimum = −0.2%, maximum = 16.0%) across 35 phenotypes. With a growing number of large-scale datasets collecting increasingly diverse phenotypes, our results represent a lower bound on the potential of meta-matching. Individual-level prediction is critical for precision medicine, but many neuroimaging prediction studies are underpowered. Here the authors present a simple yet powerful approach that effectively translates predictive models from big to small data.
2022-04-24
Nature Neuroscience (published)
doi.org
I NTRODUCING C OORDINATION IN C ONCURRENT R EIN - FORCEMENT L EARNING
Adrien Ali Taiga
Aaron Courville
Bellemare Marc-Emmanuel
Google Brain
Research on exploration in reinforcement learning has mostly focused on problems with a single agent interacting with an environment. Howeve… (see more)r many problems are better addressed by the concurrent reinforcement learning paradigm, where multiple agents operate in a common environment. Recent work has tackled the challenge of exploration in this particular setting (Dimakopoulou & Van Roy, 2018; Dimakopoulou et al., 2018). Nonetheless, they do not completely leverage the characteristics of this framework and agents end up behaving independently from each other. In this work we argue that coordination among concurrent agents is crucial for efficient exploration. We introduce coordination in Thompson Sampling based methods by drawing correlated samples from an agent’s posterior. We apply this idea to extend existing exploration schemes such as randomized least squares value iteration (RLSVI). Empirical results on simple toy tasks emphasize the merits of our approach and call attention to coordination as a key objective for efficient exploration in concurrent reinforcement learning.
2022-04-24
ICLR.cc/2022/Workshop/GMS (published)
openreview.net
openreview.net
QEN: Applicable Taxonomy Completion via Evaluating Full Taxonomic Relations
Suyuchen Wang
Ruihui Zhao
Yefeng Zheng
Bang Liu
Taxonomy is a fundamental type of knowledge graph for a wide range of web applications like searching and recommendation systems. To keep a … (see more)taxonomy automatically updated with the latest concepts, the taxonomy completion task matches a pair of proper hypernym and hyponym in the original taxonomy with the new concept as its parent and child. Previous solutions utilize term embeddings as input and only evaluate the parent-child relations between the new concept and the hypernym-hyponym pair. Such methods ignore the important sibling relations, and are not applicable in reality since term embeddings are not available for the latest concepts. They also suffer from the relational noise of the “pseudo-leaf” node, which is a null node acting as a node’s hyponym to enable the new concept to be a leaf node. To tackle the above drawbacks, we propose the Quadruple Evaluation Network (QEN), a novel taxonomy completion framework that utilizes easily accessible term descriptions as input, and applies pretrained language model and code attention for accurate inference while reducing online computation. QEN evaluates both parent-child and sibling relations to both enhance the accuracy and reduce the noise brought by pseudo-leaf. Extensive experiments on three real-world datasets in different domains with different sizes and term description sources prove the effectiveness and robustness of QEN on overall performance and especially the performance for adding non-leaf nodes, which largely surpasses previous methods and achieves the new state-of-the-art of the task.1
2022-04-24
The Web Conference (published)
doi.org
Rare CNVs and phenome-wide profiling: a tale of brain-structural divergence and phenotypical convergence
J. Kopal
K. Kumar
K. Saltoun
C. Modenato
C. A. Moreau
S. Martin-Brevet
G. Huguet
M. Jean-Louis
C.O. Martin
Z. Saci
N. Younis
P. Tamer
E. Douard
A. M. Maillard
B. Rodriguez-Herreros
A. Pain
S. Richetin
L. Kushan
A. I. Silva
M. B. M. van den Bree … (see 12 more)
D. E. J. Linden
M. J. Owen
J. Hall
S. Lippé
B. Draganski
I. E. Sønderby
O. A. Andreassen
D. C. Glahn
P. M. Thompson
C. E. Bearden
S. Jacquemont
D. Bzdok
Copy number variations (CNVs) are rare genomic deletions and duplications that can exert profound effects on brain and behavior. Previous re… (see more)ports of pleiotropy in CNVs imply that they converge on shared mechanisms at some level of pathway cascades, from genes to large-scale neural circuits to the phenome. However, studies to date have primarily examined single CNV loci in small clinical cohorts. It remains unknown how distinct CNVs escalate the risk for the same developmental and psychiatric disorders. Here, we quantitatively dissect the impact on brain organization and behavioral differentiation across eight key CNVs. In 534 clinical CNV carriers from multiple sites, we explored CNV-specific brain morphology patterns. We extensively annotated these CNV-associated patterns with deep phenotyping assays through the UK Biobank resource. Although the eight CNVs cause disparate brain changes, they are tied to similar phenotypic profiles across ∼1000 lifestyle indicators. Our population-level investigation established brain structural divergences and phenotypical convergences of CNVs, with direct relevance to major brain disorders.
2022-04-24
bioRxiv (preprint)
doi.org
Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study
Jianzhong Chen
Angela Tam
Valeria Kebets
Csaba Orban
Leon Qi Rong Ooi
Leon Qi Rong Ooi
Christopher L. Asplund
Scott Marek
Nico U. F. Dosenbach
Simon B. Eickhoff
Danilo Bzdok
Avram J. Holmes
B. T. Thomas Yeo
How individual differences in brain network organization track behavioral variability is a fundamental question in systems neuroscience. Rec… (see more)ent work suggests that resting-state and task-state functional connectivity can predict specific traits at the individual level. However, most studies focus on single behavioral traits, thus not capturing broader relationships across behaviors. In a large sample of 1858 typically developing children from the Adolescent Brain Cognitive Development (ABCD) study, we show that predictive network features are distinct across the domains of cognitive performance, personality scores and mental health assessments. On the other hand, traits within each behavioral domain are predicted by similar network features. Predictive network features and models generalize to other behavioral measures within the same behavioral domain. Although tasks are known to modulate the functional connectome, predictive network features are similar between resting and task states. Overall, our findings reveal shared brain network features that account for individual variation within broad domains of behavior in childhood.
2022-04-24
Nature Communications (published)
doi.org
Staged independent learning: Towards decentralized cooperative multi-agent Reinforcement Learning
Hadi Nekoei
Akilesh Badrinaaraayanan
Amit Sinha
Mohammad Amini
Janarthanan Rajendran
Aditya Mahajan
A. Chandar
We empirically show that classic ideas from two-time scale stochastic approximation \citep{borkar1997stochastic} can be combined with sequen… (see more)tial iterative best response (SIBR) to solve complex cooperative multi-agent reinforcement learning (MARL) problems. We first start with giving a multi-agent estimation problem as a motivating example where SIBR converges while parallel iterative best response (PIBR) does not. Then we present a general implementation of staged multi-agent RL algorithms based on SIBR and multi-time scale stochastic approximation, and show that our new methods which we call Staged Independent Proximal Policy Optimization (SIPPO) and Staged Independent Q-learning (SIQL) outperform state-of-the-art independent learning on almost all the tasks in the epymarl \citep{papoudakis2020benchmarking} benchmark. This can be seen as a first step towards more decentralized MARL methods based on SIBR and multi-time scale learning.
2022-04-24
ICLR.cc/2022/Workshop/GMS (published)
openreview.net
openreview.net