Publications

MIP-GNN: A Data-Driven Framework for Guiding Combinatorial Solvers
Elias B. Khalil
Christopher Morris
Andrea Lodi
Mixed-integer programming (MIP) technology offers a generic way of formulating and solving combinatorial optimization problems. While genera… (voir plus)lly reliable, state-of-the-art MIP solvers base many crucial decisions on hand-crafted heuristics, largely ignoring common patterns within a given instance distribution of the problem of interest. Here, we propose MIP-GNN, a general framework for enhancing such solvers with data-driven insights. By encoding the variable-constraint interactions of a given mixed-integer linear program (MILP) as a bipartite graph, we leverage state-of-the-art graph neural network architectures to predict variable biases, i.e., component-wise averages of (near) optimal solutions, indicating how likely a variable will be set to 0 or 1 in (near) optimal solutions of binary MILPs. In turn, the predicted biases stemming from a single, once-trained model are used to guide the solver, replacing heuristic components. We integrate MIP-GNN into a state-of-the-art MIP solver, applying it to tasks such as node selection and warm-starting, showing significant improvements compared to the default setting of the solver on two classes of challenging binary MILPs.
2022-06-27
AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org
PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks
Mojtaba Faramarzi
Mohammad Amini
Akilesh Badrinaaraayanan
Vikas Verma
Sarath Chandar
Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data… (voir plus). A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide a better generalization to deformed samples and is more robust against adversarial attacks.
2022-06-27
AAAI Conference on Artificial Intelligence (publié)
doi.org
arxiv.org
Proving Theorems using Incremental Learning and Hindsight Experience Replay
Maxwell Crouse
Eser Aygün
Laurent Orseau
Bassem Makni
Vernon Ralph Austel
Ankit Anand
Xavier Glorot
Cristina Cornelio
Shajith Ikbal
Stephen McAleer
Vlad Firoiu
Pavan Kapanipathi
Lei Zhang
Ndivhuwo Makondo
Doina Precup
Shibl Mourad
Traditional automated theorem provers for first-order logic depend on speed-optimized search and many handcrafted heuristics that are design… (voir plus)ed to work best over a wide range of domains. Machine learning approaches in literature either depend on these traditional provers to bootstrap themselves or fall short on reaching comparable performance. In this paper, we propose a general incremental learning algorithm for training domain specific provers for first-order logic without equality, based only on a basic given-clause algorithm, but using a learned clause-scoring function. Clauses are represented as graphs and presented to transformer networks with spectral features. To address the sparsity and the initial lack of training data as well as the lack of a natural curriculum, we adapt hindsight experience replay to theorem proving, so as to be able to learn even when no proof can be found. We show that provers trained this way can match and sometimes surpass state-of-the-art traditional provers on the TPTP dataset in terms of both quantity and quality of the proofs.
2022-06-27
Proceedings of the 39th International Conference on Machine Learning (publié)
doi.org
proceedings.mlr.press
Robust Policy Learning over Multiple Uncertainty Sets
Annie Xie
Shagun Sodhani
Chelsea Finn
Joelle Pineau
Amy Zhang
Reinforcement learning (RL) agents need to be robust to variations in safety-critical environments. While system identification methods prov… (voir plus)ide a way to infer the variation from online experience, they can fail in settings where fast identification is not possible. Another dominant approach is robust RL which produces a policy that can handle worst-case scenarios, but these methods are generally designed to achieve robustness to a single uncertainty set that must be specified at train time. Towards a more general solution, we formulate the multi-set robustness problem to learn a policy robust to different perturbation sets. We then design an algorithm that enjoys the benefits of both system identification and robust RL: it reduces uncertainty where possible given a few interactions, but can still act robustly with respect to the remaining uncertainty. On a diverse set of control tasks, our approach demonstrates improved worst-case performance on new environments compared to prior methods based on system identification and on robust RL alone.
2022-06-27
Proceedings of the 39th International Conference on Machine Learning (publié)
proceedings.mlr.press
proceedings.mlr.press
TACTiS: Transformer-Attentional Copulas for Time Series
Alexandre Drouin
Étienne Marcotte
Nicolas Chapados
The estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. However, t… (voir plus)he practical utility of such estimates is limited by how accurately they quantify predictive uncertainty. In this work, we address the problem of estimating the joint predictive distribution of high-dimensional multivariate time series. We propose a versatile method, based on the transformer architecture, that estimates joint distributions using an attention-based decoder that provably learns to mimic the properties of non-parametric copulas. The resulting model has several desirable properties: it can scale to hundreds of time series, supports both forecasting and interpolation, can handle unaligned and non-uniformly sampled data, and can seamlessly adapt to missing data during training. We demonstrate these properties empirically and show that our model produces state-of-the-art predictions on multiple real-world datasets.
2022-06-27
Proceedings of the 39th International Conference on Machine Learning (publié)
proceedings.mlr.press
proceedings.mlr.press
Towards efficient representation identification in supervised learning
Kartik Ahuja
Divyat Mahajan
Vasilis Syrgkanis
Ioannis Mitliagkas
Humans have a remarkable ability to disentangle complex sensory inputs (e.g., image, text) into simple factors of variation (e.g., shape, co… (voir plus)lor) without much supervision. This ability has inspired many works that attempt to solve the following question: how do we invert the data generation process to extract those factors with minimal or no supervision? Several works in the literature on non-linear independent component analysis have established this negative result; without some knowledge of the data generation process or appropriate inductive biases, it is impossible to perform this inversion. In recent years, a lot of progress has been made on disentanglement under structural assumptions, e.g., when we have access to auxiliary information that makes the factors of variation conditionally independent. However, existing work requires a lot of auxiliary information, e.g., in supervised classification, it prescribes that the number of label classes should be at least equal to the total dimension of all factors of variation. In this work, we depart from these assumptions and ask: a) How can we get disentanglement when the auxiliary information does not provide conditional independence over the factors of variation? b) Can we reduce the amount of auxiliary information required for disentanglement? For a class of models where auxiliary information does not ensure conditional independence, we show theoretically and experimentally that disentanglement (to a large extent) is possible even when the auxiliary information dimension is much less than the dimension of the true latent representation.
2022-06-27
Proceedings of the First Conference on Causal Learning and Reasoning (publié)
doi.org
proceedings.mlr.press
VIM: Variational Independent Modules for Video Prediction
Rim Assouel
Lluis Castrejon
Aaron Courville
Nicolas Ballas
Yoshua Bengio
2022-06-27
Proceedings of the First Conference on Causal Learning and Reasoning (publié)
proceedings.mlr.press
proceedings.mlr.press
YOUR AUTOREGRESSIVE GENERATIVE MODEL CAN BE BETTER IF YOU TREAT IT AS AN ENERGY-BASED ONE
Yezhen Wang
Tong Che
Bin Li
Kaitao Song
Hengzhi Pei
Yoshua Bengio
Dongsheng Li
2022-06-25
ArXiv (prépublication)
doi.org
openreview.net
GP.2 Deep learning prediction of response to disease modifying therapy in primary progressive multiple sclerosis
JR Falet
Joshua D. Durso-Finley
Brennan Nichyporuk
Julien Schroeter
Francesca Bovis
M Sormani
D Precup
Tal Arbel
DL Arnold
Background: Only one disease modifying therapy (DMT), ocrelizumab, was found to slow disability progression in primary progressive multiple … (voir plus)sclerosis (PPMS). Modeling the conditional average treatment effect (CATE) using deep learning could identify individuals more responsive to DMTs, allowing for predictive enrichment to increase the power of future clinical trials. Methods: Baseline clinical and MRI data were acquired as part of three placebo-controlled randomized clinical trials: ORATORIO (ocrelizumab), OLYMPUS (rituximab) and ARPEGGIO (laquinimod). Data from ORATORIO and OLYMPUS was separated into a training (70%) and testing (30%) set, while ARPEGGIO served as additional validation. An ensemble of multitask multilayer perceptrons was trained to predict the rate of disability progression on both treatment and placebo to estimate CATE. Results: The model could separate individuals based on their predicted treatment effect. The top 25% of individuals predicted to respond most have a larger effect size (HR 0.442, p=0.0497) than the entire group (HR 0.787, p=0.292). The model could also identify responders to laquinimod. A simulated study where the 50% most responsive individuals are randomized would require 6-times less participants to detect a significant effect. Conclusions: Individuals with PPMS who respond favourably to DMTs can be identified using deep learning based on their baseline clinical and imaging characteristics.
2022-06-23
Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques (publié)
doi.org
Neural Networks as Paths through the Space of Representations
Richard D Lange
Jordan Kyle Matelsky
Xinyue Wang
Devin Kwok
David Rolnick
Konrad Paul Kording
2022-06-21
ArXiv (prépublication)
doi.org
openreview.net
Relationship Between Arterial Stiffness Index, Pulse Pressure, and Magnetic Resonance Imaging Markers of White Matter Integrity: A UK Biobank Study
Atef Badji
Julien Cohen-Adad
Hélène Girouard
Alzheimer’s disease and dementia in general constitute one of the major public health problems of the 21st century. Research in arterial s… (voir plus)tiffness and pulse pressure (PP) play an important role in the quest to reduce the risk of developing dementia through controlling modifiable risk factors. The aim of the study is to investigate the association between peripheral PP, arterial stiffness index (ASI) and brain integrity, and to discover if ASI is a better predictor of white matter integrity than peripheral PP. 17,984 participants 63.09 ± 7.31 from the UK Biobank were used for this study. ASI was estimated using infrared light (photoplethysmography) and peripheral PP was calculated by subtracting the diastolic from the systolic brachial blood pressure value. Measure of fractional anisotropy (FA) was obtained from diffusion imaging to estimate white matter microstructural integrity. White matter hyperintensities were segmented from the combined T1 and T2-weighted FLAIR images as a measure of irreversible white matter damage. An important finding is that peripheral PP better predicts white matter integrity when compared to ASI. This finding is consistent until 75 years old. Interestingly, no significant relationship is found between either peripheral PP or ASI and white matter integrity after 75 years old. These results suggest that ASI from plethysmography should not be used to estimate cerebrovascular integrity in older adults and further question the relationship between arterial stiffness, blood pressure, and white matter damage after the age of 75 years old.
2022-06-20
Frontiers in Aging Neuroscience (publié)
doi.org
Healthsheet: Development of a Transparency Artifact for Health Datasets
Negar Rostamzadeh
Diana Mincu
Subhrajit Roy
Andrew J Smart
Lauren Wilcox
Mahima Pushkarna
Jessica Schrouff
Razvan Amironesei
Nyalleng Moorosi
Katherine Heller
2022-06-19
2022 ACM Conference on Fairness, Accountability, and Transparency (publié)
doi.org
arxiv.org