Publications

Small-GAN: Speeding Up GAN Training Using Core-Sets

Samarth Sinha

Han Zhang

Augustus Odena

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch siz… (see more)es. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.

2020-11-20

Proceedings of the 37th International Conference on Machine Learning (published)

doi.org

proceedings.mlr.press

Stochastic Hamiltonian Gradient Methods for Smooth Games

Nicolas Loizou

Hugo Berard

Alexia Jolicoeur-Martineau

Pascal Vincent

Simon Lacoste-Julien

Ioannis Mitliagkas

The success of adversarial formulations in machine learning has brought renewed motivation for smooth games. In this work, we focus on the c… (see more)lass of stochastic Hamiltonian methods and provide the first convergence guarantees for certain classes of stochastic smooth games. We propose a novel unbiased estimator for the stochastic Hamiltonian gradient descent (SHGD) and highlight its benefits. Using tools from the optimization literature we show that SHGD converges linearly to the neighbourhood of a stationary point. To guarantee convergence to the exact solution, we analyze SHGD with a decreasing step-size and we also present the first stochastic variance reduced Hamiltonian method. Our results provide the first global non-asymptotic last-iterate convergence guarantees for the class of stochastic unconstrained bilinear games and for the more general class of stochastic games that satisfy a "sufficiently bilinear" condition, notably including some non-convex non-concave problems. We supplement our analysis with experiments on stochastic bilinear and sufficiently bilinear games, where our theory is shown to be tight, and on simple adversarial machine learning formulations.

2020-11-20

Proceedings of the 37th International Conference on Machine Learning (published)

doi.org

proceedings.mlr.press

Universal Equivariant Multilayer Perceptrons

Siamak Ravanbakhsh

Group invariant and equivariant Multilayer Perceptrons (MLP), also known as Equivariant Networks, have achieved remarkable success in learni… (see more)ng on a variety of data structures, such as sequences, images, sets, and graphs. Using tools from group theory, this paper proves the universality of a broad class of equivariant MLPs with a single hidden layer. In particular, it is shown that having a hidden layer on which the group acts regularly is sufficient for universal equivariance (invariance). A corollary is unconditional universality of equivariant MLPs for Abelian groups, such as CNNs with a single hidden layer. A second corollary is the universality of equivariant MLPs with a high-order hidden layer, where we give both group-agnostic bounds and means for calculating group-specific bounds on the order of hidden layer that guarantees universal equivariance (invariance).

2020-11-20

Proceedings of the 37th International Conference on Machine Learning (published)

doi.org

proceedings.mlr.press

On Variational Learning of Controllable Representations for Text without Supervision

Peng Xu

Jackie Chi Kit Cheung

Yanshuai Cao

The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or … (see more)extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.

2020-11-20

Proceedings of the 37th International Conference on Machine Learning (published)

proceedings.mlr.press

What can I do here? A Theory of Affordances in Reinforcement Learning

Khimya Khetarpal

Zafarali Ahmed

Gheorghe Comanici

David Abel

Doina Precup

Reinforcement learning algorithms usually assume that all actions are always available to an agent. However, both people and animals underst… (see more)and the general link between the features of their environment and the actions that are feasible. Gibson (1977) coined the term "affordances" to describe the fact that certain states enable an agent to do certain actions, in the context of embodied agents. In this paper, we develop a theory of affordances for agents who learn and plan in Markov Decision Processes. Affordances play a dual role in this case. On one hand, they allow faster planning, by reducing the number of actions available in any given situation. On the other hand, they facilitate more efficient and precise learning of transition models from data, especially when such models require function approximation. We establish these properties through theoretical results as well as illustrative examples. We also propose an approach to learn affordances and use it to estimate transition models that are simpler and generalize better.

2020-11-20

Proceedings of the 37th International Conference on Machine Learning (published)

doi.org

proceedings.mlr.press

An Effective Anti-Aliasing Approach for Residual Networks

Cristina Vasconcelos

Hugo Larochelle

Vincent Dumoulin

Nicolas Roux

Ross Goroshin

Image pre-processing in the frequency domain has traditionally played a vital role in computer vision and was even part of the standard pipe… (see more)line in the early days of deep learning. However, with the advent of large datasets, many practitioners concluded that this was unnecessary due to the belief that these priors can be learned from the data itself. Frequency aliasing is a phenomenon that may occur when sub-sampling any signal, such as an image or feature map, causing distortion in the sub-sampled output. We show that we can mitigate this effect by placing non-trainable blur filters and using smooth activation functions at key locations, particularly where networks lack the capacity to learn them. These simple architectural changes lead to substantial improvements in out-of-distribution generalization on both image classification under natural corruptions on ImageNet-C [10] and few-shot learning on Meta-Dataset [17], without introducing additional trainable parameters and using the default hyper-parameters of open source codebases.

2020-11-19

ArXiv (preprint)

arxiv.org

Urban Night Scenery Reconstruction by Day-night Registration and Synthesis

Andi Dai

David Meger

2020-11-12

Proceedings of the 28th International Conference on Advances in Geographic Information Systems (published)

doi.org

Using Open Source Licensing to Regulate the Assembly of LAWS: A Preliminary Analysis

Cheng Lin

AJung Moon

Lethal autonomous weapons (LAWS) are an emerging technology capable of automatically targeting and exercising lethal force. Many scholars an… (see more)d advocates have petitioned to ban the technology internationally for a myriad of reasons. However, there are practical challenges to implementing a ban. One such challenge is posed by the “intangible” nature of the software that LAWS depends on, which is incompatible with implementation mechanisms such as export control. Given the dual-use nature of software, and the fact that software is developed by teams of individuals, a number of soft governance mechanisms have been proposed to regulate this technology. In this paper, we investigate the feasibility of one particular approach: leveraging open source licenses as a means to prohibit the use of certain software in LAWS. This approach is largely motivated by the fact that open source software underpins all of technology, especially AI. Through a review of the recent tech activism and open source activism, we evaluate whether open source licenses can feasibly limit the use of open source software to only non-LAWS applications. We distill the current challenges facing “ethics-driven” open source licensing efforts into three main obstacles: the need for clarity of licensing language, the lack of enforceability of licenses, and the lack of cohesiveness of the open source community. We propose that addressing these factors are also success criteria for future anti-LAWS open source initiatives. We find that open source licenses provide more theoretical than practical promise in regulating LAWS, and conclude that cohesion in the open source community is the key to their potential practical success in the future.

2020-11-11

2020 IEEE International Symposium on Technology and Society (ISTAS) (published)

doi.org

Global Surveillance of COVID-19 by mining news media using a multi-source dynamic embedded topic model.

Yue Li

Pratheeksha Nair

Zhi Wen

Imane Chafi

Anya Okhmatovskaia

Guido Powell

Yannan Shen

David L. Buckeridge

As the COVID-19 pandemic continues to unfold, understanding the global impact of non-pharmacological interventions (NPI) is important for fo… (see more)rmulating effective intervention strategies, particularly as many countries prepare for future waves. We used a machine learning approach to distill latent topics related to NPI from large-scale international news media. We hypothesize that these topics are informative about the timing and nature of implemented NPI, dependent on the source of the information (e.g., local news versus official government announcements) and the target countries. Given a set of latent topics associated with NPI (e.g., self-quarantine, social distancing, online education, etc), we assume that countries and media sources have different prior distributions over these topics, which are sampled to generate the news articles. To model the source-specific topic priors, we developed a semi-supervised, multi-source, dynamic, embedded topic model. Our model is able to simultaneously infer latent topics and learn a linear classifier to predict NPI labels using the topic mixtures as input for each news article. To learn these models, we developed an efficient end-to-end amortized variational inference algorithm. We applied our models to news data collected and labelled by the World Health Organization (WHO) and the Global Public Health Intelligence Network (GPHIN). Through comprehensive experiments, we observed superior topic quality and intervention prediction accuracy, compared to the baseline embedded topic models, which ignore information on media source and intervention labels. The inferred latent topics reveal distinct policies and media framing in different countries and media sources, and also characterize reaction to COVID-19 and NPI in a semantically meaningful manner. Our PyTorch code is available on Github (htps://github.com/li-lab-mcgill/covid19_media).

2020-11-09

Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (published)

doi.org

On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs.

Teng Long

Yanshuai Cao

Jackie CK Cheung

Variational autoencoders (VAEs) hold great potential for modelling text, as they could in theory separate high-level semantic and syntactic … (see more)properties from local regularities of natural language. Practically, however, VAEs with autoregressive decoders often suffer from posterior collapse, a phenomenon where the model learns to ignore the latent variables, causing the sequence VAE to degenerate into a language model. In this paper, we argue that posterior collapse is in part caused by the lack of dispersion in encoder features. We provide empirical evidence to verify this hypothesis, and propose a straightforward fix using pooling. This simple technique effectively prevents posterior collapse, allowing model to achieve significantly better data log-likelihood than standard sequence VAEs. Comparing to existing work, our proposed method is able to achieve comparable or superior performances while being more computationally efficient.

2020-11-09

(published)

www.semanticscholar.org

Approximate Planning and Learning for Partially Observed Systems

Aditya Mahajan

2020-11-08

International Conference of Control, Dynamic Systems, and Robotics (published)

doi.org

DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning

Timo Milbich

Karsten Roth

Homanga Bharadhwaj