Publications

Tensor Regression Networks with various Low-Rank Tensor Approximations
Xingwei Cao
Guillaume Rabusseau
Joelle Pineau
Tensor regression networks achieve high compression rate of neural networks while having slight impact on performances. They do so by imposi… (see more)ng low tensor rank structure on the weight matrices of fully connected layers. In recent years, tensor regression networks have been investigated from the perspective of their compressive power, however, the regularization effect of enforcing low-rank tensor structure has not been investigated enough. We study tensor regression networks using various low-rank tensor approximations, aiming to compare the compressive and regularization power of different low-rank constraints. We evaluate the compressive and regularization performances of the proposed model with both deep and shallow convolutional neural networks. The outcome of our experiment suggests the superiority of Global Average Pooling Layer over Tensor Regression Layer when applied to deep convolutional neural network with CIFAR-10 dataset. On the contrary, shallow convolutional neural networks with tensor regression layer and dropout achieved lower test error than both Global Average Pooling and fully-connected layer with dropout function when trained with a small number of samples.
2017-12-26
ArXiv (preprint)
arxiv.org
arxiv.org
ObamaNet: Photo-realistic lip-sync from text
Rithesh Kumar
Jose Sotelo
Kundan Kumar
Alexandre De Brébisson
Yoshua Bengio
We present ObamaNet, the first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text. Co… (see more)ntrary to other published lip-sync approaches, ours is only composed of fully trainable neural modules and does not rely on any traditional computer graphics methods. More precisely, we use three main modules: a text-to-speech network based on Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio, and a network based on Pix2Pix to generate the video frames conditioned on the keypoints.
2017-12-05
ArXiv (preprint)
arxiv.org
arxiv.org
Deep Learning @15 Petaflops/second: Semi-supervised pattern detection for 15 Terabytes of climate data
W. Collins
M. Wehner
M. Prabhat
Thorsten Kurth
Nadathur Satish
Ioannis Mitliagkas
Jie Zhang
Evan Racah
Md. Mostofa Ali Patwary
Narayanan Sundaram
Pradeep Dubey
2017-11-30
(published)
www.semanticscholar.org
Predicting Extubation Readiness in Extreme Preterm Infants based on Patterns of Breathing*
Charles C. Onu
Lara J. Kanbar
Wissam Shalish
Karen A. Brown
Guilherme M. Sant'Anna
Robert E. Kearney
Doina Precup
Extremely preterm infants commonly require intubation and invasive mechanical ventilation after birth. While the duration of mechanical vent… (see more)ilation should be minimized in order to avoid complications, extubation failure is associated with increases in morbidities and mortality. As part of a prospective observational study aimed at developing an accurate predictor of extubation readiness, Markov and semi-Markov chain models were applied to gain insight into the respiratory patterns of these infants, with more robust time-series modeling using semi-Markov models. This model revealed interesting similarities and differences between newborns who succeeded extubation and those who failed. The parameters of the model were further applied to predict extubation readiness via generative (joint likelihood) and discriminative (support vector machine) approaches. Results showed that up to 84\% of infants who failed extubation could have been accurately identified prior to extubation.
2017-11-30
2017 IEEE Symposium Series on Computational Intelligence (SSCI) (published)
doi.org
arxiv.org
Use machine learning to find energy materials.
Phil De Luna
Jennifer N. Wei
Yoshua Bengio
Al'an Aspuru-guzik
E. Sargent
2017-11-30
Nature (published)
doi.org
Learnings Options End-to-End for Continuous Action Tasks
Martin Klissarov
Pierre-Luc Bacon
Jean Harb
Doina Precup
We present new results on learning temporally extended actions for continuoustasks, using the options framework (Suttonet al.[1999b], Precup… (see more) [2000]). In orderto achieve this goal we work with the option-critic architecture (Baconet al.[2017])using a deliberation cost and train it with proximal policy optimization (Schulmanet al.[2017]) instead of vanilla policy gradient. Results on Mujoco domains arepromising, but lead to interesting questions aboutwhena given option should beused, an issue directly connected to the use of initiation sets.
2017-11-29
ArXiv (preprint)
arxiv.org
arxiv.org
Measuring the tendency of CNNs to Learn Surface Statistical Regularities
Jason Jo
Yoshua Bengio
Deep CNNs are known to exhibit the following peculiarity: on the one hand they generalize extremely well to a test set, while on the other h… (see more)and they are extremely sensitive to so-called adversarial perturbations. The extreme sensitivity of high performance CNNs to adversarial examples casts serious doubt that these networks are learning high level abstractions in the dataset. We are concerned with the following question: How can a deep CNN that does not learn any high level semantics of the dataset manage to generalize so well? The goal of this article is to measure the tendency of CNNs to learn surface statistical regularities of the dataset. To this end, we use Fourier filtering to construct datasets which share the exact same high level abstractions but exhibit qualitatively different surface statistical regularities. For the SVHN and CIFAR-10 datasets, we present two Fourier filtered variants: a low frequency variant and a randomly filtered variant. Each of the Fourier filtering schemes is tuned to preserve the recognizability of the objects. Our main finding is that CNNs exhibit a tendency to latch onto the Fourier image statistics of the training dataset, sometimes exhibiting up to a 28% generalization gap across the various test sets. Moreover, we observe that significantly increasing the depth of a network has a very marginal impact on closing the aforementioned generalization gap. Thus we provide quantitative evidence supporting the hypothesis that deep CNNs tend to learn surface statistical regularities in the dataset rather than higher-level abstract concepts.
2017-11-29
ArXiv (preprint)
arxiv.org
arxiv.org
Ubenwa: Cry-based Diagnosis of Birth Asphyxia
Charles Onu
Innocent Udeogu
Eyenimi Ndiomu
Urbain Kengni
Doina Precup
Guilherme M. Sant’Anna
E. Alikor
P. Opara
Every year, 3 million newborns die within the first month of life. Birth asphyxia and other breathing-related conditions are a leading cause… (see more) of mortality during the neonatal phase. Current diagnostic methods are too sophisticated in terms of equipment, required expertise, and general logistics. Consequently, early detection of asphyxia in newborns is very difficult in many parts of the world, especially in resource-poor settings. We are developing a machine learning system, dubbed Ubenwa, which enables diagnosis of asphyxia through automated analysis of the infant cry. Deployed via smartphone and wearable technology, Ubenwa will drastically reduce the time, cost and skill required to make accurate and potentially life-saving diagnoses.
2017-11-16
ArXiv (preprint)
arxiv.org
arxiv.org
Design of a Recognition System Automatic Vehicle License Plate through a Convolution Neural Network
P. Rajendra
K. Sudheer
Rahul Boadh
TE Campos
BR Babu
M. Varma
Ian J Goodfellow
Yoshua Bengio
Aaron
The present work is a study on the practical application of Learning process (Deep Learning) in the development of a system of Automatic rec… (see more)ognition of vehicle license plates. These systems commonly referred to as ALPR (Automatic License Plate Recognition) - are able to recognize the content of vehicles from the images captured by a camera. The system proposed in this work is based on an image classifier developed through supervised learning techniques with convolution neural network. These networks are one of the most profound learning architectures and are specifically designed to solve artificial vision, such as pattern recognition and classification of images. This paper also examines basic processing techniques and Image segmentation - such as smoothing filters, contour detection - necessary for the proposed system to be able to extract the contents of the license plates for further analysis and classification. This paper demonstrates the feasibility of an ALPR system based on a convolution neural network, noting the critical importance it has to design a network architecture and training data set appropriate to the problem to be solved.
2017-11-14
International Journal of Computer Applications (published)
doi.org
Variational Bi-LSTMs
Samira Shabanian
Devansh Arpit
Adam Trischler
Yoshua Bengio
Recurrent neural networks like long short-term memory (LSTM) are important architectures for sequential prediction tasks. LSTMs (and RNNs in… (see more) general) model sequences along the forward time direction. Bidirectional LSTMs (Bi-LSTMs), which model sequences along both forward and backward directions, generally perform better at such tasks because they capture a richer representation of the data. In the training of Bi-LSTMs, the forward and backward paths are learned independently. We propose a variant of the Bi-LSTM architecture, which we call Variational Bi-LSTM, that creates a dependence between the two paths (during training, but which may be omitted during inference). Our model acts as a regularizer and encourages the two networks to inform each other in making their respective predictions using distinct information. We perform ablation studies to better understand the different components of our model and evaluate the method on various benchmarks, showing state-of-the-art performance.
2017-11-14
ArXiv (preprint)
arxiv.org
arxiv.org
Actual: Actor-Critic Under Adversarial Learning
Anirudh Goyal
Nan Rosemary Ke
Alex Lamb
R Devon Hjelm
Chris Pal
Joelle Pineau
Yoshua Bengio
Generative Adversarial Networks (GANs) are a powerful framework for deep generative modeling. Posed as a two-player minimax problem, GANs ar… (see more)e typically trained end-to-end on real-valued data and can be used to train a generator of high-dimensional and realistic images. However, a major limitation of GANs is that training relies on passing gradients from the discriminator through the generator via back-propagation. This makes it fundamentally difficult to train GANs with discrete data, as generation in this case typically involves a non-differentiable function. These difficulties extend to the reinforcement learning setting when the action space is composed of discrete decisions. We address these issues by reframing the GAN framework so that the generator is no longer trained using gradients through the discriminator, but is instead trained using a learned critic in the actor-critic framework with a Temporal Difference (TD) objective. This is a natural fit for sequence modeling and we use it to achieve improvements on language modeling tasks over the standard Teacher-Forcing methods.
2017-11-12
ArXiv (preprint)
doi.org
arxiv.org
Three Factors Influencing Minima in SGD
Stanisław Jastrzębski
Zac Kenton
Devansh Arpit
Nicolas Ballas
Asja Fischer
Yoshua Bengio
Amos Storkey
We study the statistical properties of the endpoint of stochastic gradient descent (SGD). We approximate SGD as a stochastic differential eq… (see more)uation (SDE) and consider its Boltzmann Gibbs equilibrium distribution under the assumption of isotropic variance in loss gradients.. Through this analysis, we find that three factors – learning rate, batch size and the variance of the loss gradients – control the trade-off between the depth and width of the minima found by SGD, with wider minima favoured by a higher ratio of learning rate to batch size. In the equilibrium distribution only the ratio of learning rate to batch size appears, implying that it’s invariant under a simultaneous rescaling of each by the same amount. We experimentally show how learning rate and batch size affect SGD from two perspectives: the endpoint of SGD and the dynamics that lead up to it. For the endpoint, the experiments suggest the endpoint of SGD is similar under simultaneous rescaling of batch size and learning rate, and also that a higher ratio leads to flatter minima, both findings are consistent with our theoretical analysis. We note experimentally that the dynamics also seem to be similar under the same rescaling of learning rate and batch size, which we explore showing that one can exchange batch size and learning rate in a cyclical learning rate schedule. Next, we illustrate how noise affects memorization, showing that high noise levels lead to better generalization. Finally, we find experimentally that the similarity under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small.
2017-11-12
ArXiv (preprint)
openreview.net
openreview.net