Publications

The Consciousness Prior

Yoshua Bengio

A new prior is proposed for learning representations of high-level concepts of the kind we manipulate with language. This prior can be combi… (voir plus)ned with other priors in order to help disentangling abstract factors from each other. It is inspired by cognitive neuroscience theories of consciousness, seen as a bottleneck through which just a few elements, after having been selected by attention from a broader pool, are then broadcast and condition further processing, both in perception and decision-making. The set of recently selected elements one becomes aware of is seen as forming a low-dimensional conscious state. This conscious state is combining the few concepts constituting a conscious thought, i.e., what one is immediately conscious of at a particular moment. We claim that this architectural and information-processing constraint corresponds to assumptions about the joint distribution between high-level concepts. To the extent that these assumptions are generally true (and the form of natural language seems consistent with them), they can form a useful prior for representation learning. A low-dimensional thought or conscious state is analogous to a sentence: it involves only a few variables and yet can make a statement with very high probability of being true. This is consistent with a joint distribution (over high-level concepts) which has the form of a sparse factor graph, i.e., where the dependencies captured by each factor of the factor graph involve only very few variables while creating a strong dip in the overall energy function. The consciousness prior also makes it natural to map conscious states to natural language utterances or to express classical AI knowledge in a form similar to facts and rules, albeit capturing uncertainty as well as efficient search mechanisms implemented by attention mechanisms.

2017-09-24

ArXiv (prépublication)

Neural Network Based Nonlinear Weighted Finite Automata

Tianyu Li

Guillaume Rabusseau

Weighted finite automata (WFA) can expressively model functions defined over strings but are inherently linear models. Given the recent succ… (voir plus)esses of nonlinear models in machine learning, it is natural to wonder whether ex-tending WFA to the nonlinear setting would be beneficial. In this paper, we propose a novel model of neural network based nonlinearWFA model (NL-WFA) along with a learning algorithm. Our learning algorithm is inspired by the spectral learning algorithm for WFAand relies on a nonlinear decomposition of the so-called Hankel matrix, by means of an auto-encoder network. The expressive power of NL-WFA and the proposed learning algorithm are assessed on both synthetic and real-world data, showing that NL-WFA can lead to smaller model sizes and infer complex grammatical structures from data.

2017-09-12

ArXiv (prépublication)

Predicting Future Disease Activity and Treatment Responders for Multiple Sclerosis Patients Using a Bag-of-Lesions Brain Representation

Andrew Doyle

Douglas Arnold

Tal Arbel

2017-09-09

International Conference on Medical Image Computing and Computer-Assisted Intervention (publié)

A Deep Reinforcement Learning Chatbot

Iulian V. Serban

Mathieu Germain

Michael Pieper

Nan Rosemary Ke

Sai Mudumba

Alexandre De Brébisson

Jose M. R. Sotelo

We present MILABOT: a deep reinforcement learning chatbot developed by the Montreal Institute for Learning Algorithms (MILA) for the Amazon … (voir plus)Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models. By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed significantly better than many competing systems. Due to its machine learning architecture, the system is likely to improve with additional data.

2017-09-06

ArXiv (prépublication)

Graphs in Biomedical Image Analysis, Computational Anatomy and Imaging Genetics

M. Jorge Cardoso

Tal Arbel

Enzo Ferrante

Xavier Pennec

Adrian Dalca

Sarah Parisot

S. Joshi

Nematollah Batmanghelich

Aristeidis Sotiras

Mads Lenstrup Nielsen

Mert R. Sabuncu

Tom Fletcher

Li Shen

Stanley Durrleman

Stefan H. Sommer

2017-08-31

Lecture Notes in Computer Science (publié)

Horizontal and Vertical Self-Adaptive Cloud Controller with Reward Optimization for Resource Allocation

Jesús Alejandro Cárdenes Cabré

Ricardo Sanz

Over-booking or under-booking of computing resources leads to higher cost and performance degradation of web applications. To optimize the p… (voir plus)erformance of web applications, access to the resources has to be dynamically controlled ensuring maximum cost-performance ratio of the application while fulfilling requirements. To simplify the design of dynamic cloud controllers, we propose a horizontal and vertical scalability self-aware agent defined by a self-adaptive fuzzy logic with an oriented random optimizer based on reward and memory. The algorithm dynamically adjusts the membership functions and their relationship, maximizing the reward of the system while considering the cost related to the deployment of new resources. The evaluation of the controller under real cloud workload reveals the ability of the algorithm to maximize the performance of the web application based on the target parameters given by an operator.

2017-08-31

International Conference on Cloud and Autonomic Computing (publié)

On integrating a language model into neural machine translation

2017-08-31

Computer Speech and Language (publié)

Multi-way, multilingual neural machine translation

Orhan Firat

Kyunghyun Cho

Baskaran Sankaran

F. Yarman-Vural

Yoshua Bengio

2017-08-31

Computer Speech and Language (publié)

Piecewise Latent Variables for Neural Variational Text Processing

Iulian V. Serban

Alexander G. Ororbia II

Joelle Pineau

Aaron Courville

Advances in neural variational inference have facilitated the learning of powerful directed graphical models with continuous latent variable… (voir plus)s, such as variational autoencoders. The hope is that such models will learn to represent rich, multi-modal latent factors in real-world data, such as natural language text. However, current models often assume simplistic priors on the latent variables - such as the uni-modal Gaussian distribution - which are incapable of representing complex latent factors efficiently. To overcome this restriction, we propose the simple, but highly flexible, piecewise constant distribution. This distribution has the capacity to represent an exponential number of modes of a latent target distribution, while remaining mathematically tractable. Our results demonstrate that incorporating this new latent distribution into different models yields substantial improvements in natural language processing tasks such as document modeling and natural language generation for dialogue.

2017-08-31

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (publié)

World Knowledge for Reading Comprehension: Rare Entity Prediction with Hierarchical LSTMs Using External Descriptions

Teng Long

Emmanuel Bengio

Ryan Lowe

Jackie CK Cheung

Humans interpret texts with respect to some background information, or world knowledge, and we would like to develop automatic reading compr… (voir plus)ehension systems that can do the same. In this paper, we introduce a task and several models to drive progress towards this goal. In particular, we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources. This task is challenging due to the diversity of language styles and the extremely large number of rare entities. We propose two recurrent neural network architectures which make use of external knowledge in the form of entity descriptions. Our experiments show that our hierarchical LSTM model performs significantly better at the rare entity prediction task than those that do not make use of external resources.

2017-08-31

Conference on Empirical Methods in Natural Language Processing (publié)