Publications

Hierarchical Memory Networks

Gerald Tesauro

Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is oft… (see more)en addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.

2015-12-31

CoRR (published)

doi.org

arxiv.org

Medical Computer Vision and Bayesian and Graphical Models for Biomedical Imaging

Henning Müller

B. Kelm

Tal Arbel

Weidong (Tom) Cai

M. Jorge Cardoso

Georg Langs

Bjoern Menze

Dimitris N. Metaxas

Albert A. Montillo

William Wells

Shaoting Zhang

Albert C.S. Chung

M. Jenkinson

Annemie Ribbens

2015-12-31

Lecture Notes in Computer Science (published)

doi.org

Professor Forcing: A New Algorithm for Training Recurrent Networks

The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the networ… (see more)k’s own one-step-ahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of the recurrent network to be the same when training the network and when sampling from the network over multiple time steps. We apply Professor Forcing to language modeling, vocal synthesis on raw waveforms, handwriting generation, and image generation. Empirically we find that Professor Forcing acts as a regularizer, improving test likelihood on character level Penn Treebank and sequential MNIST. We also find that the model qualitatively improves samples, especially when sampling for a large number of time steps. This is supported by human evaluation of sample quality. Trade-offs between Professor Forcing and Scheduled Sampling are discussed. We produce T-SNEs showing that Professor Forcing successfully makes the dynamics of the network during training and sampling more similar.

2015-12-31

Advances in Neural Information Processing Systems 29 (NIPS 2016) (published)

doi.org

arxiv.org

Theano: A Python framework for fast computation of mathematical expressions

The Theano Development Team

Rami Al-Rfou

Guillaume Alain

Amjad Almahairi

Christof Angermueller

Dzmitry Bahdanau

Nicolas Ballas

Frédéric Bastien

Justin Bayer

Anatoly Belikov

Alexander Belopolsky

Josh Bleecher Snyder

Nicolas Boulanger-Lewandowski

Xavier Bouthillier

Alexandre De Brébisson … (see 93 more)

Pierre-Luc Carrier

Paul Christiano

Myriam Côté

Yann N. Dauphin

Julien Demouth

Sander Dieleman

Samira Ebrahimi Kahou

Ziye Fan

Mathieu Germain

Matt Graham

Balázs Hidasi

Arjun Jain

Kai Jia

Mikhail Korobov

Vivek Kulkarni

Alex Lamb

Pascal Lamblin

Eric Larsen

César Laurent

Sean Lee

Simon Lefrancois

Simon Lemieux

Nicholas Léonard

Zhouhan Lin

Jesse A. Livezey

Cory Lorenz

Jeremiah Lowin

Qianli Ma

Pierre-Antoine Manzagol

Robert T. McGibbon

Mehdi Mirza

Alberto Orlandi

Christopher Pal

Razvan Pascanu

Mohammad Pezeshki

Colin Raffel

Daniel Renshaw

Matthew Rocklin

Adriana Romero

Markus Roth

Peter Sadowski

John Salvatier

François Savard

Jan Schlüter

John Schulman

Gabriel Schwartz

Iulian Vlad Serban

Dmitriy Serdyuk

Samira Shabanian

Etienne Simon

Sigurd Spieckermann

S. Ramana Subramanyam

Gijs van Tulder

Sebastian Urban

Dustin J. Webb

Matthew Willson

Lijun Xue

Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficie… (see more)ntly. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models. The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.

2015-12-31

Computing Research Repository (published)

doi.org

arxiv.org

Analog-to-Stochastic Converter Using Magnetic Tunnel Junction Devices for Vision Chips

Naoya Onizawa

Daisaku Katagiri

Warren J. Gross

Takahiro Hanyu

This paper introduces an analog-to-stochastic converter using a magnetic tunnel junction (MTJ) device for vision chips based on stochastic c… (see more)omputation. Stochastic computation has been recently exploited for area-efficient hardware implementation, such as low-density parity-check (LDPC) decoders and image processors. However, power-and-area hungry two-step (analog-to-digital and digital-to-stochastic) converters are required for the analog to stochastic signal conversion. To realize a one-step conversion, an MTJ device is used as it inherently exhibits a probabilistic switching behavior between two resistance states. Exploiting the device-based probabilistic behavior, analog signals can be directly and area-efficiently converted to stochastic signals to mitigate the signal-conversion overhead. The analog-to-stochastic signal conversion is theoretically described and the conversion characteristic is evaluated using device and circuit parameters. In addition, the resistance variability of the MTJ device is considered in order to compensate the variability effect on the signal conversion. Based on the theoretical analysis, the analog-to-stochastic converter is designed in 90nm CMOS and 100nm MTJ technologies and is verified using a SPICE simulator (NS-SPICE) that handles both transistors and MTJ devices.

2015-12-21

IEEE Transactions on Nanotechnology (published)

doi.org

arxiv.org

Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks

Kyunghyun Cho

Aaron Courville

Yoshua Bengio

Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output p… (see more)roblems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input. In this paper we focus on the case where the input also has a rich structure and the input and output structures are somehow related. We describe systems that learn to attend to different places in the input, for each element of the output, for a variety of tasks: machine translation, image caption generation, video clip description, and speech recognition. All these systems are based on a shared set of building blocks: gated recurrent neural networks and convolutional neural networks, along with trained attention mechanisms. We report on experimental results with these systems, showing impressively good performance and the advantage of the attention mechanism.

2015-10-31

IEEE Transactions on Multimedia (published)

doi.org

arxiv.org

Poisson Group Testing: A Probabilistic Model for Boolean Compressed Sensing

Amin Emad

Olgica Milenkovic

We introduce a novel probabilistic group testing framework, termed Poisson group testing, in which the number of defectives follows a right-… (see more)truncated Poisson distribution. The Poisson model has a number of new applications, including dynamic testing with diminishing relative rates of defectives. We consider both nonadaptive and semi-adaptive identification methods. For nonadaptive methods, we derive a lower bound on the number of tests required to identify the defectives with a probability of error that asymptotically converges to zero; in addition, we propose test matrix constructions for which the number of tests closely matches the lower bound. For semiadaptive methods, we describe a lower bound on the expected number of tests required to identify the defectives with zero error probability. In addition, we propose a stage-wise reconstruction algorithm for which the expected number of tests is only a constant factor away from the lower bound. The methods rely only on an estimate of the average number of defectives, rather than on the individual probabilities of subjects being defective.

2015-08-14

IEEE Transactions on Signal Processing (published)

doi.org

arxiv.org

Clinical Image-Based Procedures. Translational Research in Medical Imaging

Ian J. Gerard

Marta Kersten-Oertel

Simon Drouin

Jeffery Alan Hall

Kevin Petrecca

Dante De Nigris

Tal Arbel

D. Collins

2014-12-31

Lecture Notes in Computer Science (published)

doi.org

ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation

Francesco Visin

Adriana Romero

Kyunghyun Cho

Matteo Matteucci

Marco Ciccone

Kyle Kastner

Yoshua Bengio

Aaron Courville

We propose a structured prediction architecture, which exploits the local generic features extracted by Convolutional Neural Networks and th… (see more)e capacity of Recurrent Neural Networks (RNN) to retrieve distant dependencies. The proposed architecture, called ReSeg, is based on the recently introduced ReNet model for image classification. We modify and extend it to perform the more challenging task of semantic segmentation. Each ReNet layer is composed of four RNN that sweep the image horizontally and vertically in both directions, encoding patches or activations, and providing relevant global information. Moreover, ReNet layers are stacked on top of pre-trained convolutional layers, benefiting from generic local features. Upsampling layers follow ReNet layers to recover the original image resolution in the final predictions. The proposed ReSeg architecture is efficient, flexible and suitable for a variety of semantic segmentation tasks. We evaluate ReSeg on several widely-used semantic segmentation datasets: Weizmann Horse, Oxford Flower, and CamVid; achieving state-of-the-art performance. Results show that ReSeg can act as a suitable architecture for semantic segmentation tasks, and may have further applications in other structured prediction problems. The source code and model hyperparameters are available on https://github.com/fvisin/reseg.

2014-12-31

CoRR (published)

doi.org

arxiv.org

A Scalable Successive-Cancellation Decoder for Polar Codes

Alexandre J. Raymond

Warren J. Gross

Polar codes are the first error-correcting codes to provably achieve channel capacity, asymptotically in code length, with an explicit const… (see more)ruction. However, under successive-cancellation decoding, polar codes require very long code lengths to compete with existing modern codes. Nonetheless, the successive cancellation algorithm enables very-low-complexity implementations in hardware, due to the regular structure exhibited by polar codes. In this paper, we present an improved architecture for successive-cancellation decoding of polar codes, making use of a novel semi-parallel, encoder-based partial-sum computation module. We also provide quantization results for realistic code length N=215, and explore various optimization techniques such as a chained processing element and a variable quantization scheme. This design is shown to scale to code lengths of up to N=221, enabled by its low logic use, low register use and simple datapaths, limited almost exclusively by the amount of available SRAM. It also supports an overlapped loading of frames, allowing full-throughput decoding with a single set of input buffers.

2014-10-14

IEEE Transactions on Signal Processing (published)

doi.org

arxiv.org

High-Throughput Energy-Efficient LDPC Decoders Using Differential Binary Message Passing

Kevin Cushon

Saied Hemati

Camille Leroux

Shie Mannor

Warren J. Gross

In this paper, we present energy-efficient architectures for decoders of low-density parity check (LDPC) codes using the differential decodi… (see more)ng with binary message passing (DD-BMP) algorithm and its modified variant (MDD-BMP). We also propose an improved differential binary (IDB) decoding algorithm. These algorithms offer significant intrinsic advantages in the energy domain: simple computations, low interconnect complexity, and very high throughput, while achieving error correction performance up to within 0.25 dB of the offset min-sum algorithm. We report on fully parallel decoder implementations of (273, 191), (1023, 781), and (4095, 3367) finite geometry-based LDPC codes in 65 nm CMOS. Using the MDD-BMP algorithm, these decoders achieve respective areas of 0.28 mm2, 1.38 mm2, and 15.37 mm2, average throughputs of 37 Gbps, 75 Gbps, and 141 Gbps, and energy efficiencies of 4.9 pJ/bit, 13.2 pJ/bit, and 37.9 pJ/bit with a 1.0 V supply voltage in post-layout simulations. At a reduced supply voltage of 0.8 V, these decoders achieve respective throughputs of 26 Gbps, 54 Gbps, and 94 Gbps, and energy efficiencies of 3.1 pJ/bit, 8.2 pJ/bit, and 23.5 pJ/bit. We also report on a fully parallel implementation of IDB for the (2048, 1723) LDPC code specified in the IEEE 802.3an (10GBASE-T) standard. This decoder achieves an area of 1.44 mm2, average throughput of 172 Gbps, and an energy efficiency of 2.8 pJ/bit with a 1.0 V supply voltage; at 0.8 V, it achieves throughput of 116 Gbps and energy efficiency of 1.7 pJ/bit.

2014-01-31

IEEE Transactions on Signal Processing (published)

doi.org

Bayesian and grAphical Models for Biomedical Imaging

M. Jorge Cardoso

Ivor J. A. Simpson

Arbel, Tal

Doina Precup

Annemie Ribbens

2013-12-31

Lecture Notes in Computer Science (published)

doi.org

Mila Ventures Launchpad

AI Policy Compass

Virtual Information Sessions

Publications

Mila Ventures Launchpad

AI Policy Compass

Virtual Information Sessions

Popular keywords:

Publications