Doina Precup

Sumana Basu

Collaborating Alumni - McGill University

Co-supervisor :

Adriana Romero Soriano

Collaborating Alumni - McGill University

Lynn Cherif

Collaborating Alumni - McGill University

Co-supervisor :

PhD - McGill University

Co-supervisor :

PhD - McGill University

Principal supervisor :

David Meger

Jonathan Colaço Carr

Master's Research - McGill University

Principal supervisor :

Prakash Panangaden

Élodie Coté-Gauthier

Collaborating researcher - McGill University

Co-supervisor :

Isabeau Prémont-Schwarz

Franco Del Balso

Collaborating researcher - Université de Montréal

Jesse Farebrother

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

PhD - McGill University

Principal supervisor :

Collaborating researcher - Birla Institute of Technology

Howard Huang

PhD - McGill University

Haque Ishfaq

Collaborating Alumni - McGill University

Mohammad Sami Nur Islam Islam

Master's Research - McGill University

Arushi Jain

Collaborating Alumni - McGill University

PhD - Polytechnique Montréal

Martin Klissarov

PhD - McGill University

Postdoctorate - McGill University

Jonathan Lebensold

Collaborating Alumni - McGill University

Collaborating Alumni - McGill University

Ray Luo

PhD - McGill University

Principal supervisor :

G McCracken

PhD - McGill University

Nazanin Mohammadi Sepahvand

Collaborating Alumni - McGill University

Shahrad Mohammadzadeh

Master's Research - McGill University

Principal supervisor :

Gabriela Moisescu-Pareja

Collaborating researcher - McGill University

Co-supervisor :

Irina Rish

Padideh Nouri

PhD - Université de Montréal

Co-supervisor :

PhD - McGill University

Co-supervisor :

Nate Rahn

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

Manoosh Samiei

PhD - McGill University

Co-supervisor :

PhD - McGill University

Co-supervisor :

PhD - McGill University

Nishanth Anand Vemgal

PhD - McGill University

PhD - McGill University

Co-supervisor :

Samira Ebrahimi Kahou

Zihan Wang

PhD - McGill University

Skipper: Combining Spatial and Temporal Abstraction for Better Generalization

Guangyuan Wang

Research Intern - McGill University

Steve Wen

Master's Research - McGill University

Co-supervisor :

Gregory Dudek

Zijing Wu

PhD - McGill University

Principal supervisor :

PhD - McGill University

Harry Zhao

Collaborating Alumni - McGill University

Co-supervisor :

Blog Posts

Generic thumbnail for Mila Blog articles.

February 22, 2024

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Read the article

Publications

Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control … (see more)tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.

2017-06-16

ICML.cc/2017/RML (poster)

openreview.net

Investigating Recurrence and Eligibility Traces in Deep Q-Networks

Jean Harb

Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowl… (see more)edge back over time-steps in a single update. We investigate the use of eligibility traces in combination with recurrent networks in the Atari domain. We illustrate the benefits of both recurrent nets and eligibility traces in some Atari games, and highlight also the importance of the optimization used in the training.

2017-04-17

ArXiv (preprint)

openreview.net

Multi-Timescale, Gradient Descent, Temporal Difference Learning with Linear Options

Peeyush T. Kumar

Deliberating on large or continuous state spaces have been long standing challenges in reinforcement learning. Temporal Abstraction have som… (see more)ewhat made this possible, but efficiently planing using temporal abstraction still remains an issue. Moreover using spatial abstractions to learn policies for various situations at once while using temporal abstraction models is an open problem. We propose here an efficient algorithm which is convergent under linear function approximation while planning using temporally abstract actions. We show how this algorithm can be used along with randomly generated option models over multiple time scales to plan agents which need to act real time. Using these randomly generated option models over multiple time scales are shown to reduce number of decision epochs required to solve the given task, hence effectively reducing the time needed for deliberation.

2017-03-18

ArXiv (preprint)

Learning-based interactive segmentation using the maximum mean cycle weight formalism

S. Nilufar

D. S. Wang

J. Girgis

C. G. Palii

D. Yang

A. Blais

M. Brand

T. J. Perkins

The maximum mean cycle weight (MMCW) segmentation framework is a graph-based alternative to approaches such as GraphCut or Markov Random Fie… (see more)lds. It offers time- and space-efficient computation and guaranteed optimality. However, unlike GraphCut or Markov Random Fields, MMCW does not seek to segment the entire image, but rather to find the single best object within the image, according to an objective function encoded by edge weights. Its focus on a single, best object makes MMCW attractive to interactive segmentation settings, where the user indicates which objects are to be segmented. However, a provably correct way of performing interactive segmentation using the MMCW framework has never been established. Further, the question of how to develop a good objective function based on user-provided information has never been addressed. Here, we propose a three-component objective function specifically designed for use with interactive MMCW segmentation. Two of those components, representing object boundary and object interior information, can be learned from a modest amount of user-labelled data, but in a way unique to the MMCW framework. The third component allows us to extend the MMCW framework to the situation of interactive segmentation. Specifically, we show that an appropriate weighted combination of the three components guarantees that the object produced by MMCW segmentation will enclose user-specified pixels that can be chosen interactively. The component weights can either be computed a priori based on image characteristics, or online via an adaptive reweighting scheme. We demonstrate the success of the approach on several microscope image segmentation problems.

2017-02-23

Medical Imaging (published)

The Option-Critic Architecture

Pierre-Luc Bacon

Jean Harb

Temporal abstraction is key to scaling up learning and planning in reinforcement learning. While planning with temporally extended actions i… (see more)s well understood, creating such abstractions autonomously from data has remained challenging. We tackle this problem in the framework of options [Sutton, Precup & Singh, 1999; Precup, 2000]. We derive policy gradient theorems for options and propose a new option-critic architecture capable of learning both the internal policies and the termination conditions of options, in tandem with the policy over options, and without the need to provide any additional rewards or subgoals. Experimental results in both discrete and continuous environments showcase the flexibility and efficiency of the framework.

2017-02-12

Proceedings of the AAAI Conference on Artificial Intelligence (published)

Real-Time Indoor Localization in Smart Homes Using Semi-Supervised Learning

Negar Ghourchian

Michel Allegue‐martínez

Long-term automated monitoring of residential or small in- dustrial properties is an important task within the broader scope of human activi… (see more)ty recognition. We present a device- free wifi-based localization system for smart indoor spaces, developed in a collaboration between McGill University and Aerˆıal Technologies. The system relies on existing wifi net- work signals and semi-supervised learning, in order to au- tomatically detect entrance into a residential unit, and track the location of a moving subject within the sensing area. The implemented real-time monitoring platform works by detect- ing changes in the characteristics of the wifi signals collected via existing off-the-shelf wifi-enabled devices in the environ- ment. This platform has been deployed in several apartments in the Montreal area, and the results obtained show the poten- tial of this technology to turn any regular home with an ex- isting wifi network into a smart home equipped with intruder alarm and room-level location detector. The machine learn- ing component has been devised so as to minimize the need for user annotation and overcome temporal instabilities in the input signals. We use a semi-supervised learning framework which works in two phases. First, we build a base learner for mapping wifi signals to different physical locations in the en- vironment from a small amount of labeled data; during its lifetime, the learner automatically re-trains when the uncer- tainty level rises significantly, without the need for further supervision. This paper describes the technical and practical issues arising in the design and implementation of such a sys- tem for real residential units, and illustrates its performance during on-going deployment.

2017-02-10

AAAI Conference on Artificial Intelligence (published)

Independently Controllable Factors

Valentin Thomas

Jules Pondard

Emmanuel Bengio

Marc Sarfati

Philippe Beaudoin

Marie-Jean Meurs

Joelle Pineau

Yoshua Bengio

It has been postulated that a good representation is one that disentangles the underlying explanatory factors of variation. However, it rema… (see more)ins an open question what kind of training framework could potentially achieve that. Whereas most previous work focuses on the static setting (e.g., with images), we postulate that some of the causal factors could be discovered if the learner is allowed to interact with its environment. The agent can experiment with different actions and observe their effects. More specifically, we hypothesize that some of these factors correspond to aspects of the environment which are independently controllable, i.e., that there exists a policy and a learnable feature for each such aspect of the environment, such that this policy can yield changes in that feature with minimal changes to other features that explain the statistical variations in the observed data. We propose a specific objective function to find such factors and verify experimentally that it can indeed disentangle independently controllable aspects of the environment without any extrinsic reward signal.

2016-12-31

arXiv (preprint)

Independently Controllable Features

Valentin Thomas

Philippe Beaudoin

Y. Bengio

Marie-Jean Meurs

2016-12-31

(published)

A Matrix Splitting Perspective on Planning with Options

Pierre-Luc Bacon

2016-12-02

ArXiv (preprint)

Smart Classifier Selection for Activity Recognition on Wearable Devices

Negar Ghourchian

Activity recognition is a key component of human-machine interaction applications. Information obtained from sensors in smart wearable devic… (see more)es is especially valuable, because these devices have become ubiquitous, and they record large amounts of data. Machine learning algorithms can then be used to process this data. However, wearable devices impose restrictions in terms of computation and energy resources, which need to be taken into account by a learning algorithm. We propose to use a real-time learning approach, which interactively determines the most effective set of modalities (or features) for classification, given the task at hand. Our algorithm optimizes sensor selection, in order to consume less power, while still maintaining good accuracy in classifying sequences of activities. Performance on a large, noisy dataset including four different sensing modalities shows that this is a promising approach.

2016-11-22

International Conference on Pattern Recognition Applications and Methods (published)

Verb Phrase Ellipsis Resolution Using Discriminative and Margin-Infused Algorithms

Kian Kenyon-Dean

Jackie CK Cheung

Verb Phrase Ellipsis (VPE) is an anaphoric construction in which a verb phrase has been elided. It occurs frequently in dialogue and informa… (see more)l conversational settings, but despite its evident impact on event coreference resolution and extraction, there has been relatively little work on computational methods for identifying and resolving VPE. Here, we present a novel approach to detecting and resolving VPE by using supervised discriminative machine learning techniques trained on features extracted from an automatically parsed, publicly available dataset. Our approach yields state-of-the-art results for VPE detection by improving F1 score by over 11%; additionally, we explore an approach to antecedent identiﬁ-cation that uses the Margin-Infused-Relaxed-Algorithm, which shows promising results.

2016-10-31

Conference on Empirical Methods in Natural Language Processing (published)

Prediction of Cell Type Specific Transcription Factor Binding Site Occupancy

Faizy Ahsan

Mathieu Blanchette

2016-10-01

ACM International Conference on Bioinformatics, Computational Biology and Biomedicine (published)