Portrait de Bart Van Merriënboer n'est pas disponible

Bart Van Merriënboer

Alumni

Publications

The Search for Squawk: Agile Modeling in Bioacoustics
Vincent Dumoulin
Otilia Stretcu
Jenny Hamer
Lauren Harrell
Rob Laber
Hugo Larochelle
Bart Van Merriënboer
Amanda K. Navine
Patrick Hart
Ben Williams
Timothy A. C. Lamont
Tries B. Rasak
Mars Coral Restoration Team
Sheryn Brodie
Brendan Doohan
Philip Eichinski
Paul Roe
Lin Schwarzkopf
Tom Denton
2025-05-04
ArXiv (prépublication)
arxiv.org
arxiv.org
Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis
Courtney Paquette
Bart Van Merriënboer
Fabian Pedregosa
Average-case analysis computes the complexity of an algorithm averaged over all possible inputs. Compared to worst-case analysis, it is more… (voir plus) representative of the typical behavior of an algorithm, but remains largely unexplored in optimization. One difficulty is that the analysis can depend on the probability distribution of the inputs to the model. However, we show that this is not the case for a class of large-scale problems trained with first-order methods including random least squares and one-hidden layer neural networks with random weights. In fact, the halting time exhibits a universality property: it is independent of the probability distribution. With this barrier for average-case analysis removed, we provide the first explicit average-case convergence rates showing a tighter complexity not captured by traditional worst-case analysis. Finally, numerical simulations suggest this universality property holds for a more general class of algorithms and problems.
2022-02-14
Foundations of Computational Mathematics (publié)
doi.org
arxiv.org
On the interplay between noise and curvature and its effect on optimization and generalization
Valentin Thomas
Fabian Pedregosa
Bart Van Merriënboer
Pierre-Antoine Mangazol
Yoshua Bengio
Nicolas Le Roux
The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the v… (voir plus)ariance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ultimate goal is good generalization performance, we clarify how both curvature and noise are relevant to properly estimate the generalization gap. Realizing that the limitations of some existing works stems from a confusion between these matrices, we also clarify the distinction between the Fisher matrix, the Hessian, and the covariance matrix of the gradients.
2019-12-31
AISTATS (publié)
doi.org
proceedings.mlr.press
Information matrices and generalization
Valentin Thomas
Fabian Pedregosa
Bart Van Merriënboer
Pierre-Antoine Manzagol
Yoshua Bengio
Nicolas Roux
This work revisits the use of information criteria to characterize the generalization of deep learning models. In particular, we empirically… (voir plus) demonstrate the effectiveness of the Takeuchi information criterion (TIC), an extension of the Akaike information criterion (AIC) for misspecified models, in estimating the generalization gap, shedding light on why quantities such as the number of parameters cannot quantify generalization. The TIC depends on both the Hessian of the loss H and the covariance of the gradients C. By exploring the similarities and differences between these two matrices as well as the Fisher information matrix F, we study the interplay between noise and curvature in deep models. We also address the question of whether C is a reasonable approximation to F, as is commonly assumed.
2019-06-17
arXiv.org (prépublication)
dblp.uni-trier.de
Automatic differentiation in ML: Where we are and where we should be going
Bart Van Merriënboer
Olivier Breuleux
Arnaud Bergeron
Pascal Lamblin
We review the current state of automatic differentiation (AD) for array programming in machine learning (ML), including the different approa… (voir plus)ches such as operator overloading (OO) and source transformation (ST) used for AD, graph-based intermediate representations for programs, and source languages. Based on these insights, we introduce a new graph-based intermediate representation (IR) which specifically aims to efficiently support fully-general AD for array programming. Unlike existing dataflow programming representations in ML frameworks, our IR naturally supports function calls, higher-order functions and recursion, making ML models easier to implement. The ability to represent closures allows us to perform AD using ST without a tape, making the resulting derivative (adjoint) program amenable to ahead-of-time optimization using tools from functional language compilers, and enabling higher-order derivatives. Lastly, we introduce a proof of concept compiler toolchain called Myia which uses a subset of Python as a front end.
2017-12-31
NeurIPS (publié)
arxiv.org
arxiv.org
Automatic Differentiation in Myia
Olivier Breuleux
Bart Van Merriënboer
Automatic differentiation is an essential feature of machine learning frameworks. However, its implementation in existing frameworks often h… (voir plus)as limitations. In dataflow programming frameworks such as Theano or TensorFlow the representation used makes supporting higher-order gradients difficult. On the other hand, operator overloading frameworks such as PyTorch are flexible, but do not lend themselves well to optimization. With Myia, we attempt to have the best of both worlds: Building on the work by Pearlmutter and Siskind we implement a first-order gradient operator for a subset of the Python programming language.
2017-10-27
(publié)
www.semanticscholar.org
Multiscale sequence modeling with a learned dictionary
Bart Van Merriënboer
Amartya Sanyal
Hugo Larochelle
Yoshua Bengio
We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predi… (voir plus)ctions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that the model is trained with. When applied to language modelling, our model has the flexibility of character-level models while maintaining many of the performance benefits of word-level models. Our experiments show that this model performs better than a regular LSTM on language modeling tasks, especially for smaller models.
2017-07-02
ArXiv (prépublication)
arxiv.org
arxiv.org
Theano: A Python framework for fast computation of mathematical expressions
Rami Al-Rfou
Guillaume Alain
Amjad Almahairi
Christof Angermueller
Dzmitry Bahdanau
Nicolas Ballas
Frédéric Bastien
Justin Bayer
Anatoly Belikov
Alexander Belopolsky
Yoshua Bengio
Arnaud Bergeron
James Bergstra
Valentin Bisson
Josh Bleecher Snyder
Nicolas Bouchard
Nicolas Boulanger-Lewandowski
Xavier Bouthillier
Alexandre De Brébisson
Olivier Breuleux … (voir 92 de plus)
Pierre-Luc Carrier
Kyunghyun Cho
Jan Chorowski
Paul Christiano
Tim Cooijmans
Marc-Alexandre Côté
Myriam Côté
Aaron Courville
Yann N. Dauphin
Olivier Delalleau
Julien Demouth
Guillaume Desjardins
Sander Dieleman
Laurent Dinh
Mélanie Ducoffe
Vincent Dumoulin
Samira Ebrahimi Kahou
Dumitru Erhan
Ziye Fan
Orhan Firat
Mathieu Germain
Xavier Glorot
Ian Goodfellow
Matt Graham
Caglar Gulçehre
Philippe Hamel
Iban Harlouchet
Jean-philippe Heng
Balázs Hidasi
Sina Honari
Arjun Jain
Sébastien Jean
Kai Jia
Mikhail Korobov
Vivek Kulkarni
Alex Lamb
Pascal Lamblin
Eric Larsen
César Laurent
Sean Lee
Simon Lefrancois
Simon Lemieux
Nicholas Léonard
Zhouhan Lin
Jesse A. Livezey
Cory Lorenz
Jeremiah Lowin
Qianli Ma
Pierre-Antoine Manzagol
Olivier Mastropietro
Robert T. McGibbon
Roland Memisevic
Bart Van Merriënboer
Vincent Michalski
Mehdi Mirza
Alberto Orlandi
Christopher Pal
Razvan Pascanu
Mohammad Pezeshki
Colin Raffel
Daniel Renshaw
Matthew Rocklin
Adriana Romero
Markus Roth
Peter Sadowski
John Salvatier
François Savard
Jan Schlüter
John Schulman
Gabriel Schwartz
Iulian Vlad Serban
Dmitriy Serdyuk
Samira Shabanian
Etienne Simon
Sigurd Spieckermann
S. Ramana Subramanyam
Jakub Sygnowski
Jeremie Tanguay
Gijs van Tulder
Joseph Turian
Sebastian Urban
Pascal Vincent
Francesco Visin
Harm de Vries
David Warde-Farley
Dustin J. Webb
Matthew Willson
Kelvin Xu
Lijun Xue
Li Yao
Saizheng Zhang
Ying Zhang
Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficie… (voir plus)ntly. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models. The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.
2015-12-31
arXiv (prépublication)
doi.org
arxiv.org