Publications

Benchmarking State-Merging Algorithms for Learning Regular Languages.

Adil Soubki

Jeffrey Heinz

François Coste

Faissal Ouardi

Guillaume Rabusseau

2022-12-31

International Conference on Graphics and Interaction (published)

dblp.uni-trier.de

Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision

Fernando Diaz

Across a variety of ranking tasks, researchers use reciprocal rank to measure the effectiveness for users interested in exactly one relevant… (see more) item. Despite its widespread use, evidence suggests that reciprocal rank is brittle when discriminating between systems. This brittleness, in turn, is compounded in modern evaluation settings where current, high-precision systems may be difficult to distinguish. We address the lack of sensitivity of reciprocal rank by introducing and connecting it to the concept of best-case retrieval, an evaluation method focusing on assessing the quality of a ranking for the most satisfied possible user across possible recall requirements. This perspective allows us to generalize reciprocal rank and define a new preference-based evaluation we call lexicographic precision or lexiprecision. By mathematical construction, we ensure that lexiprecision preserves differences detected by reciprocal rank, while empirically improving sensitivity and robustness across a broad set of retrieval and recommendation tasks.

2022-12-31

EVIA (published)

doi.org

arxiv.org

Block-State Transformers

Christopher Pal

State space models (SSMs) have shown impressive results on tasks that require modeling long-range dependencies and efficiently scale to long… (see more) sequences owing to their subquadratic runtime complexity. Originally designed for continuous signals, SSMs have shown superior performance on a plethora of tasks, in vision and audio; however, SSMs still lag Transformer performance in Language Modeling tasks. In this work, we propose a hybrid layer named Block-State Transformer (BST), that internally combines an SSM sublayer for long-range contextualization, and a Block Transformer sublayer for short-term representation of sequences. We study three different, and completely parallelizable, variants that integrate SSMs and block-wise attention. We show that our model outperforms similar Transformer-based architectures on language modeling perplexity and generalizes to longer sequences. In addition, the Block-State Transformer demonstrates more than tenfold increase in speed at the layer level compared to the Block-Recurrent Transformer when model parallelization is employed.

2022-12-31

arXiv (preprint)

doi.org

openreview.net

Bugs in the Data: How ImageNet Misrepresents Biodiversity

Alexandra Luccioni

David Rolnick

ImageNet-1k is a dataset often used for benchmarking machine learning (ML) models and evaluating tasks such as image recognition and object … (see more)detection. Wild animals make up 27% of ImageNet-1k but, unlike classes representing people and objects, these data have not been closely scrutinized. In the current paper, we analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set, with the participation of expert ecologists. We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled, with some classes having >90% of images incorrect. We also find that both the wildlife-related labels and images included in ImageNet-1k present significant geographical and cultural biases, as well as ambiguities such as artificial animals, multiple species in the same image, or the presence of humans. Our findings highlight serious issues with the extensive use of this dataset for evaluating ML systems, the use of such algorithms in wildlife-related tasks, and more broadly the ways in which ML datasets are commonly created and curated.

2022-12-31

AAAI (published)

doi.org

arxiv.org

Cache-Efficient Dynamic Programming MDP Solver

Jaël Champagne Gareau

Guillaume Gosset

Éric Beaudry

Vladimir Makarenkov

2022-12-31

European Conference on Artificial Intelligence (published)

doi.org

Can Ensembling Pre-processing Algorithms Lead to Better Machine Learning Fairness?

Khaled Badran

Pierre-Olivier Côté

Amanda Kolopanis

Rached Bouchoucha

Antonio Collante

Diego Elias Costa

Emad Shihab

Foutse Khomh

As machine learning (ML) systems get adopted in more critical areas, it has become increasingly crucial to address the bias that could occur… (see more) in these systems. Several fairness pre-processing algorithms are available to alleviate implicit biases during model training. These algorithms employ different concepts of fairness, often leading to conflicting strategies with consequential trade-offs between fairness and accuracy. In this work, we evaluate three popular fairness pre-processing algorithms and investigate the potential for combining all algorithms into a more robust pre-processing ensemble. We report on lessons learned that can help practitioners better select fairness algorithms for their models.

2022-12-31

Computer (published)

doi.org

arxiv.org

Can Forward Gradient Match Backpropagation?

Louis Fournier

Stéphane Rivaud

Eugene Belilovsky

Michael Eickenberg

Edouard Oyallon

Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable fo… (see more)r neural network training while avoiding problems generally associated with backpropagation gradient computation, such as locking and memorization requirements. The cost is the requirement to guess the step direction, which is hard in high dimensions. While current solutions rely on weighted averages over isotropic guess vector distributions, we propose to strongly bias our gradient guesses in directions that are much more promising, such as feedback obtained from small, local auxiliary networks. For a standard computer vision neural network, we conduct a rigorous study systematically covering a variety of combinations of gradient targets and gradient guesses, including those previously presented in the literature. We find that using gradients obtained from a local loss as a candidate direction drastically improves on random noise in Forward Gradient methods.

2022-12-31

ICML (published)

doi.org

proceedings.mlr.press

Can AI Read the Minds of Corporate Executives?

Nicolas Chapados

Zhenzhen Fan

Ruslan Goyenko

Issam Hadj Laradji

Fred Liu

Chengyu Zhang

2022-12-31