Publications

PEACE: Prompt Engineering Automation for CLIPSeg Enhancement in Aerial Robotics

Haechan Mark Bong

Rongge Zhang

Ricardo de Azambuja

Giovanni Beltrame

From industrial to space robotics, safe landing is an essential component for flight operations. With the growing interest in artificial int… (voir plus)elligence, we direct our attention to learning based safe landing approaches. This paper extends our previous work, DOVESEI, which focused on a reactive UAV system by harnessing the capabilities of open vocabulary image segmentation. Prompt-based safe landing zone segmentation using an open vocabulary based model is no more just an idea, but proven to be feasible by the work of DOVESEI. However, a heuristic selection of words for prompt is not a reliable solution since it cannot take the changing environment into consideration and detrimental consequences can occur if the observed environment is not well represented by the given prompt. Therefore, we introduce PEACE (Prompt Engineering Automation for CLIPSeg Enhancement), powering DOVESEI to automate the prompt generation and engineering to adapt to data distribution shifts. Our system is capable of performing safe landing operations with collision avoidance at altitudes as low as 20 meters using only monocular cameras and image segmentation. We take advantage of DOVESEI's dynamic focus to circumvent abrupt fluctuations in the terrain segmentation between frames in a video stream. PEACE shows promising improvements in prompt generation and engineering for aerial images compared to the standard prompt used for CLIP and CLIPSeg. Combining DOVESEI and PEACE, our system was able improve successful safe landing zone selections by 58.62% compared to using only DOVESEI. All the source code is open source and available online.

2023-09-29

ArXiv (prépublication)

Tree Cross Attention

Leo Feng

Frederick Tung

Hossein Hajimirsadeghi

Yoshua Bengio

Mohamed Osama Ahmed

2023-09-29

ArXiv (prépublication)

An Empirical Study of Retrieval-Enhanced Graph Neural Networks

Dingmin Wang

Shengchao Liu

Hanchen Wang

Bernardo Cuenca Grau

Linfeng Song

Jian Tang

Le Song

Qi Liu

Graph Neural Networks (GNNs) are effective tools for graph representation learning. Most GNNs rely on a recursive neighborhood aggregation s… (voir plus)cheme, named message passing, thereby their theoretical expressive power is limited to the first-order Weisfeiler-Lehman test (1-WL). An effective approach to this challenge is to explicitly retrieve some annotated examples used to enhance GNN models. While retrieval-enhanced models have been proved to be effective in many language and vision domains, it remains an open question how effective retrieval-enhanced GNNs are when applied to graph datasets. Motivated by this, we want to explore how the retrieval idea can help augment the useful information learned in the graph neural networks, and we design a retrieval-enhanced scheme called GRAPHRETRIEVAL, which is agnostic to the choice of graph neural network models. In GRAPHRETRIEVAL, for each input graph, similar graphs together with their ground-true labels are retrieved from an existing database. Thus they can act as a potential enhancement to complete various graph property predictive tasks. We conduct comprehensive experiments over 13 datasets, and we observe that GRAPHRETRIEVAL is able to reach substantial improvements over existing GNNs. Moreover, our empirical study also illustrates that retrieval enhancement is a promising remedy for alleviating the long-tailed label distribution problem.

2023-09-28

Frontiers in Artificial Intelligence and Applications (publié)

Influence of preprocessing, distortion correction and cardiac triggering on the quality of diffusion MR images of spinal cord

Kurt G Schilling

Anna J. E. Combes

Karthik Ramadass

Francois Rheault

Grace Sweeney

Logan Prock

Subramaniam Sriram

Julien Cohen-Adad

John C Gore

Bennett A Landman

Seth A. Smith

Kristin P. O’Grady

Diffusion MRI of the spinal cord (SC) is susceptible to geometric distortion caused by field inhomogeneities, and prone to misalignment acro… (voir plus)ss time series and signal dropout caused by biological motion. Several modifications of image acquisition and image processing techniques have been introduced to overcome these artifacts, but their specific benefits are largely unproven and warrant further investigations. We aim to evaluate two specific aspects of image acquisition and processing that address image quality in diffusion studies of the spinal cord: susceptibility corrections to reduce geometric distortions, and cardiac triggering to minimize motion artifacts. First, we evaluate 4 distortion preprocessing strategies on 7 datasets of the cervical and lumbar SC and find that while distortion correction techniques increase geometric similarity to structural images, they are largely driven by the high-contrast cerebrospinal fluid, and do not consistently improve the geometry within the cord nor improve white-to-gray matter contrast. We recommend at a minimum to perform bulk-motion correction in preprocessing and posit that improvements/adaptations are needed for spinal cord distortion preprocessing algorithms, which are currently optimized and designed for brain imaging. Second, we design experiments to evaluate the impact of removing cardiac triggering. We show that when triggering is foregone, images are qualitatively similar to triggered sequences, do not have increased prevalence of artifacts, and result in similar diffusion tensor indices with similar reproducibility to triggered acquisitions. When triggering is removed, much shorter acquisitions are possible, which are also qualitatively and quantitatively similar to triggered sequences. We suggest that removing cardiac triggering for cervical SC diffusion can be a reasonable option to save time with minimal sacrifice to image quality.

2023-09-27

bioRxiv (prépublication)

RECOVER identifies synergistic drug combinations in vitro through sequential model optimization

Paul Bertin

Jarrid Rector-Brooks

Deepak Sharma

Thomas Gaudelet

Andrew Anighoro

Torsten Gross

Francisco Martínez-Peña

Eileen L. Tang

M.S. Suraj

Cristian Regep

Jeremy B.R. Hayter

Maksym Korablyov

Nicholas Valiante

Almer van der Sloot

Mike Tyers

Charles E.S. Roberts

Michael M. Bronstein

Luke L. Lairson

Jake P. Taylor-King

Yoshua Bengio

2023-09-27

Cell Reports Methods (publié)

Laurence Perreault-Levasseur

Time Delay Cosmography with a Neural Ratio Estimator

Eve Campeau-Poirier

Adam Coogan

Yashar Hezaveh

We explore the use of a Neural Ratio Estimator (NRE) to determine the Hubble constant (…

2023-09-27

ArXiv (prépublication)

ClimateSet: A Large-Scale Climate Model Dataset for Machine Learning

Julia Kaltenborn

Charlotte Emilie Elektra Lange

Venkatesh Ramesh

Philippe Brouillard

Yaniv Gurwicz

Chandni Nagda

Jakob Runge

Peer Nowack

David Rolnick

Climate models have been key for assessing the impact of climate change and simulating future climate scenarios. The machine learning (ML) c… (voir plus)ommunity has taken an increased interest in supporting climate scientists’ efforts on various tasks such as climate model emulation, downscaling, and prediction tasks. Many of those tasks have been addressed on datasets created with single climate models. However, both the climate science and ML communities have suggested that to address those tasks at scale, we need large, consistent, and ML-ready climate model datasets. Here, we introduce ClimateSet, a dataset containing the inputs and outputs of 36 climate models from the Input4MIPs and CMIP6 archives. In addition, we provide a modular dataset pipeline for retrieving and preprocessing additional climate models and scenarios. We showcase the potential of our dataset by using it as a benchmark for ML-based climate model emulation. We gain new insights about the performance and generalization capabilities of the different ML models by analyzing their performance across different climate models. Furthermore, the dataset can be used to train an ML emulator on several climate models instead of just one. Such a “super-emulator” can quickly project new climate change scenarios, complementing existing scenarios already provided to policymakers. We believe ClimateSet will create the basis needed for the ML community to tackle climate-related tasks at scale.

Evaluating Self-Supervised Learning for Molecular Graph Embeddings

Hanchen Wang

Jean Kaddour

Shengchao Liu

Jian Tang

Matt J. Kusner

Joan Lasenby

Qi Liu

Graph Self-Supervised Learning (GSSL) provides a robust pathway for acquiring embeddings without expert labelling, a capability that carries… (voir plus) profound implications for molecular graphs due to the staggering number of potential molecules and the high cost of obtaining labels. However, GSSL methods are designed not for optimisation within a specific domain but rather for transferability across a variety of downstream tasks. This broad applicability complicates their evaluation. Addressing this challenge, we present"Molecular Graph Representation Evaluation"(MOLGRAPHEVAL), generating detailed profiles of molecular graph embeddings with interpretable and diversified attributes. MOLGRAPHEVAL offers a suite of probing tasks grouped into three categories: (i) generic graph, (ii) molecular substructure, and (iii) embedding space properties. By leveraging MOLGRAPHEVAL to benchmark existing GSSL methods against both current downstream datasets and our suite of tasks, we uncover significant inconsistencies between inferences drawn solely from existing datasets and those derived from more nuanced probing. These findings suggest that current evaluation methodologies fail to capture the entirety of the landscape.

GEO-Bench: Toward Foundation Models for Earth Monitoring

Alexandre Lacoste

Nils Lehmann

Pau Rodriguez

Evan David Sherwin

Hannah Kerner

Björn Lütjens

Jeremy Andrew Irvin

David Dao

Hamed Alemohammad

Alexandre Drouin

Mehmet Gunturkun

Gabriel Huang

David Vazquez

Dava Newman

Yoshua Bengio

Stefano Ermon

Xiao Xiang Zhu

Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to subst… (voir plus)antial increases in generalization to downstream tasks. Such models, recently coined foundation models, have been transformational to the field of natural language processing. Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited. To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results to enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models. We believe that this benchmark will be a driver of progress across a variety of Earth monitoring tasks.

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

Maxime Chevalier-Boisvert

Bolun Dai

Mark Towers

Rodrigo De Lazcano Perez-Vicente

Lucas Willems

Salem Lahlou

Suman Pal

Pablo Samuel Castro

J K Terry

We present the Minigrid and Miniworld libraries which provide a suite of goal-oriented 2D and 3D environments. The libraries were explicitly… (voir plus) created with a minimalistic design paradigm to allow users to rapidly develop new environments for a wide range of research-specific needs. As a result, both have received widescale adoption by the RL community, facilitating research in a wide range of areas. In this paper, we outline the design philosophy, environment details, and their world generation API. We also showcase the additional capabilities brought by the unified API between Minigrid and Miniworld through case studies on transfer learning (for both RL agents and humans) between the different observation spaces. The source code of Minigrid and Miniworld can be found at https://github.com/Farama-Foundation/Minigrid and https://github.com/Farama-Foundation/Miniworld along with their documentation at https://minigrid.farama.org/ and https://miniworld.farama.org/.

2023-09-25

NeurIPS.cc/2023/Track/Datasets_and_Benchmarks (poster)

PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Florian Bordes

Shashank Shekhar

Mark Ibrahim

Diane Bouchacourt

Pascal Vincent

Ari S. Morcos

Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render a… (voir plus)s many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models.