Publications

The Unsolved Challenges of LLMs as Generalist Web Agents: A Case Study
Rim Assouel
Tom Marty
Massimo Caccia
Issam Hadj Laradji
Alexandre Drouin
Sai Rajeswar
Hector Palacios
Quentin Cappart
David Vazquez
Nicolas Chapados
Maxime Gasse
Alexandre Lacoste
2023-11-07
NeurIPS.cc/2023/Workshop/FMDM (published)
openreview.net
openreview.net
30×30 biodiversity gains rely on national coordination
Isaac Eckert
Andrea Brown
Dominique Caron
Federico Riva
Laura J. Pollock
2023-11-06
Nature Communications (published)
doi.org
Coordination among leaf and fine root traits across a strong natural soil fertility gradient
Xavier Guilbeault-Mayers
Hans Lambers
Étienne Laliberté
2023-11-05
bioRxiv (preprint)
doi.org
Player-Guided AI outperforms standard AI in Sequence Alignment Puzzles
Renata Mutalova
Roman Sarrazin-Gendron
Parham Ghasemloo Gheidari
Eddie Cai
Gabriel Richard
Sébastien Caisse
Rob Knight
Mathieu Blanchette
Attila Szantner
Jérôme Waldispühl
2023-11-05
International Conference on Climate Informatics (published)
doi.org
The feature landscape of visual cortex
Rudi Tong
Ronan da Silva
Dongyan Lin
Arna Ghosh
James Wilsenach
Erica Cianfarano
Pouya Bashivan
Blake Richards
Stuart Trenholm
Understanding computations in the visual system requires a characterization of the distinct feature preferences of neurons in different visu… (see more)al cortical areas. However, we know little about how feature preferences of neurons within a given area relate to that area’s role within the global organization of visual cortex. To address this, we recorded from thousands of neurons across six visual cortical areas in mouse and leveraged generative AI methods combined with closed-loop neuronal recordings to identify each neuron’s visual feature preference. First, we discovered that the mouse’s visual system is globally organized to encode features in a manner invariant to the types of image transformations induced by self-motion. Second, we found differences in the visual feature preferences of each area and that these differences generalized across animals. Finally, we observed that a given area’s collection of preferred stimuli (‘own-stimuli’) drive neurons from the same area more effectively through their dynamic range compared to preferred stimuli from other areas (‘other-stimuli’). As a result, feature preferences of neurons within an area are organized to maximally encode differences among own-stimuli while remaining insensitive to differences among other-stimuli. These results reveal how visual areas work together to efficiently encode information about the external world.
2023-11-05
bioRxiv (preprint)
doi.org
Score-Based Likelihood Characterization for Inverse Problems in the Presence of Non-Gaussian Noise
Ronan Legin
Alexandre Adam
Yashar Hezaveh
Laurence Perreault-Levasseur
Likelihood analysis is typically limited to normally distributed noise due to the difficulty of determining the probability density function… (see more) of complex, high-dimensional, non-Gaussian, and anisotropic noise. This work presents Score-based LIkelihood Characterization (SLIC), a framework that resolves this issue by building a data-driven noise model using a set of noise realizations from observations. We show that the approach produces unbiased and precise likelihoods even in the presence of highly non-Gaussian correlated and spatially varying noise. We use diffusion generative models to estimate the gradient of the probability density of noise with respect to data elements. In combination with the Jacobian of the physical model of the signal, we use Langevin sampling to produce independent samples from the unbiased likelihood. We demonstrate the effectiveness of the method using real data from the Hubble Space Telescope and James Webb Space Telescope.
2023-11-03
NeurIPS.cc/2023/Workshop/Deep_Inverse (poster)
openreview.net
openreview.net
Empowering Clinicians with MeDT: A Framework for Sepsis Treatment
Aamer Abdul Rahman
Pranav Agarwal
Vincent Michalski
Rita Noumeir
Samira Ebrahimi Kahou
2023-11-02
NeurIPS.cc/2023/Workshop/GCRL (published)
openreview.net
openreview.net
Goal Misgeneralization as Implicit Goal Conditioning
Diego Dorn
Neel Alex
David Scott Krueger
2023-11-02
NeurIPS.cc/2023/Workshop/GCRL (published)
openreview.net
openreview.net
How does fine-tuning affect your model? Mechanistic analysis on procedural tasks
Samyak Jain
Robert Kirk
Ekdeep Singh Lubana
Robert P. Dick
Hidenori Tanaka
Tim Rocktäschel
Edward Grefenstette
David Scott Krueger
Fine-tuning large pre-trained models has become the *de facto* strategy for developing models that are safe to deploy. However, there has be… (see more)en little work that explains how fine-tuning alters the underlying capabilities learnt by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in *synthetic* settings with mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. Our extensive analysis of the effects of fine-tuning shows: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities; and (iii) further fine-tuning on a task where such wrapped capabilities are relevant leads to sample-efficient "revival'' of the capability, i.e., the model begins reusing this capability in a few gradient steps. *This indicates practitioners can unintentionally remove a model's safety wrapper by merely fine-tuning it on a superficially unrelated task.* We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.
2023-11-02
NeurIPS.cc/2023/Workshop/UniReps (poster)
openreview.net
openreview.net
Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor
Trevor Ablett
Oliver Limoyo
Adam Sigal
Affan Jilani
Jonathan Kelly
Kaleem Siddiqi
Francois R. Hogan
Gregory Dudek
Kinesthetic Teaching is a popular approach to collecting expert robotic demonstrations of contact-rich tasks for imitation learning (IL), bu… (see more)t it typically only measures motion, ignoring the force placed on the environment by the robot. Furthermore, contact-rich tasks require accurate sensing of both reaching and touching, which can be difficult to provide with conventional sensing modalities. We address these challenges with a See-Through-your-Skin (STS) visuotactile sensor, using the sensor both (i) as a measurement tool to improve kinesthetic teaching, and (ii) as a policy input in contact-rich door manipulation tasks. An STS sensor can be switched between visual and tactile modes by leveraging a semi-transparent surface and controllable lighting, allowing for both pre-contact visual sensing and during-contact tactile sensing with a single sensor. First, we propose tactile force matching, a methodology that enables a robot to match forces read during kinesthetic teaching using tactile signals. Second, we develop a policy that controls STS mode switching, allowing a policy to learn the appropriate moment to switch an STS from its visual to its tactile mode. Finally, we study multiple observation configurations to compare and contrast the value of visual and tactile data from an STS with visual data from a wrist-mounted eye-in-hand camera. With over 3,000 test episodes from real-world manipulation experiments, we find that the inclusion of force matching raises average policy success rates by 62.5%, STS mode switching by 30.3%, and STS data as a policy input by 42.5%. Our results highlight the utility of see-through tactile sensing for IL, both for data collection to allow force matching, and for policy execution to allow accurate task feedback.
2023-11-02
ArXiv (preprint)
arxiv.org
arxiv.org
Pepid: a Highly Modifiable, Bioinformatics-Oriented Peptide Search Engine
Jeremie Zumer
Sébastien Lemieux
2023-11-02
bioRxiv (preprint)
doi.org
SatBird: Bird Species Distribution Modeling with Remote Sensing and Citizen Science Data
Mélisande Teng
Amna Elmustafa
Benjamin Akera
Yoshua Bengio
Hager Radi
Hugo Larochelle
David Rolnick
Biodiversity is declining at an unprecedented rate, impacting ecosystem services necessary to ensure food, water, and human health and well-… (see more)being. Understanding the distribution of species and their habitats is crucial for conservation policy planning. However, traditional methods in ecology for species distribution models (SDMs) generally focus either on narrow sets of species or narrow geographical areas and there remain significant knowledge gaps about the distribution of species. A major reason for this is the limited availability of data traditionally used, due to the prohibitive amount of effort and expertise required for traditional field monitoring. The wide availability of remote sensing data and the growing adoption of citizen science tools to collect species observations data at low cost offer an opportunity for improving biodiversity monitoring and enabling the modelling of complex ecosystems. We introduce a novel task for mapping bird species to their habitats by predicting species encounter rates from satellite images, and present SatBird, a satellite dataset of locations in the USA with labels derived from presence-absence observation data from the citizen science database eBird, considering summer (breeding) and winter seasons. We also provide a dataset in Kenya representing low-data regimes. We additionally provide environmental data and species range maps for each location. We benchmark a set of baselines on our dataset, including SOTA models for remote sensing tasks. SatBird opens up possibilities for scalably modelling properties of ecosystems worldwide.
2023-11-02
ArXiv (preprint)
doi.org
arxiv.org