Giovanni Beltrame

haechan-mark.bong@mila.quebec

PhD - Polytechnique Montréal

Website

Google Scholar

Yann Bouteiller

Collaborating researcher - Polytechnique Montréal Montreal

yann.bouteiller@mila.quebec

PhD - Polytechnique Montréal

maeva.guerrier@mila.quebec

Github

Simon Roy

Master's Research - Université de Montréal

simon.roy@mila.quebec

Website

Github

Soma Soma

PhD - Polytechnique Montréal

karthik.soma@mila.quebec

Publications

Influence of Autonomy and Interfaces on Human and Multi-Robot Teams: a Study on Planetary Exploration

Marcel Kaufmann

Advances in robotic autonomy and interfaces have transformed human-robot teaming across domains, from disaster response to planetary science… (see more). However, critical gaps remain in understanding how autonomy and interface design affect human performance and cognitive demands, especially in large-scale, unstructured environments such as those on the Moon or Mars. We present a human-in-the-loop system comprising two (semi-)autonomous robots supervised by a single human operator. The system was evaluated in real caves at Lava Beds National Monument (California) and in a controlled within-subject study (n=38) at Polytechnique Montréal exploring both real and simulated caves. Participants interacted using either a traditional screen interface or a novel real-time, immersive VR interface, developed for this study and field-tested during NASA's BRAILLE campaign. We find that continuous physiological measurements (HRV) align with subjective NASA TLX scores in the context of human and multi-robot planetary exploration. Compared to benchmarks from prior studies, the screen interface resulted in low workload, while VR was rated in the low-to-moderate range. The low-autonomy VR-waypoint condition resulted in the least effective performance, with the fewest automated science detections, whereas both the full-autonomy VR and screen-based conditions yielded comparably higher exploration and detection performance. Both interfaces supported high situational awareness, with accuracy measures near 90%. Autonomy did not significantly affect situational awareness, but full-autonomy did reduce operator input effort. Trust levels did not significantly vary across conditions, motivating more detailed assessment methods in future studies. The results inform how to align interface design and autonomy to support effective multi-robot supervision in future missions.

2026-06-28

ACM Transactions on Human-Robot Interaction (published)

VISTA: Scale-Aware Visual Navigation via Action History Conditioning

Maeva Guerrier

Koki Kobayashi

Simon Roy

Jana Pavlasek

Vision Navigation Foundation Models (VNMs) promise end-to-end learned navigation policies capable of zero-shot deployment across diverse emb… (see more)odiments and environments. To maintain generality, many vision-based navigation models predict normalized actions. However, this normalization introduces a critical deployment vulnerability: applying different scaling factors to the same normalized trajectory alters its physical geometry, which degrades navigation performance and increases collision risks. We address this vulnerability by conditioning the model on normalized action histories alongside image observations, providing explicit context on the relationship between the model's predictions and the robot's actual physical displacement. Furthermore, current VNMs often struggle in visually repetitive environments that lack distinct features. To resolve this issue, we integrate a DINOv3 encoder, whose richer representations enable our model to capture both spatial and geometric dimensions between observations. VISTA generalizes robustly to out-of-distribution environments, achieving 100% goal prediction accuracy in zero-shot, real-world deployment in Outdoor, Forest and Office settings, and an average of 95% checkpoints crossed, demonstrating consistent path following in unseen environments.

2026-06-14

arXiv (preprint)

BlabberSeg: Semantic Perception for Reliable Open-Vocabulary UAV Safe Landing

Reliable robot autonomy requires semantic perception that remains both informative and fast enough for closed-loop safety decisions. We pres… (see more)ent BlabberSeg, an optimized CLIPSeg-based open-vocabulary segmentation pipeline for UAV emergency landing. The method targets semantic reliability under edge constraints by reusing prompt, positional, and image features and deploying floating-point 16 ONNX (TensorRT) inference. In a DOVESEI-based safe-landing workflow, BlabberSeg reaches 16.78Hz on Jetson Orin AGX (64GB), a 927.41% speed increase over the original CLIPSeg (1.81Hz), with limited degradation in segmentation agreement (2.1% relative area difference) and mIoU (9%). At the task level, safe-landing success is preserved (76/100, matching baseline) while mission time is substantially reduced. These results support semantic open-vocabulary perception as a practical component for reliable autonomous landing.

2026-05-26

SRRA @ IEEE International Conference on Robotics and Automation (poster)

openreview.net

Latent Goal Prediction from Language for Model-Based Planning

Samuel Barbeau

Simon Roy

Christian Desrosiers

Nicolas Thome

Planning with world models is bottlenecked by compounding prediction errors and the difficulty of defining optimizable goals. Visual targets… (see more) provide precise local gradients but poor distant guidance, while language is flexible yet limited by noisy cross-modal alignment or dependence on large generative models unsuited for the high-sampling nature of model-based planning. To address these challenges, we introduce Latent Goal Prediction from Language (LAGO), a framework that predicts both sequences of intermediate goal states from language instructions and action-conditioned rollouts, all within the same latent space. Rather than optimizing toward a single global objective, LAGO dynamically decomposes instructions into explicitly predicted, locally tractable latent subgoals. By updating these subgoals online and using a soft minimum trajectory cost during planning, LAGO enables an agent to follow coherent latent trajectories over long horizons. Evaluation across multiple environments planning horizons shows that LAGO avoids the sharp degradation of prior methods. By achieving robust and precise long-horizon planning purely from language, LAGO bridges the precision of visual goals with the flexibility of text-guided control.

2026-05-25

arXiv (preprint)

To Select or not to Select, that is the Question: Distilling Robot Skill Prediction into a Small Ensemble

Simon Roy

Euhid Aman

As robot fleets become more heterogeneous, including humanoids, rovers, quadrupeds, and drones, selecting the right robot for a task becomes… (see more) a core systems problem. We study robot skill prediction: mapping a natural-language task description to the physical capabilities required to execute it, such as fly, wheels, legs, surface water, under water and hands. Since labelled data that maps natural-language task descriptions to robot's physical capabilities does not exist, we construct a synthetic task-to-skill dataset using LLM-assisted generation and targeted label auditing. Trained on this data, a ~133M-parameter ensemble of two fine-tuned sentence encoders (mpnet + MiniLM) reaches 83.5% task-to-skill matching on a stratified 200 task dataset, outperforming Kimi K2 (1T MoE) at 72.0%, GPT-OSS-120B at 71.5%, and Llama-4-Scout-17B at 69.0% under the same zero-shot prompt. These results suggest that, for fixed robot skill taxonomies, small specialized models trained on synthetic data can outperform much larger general-purpose LLMs for fleet-level task routing.

2026-05-19

arXiv (preprint)

Sleep Spindle-Locked Targeted Memory Reactivation Enhances Declarative Memory Consolidation

Vaishali Mutreja

Prakriti Gupta

Ovidiu Lungu

Latifa Lazzouni

Ella Gabitov

Habib Benali

Hugo Jourde

Emily BJ Coffey

Jean-Marc Lina

Geneviève Albouy

Bradley King

Arnaud Boutin

Julie Carrier

Julien Doyon

Abstract Study Objectives Sleep spindles are implicated in memory consolidation. Yet direct evidence linking spindle dynamics to declarative… (see more) memory outcomes remains limited. We thus tested whether targeted memory reactivation (TMR) time-locked to sleep spindles enhances declarative memory, and whether the temporal organization of stimulated spindles–trains versus isolated events–is selectively associated with distinct memory outcomes. Methods Twenty-eight healthy young adults learned image locations from two categories (animals, clothing) in a grid, each paired with a distinct auditory cue. During overnight NREM sleep, one cue was replayed time-locked to spindles detected in real-time using a closed-loop system (TMR condition); the other served as the non-reactivated control (No-TMR condition). Category-cue assignment was counterbalanced. Post-sleep recall, recognition accuracy, and movement time were assessed. Results Recall accuracy was significantly higher in the TMR than the No-TMR condition (93.96% vs. 90.61%, p = .024), whereas recognition accuracy ( p = .139) and movement time ( p = .651) did not differ. Stimulation intensity within spindle trains correlated with the TMR effect on recall (Spearman ρ = .531, p = .004), whereas the proportion of isolated spindle stimulations correlated with the TMR effect on recognition (ρ = .563, p = .002). Cross-associations were not significant. Conclusions Spindle-locked TMR enhances recall-based declarative memory retention. The selective association between spindle temporal clustering and memory outcomes suggests that train-embedded and isolated spindles support different aspects of memory consolidation, highlighting spindle temporal context as a functionally relevant dimension of sleep-dependent memory processing.

2026-05-11

bioRxiv (preprint)

Safe Aerial 3D Path Planning for Autonomous UAVs using Magnetic Potential Fields

Safe autonomous Uncrewed Aerial Vehicle (UAV) navigation in urban environments requires real-time path planning that avoids obstacles. MaxCo… (see more)nvNet is a potential-field planner that leverages properties of Maxwell's equations to generate a path to the goal without local minima. We extend the 2D MaxConvNet magnetic field planner to 3D, using a convolutional autoencoder to predict obstacle-aware potential fields from LiDAR-derived 101^3 voxel grids. Evaluation across 100 randomized closed-loop trials in two distinct Cosys-AirSim urban environments, a dense night-time cityscape and a suburban district shows a 100% path planning success rate on both maps without retraining. In offline path planning, 3DMaxConvNet produces path lengths comparable to A* on unseen maps while reducing runtime from 0.155--0.17s to 0.087--0.089s, or about 1.7--1.95 times faster than A*. Against RRT*(3k), 3DMaxConvNet achieves similar path quality while reducing planning runtime from 17.2--17.5s to about 0.09s, which is roughly 193--201 times faster than RRT*(3k).

2026-05-10

arXiv (preprint)

Split over $n$ resource sharing problem: Are fewer capable agents better than many simpler ones?

Karthik Soma

Mohamed S. Talamali

Genki Miyauchi

Heiko Hamann

Roderich Groß

In multi-agent systems, should limited resources be concentrated into a few capable agents or distributed among many simpler ones? This work… (see more) formulates the split over

2026-04-28

arXiv (preprint)

Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned

Maeva Guerrier

Karthik Soma

Jana Pavlasek

Visual Navigation Models (VNMs) promise generalizable, robot navigation by learning from large-scale visual demonstrations. Despite growing … (see more)real-world deployment, existing evaluations rely almost exclusively on success rate, whether the robot reaches its goal, which conceals trajectory quality, collision behavior, and robustness to environmental change. We present a real-world evaluation of five state-of-the-art VNMs (GNM, ViNT, NoMaD, NaviBridger, and CrossFormer) across two robot platforms and five environments spanning indoor and outdoor settings. Beyond success rate, we combine path-based metrics with vision-based goal-recognition scores and assess robustness through controlled image perturbations (motion blur, sunflare). Our analysis uncovers three systematic limitations: (a) even architecturally sophisticated diffusion and transformer-based models exhibit frequent collisions, indicating limited geometric understanding; (b) models fail to discriminate between different locations that are perceptually similar, however some semantics differences are present, causing goal prediction errors in repetitive environments; and (c) performance degrades under distribution shift. We will publicly release our evaluation codebase and dataset to facilitate reproducible benchmarking of VNMs.

2026-03-25

arXiv (preprint)

Scalable Multi-Agent Reinforcement Learning Framework for Multi-Machine Tending

Abdalwhab Abdalwhab

David St-Onge

Robotic manipulators hold significant untapped potential for manufacturing industries, particularly when deployed in multi-robot configurati… (see more)ons that can enhance resource utilization, increase throughput, and reduce costs. However, industrial manipulators typically operate in isolated one-robot, one-machine setups, limiting both utilization and scalability. Even mobile robot implementations generally rely on centralized architectures, creating vulnerability to single points of failure and requiring robust communication infrastructure. This paper introduces SMAPPO (Scalable Multi-Agent Proximal Policy Optimization), a scalable input-size invariant multi-agent reinforcement learning model for decentralized multi-robot management in industrial environments. MAPPO (Multi-Agent Proximal Policy Optimization) represents the current state-of-the-art approach. We optimized an existing simulator to handle complex multi-agent reinforcement learning scenarios and designed a new multi-machine tending scenario for evaluation. Our novel observation encoder enables SMAPPO to handle varying numbers of agents, machines, and storage areas with minimal or no retraining. Results demonstrate SMAPPO's superior performance compared to the state-of-the-art MAPPO across multiple conditions: full retraining (up to 61% improvement), curriculum learning (up to 45% increased productivity and up to 49% fewer collisions), zero-shot generalization to significantly different scale scenarios (up to 272% better performance without retraining), and adaptability under extremely low initial training (up to 100% increase in parts delivery).

2026-02-28

IEEE Robotics and Automation Letters (published)

Sociodynamics of Reinforcement Learning

Yann Bouteiller

Karthik Soma

Reinforcement Learning (RL) has emerged as a core algorithmic paradigm explicitly driving innovation in a growing number of industrial appli… (see more)cations, including large language models and quantitative finance. Furthermore, computational neuroscience has long found evidence of natural forms of RL in biological brains. Therefore, it is crucial for the study of social dynamics to develop a scientific understanding of how RL shapes population behaviors. We leverage the framework of Evolutionary Game Theory (EGT) to provide building blocks and insights toward this objective. We propose a methodology that enables simulating large populations of RL agents in simple game theoretic interaction models. More specifically, we derive fast and parallelizable implementations of two fundamental revision protocols from multi-agent RL - Policy Gradient (PG) and Opponent-Learning Awareness (LOLA) - tailored for population simulations of random pairwise interactions in stateless normal-form games. Our methodology enables us to simulate large populations of 200,000 independent co-learning agents, yielding compelling insights into how non-stationarity-aware learners affect social dynamics. In particular, we find that LOLA learners promote cooperation in the Stag Hunt model, delay cooperative outcomes in the Hawk-Dove model, and reduce strategy diversity in the Rock-Paper-Scissors model.

2026-02-20

Transactions on Machine Learning Research (accepted)

openreview.net

Swarm robotics localization: comparing methods from infrared to foundation models

Ali Imran

Vivek Shankar Vardharajan

Rafael Gomes Braga

David St-Onge

2026-02-01

Swarm Intelligence (published)