Publications

Coordinating speech and body during conversation: an at-home study with 4-year-olds
Chantal-Valerie Lee
Guillaume Dumas
Ahmed Jerôme Romain
Simone Falk
2026-05-25
International Conference on Speech Prosody (published)
doi.org
How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning
Qian Yang
Ankur Sikarwar
Huy Le
Le Zhang
Zhuan Shi
Perouz Taslakian
Aishwarya Agrawal
Cross-view spatial reasoning remains a weak spot for vision-language models (VLMs): they often reason in language and lose the fine-grained … (see more)geometry needed for the task. Thinking with images aims to address this by generating an intermediate thinking image, but recent work shows that models often ignore the visual evidence in these traces. We therefore ask how to make visual thinking matter, and what kind of visual thinking works best. We study these questions in unified multimodal models (UMMs), which natively support interleaved image-text generation. For the first question, we propose View Dropout (VDrop), a training-time intervention that hides parts of one input view from the answer span while keeping them visible to the thinking-image tokens. This encourages the model to use the thinking image when answering, instead of relying only on the input views. Once the thinking image is used for answer prediction, we study which type of visual thinking is most effective. We frame this as a learnability-informativeness tradeoff and compare three thinking-image variants: top-down, panoramic, and point-matching renderings. Trained on synthetic scenes and evaluated on five real-world out-of-domain benchmarks, panoramic visual thinking with VDrop is the only configuration that is both informative and learnable, and it achieves the best out-of-domain generalization.
2026-05-25
arXiv (preprint)
doi.org
arxiv.org
Latent Goal Prediction from Language for Model-Based Planning
Samuel Barbeau
Simon Roy
Giovanni Beltrame
Christian Desrosiers
Nicolas Thome
Planning with world models is bottlenecked by compounding prediction errors and the difficulty of defining optimizable goals. Visual targets… (see more) provide precise local gradients but poor distant guidance, while language is flexible yet limited by noisy cross-modal alignment or dependence on large generative models unsuited for the high-sampling nature of model-based planning. To address these challenges, we introduce Latent Goal Prediction from Language (LAGO), a framework that predicts both sequences of intermediate goal states from language instructions and action-conditioned rollouts, all within the same latent space. Rather than optimizing toward a single global objective, LAGO dynamically decomposes instructions into explicitly predicted, locally tractable latent subgoals. By updating these subgoals online and using a soft minimum trajectory cost during planning, LAGO enables an agent to follow coherent latent trajectories over long horizons. Evaluation across multiple environments planning horizons shows that LAGO avoids the sharp degradation of prior methods. By achieving robust and precise long-horizon planning purely from language, LAGO bridges the precision of visual goals with the flexibility of text-guided control.
2026-05-25
arXiv (preprint)
doi.org
arxiv.org
A survey of agentic materials science and engineering: where are we and where are we going?
Jiayi Zhu
Longhan Zhang
Yizhang Zhu
Xiaotian Lin
Yifan Wu
Shimin Di
Bang Liu
Yuyu Luo
Tongyi Zhang
Agents, primarily built upon large language models (LLMs) and equipped with planning, tool use, memory, and self-reflection capabilities, a… (see more)re revolutionizing all aspects of materials science and engineering (MSE), from materials design and experimental execution to industrial manufacturing and deployment, thereby opening the age of agentic MSE. Rather than functioning as isolated artificial intelligence (AI) predictive models, these agents coordinate multi-step scientific workflows by retrieving and structuring knowledge, proposing and refining hypotheses, planning experiments, combining multimodal simulations and characterizations, and, when integrated with AI materials laboratories, closing the loop toward autonomous materials discovery. However, agentic systems exhibit varying degrees of autonomy, and their roles in materials research and development differ accordingly. To systematically examine the landscape of agentic MSE, this survey proposes a six-level autonomy framework (Levels 0-5) that characterizes the progression from human-only workflows to fully autonomous scientific agents. The framework aligns with key task families across the entire MSE pipeline, including information retrieval, property prediction, simulation, synthesis, and characterization. By reviewing recent advances in agentic MSE, we reveal uneven progress across these domains. Knowledge-centric capabilities often remain at an early stage, while experimental orchestration and characterization are starting to explore higher-level agent behaviors. Importantly, achieving mature autonomy requires coordinating multiple tasks rather than optimizing any single task in isolation. Collectively, these insights provide a structured roadmap for advancing agentic MSE toward higher levels of autonomy.
2026-05-25
Journal of Materials Informatics (published)
doi.org
Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors
Guozheng Ma
Lu Li
Zilin Wang
Pierre-Luc Bacon
Dacheng Tao
Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency. Originally stud… (see more)ied in offline-to-online RL, this paradigm now spans foundation model post-training and embodied intelligence, with prior types expanding from offline datasets and pre-trained policies to increasingly diverse knowledge sources such as multimodal foundation models and generative world models. Offline priors have become central to how deep RL is developed and deployed. However, this reliance introduces a challenge that the prevailing benchmark-driven paradigm cannot resolve: because prior validity varies across deployments and shifts during training, no single approach to managing it is universally optimal, and benchmark rankings offer limited guidance for real-world deployments. Rather than pursuing universal solutions, we argue that the field should shift to diagnosis-driven tension management, in which deployment-specific evidence guides how the learner relates to its priors throughout training, enabling both flexible and adaptive deployment. We support this position with a framework characterizing how priors reshape online optimization through three functional roles, controlled experiments demonstrating help-or-hurt reversals, cross-domain evidence from foundation model post-training to embodied intelligence, and engagement with five substantive counterarguments.
2026-05-24
DEMO @ International Conference on Machine Learning (poster)
openreview.net
openreview.net
Body Keypoint Detection Algorithm Based on Channel Attention Mechanism
Shaojun Yu
Wenhao Huo
Yuping Lu
Hanqing Zhao
Yilin Wang
Lili Wang
Rizwan Anjum Muhammad
With the implementation of national strategies aimed at building a leading sporting nation and promoting nationwide fitness, physical fitnes… (see more)s assessment has gained increasing attention as a crucial metric for evaluating students' physical condition and motor abilities. Concurrently, advancements in computer vision have enabled body keypoint detection technology to gradually replace traditional manual measurement methods, demonstrating significant potential for application in automated assessment systems. Accurate recognition of keypoints serves as the fundamental support for intelligent physical fitness testing and smart sports. However, existing keypoint detection algorithms often suffer from drifting of extremity keypoints, such as those of the hands and feet keypoints, in physical fitness test scenarios, thereby compromising the accuracy of the assessment. To address this challenge, this paper proposes Channel Attention BlazePose(CA-BlazePose), a body keypoint detection algorithm based on a channel attention mechanism, specifically designed for count-based physical fitness test scenarios, namely sit-ups and pull-ups. To tackle the issue of keypoint drift in motion detection, CA-BlazePose aims to enhance keypoint detection accuracy. It employs a two-stage network architecture consisting of heatmap training and regression fine-tuning, incorporating a channel attention module. This module strengthens the feature extraction process for extremity keypoints such as hands and feet, thereby improving recognition accuracy during detection.Experimental results demonstrate that, compared to mainstream keypoint detection algorithms such as OpenPose and BlazePose, the proposed CA-BlazePose algorithm achieves improvements in the PCK on two representative motion datasets, Common Objects in Context(COCO) and Leeds Sports Pose Extended(LSPET). Specifically, it shows an approximate increase of 7% for hand and foot keypoints and 8% for overall keypoints. Furthermore, in real-time detection tests for sit-ups and pull-ups captured from various viewing angles, CA-BlazePose demonstrates superior performance in handling frames with missing or drifting keypoints compared to existing algorithms, exhibiting more stable recognition performance under identical detection conditions.
2026-05-24
Journal of Intelligent Computing and Networking (published)
doi.org
From Static Policies to Adaptive Priors in Offline Reinforcement Learning
Tianwei Ni
Vineet Jain
Akash Karthikeyan
Pierre-Luc Bacon
Offline reinforcement learning (RL) has traditionally focused on learning policies for direct deployment under conservative objectives, wher… (see more)e uncertainty outside the offline dataset is treated pessimistically to ensure robustness. We argue that this formulation becomes incomplete when an offline-trained policy is subsequently updated through online interaction, as increasingly occurs in modern intelligent systems through test-time adaptation and online fine-tuning. This position paper argues that, in such settings, the objective of offline RL should extend beyond immediate deployment and instead prioritize learning *adaptive policy priors*: policies that preserve the capacity to improve during subsequent interaction through memory, exploration, and self-correction. We formalize this perspective as *adaptive offline reinforcement learning* (AORL), distinguish it from offline-to-online RL, and explain why adaptability becomes important under distributional shift, limited dataset coverage, and changing test-time conditions. We further discuss Bayesian offline RL as one principled direction for constructing adaptive policy priors by preserving epistemic uncertainty over plausible environments. Finally, we outline connections, open challenges, and research directions for treating offline RL as preparation for future experience rather than as a static deployment problem.
2026-05-24
DEMO @ International Conference on Machine Learning (poster)
openreview.net
openreview.net
HarMoEny: Efficient Inference of MoE Models
Zachary Doucet
Rishi Sharma
Martijn de Vos
Rafael Pires
Anne-Marie Kermarrec
Oana Balmau
2026-05-24
IEEE International Parallel and Distributed Processing Symposium (published)
doi.org
Language Pretraining Gives Structured Forecasters a Sequential Prior
Zhenghan Tai
Alexis Roger
Prateek Humane
Gwen Legate
Andrei Mircea
Vasilii Feofanov
Irina Rish
Structured-data foundation models now target transfer across time series, tabular data, text, and other sequential domains, but it remains u… (see more)nclear what language pretraining contributes to forecasting beyond a generic transformer architecture. We study this question by adapting Qwen3-0.6B to probabilistic time-series forecasting on GiftEval and comparing language-pretrained models with identical randomly initialized models under full and parameter-efficient finetuning. Language pretraining gives a large early optimization advantage, especially in limited-adaptation regimes: LoRA attention updates recover most of the effective transfer benefit of full finetuning. Frozen-state probes, retrieval forecasts, gradient coherence, and effective-rank dynamics indicate that language models already contain reusable sequential structure before time-series supervision. These results frame language-to-time-series transfer as efficient low-rank specialization of a pretrained sequential inductive bias, with implications for future structured-data foundation models.
2026-05-24
FMSD @ International Conference on Machine Learning (poster)
openreview.net
openreview.net
Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage
Alan Milligan
Zikun Xu
Simon Lacoste-Julien
Felix Dangel
Wu Lin
Shampoo-based methods, such as KL-Shampoo and SOAP, have demonstrated strong performance in training neural networks and rely on QR decompos… (see more)ition. Because existing QR implementations require single-precision (FP32) arithmetic and remain computationally expensive, these methods become time- and memory-intensive when their preconditioning matrices are large. Moreover, using BFloat16 (BFP16) storage to reduce memory usage can degrade the performance of Shampoo-based methods. We propose a reparametrization of the preconditioner that supports BFP16 storage and forms a complete basis by combining updated basis vectors with unchanged ones. By updating only part of the basis through QR decomposition in a subspace, our approach reduces computational overhead while mitigating the performance degradation caused by BFP16 storage. Our approach applies broadly to Shampoo-based methods that employ QR decomposition, including KL-Shampoo, SOAP, and KL-SOAP. In particular, it improves the performance of SOAP and KL-SOAP under BFP16 storage, enabling KL-SOAP to match or exceed KL-Shampoo. Overall, our approach makes Shampoo-based methods more memory- and time-efficient.
2026-05-24
arXiv (preprint)
doi.org
openreview.net
Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion
Somjit Nath
Jackson J Cone
Derek Nowrouzezahrai
Samira Ebrahimi Kahou
Neuroscientific research has revealed that the brain encodes complex behaviors by leveraging structured, low-dimensional manifolds and dynam… (see more)ically fusing multiple sources of information through adaptive gating mechanisms. Inspired by these principles, we propose a novel reinforcement learning (RL) framework that encourages the disentanglement of dynamics-specific and reward-specific features, drawing direct parallels to how neural circuits separate and integrate information for efficient decision-making. Our approach leverages locally linear embeddings (LLEs) to capture the intrinsic, locally linear structure inherent in many environments—mirroring the local smoothness observed in neural population activity—while concurrently deriving reward-specific features through the standard RL objective. An attention mechanism, analogous to cortical gating, adaptively fuses these complementary representations on a per-state basis. Experimental results on benchmark tasks demonstrate that our method, grounded in neuroscientific principles, improves learning efficiency and overall performance compared to conventional RL approaches, highlighting the benefits of explicitly modeling local state structures and adaptive feature selection as observed in biological systems.
2026-05-24
Transactions on Machine Learning Research (accepted)
openreview.net
openreview.net
Training Fair Tabular Foundation Models
Patrik Kenfack
Jesse C. Cresswell
Anthony L. Caterini
Samira Ebrahimi Kahou
Ulrich Aivodji
Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on … (see more)new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose \ftfm{}, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 120 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.
2026-05-24
FMSD @ International Conference on Machine Learning (oral)
openreview.net
openreview.net