Publications

Understanding Softmax Attention Layers: Exact Mean-Field Analysis on a Toy Problem
Elvis Dohmatob
Self-attention has emerged as a fundamental component driving the success of modern transformer architectures, which power large language mo… (see more)dels and various applications. However, a theoretical understanding of how such models actually work is still under active development. The recent work of (Marion et al., 2025) introduced the so-called "single-location regression" problem, which can provably be solved by a simplified self-attention layer but not by linear models, thereby demonstrating a striking functional separation. A rigorous analysis of self-attention with softmax for this problem is challenging due to the coupled nature of the model. In the present work, we use ideas from the classical random energy model in statistical physics to analyze softmax self-attention on the single-location problem. Our analysis yields exact analytic expressions for the population risk in terms of the overlaps between the learned model parameters and those of an oracle. Moreover, we derive a detailed description of the gradient descent dynamics for these overlaps and prove that, under broad conditions, the dynamics converge to the unique oracle attractor. Our work not only advances our understanding of self-attention but also provides key theoretical ideas that are likely to find use in further analyses of even more complex transformer architectures.
2025-12-02
Conference on Neural Information Processing Systems (Accept (poster))
openreview.net
openreview.net
Deploying Geospatial Foundation Models in the Real World: Lessons from WorldCereal
Christina Butsko
Gabriel Tseng
Kristof Van Tricht
Giorgia Milli
David Rolnick
Ruben Cartuyvels
Inbal Becker Reshef
Zoltan Szantoi
Hannah Kerner
The increasing availability of geospatial foundation models has the potential to transform remote sensing applications such as land cover cl… (see more)assification, environmental monitoring, and change detection. Despite promising benchmark results, the deployment of these models in operational settings is challenging and rare. Standardized evaluation tasks often fail to capture real-world complexities relevant for end-user adoption such as data heterogeneity, resource constraints, and application-specific requirements. This paper presents a structured approach to integrate geospatial foundation models into operational mapping systems. Our protocol has three key steps: defining application requirements, adapting the model to domain-specific data and conducting rigorous empirical testing. Using the Presto model in a case study for crop mapping, we demonstrate that fine-tuning a pre-trained model significantly improves performance over conventional supervised methods. Our results highlight the model’s strong spatial and temporal generalization capabilities. Our protocol provides a replicable blueprint for practitioners and lays the groundwork for future research to operationalize foundation models in diverse remote sensing applications. Application of the protocol to the WorldCereal global crop-mapping system showcases the framework’s scalability.
2025-12-01
Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation (published)
doi.org
proceedings.mlr.press
The Cloud-Based Geospatial Benchmark: Challenges and LLM Evaluation
Jeffrey A. Cardille
Renee Johnston
Simon Ilyushchenko
Johan Kartiwa
Zahra Shamsi
Matthew Abraham
Khashayar Azad
Kainath Ahmed
Emma Bergeron Quick
Nuala Caughie
Noah Jencz
Karen Dyson
Andrea Puzzi Nicolau
Maria Fernanda Lopez-Ornelas
David Saah
Michael Brenner
Subhashini Venugopalan
Sameera S Ponda
2025-12-01
Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation (published)
proceedings.mlr.press
proceedings.mlr.press
The Moral Consistency Pipeline: Continuous Ethical Evaluation for Large Language Models
Saeid Jamshidi
Kawser Wazed Nafi
Arghavan Moradi Dakhel
Negar Shahabi
Foutse Khomh
2025-12-01
ArXiv (preprint)
doi.org
arxiv.org
Training neural networks from scratch in a videogame leads to brittle brain encoding
François Paugam
Basile Pinsard
Marie St-Laurent
Guillaume Lajoie
Lune Bellec
Recent brain-encoding studies using videogame tasks suggest that the training objective of an artificial neural network plays a central role… (see more) in how well the network’s representations align with brain activity. This study investigates the alignment of artificial neural network activations with brain activity elicited by a video game task using models trained from scratch in controlled settings. We specifically compared three model training objectives: reinforcement learning, imitation learning, and a vision task, while accounting for other potential factors which may impact performance such as training data and model architecture. We tested models on brain encoding, i.e. their ability to predict functional magnetic resonance imaging (fMRI) signals acquired while human subjects played different levels of the video game Super Mario Bros. When tested on new playthroughs from the game levels seen at training, the reinforcement learning objective had a small but significant advantage in brain encoding, followed by the imitation learning and vision models. We hypothesized that brain-aligned representations would emerge only in task-competent models, and that the specific brain regions well encoded by a model would depend on the nature of the task it was trained on. While brain encoding did improve during model training, even an untrained model with matching architecture approached the performance of the best models. Contrary to our hypotheses, no model layers or specific training objectives aligned preferentially with specific brain areas. Large performance gaps also persisted in fully trained models across game levels, both those seen during training and entirely novel ones. Overall, even though reinforcement learning presented a small advantage to train brain encoding models for videogame data, all tested brain encoding models exhibited brittle performance with limited generalization both within- and out-of-distribution. Overall, our results suggest that training small artificial models from scratch is not sufficiently reliable, and that incorporating pretrained models such as foundation vision–action models may ultimately be necessary to support robust inferences about brain representations.
2025-12-01
bioRxiv (preprint)
doi.org
Accelerated Inorganic Materials Design with Generative Al Agents
Izumi Takahara
Teruyasu Mizoguchi
Bang Liu
Designing inorganic crystalline materials with tailored properties is critical to technological innovation, yet current generative computati… (see more)onal methods often struggle to efficiently explore desired targets with sufficient interpretability. Here, we present MatAgent, a generative approach for inorganic materials discovery that harnesses the powerful reasoning capabilities of large language models (LLMs). By combining a diffusion-based generative model for crystal structure estimation with a predictive model for property evaluation, MatAgent uses iterative, feedback-driven guidance to steer material exploration precisely toward user-defined targets. Integrated with external cognitive tools-including short-term memory, long-term memory, the periodic table, and a comprehensive materials knowledge base-MatAgent emulates human expert reasoning to vastly expand the accessible compositional space. Our results demonstrate that MatAgent robustly directs exploration toward desired properties while consistently achieving high compositional validity, uniqueness, and material novelty. This framework thus provides a highly interpretable, practical, and versatile AI-driven solution to accelerate the discovery and design of next-generation inorganic materials.
2025-11-30
Cell Reports Physical Science (published)
doi.org
arxiv.org
Biomechanical finite element simulation of the pelvic organs under dynamic loading and validation against experimental data from magnetic resonance imaging.
Camille Lafond
Louise Hohnadel
Thomas Brunel
Nicolas Pirró
Bellemare Marc-Emmanuel
Dominique Chamoret
Sébastien Roth
2025-11-30
Medical Engineering & Physics (published)
doi.org
On Global Applicability and Location Transferability of Generative Deep Learning Models for Precipitation Downscaling
Paula Harder
Christian Lessig
Matthew Chantry
Francis Pelletier
David Rolnick
2025-11-30
ArXiv (preprint)
doi.org
arxiv.org
Improving Robustness and Reliability in Medical Image Classification with Latent-Guided Diffusion and Nested-Ensembles
Xing Shen
Hengguan Huang
Brennan Nichyporuk
Tal Arbel
Once deployed, medical image analysis methods are often faced with unexpected image corruptions and noise perturbations. These unknown covar… (see more)iate shifts present significant challenges to deep learning based methods trained on "clean" images. This often results in unreliable predictions and poorly calibrated confidence, hence hindering clinical applicability. While recent methods have been developed to address specific issues such as confidence calibration or adversarial robustness, no single framework effectively tackles all these challenges simultaneously. To bridge this gap, we propose LaDiNE, a novel ensemble learning method combining the robustness of Vision Transformers with diffusion-based generative models for improved reliability in medical image classification. Specifically, transformer encoder blocks are used as hierarchical feature extractors that learn invariant features from images for each ensemble member, resulting in features that are robust to input perturbations. In addition, diffusion models are used as flexible density estimators to estimate member densities conditioned on the invariant features, leading to improved modeling of complex data distributions while retaining properly calibrated confidence. Extensive experiments on tuberculosis chest X-rays and melanoma skin cancer datasets demonstrate that LaDiNE achieves superior performance compared to a wide range of state-of-the-art methods by simultaneously improving prediction accuracy and confidence calibration under unseen noise, adversarial perturbations, and resolution degradation.
2025-11-30
IEEE Transactions on Medical Imaging (published)
doi.org
arxiv.org
LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing
Bram Bulté
Ayla Rigouts Terryn
Large Language Models (LLMs) are rapidly being adopted by users across the globe, who interact with them in a diverse range of languages. At… (see more) the same time, there are well-documented imbalances in the training data and optimisation objectives of this technology, raising doubts as to whether LLMs can represent the cultural diversity of their broad user base. In this study, we look at LLMs and cultural values and examine how prompt language and cultural framing influence model responses and their alignment with human values in different countries. We probe 10 LLMs with 63 items from the Hofstede Values Survey Module and World Values Survey, translated into 11 languages, and formulated as prompts with and without different explicit cultural perspectives. Our study confirms that both prompt language and cultural perspective produce variation in LLM outputs, but with an important caveat: While targeted prompting can, to a certain extent, steer LLM responses in the direction of the predominant values of the corresponding countries, it does not overcome the models' systematic bias toward the values associated with a restricted set of countries in our dataset: the Netherlands, Germany, the US, and Japan. All tested models, regardless of their origin, exhibit remarkably similar patterns: They produce fairly neutral responses on most topics, with selective progressive stances on issues such as social tolerance. Alignment with cultural values of human respondents is improved more with an explicit cultural perspective than with a targeted prompt language. Unexpectedly, combining both approaches is no more effective than cultural framing with an English prompt. These findings reveal that LLMs occupy an uncomfortable middle ground: They are responsive enough to changes in prompts to produce variation, but too firmly anchored to specific cultural defaults to adequately represent cultural diversity.
2025-11-30
Computational Linguistics (published)
doi.org
arxiv.org
Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models
Eva Portelance
Siva Reddy
Timothy J. O’Donnell
Semantic and syntactic bootstrapping posit that children use their prior knowledge of one linguistic domain, say syntactic relations, to hel… (see more)p later acquire another, such as the meanings of new words. Empirical results supporting both theories may tempt us to believe that these are different learning strategies, where one may precede the other. Here, we argue that they are instead both contingent on a more general learning strategy for language acquisition: joint learning. Using a series of neural visually-grounded grammar induction models, we demonstrate that both syntactic and semantic bootstrapping effects are strongest when syntax and semantics are learnt simultaneously. Joint learning results in better grammar induction, realistic lexical category learning, and better interpretations of novel sentence and verb meanings. Joint learning makes language acquisition easier for learners by mutually constraining the hypotheses spaces for both syntax and semantics. Studying the dynamics of joint inference over many input sources and modalities represents an important new direction for language modeling and learning research in both cognitive sciences and AI, as it may help us explain how language can be acquired in more constrained learning settings.
2025-11-30
OpenReview (published)
doi.org
openreview.net
What makes a good public EV charging station? A revealed preference study
Steven Lamontagne
Margarida Carvalho
Emma Frejinger
Ribal Atallah
To determine the optimal locations for electric vehicle charging stations, optimisation models need to predict which charging stations users… (see more) will select. We estimate discrete choice models to predict the usage of charging stations using only readily available information for charging network operators. Our parameter values are estimated from a unique, revealed preferences dataset of charging sessions in Montreal, Quebec. We find that user distance to stations, proximity to home areas, and the number of outlets at each station are significant factors for predicting station usage. Additionally, amenities near charging stations have a neutral effect overall, with some users demonstrating strong preference or aversion for these locations. High variability among the preferences of users highlight the importance of models which incorporate panel effects. Moreover, integrating mixed logit models within the optimization of charging station network design yields high-quality solutions, even when evaluated under other model specifications.
2025-11-30
Transportation Research Part D: Transport and Environment (published)
doi.org
arxiv.org