Publications

Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
Léo Boisvert
Abhay Puri
Chandra Kiran Reddy Evuru
Nicolas Chapados
Quentin Cappart
Alexandre Lacoste
Krishnamurthy Dj Dvijotham
Alexandre Drouin
The practice of fine-tuning AI agents on data from their own interactions--such as web browsing or tool use--, while being a strong general … (see more)recipe for improving agentic capabilities, also introduces a critical security vulnerability within the AI supply chain. In this work, we show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors that are triggerred by specific target phrases, such that when the agent encounters these triggers, it performs an unsafe or malicious action. We formalize and validate three realistic threat models targeting different layers of the supply chain: 1) direct poisoning of fine-tuning data, where an attacker controls a fraction of the training traces; 2) environmental poisoning, where malicious instructions are injected into webpages scraped or tools called while creating training data; and 3) supply chain poisoning, where a pre-backdoored base model is fine-tuned on clean data to improve its agentic capabilities. Our results are stark: by poisoning as few as 2% of the collected traces, an attacker can embed a backdoor causing an agent to leak confidential user information with over 80% success when a specific trigger is present. This vulnerability holds across all three threat models. Furthermore, we demonstrate that prominent safeguards, including two guardrail models and one weight-based defense, fail to detect or prevent the malicious behavior. These findings highlight an urgent threat to agentic AI development and underscore the critical need for rigorous security vetting of data collection processes and end-to-end model supply chains.
2025-10-03
ArXiv (preprint)
arxiv.org
arxiv.org
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu
Johan S. Obando-Ceron
Han Lu
Yancheng He
Weixun Wang
Wenbo Su
Bo Zheng
Pablo Samuel Castro
Aaron Courville
Ling Pan
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely prag… (see more)matic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu
Johan S. Obando-Ceron
Han Lu
Yancheng He
Weixun Wang
Wenbo Su
Bo Zheng
Pablo Samuel Castro
Aaron Courville
Ling Pan
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely prag… (see more)matic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu
Johan S. Obando-Ceron
Han Lu
Yancheng He
Weixun Wang
Wenbo Su
Bo Zheng
Pablo Samuel Castro
Aaron Courville
Ling Pan
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely prag… (see more)matic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu
Johan S. Obando-Ceron
Han Lu
Yancheng He
Weixun Wang
Wenbo Su
Bo Zheng
Pablo Samuel Castro
Aaron Courville
Ling Pan
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely prag… (see more)matic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Catalyst GFlowNet for electrocatalyst design: A hydrogen evolution reaction case study
Lena Podina
Christina Humer
Alexandre AGM Duval
Victor Schmidt
Ali Ramlaoui
Shahana Chatterjee
Yoshua Bengio
Alex Hernandez-Garcia
David Rolnick
Félix Therrien
Efficient and inexpensive energy storage is essential for accelerating the adoption of renewable energy and ensuring a stable supply, despit… (see more)e fluctuations in sources such as wind and solar. Electrocatalysts play a key role in hydrogen energy storage (HES), allowing the energy to be stored as hydrogen. However, the development of affordable and high-performance catalysts for this process remains a significant challenge. We introduce Catalyst GFlowNet, a generative model that leverages machine learning-based predictors of formation and adsorption energy to design crystal surfaces that act as efficient catalysts. We demonstrate the performance of the model through a proof-of-concept application to the hydrogen evolution reaction, a key reaction in HES, for which we successfully identified platinum as the most efficient known catalyst. In future work, we aim to extend this approach to the oxygen evolution reaction, where current optimal catalysts are expensive metal oxides, and open the search space to discover new materials. This generative modeling framework offers a promising pathway for accelerating the search for novel and efficient catalysts.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Catalyst GFlowNet for electrocatalyst design: A hydrogen evolution reaction case study
Lena Podina
Christina Humer
Alexandre AGM Duval
Victor Schmidt
Ali Ramlaoui
Shahana Chatterjee
Yoshua Bengio
Alex Hernandez-Garcia
David Rolnick
Félix Therrien
Efficient and inexpensive energy storage is essential for accelerating the adoption of renewable energy and ensuring a stable supply, despit… (see more)e fluctuations in sources such as wind and solar. Electrocatalysts play a key role in hydrogen energy storage (HES), allowing the energy to be stored as hydrogen. However, the development of affordable and high-performance catalysts for this process remains a significant challenge. We introduce Catalyst GFlowNet, a generative model that leverages machine learning-based predictors of formation and adsorption energy to design crystal surfaces that act as efficient catalysts. We demonstrate the performance of the model through a proof-of-concept application to the hydrogen evolution reaction, a key reaction in HES, for which we successfully identified platinum as the most efficient known catalyst. In future work, we aim to extend this approach to the oxygen evolution reaction, where current optimal catalysts are expensive metal oxides, and open the search space to discover new materials. This generative modeling framework offers a promising pathway for accelerating the search for novel and efficient catalysts.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Genetic contribution to asthma informs acute chest syndrome pathophysiology and risk stratification
Sara El Aouhel
Vanessa Bellegarde
Stennio Da
Silva Faria
Tristan St-Laurent
Estelle Lecluze
Anne-Laure Pham Hung d’Alexandry d’Orengiani
F. Galactéros
Pablo Bartolucci
Marc-André Legault
Guillaume Lettre
Thomas Pincez
2025-10-02
medRxiv (preprint)
doi.org
Genetic contribution to asthma informs acute chest syndrome pathophysiology and risk stratification
Sara El Aouhel
Vanessa Bellegarde
Stennio Da
Silva Faria
Tristan St-Laurent
Estelle Lecluze
Anne-Laure Pham Hung d’Alexandry d’Orengiani
F. Galactéros
Pablo Bartolucci
Marc-André Legault
Guillaume Lettre
Thomas Pincez
2025-10-02
medRxiv (preprint)
doi.org
GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
Silvia Sapora
(Rex) Devon Hjelm
Alexander T Toshev
Omar Attia
Bogdan Mazoure
Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield"black-box"models that… (see more) are difficult to interpret and debug. In this work, we introduce GRACE (Generating Rewards As CodE), a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function directly from expert trajectories. The resulting reward function is executable code that can be inspected and verified. We empirically validate GRACE on the BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards, even in complex, multi-task settings. Further, we demonstrate that the resulting reward leads to strong policies, compared to both competitive Imitation Learning and online RL approaches with ground-truth rewards. Finally, we show that GRACE is able to build complex reward APIs in multi-task setups.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
Silvia Sapora
(Rex) Devon Hjelm
Alexander T Toshev
Omar Attia
Bogdan Mazoure
Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield"black-box"models that… (see more) are difficult to interpret and debug. In this work, we introduce GRACE (Generating Rewards As CodE), a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function directly from expert trajectories. The resulting reward function is executable code that can be inspected and verified. We empirically validate GRACE on the BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards, even in complex, multi-task settings. Further, we demonstrate that the resulting reward leads to strong policies, compared to both competitive Imitation Learning and online RL approaches with ground-truth rewards. Finally, we show that GRACE is able to build complex reward APIs in multi-task setups.
2025-10-02
ArXiv (preprint)
arxiv.org
arxiv.org
Attention-Based Multi-Agent RL for Multi-Machine Tending Using Mobile Robots
Abdalwhab Bakheet Mohamed Abdalwhab
Giovanni Beltrame
Samira Ebrahimi Kahou
David St-Onge
2025-10-01
AI (published)
doi.org