Abhay Puri

Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain

Chandra Kiran Reddy Evuru

Nicolas Chapados

Alexandre Lacoste

Krishnamurthy Dj Dvijotham

The practice of fine-tuning AI agents on data from their own interactions--such as web browsing or tool use--, while being a strong general … (voir plus)recipe for improving agentic capabilities, also introduces a critical security vulnerability within the AI supply chain. In this work, we show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors that are triggerred by specific target phrases, such that when the agent encounters these triggers, it performs an unsafe or malicious action. We formalize and validate three realistic threat models targeting different layers of the supply chain: 1) direct poisoning of fine-tuning data, where an attacker controls a fraction of the training traces; 2) environmental poisoning, where malicious instructions are injected into webpages scraped or tools called while creating training data; and 3) supply chain poisoning, where a pre-backdoored base model is fine-tuned on clean data to improve its agentic capabilities. Our results are stark: by poisoning as few as 2% of the collected traces, an attacker can embed a backdoor causing an agent to leak confidential user information with over 80% success when a specific trigger is present. This vulnerability holds across all three threat models. Furthermore, we demonstrate that prominent safeguards, including two guardrail models and one weight-based defense, fail to detect or prevent the malicious behavior. These findings highlight an urgent threat to agentic AI development and underscore the critical need for rigorous security vetting of data collection processes and end-to-end model supply chains.

2025-10-03

ArXiv (prépublication)

Evaluating and Improving LitLLMs with Deep Research

Gaurav Sahu

Shubham Agarwal

Issam Hadj Laradji

Krishnamurthy Dj Dvijotham

Jason Stanley

Laurent Charlin

Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially du… (voir plus)e to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: (1) Retrieving related works given a query abstract and (2) Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Particularly, our ``Deep Research" retrieval variant improves coverage by over 5x compared to standard keyword search, addressing a key bottleneck in the pipeline. Further, we demonstrate that our planning-based approach achieves higher-quality reviews by minimizing hallucinated references in the generated review by 18-26\% compared to existing simpler LLM-based generation methods.

2025-07-31

colmweb.org/COLM/2025/Workshop/LM4Sci (publié)

BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning

Ahmed Masry

Masoud Hashemi

Juan A. Rodriguez

Megh Thakkar

Khyati Mahajan

Vikas Yadav

Sathwik Tejaswi Madhusudhan

Alexandre Piché

Dzmitry Bahdanau

David Vazquez

Enamul Hoque

Perouz Taslakian

Sai Rajeswar

Spandana Gella

2025-07-07

colmweb.org/COLM/2025/Conference (accepté)

DoomArena: A framework for Testing AI Agents Against Evolving Security Threats

Gabriel Huang

Mihir Bansal

Chandra Kiran Reddy Evuru

Avinandan Bose

Maryam Fazel

Alexandre Lacoste

Jason Stanley

Krishnamurthy Dj Dvijotham

We present DoomArena, a security evaluation framework for AI agents. DoomArena is designed on three principles: 1) It is a plug-in framework… (voir plus) and integrates easily into realistic agentic frameworks like BrowserGym (for web agents) and

2025-07-07

colmweb.org/COLM/2025/Conference (accepté)

Silent Sabotage: Injecting Backdoors into AI Agents Through Fine-Tuning

Chandra Kiran Reddy Evuru

Joshua Kazdan

Avinandan Bose

Maryam Fazel

Sai Rajeswar

Jason Stanley

Nicolas Chapados

Krishnamurthy Dj Dvijotham

The rise of AI agents that can use tools, browse the web and interact with computers on behalf of a user, has sparked strong interest in imp… (voir plus)roving these capabilities by explicitly fine-tuning the LLMs/VLMs that power these agents. Several researchers have proposed collecting data by letting the agents interact with their environment (e.g., a computer operating system, the web or a collection of APIs exposed as tools), and improve agent performance by fine tuning on this data. In this work, we show that such data collection can be manipulated by adversaries to insert poisoned traces. By modifying just 5% of collected traces, adversaries can embed stealthy bad behaviors into agents—like leaking confidential user information whenever the tool or webpage exposes a trigger. Our results raise important security concerns in the development of AI agents, and underscore the importance of careful scrutiny of all data collection processes used to improve agentic AI.

2025-06-08

ICML.cc/2025/Workshop/WCUA (poster)

Rendering-Aware Reinforcement Learning for Vector Graphics Generation

Juan A. Rodriguez

Haotian Zhang

Aarash Feizi

Rishav Pramanik

Pascal Wichmann

Arnab Mondal

Mohammad Reza Samsami

Rabiul Awal

Perouz Taslakian

Spandana Gella

Sai Rajeswar

David Vazquez

Marco Pedersoli

Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-lang… (voir plus)uage models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF(Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.

2025-05-27

ArXiv (prépublication)

DoomArena: A framework for Testing AI Agents Against Evolving Security Threats

Mihir Bansal

Chandra Kiran Reddy Evuru

Gabriel Huang

Avinandan Bose

Maryam Fazel

Jason Stanley

Alexandre Lacoste

Krishnamurthy Dj Dvijotham

2025-04-18

ArXiv (prépublication)

StarVector: Generating Scalable Vector Graphics Code from Images and Text

Juan A. Rodriguez

Shubham Agarwal

Issam Hadj Laradji

Pau Rodriguez

Sai Rajeswar

David Vazquez

Marco Pedersoli

2025-04-11

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

LitLLMs, LLMs for Literature Review: Are we there yet?

Shubham Agarwal

Gaurav Sahu

Issam Hadj Laradji

Krishnamurthy Dj Dvijotham

Jason Stanley

Laurent Charlin

2025-04-01

TMLR (accepté)

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

Juan A. Rodriguez

Chao Wang

Akshay Kalkunte Suresh

Xiangru Jian

Pierre-Andre Noel

Sathwik Tejaswi Madhusudhan

Enamul Hoque

Issam Hadj Laradji

David Vazquez

Perouz Taslakian … (voir 2 de plus)

Spandana Gella

Sai Rajeswar

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges… (voir plus) on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), often produce out-of-distribution or noisy inputs, leading to misalignment between the modalities. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where scanned document images must be accurately mapped to their textual content. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods. We provide further analysis demonstrating improved vision-text feature alignment and robustness to noise.

2025-03-05

ICLR.cc/2025/Workshop/Re-Align (poster)

BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks

Juan A. Rodriguez

Xiangru Jian

Siba Smarak Panigrahi

Tianyu Zhang

Aarash Feizi