Portrait of Jiaqi Deng is unavailable

Jiaqi Deng

Alumni
Website
Google Scholar

Publications

AdaptArena: Evaluating Test-Time Personalization of Web Agents
Dongchan Shin
Xing Han Lu
Jiaqi Deng
Jay Gala
Tomás Vergara Browne
Jaewon Moon
Fengyuan Liu
Alexandre Drouin
Siva Reddy
Alexandre Lacoste
Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-worl… (see more)d settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web agents. We release our code: https://github.com/McGill-NLP/web-agents-test-time-adaptations
2026-09-28
arXiv (preprint)
doi.org
arxiv.org