Portrait of Juan Duque

Juan Duque

juan.duque@mila.quebec
PhD - Université de Montréal
Supervisor
Aaron Courville
Research Topics
Game Theory
Reinforcement Learning
Website
Google Scholar
GitHub

Publications

Recurrent Structural Policy Gradient for Partially Observable Mean Field Games
Clarisse Wibault
Johannes Forkel
Sebastian Towers
Tiphaine Wibault
Juan Duque
George Whittle
Andreas Schaab
Yucheng Yang
Chiyuan Wang
Maike Osborne
Benjamin Moll
Jakob Foerster
Mean Field Games (MFGs) provide a principled framework for modeling interactions in large populations models: at scale, population dynamics … (see more)become deterministic, with uncertainty entering only through aggregate shocks, or *common noise*. However, algorithmic progress has been limited since model-free methods are too high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) use Monte Carlo rollouts for the common noise in combination with exact estimation of the expected return, conditioned on those samples. However, HSMs have not been scaled to Partially Observable settings. We propose *Recurrent Structural Policy Gradient* (RSPG), the first history-aware HSM. We also introduce MFAX, our JAX-based framework for MFGs. By leveraging known transition dynamics, RSPG achieves state-of-the-art performance as well as an order-of-magnitude faster convergence and solves, for the first time, a macroeconomics MFG with heterogeneous agents, common noise and history-aware policies. MFAX is publicly available at: .
2025-12-31
International Conference on Machine Learning (Accept (spotlight))
doi.org
openreview.net
Advantage Alignment Algorithms
Juan Duque
Milad Aghajohari
Tim Cooijmans
Razvan Ciuca
Tianyu Zhang
Gauthier Gidel
Aaron Courville
The growing presence of artificially intelligent agents in everyday decision-making, from LLM assistants to autonomous vehicles, hints at a … (see more)future in which conflicts may arise from each agent optimizing individual interests. In general-sum games these conflicts are apparent, where naive Reinforcement Learning agents get stuck in Pareto-suboptimal Nash equilibria. Consequently, opponent shaping has been introduced as a method with success at finding socially beneficial equilibria in social dilemmas. In this work, we introduce Advantage Alignment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. This is achieved by aligning the advantages of conflicting agents in a given game by increasing the probability of mutually-benefiting actions. We prove that existing opponent shaping methods, including LOLA and LOQA, implicitly perform Advantage Alignment. Compared to these works, Advantage Alignment mathematically simplifies the formulation of opponent shaping and seamlessly works for continuous action domains. We also demonstrate the effectiveness of our algorithm in a wide range of social dilemmas, achieving state of the art results in each case, including a social dilemma version of the Negotiation Game.
2025-01-21
International Conference on Learning Representations (oral)
doi.org
openreview.net