Portrait of Xiaofeng Zhang is unavailable

Xiaofeng Zhang

PhD - Université de Montréal
Supervisor
Aaron Courville
Research Topics
Computer Vision
Generative Models
Multimodal Learning
Representation Learning
Website

Publications

Beyond Reward Maximization: Evaluating the Diversity of Trajectories in Reinforcement Learning with Temporal Vendi Score
Tom Stanic
Marco Jiralerspong
Xiaofeng Zhang
Danilo Vucetic
Gauthier Gidel
In domains such as scientific discovery and automated design using reinforcement learning (RL), the final task of an agent should extend bey… (see more)ond maximising a single scalar reward; it requires identifying diverse sets of high-quality trajectories to uncover distinct solutions that can provide novel insights on how to solve the problems of interest and transfer robustly from simulation to the real world. However, the RL literature currently lacks a holistic, domain-agnostic standard for measuring trajectory diversity. Existing metrics have been developed to improve exploration at training time but not to evaluate and compare diversity induced by different agents, rendering cross-method comparisons inconsistent and challenging. To address this, we introduce the Temporal Vendi Score (TVS), a novel metric designed to evaluate the diversity of an RL agent by computing the entropy of the eigenvalues' similarity matrix of sampled trajectories. Unlike previous approaches, our metric captures the behavioural diversity of trajectories by accounting for both the sequential nature of state visitations and the temporal structure of the underlying MDP, rather than relying on order-agnostic state comparisons. We validate the TVS on simple environments where we can control the number of different ways a problem can be solved, demonstrating that it provides a more robust, semantically meaningful ranking of diversity than standard baselines. We then show that our metric can scale to a high-dimensional, continuous environment.
2026-03-01
LLA @ International Conference on Learning Representations (poster)
openreview.net
openreview.net
Inference-time Physics Alignment of Video Generative Models with Latent World Models
Jianhao Yuan
Xiaofeng Zhang
Felix Friedrich
Nicolas Beltran-Velez
Melissa Hall
Reyhane Askari Hemmat
Xiaochuang Han
Nicolas Ballas
Michal Drozdzal
Adriana Romero
State-of-the-art video generative models produce promising visual content yet often violate basic physics principles, limiting their utility… (see more). While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of video generation as an inference-time alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, in the ICCV 2025 Perception Test PhysicsIQ Challenge, we achieve a final score of 62.64%, winning first place and outperforming the previous state of the art by 7.42%. Our work demonstrates the viability of using latent world models to improve physics plausibility of video generation, beyond this specific instantiation or parameterization.
2026-01-14
ArXiv (preprint)
doi.org
arxiv.org
Inference-time Physics Alignment of Video Generative Models with Latent World Models
Jianhao Yuan
Xiaofeng Zhang
Felix Friedrich
Nicolas Beltran-Velez
Melissa Hall
Reyhane Askari-Hemmat
Xiaochuang Han
Nicolas Ballas
Michal Drozdzal
Adriana Romero-Soriano
State-of-the-art video generative models produce promising visual content yet often violate basic physics principles, limiting their utility… (see more). While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of video generation as an inference-time alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, on the challenging PhysicsIQ benchmark we achieve 62.00% final score, outperforming previous state of the art by 6.78%. Our work demonstrates the viability of using latent world models to improve physical plausibility of video generation, beyond this specific instantiation or parameterization.
2025-12-31
IEEE/CVF Conference on Computer Vision and Pattern Recognition (Accept (Highlight))
doi.org
arxiv.org
Improving the Physics of Video Generation with VJEPA-2 Reward Signal
Jianhao Yuan
Xiaofeng Zhang
Felix Friedrich
Nicolas Beltran-Velez
Melissa Hall
Reyhane Askari Hemmat
Xiaochuang Han
Nicolas Ballas
Michal Drozdzal
Adriana Romero
2025-10-21
ArXiv (preprint)
doi.org
arxiv.org
The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Models
Xiaofeng Zhang
Aaron Courville
Michal Drozdzal
Adriana Romero
2025-10-21
ArXiv (preprint)
doi.org
arxiv.org
Increasing the Utility of Synthetic Images through Chamfer Guidance
Nicola Dall'Asen
Xiaofeng Zhang
Reyhane Askari-Hemmat
Melissa Hall
Jakob Verbeek
Adriana Romero-Soriano
Michal Drozdzal
Conditional image generative models hold considerable promise to produce infinite amounts of synthetic training data. Yet, recent progress i… (see more)n generation quality has come at the expense of generation diversity, limiting the utility of these models as a source of synthetic training data. Although guidance-based approaches have been introduced to improve the utility of generated data by focusing on quality or diversity, the (implicit or explicit) utility functions oftentimes disregard the potential distribution shift between synthetic and real data. In this work, we introduce Chamfer Guidance: a training-free guidance approach which leverages a handful of real exemplar images to characterize the quality and diversity of synthetic data. We show that by leveraging the proposed Chamfer Guidance, we can boost the diversity of the generations w.r.t. a dataset of real images while maintaining or improving the generation quality on ImageNet-1k and standard geo-diversity benchmarks. Our approach achieves state-of-the-art few-shot performance with as little as 2 exemplar real images, obtaining 96.4% in terms of precision, and 86.4% in terms of distributional coverage, which increase to 97.5% and 92.7%, respectively, when using 32 real images. We showcase the benefits of the Chamfer Guidance generation by training downstream image classifiers on synthetic data, achieving accuracy boost of up to 15% for in-distribution over the baselines, and up to 16% in out-of-distribution. Furthermore, our approach does not require using the unconditional model, and thus obtains a 31% reduction in FLOPs w.r.t. classifier-free-guidance-based approaches at sampling time.
2025-09-17
NeurIPS.cc/2025/Conference (poster)
doi.org
openreview.net
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics
Shravan Nayak
Mehar Bhatia
Xiaofeng Zhang
Verena Rieser
Lisa Anne Hendricks
Sjoerd van Steenkiste
Yash Goyal
Karolina Stanczak
Aishwarya Agrawal
The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurate… (see more)ly represent diverse cultural contexts. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit as well as implicit cultural expectations. To this end, we introduce CulturalFrames, a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CulturalFrames comprises 983 prompts, 3637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that T2I models not only fail to meet the more challenging implicit expectations but also the less challenging explicit expectations. Across models and countries, cultural expectations are missed an average of 44% of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68%, while implicit expectation failures are also significant, averaging 49%. Furthermore, we demonstrate that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, providing actionable directions for developing more culturally informed T2I models and evaluation methodologies.
2025-06-09
ICML.cc/2025/Workshop/MoFA (poster)
doi.org
openreview.net