Portrait of Weikai Wang is unavailable

Weikai Wang

weikai.wang@mila.quebec
PhD - HEC Montréal
Supervisor
Erick Delage
Research Topics
Optimization
Reinforcement Learning

Publications

Online Policy Evaluation for MDPs with Dynamic UBSR Measures
Weikai Wang
Erick Delage
Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. E… (see more)xisting approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.
2026-07-24
arXiv (preprint)
doi.org
arxiv.org
Planning and Learning in Average Risk-aware MDPs
Weikai Wang
Erick Delage
For continuing tasks, average cost Markov decision processes have well-documented value and can be solved using efficient algorithms. Howeve… (see more)r, it explicitly assumes that the agent is risk-neutral. In this work, we extend risk-neutral algorithms to accommodate the more general class of dynamic risk measures. Specifically, we propose a relative value iteration (RVI) algorithm for planning and design two model-free Q-learning algorithms, namely a generic algorithm based on the multi-level Monte Carlo (MLMC) method, and an off-policy algorithm dedicated to utility-based shortfall risk measures. Both the RVI and MLMC-based Q-learning algorithms are proven to converge to optimality. Numerical experiments validate our analysis, confirm empirically the convergence of the off-policy algorithm, and demonstrate that our approach enables the identification of policies that are finely tuned to the intricate risk-awareness of the agent that they serve.
2025-12-02
Conference on Neural Information Processing Systems (Accept (poster))
doi.org
openreview.net