Portrait de Homayoun Honari

Homayoun Honari

homayoun.honari@mila.quebec
Collaborateur·rice de recherche - UdeM
Superviseur⋅e principal⋅e
Glen Berseth
Sujets de recherche
Apprentissage de représentations
Apprentissage par renforcement
Causalité
Cognition
Conscience
Généralisation
IA inspirée du cerveau
IAG (Intelligence Artificielle Générale)
Méthodes inspirées de la causalité
Raisonnement
Robotique
Théorie de l'apprentissage automatique
Google Scholar
GitHub

Publications

Align and Filter: Improving Performance in Asynchronous On-Policy RL
Homayoun Honari
Roger Creus Castanyer
Michael Przystupa
Michael Noukhovitch
Pablo Samuel Castro
Glen Berseth
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, b… (voir plus)ut both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.
2026-03-01
arXiv (prépublication)
doi.org
arxiv.org
Training PPO-Clip with Parallelized Data Generation: A Case of Fixed-Point Convergence
Homayoun Honari
Roger Creus Castanyer
Pablo Samuel Castro
Glen Berseth
In recent years, with the increase in the compute power of GPUs, parallelized data collection has become the dominant approach for training … (voir plus)reinforcement learning (RL) agents. Proximal Policy Optimization (PPO) is one of the widely-used on-policy methods for training RL agents. In this paper, we focus on the training behavior of PPO-Clip with the increase in the number of parallel environments. In particular, we show that as we increase the amount of data used to train PPO-Clip, the optimized policy would converge to a fixed distribution. We use the results to study the behavior of PPO-Clip in two case studies: the effect of change in the minibatch size and the effect of increase in the number of parallel environments versus the increase in the rollout lengths. The experiments show that settings with high-return PPO runs result in slower convergence to the fixed-distribution and higher consecutive KL divergence changes. Our results aim to offer a better understanding for the prediction of the performance of PPO with the scaling of the parallel environments.
2019-12-31
International Conference on Machine Learning (publié)
openreview.net
openreview.net