Padideh Nouri

Learning Where to Constrain: Meta-Learned Trust Allocation for Mathematical Reasoning

Long mathematical solutions expose a weakness in uniform policy-update constraints. Early token changes affect a long suffix, while late cha… (voir plus)nges affect fewer future states. Prefix divergence-based methods like CPPO encode this asymmetry with a fixed position schedule and prefix budget. We introduce Meta-TA, a bilevel method for learning both choices from post-update performance on disjoint prompts. A direct one-step design fails when behavior and current policies match because the constraint starts inactive. We prove the resulting hypergradient equals zero. A detached warm-up creates controlled policy drift. A soft-gated step then supplies the meta-gradient. On Qwen3-8B-Base, five frozen-retraining seeds raise mean AIME24/25/26 Avg@16 from 29.01 for CPPO to 31.95 for Meta-TA with closely matched realized KL. At a 16k response cap, Meta-TA lowers normalized reasoning effective horizon from 0.475 to 0.430 and importance-ratio variance from 0.0534 to 0.0475. These results link adaptive trust allocation to higher accuracy and more localized updates in long-horizon mathematical reasoning.

2026-10-03

MATH-AI @ Neural Information Processing Systems (poster)

openreview.net

Relative Trajectory Balance is equivalent to Trust-PCL

2025-08-31

ArXiv (prépublication)

doi.org

arxiv.org

Revisiting Laplacian Representations for Value Function Approximation in Deep RL

Priyesh Vijayan

Padideh Nouri

Rishav

A. Chandar

Yash Chandak

Mathieu Reymond

S Ebrahimi Kahou

Doina Precup

Proto-value functions (PVFs) introduced Laplacian embeddings as an effective feature basis for value-function approximation; however, their … (voir plus)utility remained limited to small, fully known state spaces. Recent work has scaled Laplacian embeddings to high-dimensional inputs, using them for reward shaping and option discovery in goal-directed tasks, yet only as auxiliary signals, rather than directly using them as features for value functions. In this paper, we learn Laplacian eigenvectors online and employ them as features for Q-learning in 23 Atari games. We empirically demonstrate that these online–learned embeddings substantially improve model-free RL in large, high-dimensional domains. We demonstrate that enriching state representations with action embeddings yields additional gains under both behavior-policy and uniform-random policies. Additionally, we introduce the Fusion architecture, which augments the representation with useful inductive bias at the embedding level. To assess the usefulness of each embedding used in the Fusion architecture, we use Shapley values analysis.

2025-06-21

rl-conference.cc/RLC/2025/Workshop/IBRL (publié)

openreview.net

Discrete Probabilistic Inference as Control in Multi-path Environments

Nikolay Malkin

We consider the problem of sampling from a discrete and structured distribution as a sequential decision problem, where the objective is to … (voir plus)find a stochastic policy such that objects are sampled at the end of this sequential process proportionally to some predefined reward. While we could use maximum entropy Reinforcement Learning (MaxEnt RL) to solve this problem for some distributions, it has been shown that in general, the distribution over states induced by the optimal policy may be biased in cases where there are multiple ways to generate the same object. To address this issue, Generative Flow Networks (GFlowNets) learn a stochastic policy that samples objects proportionally to their reward by approximately enforcing a conservation of flows across the whole Markov Decision Process (MDP). In this paper, we extend recent methods correcting the reward in order to guarantee that the marginal distribution induced by the optimal MaxEnt RL policy is proportional to the original reward, regardless of the structure of the underlying MDP. We also prove that some flow-matching objectives found in the GFlowNet literature are in fact equivalent to well-established MaxEnt RL algorithms with a corrected reward. Finally, we study empirically the performance of multiple MaxEnt RL and GFlowNet algorithms on multiple problems involving sampling from discrete distributions.

2024-04-25

Conference on Uncertainty in Artificial Intelligence (poster)

doi.org

proceedings.mlr.press

Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian Optimization

The ability to accelerate the design of biological sequences can have a substantial impact on the progress of the medical field. The problem… (voir plus) can be framed as a global optimization problem where the objective is an expensive black-box function such that we can query large batches restricted with a limitation of a low number of rounds. Bayesian Optimization is a principled method for tackling this problem. However, the astronomically large state space of biological sequences renders brute-force iterating over all possible sequences infeasible. In this paper, we propose MetaRLBO where we train an autoregressive generative model via Meta-Reinforcement Learning to propose promising sequences for selection via Bayesian Optimization. We pose this problem as that of finding an optimal policy over a distribution of MDPs induced by sampling subsets of the data acquired in the previous rounds. Our in-silico experiments show that meta-learning over such ensembles provides robustness against reward misspecification and achieves competitive results compared to existing strong baselines.

2022-09-12

ArXiv (prépublication)

doi.org

arxiv.org

Formation d'entrepreneurs scientifiques

Boussole des politiques en IA

Demandes de supervision

Padideh Nouri

Publications

Formation d'entrepreneurs scientifiques

Boussole des politiques en IA

Demandes de supervision

Mots-clés populaires:

Padideh Nouri

Publications