no. 16 · Tuesday 17 November 2026
An Online Multiobjective Policy Gradient for Long-run Average-reward Markov Decision Process
Abstract
We propose a policy gradient algorithm for multiobjective decision-making on continuous state spaces, in which the long-run average of a reward vector must be steered into a target set despite an adversarial disturbance, by scalarizing the reward along the direction prescribed by Blackwell's Approachability Theorem instead of tuning weights by trial and error. Existing approachability-based learning is confined to countable state spaces, because the scalarization is updated at the visits to a recurrent state, which on a continuous state space has probability zero. We update it instead at the returns to a set after a minimum number of steps, show that a geometric drift condition, uniform over the policy class, controls the cycle lengths while the policy is being learned, and replace the parallel learners of the existing algorithm by a single two-timescale actor-critic. We show that the distance to the target set is asymptotically bounded by the per-cycle remainder from Blackwell's condition, and decompose this remainder into the errors of the actor, the policy class, the direction tracking and the return set. We illustrate the algorithm on a heating, ventilation, and humidity control problem with adversarial weather and occupancy.
- fields
- Optimization and Control, Probability