Paperlane

no. 5 · Friday 30 October 2026

Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets

Eugene A. Feinberg and Gaojin He

math.OC

Abstract

This paper studies differential properties of objective functions for discrete-time discounted Markov decision processes (MDPs) with finite state and action sets, when differentiation is considered with respect to the discount factor. Given a fixed discount factor for an MDP, we estimate the norms and seminorms of the objective functions, their derivatives, and the value function at a different discount factor. We then use these estimates to prove that objective functions, their derivatives, and value functions are Lipschitz continuous in discount factors. As a result, if an MDP is solved with a fixed discount factor, then we can estimate the value function at a different discount factor without solving it again.\\ %Following these estimations, we also reveal the relations between the seminorms of objective functions, their behavior when the discount factor approaches $1,$ and the discount factors at which the set of optimal policies changes.

fields
Optimization and Control
pdf ↗ page arXiv:2610.39326 ↗