Paperlane

no. 16 · Tuesday 17 November 2026

Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear–Quadratic Reinforcement Learning Problems

Yilie Huang and Xun Yu Zhou

math.OC · v3 · 31 pp

Abstract

We study reinforcement learning (RL) for a class of continuous-time stochastic linear-quadratic (LQ) control problems, where volatilities depend on both states and controls while states are scalar-valued and there are no running control rewards, the last feature also leading to the so-called indefinite LQ controls. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in the existing literature, which require extensive tuning for implementations and ignore learning progress during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.

comments
31 pages. Author-accepted manuscript, with publication details and updated author notes
journal
IEEE Transactions on Automatic Control, Early Access, pp. 1-16, 16 September 2026
fields
Optimization and Control
pdf ↗ page arXiv:2507.00358 ↗