Paperlane

no. 12 · Tuesday 10 November 2026

Gradient Estimation for LLM-Embedded Simulation Optimization

Ruiyang Jin and Siyang Gao

math.OC · 31 pp

Abstract

Large language models (LLMs) are increasingly embedded in operational simulators to represent context-dependent human decisions, which opens new possibilities for simulation optimization. However, the LLM acts as a black box, so derivatives through its decisions are unavailable for gradient-based policy search. Finite difference estimation can avoid this requirement, but the required paired simulations that sample LLM actions independently often disagree, and such action switches persist under small perturbations, which inflates the gradient variance to order $c^{-2}$ for perturbation radius $c$. To control this variance, we develop a behavioral common random number framework. It first applies maximal coupling to the LLM's action distributions, preserving each simulation's behavioral law while making action switches as rare as possible, which reduces the variance bound to $O(c^{-1})$. It then applies first-switch conditioning, which averages over whether a switch occurs, together with an impact-aware extension that prioritizes influential switches. Under suitable stability conditions, both keep the variance bounded as $c$ shrinks. We further establish improved complexity bounds for projected stochastic approximation. Experiments on synthesized and LLM-embedded simulators confirm the theory and show substantial performance gains.

comments
31 pages, 5 figures
fields
Optimization and Control
pdf ↗ page arXiv:2611.10809 ↗