Feedback Cycles in Exploratory Equilibria

Abstract

Entropy regularization smooths equilibrium policies in time-inconsistent stochastic control. At low temperature, the same Gibbs response can strongly amplify errors in learned rewards and dynamics. We show that the derivative of an exploratory equilibrium is governed by a backward Volterra-parabolic resolvent. Along an aligned positive mode, a lower bound has the same exponential order. A block decomposition identifies the source of the amplification: causal paths contribute powers of 1/tau, whereas a positive feedback cycle can produce exponential growth. At fixed temperature, a local equilibrium branch is twice differentiable with respect to finite-dimensional model parameters, which yields a function-valued delta method. A bounded uniformly elliptic diffusion realizes this path-cycle distinction in every finite dimension. Closing one positive cycle changes the root-n linear-response boundary from a power law to order 1/log n; along the cyclic Perron mode, right-endpoint discretization is relatively consistent exactly when N tau^2 -> infinity. An affine model also gives an exact nonlinear transition at the Lambert-W temperature beta T / W(beta T sqrt(n)). Numerical calculations illustrate these rates.

Disclosure

“nd fixed positive temperature, except in the exactly solvable affine model. Action-dependent diffusion, nonsmooth estimators, and graph criteria for state-dependent equilibria remain unresolved. Declaration of generative-AI assistance. OpenAI ChatGPT/Codex (July 2026) assisted with literature searches, drafting, editing, mathematical checks, and numerical code. The author assumes responsibility for all content. Code availability. The accompanying reproducibility archive contains th”

PDF page 26

Classification

Proof ideas or individual proof-step assistance

Multiplier

8

Verified

2026-08-23T18:59:33.203811Z

Structural counts