From Stationary Claims to Safe Bets: Poisson-Profile E-Processes for Markov Chains

E-values have formalised the study of continuous, anytime valid tests of hypotheses. In this work, we apply it to data from a Markov Chain, testing claims about its stationary features, like its mean $\pi_Pf$ under an unknown irreducible kernel $P$. The foremost issue here is that a stationary claim is global, whereas an e-process, built from one-step conditional bets, is local. We show that the Poisson Equation, a classical tool in Markov chain theory, can bridge that gap. For a finite irreducible kernel $P$, the null $\pi_Pf\le c$ holds if and only if there is a function $h$ with $f-c\le h-Ph$. We call $h$ a \textit{witness} for our claim, and turn the residuals $f(X_t)-c-h(X_t)+h(X_{t+1})$ into a conditionally subfair wealth process. Profiling over witnesses, via the lower envelope of their wealth processes, yields a valid e-process for the full composite stationary-mean null. The optimal wealth factor is then a numeraire e-variable, equal to a likelihood ratio against a reverse information projection. Taking the same infimum over these projections recovers the stationary Markov reverse-KL distance to the null. A predictable transition estimator attains this log-growth rate, and a matching lower bound on stopping time shows the construction is close to optimal. Applications of our construction include anytime-valid MCMC confidence sequences and Bellman-profile tests for adaptive average-reward reinforcement learning.