Timezone: »

Variance-Aware Off-Policy Evaluation with Linear Function Approximation
Yifei Min · Tianhao Wang · Dongruo Zhou · Quanquan Gu

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @ Virtual

We study the off-policy evaluation (OPE) problem in reinforcement learning with linear function approximation, which aims to estimate the value function of a target policy based on the offline data collected by a behavior policy. We propose to incorporate the variance information of the value function to improve the sample efficiency of OPE. More specifically, for time-inhomogeneous episodic linear Markov decision processes (MDPs), we propose an algorithm, \texttt{VA-OPE}, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration. We show that our algorithm achieves a tighter error bound than the best-known result. We also provide a fine-grained characterization of the distribution shift between the behavior policy and the target policy. Extensive numerical experiments corroborate our theory.

Author Information

Yifei Min (Yale University)
Tianhao Wang (Yale University)
Dongruo Zhou (UCLA)
Quanquan Gu (UCLA)

More from the Same Authors