Timezone: »

 
Poster
SOPE: Spectrum of Off-Policy Estimators
Christina Yuan · Yash Chandak · Stephen Giguere · Philip Thomas · Scott Niekum

Fri Dec 10 08:30 AM -- 10:00 AM (PST) @
in Poster Session 8 »

Many sequential decision making problems are high-stakes and require off-policy evaluation (OPE) of a new policy using historical data collected using some other policy. One of the most common OPE techniques that provides unbiased estimates is trajectory based importance sampling (IS). However, due to the high variance of trajectory IS estimates, importance sampling methods based on state-action visitation distributions (SIS) have recently been adopted. Unfortunately, while SIS often provides lower variance estimates for long horizons, estimating the state-action distribution ratios can be challenging and lead to biased estimates. In this paper, we present a new perspective on this bias-variance trade-off and show the existence of a spectrum of estimators whose endpoints are SIS and IS. Additionally, we also establish a spectrum for doubly-robust and weighted version of these estimators. We provide empirical evidence that estimators in this spectrum can be used to trade-off between the bias and variance of IS and SIS and can achieve lower mean-squared error than both IS and SIS.

Keywords: Reinforcement Learning and Planning

Author Information

Christina Yuan (University of Texas at Austin)
Yash Chandak (University of Massachusetts Amherst)
Stephen Giguere (UMass Amherst)
Philip Thomas (University of Massachusetts Amherst)
Scott Niekum (UT Austin)

More from the Same Authors