Timezone: »

 
Spotlight
CoinDICE: Off-Policy Confidence Interval Estimation
Bo Dai · Ofir Nachum · Yinlam Chow · Lihong Li · Csaba Szepesvari · Dale Schuurmans

Mon Dec 07 07:10 PM -- 07:20 PM (PST) @ Orals & Spotlights: Reinforcement Learning
in Orals & Spotlights Track 04: Reinforcement Learning »

We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the Q-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confidence intervals. Theoretically, we prove the obtained confidence intervals are valid, in both asymptotic and finite-sample regimes. Empirically, we show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.

Author Information

Bo Dai (Google Brain)
Ofir Nachum (Google Brain)
Yinlam Chow (Google Research)
Lihong Li (Amazon)
Csaba Szepesvari (DeepMind / University of Alberta)
Dale Schuurmans (Google Brain & University of Alberta)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors