Timezone: »

 
Importance of Representation Learning for Off-Policy Fitted Q-Evaluation
Xian Wu · Nevena Lazic · Dong Yin · Cosmin Paduraru

@
in Offline Reinforcement Learning »

The goal of offline policy evaluation (OPE) is to evaluate target policies based on logged data with a possibly much different distribution. One of the most popular empirical approaches to OPE is fitted Q-evaluation (FQE). With linear function approximation, several works have found that FQE (and other OPE methods) exhibit exponential error amplification in the problem horizon, except under very strong assumptions. Given the empirical success of deep FQE, in this work we examine the effect of implicit regularization through deep architectures and loss functions on the divergence and performance of FQE. We find that divergence does occur with simple feed-forward architectures, but can be mitigated using various architectures and algorithmic techniques, such as ResNet architectures, learning a shared representation between multiple target policies, and hypermodels. Our results suggest interesting directions for future work, including analyzing the effect of architecture on stability of fixed-point updates which are ubiquitous in modern reinforcement learning.

Author Information

Xian Wu (University of California, Berkeley)
Nevena Lazic (DeepMind)
Dong Yin (DeepMind)
Cosmin Paduraru (DeepMind)

More from the Same Authors