Unbiased Efficient Feature Counts for Inverse RL

Feature counts play a crucial role when computing good reward weights in inverse reinforcement learning. Despite their importance, little work has focused on developing better methods for estimating feature counts. In this work, we propose a new method for estimating feature counts for scenarios with a small number of long demonstrations. Most existing algorithms perform poorly in this scenario. In particular, we propose two new algorithms, E-DLS and E-SLS, which can efficiently use a small number of long demonstrations to estimate feature counts. We show that E-SLS estimates are unbiased, which is the first such estimation algorithm. Our experimental results on benchmark problems demonstrate better learned reward weights when feature counts are estimated with E-DLS and E-SLS compared to other popular methods.