Timezone: »

 
Poster
Analysis and Improvement of Policy Gradient Estimation
Tingting Zhao · Hirotaka Hachiya · Gang Niu · Masashi Sugiyama

Mon Dec 12 10:00 AM -- 02:59 PM (PST) @
in Poster Session and Reception »

Policy gradient is a useful model-free reinforcement learning approach, but it tends to suffer from instability of gradient estimates. In this paper, we analyze and improve the stability of policy gradient methods. We first prove that the variance of gradient estimates in the PGPE(policy gradients with parameter-based exploration) method is smaller than that of the classical REINFORCE method under a mild assumption. We then derive the optimal baseline for PGPE, which contributes to further reducing the variance. We also theoretically show that PGPE with the optimal baseline is more preferable than REINFORCE with the optimal baseline in terms of the variance of gradient estimates. Finally, we demonstrate the usefulness of the improved PGPE method through experiments.

[ PDF] 

Author Information

Tingting Zhao (Tokyo Institute of Technology)
Hirotaka Hachiya (Tokyo Institute of Technology)
Gang Niu (RIKEN)
Gang Niu

Gang Niu is currently an indefinite-term senior research scientist at RIKEN Center for Advanced Intelligence Project.

Masashi Sugiyama (RIKEN / University of Tokyo)

More from the Same Authors