Timezone: »

 
VARIATIONAL REPARAMETRIZED POLICY LEARNING WITH DIFFERENTIABLE PHYSICS
Zhiao Huang · Litian Liang · Zhan Ling · Xuanlin Li · Chuang Gan · Hao Su

@
in Deep Reinforcement Learning Workshop »
Event URL: https://openreview.net/forum?id=-23E9fxmI4 »

We study the problem of policy parameterization for reinforcement learning (RL) with high-dimensional continuous action space. Our goal is to find a good way to parameterize the policy of continuous RL as a multi-modality distribution. To this end, we propose to treat the continuous RL policy as a generative model over the distribution of optimal trajectories. We use a diffusion process-like strategy to model the policy and derive a novel variational bound which is the optimization objective to learn the policy. To maximize the objective by gradient descent, we introduce the Reparameterized Policy Gradient Theorem. This theorem elegantly connects classical method REINFORCE and trajectory return optimization for computing the gradient of a policy. Moreover, our method enjoys strong exploration ability due to the multi-modality policy parameterization; notably, when a strong differentiable world model presents, our method also enjoys the fast convergence speed of trajectory optimization. We evaluate our method on numerical problems and manipulation tasks within a differentiable simulator. Qualitative results show its ability to capture the multi-modality distribution of optimal trajectories, and quantitative results show that it can avoid local optima and outperforms baseline approaches.

[ Poster]  [ OpenReview]  [ Topia] 

Author Information

Zhiao Huang (University of California San Diego)
Litian Liang (University of California, San Diego)
Zhan Ling (UC San Diego)
Xuanlin Li (University of California, San Diego)
Chuang Gan (UMass Amherst/ MIT-IBM Watson AI Lab)
Hao Su (UCSD)

More from the Same Authors