Timezone: »

 
How crucial is Transformer in Decision Transformer?
Max Siebenborn · Boris Belousov · Junning Huang · Jan Peters

@
in Foundation Models for Decision Making »
Event URL: https://openreview.net/forum?id=RV6fghh1As_ »

Decision Transformer (DT) is a recently proposed architecture for Reinforcement Learning that frames the decision-making process as an auto-regressive sequence modeling problem and uses a Transformer model to predict the next action in a sequence of states, actions, and rewards. In this paper, we analyze how crucial the Transformer model is in the complete DT architecture. Namely, we replace the Transformer by an LSTM model while keeping the other parts unchanged to obtain what we call a Decision LSTM model. We compare it to the Decision Transformer on continuous control tasks, including pendulum swing-up and stabilization tasks in simulation and on physical hardware. Our experiments show that Decision Transformer struggles with stabilization tasks, such as inverted pendulum and Furuta pendulum stabilization. On the other hand, the proposed Decision LSTM is able to achieve expert-level performance on these tasks, in addition to learning a swing-up controller on the real system. These results indicate that the strength of the Decision Transformer may lie in the overall sequential modeling architecture and not in the Transformer per se. Therefore, a further investigation into the effects of employing other sequence models in place of the Transformer is desirable.

Author Information

Max Siebenborn (Technische Universität Darmstadt)
Boris Belousov (DFKI Darmstadt)
Junning Huang (TU Darmstadt)
Jan Peters (TU Darmstadt & MPI Intelligent Systems)

Jan Peters is a full professor (W3) for Intelligent Autonomous Systems at the Computer Science Department of the Technische Universitaet Darmstadt and at the same time a senior research scientist and group leader at the Max-Planck Institute for Intelligent Systems, where he heads the interdepartmental Robot Learning Group. Jan Peters has received the Dick Volz Best 2007 US PhD Thesis Runner-Up Award, the Robotics: Science & Systems - Early Career Spotlight, the INNS Young Investigator Award, and the IEEE Robotics & Automation Society‘s Early Career Award as well as numerous best paper awards. In 2015, he was awarded an ERC Starting Grant. Jan Peters has studied Computer Science, Electrical, Mechanical and Control Engineering at TU Munich and FernUni Hagen in Germany, at the National University of Singapore (NUS) and the University of Southern California (USC). He has received four Master‘s degrees in these disciplines as well as a Computer Science PhD from USC.

More from the Same Authors