How crucial is Transformer in Decision Transformer?

Decision Transformer (DT) is a recently proposed architecture for Reinforcement Learning that frames the decision-making process as an auto-regressive sequence modeling problem and uses a Transformer model to predict the next action in a sequence of states, actions, and rewards. In this paper, we analyze how crucial the Transformer model is in the complete DT architecture. Namely, we replace the Transformer by an LSTM model while keeping the other parts unchanged to obtain what we call a Decision LSTM model. We compare it to the Decision Transformer on continuous control tasks, including pendulum swing-up and stabilization tasks in simulation and on physical hardware. Our experiments show that Decision Transformer struggles with stabilization tasks, such as inverted pendulum and Furuta pendulum stabilization. On the other hand, the proposed Decision LSTM is able to achieve expert-level performance on these tasks, in addition to learning a swing-up controller on the real system. These results indicate that the strength of the Decision Transformer may lie in the overall sequential modeling architecture and not in the Transformer per se. Therefore, a further investigation into the effects of employing other sequence models in place of the Transformer is desirable.