Offline reinforcement learning has shown promise for solving tasks in safety-critical settings, such as clinical decision support. Its application, however, has been limited by the need for interpretability and interactivity for clinicians. To address these challenges, we propose medical decision transformer (MeDT), a novel and versatile framework based on the goal-conditioned reinforcement learning (RL) paradigm for sepsis treatment recommendation. MeDT is based on the decision transformer architecture, and conditions the model on expected treatment outcomes, hindsight patient acuity scores, past dosages and the patient’s current and past medical state at every timestep. This allows it to consider the complete context of a patient’s medical history, enabling more informed decision-making. By conditioning the policy’s generation of actions on user-specified goals at every timestep, MeDT enables clinician interactability while avoiding the problem of sparse rewards. Using data from the MIMIC-III dataset, we show that MeDT produces interventions that outperform or are competitive with existing methods while enabling a more interpretable, personalized and clinician-directed approach.