Online Fine-Tuning with Uncertainty Quantification for Offline Pre-Trained Agents

This paper proposes an online fine-tuning with uncertainty quantification for offline pre-trained agents in deep reinforcement learning (RL). Offline RL allows agents to learn from pre-collected datasets without additional environment interactions, but faces challenges like distributional shifts and uncertainty during online fine-tuning. Our method incorporates uncertainty quantification into an ensemble of pessimistic Q-functions. The uncertainty-based penalization mitigates the effects of distributional shift during online fine-tuning, resulting in more stable and sample-efficient learning. Through experiments on D4RL locomotion tasks with various datasets, we demonstrate that the proposed method outperforms existing baseline methods, achieving superior performance with fewer environment interactions. The results highlight the effectiveness of uncertainty quantification in managing distributional shift and improving the robustness of online fine-tuning from offline pre-trained agents.