Poster
Near-Optimal Goal-Oriented Reinforcement Learning in Non-Stationary Environments
Liyu Chen · Haipeng Luo
Hall J (level 1) #822
Keywords: [ dynamic regret minimization ] [ non-stationary environments ] [ Reinforcement Learning ] [ stochastic shortest path ]
Abstract:
We initiate the study of dynamic regret minimization for goal-oriented reinforcement learning modeled by a non-stationary stochastic shortest path problem with changing cost and transition functions.We start by establishing a lower bound , where is the maximum expected cost of the optimal policy of any episode starting from any state, is the maximum hitting time of the optimal policy of any episode starting from the initial state, is the number of state-action pairs, and are the amount of changes of the cost and transition functions respectively, and is the number of episodes.The different roles of and in this lower bound inspire us to design algorithms that estimate costs and transitions separately.Specifically, assuming the knowledge of and , we develop a simple but sub-optimal algorithm and another more involved minimax optimal algorithm (up to logarithmic terms).These algorithms combine the ideas of finite-horizon approximation [Chen et al., 2021b], special Bernstein-style bonuses of the MVP algorithm [Zhang et al., 2020], adaptive confidence widening [Wei and Luo, 2021], as well as some new techniques such as properly penalizing long-horizon policies.Finally, when and are unknown, we develop a variant of the MASTER algorithm [Wei and Luo, 2021] and integrate the aforementioned ideas into it to achieve regret, where is the unknown number of changes of the environment.
Chat is not available.