Timezone: »

 
Poster
Context-dependent upper-confidence bounds for directed exploration
Raksha Kumaraswamy · Matthew Schlegel · Adam White · Martha White

Wed Dec 05 02:00 PM -- 04:00 PM (PST) @ Room 517 AB #156
in Wed Poster Session B »

Directed exploration strategies for reinforcement learning are critical for learning an optimal policy in a minimal number of interactions with the environment. Many algorithms use optimism to direct exploration, either through visitation estimates or upper confidence bounds, as opposed to data-inefficient strategies like e-greedy that use random, undirected exploration. Most data-efficient exploration methods require significant computation, typically relying on a learned model to guide exploration. Least-squares methods have the potential to provide some of the data-efficiency benefits of model-based approaches—because they summarize past interactions—with the computation closer to that of model-free approaches. In this work, we provide a novel, computationally efficient, incremental exploration strategy, leveraging this property of least-squares temporal difference learning (LSTD). We derive upper confidence bounds on the action-values learned by LSTD, with context-dependent (or state-dependent) noise variance. Such context-dependent noise focuses exploration on a subset of variable states, and allows for reduced exploration in other states. We empirically demonstrate that our algorithm can converge more quickly than other incremental exploration strategies using confidence estimates on action-values.

Keywords: Decision and Control · Reinforcement Learning · Markov Decision Processes · Exploration
[ Paper]  [ Poster] 

Author Information

Raksha Kumaraswamy (University of Alberta)
Matthew Schlegel (University of Alberta)

An AI and coffee enthusiast with research experience in RL and ML. Currently pursuing a PhD at the University of Alberta! Excited about off-policy policy evaluation, general value functions, understanding the behavior of artificial neural networks, and cognitive science (specifically cognitive neuroscience).

Adam White (University of Alberta; DeepMind)
Martha White (University of Alberta)

More from the Same Authors