RL-Glue: From Grid Worlds to Sensor Rich Robots

We propose to demonstrate a suite of software and robotics projects for reinforcement learning (RL) that have been in development for several years in Dr. Richard Sutton's group at the University of Alberta. Specifically, the three projects that we intend to showcase are: RL-Glue interface, the CritterBot robotic platform, and RL-Viz experimentation platform. The demonstration will illustrate how these projects allow researchers to develop learning agents that can be evaluated in a graphical simulation (RL-Viz) and on a mobile robot (CritterBot). RL-Glue is a language- and platform-independent protocol for evaluating reinforcement learning agents with environment programs. RL-Glue separates the agent- and environment-development process so that each can be written in different languages and even executed over the Internet from different computers. RL-Glue have had a significant influence on the way empirical comparisons are done in reinforcement learning. RL-Glue has been used to evaluate agents in four international competitions at high profile machine learning conferences. The most recent competition, held in conjunction with ICML 08, attracted over 150 teams. The final test phase of the competition included over 20 teams comprised of more than 40 participants. RL-Glue has been used by several university instructors, in several countries, to teach reinforcement learning. Several researchers have used RL-Glue to benchmark their agents in papers published in top international conferences, including NIPS. The CritterBot is an ongoing project at the University of Alberta whose goal is to add a further robotics effort to challenge, direct, and inspire the research on grounded artificial intelligence. This robot is small and mobile and outfitted with an unusually rich set of sensors, including sensors for touch, acceleration, motion, sound, vision, and several kinds of proximity. The initial objective is for the robot to form an extended multi-level model of the relationships among its sensors and between its sensors and its actuators. We have proposed that higher-level knowledge can be grounded in raw data of sensations and actions; this robotic platform will challenge and inspire us to see if it can really be done. We also plan to use this platform as a test case for rapid learning and for the use of reinforcement learning by non-experts. We would like a person whose has no training to be able to teach the system new ways of behaving in an intuitive manner much as one might train a particularly cooperative dog. Learning agents can interact with the CritterBot through RL-Glue just like with any other RL-Glue environment. RL-Viz provides the reinforcement learning community for the first time ever with a flexible, general, standardized, cross language and cross platform protocol/framework for managing and visualizing the interaction between agents and environments in reinforcement learning experiments. The RL-Viz project includes several state-of-the-art tasks used in learning research including, Tetris, a remote-controlled helicopter simulator provided by Andrew Ng's team at Stanford, keep-away soccer and a real-time strategy engine. RL-Viz is a protocol and library layered on top of RL-Glue. RL-Viz supports advanced features such as visualization of environments and agents and run-time loading of agents and environments. The software for most recent RL competition (mentioned above) was based on RL-Viz. We will present the latest developments in the RL-Glue project and demonstrate how RL-Glue provides a novel, unified architecture for developing reinforcement learning algorithms for simulation and physical experiments. This framework makes it easier to compare the performance of agents in a variety of simulated and physical tasks.