Interactive fiction (IF) games present very different challenges than the vision and control-based games that learning agents have previously excelled at. Solving IF games requires human-like language understanding, commonsense reasoning, planning, and deduction skills. This paper provides a testbed for rapid development of new agents that exhibit these skills by introducing Jericho, a fast, fully-featured interface to fifty-six popular and challenging IF games. We also present initial work towards solving these games in the form of an agent that won the 2018 Text-Based Adventure AI Competition. Finally, we conduct a comprehensive evaluation between NAIL, our agent, and several other IF agents in a richer set of text game environments, and point to directions in which agents can improve. We are optimistic that tools such as Jericho and NAIL will help the community make progress towards language-understanding agents.