Title: Sensing and simulating the real world for next generation autonomous mobility

Abstract: Zoox is developing the first ground-up, fully autonomous vehicle fleet and the supporting ecosystem required to bring this technology to market. Sitting at the intersection of robotics, machine learning, and design, Zoox aims to provide the next generation of mobility-as-a-service in urban environments. A key part of our design is safety: in addition to providing a great user experience, we aim to design robots that are significantly safer than human drivers. To ensure this, it is critical to maintain accurate perception of objects in the world that our robots need to react to. To that end, Zoox has taken a holistic approach to its sensor choice and placement, computational power, and algorithms. In the first half of our talk we will describe some of the sensors and algorithms we use to ensure that the robot is able to perceive all objects that it needs to react to, and that it is able to do so with sufficiently low latency.

In addition to developing the necessary technology, it is imperative to be able to validate it. Zoox is developing an advanced 3D simulation framework to help verify that our vehicle is safe while also being able to complete its missions successfully. This framework provides the foundation for generating highly realistic simulated data which is used as ground truth for testing algorithms as well as to train machine learning algorithms in cases when sufficient real-world data is not readily available. The second part of the talk will provide an overview of this framework and, in particular, discuss how we quantify the fidelity of the various simulated sensor types used by our robot in its perception stack.

Bios: Sarah is the Director of Vision Detection and Tracking at Zoox, where her team focuses on perception for cameras, including detecting and tracking objects of interest reliably and in real time. Sarah has been at Zoox for over three years and before that she has almost a decade of experience working at NVIDIA across multiple roles. Amongst her many achievements at NVIDIA, she contributed to the implementation of novel real-time simulation and rendering of algorithms for video games, managed a team working on profiling and optimizing code for high performance computing and super computers, and served as a technical lead for the computer vision team focusing on self-driving technology.

Ekaterina is a senior research engineer at Zoox. Her goal is to quantify how realistic simulated sensors need to be to to enable end-to-end testing of the software stack and create synthetic training data to help improve perception models. Before Zoox, she was a postdoc with Tony Jebara and Rafael Yuste at Columbia University, where she developed large scale graphical models to quantify neural activity in the mouse visual cortex. Ekaterina obtained her PhD with Martial Hebert and Fernando De la Torre at Carnegie Mellon University, where her thesis work was on action classification and segmentation in videos.