Timezone: »

 
Workshop
HAMLETS: Human And Model in the Loop Evaluation and Training Strategies
Divyansh Kaushik · Bhargavi Paranjape · Forough Arabshahi · Yanai Elazar · Yixin Nie · Max Bartolo · Polina Kirichenko · Pontus Lars Erik Saito Stenetorp · Mohit Bansal · Zachary Lipton · Douwe Kiela

Sat Dec 12 08:15 AM -- 08:00 PM (PST) @ None
in Workshops Sat »
Event URL: https://hamlets-workshop.github.io »

Human involvement in AI system design, development, and evaluation is critical to ensure that the insights being derived are practical, and the systems built are meaningful, reliable, and relatable to those who need them. Humans play an integral role in all stages of machine learning development, be it during data generation, interactively teaching machines, or interpreting, evaluating and debugging models. With growing interest in such “human in the loop” learning, we aim to highlight new and emerging research opportunities for the ML community that arise from the evolving needs to design evaluation and training strategies for humans and models in the loop. The specific focus of this workshop is on emerging and under-explored areas of human- and model-in-the-loop learning, such as employing humans to seek richer forms of feedback for data than labels alone, learning from dynamic adversarial data collection with humans employed to find weaknesses in models, learning from human teachers instructing computers through conversation and/or demonstration, investigating the role of humans in model interpretability, and assessing social impact of ML systems. This workshop aims to bring together interdisciplinary researchers from academia and industry to discuss major challenges, outline recent advances, and facilitate future research in these areas.

Sat 8:15 a.m. - 8:30 a.m.
Opening Remarks
Divyansh Kaushik, Bhargavi Paranjape, Douwe Kiela
Sat 8:30 a.m. - 8:55 a.m.
Invited Talk by Tom Mitchell (Carnegie Mellon) (Talk) Video
Divyansh Kaushik
Sat 8:55 a.m. - 9:00 a.m.
Break Video
Divyansh Kaushik
Sat 9:00 a.m. - 9:25 a.m.
Invited Talk by Jenn Wortman Vaughan (Microsoft Research) (Talk) Video
Divyansh Kaushik
Sat 9:25 a.m. - 9:30 a.m.
Break Video
Divyansh Kaushik
Sat 9:30 a.m. - 9:55 a.m.
Invited Talk by Sanjoy Dasgupta (UCSD) (Talk) Video
Divyansh Kaushik
Sat 9:55 a.m. - 10:00 a.m.
Break Video
Divyansh Kaushik
Sat 10:00 a.m. - 10:25 a.m.
Invited Talk by Finale Doshi-Velez (Harvard) (Talk) Video
Divyansh Kaushik
Sat 10:25 a.m. - 11:30 a.m.
Q & A and Panel Session with Tom Mitchell, Jenn Wortman Vaughan, Sanjoy Dasgupta, and Finale Doshi-Velez (Discussion Panel)
Tom Mitchell, Jenn Wortman Vaughan, Sanjoy Dasgupta, Finale Doshi-Velez, Zachary Lipton
Sat 11:30 a.m. - 12:45 p.m.
Poster Session 1 (GatherTown Posters)
Sat 12:45 p.m. - 1:55 p.m.
Break (Or meet on GatherTown) (Break)
Sat 1:55 p.m. - 2:00 p.m.
Break Video
Divyansh Kaushik
Sat 2:00 p.m. - 2:25 p.m.
Invited Talk by Dan Weld (University of Washington) (Talk) Video
Divyansh Kaushik
Sat 2:25 p.m. - 2:30 p.m.
Break Video
Divyansh Kaushik
Sat 2:30 p.m. - 2:55 p.m.
Invited Talk by Kristen Grauman (UT Austin, FAIR) (Talk) Video
Divyansh Kaushik
Sat 2:55 p.m. - 3:00 p.m.
Break Video
Divyansh Kaushik
Sat 3:00 p.m. - 3:25 p.m.
Invited Talk by Scott Yih (FAIR) (Talk) Video
Divyansh Kaushik
Sat 3:25 p.m. - 3:30 p.m.
Break Video
Divyansh Kaushik
Sat 3:30 p.m. - 3:55 p.m.
Invited Talk by Emma Brunskill (Stanford) (Talk) Video
Divyansh Kaushik
Sat 3:55 p.m. - 4:00 p.m.
Break Video
Divyansh Kaushik
Sat 4:00 p.m. - 4:25 p.m.
Invited Talk by Alex Ratner (Snorkel, University of Washington) (Talk) Video
Divyansh Kaushik
Sat 4:25 p.m. - 5:30 p.m.
Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner (Discussion Panel)
Kristen Grauman, Scott Yih, Alexander Ratner, Emma Brunskill, Douwe Kiela, Dan S. Weld
Sat 5:30 p.m. - 6:45 p.m.
Poster Session 2 (GatherTown Posters)
Sat 6:45 p.m. - 7:00 p.m.
Closing Remarks (Closing)
Sat 7:00 p.m. - 8:00 p.m.
Social/Get-Together on GatherTown (GatherTown)

Author Information

Divyansh Kaushik (Carnegie Mellon University)
Bhargavi Paranjape (University of Washington)
Forough Arabshahi (Facebook)
Yanai Elazar (Bar-Ilan University)
Yixin Nie (UNC-Chapel Hill)
Max Bartolo (UCL)
Polina Kirichenko (New York University)
Pontus Lars Erik Saito Stenetorp (University of Tokyo)
Mohit Bansal (UNC Chapel Hill)
Zachary Lipton (Carnegie Mellon University)
Douwe Kiela (Facebook AI Research)

More from the Same Authors