`

Timezone: »

 
Poster
Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction
Jacob Steinhardt · Gregory Valiant · Moses Charikar

Wed Dec 07 09:00 AM -- 12:30 PM (PST) @ Area 5+6+7+8 #26 #None
We consider a crowdsourcing model in which n workers are asked to rate the quality of n items previously generated by other workers. An unknown set of $\alpha n$ workers generate reliable ratings, while the remaining workers may behave arbitrarily and possibly adversarially. The manager of the experiment can also manually evaluate the quality of a small number of items, and wishes to curate together almost all of the high-quality items with at most an fraction of low-quality items. Perhaps surprisingly, we show that this is possible with an amount of work required of the manager, and each worker, that does not scale with n: the dataset can be curated with $\tilde{O}(1/\beta\alpha\epsilon^4)$ ratings per worker, and $\tilde{O}(1/\beta\epsilon^2)$ ratings by the manager, where $\beta$ is the fraction of high-quality items. Our results extend to the more general setting of peer prediction, including peer grading in online classrooms.

Author Information

Jacob Steinhardt (Stanford University)
Gregory Valiant (Stanford University)
Moses Charikar (Stanford University)

More from the Same Authors

  • 2020 Poster: Instance Based Approximations to Profile Maximum Likelihood »
    Nima Anari · Moses Charikar · Kirankumar Shiragur · Aaron Sidford
  • 2019 : Poster Session »
    Gergely Flamich · Shashanka Ubaru · Charles Zheng · Josip Djolonga · Kristoffer Wickstrøm · Diego Granziol · Konstantinos Pitas · Jun Li · Robert Williamson · Sangwoong Yoon · Kwot Sin Lee · Julian Zilly · Linda Petrini · Ian Fischer · Zhe Dong · Alexander Alemi · Bao-Ngoc Nguyen · Rob Brekelmans · Tailin Wu · Aditya Mahajan · Alexander Li · Kirankumar Shiragur · Yair Carmon · Linara Adilova · SHIYU LIU · Bang An · Sanjeeb Dash · Oktay Gunluk · Arya Mazumdar · Mehul Motani · Julia Rosenzweig · Michael Kamp · Marton Havasi · Leighton P Barnes · Zhengqing Zhou · Yi Hao · Dylan Foster · Yuval Benjamini · Nati Srebro · Michael Tschannen · Paul Rubenstein · Sylvain Gelly · John Duchi · Aaron Sidford · Robin Ru · Stefan Zohren · Murtaza Dalal · Michael A Osborne · Stephen J Roberts · Moses Charikar · Jayakumar Subramanian · Xiaodi Fan · Max Schwarzer · Nicholas Roberts · Simon Lacoste-Julien · Vinay Prabhu · Aram Galstyan · Greg Ver Steeg · Lalitha Sankar · Yung-Kyun Noh · Gautam Dasarathy · Frank Park · Ngai-Man (Man) Cheung · Ngoc-Trung Tran · Linxiao Yang · Ben Poole · Andrea Censi · Tristan Sylvain · R Devon Hjelm · Bangjie Liu · Jose Gallego-Posada · Tyler Sypherd · Kai Yang · Jan Nikolas Morshuis
  • 2019 Poster: A General Framework for Symmetric Property Estimation »
    Moses Charikar · Kirankumar Shiragur · Aaron Sidford
  • 2017 Poster: Learning Overcomplete HMMs »
    Vatsal Sharan · Sham Kakade · Percy Liang · Gregory Valiant
  • 2017 Poster: Learning Populations of Parameters »
    Kevin Tian · Weihao Kong · Gregory Valiant
  • 2016 Poster: Unsupervised Risk Estimation Using Only Conditional Independence Structure »
    Jacob Steinhardt · Percy Liang
  • 2015 : When Your Big Data Seems Too Small »
    Gregory Valiant
  • 2015 Poster: Testing Closeness With Unequal Sized Samples »
    Bhaswar Bhattacharya · Gregory Valiant