Timezone: »

 
Poster
Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
Nihar Bhadresh Shah · Denny Zhou

Mon Dec 07 04:00 PM -- 08:59 PM (PST) @ 210 C #45
in Poster Session »

Crowdsourcing has gained immense popularity in machine learning applications for obtaining large amounts of labeled data. Crowdsourcing is cheap and fast, but suffers from the problem of low-quality data. To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest. We show that surprisingly, under a mild and natural "no-free-lunch" requirement, this mechanism is the one and only incentive-compatible payment mechanism possible. We also show that among all possible incentive-compatible mechanisms (that may or may not satisfy no-free-lunch), our mechanism makes the smallest possible payment to spammers. Interestingly, this unique mechanism takes a "multiplicative" form. The simplicity of the mechanism is an added benefit. In preliminary experiments involving over several hundred workers, we observe a significant reduction in the error rates under our unique mechanism for the same or lower monetary expenditure.

[ PDF] 

Author Information

Nihar Bhadresh Shah (UC Berkeley)
Denny Zhou (MSR)

More from the Same Authors