Speakeasy: Auditing Cheap Signals in Peer Prediction

How can we elicit truthful information from strategic agents? Traditional peer prediction mechanisms incentivize truthful reporting by rewarding agreement between agents' reports, but break down when agents can submit cheap but correlated signals---such as outputs from different LLMs. We introduce the Speakeasy mechanism, a framework for peer prediction when the principal can also sample these cheap signals. The mechanism augments any peer prediction rule with an audit strategy that penalizes reports for matching the principal's samples. We give a polynomial-time algorithm for the optimal audit strategy, so that truthful reporting is a strict Bayes--Nash equilibrium and yields the highest welfare against agents choosing between reporting truthfully and copying a single cheap signal. We extend the guarantee to more complex misreport strategies that mix multiple cheap signals, showing that Speakeasy preserves the truthfulness of the base mechanism. We empirically test the Speakeasy mechanism against classical peer prediction mechanisms on a peer-review dataset of ICLR submissions, with reviews from human reviewers and several LLMs. Several classical mechanisms fail to make truthful reporting a Bayes--Nash equilibrium once LLMs are available, while Speakeasy restores it for all of them.