Epistemic Exams

Despite AI agents reportedly being able to autonomously complete many knowledge work tasks, the economic impacts of AI in many companies have been relatively muted. We argue that a key disanalogy between these AI benchmarks and creating value in the world is that current benchmarks measure the skill of an agent, what they know. However, for value creation, we often would like agents to have good epistemics, that is, to know what they know, so that their work can be appropriately deferred to, trusted only with verification, or discarded as appropriate. We introduce the concept of an epistemic exam as a natural means to test and train AI systems, and develop the general theory for how to score them. An epistemic exam is an exam which asks the test taker to report their answer and the probability they believe it to be correct. An exam measures what you know. An epistemic exam, in addition, measures what you \emph{know you know}. We derive BIFOCAL as a natural scoring rule with which to measure AI capabilities via epistemic exams and test frontier LLMs on the epistemic exam variants of current benchmarks. We believe epistemic exams can help AI developers, governments, and the public to have a clearer picture of emerging AI capabilities, which can in turn allow for better forecasting of automation, encourage the development of safer and more useful AI products, and refine discussions about the benefits and risks of AI systems.