The Future of AI Evaluation

AI evaluation is finally shifting from aggregate performance metrics on specialised benchmarks to rich profiles of capabilities and propensities. Determining task demands and incentives, and comparing them against these profiles makes it possible to predict and explain performance and harm; this is the true objective of AI evaluation. Just as we are beginning to extend this to evaluations that are more human-centred and society-relevant, new challenges arise. In performance evaluations, we struggle to build verifiable, superhuman tasks of arbitrary difficulty. In safety evaluations, we can no longer build testing sandboxes that are realistic, contained and secure. In both cases, we face continual learning, with ever-evolving AI systems, and evaluation awareness, with deliberate sandbagging or judge manipulation. In this new world for AI evaluation, I propose a blend of pragmatism and foundational work. We must accept that AI systems will be unreliable, insecure, and misaligned, requiring evidence-based red lines and monitoring. We must also overhaul scaling laws and supervision, and explore new formalisms, such as relative 'supraintelligence', back in computational terms. And we must choose the right level and extent of AI evaluation automation to make AI oversight both scalable and controllable. AI development may have entered open-ended recursive self-improvement, but we cannot afford to let that happen with AI evaluation.