Reference Misgeneralization: Why Orthogonality Thesis Doesn’t Go Far Enough, and the Problem of Deep Brittleness

The orthogonality thesis (OT) holds that more or less any level of intelligence can be combined with more or less any final goal. We argue that the moves which make OT compelling make a second thesis compelling: the Orthogonality of Referentiality (ORT), that more or less any level of intelligence could in principle be situated within more or less any meaningful web references, given the broad degrees of freedom in the possible routes of embedding referential content. Work on AI risk focuses on value alignment and treats referential alignment as given; we argue it cannot be. Two consequences follow. First, deep brittleness: selection pressure rewards systems whose concepts pass our evaluations, so apparent referential agreement can hide divergence that our evaluations do not notice. Second, instrumental convergence presupposes a determinate self, but what a system preserves, and what counts as power for it, depend on its embedding. Such a system may look disordered from our standpoint. Strange-from-our-standpoint is not safe-from-our-standpoint.