World Drift or Question Drift? Question Identity as a Precondition for Continual World Model Evaluation

Continual world models are commonly evaluated by tracking their predictions under fixed labels across model updates. We argue that such comparisons are meaningful only if the underlying question (what is predicted, under which conditions and within which horizon) stays the same. We call a change of this question under an unchanged label question drift. For example, a predicted collision risk can increase even as the environment becomes safer, simply because its deadline was extended. We decompose a change in prediction into world drift, question drift and a change in prediction error, and prove that when only the current question definition is logged, the record may not even reveal whether the question changed. Question drift also distorts scoring: even correctly matched risk varies with event uncertainty, scoring revised predictions against the previous question's targets can overstate or understate error, and replacing historical questions hides forgetting. Controlled grid-world experiments with exact simulator probabilities illustrate each effect. We conclude with three recommendations for continual evaluation: log question definitions and evaluation protocols, score each prediction against its own target, and evaluate new checkpoints on fixed historical questions.