What Does a Model Learn from a Changing World?

Models absorb world knowledge as they train, yet the world is not static: some facts remain fixed while others change, sometimes predictably. In this work, we study what models learn from observing changing knowledge during training. In a controlled setting, we show that models exposed to temporal change acquire an \textit{inductive bias} over future updates: they preserve constant facts under an update while more readily absorbing updates consistent with the historical dynamics. We additionally discover that models learn one of two distinct fact storage mechanisms, depending on how they observe change: observing changing facts in chronological order causes facts to be stored in \textit{individualized slots}, in which updates stay local. When facts from different time periods are shuffled together, models store facts in co-evolving \textit{equivalence classes}, which continue to evolve together under subsequent updates. We trace these two mechanisms to opposing training dynamics, in which ordered training learns in a high-rank regime while shuffled training remains low-rank. Our findings demonstrate that observing change during training can shape how models respond to future updates, providing a promising avenue for training inherently updatable models.