In-Context Learning of Hidden Markov Models by Loop Transformers

Markovian data have become a standard testbed for understanding in-context learning in Transformers. Hidden Markov Models (HMMs), however, remain comparatively underexplored because prediction requires inference over unobserved latent belief states. In this paper, we show that this challenge can be addressed by loop Transformers. We give an explicit encoder-decoder construction in which the encoder performs spectral learning from the prompt via a fixed Jacobi-SVD loop to recover observable operators, and the decoder predicts through a length-adaptive loop over the recovered operators. Our construction shows that looping enables latent-state inference whose computational depth grows with the prediction horizon, highlighting the benefit of loop architectures.