Latent Thought Models with Variational Bayes Inference-Time Computation

This talk introduces Latent Thought Models (LTMs), a novel class of language models that incorporate explicit latent thought vectors following a prior model in latent space. These vectors, inferred from observed ground tokens via posterior inference within the classical variational Bayes framework, are refined through inference-time computation. This process enables explicit abstraction and reasoning in a compact latent space, distinct from standard LLM’s unstructured embeddings. Experiments show that this new paradigm achieves superior sample and parameter efficiency compared to autoregressive models and introduces inference-time computation as a new scaling dimension beyond traditional LLMs.