Two Lenses, Two Inference Modes: Comparing Autoregressive and Diffusion Language Models

The same language model can produce the same answer through two different routes. In autoregressive inference, it commits to tokens from left to right. In diffusion inference, it refines masked positions through denoising. The output may look similar, but does the answer take shape in the same way inside the model? We study this question using a controlled experiment in which only the inference procedure changes. Across 11 language understanding tasks, we compare autoregressive (AR) and diffusion (DLM) inference in the 3B and 8B Nemotron Labs Diffusion checkpoints, holding all model components fixed. Tests on separate validation prompts identify layers where the Logit Lens and a shared Tuned Lens reliably reveal preferences among the answer choices. We then causally test late computation by attenuating selected MLP updates at the answer position and measuring the resulting change in the final answer margin. The resulting picture contains a tension. At a fixed point in denoising, DLM representations look more like their final answer distributions. Yet the second-to-last MLP has greater causal influence under AR at both model scales. This contrast between representational alignment and sensitivity to a specific internal update motivates a closer look at how the two inference modes form an answer.