The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

Does continued scaling of large language models (LLMs) yield diminishing returns? Real-world value often stems from the length of task an agent can complete. We start this work by observing the simple, but counter-intuitive fact that marginal gains in single-step accuracy can compound into exponential improvements in the length of a task the model can successfully complete. Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than an inability to reason. We propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve a long-horizon task. We find that larger models correctly execute for far more turns even when a small model has similarly high single-turn accuracy. Curiously, this is not just due to long-context limitations; we also observe a self-conditioning effect—models become more likely to make mistakes when the context contains their errors in prior turns. Self-conditioning does not reduce by just scaling model size. We find recent thinking models can not only perform long tasks in a single turn, but also self-condition less. We conclude by benchmarking frontier thinking models on the length of task they can execute in a single turn. Overall, by focusing on execution, we hope to reconcile debates on how LLMs can solve complex reasoning problems yet fail at simple tasks when made longer, and highlight the massive benefits of scaling model size and sequential test-time compute for long-horizon tasks.