From Budget Awareness to Budget Control: Confidence-Calibrated Intervention for Agentic AI

Large language model agents can produce long trajectories of interleaved reasoning and tool use, yet much of this computation may not improve the final answer. A serving system would therefore benefit from stopping a run once further computation provides little or no additional accuracy gain. A fixed token budget cannot adapt to whether an answer has already converged or is still evolving, while stopping when the model's confidence crosses a fixed threshold relies on a raw score that can be systematically overconfident. This paper studies whether an agent's own answer confidence, once calibrated into a probability of correctness, can support online stopping, and under what conditions doing so saves tokens without substantial accuracy loss. By replaying logged agent runs across several model sizes and tasks with tool use, we identify three findings. First, early stopping is beneficial only when trajectories contain a sufficiently long tail of low-yield steps; otherwise, reducing token consumption sacrifices accuracy. Second, confidence must be calibrated before a fixed threshold on its value can reliably support stopping, whereas interventions based on changes in confidence are less sensitive to such miscalibration. Third, when trajectories provide sufficient opportunity for early stopping, simple change detectors achieve modest token savings with limited accuracy loss, while more aggressive interventions obtain larger savings at the cost of accuracy. Guided by these findings, we propose a lightweight controller that considers stopping only after a minimum budget has been consumed and the answer has stabilized, and then learns which eligible runs can be stopped early. Across the models and tasks we study, the controller keeps the average accuracy loss near two percentage points while reducing token consumption, and avoids the larger accuracy degradation observed with more aggressive interventions.