Which Model for Which Role? Cost-Aware Pareto Search over LLM Agent Configurations

LLM agentic workflows increasingly employ different models for specialized roles. Users may prefer different trade-offs between task quality and deployment cost, sometimes also considering latency. Finding representative Pareto-optimal assignments incurs a separate search cost from evaluating multiple candidates. We present AgentOpt, our framework for evaluating and optimizing role-level model assignments, and introduce its search algorithm, Cost-Coupled Gittins (CC-Gittins), a multi-objective extension of single-objective cost-aware Gittins search for Pareto recommendation. It exploits the coupling between evaluation and deployment costs: assignments that cost more to deploy also cost more to evaluate. The search alternates between seeking higher quality and lower deployment cost, reusing each evaluation to inform both searches. After each batch, CC-Gittins recommends assignments that are nondominated under uncertainty-adjusted estimates of quality and deployment cost. We evaluate CC-Gittins on complete HotpotQA and MathQA matrices collected with our AgentOpt harness. We also evaluate the method with latency as a third objective. Across the two benchmarks, CC-Gittins recovers most of the true Pareto frontier with few false positives by 30% of the exhaustive-evaluation cost.