SeqPAC: PAC-Certified Labeling with Sequential Ensembles

Synthetic labels are widely used to build large training and evaluation datasets, but optimizing weighted LLM ensembles for accuracy alone leaves a central deployment question unanswered: how should a system trade automatic coverage against model-call and human-annotation cost while controlling population error? We introduce SeqPAC, a \emph{probably approximately correct} (PAC)-certified sequential weighted-ensemble framework for this problem. For each input, SeqPAC learns weights that determine both which models to query and how to aggregate their responses. After every response, the accumulated evidence is used to release the current label, query the next precommitted model, or defer to a human. We characterize the fixed-aggregation oracle stopping policy and show that, under directed single-crossing conditions, its decisions reduce to two thresholds on an ordered scalar score. This analysis motivates SeqPAC-TwoTail, a practical policy based on a single round-aware $1$-D score. We calibrate complete stopping policies using fixed-sequence learn-then-test and select the certified member with the smallest development-estimated cost. Across eleven reasoning and knowledge benchmarks---four in-distribution and seven held out---and eighteen open $4$--$9$B model instances, SeqPAC-TwoTail attains the highest automatic coverage among the scalar policies at the evSynthetic labels are widely used to build large training and evaluation datasets, but aluated error targets while retaining early exits and being substantially easier to train than a backward-fitted value policy. The experiments further assess input-dependent weighting, early stopping, and out-of-distribution transfer, tracing the resulting certified coverage--computation tradeoffs. Together, SeqPAC provides a practical route to cost-efficient synthetic labeling with a population-level statistical guarantee.