Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking

This position paper argues that ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics. We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band before human reviewing. Given a venue-specific minimum review target (e.g., 3 or 4), we propose using this LLM based borderline signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for any accept/reject decision. We provide a simple expected-impact estimate based on (i) the overlap between the predicted and true borderline sets ($\rho$) and (ii) the incremental value of an extra review near the boundary ($\Delta$), using review data from ICLR 2025.