ProxySearch: Decoupled Inference-Time Scaling for Diffusion Models via Asymmetric Noise-Rank Transfer

Noise search at inference time improves text-to-image diffusion quality but requires many multi-step teacher rollouts per query. Distilled few-step models are now routinely released for efficient inference, but their potential as proxies inside noise search is uncharacterised. We measure proxy--teacher rank agreement across three distillation regimes and three backbones spanning DiT, MMDiT, and UNet, and find a sharp asymmetry: eight closed-form image statistics preserve rank from the distilled proxy to the teacher (Spearman $\rho$ up to $0.80$), while learned semantic scorers decorrelate ($\rho \le 0.12$). Null controls localise the transfer to distillation training rather than few-step denoising or initial noise characteristics. We therefore reformulate noise search as a \emph{joint constraint} and propose ProxySearch: the proxy filters candidates by statistical match in $1$ step, the teacher argmaxes a user-chosen semantic reward over the survivors without a tunable weight trading semantic reward against the statistical target. \textsc{ProxySearch} matches or beats teacher-only argmax on compositional accuracy and human preference, while tightening the statistical target by $\sim 23\%$ in paired statistical $\ell_1$ relative to teacher-only argmax at matched compute on FLUX-$1024$.