WayraPPL: Capability-Dependent Perplexity Skipping in Continual Pretraining

Continual pretraining (CPT) adapts a base model to a new domain under a compute budget, and perplexity-based data pruning is an attractive way to spend that budget efficiently. For pretraining from scratch, prior work established that removing low-perplexity examples improves data efficiency, and that a small fraction of tokens suffices for general evaluations. We show that in continual pretraining this prescription is \emph{capability-dependent} and must be applied with care. We first introduce \textbf{WayraPPL}, a 55M-parameter multilingual student distilled from a 1B teacher that predicts teacher perplexity at $0.94$ Spearman correlation while running $\sim$10.5$\times$ faster and using $4\times$ less memory, making corpus-scale difficulty scoring affordable and reusable across an entire recipe sweep. Using WayraPPL to score a 342B-token English/Spanish/Portuguese corpus, we continually pretrain Llama~3.1~8B while removing the lowest-perplexity 25/50/75\% of examples, and evaluate on general knowledge (MMLU), commonsense (HellaSwag), Latin American cultural knowledge (Choclo), and regional factuality (Trueque). We find a clean dissociation: general and cultural-knowledge benchmarks stay flat as easy data is removed, while regional factuality declines monotonically, because its signal is spread across the perplexity range rather than concentrated in the high-perplexity tail. Consequently the easy quartile can be dropped for a $25\%$ compute reduction at no measured cost, but more aggressive filtering trades regional adaptation for compute. We connect this to a saturation analysis across the 1B, 8B, and 70B scales that predicts which benchmarks a recipe can move, and we corroborate the finding on a Llama~3.1~70B CPT model. Our results reframe perplexity in continual pretraining as a signal of \emph{per-example adaptation gain} rather than of data quality. We release the scorer weights and code.