Dense Cross-Tokenizer Distillation via Semantic Optimal Local Alignment

Cross-tokenizer knowledge distillation lacks a shared coordinatesystem between teacher and student vocabularies, and existingmethods each drop one of the two ingredients comparison needs:shape-based objectives preserve density but discard token identity,lexical alignment restores identity but uses surface form as a proxyfor meaning, and text-space matching compresses the teacher signalinto chunk-level events. We introduce SOLA (Semantic Optimal LocalAlignment), which reconstructs the missing coordinate systemexplicitly: it partitions inputs into decoded-text blocks viaMinimal Complete Correspondence, estimates a cross-vocabularycorrespondence cost from calibration text, and runs entropicoptimal transport on a bidirectional local support retrieved aroundeach teacher prediction. Under matched training and decodingprotocols, SOLA outperforms direct cross-tokenizer baselines onDolly instruction following, an UltraChat general-instructionbenchmark, and code and math domain transfer; a coverage--scorecorrelation and component ablations attribute the gain to theconstructed support.