Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

Finetuning Language Models often requires enforcing constraints on individual inputs without compromising performance. However, current alignment methods typically impose constraints only on average, which can induce undesirable disparities across inputs or users. We propose a finetuning framework that addresses this limitation by enforcing alignment requirements as per-sample constraints. To handle the optimization challenges inherent in this approach, we utilize an augmented Lagrangian formulation in the dual domain. Since pointwise constraints can be overly restrictive in low-probability regions or in the presence of outliers, we introduce a learned, sample-dependent relaxation that minimally relaxes constraints to optimize objective performance. We demonstrate the versatility of our framework across three small language model tasks: safety in instruction following, preference satisfaction in function calling, and length-aware re-ranking. Across these settings, our approach reduces tail constraint violations while largely preserving or improving downstream performance