MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization

Matryoshka Quantization (MatQuant) is a recent quantization approach showing that a single integer-quantized model can be served across multiple precisions, by slicing the most significant bits (MSB) at inference time. This enables a single checkpoint to cover a wide range of memory and latency budgets, but renders quantization much more challenging. In particular, the initial MatQuant relies either on methods using backpropagation in the form of expensive quantization-aware training (QAT) or post training quantization (PTQ) method OmniQuant, rather than fast one–shot forward-only PTQ methods. We address these limitations by introducing Post-Training Matryoshka Quantization (MatGPTQ), a new PTQ pipeline that produces a single parent model jointly optimized for multiple target precisions in one-shot, based on a small calibration set. MatGPTQ casts Matryoshka quantization as a multi–precision objective with improved bit-slicing and cross–bit error compensation, resulting in an algorithm that produces a multi-bit-width, ``sliceable'' model in a single pass. Across standard LLMs and benchmarks, MatGPTQ models are superior to MatQuant model, as well as Pareto optimal through non-uniform bit-widths allocation and mixed–precision execution. Overall, we establish a new state of the art for Matryoshka–style (nested) post–training quantization and make single–checkpoint, multi–precision deployment open and practical.