OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Key points

• However, preserving accuracy using NVFP4 requires careful quantization.1
• In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4).1
• For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation).1
• This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one.1

Sources (1)

• [1]OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 12:17 PM

  NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration.

  However, preserving accuracy using NVFP4 requires careful quantization.

Extractive summary: sentences quoted from the sources.