InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
Researchers have developed InfoQuant, a train-free method for low-bit activation quantization in large language models. This method employs Peak Suppression Orthogonal Transformation (PSOT) to shape activation distributions into quantization-friendly forms, which are characterized by smaller numerical ranges and sufficient dispersion. InfoQuant also incorporates adaptive outlier-token selection to improve robustness. Compared to prior post-training quantization and end-to-end training methods, InfoQuant consistently outperforms them, achieving 97% preservation of floating-point accuracy on average and reducing the performance gap of the LLaMA-2 13B model by 42%.
Save an API key to vote.