Quantization

As language model sizes continue to grow, the difficulty and cost of training have become well recognized. As the number of users increases, the cost of model inference is also rising, and may even become the primary factor limiting model deployment. Therefore, we need to compress models to accelerate the inference process, and model quantization is one effective method.

Parameters of large language models are typically stored as high-precision floating-point numbers, which requires substantial computational resources for model inference. Quantization techniques convert parameters stored in high-precision data types to low-precision data types, which can accelerate the inference process without changing the model’s parameter count and architecture. This method makes model deployment more cost-effective and feasible.

source

Floating-point numbers generally consist of 3 parts: sign bit, exponent bits, and mantissa bits. The larger the exponent bits, the larger the range of representable numbers. The larger the mantissa bits, the higher the precision of the number.

Quantization can be divided based on when quantization occurs into: post-training quantization and quantization-aware training, or based on how quantization parameters are determined into: static quantization and dynamic quantization.

PTQ

Post-Training Quantization (PTQ) generally refers to quantizing a model after pre-training is complete, determining quantization parameters based on a calibration dataset and then quantizing the model.

GPTQ

GPTQ (Group-wise Precision Tuning Quantization) is a static post-training quantization technique. “Static” means that once the pre-trained model is determined, the quantization parameters do not change after quantization. GPTQ quantization quantizes fp16 precision models to 4-bit, saving about 75% of memory while significantly improving inference speed. To use GPTQ quantized models, you need to specify the quantized model name or path, for example model_name_or_path: TechxGenus/Meta-Llama-3-8B-Instruct-GPTQ

QAT

In Quantization-Aware Training (QAT), the model is typically quantized during the pre-training process, then fine-tuned again on training data to obtain the final quantized model.

AWQ

AWQ (Activation-Aware Layer Quantization) is a static post-training quantization technique. Its idea is based on: a very small portion of weights are extremely important, and these weights will not be quantized to maintain performance. AWQ has the advantage of requiring a smaller calibration dataset and performs well on instruction-tuned and multi-modal models. To use AWQ quantized models, you need to specify the quantized model name or path, for example model_name_or_path: TechxGenus/Meta-Llama-3-8B-Instruct-AWQ

AQLM

AQLM (Additive Quantization of Language Models), as a PTQ method that only quantizes model weights, achieved the best performance at the time under 2-bit quantization, and also demonstrated performance improvements under 3-bit and 4-bit quantization. Although AQLM does not show the most significant improvement in model inference speed, its excellent performance under 2-bit quantization means you can deploy large models with extremely low memory usage.

OFTQ

OFTQ (On-the-fly Quantization) refers to quantizing the model directly during the inference stage without requiring a calibration dataset. OFTQ is a dynamic post-training quantization technique. OFTQ maintains performance while reducing memory usage. Therefore, when using the OFTQ quantization method, you need to specify the pre-trained model, specify the quantization method quantization_method, and specify the quantization bit quantization_bit. Below is an example configuration using the bitsandbytes quantization method:

model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
quantization_bit: 4
quantization_method: bitsandbytes  # choices: [bitsandbytes (4/8), hqq (2/3/4/5/6/8), eetq (8)]

bitsandbytes

Unlike GPTQ, bitsandbytes is a dynamic post-training quantization technique. bitsandbytes enables language models larger than 1B to maintain performance after 8-bit quantization without excessive performance loss. Models quantized with bitsandbytes 8-bit can save about 50% of memory while maintaining performance.

HQQ

Methods that rely on calibration datasets tend to have higher accuracy, while methods that do not rely on calibration datasets tend to be faster. HQQ (Half-Quadratic Quantization) aims to achieve a good balance between accuracy and speed. As a dynamic post-training quantization method, HQQ does not require a calibration phase, but can achieve accuracy comparable to methods that require calibration datasets, with extremely fast inference speed.

EETQ

EETQ (Easy and Efficient Quantization for Transformers) is a PTQ method that only quantizes model weights. It features fast speed and ease of use.