LLM/Model & Papers
LLM Quantization
Accurate Post-Training Quantization for Generative Pre-trained Transformers compresses LLMs by reducing weight bit depth, maintaining processing speed…
🧐 LLM Quantization - GPTQ | QAT | AWQ | GGUF | GGML | PTQ ( Created By Abonia Sojasingarayar)
🚩 GPTQ
Accurate Post-Training Quantization for Generative Pre-trained Transformers compresses LLMs by reducing weight bit depth, maintaining processing speed : https://github.com/IST-DASLab/gptq
🚩 AWQ
Activation-aware Weight Quantization focuses on activation levels for weight quantization, enhancing model performance : https://github.com/mit-han-lab/llm-awq
🚩 QAT
Quantization-Aware Training quantizes specific operators to INT8 precision, offering flexibility to tailor quantization parameters to different parts of the network : https://arxiv.org/abs/2305.17888
🚩 GGML
GGML is a C++ replica of LLM library, supporting multiple LLMs like LLaMA series & Falcon, optimized for CPU performance : https://github.com/ggerganov/ggml
🚩 GGUF
GPT-Generated unified Format introduced by llama.cpp team in 2023, replacing GGML. It offers unified file structure, *.safetensors to *.gguf conversion support, cross-platform compatibility inference on CPUs, GPUs, and MPUs : https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
🚩 PTQ
Post-Training Quantization reduces model parameters’ precision post-training, offering reduced memory consumption, faster inference times, and improved energy efficiency.
🚩 K-quants
It adjust bit precision for model weights based on importance, improving efficiency.Examples such as q2_K, q3_K_S, and q3_K_L, each with varying bit widths for different tensors.