File size: 485 Bytes
5fa1a76 |
1 2 3 4 5 6 7 |
Before you begin, make sure the following libraries are installed: pip install auto-gptq pip install git+https://github.com/huggingface/optimum.git pip install git+https://github.com/huggingface/transformers.git pip install --upgrade accelerate To quantize a model (currently only supported for text models), you need to create a [GPTQConfig] class and set the number of bits to quantize to, a dataset to calibrate the weights for quantization, and a tokenizer to prepare the dataset. |