from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig | |
model_id = "facebook/opt-125m" | |
tokenizer = AutoTokenizer.from_pretrained(model_id) | |
gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer) | |
You could also pass your own dataset as a list of strings, but it is highly recommended to use the same dataset from the GPTQ paper. |