File size: 433 Bytes
5fa1a76 |
1 2 3 4 5 6 7 8 |
You can change the data type of these modules with the torch_dtype parameter if you want:
import torch
from transformers import AutoModelForCausalLM
model_4bit = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", load_in_4bit=True, torch_dtype=torch.float32)
model_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype
If you have bitsandbytes>=0.41.3, you can serialize 4-bit models and push them on Hugging Face Hub. |