You can change the data type of these modules with the torch_dtype parameter if you want: | |
import torch | |
from transformers import AutoModelForCausalLM | |
model_4bit = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", load_in_4bit=True, torch_dtype=torch.float32) | |
model_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype | |
If you have bitsandbytes>=0.41.3, you can serialize 4-bit models and push them on Hugging Face Hub. |