OLoRA residual initialized with:
from peft import LoraConfig, get_peft_model
config = LoraConfig(
target_modules='all-linear',
r=64,
lora_alpha=64,
init_lora_weights='olora',
modules_to_save=["lm_head",'embed_tokens']
)
model = get_peft_model(model, config)
- Downloads last month
- 13