codelion/Qwen3-0.6B-accuracy-recovery-lora

🎯 Accuracy Recovery LoRA Adapter

This LoRA adapter helps recover accuracy when using INT4 quantized versions of Qwen/Qwen3-0.6B. It was trained using self-distillation with Magpie-generated data.

📊 Performance Metrics

• Base Model: Qwen/Qwen3-0.6B
• Quantization: INT4 with NF4
• LoRA Rank: 64
• LoRA Alpha: 128
• Training Samples: 610
• Target Performance Gap: <5% perplexity increase

🔧 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model with quantization
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, "codelion/Qwen3-0.6B-accuracy-recovery-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

# Use the model
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

🧪 Training Details

• Method: Self-distillation using Magpie data generation
• Framework: PEFT + LoRA
• Loss Function: Combined KL divergence + MSE loss
• Temperature: 1.0
• Alpha (distillation weight): 0.01

📈 Expected Benefits

• ✅ Maintains accuracy close to FP16 baseline
• ✅ ~75% reduction in memory usage
• ✅ 2-3x faster inference than FP16
• ✅ Easy to integrate with existing workflows

🏷️ Related

• Dataset: codelion/Qwen3-0.6B-magpie
• Base Model: Qwen/Qwen3-0.6B
• Framework: PEFT

---

This adapter is part of the Ellora project - standardized recipes for enhancing LLM capabilities.