|
|
--- |
|
|
library_name: peft |
|
|
base_model: Qwen/Qwen2.5-Coder-3B-Instruct-AWQ |
|
|
tags: |
|
|
- knowledge-distillation |
|
|
- code-generation |
|
|
- qwen |
|
|
- lora |
|
|
- distilled |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Qwen2.5-Coder-3B Distilled Model |
|
|
|
|
|
This is a **knowledge-distilled** version of Qwen2.5-Coder-3B-Instruct-AWQ, trained using knowledge distillation from Qwen2.5-Coder-7B-Instruct-AWQ. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: Qwen/Qwen2.5-Coder-3B-Instruct-AWQ |
|
|
- **Teacher Model**: Qwen/Qwen2.5-Coder-7B-Instruct-AWQ |
|
|
- **Training Method**: Knowledge Distillation with LoRA |
|
|
- **Best Validation Loss**: 1.9286 |
|
|
- **Training Time**: ~5 minutes |
|
|
- **Parameters Trained**: 14.9M (4.59% of base model) |
|
|
|
|
|
## Training Configuration |
|
|
|
|
|
- **Temperature**: 2.0 (optimal) |
|
|
- **Alpha**: 0.95 (95% distillation weight) |
|
|
- **LoRA Rank**: 8 |
|
|
- **Target Modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel |
|
|
|
|
|
# Load base model and tokenizer |
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
|
"Qwen/Qwen2.5-Coder-3B-Instruct-AWQ", |
|
|
torch_dtype=torch.float16, |
|
|
device_map="auto" |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-3B-Instruct-AWQ") |
|
|
|
|
|
# Load distilled adapter |
|
|
model = PeftModel.from_pretrained(base_model, "Vinitha2004/qwen-coder-1.5B-Instruct-AWQ-t2") |
|
|
|
|
|
# Generate code |
|
|
input_text = "Original Code:\ndef add(a, b):\n return a + b\n\nUpdate Snippet:\n// ... existing code ...\ndef add(a: int, b: int) -> int:\n// ... existing code ...\n\nUpdated Code:\n" |
|
|
inputs = tokenizer(input_text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
result = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(result) |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
This distilled model retains the knowledge from the 7B teacher model while being significantly more efficient: |
|
|
- **Faster inference** (3B vs 7B parameters) |
|
|
- **Lower memory usage** |
|
|
- **Maintained code generation quality** |
|
|
|
|
|
## Training Dataset |
|
|
|
|
|
Trained on 5000 code editing examples from custom dataset. |
|
|
|
|
|
## Files |
|
|
|
|
|
- `adapter_config.json`: LoRA configuration |
|
|
- `adapter_model.safetensors`: Trained LoRA weights (59MB) |
|
|
- Other standard tokenizer files |
|
|
|
