SkyplerCoder-1.0-SFT

SkyplerCoder-1.0-SFT is a CodeLlama-7B model finetuned using LoRA (PEFT) on a curated dataset of code instructions and completions. This model is designed for code generation, code completion, and code understanding tasks, supporting both Python and TypeScript, and is optimized for instruction-following and developer productivity.

Model Details

• Base Model: meta-llama/CodeLlama-7b-hf
• Finetuning Method: LoRA (PEFT) with SFT (Supervised Fine-Tuning)
• Framework: PyTorch, Hugging Face Transformers, PEFT, TRL
• Quantization: Optional (BitsAndBytes config available)
• Parameters: 7B (with LoRA adapters merged)
• Author: Abinayasankar M(Skypler AI)
• Contact: abinayasankar2004@gmail.com

Training Data

• Sources:
  • mhhmm/typescript-instruct-20k
  • Custom CSVs: trainable_data.csv, trainable_data_concert.csv
• Format: Instruction/Completion pairs for code tasks in Python and TypeScript.

Training Procedure

• Hardware: NVIDIA A100 GPU
• Batch Size: 8
• Epochs: 5
• Learning Rate: 2e-5
• Optimizer: AdamW
• Scheduler: Cosine
• LoRA Config: r=8, alpha=16, dropout=0.1, target_modules=["q_proj", "v_proj"]
• Max Sequence Length: 1024 (input), 2048 (labels)
• Callbacks: EarlyStopping, GradientNormLog, SaveCheckpoints

Intended Use

• Primary: Code generation, code completion, code review, and code understanding.
• Secondary: Instruction-following for code tasks, code explanation, and code translation.

Limitations

• May generate incorrect or insecure code.
• Not suitable for production without human review.
• Trained on public and synthetic data; may not generalize to all codebases.

Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("Abinayasankar/SkyplerCoder-1.0-SFT")
model = AutoModelForCausalLM.from_pretrained("Abinayasankar/SkyplerCoder-1.0-SFT").to("cuda")

prompt = "Write a Python function to reverse a string."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

• Perplexity: Use the provided compute_perplexity function in the repo to evaluate on your own validation set.

Citation

If you use this model, please cite:

@misc{skyplercoder2025,
  title={SkyplerCoder-1.0-SFT: LoRA-Finetuned CodeLlama-7B for Code Generation},
  author={Abinayasankar M},
  year={2025},
  howpublished={\url{https://huggingface.co/Abinayasankar/SkyplerCoder-1.0-SFT}}
}

License

mit

---

Disclaimer:
This model is provided as-is and comes with no warranty. Always review generated code for correctness and security before use