Model Card for Llama-3.2-3B ORPO Fine-Tuned Model with LoRA

This model is a fine-tuned version of the base model unsloth/Llama-3.2-3B-Instruct-bnb-4bit using Odds Ratio Preference Optimization (ORPO) with LoRA-based adaptation. The training leverages a dataset of pairwise (chosen vs. rejected) responses to align the model with human preferences without the need for a separate reward or reference model.

Model Details

Model Description

This is a fine-tuned language model that has been optimized using ORPO—a direct preference optimization method that eliminates the need for a reference model. The base model, unsloth/Llama-3.2-3B-Instruct-bnb-4bit, is adapted using Low-Rank Adaptation (LoRA) with a rank and alpha of 64, allowing for efficient fine-tuning with only a small fraction of the model's parameters updated. The fine-tuning is performed on a dataset consisting of approximately 1,600 examples (sampled from "mlabonne/orpo-dpo-mix-40k"), where the model learns to favor the "chosen" response over the "rejected" one directly through odds ratio optimization.

Developed by: [Your Name or Organization]
Model Type: Causal Language Model (Instruction-Finetuned)
Base Model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
Training Method: ORPO (Odds Ratio Preference Optimization) with LoRA
Quantization: 4-bit
Language: English (primarily)
License: [Specify License, e.g., Apache-2.0]

Model Sources

Repository: [Link to the repository on Hugging Face]
Paper: [Reference any paper if available, or "N/A"]
Demo: [Link to a demo if available]

Uses

Direct Use

This model is intended for tasks that benefit from preference-aligned generation, such as:

Instruction following
Chatbot response generation
Content creation where human-aligned quality is crucial

Downstream Use

This model can be further fine-tuned or adapted for domain-specific applications where human preferences play a significant role in output quality.

Out-of-Scope Use

Applications requiring rigorous factual correctness (e.g., medical or legal advice) without further domain-specific fine-tuning.
Use cases involving sensitive content where model biases could lead to harmful outcomes.

Bias, Risks, and Limitations

Bias: The model may still exhibit biases inherited from the base model and the fine-tuning data.
Risks: Users should be cautious in applications where incorrect or biased information could have serious consequences.
Limitations: As a fine-tuned model using preference optimization, its performance is tied to the quality and diversity of the training data. It may not generalize well to contexts significantly different from its training set.

Recommendations

Users should:

Evaluate the model on their specific use case.
Monitor outputs for potential bias or factual inaccuracies.
Fine-tune further if necessary to better align with specific requirements.

How to Get Started with the Model

Below is an example code snippet to load and use the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("your-username/llama-3.2-3b-orpo-lora64")
tokenizer = AutoTokenizer.from_pretrained("your-username/llama-3.2-3b-orpo-lora64")

input_text = "Please explain the benefits of using ORPO for fine-tuning language models."
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))