--- library_name: peft license: llama3.2 base_model: meta-llama/Llama-3.2-3B-Instruct tags: - axolotl - generated_from_trainer datasets: - minpeter/xlam-function-calling-60k-hermes - minpeter/xlam-irrelevance-7.5k-qwen2.5-72b-distill-hermes - minpeter/hermes-function-calling-v1-jsonl - minpeter/hermes-function-calling-v1-jsonl model-index: - name: m-3b-v1-iteration-00-sf-xlam-10 results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.7.0` ```yaml base_model: meta-llama/Llama-3.2-3B-Instruct hub_model_id: morsmordre/m-3b-v1-iteration-00-sf-xlam-10 load_in_8bit: false load_in_4bit: false strict: false datasets: # 0.5k - path: minpeter/xlam-function-calling-60k-hermes data_files: - result.parquet type: chat_template chat_template: llama3 field_messages: conversations message_field_role: from message_field_content: value shards: 120 # 0.35k - path: minpeter/xlam-irrelevance-7.5k-qwen2.5-72b-distill-hermes data_files: - result.parquet type: chat_template chat_template: llama3 field_messages: conversations message_field_role: from message_field_content: value shards: 15 # 1.2k - path: minpeter/hermes-function-calling-v1-jsonl data_files: - func-calling-singleturn.jsonl - func-calling.jsonl type: chat_template chat_template: llama3 field_messages: conversations message_field_role: from message_field_content: value shards: 3 # 1k - path: minpeter/hermes-function-calling-v1-jsonl data_files: - glaive-function-calling-5k.jsonl type: chat_template chat_template: llama3 field_messages: conversations message_field_role: from message_field_content: value shards: 5 chat_template: llama3 dataset_prepared_path: last_run_prepared output_dir: ./output adapter: lora lora_model_dir: sequence_len: 4096 pad_to_sequence_len: true sample_packing: true val_set_size: 0.05 eval_sample_packing: true evals_per_epoch: 3 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_fan_in_fan_out: lora_target_modules: - gate_proj - down_proj - up_proj - q_proj - v_proj - k_proj - o_proj wandb_project: "axolotl" wandb_entity: "kasfiekfs-e" wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 2 micro_batch_size: 2 num_epochs: 2 optimizer: adamw_8bit lr_scheduler: cosine learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true loss_watchdog_threshold: 5.0 loss_watchdog_patience: 3 warmup_steps: 10 saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: pad_token: "<|finetune_right_pad_id|>" ```

# m-3b-v1-iteration-00-sf-xlam-10 This model is a fine-tuned version of [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) on the minpeter/xlam-function-calling-60k-hermes, the minpeter/xlam-irrelevance-7.5k-qwen2.5-72b-distill-hermes, the minpeter/hermes-function-calling-v1-jsonl and the minpeter/hermes-function-calling-v1-jsonl datasets. It achieves the following results on the evaluation set: - Loss: 0.3335 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 4 - optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 2.0 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.5354 | 0.0039 | 1 | 0.7727 | | 0.4667 | 0.3327 | 85 | 0.3745 | | 0.1858 | 0.6654 | 170 | 0.3515 | | 0.5982 | 0.9980 | 255 | 0.3440 | | 0.1452 | 1.3288 | 340 | 0.3389 | | 0.2287 | 1.6614 | 425 | 0.3344 | | 0.1441 | 1.9941 | 510 | 0.3335 | ### Framework versions - PEFT 0.14.0 - Transformers 4.48.3 - Pytorch 2.5.1+cu124 - Datasets 3.2.0 - Tokenizers 0.21.0