Duplicate from HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

Co-authored-by: Lewis Tunstall <lewtun@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,155 @@
+---
+license: apache-2.0
+base_model: mistral-community/Mixtral-8x22B-v0.1
+tags:
+- trl
+- orpo
+- generated_from_trainer
+datasets:
+- argilla/distilabel-capybara-dpo-7k-binarized
+model-index:
+- name: zephyr-orpo-141b-A35b-v0.1
+  results: []
+inference:
+  parameters:
+    temperature: 0.7
+---
+
+<img src="https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1/resolve/main/logo.png" alt="Zephyr 141B Logo" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+
+
+# Model Card for Zephyr 141B-A35B
+
+Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr 141B-A35B is the latest model in the series, and is a fine-tuned version of [mistral-community/Mixtral-8x22B-v0.1](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1) that was trained using a novel alignment algorithm called [Odds Ratio Preference Optimization (ORPO)](https://huggingface.co/papers/2403.07691) with **7k instances** for **1.3 hours** on 4 nodes of 8 x H100s. ORPO does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. To train Zephyr-141B-A35B, we used the [`argilla/distilabel-capybara-dpo-7k-binarized`](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized) preference dataset, which consists of synthetic, high-quality, multi-turn preferences that have been scored via LLMs.
+
+> [!NOTE]
+> This model was trained collaboratively between Argilla, KAIST, and Hugging Face
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Model type:** A Mixture of Experts (MoE) model with 141B total parameters and 35B active parameters. Fine-tuned on a mix of publicly available, synthetic datasets.
+- **Language(s) (NLP):** Primarily English.
+- **License:** Apache 2.0
+- **Finetuned from model:** [mistral-community/Mixtral-8x22B-v0.1](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1)
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** https://github.com/huggingface/alignment-handbook
+- **Dataset:** https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized
+
+## Performance
+
+Zephyr 141B-A35B was trained to test the effectiveness of ORPO at scale and the underlying dataset contains a mix of general chat capabilities. It achieves strong performance on chat benchmarks like [MT Bench](https://huggingface.co/spaces/lmsys/mt-bench) and [IFEval](https://arxiv.org/abs/2311.07911). The scores reported below were obtained using the [LightEval](https://github.com/huggingface/lighteval) evaluation suite and each prompt has been formatted with the model's corresponding chat template to simulate real-world usage. This is why some scores may differ from those reported in technical reports or on the Open LLM Leaderboard. 
+
+| Model                                                                                               | MT Bench | IFEval |   BBH | AGIEval |
+|-----------------------------------------------------------------------------------------------------|---------:|-------:|------:|--------:|
+| [zephyr-orpo-141b-A35b-v0.1](https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1)       |     8.17 |  65.06 | 58.96 |   44.16 |
+| [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct)                         |     8.26 |  52.13 | 48.50 |   41.16 |
+| [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) |     8.30 |  55.08 | 45.31 |   47.68 |
+
+
+## Intended uses & limitations
+
+The model was fine-tuned on a blend of chat, code, math, and reasoning data. Here's how you can run the model using the `pipeline()` function from 🤗 Transformers:
+
+```python
+# pip install 'transformers>=4.39.3'
+# pip install accelerate
+
+import torch
+from transformers import pipeline
+
+pipe = pipeline(
+    "text-generation",
+    model="HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1",
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+messages = [
+    {
+        "role": "system",
+        "content": "You are Zephyr, a helpful assistant.",
+    },
+    {"role": "user", "content": "Explain how Mixture of Experts work in language a child would understand."},
+]
+outputs = pipe(
+    messages,
+    max_new_tokens=512,
+    do_sample=True,
+    temperature=0.7,
+    top_k=50,
+    top_p=0.95,
+)
+print(outputs[0]["generated_text"][-1]["content"])
+```
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+Zephyr 141B-A35B has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). 
+It is also unknown what the size and composition of the corpus was used to train the base model (`mistral-community/Mixtral-8x22B-v0.1`), however it is likely to have included a mix of Web data and technical sources like books and code. See the [Falcon 180B model card](https://huggingface.co/tiiuae/falcon-180B#training-data) for an example of this.
+
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 32
+- total_train_batch_size: 32
+- total_eval_batch_size: 256
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: inverse_sqrt
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 3
+
+### Training results
+
+
+
+### Framework versions
+
+- Transformers 4.39.3
+- Pytorch 2.1.2+cu121
+- Datasets 2.18.0
+- Tokenizers 0.15.1
+
+## Citation
+
+If you find Zephyr 141B-A35B is useful in your work, please cite the ORPO paper:
+
+```
+@misc{hong2024orpo,
+      title={ORPO: Monolithic Preference Optimization without Reference Model}, 
+      author={Jiwoo Hong and Noah Lee and James Thorne},
+      year={2024},
+      eprint={2403.07691},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+You may also wish to cite the creators of this model:
+
+```
+@misc{zephyr_141b,
+  author = {Alvaro Bartolome and Jiwoo Hong and Noah Lee and Kashif Rasul and Lewis Tunstall},
+  title = {Zephyr 141B A35B},
+  year = {2024},
+  publisher = {Hugging Face},
+  journal = {Hugging Face repository},
+  howpublished = {\url{https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1}}
+}
+```

all_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 3.0,
+    "train_loss": 0.812556631554107,
+    "train_runtime": 4771.9621,
+    "train_samples": 6932,
+    "train_samples_per_second": 4.358,
+    "train_steps_per_second": 0.136
+}

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "mistral-community/Mixtral-8x22B-v0.1",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 6144,
+  "initializer_range": 0.02,
+  "intermediate_size": 16384,
+  "max_position_embeddings": 65536,
+  "model_type": "mixtral",
+  "num_attention_heads": 48,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 56,
+  "num_key_value_heads": 8,
+  "num_local_experts": 8,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000,
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.39.3",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.39.3"
+}

model-00001-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f623d7c12dc3c39e3f011ac10e2894bb97885723967b646db09aa69a467b84e2
+size 4998663696

model-00002-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79894cf86c8a1c824eae0fb8dce3dda2f88192d16a04ad617250f5b557e6418d
+size 4806799120

model-00003-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4737356e72afbd1d899e51183a7a921c926ea3e2fa1aa03f6c24949ed54c806c
+size 4806799120

model-00004-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07b1544d0594c8eace33d383d5f0c8cc80186b57d2d5924f1f62fbf37edb1850
+size 4806799120

model-00005-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14dedde5e9178654af879895c81d86cdb984d5d2f3ab773c8982026d69c0e8eb
+size 4806799120

model-00006-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f8522fe23daed9d69673009e8c97081e37a98b1c9d8d08fe5d4c94969e40fda
+size 4806799120

model-00007-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:282b30abd7f2acafc3b3e4089a6a832655b9013c3fe4d65a6d3ad7f145a2a762
+size 4806799120

model-00008-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26e9e370572be671609cfc77675d8f89c59afaf9ad69723d9f29a70e6fbd5a02
+size 4806799120

model-00009-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a8c348a223a71b871c2557157eeb11e229c52359ce68a571070df11d9fd40ea
+size 4806799120

model-00010-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a25a5ca1d0e4674c7c3548f4b4c113ed811ccd14c1cc908b6fd88c5f456975f9
+size 4806799120

model-00011-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02b5f57f8a14c8081a5a87b0d51d807070dfa5888387b7320815044b93a60e33
+size 4806799136

model-00012-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b87b4d457274faef5cdb2849c36b8c2a1fd92be55a7b6b9ae498d823aabd2b9
+size 4806799152

model-00013-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:698074e43a07c2d1fc1eded8ad2fe359c157ef8a157e917e7701d9d38094a55a
+size 4806799152

model-00014-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efc7c5861083c569421c04e0606631140f1982c7a88ce661291583e751016a2e
+size 4806799152

model-00015-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:149f45ece2a1ade674f568b9f14b3a93ed271998593f9a9a598d108420e00bde
+size 4806799152

model-00016-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1076aed472ed5117df7922e0597993c620f8b2add0679463c9a1570eb444ffe
+size 4806799152

model-00017-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db4e673a4df8c01b7728cc7200c0c514863702481d6191a6c06916f13a2221e1
+size 4806799152

model-00018-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6797d2776235c46859055ec897edbb42ec6570d73955393bbc338f3c019afb0e
+size 4806799152

model-00019-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:175150f016318ac04287e9f94d087b9ed82baf237ef577717bb7896656cf7085
+size 4806799152

model-00020-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbc3c4fa6b4616f70e9d19d89ecc381c88ea7e53aab09b55815037452c65e022
+size 4806799152

model-00021-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2424009021647ebb60abce2b3ca56a66cc40bc52b1c5cb15ca9df94d212738ae
+size 4806799152

model-00022-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:270cba731b5358645cf62b8675245a4e1bc66a520d292e75bc4f81d465b34188
+size 4806799152

model-00023-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f450ec6f7509868e884ca80ef96a5d5f970297c57720a8d467c3ffd3292817e2
+size 4806799152

model-00024-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2dbad77697665ec3a81523d8164c2a5eae4e362978be300e6efa1ee0af7a103
+size 4932529864

model-00025-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9a7c3feac72afea74735cc8d372b1f4d0e107bdab3cdea39861e1ad489b00db
+size 4995542848

model-00026-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f57ec4d67f24ade47ef0a8b09adb0f4dd8f6fc1685d2890141e02d11ca5d739
+size 4995542848

model-00027-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ebe99b1f40c4b1740579d274ae2da8a732424380c851a36ede3a0142b080056
+size 4932628288

model-00028-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d34357259ab1d1fd4b2e3857c3f7dbfc68ad8a1ec559fea465953616bb9857b
+size 4806774344

model-00029-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2cc629634bf49988aaadac27c1daff0d8fa3350aea7aa98cb8d810f7e748d549
+size 4806799144

model-00030-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf21f65b23fcdc9cb726fe257a545181ec652fb416815670ce6113f18c15adce
+size 4806799144

model-00031-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:526d7e75300daf3a153b6611e3b2a53820f3e59137f9443d3c6c50804009978c
+size 4806799144

model-00032-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fd1858ccb1779be8e4b4b79466d015a453052703a766a772f54b0f086fb8a2e
+size 4806799144

model-00033-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c18ca349d07cc427ce05b15b05073f3b8020715260780c80697766290dbe3c95
+size 4806799152

model-00034-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5801d046f2895b838c5e29ed0a1d92c9fd5acffc43be55086aff34aa4d353eba
+size 4806799152

model-00035-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aef8078724b01727ea5f3fc6feae0251e058b813a1d5b3e2813d4e76f39d46c9
+size 4806799152

model-00036-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65b826deb2e5a99114667e8317517b0ea244e90a277631fc257512cf7ea65fa6
+size 4806799152

model-00037-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b63df2d0a8e042275ccd0725e7c2b799184e52e550fcc75dfddd29bf3b573a8b
+size 4806799152

model-00038-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3327092547270cb9f680b3e9061243833d8cda2fa7866038e5c3d9c8e387237
+size 4806799152

model-00039-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:180fbadb648dc8150c4171199cb1f0bfb490b78b2f997870e357cab00d8eb3bb
+size 4806799152

model-00040-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c9cef16e5582f606decaf7e7d6c76f29c393b3db016f2959d755688d656f56d
+size 4806799152

model-00041-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff7e2363129f1fc3797d65bdc2db8d969c2901aa0bf50c0697de1143c3c86d52
+size 4806799152

model-00042-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3ad6e4d5dce397995a5db3453cb4266a36e5ac60c8a4b04f7c093670807ceeb
+size 4806799152

model-00043-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9485220622e654a2b950d758102cef11fb04207f691199215198d2c0cc58d87
+size 4806799152

model-00044-of-00059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a03e4cea8cc668179e8ca768eeabb83c0f7d7f432208bed002bfa068387e66bb
+size 4806799152