Hugging Face's logo

rhplus0831
/
maid-yuzu-v8

Text Generation
Transformers
Safetensors
mixtral
mergekit
Merge
text-generation-inference
Model card Files Community
rhplus0831 commited on
Commit
fe8012b
·
verified ·
1 Parent(s): d9dafe9

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +57 -0
  2. config.json +30 -0
  3. mergekit_config.yml +18 -0
  4. model-00001-of-00048.safetensors +3 -0
  5. model-00002-of-00048.safetensors +3 -0
  6. model-00003-of-00048.safetensors +3 -0
  7. model-00004-of-00048.safetensors +3 -0
  8. model-00005-of-00048.safetensors +3 -0
  9. model-00006-of-00048.safetensors +3 -0
  10. model-00007-of-00048.safetensors +3 -0
  11. model-00008-of-00048.safetensors +3 -0
  12. model-00009-of-00048.safetensors +3 -0
  13. model-00010-of-00048.safetensors +3 -0
  14. model-00011-of-00048.safetensors +3 -0
  15. model-00012-of-00048.safetensors +3 -0
  16. model-00013-of-00048.safetensors +3 -0
  17. model-00014-of-00048.safetensors +3 -0
  18. model-00015-of-00048.safetensors +3 -0
  19. model-00016-of-00048.safetensors +3 -0
  20. model-00017-of-00048.safetensors +3 -0
  21. model-00018-of-00048.safetensors +3 -0
  22. model-00019-of-00048.safetensors +3 -0
  23. model-00020-of-00048.safetensors +3 -0
  24. model-00021-of-00048.safetensors +3 -0
  25. model-00022-of-00048.safetensors +3 -0
  26. model-00023-of-00048.safetensors +3 -0
  27. model-00024-of-00048.safetensors +3 -0
  28. model-00025-of-00048.safetensors +3 -0
  29. model-00026-of-00048.safetensors +3 -0
  30. model-00027-of-00048.safetensors +3 -0
  31. model-00028-of-00048.safetensors +3 -0
  32. model-00029-of-00048.safetensors +3 -0
  33. model-00030-of-00048.safetensors +3 -0
  34. model-00031-of-00048.safetensors +3 -0
  35. model-00032-of-00048.safetensors +3 -0
  36. model-00033-of-00048.safetensors +3 -0
  37. model-00034-of-00048.safetensors +3 -0
  38. model-00035-of-00048.safetensors +3 -0
  39. model-00036-of-00048.safetensors +3 -0
  40. model-00037-of-00048.safetensors +3 -0
  41. model-00038-of-00048.safetensors +3 -0
  42. model-00039-of-00048.safetensors +3 -0
  43. model-00040-of-00048.safetensors +3 -0
  44. model-00041-of-00048.safetensors +3 -0
  45. model-00042-of-00048.safetensors +3 -0
  46. model-00043-of-00048.safetensors +3 -0
  47. model-00044-of-00048.safetensors +3 -0
  48. model-00045-of-00048.safetensors +3 -0
  49. model-00046-of-00048.safetensors +3 -0
  50. model-00047-of-00048.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - smelborp/MixtralOrochi8x7B
4
+ library_name: transformers
5
+ tags:
6
+ - mergekit
7
+ - merge
8
+
9
+ ---
10
+ # maid-yuzu-v8
11
+
12
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
13
+
14
+ v7's approach worked better than I thought, so I tried something even weirder as a test. I don't think a proper model will come out, but I'm curious about the results.
15
+
16
+ ## Merge Details
17
+ ### Merge Method
18
+
19
+ This models were merged using the SLERP method in the following order:
20
+
21
+ maid-yuzu-v8-base: mistralai/Mixtral-8x7B-v0.1 + mistralai/Mixtral-8x7B-Instruct-v0.1 = 0.5
22
+ maid-yuzu-v8-step1: above + jondurbin/bagel-dpo-8x7b-v0.2 = 0.25
23
+ maid-yuzu-v8-step2: above + cognitivecomputations/dolphin-2.7-mixtral-8x7b = 0.25
24
+ maid-yuzu-v8-step3: above + NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss = 0.25
25
+ maid-yuzu-v8-step4: above + ycros/BagelMIsteryTour-v2-8x7B = 0.25
26
+ maid-yuzu-v8: above + smelborp/MixtralOrochi8x7B = 0.25
27
+
28
+ ### Models Merged
29
+
30
+ The following models were included in the merge:
31
+ * [smelborp/MixtralOrochi8x7B](https://huggingface.co/smelborp/MixtralOrochi8x7B)
32
+ * ../maid-yuzu-v8-step4
33
+
34
+ ### Configuration
35
+
36
+ The following YAML configuration was used to produce this model:
37
+
38
+ ```yaml
39
+ base_model:
40
+ model:
41
+ path: ../maid-yuzu-v8-step4
42
+ dtype: bfloat16
43
+ merge_method: slerp
44
+ parameters:
45
+ t:
46
+ - value: 0.25
47
+ slices:
48
+ - sources:
49
+ - layer_range: [0, 32]
50
+ model:
51
+ model:
52
+ path: ../maid-yuzu-v8-step4
53
+ - layer_range: [0, 32]
54
+ model:
55
+ model:
56
+ path: smelborp/MixtralOrochi8x7B
57
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "../maid-yuzu-v8-step4",
3
+ "architectures": [
4
+ "MixtralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mixtral",
15
+ "num_attention_heads": 32,
16
+ "num_experts_per_tok": 2,
17
+ "num_hidden_layers": 32,
18
+ "num_key_value_heads": 8,
19
+ "num_local_experts": 8,
20
+ "output_router_logits": false,
21
+ "rms_norm_eps": 1e-05,
22
+ "rope_theta": 1000000.0,
23
+ "router_aux_loss_coef": 0.02,
24
+ "sliding_window": null,
25
+ "tie_word_embeddings": false,
26
+ "torch_dtype": "bfloat16",
27
+ "transformers_version": "4.37.2",
28
+ "use_cache": true,
29
+ "vocab_size": 32000
30
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ base_model:
2
+ model:
3
+ path: ../maid-yuzu-v8-step4
4
+ dtype: bfloat16
5
+ merge_method: slerp
6
+ parameters:
7
+ t:
8
+ - value: 0.25
9
+ slices:
10
+ - sources:
11
+ - layer_range: [0, 32]
12
+ model:
13
+ model:
14
+ path: ../maid-yuzu-v8-step4
15
+ - layer_range: [0, 32]
16
+ model:
17
+ model:
18
+ path: smelborp/MixtralOrochi8x7B
model-00001-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed4ae0a36bbe8adf60d96bc0addef738ed0191821009118e3fd52c8347a87442
3
+ size 1963199472
model-00002-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71fa8e226c606baf1b686623d80ffc90e643bdbd55285b438fb60af81a584487
3
+ size 1996490952
model-00003-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f970f1ba79117c87db44684e71c268de39800dd474d9967d36966c57655c5a4
3
+ size 1956793496
model-00004-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a44660eefdc59de6cf95f26052f34124b70ef5398e3ef5773edc2ca98c813fd0
3
+ size 1996490952
model-00005-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03e9cfede6248761bc97e38290db30cd1962327c936df0194338fab4763745a2
3
+ size 1996590288
model-00006-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c8c41a5f388fba8b3bbf28751b60ac00fb31c54b6227342eea131b72d3f67f0
3
+ size 1929447960
model-00007-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2eeeb6d1f08a019e76e5799c8afcc4d368b482bcf577a328793ef667c9ea12a
3
+ size 1962953472
model-00008-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3417f6e77950697a0a0129e4a9cb7893fb18b9ee66c5e4e68dcf3d3fb02ec7d
3
+ size 1996573208
model-00009-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b79dc7ef1935cd65f558948f5da1600973a66dff417db3acbb595f0934b5b9cf
3
+ size 1962936856
model-00010-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f7c835c95e00f99797bd0e878e48c8377b2544926a6834cbe071d5a7aa3083b
3
+ size 1996507568
model-00011-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4f3a5d8bdf09ab9abaa3341f395d5205d5ea4ed3f22a84b24b6fb6aaa9d03de
3
+ size 1963068152
model-00012-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:21b8bbeb43885caabf1d5835efac95348af6a85e6768de9006a8af0b8d109af5
3
+ size 1996490952
model-00013-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:505053d49e8b81aeead2015161932056303c65c7cdf155853c74770e29895822
3
+ size 1929547296
model-00014-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3743cda523a644105537ae3820fdc143cad70f3a624bd4538a4c73e639845ab6
3
+ size 1996490952
model-00015-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e70b5364e3e9de44b5af266fb644989bc0d2bc249f298909babbdb2e3d256791
3
+ size 1962953472
model-00016-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5546ffd3d0d544eccddfd71f1035d0ea18df8163a607f878696b03ae8017b06d
3
+ size 1996573240
model-00017-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ebd41d34693dc46224e1fcb3bd789420cc4ee32f9f3b9ec174021427b9415c83
3
+ size 1962936872
model-00018-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06269dced916505007054b539136b689601644ac65c29335f6737bc46cb866eb
3
+ size 1996573240
model-00019-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6492ebf8d0cb330b68dd41811fed91dd4cef09e07e9650baa4c1b5b120912363
3
+ size 1963002528
model-00020-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:931f3840992f181b347e9c13fdfa604fe9f9ca1c7cfea7d8201a42be5b8517ce
3
+ size 1962953488
model-00021-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:601d6b42a57d0783ebd75e8c86d48a4ec0302decce7972b89c5e29e9505da214
3
+ size 1963084800
model-00022-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1eadcddf8e0a464069ddd7d089db24d80179ae2d1a1f3748dcde935906ca5ba1
3
+ size 1962953488
model-00023-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f6629f337ac9229f557e16e30799bbdcbc67155766d355008af08a9f0c97d41e
3
+ size 1996573240
model-00024-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c273c12d20a10ebb02758ce77fa3336c2c003d0be1a013b57ca352cdfb599eaa
3
+ size 1962936872
model-00025-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c05ae2f3b314694034789855994b7e8e44a1c8f4e6c0d4582752efe6949489c2
3
+ size 1996490968
model-00026-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:74e94d02a37c66cfc23921beb4a9b842e0441ee7870d97698a8e0baff76cddd8
3
+ size 1996573232
model-00027-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:105fcc4cbaf0ab869ff5949645776950c7eeffcffdb93908b87c4fe9a2bc3ec0
3
+ size 1963002528
model-00028-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a52b8bafce052c29beaeca472d26c5e841c84e61e7c5fa55425ad3f863bfa772
3
+ size 1963035760
model-00029-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4d3c9514281f728e630b6b25fc936ed1386e524f9386d6d20a7dd0976d2df7d7
3
+ size 1963002528
model-00030-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68e8b910128c27135d15e155628ee1c74d626b84ae7ee57353b3fdb905fb0195
3
+ size 1962953488
model-00031-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:596609ac849806da29b899d4b5208d92f03e1131d53c196f6e4098309c880a8e
3
+ size 1963084800
model-00032-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f3e6dddf13a4ea7c6b99c4b6880332628da83fc0f83b9c6db513bc9faba7475e
3
+ size 1962953488
model-00033-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e04d3b954db49fa2b1fef1b8feae09ae7e618761d6fe27e11b94c3ccfde0637b
3
+ size 1996573240
model-00034-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:53b7abc8c1aa0c2bea113d4922839c8ccd98483cca4bcab34d41b59aa8d2087e
3
+ size 1962936872
model-00035-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:063b651569a4c2d8e399202ca2a022bad05f6ba289ae29014a358b5704629a6d
3
+ size 1996490968
model-00036-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:323fb50fac43f85aed99159e2307cd472d1a5d89f78bc65be913b61708620b20
3
+ size 1963084800
model-00037-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35b9f2819c66c04884b5ce5ecd3afdd404fca2b3c3f6ab66d9295a381c754a24
3
+ size 1996490968
model-00038-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fa451b5e0c5a7f09c377944d801c869bec840187f0aa379f4e822060099afb3
3
+ size 1929547328
model-00039-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e6fdfad7efddda97fd228c3f6ce84bb9487dc0c793b2f474979883dc43d4aced
3
+ size 1996490968
model-00040-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b8dd5c75df5fea1d247fe9d3ee048053870aafe4263278033881c112862ee3f
3
+ size 1962953488
model-00041-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99d53fbb16a0fc46e992348008ef978fb80476dcab0fb00a0eee03c86701aadf
3
+ size 1996507576
model-00042-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd11830c5a9db45bff5d13eed0acb614264fa064824eb739fbb6223ab3b183ce
3
+ size 1962936872
model-00043-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dbc00be221f29fa4ab4c62ad811a76772007ab5e6f7488bcb8b208b0673cddda
3
+ size 1996507584
model-00044-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36ab329322b650340cc53fb5a6dbd46c430309bfc5d77460ce5221b4f3de444b
3
+ size 1963002528
model-00045-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb9994a46b93a9c4b9e2dbf2fd84c1f7302fcb23da662b5077ed429ce1e0e000
3
+ size 1996490968
model-00046-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e375fb9f8a30963acf796b2f66343c72e87f145374e2988671afd8bf21dbab8
3
+ size 1990306840
model-00047-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:228b0131e8ddadc6f46a1a9cadb1590b3ba276bafd6bf5bd2476c822cb56ff87
3
+ size 1962953488