mihaimasala commited on
Commit
c41cd02
·
verified ·
1 Parent(s): b395b08

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +760 -3
README.md CHANGED
@@ -1,3 +1,760 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ language:
4
+ - ro
5
+ base_model:
6
+ - mistralai/Mistral-7B-v0.1
7
+ datasets:
8
+ - OpenLLM-Ro/ro_sft_alpaca
9
+ - OpenLLM-Ro/ro_sft_alpaca_gpt4
10
+ - OpenLLM-Ro/ro_sft_dolly
11
+ - OpenLLM-Ro/ro_sft_selfinstruct_gpt4
12
+ - OpenLLM-Ro/ro_sft_norobots
13
+ - OpenLLM-Ro/ro_sft_orca
14
+ - OpenLLM-Ro/ro_sft_camel
15
+ - OpenLLM-Ro/ro_sft_oasst
16
+ - OpenLLM-Ro/ro_sft_ultrachat
17
+ model-index:
18
+ - name: OpenLLM-Ro/RoMistral-7b-Instruct-2024-10-09
19
+ results:
20
+ - task:
21
+ type: text-generation
22
+ dataset:
23
+ name: RoMT-Bench
24
+ type: RoMT-Bench
25
+ metrics:
26
+ - name: Score
27
+ type: Score
28
+ value: 5.29
29
+ - task:
30
+ type: text-generation
31
+ dataset:
32
+ name: RoCulturaBench
33
+ type: RoCulturaBench
34
+ metrics:
35
+ - name: Score
36
+ type: Score
37
+ value: 3.99
38
+ - task:
39
+ type: text-generation
40
+ dataset:
41
+ name: Romanian_Academic_Benchmarks
42
+ type: Romanian_Academic_Benchmarks
43
+ metrics:
44
+ - name: Average accuracy
45
+ type: accuracy
46
+ value: 52.91
47
+ - task:
48
+ type: text-generation
49
+ dataset:
50
+ name: OpenLLM-Ro/ro_arc_challenge
51
+ type: OpenLLM-Ro/ro_arc_challenge
52
+ metrics:
53
+ - name: Average accuracy
54
+ type: accuracy
55
+ value: 52.27
56
+ - task:
57
+ type: text-generation
58
+ dataset:
59
+ name: OpenLLM-Ro/ro_mmlu
60
+ type: OpenLLM-Ro/ro_mmlu
61
+ metrics:
62
+ - name: Average accuracy
63
+ type: accuracy
64
+ value: 49.33
65
+ - task:
66
+ type: text-generation
67
+ dataset:
68
+ name: OpenLLM-Ro/ro_winogrande
69
+ type: OpenLLM-Ro/ro_winogrande
70
+ metrics:
71
+ - name: Average accuracy
72
+ type: accuracy
73
+ value: 70.03
74
+ - task:
75
+ type: text-generation
76
+ dataset:
77
+ name: OpenLLM-Ro/ro_hellaswag
78
+ type: OpenLLM-Ro/ro_hellaswag
79
+ metrics:
80
+ - name: Average accuracy
81
+ type: accuracy
82
+ value: 62.88
83
+ - task:
84
+ type: text-generation
85
+ dataset:
86
+ name: OpenLLM-Ro/ro_gsm8k
87
+ type: OpenLLM-Ro/ro_gsm8k
88
+ metrics:
89
+ - name: Average accuracy
90
+ type: accuracy
91
+ value: 32.42
92
+ - task:
93
+ type: text-generation
94
+ dataset:
95
+ name: OpenLLM-Ro/ro_truthfulqa
96
+ type: OpenLLM-Ro/ro_truthfulqa
97
+ metrics:
98
+ - name: Average accuracy
99
+ type: accuracy
100
+ value: 50.51
101
+ - task:
102
+ type: text-generation
103
+ dataset:
104
+ name: LaRoSeDa_binary
105
+ type: LaRoSeDa_binary
106
+ metrics:
107
+ - name: Average macro-f1
108
+ type: macro-f1
109
+ value: 95.56
110
+ - task:
111
+ type: text-generation
112
+ dataset:
113
+ name: LaRoSeDa_multiclass
114
+ type: LaRoSeDa_multiclass
115
+ metrics:
116
+ - name: Average macro-f1
117
+ type: macro-f1
118
+ value: 67.83
119
+ - task:
120
+ type: text-generation
121
+ dataset:
122
+ name: LaRoSeDa_binary_finetuned
123
+ type: LaRoSeDa_binary_finetuned
124
+ metrics:
125
+ - name: Average macro-f1
126
+ type: macro-f1
127
+ value: 99.00
128
+ - task:
129
+ type: text-generation
130
+ dataset:
131
+ name: LaRoSeDa_multiclass_finetuned
132
+ type: LaRoSeDa_multiclass_finetuned
133
+ metrics:
134
+ - name: Average macro-f1
135
+ type: macro-f1
136
+ value: 87.57
137
+ - task:
138
+ type: text-generation
139
+ dataset:
140
+ name: WMT_EN-RO
141
+ type: WMT_EN-RO
142
+ metrics:
143
+ - name: Average bleu
144
+ type: bleu
145
+ value: 28.28
146
+ - task:
147
+ type: text-generation
148
+ dataset:
149
+ name: WMT_RO-EN
150
+ type: WMT_RO-EN
151
+ metrics:
152
+ - name: Average bleu
153
+ type: bleu
154
+ value: 6.10
155
+ - task:
156
+ type: text-generation
157
+ dataset:
158
+ name: WMT_EN-RO_finetuned
159
+ type: WMT_EN-RO_finetuned
160
+ metrics:
161
+ - name: Average bleu
162
+ type: bleu
163
+ value: 27.70
164
+ - task:
165
+ type: text-generation
166
+ dataset:
167
+ name: WMT_RO-EN_finetuned
168
+ type: WMT_RO-EN_finetuned
169
+ metrics:
170
+ - name: Average bleu
171
+ type: bleu
172
+ value: 40.36
173
+ - task:
174
+ type: text-generation
175
+ dataset:
176
+ name: XQuAD
177
+ type: XQuAD
178
+ metrics:
179
+ - name: Average exact_match
180
+ type: exact_match
181
+ value: 41.09
182
+ - task:
183
+ type: text-generation
184
+ dataset:
185
+ name: XQuAD
186
+ type: XQuAD
187
+ metrics:
188
+ - name: Average f1
189
+ type: f1
190
+ value: 63.21
191
+ - task:
192
+ type: text-generation
193
+ dataset:
194
+ name: XQuAD_finetuned
195
+ type: XQuAD_finetuned
196
+ metrics:
197
+ - name: Average exact_match
198
+ type: exact_match
199
+ value: 47.56
200
+ - task:
201
+ type: text-generation
202
+ dataset:
203
+ name: XQuAD_finetuned
204
+ type: XQuAD_finetuned
205
+ metrics:
206
+ - name: Average f1
207
+ type: f1
208
+ value: 62.69
209
+ - task:
210
+ type: text-generation
211
+ dataset:
212
+ name: STS
213
+ type: STS
214
+ metrics:
215
+ - name: Average spearman
216
+ type: spearman
217
+ value: 78.47
218
+ - task:
219
+ type: text-generation
220
+ dataset:
221
+ name: STS
222
+ type: STS
223
+ metrics:
224
+ - name: Average pearson
225
+ type: pearson
226
+ value: 77.24
227
+ - task:
228
+ type: text-generation
229
+ dataset:
230
+ name: STS_finetuned
231
+ type: STS_finetuned
232
+ metrics:
233
+ - name: Average spearman
234
+ type: spearman
235
+ value: 87.28
236
+ - task:
237
+ type: text-generation
238
+ dataset:
239
+ name: STS_finetuned
240
+ type: STS_finetuned
241
+ metrics:
242
+ - name: Average pearson
243
+ type: pearson
244
+ value: 87.88
245
+ - task:
246
+ type: text-generation
247
+ dataset:
248
+ name: RoMT-Bench
249
+ type: RoMT-Bench
250
+ metrics:
251
+ - name: First turn
252
+ type: Score
253
+ value: 5.86
254
+ - name: Second turn
255
+ type: Score
256
+ value: 4.72
257
+ - task:
258
+ type: text-generation
259
+ dataset:
260
+ name: OpenLLM-Ro/ro_arc_challenge
261
+ type: OpenLLM-Ro/ro_arc_challenge
262
+ metrics:
263
+ - name: 0-shot
264
+ type: accuracy
265
+ value: 52.10
266
+ - name: 1-shot
267
+ type: accuracy
268
+ value: 49.87
269
+ - name: 3-shot
270
+ type: accuracy
271
+ value: 51.76
272
+ - name: 5-shot
273
+ type: accuracy
274
+ value: 52.10
275
+ - name: 10-shot
276
+ type: accuracy
277
+ value: 53.64
278
+ - name: 25-shot
279
+ type: accuracy
280
+ value: 54.16
281
+ - task:
282
+ type: text-generation
283
+ dataset:
284
+ name: OpenLLM-Ro/ro_mmlu
285
+ type: OpenLLM-Ro/ro_mmlu
286
+ metrics:
287
+ - name: 0-shot
288
+ type: accuracy
289
+ value: 43.86
290
+ - name: 1-shot
291
+ type: accuracy
292
+ value: 47.70
293
+ - name: 3-shot
294
+ type: accuracy
295
+ value: 52.48
296
+ - name: 5-shot
297
+ type: accuracy
298
+ value: 53.29
299
+ - task:
300
+ type: text-generation
301
+ dataset:
302
+ name: OpenLLM-Ro/ro_winogrande
303
+ type: OpenLLM-Ro/ro_winogrande
304
+ metrics:
305
+ - name: 0-shot
306
+ type: accuracy
307
+ value: 68.27
308
+ - name: 1-shot
309
+ type: accuracy
310
+ value: 69.30
311
+ - name: 3-shot
312
+ type: accuracy
313
+ value: 70.56
314
+ - name: 5-shot
315
+ type: accuracy
316
+ value: 71.98
317
+ - task:
318
+ type: text-generation
319
+ dataset:
320
+ name: OpenLLM-Ro/ro_hellaswag
321
+ type: OpenLLM-Ro/ro_hellaswag
322
+ metrics:
323
+ - name: 0-shot
324
+ type: accuracy
325
+ value: 63.03
326
+ - name: 1-shot
327
+ type: accuracy
328
+ value: 62.39
329
+ - name: 3-shot
330
+ type: accuracy
331
+ value: 62.54
332
+ - name: 5-shot
333
+ type: accuracy
334
+ value: 62.95
335
+ - name: 10-shot
336
+ type: accuracy
337
+ value: 63.47
338
+ - task:
339
+ type: text-generation
340
+ dataset:
341
+ name: OpenLLM-Ro/ro_gsm8k
342
+ type: OpenLLM-Ro/ro_gsm8k
343
+ metrics:
344
+ - name: 0-shot
345
+ type: accuracy
346
+ value: 25.47
347
+ - name: 1-shot
348
+ type: accuracy
349
+ value: 33.06
350
+ - name: 3-shot
351
+ type: accuracy
352
+ value: 38.74
353
+ - task:
354
+ type: text-generation
355
+ dataset:
356
+ name: LaRoSeDa_binary
357
+ type: LaRoSeDa_binary
358
+ metrics:
359
+ - name: 0-shot
360
+ type: macro-f1
361
+ value: 88.87
362
+ - name: 1-shot
363
+ type: macro-f1
364
+ value: 97.40
365
+ - name: 3-shot
366
+ type: macro-f1
367
+ value: 98.13
368
+ - name: 5-shot
369
+ type: macro-f1
370
+ value: 97.83
371
+ - task:
372
+ type: text-generation
373
+ dataset:
374
+ name: LaRoSeDa_multiclass
375
+ type: LaRoSeDa_multiclass
376
+ metrics:
377
+ - name: 0-shot
378
+ type: macro-f1
379
+ value: 66.79
380
+ - name: 1-shot
381
+ type: macro-f1
382
+ value: 67.00
383
+ - name: 3-shot
384
+ type: macro-f1
385
+ value: 67.63
386
+ - name: 5-shot
387
+ type: macro-f1
388
+ value: 69.88
389
+ - task:
390
+ type: text-generation
391
+ dataset:
392
+ name: WMT_EN-RO
393
+ type: WMT_EN-RO
394
+ metrics:
395
+ - name: 0-shot
396
+ type: bleu
397
+ value: 23.84
398
+ - name: 1-shot
399
+ type: bleu
400
+ value: 29.49
401
+ - name: 3-shot
402
+ type: bleu
403
+ value: 30.29
404
+ - name: 5-shot
405
+ type: bleu
406
+ value: 29.49
407
+ - task:
408
+ type: text-generation
409
+ dataset:
410
+ name: WMT_RO-EN
411
+ type: WMT_RO-EN
412
+ metrics:
413
+ - name: 0-shot
414
+ type: bleu
415
+ value: 3.14
416
+ - name: 1-shot
417
+ type: bleu
418
+ value: 3.18
419
+ - name: 3-shot
420
+ type: bleu
421
+ value: 6.72
422
+ - name: 5-shot
423
+ type: bleu
424
+ value: 11.35
425
+ - task:
426
+ type: text-generation
427
+ dataset:
428
+ name: XQuAD_EM
429
+ type: XQuAD_EM
430
+ metrics:
431
+ - name: 0-shot
432
+ type: exact_match
433
+ value: 35.21
434
+ - name: 1-shot
435
+ type: exact_match
436
+ value: 40.76
437
+ - name: 3-shot
438
+ type: exact_match
439
+ value: 43.70
440
+ - name: 5-shot
441
+ type: exact_match
442
+ value: 44.71
443
+ - task:
444
+ type: text-generation
445
+ dataset:
446
+ name: XQuAD_F1
447
+ type: XQuAD_F1
448
+ metrics:
449
+ - name: 0-shot
450
+ type: f1
451
+ value: 57.74
452
+ - name: 1-shot
453
+ type: f1
454
+ value: 61.96
455
+ - name: 3-shot
456
+ type: f1
457
+ value: 65.55
458
+ - name: 5-shot
459
+ type: f1
460
+ value: 67.59
461
+ - task:
462
+ type: text-generation
463
+ dataset:
464
+ name: STS
465
+ type: STS
466
+ metrics:
467
+ - name: 0-shot
468
+ type: spearman
469
+ value: 77.38
470
+ - name: 1-shot
471
+ type: spearman
472
+ value: 79.28
473
+ - name: 3-shot
474
+ type: spearman
475
+ value: 78.75
476
+ - task:
477
+ type: text-generation
478
+ dataset:
479
+ name: STS
480
+ type: STS
481
+ metrics:
482
+ - name: 0-shot
483
+ type: pearson
484
+ value: 77.10
485
+ - name: 1-shot
486
+ type: pearson
487
+ value: 77.70
488
+ - name: 3-shot
489
+ type: pearson
490
+ value: 76.91
491
+
492
+ ---
493
+
494
+ # Model Card for Model ID
495
+
496
+
497
+ This model points/is identical to [RoMistral-7b-Instruct-2024-10-09](https://huggingface.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-10-09).
498
+
499
+
500
+ <!-- Provide a quick summary of what the model is/does. -->
501
+
502
+ RoMistral is a family of pretrained and fine-tuned generative text models for Romanian. This is the repository for the **instruct 7B model**. Links to other models can be found at the bottom of this page.
503
+
504
+ ## Model Details
505
+
506
+ ### Model Description
507
+
508
+
509
+
510
+ <!-- Provide a longer summary of what this model is. -->
511
+ OpenLLM-Ro represents the first open-source effort to build a LLM specialized for Romanian. OpenLLM-Ro developed and publicly releases a collection of Romanian LLMs, both in the form of foundational model and instruct and chat variants.
512
+
513
+
514
+ - **Developed by:** OpenLLM-Ro
515
+ <!-- - **Funded by [optional]:** [More Information Needed] -->
516
+ <!-- - **Shared by [optional]:** [More Information Needed] -->
517
+ <!-- - **Model type:** [More Information Needed] -->
518
+ - **Language(s):** Romanian
519
+ - **License:** cc-by-nc-4.0
520
+ - **Finetuned from model:** [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
521
+ - **Trained using:** [RoAlpaca](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_alpaca), [RoAlpacaGPT4](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_alpaca_gpt4), [RoDolly](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_dolly), [RoSelfInstruct](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_selfinstruct_gpt4), [RoNoRobots](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_norobots), [RoOrca](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_orca), [RoCamel](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_camel), [RoOpenAssistant](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_oasst), [RoUltraChat](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_ultrachat)
522
+
523
+
524
+ <!-- - **Finetuned from model [optional]:** [More Information Needed] -->
525
+
526
+ ### Model Sources
527
+
528
+ <!-- Provide the basic links for the model. -->
529
+
530
+ - **Repository:** https://github.com/OpenLLM-Ro/LLaMA-Factory
531
+ - **Paper:** https://arxiv.org/abs/2406.18266
532
+
533
+ ## Intended Use
534
+
535
+ ### Intended Use Cases
536
+
537
+ RoMistral is intented for research use in Romanian. Base models can be adapted for a variety of natural language tasks while instruction and chat tuned models are intended for assistant-like chat.
538
+
539
+ ### Out-of-Scope Use
540
+
541
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
542
+
543
+ Use in any manner that violates the license, any applicable laws or regluations, use in languages other than Romanian.
544
+
545
+
546
+
547
+ ## How to Get Started with the Model
548
+
549
+ Use the code below to get started with the model.
550
+
551
+ ```python
552
+ from transformers import AutoTokenizer, AutoModelForCausalLM
553
+
554
+ tokenizer = AutoTokenizer.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct")
555
+ model = AutoModelForCausalLM.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct")
556
+
557
+ instruction = "Ce jocuri de societate pot juca cu prietenii mei?"
558
+ chat = [
559
+ {"role": "user", "content": instruction},
560
+ ]
561
+ prompt = tokenizer.apply_chat_template(chat, tokenize=False, system_message="")
562
+
563
+ inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
564
+ outputs = model.generate(input_ids=inputs, max_new_tokens=128)
565
+ print(tokenizer.decode(outputs[0]))
566
+ ```
567
+
568
+ ## Academic Benchmarks
569
+
570
+
571
+ <table>
572
+ <tbody>
573
+ <tr>
574
+ <td><strong>Model</strong></td>
575
+ <td><strong><center>Average</center></strong></td>
576
+ <td><strong><center>ARC</center></strong></td>
577
+ <td><strong><center>MMLU</center></strong></td>
578
+ <td><strong><center>Winogrande</center></strong></td>
579
+ <td><strong><center>Hellaswag</center></strong></td>
580
+ <td><strong><center>GSM8k</center></strong></td>
581
+ <td><strong><center>TruthfulQA</center></strong></td>
582
+ </tr>
583
+ <tr>
584
+ <td>Mistral-7B-Instruct-v0.2</td><td><center>47.40</center></td><td><center>46.29</center></td><td><center>47.00</center></td><td><center>58.78</center></td><td><center>54.27</center></td><td><center>13.47</center></td><td><center><strong>64.59</strong></center></td>
585
+ </tr>
586
+ <tr>
587
+ <td>RoMistral-7b-Instruct-2024-05-17</td><td><center>52.54</center></td><td><center>50.41</center></td><td><center><strong>51.61</strong></center></td><td><center>66.48</center></td><td><center>60.27</center></td><td><center><strong>34.19</strong></center></td><td><center>52.30</center></td>
588
+ </tr>
589
+ <tr>
590
+ <td><em>RoMistral-7b-Instruct-2024-10-09</em></td><td><center><em><strong>52.91</strong></em></center></td><td><center><em><strong>52.27</strong></em></center></td><td><center><em>49.33</em></center></td><td><center><em><strong>70.03</strong></em></center></td><td><center><em><strong>62.88</strong></em></center></td><td><center><em>32.42</em></center></td><td><center><em>50.51</em></center></td>
591
+ </tr>
592
+ <tr>
593
+ <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>51.95</center></td><td><center>50.73</center></td><td><center>47.88</center></td><td><center>68.41</center></td><td><center>62.27</center></td><td><center>32.27</center></td><td><center>50.12</center></td>
594
+ </tr>
595
+ </tbody>
596
+ </table>
597
+
598
+
599
+ ## Downstream tasks
600
+
601
+ <table>
602
+ <tbody>
603
+ <tr>
604
+ <td></td>
605
+ <td colspan="4"><center><strong>LaRoSeDa</strong></center></td>
606
+ <td colspan="4"><center><strong>WMT</strong></center></td>
607
+ </tr>
608
+ <tr>
609
+ <td></td>
610
+ <td colspan="2"><center><strong>Few-shot</strong></center></td>
611
+ <td colspan="2"><center><strong>Finetuned</strong></center></td>
612
+ <td colspan="2"><center><strong>Few-shot</strong></center></td>
613
+ <td colspan="2"><center><strong>Finetuned</strong></center></td>
614
+ </tr>
615
+ <tr>
616
+ <td><strong>Model</strong></td>
617
+ <td><center><strong>Binary<br>(Macro F1)</strong></center></td>
618
+ <td><center><strong>Multiclass<br>(Macro F1)</strong></center></td>
619
+ <td><center><strong>Binary<br>(Macro F1)</strong></center></td>
620
+ <td><center><strong>Multiclass<br>(Macro F1)</strong></center></td>
621
+ <td><center><strong>EN-RO<br>(Bleu)</strong></center></td>
622
+ <td><center><strong>RO-EN<br>(Bleu)</strong></center></td>
623
+ <td><center><strong>EN-RO<br>(Bleu)</strong></center></td>
624
+ <td><center><strong>RO-EN<br>(Bleu)</strong></center>
625
+ </tr>
626
+ <tr>
627
+ <td>Mistral-7B-Instruct-v0.2</td><td><center>96.97</center></td><td><center>56.66</center></td><td><center>98.83</center></td><td><center>87.32</center></td><td><center>18.60</center></td><td><center><strong>33.99</strong></center></td><td><center>26.19</center></td><td><center>39.88</center></td>
628
+ </tr>
629
+ <tr>
630
+ <td>RoMistral-7b-Instruct-2024-05-17</td><td><center><strong>97.36</strong></center></td><td><center>67.55</center></td><td><center>98.80</center></td><td><center><strong>88.28</strong></center></td><td><center>27.93</center></td><td><center>13.21</center></td><td><center><strong>28.72</strong></center></td><td><center><strong>40.86</strong></center></td>
631
+ </tr>
632
+ <tr>
633
+ <td><em>RoMistral-7b-Instruct-2024-10-09</em></td><td><center><em>95.56</em></center></td><td><center><em><strong>67.83</strong></em></center></td><td><center><em><strong>99.00</strong></em></center></td><td><center><em>87.57</em></center></td><td><center><em><strong>28.28</strong></em></center></td><td><center><em>6.10</em></center></td><td><center><em>27.70</em></center></td><td><center><em>40.36</em></center></td>
634
+ </tr>
635
+ <tr>
636
+ <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>82.13</center></td><td><center>65.24</center></td><td><center>-</center></td><td><center>-</center></td><td><center>26.25</center></td><td><center>6.09</center></td><td><center>-</center></td><td><center>-</center></td>
637
+ </tr>
638
+ </tbody>
639
+ </table>
640
+
641
+
642
+ <table>
643
+ <tbody>
644
+ <tr>
645
+ <td></td>
646
+ <td colspan="4"><center><strong>XQuAD</strong></center></td>
647
+ <td colspan="4"><center><strong>STS</strong></center></td>
648
+ </tr>
649
+ <tr>
650
+ <td></td>
651
+ <td colspan="2"><center><strong>Few-shot</strong></center></td>
652
+ <td colspan="2"><center><strong>Finetuned</strong></center></td>
653
+ <td colspan="2"><center><strong>Few-shot</strong></center></td>
654
+ <td colspan="2"><center><strong>Finetuned</strong></center></td>
655
+ </tr>
656
+ <tr>
657
+ <td><strong>Model</strong></td>
658
+ <td><center><strong>(EM)</strong></center></td>
659
+ <td><center><strong>(F1)</strong></center></td>
660
+ <td><center><strong>(EM)</strong></center></td>
661
+ <td><center><strong>(F1)</strong></center></td>
662
+ <td><center><strong>(Spearman)</strong></center></td>
663
+ <td><center><strong>(Pearson)</strong></center></td>
664
+ <td><center><strong>(Spearman)</strong></center></td>
665
+ <td><center><strong>(Pearson)</strong></center></td>
666
+ </tr>
667
+ <tr>
668
+ <td>Mistral-7B-Instruct-v0.2</td><td><center>27.92</center></td><td><center>50.71</center></td><td><center><strong>65.46</strong></center></td><td><center><strong>79.73</strong></center></td><td><center>62.62</center></td><td><center>60.86</center></td><td><center>84.92</center></td><td><center>85.44</center></td>
669
+ </tr>
670
+ <tr>
671
+ <td>RoMistral-7b-Instruct-2024-05-17</td><td><center><strong>43.66</strong></center></td><td><center><strong>63.70</strong></center></td><td><center>55.04</center></td><td><center>72.31</center></td><td><center>77.43</center></td><td><center><strong>78.43</strong></center></td><td><center>87.25</center></td><td><center>87.79</center></td>
672
+ </tr>
673
+ <tr>
674
+ <td><em>RoMistral-7b-Instruct-2024-10-09</em></td><td><center><em>41.09</em></center></td><td><center><em>63.21</em></center></td><td><center><em>47.56</em></center></td><td><center><em>62.69</em></center></td><td><center><em><strong>78.47</strong></em></center></td><td><center><em>77.24</em></center></td><td><center><em><strong>87.28</strong></em></center></td><td><center><em><strong>87.88</strong></em></center></td>
675
+ </tr>
676
+ <tr>
677
+ <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>23.40</center></td><td><center>45.80</center></td><td><center>-</center></td><td><center>-</center></td><td><center>77.33</center></td><td><center>76.60</center></td><td><center>-</center></td><td><center>-</center></td>
678
+ </tr>
679
+ </tbody>
680
+ </table>
681
+
682
+
683
+ ## MT-Bench
684
+
685
+ <table>
686
+ <tbody>
687
+ <tr>
688
+ <td><strong>Model</strong></td>
689
+ <td><strong><center>Average</center></strong></td>
690
+ <td><strong><center>1st turn</center></strong></td>
691
+ <td><strong><center>2nd turn</center></strong></td>
692
+ <td><strong><center>Answers in Ro</center></strong></td>
693
+ </tr>
694
+ <tr>
695
+ <td>Mistral-7B-Instruct-v0.2</td><td><center>5.03</center></td><td><center>5.05</center></td><td><center>5.00</center></td><td><center>154/160</center></td>
696
+ </tr>
697
+ <tr>
698
+ <td>RoMistral-7b-Instruct-2024-05-17</td><td><center>4.99</center></td><td><center>5.46</center></td><td><center>4.53</center></td><td><center><strong>160/160</strong></center></td>
699
+ </tr>
700
+ <tr>
701
+ <td><em>RoMistral-7b-Instruct-2024-10-09</em></td><td><center><em>5.29</em></center></td><td><center><em>5.86</em></center></td><td><center><em>4.72</em></center></td><td><center><em><strong>160/160</strong></em></center></td>
702
+ </tr>
703
+ <tr>
704
+ <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center><strong>5.88</strong></center></td><td><center><strong>6.44</strong></center></td><td><center><strong>5.33</strong></center></td><td><center><strong>160/160</strong></center></td>
705
+ </tr>
706
+ </tbody>
707
+ </table>
708
+
709
+
710
+ ## RoCulturaBench
711
+
712
+ <table>
713
+ <tbody>
714
+ <tr>
715
+ <td><strong>Model</strong></td>
716
+ <td><strong><center>Average</center></strong></td>
717
+ <td><strong><center>Answers in Ro</center></strong></td>
718
+ </tr>
719
+ <tr>
720
+ <td>Mistral-7B-Instruct-v0.2</td><td><center>3.68</center></td><td><center>97/100</center></td>
721
+ </tr>
722
+ <tr>
723
+ <td>RoMistral-7b-Instruct-2024-05-17</td><td><center>3.38</center></td><td><center><strong>100/100</strong></center></td>
724
+ </tr>
725
+ <tr>
726
+ <td><em>RoMistral-7b-Instruct-2024-10-09</em></td><td><center><em>3.99</em></center></td><td><center><em><strong>100/100</strong></em></center></td>
727
+ </tr>
728
+ <tr>
729
+ <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center><strong>4.72</strong></center></td><td><center><strong>100/100</strong></center></td>
730
+ </tr>
731
+ </tbody>
732
+ </table>
733
+
734
+
735
+
736
+ ## RoMistral Model Family
737
+
738
+ | Model | Link |
739
+ |--------------------|:--------:|
740
+ |RoMistral-7b-Instruct-2024-05-17| [link](https://huggingface.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17) |
741
+ |*RoMistral-7b-Instruct-2024-10-09*| [link](https://huggingface.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-10-09) |
742
+ |RoMistral-7b-Instruct-DPO-2024-10-09| [link](https://huggingface.co/OpenLLM-Ro/RoMistral-7b-Instruct-DPO-2024-10-09) |
743
+
744
+
745
+ ## Citation
746
+
747
+ ```
748
+ @misc{masala2024vorbecstiromanecsterecipetrain,
749
+ title={"Vorbe\c{s}ti Rom\^ane\c{s}te?" A Recipe to Train Powerful Romanian LLMs with English Instructions},
750
+ author={Mihai Masala and Denis C. Ilie-Ablachim and Alexandru Dima and Dragos Corlatescu and Miruna Zavelca and Ovio Olaru and Simina Terian-Dan and Andrei Terian-Dan and Marius Leordeanu and Horia Velicu and Marius Popescu and Mihai Dascalu and Traian Rebedea},
751
+ year={2024},
752
+ eprint={2406.18266},
753
+ archivePrefix={arXiv},
754
+ primaryClass={cs.CL},
755
+ url={https://arxiv.org/abs/2406.18266},
756
+ }
757
+ ```
758
+ <!-- **APA:**
759
+
760
+ [More Information Needed] -->