Hugging Face's logo

xuan-luo
/
FlexiDepth-Llama-3-8B-Instruct

Text Generation
Transformers
Safetensors
English
ddllama
conversational
custom_code
Model card Files Community
1
xuan-luo commited on
Commit
6c7b28c
·
verified ·
1 Parent(s): 793dca0

Update evals/1-mmlu.out

Browse files
Files changed (1) hide show
  1. evals/1-mmlu.out +53 -55
evals/1-mmlu.out CHANGED
@@ -1,75 +1,73 @@
1
- 2025-03-30:04:03:05 INFO [loggers.evaluation_tracker:272] Output path not provided, skipping saving results aggregated
2
- hf (pretrained=../,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 4
3
  | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
4
  |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
5
- |mmlu | 2|none | |acc |↑ |0.6642|± |0.0038|
6
- | - humanities | 2|none | |acc |↑ |0.6091|± |0.0068|
7
- | - formal_logic | 1|none | 5|acc |↑ |0.5000|± |0.0447|
8
- | - high_school_european_history | 1|none | 5|acc |↑ |0.7515|± |0.0337|
9
- | - high_school_us_history | 1|none | 5|acc |↑ |0.8382|± |0.0258|
10
- | - high_school_world_history | 1|none | 5|acc |↑ |0.8270|± |0.0246|
11
  | - international_law | 1|none | 5|acc |↑ |0.7769|± |0.0380|
12
  | - jurisprudence | 1|none | 5|acc |↑ |0.7870|± |0.0396|
13
  | - logical_fallacies | 1|none | 5|acc |↑ |0.7546|± |0.0338|
14
  | - moral_disputes | 1|none | 5|acc |↑ |0.7457|± |0.0234|
15
- | - moral_scenarios | 1|none | 5|acc |↑ |0.4447|± |0.0166|
16
- | - philosophy | 1|none | 5|acc |↑ |0.7492|± |0.0246|
17
- | - prehistory | 1|none | 5|acc |↑ |0.7438|± |0.0243|
18
- | - professional_law | 1|none | 5|acc |↑ |0.4817|± |0.0128|
19
- | - world_religions | 1|none | 5|acc |↑ |0.8246|± |0.0292|
20
- | - other | 2|none | |acc |↑ |0.7422|± |0.0076|
21
- | - business_ethics | 1|none | 5|acc |↑ |0.6600|± |0.0476|
22
- | - clinical_knowledge | 1|none | 5|acc |↑ |0.7434|± |0.0269|
23
- | - college_medicine | 1|none | 5|acc |↑ |0.6705|± |0.0358|
24
- | - global_facts | 1|none | 5|acc |↑ |0.5000|± |0.0503|
25
- | - human_aging | 1|none | 5|acc |↑ |0.7309|± |0.0298|
26
  | - management | 1|none | 5|acc |↑ |0.8155|± |0.0384|
27
  | - marketing | 1|none | 5|acc |↑ |0.8974|± |0.0199|
28
- | - medical_genetics | 1|none | 5|acc |↑ |0.8200|± |0.0386|
29
- | - miscellaneous | 1|none | 5|acc |↑ |0.8506|± |0.0127|
30
- | - nutrition | 1|none | 5|acc |↑ |0.7484|± |0.0248|
31
- | - professional_accounting | 1|none | 5|acc |↑ |0.5390|± |0.0297|
32
- | - professional_medicine | 1|none | 5|acc |↑ |0.7390|± |0.0267|
33
  | - virology | 1|none | 5|acc |↑ |0.5422|± |0.0388|
34
- | - social sciences | 2|none | |acc |↑ |0.7618|± |0.0075|
35
- | - econometrics | 1|none | 5|acc |↑ |0.5702|± |0.0466|
36
  | - high_school_geography | 1|none | 5|acc |↑ |0.8283|± |0.0269|
37
  | - high_school_government_and_politics| 1|none | 5|acc |↑ |0.9171|± |0.0199|
38
- | - high_school_macroeconomics | 1|none | 5|acc |↑ |0.6718|± |0.0238|
39
- | - high_school_microeconomics | 1|none | 5|acc |↑ |0.7227|± |0.0291|
40
  | - high_school_psychology | 1|none | 5|acc |↑ |0.8587|± |0.0149|
41
- | - human_sexuality | 1|none | 5|acc |↑ |0.7634|± |0.0373|
42
  | - professional_psychology | 1|none | 5|acc |↑ |0.7026|± |0.0185|
43
- | - public_relations | 1|none | 5|acc |↑ |0.6727|± |0.0449|
44
- | - security_studies | 1|none | 5|acc |↑ |0.7184|± |0.0288|
45
- | - sociology | 1|none | 5|acc |↑ |0.8557|± |0.0248|
46
  | - us_foreign_policy | 1|none | 5|acc |↑ |0.8400|± |0.0368|
47
- | - stem | 2|none | |acc |↑ |0.5744|± |0.0085|
48
  | - abstract_algebra | 1|none | 5|acc |↑ |0.3700|± |0.0485|
49
  | - anatomy | 1|none | 5|acc |↑ |0.7037|± |0.0394|
50
- | - astronomy | 1|none | 5|acc |↑ |0.7500|± |0.0352|
51
- | - college_biology | 1|none | 5|acc |↑ |0.8125|± |0.0326|
52
- | - college_chemistry | 1|none | 5|acc |↑ |0.5000|± |0.0503|
53
- | - college_computer_science | 1|none | 5|acc |↑ |0.5400|± |0.0501|
54
- | - college_mathematics | 1|none | 5|acc |↑ |0.3600|± |0.0482|
55
- | - college_physics | 1|none | 5|acc |↑ |0.4412|± |0.0494|
56
- | - computer_security | 1|none | 5|acc |↑ |0.7500|± |0.0435|
57
  | - conceptual_physics | 1|none | 5|acc |↑ |0.5957|± |0.0321|
58
- | - electrical_engineering | 1|none | 5|acc |↑ |0.6138|± |0.0406|
59
- | - elementary_mathematics | 1|none | 5|acc |↑ |0.4735|± |0.0257|
60
- | - high_school_biology | 1|none | 5|acc |↑ |0.8065|± |0.0225|
61
- | - high_school_chemistry | 1|none | 5|acc |↑ |0.5517|± |0.0350|
62
- | - high_school_computer_science | 1|none | 5|acc |↑ |0.7000|± |0.0461|
63
- | - high_school_mathematics | 1|none | 5|acc |↑ |0.3889|± |0.0297|
64
- | - high_school_physics | 1|none | 5|acc |↑ |0.4901|± |0.0408|
65
- | - high_school_statistics | 1|none | 5|acc |↑ |0.5648|± |0.0338|
66
- | - machine_learning | 1|none | 5|acc |↑ |0.4196|± |0.0468|
67
 
68
  | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
69
  |------------------|------:|------|------|------|---|-----:|---|-----:|
70
- |mmlu | 2|none | |acc |↑ |0.6642|± |0.0038|
71
- | - humanities | 2|none | |acc |↑ |0.6091|± |0.0068|
72
- | - other | 2|none | |acc |↑ |0.7422|± |0.0076|
73
- | - social sciences| 2|none | |acc |↑ |0.7618|± |0.0075|
74
- | - stem | 2|none | |acc |↑ |0.5744|± |0.0085|
75
-
 
1
+ f (pretrained=../,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
 
2
  | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
3
  |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
4
+ |mmlu | 2|none | |acc |↑ |0.6634|± |0.0038|
5
+ | - humanities | 2|none | |acc |↑ |0.6077|± |0.0068|
6
+ | - formal_logic | 1|none | 5|acc |↑ |0.5238|± |0.0447|
7
+ | - high_school_european_history | 1|none | 5|acc |↑ |0.7455|± |0.0340|
8
+ | - high_school_us_history | 1|none | 5|acc |↑ |0.8480|± |0.0252|
9
+ | - high_school_world_history | 1|none | 5|acc |↑ |0.8312|± |0.0244|
10
  | - international_law | 1|none | 5|acc |↑ |0.7769|± |0.0380|
11
  | - jurisprudence | 1|none | 5|acc |↑ |0.7870|± |0.0396|
12
  | - logical_fallacies | 1|none | 5|acc |↑ |0.7546|± |0.0338|
13
  | - moral_disputes | 1|none | 5|acc |↑ |0.7457|± |0.0234|
14
+ | - moral_scenarios | 1|none | 5|acc |↑ |0.4458|± |0.0166|
15
+ | - philosophy | 1|none | 5|acc |↑ |0.7460|± |0.0247|
16
+ | - prehistory | 1|none | 5|acc |↑ |0.7407|± |0.0244|
17
+ | - professional_law | 1|none | 5|acc |↑ |0.4752|± |0.0128|
18
+ | - world_religions | 1|none | 5|acc |↑ |0.8187|± |0.0295|
19
+ | - other | 2|none | |acc |↑ |0.7441|± |0.0076|
20
+ | - business_ethics | 1|none | 5|acc |↑ |0.6800|± |0.0469|
21
+ | - clinical_knowledge | 1|none | 5|acc |↑ |0.7509|± |0.0266|
22
+ | - college_medicine | 1|none | 5|acc |↑ |0.6647|± |0.0360|
23
+ | - global_facts | 1|none | 5|acc |↑ |0.4700|± |0.0502|
24
+ | - human_aging | 1|none | 5|acc |↑ |0.7265|± |0.0299|
25
  | - management | 1|none | 5|acc |↑ |0.8155|± |0.0384|
26
  | - marketing | 1|none | 5|acc |↑ |0.8974|± |0.0199|
27
+ | - medical_genetics | 1|none | 5|acc |↑ |0.8300|± |0.0378|
28
+ | - miscellaneous | 1|none | 5|acc |↑ |0.8519|± |0.0127|
29
+ | - nutrition | 1|none | 5|acc |↑ |0.7582|± |0.0245|
30
+ | - professional_accounting | 1|none | 5|acc |↑ |0.5426|± |0.0297|
31
+ | - professional_medicine | 1|none | 5|acc |↑ |0.7426|± |0.0266|
32
  | - virology | 1|none | 5|acc |↑ |0.5422|± |0.0388|
33
+ | - social sciences | 2|none | |acc |↑ |0.7595|± |0.0075|
34
+ | - econometrics | 1|none | 5|acc |↑ |0.5263|± |0.0470|
35
  | - high_school_geography | 1|none | 5|acc |↑ |0.8283|± |0.0269|
36
  | - high_school_government_and_politics| 1|none | 5|acc |↑ |0.9171|± |0.0199|
37
+ | - high_school_macroeconomics | 1|none | 5|acc |↑ |0.6615|± |0.0240|
38
+ | - high_school_microeconomics | 1|none | 5|acc |↑ |0.7395|± |0.0285|
39
  | - high_school_psychology | 1|none | 5|acc |↑ |0.8587|± |0.0149|
40
+ | - human_sexuality | 1|none | 5|acc |↑ |0.7786|± |0.0364|
41
  | - professional_psychology | 1|none | 5|acc |↑ |0.7026|± |0.0185|
42
+ | - public_relations | 1|none | 5|acc |↑ |0.6545|± |0.0455|
43
+ | - security_studies | 1|none | 5|acc |↑ |0.7143|± |0.0289|
44
+ | - sociology | 1|none | 5|acc |↑ |0.8507|± |0.0252|
45
  | - us_foreign_policy | 1|none | 5|acc |↑ |0.8400|± |0.0368|
46
+ | - stem | 2|none | |acc |↑ |0.5734|± |0.0084|
47
  | - abstract_algebra | 1|none | 5|acc |↑ |0.3700|± |0.0485|
48
  | - anatomy | 1|none | 5|acc |↑ |0.7037|± |0.0394|
49
+ | - astronomy | 1|none | 5|acc |↑ |0.7566|± |0.0349|
50
+ | - college_biology | 1|none | 5|acc |↑ |0.8264|± |0.0317|
51
+ | - college_chemistry | 1|none | 5|acc |↑ |0.4800|± |0.0502|
52
+ | - college_computer_science | 1|none | 5|acc |↑ |0.5300|± |0.0502|
53
+ | - college_mathematics | 1|none | 5|acc |↑ |0.3500|± |0.0479|
54
+ | - college_physics | 1|none | 5|acc |↑ |0.4902|± |0.0497|
55
+ | - computer_security | 1|none | 5|acc |↑ |0.7600|± |0.0429|
56
  | - conceptual_physics | 1|none | 5|acc |↑ |0.5957|± |0.0321|
57
+ | - electrical_engineering | 1|none | 5|acc |↑ |0.6069|± |0.0407|
58
+ | - elementary_mathematics | 1|none | 5|acc |↑ |0.4656|± |0.0257|
59
+ | - high_school_biology | 1|none | 5|acc |↑ |0.8032|± |0.0226|
60
+ | - high_school_chemistry | 1|none | 5|acc |↑ |0.5567|± |0.0350|
61
+ | - high_school_computer_science | 1|none | 5|acc |↑ |0.6900|± |0.0465|
62
+ | - high_school_mathematics | 1|none | 5|acc |↑ |0.3704|± |0.0294|
63
+ | - high_school_physics | 1|none | 5|acc |↑ |0.4702|± |0.0408|
64
+ | - high_school_statistics | 1|none | 5|acc |↑ |0.5694|± |0.0338|
65
+ | - machine_learning | 1|none | 5|acc |↑ |0.4554|± |0.0473|
66
 
67
  | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
68
  |------------------|------:|------|------|------|---|-----:|---|-----:|
69
+ |mmlu | 2|none | |acc |↑ |0.6634|± |0.0038|
70
+ | - humanities | 2|none | |acc |↑ |0.6077|± |0.0068|
71
+ | - other | 2|none | |acc |↑ |0.7441|± |0.0076|
72
+ | - social sciences| 2|none | |acc |↑ |0.7595|± |0.0075|
73
+ | - stem | 2|none | |acc |↑ |0.5734|± |0.0084|