| f (pretrained=../,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16 | |
| | Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr| | |
| |---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:| | |
| |mmlu | 2|none | |acc |↑ |0.6634|± |0.0038| | |
| | - humanities | 2|none | |acc |↑ |0.6077|± |0.0068| | |
| | - formal_logic | 1|none | 5|acc |↑ |0.5238|± |0.0447| | |
| | - high_school_european_history | 1|none | 5|acc |↑ |0.7455|± |0.0340| | |
| | - high_school_us_history | 1|none | 5|acc |↑ |0.8480|± |0.0252| | |
| | - high_school_world_history | 1|none | 5|acc |↑ |0.8312|± |0.0244| | |
| | - international_law | 1|none | 5|acc |↑ |0.7769|± |0.0380| | |
| | - jurisprudence | 1|none | 5|acc |↑ |0.7870|± |0.0396| | |
| | - logical_fallacies | 1|none | 5|acc |↑ |0.7546|± |0.0338| | |
| | - moral_disputes | 1|none | 5|acc |↑ |0.7457|± |0.0234| | |
| | - moral_scenarios | 1|none | 5|acc |↑ |0.4458|± |0.0166| | |
| | - philosophy | 1|none | 5|acc |↑ |0.7460|± |0.0247| | |
| | - prehistory | 1|none | 5|acc |↑ |0.7407|± |0.0244| | |
| | - professional_law | 1|none | 5|acc |↑ |0.4752|± |0.0128| | |
| | - world_religions | 1|none | 5|acc |↑ |0.8187|± |0.0295| | |
| | - other | 2|none | |acc |↑ |0.7441|± |0.0076| | |
| | - business_ethics | 1|none | 5|acc |↑ |0.6800|± |0.0469| | |
| | - clinical_knowledge | 1|none | 5|acc |↑ |0.7509|± |0.0266| | |
| | - college_medicine | 1|none | 5|acc |↑ |0.6647|± |0.0360| | |
| | - global_facts | 1|none | 5|acc |↑ |0.4700|± |0.0502| | |
| | - human_aging | 1|none | 5|acc |↑ |0.7265|± |0.0299| | |
| | - management | 1|none | 5|acc |↑ |0.8155|± |0.0384| | |
| | - marketing | 1|none | 5|acc |↑ |0.8974|± |0.0199| | |
| | - medical_genetics | 1|none | 5|acc |↑ |0.8300|± |0.0378| | |
| | - miscellaneous | 1|none | 5|acc |↑ |0.8519|± |0.0127| | |
| | - nutrition | 1|none | 5|acc |↑ |0.7582|± |0.0245| | |
| | - professional_accounting | 1|none | 5|acc |↑ |0.5426|± |0.0297| | |
| | - professional_medicine | 1|none | 5|acc |↑ |0.7426|± |0.0266| | |
| | - virology | 1|none | 5|acc |↑ |0.5422|± |0.0388| | |
| | - social sciences | 2|none | |acc |↑ |0.7595|± |0.0075| | |
| | - econometrics | 1|none | 5|acc |↑ |0.5263|± |0.0470| | |
| | - high_school_geography | 1|none | 5|acc |↑ |0.8283|± |0.0269| | |
| | - high_school_government_and_politics| 1|none | 5|acc |↑ |0.9171|± |0.0199| | |
| | - high_school_macroeconomics | 1|none | 5|acc |↑ |0.6615|± |0.0240| | |
| | - high_school_microeconomics | 1|none | 5|acc |↑ |0.7395|± |0.0285| | |
| | - high_school_psychology | 1|none | 5|acc |↑ |0.8587|± |0.0149| | |
| | - human_sexuality | 1|none | 5|acc |↑ |0.7786|± |0.0364| | |
| | - professional_psychology | 1|none | 5|acc |↑ |0.7026|± |0.0185| | |
| | - public_relations | 1|none | 5|acc |↑ |0.6545|± |0.0455| | |
| | - security_studies | 1|none | 5|acc |↑ |0.7143|± |0.0289| | |
| | - sociology | 1|none | 5|acc |↑ |0.8507|± |0.0252| | |
| | - us_foreign_policy | 1|none | 5|acc |↑ |0.8400|± |0.0368| | |
| | - stem | 2|none | |acc |↑ |0.5734|± |0.0084| | |
| | - abstract_algebra | 1|none | 5|acc |↑ |0.3700|± |0.0485| | |
| | - anatomy | 1|none | 5|acc |↑ |0.7037|± |0.0394| | |
| | - astronomy | 1|none | 5|acc |↑ |0.7566|± |0.0349| | |
| | - college_biology | 1|none | 5|acc |↑ |0.8264|± |0.0317| | |
| | - college_chemistry | 1|none | 5|acc |↑ |0.4800|± |0.0502| | |
| | - college_computer_science | 1|none | 5|acc |↑ |0.5300|± |0.0502| | |
| | - college_mathematics | 1|none | 5|acc |↑ |0.3500|± |0.0479| | |
| | - college_physics | 1|none | 5|acc |↑ |0.4902|± |0.0497| | |
| | - computer_security | 1|none | 5|acc |↑ |0.7600|± |0.0429| | |
| | - conceptual_physics | 1|none | 5|acc |↑ |0.5957|± |0.0321| | |
| | - electrical_engineering | 1|none | 5|acc |↑ |0.6069|± |0.0407| | |
| | - elementary_mathematics | 1|none | 5|acc |↑ |0.4656|± |0.0257| | |
| | - high_school_biology | 1|none | 5|acc |↑ |0.8032|± |0.0226| | |
| | - high_school_chemistry | 1|none | 5|acc |↑ |0.5567|± |0.0350| | |
| | - high_school_computer_science | 1|none | 5|acc |↑ |0.6900|± |0.0465| | |
| | - high_school_mathematics | 1|none | 5|acc |↑ |0.3704|± |0.0294| | |
| | - high_school_physics | 1|none | 5|acc |↑ |0.4702|± |0.0408| | |
| | - high_school_statistics | 1|none | 5|acc |↑ |0.5694|± |0.0338| | |
| | - machine_learning | 1|none | 5|acc |↑ |0.4554|± |0.0473| | |
| | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| | |
| |------------------|------:|------|------|------|---|-----:|---|-----:| | |
| |mmlu | 2|none | |acc |↑ |0.6634|± |0.0038| | |
| | - humanities | 2|none | |acc |↑ |0.6077|± |0.0068| | |
| | - other | 2|none | |acc |↑ |0.7441|± |0.0076| | |
| | - social sciences| 2|none | |acc |↑ |0.7595|± |0.0075| | |
| | - stem | 2|none | |acc |↑ |0.5734|± |0.0084| | |