seth-zou/dpsk02
Updated
•
6
lingang
seth-zou
·
AI & ML interests
None yet
Recent Activity
updated
a model
4 days ago
seth-zou/dpsk02
published
a model
4 days ago
seth-zou/dpsk02
new activity
9 days ago
deepseek-ai/DeepSeek-V3.1-Base:Modelcard come On ~~
Organizations
updated
a
model
4 days ago
published
a
model
4 days ago
Modelcard come On ~~
#23 opened 9 days ago
by
seth-zou
published
a
dataset
2 months ago
reacted to
clefourrier's
post with 🤯
over 1 year ago
Post
2257
Fun fact about evaluation, part 2!
How much do scores change depending on prompt format choice?
Using different prompts (all present in the literature, from
10 points for a single model!
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt...
How much do scores change depending on prompt format choice?
Using different prompts (all present in the literature, from
Prompt question? to Question: prompt question?\nChoices: enumeration of all choices\nAnswer: ), we get a score range of...10 points for a single model!
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt...
updated
a
model
over 1 year ago