You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Increase default max_gen_toks to 2048 and max_length to 8192 for MMLU Pro tests (#2824)
* Changed default max_length from 2048 to 8192 and max_gen_toks from 256 to 2048 fro MMLU Pro tasks.
* Update lm_eval/tasks/mmlu_pro/_default_template_yaml
* pre-commit
* nit
---------
Copy file name to clipboardexpand all lines: lm_eval/tasks/mmlu_pro/README.md
+2
Original file line number
Diff line number
Diff line change
@@ -62,3 +62,5 @@ If other tasks on this dataset are already supported:
62
62
63
63
* (tasks, group) 2024-09-23 -- (version 1 --> version 2)
64
64
* Added one newline to task description(s) as per [reference implementation](https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/47b9891aacb8bd7cda29d5c5ba17b9434dd333bc/evaluate_from_local.py#L93)
65
+
* (tasks, group) 2025-03-20 -- (version 2.0 --> version 2.1)
66
+
* Changed default max_length from 2048 to 8192 and max_gen_toks from 256 to 2048.
0 commit comments