Skip to content

Commit 8850ebc

Browse files
authoredMar 28, 2025··
Increase default max_gen_toks to 2048 and max_length to 8192 for MMLU Pro tests (#2824)
* Changed default max_length from 2048 to 8192 and max_gen_toks from 256 to 2048 fro MMLU Pro tasks. * Update lm_eval/tasks/mmlu_pro/_default_template_yaml * pre-commit * nit ---------
1 parent ebbbb96 commit 8850ebc

File tree

2 files changed

+4
-1
lines changed

2 files changed

+4
-1
lines changed
 

‎lm_eval/tasks/mmlu_pro/README.md

+2
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,5 @@ If other tasks on this dataset are already supported:
6262

6363
* (tasks, group) 2024-09-23 -- (version 1 --> version 2)
6464
* Added one newline to task description(s) as per [reference implementation](https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/47b9891aacb8bd7cda29d5c5ba17b9434dd333bc/evaluate_from_local.py#L93)
65+
* (tasks, group) 2025-03-20 -- (version 2.0 --> version 2.1)
66+
* Changed default max_length from 2048 to 8192 and max_gen_toks from 256 to 2048.

‎lm_eval/tasks/mmlu_pro/_default_template_yaml

+2-1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ generation_kwargs:
2020
- "</s>"
2121
- "Q:"
2222
- "<|im_end|>"
23+
max_gen_toks: 2048
2324
do_sample: false
2425
temperature: 0.0
2526
num_fewshot: 5
@@ -30,4 +31,4 @@ metric_list:
3031
ignore_case: true
3132
ignore_punctuation: true
3233
metadata:
33-
version: 1.0
34+
version: 2.1

0 commit comments

Comments
 (0)
Please sign in to comment.