Sets the num_proc argument for all calls to load_dataset in the repo.

[finbarrtimbers]

This lets us use multiple processes to load the dataset.

Runs:

• Single GPU finetune job: Beaker
• Singe GPU GRPO run: Beaker

---

Note

Standardizes parallel dataset loading by using max_num_processes() for num_proc across the repository.

• Performance/Parallelism:
  • Add open_instruct.utils.max_num_processes() helper and apply it to nearly all datasets.load_dataset(...) calls across the codebase to set num_proc for parallel data loading.
  • Update modules and scripts (e.g., decontamination/*, open_instruct/dataset_transformation.py, quantize/*, scripts/data/*, evaluation/util scripts) to import the helper and pass num_proc consistently.
• Refactor:
  • Minor import adjustments to reference open_instruct.utils and max_num_processes where used.

^{Written by Cursor Bugbot for commit a5e6d55. This will update automatically on new commits. Configure here.}

[natolambert]

LGTM - I didn't explicitly test any of the changes