Update for human-eval json input

[lmzuccarelli]

The human eval has a json file with the following structure

[
   {
      "task_id": "xxx",
       "prompt": "xxx",
       "entry_point": "xxx",
       "canonical_solution": "xxx",
       "test": "xxx"
   },{ ...
   }
]

This small change allows for the generation function to work with this compounded json structure

I also added the "api_key" override (as I was using a gated llama model to test)

Summary by CodeRabbit

• Enhancements
  • Inference worker now accepts custom API keys instead of using fixed defaults
  • Batch generation method now supports optional prompts, extracting them from instance data using configurable keys for greater flexibility

[coderabbitai]

📝 Walkthrough

Walkthrough

The InferenceWorker class is updated to accept a configurable API key parameter (key) in its constructor, replacing the hardcoded "not-needed" value. The generate_batch method now accepts an optional prompts parameter and extracts prompts directly from instance dictionaries using a configurable prompt_key when prompts are not provided.

Changes

Cohort / File(s)	Summary
InferenceWorker Configuration evals/common/inference_worker.py	Added key parameter to __init__ for customizable API key configuration. Made prompts parameter optional in generate_batch with logic to extract prompts from instances via configurable prompt_key when not provided.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'Update for human-eval json input' is vague and doesn't clearly convey the specific technical changes made (api_key parameter support and optional prompts handling).	Consider a more specific title that highlights the main changes, such as 'Support api_key parameter and optional prompts in InferenceWorker' or 'Enable human-eval json structure support with configurable api_key'.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}