Interop: a converter from EvalPort ResultSets into eval.schema.json records?

[adhabnr-ux]

Hi — I maintain EvalPort, a smaller, differently-scoped project in a similar space, and wanted to raise a genuine interop question rather than pitch an "adapter" in the usual sense.

What I read before writing this

I went through every_eval_ever/schemas/eval.schema.json directly (not just the README/paper), and this is a considerably more developed schema than I expected for a "leaderboard metadata" project — the evaluation_results[].metric_config block alone (score_type: binary/continuous/levels, min_score/max_score, level_names, llm_scoring.judges[] with per-judge weight/aggregation, uncertainty.confidence_interval/standard_error) covers cases EvalPort's own result schema doesn't attempt — and source_metadata.source_type (documentation vs evaluation_run) cleanly distinguishes "scraped from a paper" from "we ran this," which is a distinction our schema doesn't make explicit. The converters for Inspect AI/HELM/lm-eval-harness under every_eval_ever/converters/ and the adapter catalog in every_eval_ever/adapters/catalog.py also make clear this is a live, actively-converged-on format (I see the ACL 2026 shared task and the recent cron-ingestion work in the commit history), not a paper spec.

Where I think the overlap actually is

EEE's eval.schema.json is built for aggregating heterogeneous evaluation results (leaderboard scrapes, papers, local runs) into one crowdsourced, model-indexed datastore — it's fundamentally a record format for a database. EvalPort's TestCase/GraderResult/ResultSet schema is built for one tool's single eval run against one suite (portable between DeepEval, Promptfoo, Inspect AI, etc. — the unit is "this run, this suite, these graders"), with no concept of leaderboard aggregation, model registries, or crowdsourced review. So I don't think a general bidirectional converter is honest — the two schemas answer different questions and I'd be forcing a mapping neither format asked for.

What might genuinely be useful: EvalPort now has adapters that normalize output from a dozen-plus tools into a single ResultSet shape. A one-way converter — ResultSet (+ whatever source_metadata/model_info context the caller supplies) → one or more eval.schema.json records — could be a lower-friction on-ramp for someone who already has EvalPort-formatted output and wants to contribute it to your datastore, sitting alongside your existing Inspect AI/HELM/lm-eval-harness converters. It would only ever produce source_type: "evaluation_run" records, since a ResultSet doesn't carry the leaderboard/paper-scrape provenance documentation implies.

I'm not certain this clears the bar your CONTRIBUTING.md sets for a new converter, and I'd rather ask than build it unprompted. Given your review lanes ("Planning something structural? Agree the approach before opening a PR"), does this sound useful, or does the scope mismatch make it not worth the added surface area? Also happy to just compare notes on schema design if that's more valuable than code — genuinely curious how you settled on some of the metric_config tradeoffs (e.g. requiring min_score/max_score for continuous scores, the levels + has_unknown_level design).