feat(cli/evals): Generate metricSummaries #2768
Conversation
| const booleanScores = reMapScores(simpleEvalOutput, (response, i) => ({ | ||
| testCaseId: response.testCaseId, | ||
| evaluation: { | ||
| score: i % 2 === 0, |
There was a problem hiding this comment.
This is hard to read. I will have to look at simpleEvalOutput and do the math to understand what this should be. Please just define the test case here.
There was a problem hiding this comment.
I was trying to avoid redefining the EvalResponse object for every test case since it is already a large object, I can add comments to the override function here to make the scores more obvious. LMK if that is helpful.
If not, I can definitely repeat the EvalResponse for each test case.
There was a problem hiding this comment.
Thank you, this is better, but I would prefer having a different helper like I suggested below: "Maybe instead of reMapScores you can have a helper that takes an array of Score objects and generates Record<string, EvalResponse> with testcaseIds. That might be concise enough and easier to read."
Please check if we can do this.
| const stringScores = reMapScores(simpleEvalOutput, (response, i) => ({ | ||
| testCaseId: response.testCaseId, | ||
| evaluation: { | ||
| score: `TYPE_${i % 2}`, |
There was a problem hiding this comment.
ditto. Maybe instead of reMapScores you can have a helper that takes an array of Score objects and generates Record<string, EvalResponse> with testcaseIds. That might be concise enough and easier to read.
|
nit: PR & commit messags title should be |
This is not specific to the CLI, but okay. |
ssbushi
changed the title
| const booleanScores = reMapScores(simpleEvalOutput, (response, i) => ({ | ||
| testCaseId: response.testCaseId, | ||
| evaluation: { | ||
| score: i % 2 === 0, |
There was a problem hiding this comment.
Thank you, this is better, but I would prefer having a different helper like I suggested below: "Maybe instead of reMapScores you can have a helper that takes an array of Score objects and generates Record<string, EvalResponse> with testcaseIds. That might be concise enough and easier to read."
Please check if we can do this.
Tooling part of #1642
Checklist (if applicable):