How to get the EvalSet results when running in pytest

[khaledhikmat]

Is there a reason why evaluate_eval_set does not return eval set result?

result = await AgentEvaluator.evaluate_eval_set(...)

Is there a way to retrieve the result without resorting to some internal/private classes or methods? Thanks.

I am using google-adk 1.27.2

[surajksharma07]

@khaledhikmat evaluate_eval_set currently doesn't return anything; it only asserts on failures internally.

As a workaround, you could try overriding the method and adding a return right after the assert not failures line in agent_evaluator.py:

return EvalSetResult( eval_set_result_id=str(uuid.uuid4()), eval_set_id=eval_set.eval_set_id, eval_case_results=[r for results in eval_results_by_eval_id.values() for r in results] )

[khaledhikmat]

Ok thank you @surajksharma07 . I sub-classed the AgentEvaluator class to override the method. It seems to work for now. Is this something that is planned to be fixed in upcoming releases? Thanks again.

[surajksharma07]

@khaledhikmat Glad the subclass approach is working for now.

Yes, this is a real bug — evaluate_eval_set should be returning an EvalSetResult and we've flagged it for review.

A proper fix would just be adding the return statement directly in agent_evaluator.py rather than relying on a subclass.

If you'd like to contribute a PR for it, that would be very welcome — it's a small change and should be straightforward to get merged.

Team is looking more into it.

[surajksharma07]

@ankursharmas Please have a look into it once.

[khaledhikmat]

Hi @surajksharma07, sorry for my delay. I wanted to say that the way I implemented this is to omit the assert in the method itself and make the caller responsible for asserting. This is because I wanted to make sure that the caller always receives the eval result so I can generate an HTML report (of the result). Once the report is generated, I then check for EvalStatus.FAILED in any of the cases and assert if need be.

While this delegates the assert responsibility to the caller (which is probably not great), I found this solution to be the most elegant for my need. Please let me know if this makes sense to you. Thanks.

[surajksharma07]

@khaledhikmat That approach makes total sense — and honestly it's the cleaner design.
Mixing result retrieval and assertion in the same method is a separation-of-concerns issue, so having the caller control when to assert (especially for a reporting use case like yours) is the right call.

When the fix lands in agent_evaluator.py, it'd be worth proposing this same pattern in the PR thread: return the result unconditionally, drop the internal assert, and let callers decide what to do with it.
Worth flagging that explicitly so the fix goes in the right direction rather than just bolting a return onto the current assert-first flow!

[khaledhikmat]

Hi @surajksharma07 , thanks for your comment. I will be on the lookout for the fix.

[surajksharma07]

Sounds good @khaledhikmat — and honestly, since you already have the pattern working (return unconditionally, let the caller assert), you're basically 90% of the way to a PR yourself!

The change in agent_evaluator.py would just be adding the return before the assert and dropping the internal assert — it's a quick one, and if you ever want to contribute it, it'd be very welcome!