Revisit RCA oracle

[tianyin]

The current oracle is designed by @EvanTian233 through an extensive discussion,
SREGym#651

Personally, I like the current oracle for its simplicity. It's certainly not perfect. I wish we could have a grounded model of failures, but I myself fail to do so.

Unfortunately, I don't have an idea how good or bad it is -- do you encounter a problem that the oracle misgrades the agent's results? If so, please report so we can take a look at it and improve the oracle.

There is at least one outstanding issue to close
SREGym#745

There was also a historical issue here,
SREGym#747

@SuMayaBee Could you take charge of this task and hopefully to close SREGym#745?

[SuMayaBee]

Thanks for assigning this to me, @tianyin. I'll go through SREGym#745 and SREGym#747 first to get the full context, then take a look at the oracle and report back here with what I find.

[varuni7]

Will go through and take a look!

[varuni7]

The main issue: The current oracle scores the agent's diagnosis text and pattern-matching behaviour, with the same weightage if the agent actually ran the correct investigation.
For example for the CFS_CPU_throttling fault , the agent can just check the cpu limt set for other services and guess the fault. SREGym#807 (review)

Proposed Solution: The ProcessOracle
I propose introducing a ProcessOracle that evaluates the agent’s tool-call trace in tandem with its textual diagnosis. This ensures that the agent's conclusion is grounded in actual observation.

1. The Fault Specification Artefact (fault_spec.yaml)
   Every fault injector should be accompanied by a YAML specification file defined at contribution time. This spec outlines all the valid solutions that the contributor thinks the agent can take, and is a valid way to fix it.

Investigation Paths: A list of known, valid sequences of tool calls or telemetry queries that conclusively reveal the fault.

Corroboration Section: A set of mandatory or supporting data points the agent should have encountered during its execution, regardless of the specific troubleshooting path taken.

On a per-fault basis , the contributor can choose not to include a particular section.

2. Handling Novel Paths
   Strict rule-matching against a static spec risks penalizing agents that find novel, valid troubleshooting paths we didn't anticipate. To handle this, the framework will use a two-tiered evaluation:

Deterministic Match: If the agent's tool trace matches a known path in the fault_spec.yaml, it passes the process check.

Epistemic Fallback (LLM Judge): If the path is novel, the LLM judge evaluates the raw trace as usual. Crucially, the judge is not shown the known paths from the spec to prevent anchoring any bias.

A sample yaml file for SREGym#807 is attached here

cfs_cpu_throttling_hotel_reservation.yaml

[SuMayaBee]

While working on SREGym#745 I ran into a case where the mitigation AlertOracle can misgrade the agent, so I wanted to report it and propose a small fix that keeps the oracle simple.

Right now the mitigation AlertOracle checks only one thing after the agent is done. It looks at whether any alerts are still firing in the namespace. If even one alert is still firing, the agent is marked as failed. If no alerts are firing, the agent is marked as passed.

Because it only looks at the final state, it can get the grade wrong in two opposite ways.

First, it can fail a good agent. Some alerts are always firing for reasons that have nothing to do with the injected fault. A clear example is ContainerCPUThrottling from the astronomy-shop Grafana sidecar, which fires constantly because the sidecar has a very low CPU limit. The oracle sees that alert at the end and marks the agent as failed, even if the agent actually fixed the real problem. This is SREGym#745.

Second, it can pass an agent that did nothing. The oracle only checks that the namespace is quiet at the end. It never checks that the injected fault produced any alert in the first place. So the oracle is only correct as long as every fault reliably fires an alert, and nothing in the code enforces that. If a fault ever fires no alert (for example a missing or mismatched alert rule), the namespace stays quiet the whole time and an agent that did nothing will pass, because the oracle cannot tell "the agent fixed it" apart from "nothing ever broke."

My proposal is to have the oracle look at three moments instead of one, using the same alert query it already uses.

1. Before the fault is injected, record which alerts are already firing and ignore them for the rest of the run. This is the issue astronomy-shop Grafana sidecar trips ContainerCPUThrottling, breaking AlertOracle mitigation grading SREGym/SREGym#745 fix, and I already have this part drafted.
2. Right after the fault is injected, check that at least one new alert fired compared to the baseline. If nothing new fires, treat the run as invalid or mis-specified instead of a pass, because there was nothing for the agent to fix.
3. At the end, require the new alerts to clear. This is what the oracle already does today.

This keeps the oracle as simple as it is now. Instead of only confirming that the namespace is quiet at the end, it also confirms that something actually broke and that the agent actually cleared it.

I am still finishing SREGym#745 (the first step above) and will open the PR for it shortly. Let me know if this direction makes sense.

[EvanTian233]

@varuni7 I like your idea of process oracle - it was part of the original DiagnosisJudge design but we dropped it for paper submission. After reading your proposal here's a couple questions on my mind:

• How is this sample yaml generated? Can the process oracle generation be formalized into a process that's applicable to all questions?
• How to position the process score in evaluation? If the outcome is correct, does process matter?
• What is the cost at eval time and oracle generation time?

Sharing some relevant papers for you to refine the design:
https://arxiv.org/pdf/2606.27154
https://arxiv.org/abs/2601.22208

[EvanTian233]

@SuMayaBee Thanks for spotting this issue and looking into this. What you said in the first point makes a lot of sense to me.

On the second point, in my understanding since AlertOracle is only looking at prometheus, there could be other symptoms than alerts across the failure cases. If nothing new fires, the run may still be valid as there could be other things for agent to fix. So how to handle this may require more thinking with consideration of the mitigation oracle as a whole. @Saadmrp1038 Please correct me if I'm wrong.