7.1 What the public benchmarks say
Open, independent measurement of operations agents now exists. Sources for every figure here are in Appendix D.
These are diagnosis and mitigation tasks in controlled environments. They are easier than your production in some ways and harder in others, and the best public results still fail most of the time.
Teams still run agents in production successfully, so both facts hold. The difference is that a production deployment is not a bare model on an unfamiliar cluster. It is a model inside a harness: curated context about a known estate, purpose-built tools, runbooks with your team’s knowledge, memory of past incidents, and policy that routes agents to the incident classes they handle well.
Benchmarks measure the bare model. Production measures the harness. So “which model do you use?” tells you little about a platform. Test the harness instead, which is what the rest of this chapter covers.
7.2 Benchmarks can be gamed, and so can yours
The SREGym research (2026) documented two exploits to remember whenever you read an evaluation claim:- Reward hacking. When the fault injector ran inside the environment the agent could inspect, agents found it and disabled it instead of fixing the fault.
- Alert-clearing as false success. A large share of one benchmark’s mitigation tasks could be “solved” by a generic pod-restart loop that made the alert go away without touching the defect.
- Score state, not symptoms. Check independently that the defect is gone and the system is healthy. Whether the alert cleared proves nothing.
- Hide the test machinery. The agent under test must not be able to see or reach the evaluation harness.
- Score diagnosis and mitigation separately. They fail independently and improve independently.
7.3 Build a golden set from your own incidents
Public benchmarks calibrate the field. Only your incidents calibrate your deployment. The asset to build is a golden set: a curated, versioned collection of test scenarios drawn from your production history.- Harvest from postmortems. Each real incident becomes a candidate scenario with five parts: the telemetry at detection, the environment state, the true root cause, the accepted fix, and the evidence that the fix worked. Twenty to fifty scenarios across your top incident classes is a working golden set. For a first proof of concept, ten is the minimum (7.5). Add every new kind of incident after that.
- Replay, don’t re-break. Feed the recorded signals to the agent and let it investigate against snapshotted or simulated state. Fault injection in staging adds mitigation scoring, with the isolation rules from 7.2 applied.
- Score in layers. Score each scenario on every layer in the table below.
- Use model judges, and check the judges. A model can grade the rubric at scale. Each cycle, compare its grades with human grades on a sample, and keep people grading a rotating slice. On-call engineers rating agent analyses gives you evaluation data and builds trust at the same time.
Fabricated evidence scores zero on the whole scenario, whatever the answer. A right answer built on data the agent never retrieved will fail you on the next incident.
healthy_when line is the state-based check from 7.2. Write it before you run the agent, so nobody grades the answer to fit.
7.4 Treat evals as CI
The agent stack changes constantly: prompts, tools, skills, policies, retrieval, and models, which vendors can swap under you. Each of those is a deploy, and deploys need gates.- Run the golden set on every change.
- Block on per-class regressions, not just the average. A new model that gains on database scenarios and quietly loses on Kubernetes ones is a regression, even if the total goes up.
- Canary in production with online metrics. Track acceptance rate, rollback rate, and autonomous-resolution rate per class. They are your evals running continuously against reality.
7.5 How this maps to CloudThinker
Use the golden set to run a proof of concept (POC) on any platform, including CloudThinker. Appendix A gives the protocol: bring ten or more replayable scenarios from your own history, and score detection, diagnosis, plan, mitigation, and evidence integrity for each one. Running your scenarios in CloudThinker. You can replay a recorded alert by posting its JSON to a Resolve webhook, which sends it through Pulse and can open an incident and start root cause analysis. CloudThinker then investigates the live systems behind your connections, not a snapshot. So point it at a staging account or cluster where you’ve injected the same fault, and score the result by hand against your scenario record. CloudThinker doesn’t gate releases on your golden set or run a model judge for you, so the CI gate in 7.4 and any judge stay in your own tooling. For the online half, Resolve Insights has an Accuracy & trust tab. Resolve is CloudThinker’s incident response module, and this tab is built only from your team’s verdicts, never from the agent scoring itself.Start your online evaluation
1
Grade a resolved incident
Go to Resolve → Investigation and open a resolved incident. Below the analysis, under Was Resolve’s diagnosis right?, click thumbs-up or thumbs-down. If it was wrong, pick a reason under What was wrong?. Only the workspace Admin role can grade, and the question appears once the incident has a verdict.Success state: the incident shows Marked accurate or Marked not accurate.
2
Read the scoreboard
In the Resolve sidebar, find the Analytics heading, click Insights, and choose the Accuracy & trust tab.Success state: your verdict counts toward Root-cause accuracy, and Your review coverage shows how many resolved incidents someone has reviewed.
Next steps
Chapter 8 · Observing the Observer
Trace, measure, and set SLOs on the agents themselves.
Appendix A · The POC Test Plan
Run an evaluation against your own incidents, scored scenario by scenario.