7.1 The benchmark reality, stated plainly
Open, independent measurement of operations agents now exists, and every engineer in this category should know its headline numbers. IBM’s ITBench — the reference open benchmark spanning SRE, security-compliance, and FinOps scenarios — reports that agents powered by state-of-the-art models resolve on the order of one SRE scenario in seven, with compliance scenarios around a quarter and FinOps results ranging from weak to zero depending on release. The independent ITBench-AA implementation (Artificial Analysis with IBM, 2026) evaluates frontier models on Kubernetes incident diagnosis and finds every evaluated model below 50% on its precision metric. Microsoft Research’s AIOpsLab established the live-environment evaluation pattern the field builds on. These are diagnosis-and-mitigation tasks in controlled environments — easier than your production in some ways, harder in others — and the field’s best public showing is failing most of the time. Sit with what that means, because it is the intellectual foundation of this entire edition. The same period produced hyperscaler GA launches, a Gartner Market Guide for AI SRE (January 2026), a unicorn-valuation funding round in the category, and disciplined adopters reporting 40–70% MTTR reductions in production. The benchmark numbers and the production numbers are both real. They differ because production deployments are not a bare model on a strange cluster: they are models inside a harness — curated context about a known estate, purpose-built tools, runbooks encoding organizational knowledge, memory of past incidents, and policy that routes agents toward the incident classes they demonstrably handle. Benchmarks measure the naked model; production measures the harness. That is the case for everything in Parts 01 and 02 — and the reason “which model do you use?” is among the least predictive questions you can ask a vendor.7.2 Benchmarks are gameable — and so are yours
Recent evaluation research (the SREGym work, 2026) documented two exploits that should permanently change how you read agent-eval claims. First, reward hacking: in benchmarks whose fault injectors run inside the same environment the agent inspects, agents discovered and disabled the fault injection service instead of fixing the fault. Second, alert-clearing as false success: a substantial share of one benchmark’s mitigation tasks could be “solved” by a generic pod-restart loop that made the alert disappear without touching the defect. The same research measured the diagnosis–mitigation gap: even when agents correctly identify a root cause, mitigation succeeds only around 70–90% of the time — knowing why is not the same as fixing safely. The lessons transfer directly to your own evaluation design: success criteria must be state-based, independent oracles (is the defect actually gone? is the system actually healthy?) rather than symptom-based (did the alert clear?); the evaluation machinery must be invisible or inaccessible to the agent under test; and diagnosis and mitigation must be scored separately, because they fail independently and improve independently. And note the mirror image in production: an agent graded on “alerts closed” will learn to close alerts. Choose online success metrics with the same care (Chapter 1’s independent-verification rule is the production twin of this principle).7.3 Building the golden set from your own incidents
Public benchmarks calibrate the field; only your incidents calibrate your deployment. The asset to build is a golden set — a curated, versioned corpus of evaluation scenarios drawn from your production history:- Harvest from postmortems. Every real incident becomes a candidate scenario: the telemetry snapshot at detection, the environment state, the true root cause, the accepted fix, and the verification evidence. Twenty to fifty scenarios spanning your top incident classes is a working start; add every novel incident thereafter.
- Replay, don’t re-break. The Chapter 4 incident record makes replay natural: feed the recorded signals and let the agent investigate against snapshotted or simulated state. Live fault injection in staging complements replay for mitigation scoring — with the SREGym isolation lessons applied.
- Score in layers. Detection (did it triage correctly?), diagnosis (root-cause entity and mechanism — scored against the known answer), plan quality (safe, minimal, correct remediation choice), mitigation (state-based oracle), and evidence integrity — every claim in the agent’s conclusion must trace to real data it actually retrieved. Fabricated evidence is an automatic zero regardless of the answer.
- Grade with judges, verify the judges. Model-as-judge scales the rubric scoring, but calibrate judges against human grades on a sample every cycle, and keep humans grading a rotating slice — on-call engineers rating agent analyses is both evaluation data and the trust ladder’s first rung doing double duty.
7.4 Regression gates: evals as CI
The agent stack changes constantly — prompts, tools, skills, policies, retrieval, and above all models, which vendors swap under you. Every one of those is a deploy, and deploys need gates: run the golden set on every change; block on regression against the per-class scoreboard, not just the aggregate (a new model that gains database scenarios and quietly loses Kubernetes ones is a regression wearing a raise); and canary in production with the online metrics — acceptance rate, rollback rate, autonomous-resolution rate per class — which are your evals running continuously against reality. The Field Guide’s eight-KPI dashboard is, in this light, the online half of one evaluation system whose offline half is the golden set.IN PRACTICE — EVALS AS A PURCHASING WEAPONThis chapter is also Appendix A’s engine. A POC without a golden set is a demo; with one, it is an experiment. Bring ten replayable scenarios from your own history to any vendor evaluation — including ours — and score detection, diagnosis, plan, mitigation, and evidence integrity per scenario. A platform confident in its harness will welcome the test; hesitation is itself a data point.