8.1 What improves, and in what order
Gains arrive in a predictable order. Plan your reporting around it, so nobody expects autonomous fixes in week two.
How large each gain gets depends on your data quality and how far you’ve climbed the trust ladder (§7.3).
The strongest published figures are vendor numbers. AWS reports DevOps Agent preview customers seeing up to 75% lower MTTR, 80% faster investigations, and 94% root-cause accuracy. In the same announcement, Western Governors University (WGU) describes an estimated two-hour resolution cut to 28 minutes.
Treat these as claims to test in your own pilot, not results to plan around.
8.2 The operating dashboard
Run the program on eight KPIs. Review them monthly with the on-call rotation.
The autonomy levels (L0 to L4) are defined in §3.2.
8.3 Building the business case
The return side has three lines. Use your own inputs for each.
The cost side covers the platform subscription, model usage, and the engineering time to integrate and govern the system. Include the autonomy-policy owner’s time.
Test the case with one rule: if it only works with the downtime line, you picked the wrong pilot domain. Choose a domain where toil savings carry the case, and treat avoided downtime as upside.
8.4 The unit economics of an agent
Most of an agentic program’s running cost is model inference. Most inference cost depends on where the expensive reasoning runs. So keep cheap perception always on, and save frontier reasoning for the few signals that need it. §4.3 calls this two-tier sensing. Read one incident as a cost object:
Deep resolution costs the most per run but fires rarely, so the expensive tier stays mostly idle while every signal is still covered. We don’t publish typical ratios, because they vary too much between environments. Measure three numbers in your own: signals per day, the share that escalates to deep resolution, and the cost of one deep run. Multiply them to get your daily deep-resolution spend.
That’s why cost per resolved incident and cost per covered service are on the dashboard. An architecture that runs frontier reasoning on every noisy signal erases its own return before the first renewal. You won’t see it happen unless you track the per-incident number from the start.
When you compare vendors, ask how pricing tracks value. An outcome-based option, such as a fee tied to verified savings, removes the vendor’s incentive to run up inference for its own sake. Whatever the structure, the figure in your business case should be the one you measured against your own 90-day baseline (§9.2).
8.5 Testing an agent before you trust it
Earlier chapters describe agentic operations as self-verifying. This section shows what that check looks like, and how to catch a confidently wrong fix before it reaches production. An evaluation harness has four parts:
Agents aren’t deterministic. The same scenario can produce different behavior on different runs. So run each scenario many times and score the spread of outcomes. A fix that’s right 70% of the time is a different risk from one that’s right 99% of the time, and only repeated runs show which one you have.
A concrete check, for a database connection-exhaustion scenario. The harness asserts that the agent:
- Names connection-pool exhaustion as the root cause.
- Restores the correct connection limit, instead of only restarting the service.
- Confirms the connection count returns to baseline and stays there.
8.6 Where to read these numbers in CloudThinker
If you’re running a pilot on CloudThinker, these pages report most of the dashboard. Track on-call load and rollback rate in your paging and change tools.
Root-cause accuracy only means something once your team reviews enough incidents. Check Your review coverage on the same tab before you report it.
Next: Chapter 9 · The Implementation Roadmap turns these measures into a 90-day pilot with exit criteria for each phase.