Appendix D · Sources and How to Read the Numbers

Every figure in this edition comes from one of the primary sources below. Use this page to check a number before you repeat it, and to decide how far it should move your own evaluation.

How to read the numbers

Each number in the book belongs to one class. The class tells you how much to trust it.

Class	Example in this book	How to weigh it
Independent benchmark	ITBench, ITBench-AA	Strongest public signal for models. It measures a model under the benchmark’s harness, not a platform under yours.
Research paper or preprint	SREGym, AIOpsLab	Good for failure modes and method. Check the setup before you reuse a percentage.
Vendor first-party	Microsoft’s Azure SRE Agent posts, Anthropic’s multi-agent write-up	Useful engineering lessons. Treat outcome numbers as the vendor’s own claim.
Standard or specification	OWASP Top 10 for LLM Applications, OpenTelemetry GenAI conventions, Model Context Protocol	Defines terms and shared schemas. Not evidence that any product works.

Four rules apply to every figure:

Provenance is stated. The chapter names the source and its class next to the number. If a figure has no source, treat it as a guess.
Ranges beat points. “About 69–88%” is more honest than one precise-looking number. Be wary of a single decimal-point figure with no method behind it.
Your baseline beats every benchmark. A public score tells you where the field is. Your own incidents, replayed under your own harness, tell you whether a platform works for you.
Scores don’t transfer. A benchmark scores a model under its harness. Your proof of concept scores a platform under yours. Neither predicts the other, which is why Appendix A exists.

Sources by chapter

Source	Class	What this book uses from it	Cited in
IBM Research, ITBench (ICML 2025, github.com/itbench-hub/ITBench)	Independent benchmark	Open scenarios across SRE, compliance, and FinOps. Agents on state-of-the-art models resolve roughly 11–14% of SRE scenarios, about one in seven.	Introduction, Chapter 7
Artificial Analysis with IBM, ITBench-AA (May 2026)	Independent benchmark	Frontier models diagnosing Kubernetes incidents. Every evaluated model scored below 50% on the headline precision metric. Turn, token, and cost curves are published.	Chapter 7
SREGym (2026, arXiv)	Research paper or preprint	Reward hacking against fault injectors, alert-clearing counted as false success, and the gap between diagnosis and fix: mitigation succeeds about 69–88% of the time even with the right root cause.	Introduction, Chapter 7
Microsoft Research, AIOpsLab (github.com/microsoft/AIOpsLab)	Research paper or preprint	The live-environment evaluation pattern that the book’s replay guidance builds on.	Chapter 7
Microsoft, Azure SRE Agent engineering posts	Vendor first-party	The move from more than a hundred tools across narrow agents to a few core tools and more generalist agents.	Chapter 3, Chapter 4
AWS DevOps Agent launch and adoption guidance (2026)	Vendor first-party	Start recommendation-only and scope to one service. Pilot outcomes are the vendor’s own reports.	Background for the read-only start in Appendix A
Anthropic, multi-agent research system write-up (anthropic.com/engineering/multi-agent-research-system)	Vendor first-party	Orchestrator-worker designs with parallel helpers gain quality at a materially higher token cost.	Chapter 4
Model Context Protocol specification (modelcontextprotocol.io/specification)	Standard or specification	The protocol layer between agents and tools.	Chapter 2, Chapter 3, Chapter 5
OpenTelemetry GenAI semantic conventions (opentelemetry.io/docs/specs/semconv/gen-ai)	Standard or specification (in development)	The shared schema for model calls, agent and tool spans, token usage, and cost. Multi-agent conventions are still in draft.	Chapter 8, Appendix B
OWASP Top 10 for LLM Applications (LLM01: Prompt Injection, genai.owasp.org)	Standard or specification	Prompt injection as the top risk for LLM applications, and the advice to layer mitigations instead of relying on one filter.	Chapter 5
CrowdStrike prompt-injection technique taxonomy	Vendor first-party	The catalog of well over a hundred distinct injection techniques behind the Chapter 5 threat model.	Chapter 5

We cut any figure we couldn’t source to one of these classes. If you find a number in this book with no matching row here, treat it as unsourced.

Check a figure before you use it

Run these checks before a number goes into a business case or a vendor scorecard.

Find the class

Match the figure to its row above. A vendor first-party number needs more checking than an independent benchmark.Success state: you can name the source and its class in one line, for example “ITBench, independent benchmark, ICML 2025”.

Read the harness

Note what the score measured: which model, which tools, which scenarios, and what counted as success. SREGym shows that “the alert cleared” and “the defect is fixed” are different outcomes.Success state: you know whether the figure measures detection, diagnosis, or a verified fix.

Replace it with your own number

Replay 10 or more of your own incidents through the platform using the golden set from Appendix A. Score them with the Chapter 7 rubric.Success state: your business case cites your own replay scores, with public benchmarks only as context.

Learn

Workshops

Scenarios

Appendix D · Sources and How to Read the Numbers

How to read the numbers

Sources by chapter

Check a figure before you use it

Next steps

Appendix A · The POC Test Plan

Chapter 7 · Evaluating Agents

​How to read the numbers

​Sources by chapter

​Check a figure before you use it

​Next steps

Appendix A · The POC Test Plan

Chapter 7 · Evaluating Agents

How to read the numbers

Sources by chapter

Check a figure before you use it

Next steps