> ## Documentation Index > Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt > Use this file to discover all available pages before exploring further. # Chapter 11 · The Failure Catalog — and Build vs. Buy for Engineers > Twelve production failure modes, each with its detection signal and its antidote — then the build-vs-buy question answered the way engineers actually face it. *Gartner’s canceled-40% is an aggregate; failures are specific. Twelve production failure modes, each with its detection signal and its antidote — then the build-vs-buy question answered the way engineers actually face it.* ## 11.1 The catalog | Failure mode | What it looks like | Detection signal | Antidote | | ------------------------------ | --------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | 1 · The infinite investigation | Agent probes forever, concludes nothing, bills plenty | Step counts and cost per investigation trending up; time-to-first-finding SLO burn | Step/token/time budgets; loop detector; escalate-on-budget with partial findings (Ch. 1) | | 2 · Context flood | Quality collapses as raw tool output swamps the window | Tokens per step ballooning; accuracy drop on long incidents | Size-capped tool results; compaction with pointers; retrieval re-ranking (Ch. 2–3) | | 3 · Confidently wrong | Fluent, evidenced-looking conclusion; wrong mechanism | Evidence-integrity failures; rejection reasons = “wrong diagnosis” | Evidence-traceable claims; layered eval scoring; human grading slice (Ch. 7) | | 4 · Validation theater | “Fixed” because the alert cleared; defect intact | Repeat-incident rate; diagnosis–mitigation gap in evals | Independent state-based verification; soak windows; separate mitigation scoring (Ch. 1, 7) | | 5 · Alert storm → cost storm | An incident wave wakes the resolver hundreds of times | Promotion-rate spike; spend anomaly at the seam | Sensing-side correlation and storm dampening; per-window resolver ceilings (Ch. 4) | | 6 · Injection incident | Agent acts on instructions found in telemetry or tooling | Injection-pattern detections; novel egress; anomalous tool sequences | Full Ch. 5 stack; red-team cadence; memory provenance | | 7 · Approval fatigue | Rubber-stamped approvals; gate becomes theater | Acceptance >95% with near-zero review latency | Graduate the class or improve the artifact; meter the gate (Ch. 6) | | 8 · Memory poisoning | One bad conclusion steers future incidents | Contradiction flags; repeat wrong diagnoses of one class | Provenance, decay, human-curated tier; hygiene cadence (Ch. 2, 9) | | 9 · Tool sprawl | Dozens of thin tools; wrong-tool choices; brittle prompts | Tool-choice error rate; context cost per step | Consolidate into intent-shaped capabilities; the fewer-better rule (Ch. 3) | | 10 · Invisible change | Behavior shifts; no diffable cause | Drift metrics move with no bundle release | Everything-is-an-artifact; pins; gated deploys (Ch. 9) | | 11 · Model-swap regression | Vendor upgrade quietly loses a class you had graduated | Per-class golden-set diff; step-count drift post-swap | Migration playbook; canary lanes; dual-run (Ch. 9) | | 12 · The frozen pilot | Advise-only forever; no graduation evidence; renewal dies | Autonomy level flat across quarters; no MTTR delta | Signed graduation criteria; boring-domain-first; Appendix A gates | Two meta-observations. First, every failure mode above is visible on a dashboard before it is visible in an outage — but only if Chapters 7 and 8 were built. Programs that skip evaluation and observability do not avoid these failures; they meet them without instruments. Second, the catalog is the interview: ask any platform vendor which of these twelve they have personally hit and what they shipped in response. Scar tissue is the only credential in a category this young. ## 11.2 Build vs. buy, the engineering edition Capable teams can absolutely build an operations agent — a weekend with a frontier model, a ReAct loop, and kubectl produces a demo that will impress your leadership. The decision is not whether you can build the agent; it is whether you should own the harness. Tally what this book actually specified: the context subsystem with discovery, budgets, memory, and provenance (Ch. 2); the tool and skill layer with risk metadata and contract tests (Ch. 3); orchestration with typed handoffs and two-engine cost control (Ch. 4); the injection defense stack with sandboxing, egress control, and tokenization (Ch. 5); policy-as-code, approval engineering, execution safety, and a regulator-grade audit schema (Ch. 6); a golden set, judges, and regression gates (Ch. 7); decision-graph observability (Ch. 8); and the day-2 operating discipline (Ch. 9). The demo is a fifth of the system, and it is the fun fifth. The other four-fifths is undifferentiated heavy lifting for most organizations — and the exact surface where the twelve failure modes live. The honest decision rule: build if agent operations is your product, or your constraints are so unusual that no platform’s trust architecture fits — and staff it as a product team with a roadmap, not a side quest. Buy if your differentiation lies elsewhere, and spend your engineering on what no vendor can ship: your golden set, your context curation, your policy design, and your operating discipline. Either way, hold the same bar — Appendix B is written to audit a vendor and to scope an internal build with equal precision, and Appendix A will pressure-test whichever path you choose against your own incidents. **KEY TAKEAWAY** Failures are specific, instrumented, and mostly self-inflicted at the harness layer. Whether you build or buy, you own the golden set, the policy, the context, and the discipline — and you should demand scar tissue, not slideware, from anyone who wants to own the rest.