> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Chapter 8 · Observing the Observer

> The agent fleet is a production service — one whose failures are decisions, not exceptions.

*You would never run a production service without traces, metrics, and SLOs. The agent fleet is a production service — one whose failures are decisions, not exceptions.*

## 8.1 Why agent observability is different

An agent can fail gracefully by every classical signal — no exception, fast response, HTTP 200 — while being confidently wrong. Observability for agents therefore has to capture the decision graph, not just the I/O boundary: what the agent saw, what it considered, which tools it called with which arguments, what came back, what it concluded, and what it spent. The industry answer is converging on OpenTelemetry’s GenAI semantic conventions — a standard schema for model calls, agent and tool spans, token usage, and cost attributes — now supported natively by major observability vendors and emitted by the mainstream agent frameworks, with multi-agent conventions (tasks, handoffs, memory operations) actively developing in the OTel GenAI working group. The strategic point for buyers and builders alike: agent telemetry in an open schema is portability and auditability; agent telemetry in a proprietary blob is lock-in at the layer you can least afford it.

## 8.2 The trace anatomy

One incident, one trace. Within it: a root span per task; child spans per agent invocation (orchestrator and each specialist); grandchildren per step — model call spans (model, tokens in/out, latency, cost) and tool spans (tool, arguments with secrets tokenized, result size, duration, errors); plus events for the semantic milestones: hypothesis formed, evidence attached, policy evaluated, approval requested and resolved, action executed, verification passed or failed, escalation. Link the trace to the Chapter 4 incident record and the Chapter 6 audit schema — three views of one truth, sharing IDs, so an engineer can pivot from “this felt slow” to “step 23 pulled 400KB of logs and re-sent them four times” in one query.

> *Figure 4 — One incident as one trace: agent spans, step spans, tool and model calls, and semantic events, tied by shared IDs to the incident record and audit trail.*

## 8.3 Metrics and SLOs for the fleet

The Field Guide’s eight program KPIs measure outcomes; the fleet needs an engineering layer beneath them:

* **Quality:** diagnosis accuracy on the golden set (per class), online acceptance rate, rollback/intervention rate, evidence-integrity failures caught. These are the agent’s error budget.

* **Latency:** time-to-first-finding, time-to-plan, end-to-end resolution time — percentiles, split by incident class and by engine (sensing promotion lag vs. resolver duration).

* **Cost:** tokens and spend per step, per investigation, per resolved incident; promotion rate at the two-engine seam; spend by model and by agent. Alert on cost anomalies exactly as you would on cloud-spend anomalies — an agent loop is a cost incident with excellent prose.

* **Behavior drift:** tool-call distribution per incident class, step counts, escalation rate, refusal rate. Drift in these — after a model swap, a prompt change, or silently — is your early-warning system, and it feeds the Chapter 7 regression process.

* **Security signals:** injection-pattern detections in context, novel egress destinations, policy-denial spikes, anomalous tool sequences (Chapter 5’s detection layer, materialized as metrics).

Set SLOs on the ones that carry the program: e.g., time-to-first-finding for promoted P1s, diagnosis accuracy per graduated class, cost per resolved incident. An agent fleet with SLOs and error budgets can earn autonomy quantitatively — the trust ladder’s graduation criteria stop being vibes and start being burn rates.

<Info>
  **IN PRACTICE — WHO WATCHES THE WATCHERS**

  This chapter is also the honest answer to a fair objection: “we are deploying an autonomous system to reduce toil — who operates it?” The answer is: instrumented software operated by the same discipline it brings to your estate. CloudThinker exposes its own decision traces, cost lines, and policy events to the customer — because a platform asking to be trusted on your production should be observable on the same terms as your production. Ask every vendor to show you their traces of their agents, in a schema you could export.
</Info>

<Tip>
  **KEY TAKEAWAY**

  Instrument decisions, not just calls: one trace per incident with model, tool, and semantic-event spans in an open schema; fleet metrics across quality, latency, cost, drift, and security; SLOs that turn trust-ladder graduation into arithmetic. If you cannot see what the agent decided and spent, you are not supervising it — you are hoping.
</Tip>