8.1 Why agent observability is different
An agent can look healthy on every classic signal (no exception, fast response, HTTP 200) and still be confidently wrong. So request and response logs aren’t enough. You need the decision graph: what the agent saw, what it considered, which tools it called with which arguments, what came back, what it concluded, and what it spent. The industry is converging on the OpenTelemetry GenAI semantic conventions, a shared schema for model calls, agent and tool spans, token usage, and cost. Major observability vendors ingest it, and mainstream agent frameworks emit it. Conventions for multi-agent work (tasks, handoffs, memory operations) are still being drafted in the OpenTelemetry GenAI working group. The buying rule follows from that. Agent telemetry in an open schema means you can move it and audit it. Telemetry in a proprietary format locks you in at the layer you can least afford. Hold CloudThinker to the same rule. Today it doesn’t export OpenTelemetry GenAI traces to your own backend. What you can take out is the Audit Logs CSV (every agent and human action), the Approvals → Activity CSV (what each Rule decided), and the views on the Usage page. The table at the end of this chapter says where each one lives.8.2 The trace anatomy
Record one trace per incident. Nest it like this:
Give the trace the same IDs as the incident record (Chapter 4) and the audit record (Chapter 6). That gives you three views of one event, joined by shared IDs. An engineer can then go from “this felt slow” to “step 23 pulled 400 KB of logs and re-sent them four times” in one query.
8.3 Metrics and SLOs for the fleet
The Field Guide’s program KPIs (§8) measure outcomes: MTTD and MTTR, autonomous resolution rate, recommendation acceptance rate, rollback and intervention rate, repeat-incident rate, coverage ratio, on-call load, and agent unit cost. Underneath them, the fleet needs an engineering layer. Two of the rows below mention “engines”. These are the cheap sensing layer that watches every signal and the resolver that does deep reasoning on the few signals it promotes (Chapter 4).
Don’t put an SLO on everything. Pick the few that carry the program. These example targets are starting points to tune against your own baseline:
With SLOs and error budgets, the trust ladder (Field Guide §7.3) stops being a judgment call. A class graduates when it stays inside its budget, and drops back when it burns through it.
IN PRACTICE: WHO WATCHES THE AGENTS?A fair objection to any agent platform: “we’re deploying it to cut toil, so who operates it?” The answer should be instrumented software, held to the same discipline it brings to your estate. Here is where CloudThinker shows you its own agents:
See Audit Logs, Usage, Credit limits, and Resolve Insights. Admin Settings is the organization menu, and only organization Owners and Admins see it. Ask every vendor, us included, to show you these records for their own agents and how you get them out.
Set up your first agent controls
1
Cap spend per workspace
Go to Admin Settings → Credit limits. Under When a workspace hits its monthly limit, pick Throttle for sandbox workspaces or Notify only for production. Set Default monthly limit per workspace about 20% above a typical workspace’s monthly spend, check both alerts under Alert the organization owner at, and click Save.Success state: the save bar disappears, and Per-workspace limits shows each workspace’s usage against its limit.
2
Pull a week of agent actions
Go to Admin Settings → Audit Logs. Set Actor Type to Agent and the date range to Last 7 days, then click Export CSV.Success state: your browser downloads
audit-trail.csv, with one row per agent action and an agent_name column. Rename it with its date range, since every export has the same name.Next steps
Chapter 9 · AgentOps: Running the Fleet on Day 2
Version changes, migrate models, keep memory clean, and run postmortems for agents.
Audit Logs
Filter, inspect, and export every action agents and people take.