Appendix B · The Agent Platform Engineering Checklist

38 items across eight surfaces, each with the evidence to ask for. Use the same bar to grade a vendor in an evaluation or to scope and review an internal build.

How to score

Score each item on evidence you have seen, not on a slide or a roadmap. If the team can’t show it running, it’s a 0 or a 1.

Score	Meaning	Typical evidence
0	Absent	Not built, or only on the roadmap
1	Partial	Works for some agents, environments or action classes, or only as a manual process
2	Solid	Works everywhere it should, and you saw it in a demo, a trace or your own test

The maximum is 76. Read the total per surface, not only the grand total. A platform that scores 2 everywhere except Security and Policy and approvals isn’t ready for write access, whatever the sum says. As a floor, don’t grant write access while any Security or Policy and approvals item scores 0.

Surface	Items	Max
Loop and economics	5	10
Context	5	10
Tools and skills	5	10
Security	6	12
Policy and approvals	5	10
Evaluation	4	8
Observability	4	8
Day 2	4	8

For an internal build, read Ask to see as the artifact your team must produce. Items scored 1 are the ones to test. The Appendix A POC test plan turns them into a 0 or a 2 on your own incidents.

Loop and economics

From chapters 1 and 4.

Item	Ask to see
Per-task budgets for steps, tokens, time and spend, with loop detection	What happens when an investigation hits its budget or starts repeating itself
Deterministic rails: the model selects typed actions, code executes them	The code path a write action takes, instead of a shell string the model composed
Two-engine separation (cheap sensing, expensive resolving) with a measurable promotion seam	The promotion rate: what fraction of signals wake the resolver
Cost per resolved incident, reported at a reference scale	That number for an estate the size of yours
Independent, state-based verification with soak windows and staged rollback	How a fix is proven, beyond “the alert cleared”

Context

From chapter 2.

Item	Ask to see
Automated discovery that produces a system map you can review	The map it built of your estate in the first week
Change-stream ingestion that is time-ordered and attributable	Which deploy or config change it ties to an incident, and who made it
Retrieval with re-ranking, context budgets and traceable compaction	A compacted trajectory with pointers back to the raw evidence
Memory with provenance, decay, contradiction flags and a human-curated tier	Where a remembered fact came from, and how a person corrects it
Knowledge and topology refresh with staleness handling	What it does when a runbook or topology entry is out of date

Tools and skills

From chapter 3.

Item	Ask to see
Intent-shaped tools, with read and write paths separated structurally	Proof that a read-only agent can’t reach a write tool at all
Tool results that are size-capped and structured, with pointers to full data	A large log query result as the model actually saw it
Skills versioned in source control, with owners, tests and changelogs	The history of one skill
Skill metadata that declares access class, reversibility and blast radius	Where that metadata lives, and what reads it
An MCP supply chain with pinned versions, allowlists, contract tests and an egress map	The list of MCP servers in use, and how a new one gets approved

Security

From chapter 5.

Item	Ask to see
Per-agent identity, short-lived scoped credentials and environment separation	Which credential an agent used for one action, and when it expired
Taint-tagged context, with untrusted material delimited	How a log line is marked differently from an operator’s instruction
Sandboxed execution (microVM class: each command runs in its own lightweight virtual machine with its own kernel, not a shared container) with no ambient credentials	What an executed command can reach on the host and the network
Default-deny egress with allowlists and alerts on new destinations	The alert raised when an agent calls a domain it never called before
PII and secret tokenization before every model boundary, with guarded de-tokenization	A prompt as sent to the model, with account numbers replaced by tokens
Injection detection in telemetry, plus a standing red-team cadence	Results from the last red-team run, including the injection seeds from Appendix A

Policy and approvals

From chapter 6.

Item	Ask to see
Two-axis action classification (reversibility and blast radius), derived from skill metadata	How a new skill gets its class without a person assigning it
Versioned policy-as-code with a review workflow, per environment and per agent	A policy change going through review
One-screen approval requests, with structured capture of rejections	An approval request, and what gets recorded when someone says no
Pre-flight checks, staged application, circuit breakers and rate limits	What halts a fleet that keeps repeating a failing action
Pre-staged rollback on every consequential action, and a kill switch that has been drilled	The last kill-switch drill, and how long it took

Evaluation

From chapter 7.

Item	Ask to see
A golden set harvested from real incidents, plus a replay harness	A replay of one past incident
Layered scoring that includes evidence integrity, with state-based mitigation oracles	A scored run where every claim traces to retrieved data
Judges calibrated against human grades, plus a standing human grading slice	The agreement rate between the judge model and human graders
Regression gates on every prompt, tool, policy and model change, per incident class	A change the gate blocked

Observability

From chapter 8.

Item	Ask to see
One trace per incident with agent, step, tool and model spans in an open schema (OpenTelemetry GenAI conventions)	The trace for one incident, opened in your own tooling
Fleet metrics for quality, latency, cost, drift and security signals	The fleet dashboard
SLOs tied to the criteria for granting an agent more autonomy	Which SLO decides when an action class moves from approval to automatic
Traces and audit data you can export	An export, in a format your tools can read

Day 2

From chapter 9.

Item	Ask to see
Versioned bundles (prompts, tools, skills, policy, model pins), so no change is invisible	The diff between two deployed bundles
A model migration playbook with per-class diffs and canary lanes	The last model upgrade, and how it was rolled out
A memory hygiene cadence with a named owner	Who reviews agent memory, and how often
Blameless agent postmortems that feed the golden set	A postmortem, and the scenario it added

Check CloudThinker against the list

If you’re scoring CloudThinker, these pages show where items live in the product. Score them on what you see in your own workspace, not on this table.

Checklist item	Where to look
Policy that decides which actions need approval	Rules decide whether an agent write runs on its own, waits for a person, or never runs. A built-in Always blocked list refuses some calls outright, and no Rule or person can override it.
One-screen approval requests	Approvers covers who answers each approval card, the two-person rule, and break-glass.
PII and secret tokenization	Custom guardrails sets which identifiers agents see only as placeholders.
Skills you can review	Skill format shows the `SKILL.md` file you upload: frontmatter plus Markdown instructions. Risk is checked on each call by Approvals, not read from the skill. CloudThinker doesn’t version skills: Replace deletes the old one, so keep each skill zip in Git to roll back (§3.2).
Exportable audit data	Audit logs filters events by actor, category, severity and date, and exports them as CSV. Approvals → Activity exports what each Rule decided as CSV.
Per-task and spend budgets	Credit limits set a monthly budget per workspace, with Throttle or Notify only (Scale, Scale Plus, and Enterprise plans).
Two-engine separation	Pulse groups alerts and drops noise. Investigations run only on incidents.
Automated discovery	Discovery runs once when a connection first reaches Connected. Results fill the Discovery section of the connection’s detail panel.
Memory with a human-curated tier	Incident memory records symptom, cause, and fix. Admins can edit or delete any memory item.
MCP supply chain	Your own MCP server becomes a connection, with its tools sorted into reads and writes.
Sandboxed execution	Commands run in the CloudThinker managed sandbox, isolated from your machines. Shell commands on a Worker run on your machine. See the security overview.
Kill switch	No single halt-all switch. The control that works today is Allow writes: turn it off on a connection and every change on that connection is refused. Setting each run type to Manual sends its writes to a person, but that alone isn’t a halt, because a chat can pick its own mode in the composer. Turn off On duty in Resolve to stop automatic investigations, and switch off an automation’s triggers to stop its runs. Once CloudThinker turns on Rule enforcement for your workspace, Rules add a broader halt. A Deny Rule for every operation, on every connection, with Never allow break-glass on, refuses every agent write in every mode while reads keep running.

Not covered today: OpenTelemetry trace export (audit export is CSV) and a golden-set replay harness. Score those items on your own build or on what the vendor shows you.

Use it next

Score the platform, then test the weak surfaces on your own incidents with the Appendix A POC test plan. Terms used here are defined in the Appendix C glossary.

Learn

Workshops

Scenarios

Appendix B · The Agent Platform Engineering Checklist

How to score

Loop and economics

Context

Tools and skills

Security

Policy and approvals

Evaluation

Observability

Day 2

Check CloudThinker against the list

Use it next

​How to score

​Loop and economics

​Context

​Tools and skills

​Security

​Policy and approvals

​Evaluation

​Observability

​Day 2

​Check CloudThinker against the list

​Use it next

How to score

Loop and economics

Context

Tools and skills

Security

Policy and approvals

Evaluation

Observability

Day 2

Check CloudThinker against the list

Use it next