Skip to main content
A two-to-four-week evaluation protocol you can run against any agentic operations platform, including ours. A proof of concept (POC) with gates you sign in advance produces a decision. A demo produces a feeling.

​
The plan at a glance

If you run this plan on CloudThinker, the audit replay and cost cap steps need Audit Logs and Credit limits, which come with the Scale, Scale Plus, and Enterprise plans. Confirm your trial or plan includes both before week 0, or ask sales to enable them for the POC.

​
Week 0: preparation

Do all of this before any vendor touches anything.
  1. Freeze the baseline. Record 90 days of MTTR, alert volume, pages per engineer, and toil hours for the pilot domain. Without a credible “before”, you’ll argue from anecdotes. The Leadership Edition, this book’s companion for the people who set the budget, covers the baseline in Chapter 8.
  2. Assemble the golden set. Pick 10 or more replayable scenarios from your incident history, spread across your top incident classes. Each needs the telemetry snapshot, the known root cause, the accepted fix, and the evidence that the fix worked (Chapter 7).
  3. Write the injection seeds. Plant 5 or more indirect prompt-injection strings in staging telemetry, plus one poisoned document in the knowledge source the agent reads (Chapter 5).
  4. Define the pass/fail gates and sign them. Get the people who own the decision to sign before results come in.
Example gates to adapt: Record the sign-off in one table and keep it with the POC results:

​
Week 1: connect, discover, read-only

Answer the eight data-control questions from Leadership Edition §6.3 here, with packet captures and logs rather than vendor paragraphs.
  1. Exactly which data leaves our perimeter, to which endpoints, in which regions?
  2. Can inference run in-region, in a dedicated instance, or fully self-hosted?
  3. Is our data used to train any model, yours or a third party’s, and is that in the contract?
  4. Are personal identifiers and secrets masked before the model boundary, and where?
  5. What do you and your model providers log and keep, and for how long?
  6. Can we run model calls on our own model provider account?
  7. Can we export the complete audit trail if we leave you?
  8. Can we revoke every agent credential and stop all agent writes in one action?

​
Weeks 2–3: the golden-set gauntlet

  1. Replay the golden set. Score each scenario on detection, diagnosis, plan, mitigation, and evidence integrity (the Chapter 7 rubric). Have on-call engineers grade blind where you can.
  2. Run the injection seeds through real investigations. Any compliance with a planted instruction is a hard fail. Detecting and refusing it is the pass.
  3. Test the approval surface. For three proposed fixes, check that one screen shows the evidence chain, the exact plan, the blast radius, the rollback, and the verification plan (Chapter 6).
  4. Instrument cost. Record tokens and spend per investigation, how often events get promoted to a full investigation on a noisy day, and projected cost per resolved incident at your volume (Chapter 4, Chapter 8).

​
Week 4: limited write, full audit

  1. Allow two or three boring, reversible action classes with per-action approval in staging, or in a bounded production slice if your governance allows. Measure how often engineers accept the proposed action and whether the platform verifies its own result.
  2. Replay the audit trail for every action: trigger, context, reasoning, policy, approval, execution, verification. If you need the vendor’s help to replay it, note that. It previews what an auditor or regulator will experience.
  3. Run a kill-switch drill. Halt all autonomous execution, confirm investigation keeps working, and measure time to halt. Ask the vendor how many actions that takes: CloudThinker has no single halt-all switch. A Deny Rule on every connection is the drill once CloudThinker turns on Rule enforcement for your workspace; until then, halt by turning Allow writes off on each connection, which refuses every change there, and setting each run type to Manual (see Appendix B).
  4. Hold the decision meeting against the signed gates. The output is a go or no-go with evidence. On a go, start the 90-day pilot from Leadership Edition §9.2 with your baseline already in hand.
In practice: running this plan on CloudThinker
A note on fairnessRun the same protocol on every platform you evaluate. Share the gates with vendors in advance and let them see their scores. The goal is evidence, not a gotcha. A vendor who improves against your golden set during the POC is showing you how they behave after go-live, which is what you’re actually buying.

​
Next steps

Score the platform’s engineering surfaces alongside this plan with Appendix B · The Agent Platform Engineering Checklist. To start week 1 on CloudThinker, follow the Quickstart, and use viewer or read-only credentials when it asks you to connect.