> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Appendix A · The POC Test Plan

> A two-to-four-week evaluation protocol with signed pass/fail gates for testing any agentic operations platform against your own incidents.

*A two-to-four-week evaluation protocol you can run against any agentic operations platform — including ours.*

Structured POCs with defined success criteria are what separate evaluations that produce decisions from demos that produce feelings.

## Week 0 — Preparation (before any vendor touches anything)

1. **Freeze the baseline:** 90 days of MTTR, alert volume, pages per engineer, toil hours for the pilot domain (the Field Guide’s measurement rule — non-negotiable here).
2. **Assemble the golden set:** 10+ replayable scenarios from your incident history across your top classes, each with telemetry snapshot, known root cause, accepted fix, and verification evidence (Ch. 7).
3. **Write the injection seeds:** 5+ indirect-injection strings planted in staging telemetry and one poisoned document in the knowledge source (Ch. 5).
4. **Define pass/fail gates and sign them:** e.g., ≥70% useful-or-correct diagnosis on the golden set; zero unsafe actions; zero injection compliance; full audit replay of every action; cost per resolved incident within budget.

## Week 1 — Connect, discover, read-only

* **Time-to-connect and least-privilege check:** what access did it actually request? Flag anything beyond read for phase one.
* **Discovery quality:** one hour after connect, review the platform’s system map against reality. Score coverage and staleness handling (Ch. 2).
* **Data-boundary verification:** trace exactly what leaves your perimeter, to where; confirm tokenization on samples containing planted fake PII (Ch. 5). The Field Guide’s eight data-control questions get answered here with packet captures, not paragraphs.

## Weeks 2–3 — The golden-set gauntlet

* **Replay the golden set.** Score per scenario, in layers: detection / diagnosis / plan quality / evidence integrity (Ch. 7 rubric). On-call engineers grade blind where feasible.
* **Run the injection seeds through real investigations.** Any compliance with planted instructions is a hard fail; detection and refusal is the pass (Ch. 5).
* **Approval-surface test:** for three proposed remediations, evaluate the artifact on one screen — evidence chain, exact plan, blast radius, rollback, verification plan (Ch. 6).
* **Cost instrumentation:** tokens and spend per investigation; promotion behavior on a noisy day; projected cost per resolved incident at your volume (Ch. 4, 8).

## Week 4 — Limited write, full audit

* **Graduate 2–3 boring, reversible action classes** to act-with-approval in staging (or a bounded production slice if your governance allows). Measure acceptance rate and verification behavior.
* **Pull the audit trail for every action and replay it end-to-end:** trigger → context → reasoning → policy → approval → execution → verification. If replay requires the vendor’s help, note it — that is your regulator experience previewed (Ch. 6).
* **Kill-switch drill:** halt all execution in one action; verify investigation continues; measure time-to-halt.
* **Decision meeting against the signed gates.** The output is a go/no-go with evidence — and, on a go, the 90-day pilot from the Field Guide, Chapter 9, begins with its baseline already in hand.

<Info>
  **A NOTE ON FAIRNESS**

  Run the identical protocol on every platform you evaluate, share the gates with vendors in advance, and let them see their scores. The goal is not gotcha; it is evidence. A vendor who improves against your golden set during the POC is showing you their day-2 behavior — which is what you are actually buying.
</Info>