Prove It Day 02: Investigate

Day 02 of Prove It. By the end you’ll have connected a read-only agent to your own Kubernetes cluster, held at least two explanations for a live incident, run the query that rules one out, and handed in one incident report per team. The live session ran on Sep 12, 2026. You can repeat the lab on your own on any cluster you control. Day 01 asked what exists in an account. Day 02 asks what caused this, right now. Access doesn’t change: credentials stay with the owner, and the agent still can’t change anything.

Use this page with CloudThinker. Paste this page’s URL into a chat and try:

“Explain the noise reduction section of this page like I’m new to on-call.”
“Quiz me on the difference between a trigger, a root cause and a contributing factor.”
“Help me outline my Day 02 incident report using the five blocks on this page.”

Rules of the room

Four blocks. Learn 35 minutes, a keynote 25, Practice 30 with no break, Share 20.
Your Kubernetes cluster is connected read-only. Use a staging cluster if you can. The agent can inspect. It can’t restart, scale, deploy or edit anything, and it can’t read secrets.
Credentials stay with the owner. The cluster owner connects it on their own screen with a read-only kubeconfig (get, list, watch).
The agent recommends. People decide. No fix is applied today.
Redact. Account IDs, ARNs, endpoints, hostnames and customer data come out of the report before it’s shared or shown on screen.
One report file per team, submitted before Share starts.

Learn: 35 minutes

Day 01 gave the agent a boundary. Day 02 starts one step earlier, with the boundary most teams never set: which signals deserve attention at all. Ideas 1 to 4 are background on alert noise. Ideas 5 to 8 are the ones you use in the lab.

1. You can’t investigate what you can’t rank

A bad night doesn’t hand you one incident. It hands you a stream: an access alert on a production role, a burst of AccessDenied lines in the audit log in the same minute, three near-identical alerts from three services, and five pages open for one problem. The investigation isn’t the hard part at that hour. Choosing what to investigate is. Every team already has an answer to alert volume, and each one costs something:

The usual fix	What it costs
Hire more on-call	Burnout, and the same volume again next quarter
Raise the thresholds	The one critical alert that mattered is now under the line
Mute the noisy channel	A blind spot nobody owns or remembers creating
Add another dashboard	One more screen nobody reads at 2am

Each is a decision about what you agree to lose, usually made once, informally, and never written down. That’s the Day 01 governance gap again, pointed at the signal instead of the agent.

2. Noise dies in ordered layers

Noise reduction is a pipeline: ingest, suppress, group, then classify and route. Suppression isn’t one filter. It’s a series of checks run in a fixed order, and the first one that matches wins, so the order is the policy. Common layers, whatever tool you use:

Layer	What it drops
Deduplication	Exact repeats of the same event
Rate limit	Bursts above a ceiling for one source
Snooze	Signals a person muted on purpose
Prior verdict	A pattern already judged to be noise
Flapping	A metric bouncing across a threshold, firing again and again for one condition
Cascade	Downstream effects of a root signal you already know about

Cascade suppression removes the most, and it will hide a real outage if the root signal it collapsed onto was wrong. Severity is a guess too: the severity a tool stamps on an event is its opinion, and you’re allowed to re-grade it. In CloudThinker, this pipeline is Pulse. Pulse runs four checks in order (dedup, rate limit, snooze, verdict), then classifies what survives and drops signals it judges not actionable. The verdict check is the one that learns: when CloudThinker confirms a pattern is noise while investigating, repeats of that pattern are suppressed. Details: Noise suppression.

3. Every suppression rule needs an escape hatch

Silence has to expire and has to be overridable, or it turns into a blind spot. Pulse’s verdict check shows both properties:

Property	How it behaves in Pulse
The window grows	A pattern confirmed as noise is quiet for 6 hours. Confirmed again, 24 hours. Again, 7 days.
The bypass wins	Critical and high events always pass the rate limit and verdict checks. They can still be deduplicated, and a snooze a person set still applies.

A rule with no bypass will eventually hide the outage it was written for. An expiring window forces the system to re-confirm that a pattern is still noise. A permanent mute is a decision nobody revisits.

4. Reduction is what makes the agent affordable

An investigator agent is the expensive step. Picture one alarm that flaps all morning, where every fire opens its own root cause analysis: many investigations of one problem, many answers to the same question, and a bill for each. Point an agent at a raw alert stream and that’s what you buy. Three gates to apply before a group of signals is worth an investigation, with or without an agent:

Gate	Why
At least one actionable signal	A group made only of suppressed signals never pays for an investigation
Something new, not a repeat	Volume alone shouldn’t re-open the question. Novelty should.
No recent investigation of the same group	A cooldown stops the same question being asked twice in an hour

In CloudThinker, related alerts that share a likely cause are grouped under one parent incident with a single investigation. See Investigation and RCA.

5. Split the question, not the work

A good investigation gathers context, then splits into hypotheses: explanations stated precisely enough that a query could contradict them. Each one is tested against the same evidence, and one verdict is written at the end. The skill to take home is the question each split is looking for: the query whose result differs depending on which hypothesis is true. That’s the query that rules one out. Every other query confirms what you already believe. In CloudThinker, the investigation can hand focused checks to temporary subagents (short-lived helper agents), and the incident’s Investigation timeline records each explanation tested and whether it was confirmed or ruled out. To see it, go to Resolve → Investigation, select an incident, and scroll to Investigation timeline. Read-only also means the agent can’t test an idea by changing something. It has to reason, and you have to be able to check the reasoning, which is why the timeline matters more than the answer.

6. Order in time isn’t proof of cause

Three ways a confident timeline proves nothing:

Shape	What it looks like
After isn’t because	Deploy at 10:02, latency at 10:05, and nobody can draw a path between them. That’s a coincidence with good timing.
Both are symptoms	Two metrics rise together because one thing upstream moved. Neither caused the other.
The trigger is gone	The surge that started it ended an hour ago, and the system still hasn’t recovered. What changed is no longer there to find.

If you can’t draw the path, you have a coincidence, not a cause. The third shape is the one nobody believes until they’ve seen it:

Time	What happens
10:00	A database fails over to its standby. It takes thirty seconds, as a failover should.
10:01	Every open connection dropped at once, so every client fails at once and retries at once.
10:06	The failover is long done and outside traffic is back to normal.
10:30	Still slow.

The retries are the load now. Timeouts cause retries, retries cause load, load causes timeouts. Name the three parts separately:

Trigger: the failover. It’s over, and there’s nothing to roll back.
Root cause: the retry policy with no cap and no backoff, shipped in last week’s client release. It’s the resource you can date a change on, and it’s still wrong right now.
Contributing factor: the missing circuit breaker. It would have stopped this, but you can’t show it ever changed.

7. The root cause is the resource that changed

A causal graph has five kinds of node, and only one is the answer.

Node	Meaning
Trigger	The event that set the incident off. It may already be over.
Root cause	The one resource whose own configuration diverged from its baseline, its previous revision, or its siblings
Contributing factor	A condition you can’t show has changed
Impact	What the customer lost, in their words
Recovery	What brought it back, if anything did

Divergence is a claim about time, so settle it by dating each candidate: creation timestamp, revision history, last-applied configuration. Never by which resource looks most abnormal now. The resource with the worst symptoms is usually a victim, and fixing a victim is a workaround. Contributing factor is the honest label for anything you suspect but can’t date. It isn’t a weaker root cause. It’s a different kind of claim.

8. A system that grades itself proves nothing

Hold every claim, the agent’s and your own, to what backs it:

The claim	What has to back it
”This caused that”	A path you can trace from the change to the customer impact
”The fix held”	A re-check that actually ran, and what it returned
”I’m confident”	A rival explanation that was tested and lost
”It’s accurate”	A verdict a person wrote, never the agent scoring its own run

CloudThinker applies the same idea. Every verdict carries an evidence label: Verified (confirmed against data from your systems), Inferred (reasoned, not confirmed) or Cause unproven (rests on an assumption). In your report, Assumed matches Cause unproven, and Blocked has no product label. Any change goes through a runbook and your Approvals settings. And with Resolve’s On duty paused, Pulse still detects and groups alerts and still files incidents as Open, but no investigation starts on its own until someone clicks Investigate. An incident a person logs still investigates right away. See What starts an investigation.

Keynote: 25 minutes

The Integration Problem Was a Context Problem: why an agent investigates well only when it can reach the data.

Practice: 30 minutes

The task: one live incident on a Kubernetes cluster. One team, one read-only agent, one report. Thirty minutes, no break. At the event, every team worked the incident below. On your own cluster, pick a real symptom from staging, such as a slow endpoint or a recent alert, and run the same steps.

Hotel Search Degradation After Traffic Recovery

That’s the title the on-call would get. The dashboards won’t agree with the users, so start from what the customer lost.

You can	You can’t
Read deployments, pods, events, logs, configuration and traffic telemetry	Restart, scale, deploy, edit anything, or read secrets

The permissions are real. If your team hits a wall, that wall is part of the exercise. Label it blocked, as on Day 01.

Time	Step	Done when
T+00	Connect and read the symptom	A chat is open on the cluster
T+05	Write the incident question	You have one line: what is broken, for whom, since when
T+10	Hold at least two hypotheses	Two explanations, both consistent with the evidence so far
T+18	Find the query that rules one out	You have its raw output
T+25	Finish the report	The file is submitted

Draft blocks 01 and 02 as you go from T+05. Five minutes is enough to finish a report, not to start one. The rail is a pace, not a rule. Two hypotheses is the floor, not the target. A team that reaches T+18 with one hypothesis has skipped the part that matters.

Connect your Kubernetes cluster

Use your own cluster, ideally staging, through a read-only kubeconfig. The cluster owner connects it on their own screen, and adding a connection needs the workspace Admin role. The full reference is Connect Kubernetes.

Create a read-only kubeconfig

Follow Connect Kubernetes: apply the cloudthinker-reader manifest, then build cloudthinker-kubeconfig.yaml for that service account. The ClusterRole grants only get, list and watch, and leaves out Secrets.Success state: kubectl --kubeconfig cloudthinker-kubeconfig.yaml get nodes lists your nodes.

Upload it in Connections

Sign in at app.cloudthinker.io. In the sidebar, click More → Workspace settings and select Connections. Select Kubernetes, upload the file in the KUBECONFIG field, and click Connect.Success state: Kubernetes shows a Connected status.

Open a chat

Start a new chat and send “Give me an overview of this Kubernetes cluster.”Success state: the agent answers with the namespaces and workloads it can see.

Can’t add the connection? Adding a connection needs connector edit permission, which only the workspace Admin role has. Ask a workspace Admin to connect the cluster, or to change your role. See Workspace roles.

Nobody types credentials into a slide, a chat or a shared document. The cluster owner connects it on their own screen.

What to ask the agent

Most teams ask what’s wrong and accept the first answer. Push past it. Start with these, in order:

Something is degraded in this cluster. Describe the symptom in terms of what a user would notice, and tell me what your role could not read.

A useful answer describes the user’s experience (slow or failed searches), not pod health. The “could not read” list goes straight into your scope.

Give me at least two different explanations for this symptom that are both consistent with the evidence so far.

A useful answer gives explanations that predict different things. If both would show the same data, ask again.

Which single query would come out differently depending on which of those explanations is true? Run it and show me the raw output.

This is the T+18 step. Paste the raw output into your report as the evidence that ruled one explanation out.

Which resource here can you show actually changed, with a timestamp? Which ones are you only assuming changed?

The resource with a dated change is your root cause candidate. Everything without a date is a contributing factor. Keep the raw output. You need it for every claim in the report.

Stuck?

Problem	What to do
The agent reports a healthy system	Ask what users are seeing, not what the dashboards say. Healthy dashboards are part of the exercise.
The agent gives one confident cause right away	Ask for the rival explanation, then for the query that separates the two.
The agent says it can’t read something	Good. Label it blocked. A blocked check is a result.
You run out of time	Submit what you have with the gap named. That’s a complete report.

Your output: one report

PDF, doc or markdown. There’s no template. These five blocks are the minimum.

Block	What it contains
01: Incident question and scope	What is broken, for whom, and since when. What the role couldn’t read.
02: At least two hypotheses, one ruled out	The evidence for each, and the query that ruled the loser out
03: Key evidence	Source, scope and time for each piece, as on Day 01
04: Your mechanism	What caused what, and the link you couldn’t show
05: Root cause vs contributing factors	Which one you could date, and which you couldn’t

Sample output

Working alone, check your report against this sample. It’s an illustration of the format, not a real result: your cluster, times and cause will differ.

# Day 02: Incident report

## 01 Incident question and scope
- Hotel search is slow or returns no results for some users, since 14:05 ICT
- Scope: staging cluster, namespace search, read-only kubeconfig, checked 2026-09-12 14:40 ICT
- Blocked: Secrets (not granted to this role)

## 02 Hypotheses
1. search-api pods ran short of CPU when traffic came back (ruled out)
   - Query: CPU use for search-api pods since 14:00. It stayed under 40% of the limit.
2. A new search-api revision cut the result cache size (kept)

## 03 Key evidence
- Deployment search-api revision 7 sets CACHE_SIZE lower than revision 6; its ReplicaSet was created at 14:02 (Verified, raw output in appendix A)
- p95 search latency rose from about 0.3 s to 2 s from 14:05 (Verified, appendix B)

## 04 Mechanism
- Smaller cache, more calls to the search backend, slower searches
- Not shown: load on the search backend (Blocked: outside this cluster)

## 05 Root cause vs contributing factors
- Root cause: search-api revision 7, dated 14:02
- Contributing: traffic recovery raised load (no dated change)

Check three things: each hypothesis has its own evidence, the losing one names the query that ruled it out, and the root cause has a date. A report that says “insufficient evidence” and shows the log is complete. A confident answer with no path is worth less. No fix proposal is required. Submit one file per team before Share starts. Redact account IDs, ARNs, endpoints, hostnames and customer data first. When you’re done, the connection stays in your workspace until a workspace Admin removes it: More → Workspace settings → Connections, select Kubernetes, then Disconnect. Volunteers present, 5 minutes each, so about four teams fit. Every team that presents gets a prize. No scoring, ranking or vote.

Minutes	Show
3	Mechanism and evidence. Which hypotheses you held, which query ruled one out, and what it returned.
2	Questions from the room.

Then the room compares: same cluster, same access, different answers. That comparison is why the log and the evidence fields matter more than the verdict.

Glossary

Term	Meaning on Day 02
Signal	One operational event that survived the noise checks
Signal cluster	Related signals grouped as one candidate problem (Pulse’s term). Not the same as the Kubernetes cluster in the lab.
Suppression	Dropping or silencing an event by a stated, ordered rule, rather than muting a channel
Flapping	A metric bouncing across a threshold, firing repeatedly for one condition
Cascade	Downstream effects of a root signal that’s already known
Trigger	The event that set the incident off. It may already be over.
Root cause	The one resource whose own configuration diverged from its baseline, previous revision, or siblings
Contributing factor	A condition you can’t show has changed
Hypothesis	An explanation stated well enough that a query could contradict it
Blocked	The role couldn’t look. Write it down; it’s a valid result.

Next step

Day 03: Prove It Arena. An unfamiliar environment, an unexpected incident, and judges. Bring today’s vocabulary: trigger, root cause, contributing factor, and the gap you can’t close.

Day 03: Prove It Arena

The rounds, how reports are judged, and how to join your team in the Arena.

Investigation and RCA

See how CloudThinker runs and records a real investigation.

Learn

Workshops

Scenarios

Prove It Day 02: Investigate

Rules of the room

Learn: 35 minutes

1. You can’t investigate what you can’t rank

2. Noise dies in ordered layers

3. Every suppression rule needs an escape hatch

4. Reduction is what makes the agent affordable

5. Split the question, not the work

6. Order in time isn’t proof of cause

7. The root cause is the resource that changed

8. A system that grades itself proves nothing

Keynote: 25 minutes

Practice: 30 minutes

Connect your Kubernetes cluster

What to ask the agent

Stuck?

Your output: one report

Sample output

Glossary

Further reading

Next step

Day 03: Prove It Arena

Investigation and RCA

​Rules of the room

​Learn: 35 minutes

​1. You can’t investigate what you can’t rank

​2. Noise dies in ordered layers

​3. Every suppression rule needs an escape hatch

​4. Reduction is what makes the agent affordable

​5. Split the question, not the work

​6. Order in time isn’t proof of cause

​7. The root cause is the resource that changed

​8. A system that grades itself proves nothing

​Keynote: 25 minutes

​Practice: 30 minutes

​Connect your Kubernetes cluster

​What to ask the agent

​Stuck?

​Your output: one report

​Sample output

​Share: 20 minutes

​Glossary

​Further reading

​Next step

Day 03: Prove It Arena

Investigation and RCA

Rules of the room

Learn: 35 minutes

1. You can’t investigate what you can’t rank

2. Noise dies in ordered layers

3. Every suppression rule needs an escape hatch

4. Reduction is what makes the agent affordable

5. Split the question, not the work

6. Order in time isn’t proof of cause

7. The root cause is the resource that changed

8. A system that grades itself proves nothing

Keynote: 25 minutes

Practice: 30 minutes

Connect your Kubernetes cluster

What to ask the agent

Stuck?

Your output: one report

Sample output

Share: 20 minutes

Glossary

Further reading

Next step