Coming soon — Detectors is not available in workspaces yet. This page describes how it will work. Contact us to hear when it launches.
What a Detector catches that alert rules miss
Your observability tool fires when a metric crosses a line you set in advance. A Detector starts from the other side: the incidents your alerts missed. CloudThinker studies each miss, finds the pattern that came before it, and writes a check for that pattern.
Patterns a Detector can catch, for example:
- A slow burn. An error budget that burns too fast over both a short and a long window, before any single metric looks bad.
- Unusual for this hour. Traffic or latency outside its normal range for the same hour of the week, measured over the last 4 weeks, instead of one fixed line.
- A limit nobody alerted on. A connection pool near its maximum, a certificate close to expiry, or a quota almost used up.
- A problem only two tools show together. A check can read from more than one connected tool, so it can compare what each one sees.
The idea in one picture
You keep your current alerts. CloudThinker adds Detectors next to them. Each Detector is tested on past incidents first, and starts working only after you accept it. A Detector’s fire takes the same path as your own alerts: Pulse, then AI triage, then RCA. Triage decides whether anyone is paged. Three rules keep this safe:- Nothing replaces your alerts. Detectors work next to the rules you already have.
- Every Detector is tested before you see it. CloudThinker runs it once and backtests it on past incidents, so you can judge it before it runs.
- Only a person can turn a Detector on. CloudThinker can suggest and fix Detectors, but it can’t accept one by itself.
Words used on this page
Get started
1
Connect a monitoring tool
In Resolve → Integrations, connect at least one monitoring tool, such as Datadog or Grafana. CloudThinker needs it to read your alert rules and metrics. See Resolve integrations.
2
Start the first analysis
Open Resolve → Detectors and click Analyze my monitoring. This turns Detectors on. CloudThinker starts reviewing your alert rules, recent signals, and past incidents right away.
3
Accept or dismiss each suggestion
When the analysis ends, go to the Needs you tab. Each suggested Detector shows why it is needed and how it did in its tests. Click Accept to turn it on, or Dismiss to drop it.
How it works
Detectors run in a weekly cycle. You only step in at the points marked You.- Check your alerts. CloudThinker finds rules that are noisy, rarely useful, or can never fire. You see them on the Alert health tab.
- Suggest and test Detectors. For each gap, CloudThinker writes a Detector, runs it once, and backtests it on the last 30 days.
- You accept. Nothing runs until a person clicks Accept.
- Runs beside your alerts. The Detector runs on a schedule. When it fires, the fire goes to Pulse and triage, the same as any alert.
- Weekly review. Once a week, CloudThinker marks each fire as real or noise, fixes noisy Detectors, removes the ones that stay noisy, and sends you a scorecard.
- Turn it into your own rule. CloudThinker can copy a proven Detector into your alerting tool as a normal alert rule.
Detector states
A Detector is always in one of four states.
Open a Detector to Pause, Resume, or Retire it. You also find Tune, Run backtest, and Promote to rule there. Accept and Retire ask you to confirm.
What CloudThinker can do on its own
In a chat, CloudThinker has your permissions, so it can’t do anything the page wouldn’t let you do. Members with read-only access can see Detectors and fires, but can’t change them.
The Detectors page
Resolve → Detectors has four tabs. Each one answers one question.
Click Ask CloudThinker at the top of the page to ask a question or tell it what to watch:
Read a Detector’s results
Open a Detector to see its scorecard, its backtest, and its fires.
An example scorecard. The numbers are made up to show the layout of the results.
Caught first and lead time
The scorecard tells you how often the Detector found a problem before your own alerts did, and how much earlier.
The Detector fired 12 minutes before your first alert.
In this example the Detector caught it first, with a lead time of 12 minutes. If no alert had fired at all, the lead time would be counted up to when the incident was created.Fire labels
Each fire gets one of these labels:
Only you or the weekly review labels a fire as real or noise. CloudThinker never decides that on its own.
Backtest
A backtest runs the Detector against past data. It covers the last 30 days by default, and at most 90 days. A past fire counts as a catch only if it lands in the catch window: from one hour before the problem started until the incident was created.
Only the fire inside the catch window counts.
Good to know
- Results count CloudThinker incidents only. Incidents closed as false alarms don’t count.
- With fewer than three incidents, the page shows plain counts instead of percentages.
- If a Detector fails three times in a row, CloudThinker tries to fix it. If it can’t, it pauses the Detector.
- If someone changes a Detector’s script outside CloudThinker, the Detector shows Changed outside Resolve and its backtest is marked out of date.
Turn a Detector into your own alert rule
When a Detector has proven itself, click Promote to rule. What happens next depends on whether you set up a rule repository:
To set the repository, open Resolve integrations and turn on Propose alert rule changes for GitLab, GitHub, Bitbucket, or Azure DevOps. CloudThinker never changes the rules inside your alerting tool directly.
Settings
Find the Detectors section in Resolve settings. Changes save right away.Limits
Related
Understand Pulse
See how Detector fires sit next to your own alerts.
Manage integrations
Connect monitoring tools and the rule repository.
Tune Resolve behavior
Change automatic RCA and the Detectors settings.
Review RCA
Follow the investigation of an incident a Detector raised.