Connect CloudWatch Alarms

This guide sends every CloudWatch alarm state change to Resolve, so CloudThinker sees the alarm within seconds and can open an incident and start a root cause analysis (RCA). Amazon EventBridge does the delivery: no Lambda function and no code to maintain.

How it works

A CloudWatch alarm state change flows through an EventBridge rule and API destination to the CloudThinker webhook URL and into Pulse, with other accounts or regions forwarding alarms to a central bus. — Alarm state changes reach Pulse through one EventBridge rule and API destination; other accounts can forward to a central bus.

When an alarm changes state, EventBridge forwards the event to your CloudThinker webhook URL. Resolve turns it into a signal (an event that survived noise suppression) in Pulse. Pulse groups related signals into a cluster, one problem, and triage decides whether an actionable cluster becomes an incident.

Prerequisites

The workspace Admin role. Developers and Viewers can see Resolve Triggers & Outcomes but can’t add or change them. See Roles.
An AWS account where you can create EventBridge connections, API destinations, and rules.
At least one CloudWatch alarm in the region you’re setting up.
The AWS CLI, if you want to test with a simulated alarm (recommended). The caller needs the IAM permission cloudwatch:SetAlarmState.

Create the webhook in CloudThinker

Open the AWS platform

Go to Resolve → Triggers & Outcomes and select Amazon Web Services. Under Signals to Resolve, click Add signal. The Create Webhook dialog opens with the AWS CloudWatch preset.You don’t need an AWS connection first. Without one, the platform reads Not configured or Needs setup (never Needs connection), and Add signal still works. An AWS connection is only needed if you want CloudThinker to query the affected resources during RCA.

Basic Info

Keep the default Webhook Name or rename it, for example Production CloudWatch. Click Next.

Configure

Field Mapping and Severity Mapping are pre-filled for EventBridge events (see the tables below). Leave them as they are for a first setup. For a production account, consider lowering INSUFFICIENT_DATA now (see Field and severity mapping). Click Next.

Advanced

Leave Auth Method as it is: API Key with the header x-api-key, which is what EventBridge sends. Then decide two options:

Option	Default	Change it when
Auto-investigate	On	You want alarms to cluster in Pulse but start nothing until a person acts. Turn it off.
Only from a minimum severity	Off	You want only some alarms to start an investigation. Turn it on and pick a level, for example High and above.

With both at their defaults, every alarm that becomes an incident starts an investigation. See Webhook options. Click Create.

Copy the URL and secret

Copy the Webhook URL and the API Secret from the dialog that opens. The URL goes into the EventBridge API destination, and the API Secret goes into the connection’s API key Value.

Raise the rate limit (production)

Open the webhook’s … menu under Signals to Resolve, choose Edit, and set Rate limit (requests per hour) to at least 3600. The default of 1,000 is below what an API destination sending 1 event per second can reach in an alarm storm, and events over the limit are dropped without an error. The Create Webhook dialog doesn’t have this field.

Success state: the webhook appears under Signals to Resolve on the Amazon Web Services platform, and the platform’s readiness badge reads Ready. Ready means the webhook is set up and active. It doesn’t prove an alarm arrived, so run the test below.

The API Secret is shown only once. If you close the dialog without copying it, open the webhook’s … menu under Signals to Resolve and choose Regenerate credentials. That replaces both the secret and the Webhook URL, and the old pair stops working at once. Update the EventBridge connection’s Value and the API destination’s endpoint right after.

Set up EventBridge in AWS

Do these steps in the AWS console, in the same region as your alarms.

Create a connection

Open Amazon EventBridge → Integration → Connections and click Create connection. Set:

Field	Value
Name	`cloudthinker-webhook`
Authorization type	API Key
API key name	`x-api-key`
Value	The API Secret from CloudThinker

Create an API destination

Open Integration → API destinations and click Create API destination. Set:

Field	Value
Name	`cloudthinker-resolve`
API destination endpoint	The Webhook URL from CloudThinker
HTTP method	`POST`
Connection	`cloudthinker-webhook`

Set Invocation rate limit per second to 1. Make sure you raised the webhook’s rate limit above.

Create a rule

Open Rules, select the default event bus, and click Create rule. Choose Rule with an event pattern and paste this pattern:

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"]
}

For the target, choose EventBridge API destination, select cloudthinker-resolve, and under Execution role pick Create a new role for this specific resource (it gets events:InvokeApiDestination).Optional, recommended for production: in the target’s Additional settings, set the retry policy and a dead-letter queue, so alarms that fail delivery during an outage are kept instead of lost. A good starting point is Maximum age of event 1 hour and Retry attempts 10: an alarm older than an hour is rarely worth a fresh investigation. Attach an SQS queue as the dead-letter queue. Click Create rule.

Success state: the rule shows as Enabled with your API destination as its target.

Test the integration

The test alarm is treated like a real one. With Auto-investigate on, it opens an incident and CloudThinker starts an investigation. To test delivery only, turn Auto-investigate off on this webhook first (… menu → Edit), and turn it back on after. Don’t pause On duty for this in a live workspace: it stops automatic triage and investigation for every source, not just this one (see What starts an investigation).

Force an alarm

Replace YourAlarmName with a real alarm that’s currently OK, and us-east-1 with the alarm’s region. A wrong region fails with an alarm-not-found error. CloudWatch sends an event only when the state actually changes, so an alarm already in ALARM sends nothing. If yours is, set it to OK first, then to ALARM.

aws cloudwatch set-alarm-state \
  --alarm-name "YourAlarmName" \
  --region us-east-1 \
  --state-value ALARM \
  --state-reason "Testing CloudThinker integration"

Check Pulse

Open Resolve → Pulse. Within a few seconds a signal appears, titled with the alarm name and marked High severity.

Reset the alarm

Run the same command with --state-value OK. CloudWatch then evaluates the alarm normally again. It may also return the alarm to OK by itself at its next evaluation, so an OK signal can arrive before you reset it.The OK event arrives as an Info signal. Pulse’s classifier treats a recovery as not actionable, so it usually doesn’t open a second incident. That check is a model judgment, so a recovery can still slip through to triage. It also doesn’t resolve the incident the ALARM opened. That incident stays open until someone closes it.

Success state: the signal shows the alarm name as its title, the state reason as its description, and the metric namespace (for example AWS/EC2) as its service.

Field and severity mapping

The preset reads these fields from the EventBridge event:

Field	Path	Example value
Title Path	`detail.alarmName`	`HighCPUUtilization`
Description Path	`detail.state.reason`	`Threshold Crossed: 1 out of the last 1 datapoints [85.0] was greater than the threshold (80.0)`
Severity Path	`detail.state.value`	`ALARM`
Service Path	`detail.configuration.metrics[0].metricStat.metric.namespace`	`AWS/EC2`

CloudWatch state	CloudThinker severity
`ALARM`	High
`INSUFFICIENT_DATA`	Medium
`OK`	Info

With the defaults, an INSUFFICIENT_DATA flap (a metric gap after a deploy, for example) can start an investigation. To stop that, find the INSUFFICIENT_DATA row in Severity Mapping, open its Maps To dropdown and pick Low or Info. Or turn on Only from a minimum severity and pick High and above. Pulse’s noise limit (100 alerts a minute) always lets Critical and High through and can drop the rest in a storm. Map any state you can’t afford to lose to High or above. Want alarms to be critical? Edit the webhook and change the ALARM row in Severity Mapping. The mapping applies to every alarm the webhook receives. To treat production and non-production differently, create two webhooks and two EventBridge rules, each filtered by alarm name prefix (see Send only the alarms you care about). A state with no mapping row falls back to Medium. Composite alarms and metric-math alarms have no metricStat block, so Service comes out as unknown. If you rely on those alarms, point Service Path at a field they do carry, such as detail.alarmName, and name alarms after their service. Repeats of one alarm group by the alarm’s ARN, whatever Service says. Service still counts: alarms that share a Service value within 6 hours can join one cluster, so two different AWS/EC2 alarms firing together can land in the same cluster. See How signals join a cluster.

Send only the alarms you care about

Narrow the EventBridge rule’s event pattern instead of filtering in CloudThinker. Add a detail block to the base pattern:

Goal	Add to the pattern
Only alarms named `prod-*`	`"detail": { "alarmName": [{ "prefix": "prod-" }] }`
Skip recoveries	`"detail": { "state": { "value": ["ALARM", "INSUFFICIENT_DATA"] } }`. You lose the `OK` signal, so Pulse no longer shows when an alarm cleared.

More than one region or account

CloudWatch sends alarm events to EventBridge only in the alarm’s own region and account. Pick one of these layouts:

Layout	What you build	Best when
Per-region rules	The connection, API destination and rule in every region and account, all pointing at the same webhook URL	A few regions in one account
Central event bus	In each source region or account, a rule that forwards alarm events to an event bus in one monitoring account. The connection and API destination exist only in that account.	AWS Organizations, or many accounts

Deploy with CloudFormation

This template builds the connection, API destination, rule and dead-letter queue, and outputs the queue URL so you can monitor it. Deploy it to each account and region for the per-region layout (for example with StackSets), or once in the monitoring account for the central bus, with EventBusName set to that bus. Deploy one stack per webhook in each account and region. To send prod and non-prod alarms to two webhooks from the same region, deploy it twice with different NamePrefix values, and narrow each rule’s pattern (see Send only the alarms you care about).

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  NamePrefix:
    Type: String
    Default: cloudthinker
    Description: Prefix for the connection and API destination names. Unique per stack in a region.
  EventBusName:
    Type: String
    Default: default
    Description: The bus that receives the alarm events. Use your central bus name for the central layout.
  WebhookUrl:
    Type: String
    Description: The Webhook URL from the CloudThinker webhook dialog
  ApiSecret:
    Type: String
    NoEcho: true
    Description: The API Secret from the CloudThinker webhook dialog
Resources:
  DeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 1209600
  DeadLetterQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues: [!Ref DeadLetterQueue]
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: events.amazonaws.com }
            Action: sqs:SendMessage
            Resource: !GetAtt DeadLetterQueue.Arn
            Condition:
              ArnEquals: { "aws:SourceArn": !GetAtt AlarmRule.Arn }
  Connection:
    Type: AWS::Events::Connection
    Properties:
      Name: !Sub "${NamePrefix}-webhook"
      AuthorizationType: API_KEY
      AuthParameters:
        ApiKeyAuthParameters:
          ApiKeyName: x-api-key
          ApiKeyValue: !Ref ApiSecret
  Destination:
    Type: AWS::Events::ApiDestination
    Properties:
      Name: !Sub "${NamePrefix}-resolve"
      ConnectionArn: !GetAtt Connection.Arn
      InvocationEndpoint: !Ref WebhookUrl
      HttpMethod: POST
      InvocationRateLimitPerSecond: 1
  InvokeRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: events.amazonaws.com }
            Action: sts:AssumeRole
      Policies:
        - PolicyName: invoke-cloudthinker
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: events:InvokeApiDestination
                Resource: !GetAtt Destination.Arn
  AlarmRule:
    Type: AWS::Events::Rule
    Properties:
      EventBusName: !Ref EventBusName
      EventPattern:
        source: ["aws.cloudwatch"]
        detail-type: ["CloudWatch Alarm State Change"]
      Targets:
        - Id: cloudthinker
          Arn: !GetAtt Destination.Arn
          RoleArn: !GetAtt InvokeRole.Arn
          RetryPolicy:
            MaximumEventAgeInSeconds: 3600
            MaximumRetryAttempts: 10
          DeadLetterConfig:
            Arn: !GetAtt DeadLetterQueue.Arn
Outputs:
  DeadLetterQueueUrl:
    Value: !Ref DeadLetterQueue

Deploy it with the AWS CLI. Read the secret into an environment variable first (read -rs doesn’t echo it or write it to shell history):

read -rs CT_API_SECRET   # paste the API Secret, then press Enter
aws cloudformation deploy \
  --stack-name cloudthinker-cloudwatch \
  --template-file cloudthinker-cloudwatch.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    WebhookUrl="https://your-webhook-url" \
    ApiSecret="$CT_API_SECRET"

The connection stores the API key in an AWS Secrets Manager secret that EventBridge creates and manages for you, so the secret isn’t kept in the template or the rule. Alarms that still fail after the retries land in the SQS dead-letter queue, kept for 14 days.

Forward alarms to a central bus

For the central layout, deploy the template above in the monitoring account with EventBusName set to your bus. Then add two pieces. In the monitoring account, give the central bus a resource policy that lets your organization put events. Replace the bus name, region, account ID and organization ID:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowOrgPutEvents",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "events:PutEvents",
    "Resource": "arn:aws:events:us-east-1:111111111111:event-bus/central-alarms",
    "Condition": { "StringEquals": { "aws:PrincipalOrgID": "o-exampleorgid" } }
  }]
}

In each source account and region, deploy a forwarding rule on the default bus:

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  CentralBusArn:
    Type: String
    Description: ARN of the central bus in the monitoring account
Resources:
  ForwardRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: events.amazonaws.com }
            Action: sts:AssumeRole
      Policies:
        - PolicyName: put-central-bus
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: events:PutEvents
                Resource: !Ref CentralBusArn
  ForwardRule:
    Type: AWS::Events::Rule
    Properties:
      EventBusName: default
      EventPattern:
        source: ["aws.cloudwatch"]
        detail-type: ["CloudWatch Alarm State Change"]
      Targets:
        - Id: central-bus
          Arn: !Ref CentralBusArn
          RoleArn: !GetAtt ForwardRole.Arn

The forwarded events keep their source and detail-type, so the rule on the central bus matches them unchanged. For more options, see the AWS guides for cross-account and cross-region delivery.

Amazon SNS works but isn’t recommended. SNS can’t send the x-api-key header, so the webhook needs Auth Method set to No Authentication, and the URL becomes the only credential. If you use it anyway, subscribe the Webhook URL to the topic with the HTTPS protocol. CloudThinker confirms the subscription for you.

Troubleshooting

No signal appears in Pulse

Check the rule: in EventBridge, open the rule’s Monitoring tab. If Invocations stays at 0, the pattern isn’t matching (see the next item).
Check the destination: the API destination endpoint must be the exact Webhook URL from CloudThinker. After Regenerate credentials, the URL changes too.
Check failed deliveries: on the rule’s Monitoring tab, look at FailedInvocations, then check the SQS dead-letter queue. Messages there carry the HTTP error that came back.
Check Pulse: on Resolve → Pulse, pick the All lane and check the time window selector in the page header (it defaults to Last 7 days). Pulse can also drop an alert as noise, see noise suppression.

The rule never matches

The rule must be on the bus that receives the alarm events: the default bus in the alarm’s region, or your central bus. The detail-type value is case-sensitive: CloudWatch Alarm State Change.

EventBridge reports 401 or 403

A 401 means the x-api-key header is missing, or the endpoint isn’t a current Webhook URL. A 403 means the key’s value is wrong. The connection’s API key name must be x-api-key and its value must be the API Secret. If you lost the secret, open Resolve → Triggers & Outcomes → Amazon Web Services, open the webhook’s … menu under Signals to Resolve, and choose Regenerate credentials. Then update both the connection’s Value and the API destination’s endpoint, because the URL changes too.

Signals arrive but the title or severity is empty

The field paths must start with detail., because that’s where EventBridge puts the alarm. CloudThinker converts SNS alarm messages to the same shape, so paths like AlarmName or NewStateValue never match, even for SNS. An empty service on a composite or metric-math alarm is expected, see Field and severity mapping.

Alarms go missing during a storm, or EventBridge reports 429

The webhook has an hourly Rate limit (requests per hour). It defaults to 1,000, goes up to 10,000, and 0 means no limit. Events over the limit still get a success response, then are dropped before they reach Pulse, whatever their severity, so EventBridge shows no error. This is a different limit from Pulse’s 100-per-minute noise limit, which always lets Critical and High through.A 429 comes back only at 10 times the hourly limit: at the default 1,000, after 10,000 requests within the same hour. It also comes back after 10 failed authentication attempts within 15 minutes, or while CloudThinker is briefly busy (with a Retry-After header). EventBridge retries a 429 within the rule’s retry policy.Check: drops leave no error, so compare counts. Right after a burst, set the Resolve → Pulse time window to Last 1 hour and read the raw events count in the header. Compare it with the rule’s Invocations metric (its Monitoring tab) over the same hour. Raw events counts every source in the workspace, so a lower number than EventBridge sent means events were dropped.Fix: open the webhook’s … menu, choose Edit, and raise Rate limit (requests per hour). Or narrow the rule’s event pattern. If 401s or 403s came first, fix the secret.

An alarm reached Pulse but no RCA started

Auto-investigate is off, the severity is below its minimum, or On duty is paused. See What starts an investigation.

Get started

Set up

Work incidents

Learn and improve

Help

Connect CloudWatch Alarms

How it works

Prerequisites

Create the webhook in CloudThinker

Set up EventBridge in AWS

Test the integration

Field and severity mapping

Send only the alarms you care about

More than one region or account

Deploy with CloudFormation

Forward alarms to a central bus

Troubleshooting

Next steps

Watch the investigation

Tune Pulse

​How it works

​Prerequisites

​Create the webhook in CloudThinker

​Set up EventBridge in AWS

​Test the integration

​Field and severity mapping

​Send only the alarms you care about

​More than one region or account

​Deploy with CloudFormation

​Forward alarms to a central bus

​Troubleshooting

​Next steps

Watch the investigation

Tune Pulse

How it works

Prerequisites

Create the webhook in CloudThinker

Set up EventBridge in AWS

Test the integration

Field and severity mapping

Send only the alarms you care about

More than one region or account

Deploy with CloudFormation

Forward alarms to a central bus

Troubleshooting

Next steps