Skip to main content

​
Elasticsearch

Connect your Elasticsearch clusters to enable CloudThinker agents to search logs, analyze observability data, and investigate incidents.

​
Supported Platforms

PlatformSupport
Elasticsearch7.x, 8.x
OpenSearch1.x, 2.x
AWS OpenSearch ServiceAll versions
Elastic CloudAll deployments

​
Setup

1

Open Kibana

Navigate to your Kibana instance and sign in with admin access.
2

Navigate to API Keys

Go to Stack Management → Security → API Keys.
3

Create API Key

Click Create API key and enter:
  • Name: cloudthinker-readonly
4

Set Permissions

For read-only access, add role descriptors:
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}
5

Generate and Copy

Click Create API key and immediately copy the encoded key (it won’t be shown again).
6

Note Cluster Endpoint

Get your Elasticsearch endpoint:
  • Self-hosted: https://elasticsearch.your-domain.com:9200
  • Elastic Cloud: Found in deployment settings
  • AWS OpenSearch: Domain endpoint from AWS console
7

Add Connection in CloudThinker

Navigate to Connections → Elasticsearch and enter:
  • Cluster URL: Your Elasticsearch endpoint
  • API Key: The encoded key you copied
  • Index patterns: Patterns to access (e.g., logs-*)
Copy the API key immediately after creation. Elasticsearch will not show it again, and you’ll need to create a new key if lost.

​
Connection Details

FieldDescriptionExample
ES_URLElasticsearch cluster URLhttps://your-cluster.es.region.cloud.es.io:9243
ES_API_KEYBase64 encoded API keyVnVhQk...

​
Required Permissions

​
Minimum

{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*"],
      "privileges": ["read"]
    }
  ]
}

{
  "cluster": ["monitor", "read_ilm"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

​
Agent Capabilities

Once connected, agents can:
CapabilityDescription
Log SearchSearch application and infrastructure logs
Error AnalysisFind error patterns and anomalies
Incident InvestigationCorrelate events across services
Metrics QueryQuery stored metrics data

​
Example Prompts

@oliver search logs for authentication failures in the last 24 hours
@alex find error patterns in application logs for service-api
@kai search for OOMKilled events in Kubernetes logs
@tony analyze database error logs for connection issues

​
Common Index Patterns

PatternUse Case
logs-*Application logs
filebeat-*Filebeat-collected logs
metricbeat-*Infrastructure metrics
apm-*Application performance data
kubernetes-*Kubernetes cluster logs

​
Troubleshooting

  • Verify cluster URL is correct and accessible
  • Check firewall allows CloudThinker IPs
  • For AWS OpenSearch: Enable public access or VPC endpoint
  • Confirm Elasticsearch is running
  • Verify API key is correct
  • Check API key hasn’t expired
  • Ensure user/role has required permissions
  • For AWS: Verify IAM role or access policy
  • Navigate to Stack Management → Security → API Keys
  • Create a new API key with the same permissions
  • Update the key in CloudThinker connection settings
  • Verify index patterns match existing indices
  • Check user has read permission on indices
  • Ensure indices contain data in the time range
  • Confirm field mappings are correct

​
Security Best Practices

  • Read-only access - Grant only read privileges, never write
  • Limited indices - Restrict access to necessary index patterns only
  • API key rotation - Rotate API keys every 90 days
  • Expiration - Set appropriate expiration on API keys
  • Audit logging - Enable Elasticsearch audit logs

Oliver Agent

Security log analysis

Grafana Connection

Visualize Elasticsearch data