> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch

> Connect Elasticsearch to CloudThinker for log analysis, index management, and search performance optimization

Connect your Elasticsearch clusters to enable CloudThinker agents to search logs, analyze observability data, and investigate incidents.

Elasticsearch authenticates with an **API key** scoped to specific indices and cluster permissions.

***

## Prerequisites

* An **Elasticsearch** (7.x, 8.x), **OpenSearch** (1.x, 2.x), **AWS OpenSearch Service**, or **Elastic Cloud** cluster reachable from CloudThinker.
* Admin access to Kibana (**Stack Management → Security → API Keys**) or the Elasticsearch API to create an API key.
* The cluster endpoint URL.

***

## Setup

<Steps>
  <Step title="Open Kibana">
    Navigate to your Kibana instance and sign in with admin access.
  </Step>

  <Step title="Navigate to API Keys">
    Go to **Stack Management → Security → API Keys**.
  </Step>

  <Step title="Create API key">
    Click **Create API key** and enter:

    * **Name**: `cloudthinker-readonly`
  </Step>

  <Step title="Set permissions">
    Add role descriptors for read-only access (see [Required permissions](#required-permissions) for the full JSON). At minimum, grant `monitor` on the cluster and `read` + `view_index_metadata` on the index patterns you need.
  </Step>

  <Step title="Generate and copy">
    Click **Create API key** and immediately copy the encoded key — Elasticsearch will not show it again.
  </Step>

  <Step title="Note the cluster endpoint">
    Copy your Elasticsearch endpoint:

    * Self-hosted: `https://elasticsearch.your-domain.com:9200`
    * Elastic Cloud: found in deployment settings
    * AWS OpenSearch: domain endpoint from the AWS console
  </Step>

  <Step title="Add connection in CloudThinker">
    Navigate to **Connections → Elasticsearch** and enter:

    * **Cluster URL**: your Elasticsearch endpoint
    * **API Key**: the encoded key you copied
    * **Index patterns**: patterns to access (e.g., `logs-*`)

    Click **Connect**. CloudThinker verifies the credentials and shows a **Connected** status.
  </Step>
</Steps>

<Warning>
  Copy the API key immediately after creation. Elasticsearch will not show it again, and you'll need to create a new key if lost.
</Warning>

***

## Connection details

| Field            | Description               | Example                                           |
| ---------------- | ------------------------- | ------------------------------------------------- |
| **ES\_URL**      | Elasticsearch cluster URL | `https://your-cluster.es.region.cloud.es.io:9243` |
| **ES\_API\_KEY** | Base64-encoded API key    | `VnVhQk...`                                       |

Common index patterns to configure:

| Pattern        | Use case                     |
| -------------- | ---------------------------- |
| `logs-*`       | Application logs             |
| `filebeat-*`   | Filebeat-collected logs      |
| `metricbeat-*` | Infrastructure metrics       |
| `apm-*`        | Application performance data |
| `kubernetes-*` | Kubernetes cluster logs      |

***

## Required permissions

### Minimum

```json theme={null}
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*"],
      "privileges": ["read"]
    }
  ]
}
```

### Recommended

```json theme={null}
{
  "cluster": ["monitor", "read_ilm"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}
```

<Tip>
  Grant only `read` privileges on the index patterns CloudThinker needs. Never grant `write` or cluster-admin permissions to the CloudThinker API key.
</Tip>

***

## Agent capabilities

Once connected, agents can:

| Capability                 | Description                                |
| -------------------------- | ------------------------------------------ |
| **Log search**             | Search application and infrastructure logs |
| **Error analysis**         | Find error patterns and anomalies          |
| **Incident investigation** | Correlate events across services           |
| **Metrics query**          | Query stored metrics data                  |

### Verify the connection

```text theme={null}
@oliver search Elasticsearch for application errors in the last hour
```

### Example prompts

```text theme={null}
@oliver search logs for authentication failures in the last 24 hours
@alex find error patterns in application logs for the api-gateway service
@kai search for pod crash events in Kubernetes logs
```

***

## Troubleshooting

<Accordion title="Connection refused">
  * Verify the cluster URL is correct and accessible from CloudThinker.
  * Check that firewall rules allow CloudThinker's egress IPs.
  * For AWS OpenSearch, enable public access or configure a VPC endpoint.
  * Confirm Elasticsearch is running and accepting requests.
</Accordion>

<Accordion title="Authentication failed">
  * Verify the API key is correct and has not expired.
  * Confirm the key has the required index and cluster permissions.
  * For AWS OpenSearch, verify the IAM role or access policy is correctly configured.
</Accordion>

<Accordion title="API key expired">
  * Navigate to **Stack Management → Security → API Keys**.
  * Create a new key with the same permissions.
  * Update the key in CloudThinker connection settings.
</Accordion>

<Accordion title="No data returned">
  * Verify index patterns match existing indices in the cluster.
  * Check the API key has `read` permission on those indices.
  * Ensure the indices contain data in the requested time range.
</Accordion>

***

## Security

* **Least privilege** — grant only the permissions the agents need for your use case; start read-only and widen later.
* **Read-only by default** — use read-only credentials unless you want agents to make changes through this connection.
* **Rotate credentials** — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
* **Revoke on offboarding** — remove the credential at the provider when you delete a connection or a teammate leaves.

- **API key rotation** — rotate API keys every 90 days; set an expiration date when creating the key
- **Limited indices** — restrict the key to the index patterns CloudThinker actually needs rather than granting access to `*`

***

## Related

<CardGroup cols={2}>
  <Card title="Oliver Agent" icon="shield-check" href="/guide/agents/oliver">
    Security log analysis
  </Card>

  <Card title="Grafana Connection" icon="https://mintcdn.com/cloudthinker/aLd-ttc-SCW-aFky/images/icons/grafana.svg?fit=max&auto=format&n=aLd-ttc-SCW-aFky&q=85&s=c1329049025cd3c3a0909b400baef7be" href="/guide/connections/grafana" width="24" height="24" data-path="images/icons/grafana.svg">
    Visualize Elasticsearch data
  </Card>
</CardGroup>