> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch

> Connect Elasticsearch to CloudThinker for log analysis, index management, and search performance optimization

Connect your Elasticsearch clusters to enable CloudThinker agents to search logs, analyze observability data, and investigate incidents.

***

## Supported Platforms

| Platform                   | Support         |
| -------------------------- | --------------- |
| **Elasticsearch**          | 7.x, 8.x        |
| **OpenSearch**             | 1.x, 2.x        |
| **AWS OpenSearch Service** | All versions    |
| **Elastic Cloud**          | All deployments |

***

## Setup

<Steps>
  <Step title="Open Kibana">
    Navigate to your Kibana instance and sign in with admin access.
  </Step>

  <Step title="Navigate to API Keys">
    Go to **Stack Management → Security → API Keys**.
  </Step>

  <Step title="Create API Key">
    Click **Create API key** and enter:

    * **Name**: `cloudthinker-readonly`
  </Step>

  <Step title="Set Permissions">
    For read-only access, add role descriptors:

    ```json theme={null}
    {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["*"],
          "privileges": ["read", "view_index_metadata"]
        }
      ]
    }
    ```
  </Step>

  <Step title="Generate and Copy">
    Click **Create API key** and **immediately copy the encoded key** (it won't be shown again).
  </Step>

  <Step title="Note Cluster Endpoint">
    Get your Elasticsearch endpoint:

    * Self-hosted: `https://elasticsearch.your-domain.com:9200`
    * Elastic Cloud: Found in deployment settings
    * AWS OpenSearch: Domain endpoint from AWS console
  </Step>

  <Step title="Add Connection in CloudThinker">
    Navigate to **Connections → Elasticsearch** and enter:

    * **Cluster URL**: Your Elasticsearch endpoint
    * **API Key**: The encoded key you copied
    * **Index patterns**: Patterns to access (e.g., `logs-*`)
  </Step>
</Steps>

<Warning>
  Copy the API key immediately after creation. Elasticsearch will not show it again, and you'll need to create a new key if lost.
</Warning>

***

## Connection Details

| Field            | Description               | Example                                           |
| ---------------- | ------------------------- | ------------------------------------------------- |
| **ES\_URL**      | Elasticsearch cluster URL | `https://your-cluster.es.region.cloud.es.io:9243` |
| **ES\_API\_KEY** | Base64 encoded API key    | `VnVhQk...`                                       |

***

## Required Permissions

### Minimum

```json theme={null}
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*"],
      "privileges": ["read"]
    }
  ]
}
```

### Recommended

```json theme={null}
{
  "cluster": ["monitor", "read_ilm"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}
```

***

## Agent Capabilities

Once connected, agents can:

| Capability                 | Description                                |
| -------------------------- | ------------------------------------------ |
| **Log Search**             | Search application and infrastructure logs |
| **Error Analysis**         | Find error patterns and anomalies          |
| **Incident Investigation** | Correlate events across services           |
| **Metrics Query**          | Query stored metrics data                  |

### Example Prompts

```bash theme={null}
@oliver search logs for authentication failures in the last 24 hours
@alex find error patterns in application logs for service-api
@kai search for OOMKilled events in Kubernetes logs
@tony analyze database error logs for connection issues
```

***

## Common Index Patterns

| Pattern        | Use Case                     |
| -------------- | ---------------------------- |
| `logs-*`       | Application logs             |
| `filebeat-*`   | Filebeat-collected logs      |
| `metricbeat-*` | Infrastructure metrics       |
| `apm-*`        | Application performance data |
| `kubernetes-*` | Kubernetes cluster logs      |

***

## Troubleshooting

<Accordion title="Connection refused">
  * Verify cluster URL is correct and accessible
  * Check firewall allows CloudThinker IPs
  * For AWS OpenSearch: Enable public access or VPC endpoint
  * Confirm Elasticsearch is running
</Accordion>

<Accordion title="Authentication failed">
  * Verify API key is correct
  * Check API key hasn't expired
  * Ensure user/role has required permissions
  * For AWS: Verify IAM role or access policy
</Accordion>

<Accordion title="API key expired">
  * Navigate to **Stack Management → Security → API Keys**
  * Create a new API key with the same permissions
  * Update the key in CloudThinker connection settings
</Accordion>

<Accordion title="No data returned">
  * Verify index patterns match existing indices
  * Check user has read permission on indices
  * Ensure indices contain data in the time range
  * Confirm field mappings are correct
</Accordion>

***

## Security Best Practices

* **Read-only access** - Grant only `read` privileges, never `write`
* **Limited indices** - Restrict access to necessary index patterns only
* **API key rotation** - Rotate API keys every 90 days
* **Expiration** - Set appropriate expiration on API keys
* **Audit logging** - Enable Elasticsearch audit logs

***

## Related

<CardGroup cols={2}>
  <Card title="Oliver Agent" icon="shield-check" href="/guide/agents/oliver">
    Security log analysis
  </Card>

  <Card title="Grafana Connection" icon="https://mintcdn.com/cloudthinker/aLd-ttc-SCW-aFky/images/icons/grafana.svg?fit=max&auto=format&n=aLd-ttc-SCW-aFky&q=85&s=c1329049025cd3c3a0909b400baef7be" href="/guide/connections/grafana" width="24" height="24" data-path="images/icons/grafana.svg">
    Visualize Elasticsearch data
  </Card>
</CardGroup>