> ## Documentation Index > Fetch the complete documentation index at: https://docs.cloudthinker.io/llms.txt > Use this file to discover all available pages before exploring further. # Splunk Observability > Connect Splunk Observability Cloud to CloudThinker for read-only APM investigation, service latency and error breakdowns, exemplar traces, metric discovery, and SignalFlow queries Connect your Splunk Observability Cloud organization to let [Alex](/guide/agents/alex) (Cloud Engineer) list APM environments and services, break latency and errors down by tag, follow a dependency to the service that is failing, pull exemplar traces, search your metric catalog, and run SignalFlow queries. The connection reads. It uses Splunk's hosted MCP Gateway and needs your realm and an access token. ## Prerequisites * A **Splunk Observability Cloud** organization, and the **realm** that holds it. A realm is the self-contained deployment your organization runs in, named like `us1`, `eu0`, or `mon0`. * An **organization administrator**, since Splunk requires the admin role to create an access token. * A **supported, authorized organization**. Splunk supports the MCP server in every production realm except Google Cloud Platform and GovCloud realms, and an organization that is not authorized for the Observability MCP tools answers `Org Not Allowed`. * **APM instrumentation**, for the APM answers. An organization that reports metrics but instruments no services returns zero environments; metric and SignalFlow questions still work. ## Setup In Splunk Observability Cloud, open **Settings → Access Tokens**, select **New Token**, and name it. Select the **API** authorization scope, then the **read\_only** role when Splunk asks which roles to attach. Choose who can read the token and set its expiration date. Splunk expires an access token 30 days after creation unless you change that here. Select the token name, select **Show Token**, then **Copy**. Open **Settings**, select your user name at the top of the menu, then open the **Organizations** tab. Your realm is there to view or copy. Go to **Connections → Splunk Observability**, paste the token into **Access token**, enter your **Realm**, choose a **Gateway region**, then click **Connect**. ## Connection details | Field | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------- | | **Access token** | An API-scope org access token with the `read_only` role | | **Realm** | The realm that holds your organization, for example `us1` or `eu0` | | **Gateway region** | Which Splunk Cloud Services region serves the MCP Gateway. Your requests and their results pass through it | Splunk's rule is that the gateway region must correspond to your realm, and it points at a supported-regions table that is not published yet. Until it is, pick the region in your realm's geography; a region that does not serve your realm answers `404`. Splunk's other credential, a **user API access token**, also reaches the gateway but expires when you log out of Splunk Observability Cloud, or after 30 days, whichever comes first. That takes the connection down with no warning, so use one only for a quick trial. ## Required permissions `read_only` is enough. Splunk describes it as reaching every page and object a `power` user can, without creating, editing, or deleting anything, and every tool in this connection reads. `power` and `usage` both grant more than it needs. Splunk enforces the role at its own API. The connection has no write lane, so an agent cannot create, edit, mute, or delete a detector, dashboard, chart, or alert here, and a change you want stays yours to make in Splunk. ## Agent capabilities Once connected, Alex can: | Capability | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------ | | **Environment and service discovery** | List APM environments and the services reporting in each one | | **Service health** | Break latency, errors, and request counts down by tag for one service | | **Dependency mapping** | Read inbound and outbound dependencies with their requests and latencies | | **Trace investigation** | Pull exemplar traces for errors or latency buckets, then open the spans of one trace by ID | | **Metric discovery** | Search the metric catalog, then read a metric's dimensions, properties, and sample values | | **SignalFlow queries** | Write a program grounded in real metric names and execute it for a time series | | **Alert review** | Search triggered alerts and incidents | ### Verify the connection ```text theme={null} @alex #report summarize my Splunk Observability setup: realm, APM environments, services, and whether APM instrumentation is reporting ``` ### Example prompts ```text theme={null} @alex which service in the prod environment has the worst error rate @alex why is checkout slow - break its latency down by tag and show me an error trace @alex what metrics do we have for kafka, and chart the busiest one ``` ## Troubleshooting Splunk's `-32005`. Your organization is not authorized to reach the Observability MCP tools, and no credential or region change fixes it. Ask your Splunk administrator. Splunk's `-32003`. The access token is invalid or expired. Check the **Access token** field, then confirm the token was issued in the organization your **Realm** names. Splunk's `404`. The MCP endpoint is not served at the address used, so change the **Gateway region** to the one that corresponds to your realm. Expect this from a token that does not match the realm: rather than failing, the gateway can answer with an empty tool set. Reissue the token in the organization that owns your realm. Both credentials expire in 30 days by default, and a user API access token also expires the moment you log out. Create a new org access token with a longer expiration, or rotate the existing one. Empty is often the true state: an organization with no APM instrumentation reports zero environments, and a catalogued metric can simply be idle. The agent says when a search did not finish instead of reporting zero, and trace search can outrun the time budget for one call. In either case, ask again over a different time range. The catalog search matches loosely and returns only a few names per call, so a name you know exists can be missed. Ask for the exact name, or search with different terms. ## Security * **Least privilege** — grant only the permissions the agents need for your use case; start read-only and widen later. * **Read-only by default** — use read-only credentials unless you want agents to make changes through this connection. * **Rotate credentials** — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection. * **Revoke on offboarding** — remove the credential at the provider when you delete a connection or a teammate leaves. - **Least privilege by role** — an API-scope token with `read_only` is the narrowest credential that runs every tool here, and Splunk enforces it server-side. - **The gateway region is a data path** — your requests and their results travel through the region you select, so choose it deliberately when data residency matters. The realm and region are routing values, not credentials. ## Related Trace queries, BubbleUp root-cause analysis, and approval-gated board and trigger changes DQL queries, problem investigation, and vulnerability review