Prerequisites
- A Splunk Observability Cloud organization, and the realm that holds it. A realm is the self-contained deployment your organization runs in, named like
us1,eu0, ormon0. - An organization administrator, since Splunk requires the admin role to create an access token.
- A supported, authorized organization. Splunk supports the MCP server in every production realm except Google Cloud Platform and GovCloud realms, and an organization that is not authorized for the Observability MCP tools answers
Org Not Allowed. - APM instrumentation, for the APM answers. An organization that reports metrics but instruments no services returns zero environments; metric and SignalFlow questions still work.
Setup
1
Create the access token
In Splunk Observability Cloud, open Settings → Access Tokens, select New Token, and name it. Select the API authorization scope, then the read_only role when Splunk asks which roles to attach.
2
Set visibility and expiration
Choose who can read the token and set its expiration date. Splunk expires an access token 30 days after creation unless you change that here.
3
Copy the token secret
Select the token name, select Show Token, then Copy.
4
Read your realm
Open Settings, select your user name at the top of the menu, then open the Organizations tab. Your realm is there to view or copy.
5
Connect in CloudThinker
Go to Connections → Splunk Observability, paste the token into Access token, enter your Realm, choose a Gateway region, then click Connect.
Connection details
Splunk’s rule is that the gateway region must correspond to your realm, and it points at a supported-regions table that is not published yet. Until it is, pick the region in your realm’s geography; a region that does not serve your realm answers
404.
Required permissions
read_only is enough. Splunk describes it as reaching every page and object a power user can, without creating, editing, or deleting anything, and every tool in this connection reads. power and usage both grant more than it needs.
Splunk enforces the role at its own API. The connection has no write lane, so an agent cannot create, edit, mute, or delete a detector, dashboard, chart, or alert here, and a change you want stays yours to make in Splunk.
Agent capabilities
Once connected, Alex can:Verify the connection
Example prompts
Troubleshooting
Org Not Allowed
Org Not Allowed
Splunk’s
-32005. Your organization is not authorized to reach the Observability MCP tools, and no credential or region change fixes it. Ask your Splunk administrator.Session Expired
Session Expired
Splunk’s
-32003. The access token is invalid or expired. Check the Access token field, then confirm the token was issued in the organization your Realm names.Not Found
Not Found
Splunk’s
404. The MCP endpoint is not served at the address used, so change the Gateway region to the one that corresponds to your realm.The connection reports no tools at all
The connection reports no tools at all
Expect this from a token that does not match the realm: rather than failing, the gateway can answer with an empty tool set. Reissue the token in the organization that owns your realm.
The connection stopped working after about a month
The connection stopped working after about a month
Both credentials expire in 30 days by default, and a user API access token also expires the moment you log out. Create a new org access token with a longer expiration, or rotate the existing one.
An answer comes back empty
An answer comes back empty
Empty is often the true state: an organization with no APM instrumentation reports zero environments, and a catalogued metric can simply be idle. The agent says when a search did not finish instead of reporting zero, and trace search can outrun the time budget for one call. In either case, ask again over a different time range.
A metric name comes back as not found
A metric name comes back as not found
The catalog search matches loosely and returns only a few names per call, so a name you know exists can be missed. Ask for the exact name, or search with different terms.
Security
- Least privilege — grant only the permissions the agents need for your use case; start read-only and widen later.
- Read-only by default — use read-only credentials unless you want agents to make changes through this connection.
- Rotate credentials — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
- Revoke on offboarding — remove the credential at the provider when you delete a connection or a teammate leaves.
- Least privilege by role — an API-scope token with
read_onlyis the narrowest credential that runs every tool here, and Splunk enforces it server-side. - The gateway region is a data path — your requests and their results travel through the region you select, so choose it deliberately when data residency matters. The realm and region are routing values, not credentials.
Related
Honeycomb Connection
Trace queries, BubbleUp root-cause analysis, and approval-gated board and trigger changes
Dynatrace Connection
DQL queries, problem investigation, and vulnerability review