Splunk Observability Thousandeyes Integration

Splunk Observability ThousandEyes Integration

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Wire ThousandEyes telemetry into Splunk Observability Cloud, configure Integrations 2.0 APM trace linking, manage TE tests/alert rules/templates for an O11y integration, or produce the per-test-type O11y dashboards.
• Preview and review the splunk observability thousandeyes integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-thousandeyes-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-thousandeyes-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

This is a generalized TE -> Splunk Observability Cloud skill, NOT tied to any one demo. A private RTSP/UDP/RTP demo repo was used during initial development to validate the TE Streams API + Integrations 2.0 mechanics; that demo's test taxonomy is demo-specific and is NOT carried into this skill. Source of truth: the public ThousandEyes for OpenTelemetry Data Model v2 (docs.thousandeyes.com/.../opentelemetry/data-model/data-model-v2/metrics) and the TE API v7 schemas (developer.cisco.com/docs/thousandeyes/).

Three TE-side surfaces

1. Integration 1.0 OpenTelemetry stream — POST /v7/streams with type=opentelemetry, signal=metric|trace|log (default metric), endpointType=http|grpc, streamEndpointUrl=https://ingest.<realm>.signalfx.com/v2/datapoint/otlp, customHeaders.X-SF-Token, dataModelVersion=v2, testMatch[], optional filters.testTypes[].
2. Integrations 2.0 Splunk Observability APM connector — generic connector targeting https://api.<realm>.signalfx.com with X-SF-Token; assigned to the splunk-observability-apm operation for trace linking.
3. TE assets — render across the canonical taxonomy, with apply limited to operations that have ID-based readback:
   • Tests: POST /v7/tests/{type} for http-server, page-load, web-transactions, api, agent-to-server, agent-to-agent, bgp, dns-server, dns-trace, dnssec, sip-server, voice, ftp-server.
   • Alert Rules: POST /v7/alerts/rules aligned with the SignalFlow detector specs we ship for O11y.
   • Labels, Tags, and TE-side Dashboards are rendered for operator/API handoff; their generated apply scripts fail closed before mutation.
   • TE Templates: verified POST /v7/templates creation with Handlebars-only credential placeholders. Deployment remains a TE UI handoff because template-resource readback does not prove deploy completion and cannot make an interrupted deploy POST retry-safe.

Out of scope (handed off)

• Splunk Platform ta_cisco_thousandeyes add-on -> cisco-thousandeyes-setup.
• ThousandEyes MCP Server registration with Cursor / Claude / Codex / VS Code / Kiro -> cisco-thousandeyes-mcp-setup.
• TE Enterprise Agent K8s/VM deployment.
• O11y dashboard apply -> splunk-observability-dashboard-builder.
• O11y detector apply -> splunk-observability-native-ops.
• signal=log and signal=trace deep targets — render the payload shape and document that O11y's /v2/datapoint/otlp endpoint is metrics-only.

For the Splunk TA or any dashboard-companion handoff, follow the shared TA completion gate: package installation alone is not completion; validate ingest and dashboard visibility, macro alignment, and data, or record evidence that no pre-built dashboard ships.

Safety Rules

• Never ask for the ThousandEyes API token, the Splunk Observability ingest token, or the Splunk Observability API token in conversation.
• Never pass any token on the command line or as an environment-variable prefix.
• Use file-based secret flags only:
  • --te-token-file for the TE bearer token (used for Streams, Tests, Alert Rules, Templates, Dashboards).
  • --o11y-ingest-token-file for the Splunk Observability Org access token with ingest authorization (used as X-SF-Token in the OTLP metric stream customHeaders).
  • --o11y-api-token-file for the Splunk Observability User API access token (used as X-SF-Token in the Integrations 2.0 APM connector and SignalFlow validate calls).
• Reject every direct token flag (--te-token, --access-token, --token, --bearer-token, --api-token, --o11y-token, --sf-token).
• Token files must be single-link, non-symlink regular files containing exactly one non-empty UTF-8 line and must be chmod 600. Live validators open them with no-follow descriptor checks. There is no permission bypass.
• TE Templates render with Handlebars placeholders only — TE API rejects plain-text credentials with HTTP 400.
• Apply scripts read token files at runtime through the fixed-origin HTTPS client; the renderer never reads token files. Every TE request is scoped with the rendered numeric account_group_id as ?aid=.

Primary Workflow

1. Collect non-secret values: realm (us0/us1/eu0/...), account group ID, list of TE test IDs or test types to include, optional alert rules / labels / tags / dashboards / templates.

2. Create or update a YAML/JSON spec from template.example. Spec supports test selection via:

   • explicit test_match[] (list of {id, domain: cea|endpoint}),
   • filters.test_types[] (any of the canonical TE OTel v2 types),
   • or mode: all (stream every enabled test).

3. Render and validate:

   bash skills/splunk-observability-thousandeyes-integration/scripts/setup.sh \
     --render \
     --validate \
     --spec skills/splunk-observability-thousandeyes-integration/template.example \
     --output-dir splunk-observability-thousandeyes-rendered
   

4. Review splunk-observability-thousandeyes-rendered/:

   • .splunk-observability-thousandeyes-bundle.json — private exclusive-root marker required before any managed directory can be cleaned on rerender.
   • te-payloads/ — request bodies for POST/PUT /v7/streams, connector + APM operation, per-test JSON, alert rules, labels, tags, TE dashboards, templates.
   • dashboards/ — one SignalFlow spec per selected test type (consumable by splunk-observability-dashboard-builder).
   • detectors/ — starter detector specs (consumable by splunk-observability-native-ops).
   • scripts/ — per-step apply scripts, list helpers, SignalFlow validation, hand-off drivers.
   • metadata.json — non-secret plan summary.

5. Apply only when explicitly requested:

   bash skills/splunk-observability-thousandeyes-integration/scripts/setup.sh \
     --apply stream,apm,tests,alert_rules,templates \
     --spec my-integration.yaml \
     --te-token-file /secure/path/te-token \
     --o11y-ingest-token-file /secure/path/o11y-ingest-token \
     --o11y-api-token-file /secure/path/o11y-api-token \
     --i-accept-te-mutations
   

   To apply only a subset:

   bash skills/splunk-observability-thousandeyes-integration/scripts/setup.sh \
     --apply stream,apm \
     --spec my-integration.yaml \
     --te-token-file /secure/path/te-token \
     --o11y-ingest-token-file /secure/path/o11y-ingest-token \
     --o11y-api-token-file /secure/path/o11y-api-token \
     --i-accept-te-mutations
   

   Every live apply requires an explicit section list (or all) and --i-accept-te-mutations. all means the currently automatable sections: stream, APM, tests, alert rules, and templates. Labels, tags, and TE dashboards remain render-only and fail closed if selected.

   Successful creates retain server-returned IDs under the rendered output's mode-700 state/ directory and verify them by collection/item readback. Each logical object has a private cross-process lock, and an fsynced in_progress intent is written before POST. A missing ID, failed exact readback, or uncertain transport becomes ambiguous with manual_reconcile: true; later applies block before any second POST. Never delete or hand-edit ambiguous state to force a retry—reconcile the live TE object and retained ID first.

   Rerender preserves state/ intentionally. It only cleans generated directories after validating the exclusive marker, canonical bundle root, expected top-level layout, same-filesystem trees, and single-link regular files. Root, home, repository-root, unmarked non-empty, symlinked, and hardlinked output targets fail without recursive deletion.

Per-test-type metric coverage (TE OpenTelemetry Data Model v2)

All charts are filtered by thousandeyes.account.id and thousandeyes.test.id.

Hand-offs

• Dashboards: scripts/handoff-dashboards.sh emits the exact splunk-observability-dashboard-builder invocation.
• Detectors: scripts/handoff-detectors.sh emits the exact splunk-observability-native-ops invocation.
• TE MCP registration: scripts/handoff-mcp.sh emits the cisco-thousandeyes-mcp-setup invocation.
• Splunk Platform TA: scripts/handoff-ta.sh emits the cisco-thousandeyes-setup invocation.

See reference.md for option details and the references/ annexes for the per-test-type catalog, TE Templates, alert rules, Integrations 2.0 APM, dashboards catalog, and SignalFlow validation.