Galileo Lemonade Instrumentation Setup

Galileo Lemonade Instrumentation Setup

Prerequisites

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Adding Galileo OTLP fan-out to a Lemonade collector, capturing agent/workflow/tool traces around Lemonade, avoiding duplicate LLM records, or validating privacy-safe Galileo ingestion from an AMD Ryzen AI host.
• Preview and review the galileo lemonade instrumentation setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Purpose

Use this skill after $lemonade-splunk-otel has established a healthy, privacy-safe Lemonade-to-Splunk trace path. It adds Galileo in one of two explicit modes and proves delivery by backend readback.

The tested baseline is Lemonade v10.10 with Splunk OTel Collector v0.156; Galileo SaaS and Enterprise still require an exact tenant intake endpoint. Run the tools from an isolated Python environment containing requirements-dev.txt (PyYAML 6.x). Rendering is safe offline; production validation also requires the exact installed collector binary.

This skill does not treat Lemonade's native inference span as a complete AI agent trace. Lemonade v10.10 creates one root SERVER span per request, does not extract traceparent, and has no workflow/tool hierarchy.

Required Intake

Ask for and record the exact Galileo instance console URL. A copied console link may include a navigation route after the host. Validate that exact link with --galileo-console-url, but pass only its reported HTTPS origin to $galileo-platform-setup. For example, https://console.example.invalid/tenant-navigation normalizes to https://console.example.invalid/ for endpoint derivation. Treat the path as navigation context; never infer a project or Log stream from it. The demo-v2 deployment used by the production example starts at https://console.demo-v2.galileocloud.io/; still require and validate the operator's full navigation URL instead of silently assuming that instance.

Resolve the API base, exact OTLP traces endpoint, project, and Log stream independently. The renderer reports an endpoint candidate but keeps the runtime value in GALILEO_OTLP_TRACES_ENDPOINT until tenant validation.

Choose One Galileo Source

Do not send both native and caller-instrumented LLM spans to the same Galileo Log stream. They have unrelated trace IDs and appear as duplicate observations. If temporary comparison is required, use separate Log streams and label it.

Direct Lemonade-to-Galileo is technically possible but is not the default: the server has one OTLP destination, so it displaces Splunk and exposes the Galileo key to the Lemonade service.

Safety Contract

• Validate the exact Galileo console URL, normalize a copied navigation link to its origin, and do not assume public Galileo Cloud.
• Use $galileo-platform-setup for tenant readiness and project/Log stream lifecycle. Prefer immutable project and Log stream IDs after discovery.
• Require the exact tenant-supported traces_endpoint. Current Galileo docs show both /otel/v1/traces for raw OTLP POST and /otel/traces for several exporter/SDK integrations. Do not derive one from the client type or let the collector append a path.
• Stock Collector v0.156 follows HTTP redirects and can copy custom headers to the redirect target. Pin the endpoint to a separately discovered GALILEO_EXPECTED_ORIGIN, then route only otlp_http/galileo_lemonade through the literal Collector v0.156 proxy_url field. The proxy must be a dedicated loopback tinyproxy with FilterDefaultDeny Yes; its only filter rule is the anchored, regex-escaped exact host derived from that pinned origin. Production revalidates protected binary/config/filter path identity, inode, metadata, and SHA-256, then runs bounded credential-free CONNECT probes that require HTTP 403 for an unlisted host and 2xx for the exact Galileo host. All Splunk exporters remain direct. Ambient proxy variables are stripped and are never part of this contract. Install evidence as root-owned, collector-group-readable mode 0440; a root 0400 file cannot be read by the non-root collector wrapper.
• Use the current collector component type otlp_http, not legacy otlphttp. This skill names its instance otlp_http/galileo_lemonade so it cannot overwrite another application's Galileo exporter.
• Keep Galileo credentials in a dedicated service-user-owned 0600 file (root-owned only when the collector runs as root). Generated YAML contains only ${env:...} placeholders.
• Use scripts/galileo_bootstrap_transaction.py when a broad bootstrap key must create or adopt the exact target and mint the project-scoped runtime key. The bootstrap secret is accepted only from a current-user-owned, single-link protected file; it is never accepted through argv or the environment. Keep the private journal and one-time runtime-key output until the transaction is finalized or rolled back.
• Bootstrap stops at RUNTIME_KEY_CREATED. Never revoke the old key in that invocation. Record fresh, exact cutover evidence and run the separate finalize command with a distinct reviewed unscoped revoker credential only after host cutover, Galileo API trace/hierarchy and privacy readback, and unchanged Splunk backend readback all pass. Console UI review is not inferred from API evidence and is not a revocation gate.
• Use GALILEO_API_KEY_FILE with the packaged collector runtime wrapper. Do not source a plaintext key into an interactive shell or store it in the non-secret collector environment file.
• Pin the collector the wrapper may exec with GALILEO_COLLECTOR_BINARY and GALILEO_COLLECTOR_BINARY_SHA256. Exec mode requires both and fails closed without them, before the API key is read. Exec mode also requires Linux and refuses to run elsewhere, because the binary and every ancestor directory must be proven root-owned, link-free, and not group/other-writable. Recompute the digest after every collector package change.
• Render one complete config from the live base, review its diff, validate with the exact installed collector binary, back up, then apply transactionally.
• Keep both Lemonade and client OTLP receivers loopback-bound.
• Treat service.name filtering as classification, not authentication. Use server-fanout only within a loopback/single-host trust boundary and accept its shared-receiver replay risk explicitly during production validation.
• Use the persistent Galileo queue for production. The memory queue is an accepted-loss development option and cannot pass --production validation.
• Bind every persistent queue to the SHA-256 of its validated endpoint and selector pair. The queue directory's final component must be that fingerprint; never reuse, rename, or copy it to a different destination.
• Delete every galileo.* resource, span, and event attribute immediately before the Galileo exporter so in-band project, Log stream, experiment, or dataset fields cannot override the fixed exporter headers.
• Never disable content hiding without explicit approval of every backend that will receive the affected pipeline.

Workflow

1. Read reference.md, then load the architecture, application, or validation reference required by the chosen mode.

2. Run $lemonade-splunk-otel discovery and confirm the existing Splunk path, privacy flags, native trace pipeline name, collector config, and receiver.

3. Run $galileo-platform-setup readiness for the user-confirmed instance. Put the existing bootstrap key in a protected file without printing it and identify its exact API-key ID. For read-only inventory, resolve immutable project and Log stream IDs without creating objects:

   python3 skills/galileo-lemonade-instrumentation-setup/scripts/galileo_target_discovery.py \
     --api-base "$GALILEO_API_BASE" \
     --api-key-file "$GALILEO_API_KEY_FILE" \
     --api-key-header Splunk-AO-API-Key
   

   Filter with exact --project-name or --project-id when the tenant has many projects. Current v2 project/read APIs document Splunk-AO-API-Key; OTLP ingest separately uses Galileo-API-Key. Confirm the selected Log stream before rendering.

   When the target or project-scoped runtime key must be created, use the phased bootstrap transaction in references/runtime-credentials.md. Existing targets require explicit adoption, preferably by exact IDs. The default candidate role is annotator; it is accepted only if a live API probe proves exact project-only visibility and log_data. If that probe fails, roll back the exact owned key and start a new transaction before trying editor; do not claim either role is least privilege without the live permission proof.

4. State the selected Galileo source and content policy before rendering.

5. With the confirmed endpoint, expected origin, and exactly one selector pair set in the protected runtime environment, calculate the non-secret destination fingerprint. This command emits only the lowercase digest and does not require or print the API key:

   python3 skills/galileo-lemonade-instrumentation-setup/scripts/collector_runtime_wrapper.py \
     --print-destination-fingerprint
   

   Record it as GALILEO_DESTINATION_FINGERPRINT, and set GALILEO_QUEUE_STORAGE_DIRECTORY to a new private directory ending with that exact digest.

   Pin the collector the wrapper may exec. Record the reviewed binary as GALILEO_COLLECTOR_BINARY and its digest as GALILEO_COLLECTOR_BINARY_SHA256, read from the installed file itself, not from a vendor download page:

   sha256sum /usr/bin/otelcol
   

   Render from the existing full collector config:

   bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh \
     --galileo-console-url "$GALILEO_CONSOLE_URL" \
     --base /etc/otel/collector/agent_config.yaml \
     --output /tmp/lemonade-galileo-agent_config.yaml \
     --mode client-fanout \
     --routing ids \
     --galileo-proxy-url http://127.0.0.1:18888 \
     --queue-policy persistent \
     --production \
     --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \
     --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY"
   

6. Review the diff, then apply four independently journaled layers in dependency order: create the destination-fingerprinted directory with transactional_queue_directory.py; install and probe the dedicated tinyproxy package/config/filter/unit with transactional_proxy_bundle.py; render protected proxy identity evidence from those installed assets; and install the routing environment, evidence, wrapper, key, and drop-in with transactional_runtime_bundle.py. Follow references/queue-directory-transaction.md, references/proxy-bundle-transaction.md, references/runtime-bundle-transaction.md, after reading the underlying credential contract in references/runtime-credentials.md. Do not apply the collector YAML yet. Start the proxy and run the wrapper's --check; it must prove the protected assets plus credential-free live allow/deny probes. Then validate the staged YAML statically and with the installed collector:

   Use $lemonade-splunk-otel's value-free config_change_summary.py first; PyYAML normalizes formatting and comments, so retain the exact source backup.

   The production validator inspects service-owned queue files and collector-group-readable proxy evidence. Run the command under the exact Collector UID, primary GID, and supplementary groups, while inheriting the protected systemd environment without copying secret values into argv. Do not run it as root: root's group set is not proof that the Collector can read or safely own those assets.

   # Execute as the discovered Collector service identity, not as root.
   bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh \
     --collector-config /tmp/lemonade-galileo-agent_config.yaml \
     --mode client-fanout \
     --queue-policy persistent \
     --production \
     --galileo-proxy-url http://127.0.0.1:18888 \
     --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \
     --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY" \
     --collector-binary /usr/bin/otelcol
   

7. Back up the live collector config and service state, preserving the exact original command/arguments. After the three prerequisite transactions and service-identity validation pass, use the baseline skill's SHA-gated transactional apply helper for the validated YAML and restart, with /etc/splunk-otel-collector/lemonade-agent-config.yaml as the live config path pinned by the runtime manifest. The wrapper validates endpoint/origin, selectors, destination fingerprint, proxy assets/live probes, and queue before loading the protected key only in the collector child. Restore the collector YAML transaction immediately if Splunk regresses, then restore the exact Collector YAML manifest, runtime manifest, proxy manifest, and queue manifest in that reverse order. Queue restore is last and quarantines nonempty or uncertain data; do not roll back only one proxy/runtime file or delete a queue database manually.

8. For client-fanout, adapt skills/galileo-lemonade-instrumentation-setup/assets/lemonade_openinference_client.py. It sends to the dedicated loopback receiver and keeps Galileo credentials out of the application. Install its pinned requirements in an isolated environment, then run lemonade_openinference_client.py --check under the application identity before sending the real canary.

9. Send the synthetic canary through the receiver selected by the mode. A receiver success is only pipeline evidence; continue to Galileo readback.

10. Follow references/validation.md, including a real non-sensitive Lemonade request, collector counter deltas, Galileo API trace/hierarchy readback, privacy assertions, and unchanged Splunk backend readback. Perform Console trace-shape review only when a signed-in browser is actually available, and report its absence truthfully.

11. Build a fresh schema-v2 cutover document from the actual host, Galileo API, and Splunk backend results. Record it with the transaction's record-cutover-evidence command. Omit console_review or record only {"status":"not_observed"}; any signed-in UI review is separate evidence.

12. In a distinct invocation, run finalize with the same protected bootstrap file plus a distinct protected unscoped revoker file and its exact key ID. It revalidates the evidence and runtime key, verifies that the revoker is neither the old nor runtime key, revokes only the bound old key ID through that revoker, reconciles full-inventory absence, requires the old key to return 401 Unauthorized from both /v2/current_user and /v2/token, rechecks the runtime key, and reaches FINALIZED. Before revocation, rollback deletes only exact transaction-owned IDs and preserves adopted targets. The separately documented reconcile-legacy-revocation command is only for the exact retired, already-started self-delete journal schema; it performs no DELETE and must never replace the fresh revoker policy.

Rendered Collector Shapes

server-fanout leaves the shared native traces pipeline's exporters unchanged and adds traces/lemonade_galileo_server. The new branch shares the reviewed receiver set but fail-closed filters on Lemonade's native service.name=lemonade-server before the Galileo-only exporter and preserves the baseline Lemonade deployment/privacy transform. The attribute filter does not establish provenance; production validation rejects externally bound receivers.

client-fanout keeps the native traces pipeline Splunk-only and creates:

OpenAI-compatible caller
  -> 127.0.0.1:14318/v1/traces
  -> traces/lemonade_galileo_client
  -> otlp_http/galileo_lemonade

This default avoids duplicate LLM/cost records in both backends and permits a caller-specific Galileo content policy. --mirror-client-to-native-exporters is an explicit exception for users who want the richer caller hierarchy in Splunk too; it requires --allow-client-mirror during validation and an approved duplicate/content-handling plan.

The renderer inherits only baseline memory_limiter, resource_detection (or resourcedetection when that is the live distro ID), and batch processor types. Use repeated --client-processor only after reviewing the component for source-specific filters, transforms, and content expansion; validation then requires --allow-custom-client-processors. Every custom processor must precede the managed client privacy transform. The privacy transform and Galileo route guard are the final two non-batch processors, followed only by a terminal batch suffix or direct export; rendering and validation fail if any processor could mutate spans after them. The managed resource processor runs after inherited resource detection and upserts the dedicated client service.name; validation rejects an insert action that could retain the host application's identity.

Both modes redact error status text; client mode also deletes known content-bearing span/event attributes—including multimodal message payloads and message-level function/tool arguments and results—while retaining roles, models, providers, structural IDs, and content types. It deletes user.id and arbitrary tags/metadata, but retains opaque session.id for session grouping; never put personal data in that ID. It also redacts exception messages and stack traces. Agent/Tool input.value, output.value, and tool call arguments/results are restored only as the constant [REDACTED] after deletion so required hierarchy fields remain without source content. Source-side OpenInference hiding remains mandatory as defense in depth, with GenAI semantic convention duplication explicitly disabled in the reference client. Lemonade v10.10's native hide_outputs does not cover status.message, so removing these transforms can expose an error that echoes sensitive content.

The renderer strips only an exact recognized prior render before adding the requested mode, so repeated renders and mode switches are deterministic. It preserves unrelated components but fails closed on managed drift, foreign references, every custom Galileo-shaped or Galileo-named exporter/route in all modes, and any extra pipeline sharing the dedicated client receiver.

Application Choices

• OpenInference + standard OpenAI instrumentation: preferred for sync/async, streaming, OTel context propagation, collector fan-out, and strong hide controls. The packaged client demonstrates this path.
• Galileo OpenAI wrapper: smallest synchronous change and supports OpenAI-compatible model servers, but captures raw input/output by default.
• Galileo OpenAI Agents tracing processor: best when the application actually uses OpenAI Agents and needs generations, tools, and handoffs.
• Galileo manual logger or @log: use for custom agent/framework semantics or explicit redacted fields.

Read references/application-instrumentation.md before choosing a client library. Content capture is opt-in.

Completion Gate

Report all of the following:

• chosen mode and why the other source is excluded;
• exact Galileo instance/API endpoint and project/Log stream IDs, with secrets omitted;
• Lemonade version, health, semantics, and privacy flags;
• collector config validation, loopback binds, service health, exact accepted/failed/refused and sent/send-failed/enqueue-failed deltas, queue/in-flight state, and unchanged Splunk readback;
• Galileo API readback proving the expected trace, Agent/Tool/LLM hierarchy, and privacy state. Report signed-in Console confirmation separately when it was actually observed; otherwise report it as not observed;
• bootstrap transaction phase and sanitized IDs, with FINALIZED required only after the separate fresh-evidence revocation gate;
• backup and tested rollback path.

Production completion still requires live Galileo backend readback of the Agent/Tool/LLM hierarchy. Static validation, collector acceptance, and an empty or healthy queue do not substitute for that deployment gate.

HTTP 200 alone is insufficient: OTLP responses can contain partialSuccess.rejectedSpans, and collector counters do not prove backend storage.