Galileo Lemonade Instrumentation Setup
Prerequisites
| Tool or access |
Purpose |
Verify |
| Bash and Python 3 |
Run bundled setup and validation helpers |
bash --version && python3 --version |
| Required product/platform access |
Inspect or configure the selected target |
Complete the documented preflight |
| Credential files for live modes |
Keep secrets out of chat |
Verify paths only |
Workflow Overview
┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘
When to Activate
- Adding Galileo OTLP fan-out to a Lemonade collector, capturing agent/workflow/tool traces around Lemonade,
avoiding duplicate LLM records, or validating privacy-safe Galileo ingestion from an AMD Ryzen AI host.
- Preview and review the galileo lemonade instrumentation setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.
Scope
Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.
Examples
Inspect the supported setup modes before selecting one:
bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh --help
Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.
Inspect validation modes before running completion checks:
bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh --help
Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.
Troubleshooting
| Issue |
Cause |
Resolution |
| Preflight fails |
A required tool or access path is missing |
Resolve it before rendering or applying |
| Rendered assets are incomplete |
Required non-secret inputs are absent |
Complete intake and render again |
| Apply is blocked |
Review, credentials, or explicit acceptance is missing |
Use the documented handoff |
| Validation is incomplete |
Live evidence is unavailable |
Record the gap and keep completion open |
Purpose
Use this skill after $lemonade-splunk-otel has established a healthy,
privacy-safe Lemonade-to-Splunk trace path. It adds Galileo in one of two
explicit modes and proves delivery by backend readback.
The tested baseline is Lemonade v10.10 with Splunk OTel Collector v0.156;
Galileo SaaS and Enterprise still require an exact tenant intake endpoint.
Run the tools from an isolated Python environment containing
requirements-dev.txt (PyYAML 6.x). Rendering is safe offline; production
validation also requires the exact installed collector binary.
This skill does not treat Lemonade's native inference span as a complete AI
agent trace. Lemonade v10.10 creates one root SERVER span per request, does not
extract traceparent, and has no workflow/tool hierarchy.
Required Intake
Ask for and record the exact Galileo instance console URL. A copied console
link may include a navigation route after the host. Validate that exact link
with --galileo-console-url, but pass only its reported HTTPS origin to
$galileo-platform-setup. For example,
https://console.example.invalid/tenant-navigation normalizes to
https://console.example.invalid/ for endpoint derivation. Treat the
path as navigation context; never infer a project or Log stream from it.
The demo-v2 deployment used by the production example starts at
https://console.demo-v2.galileocloud.io/; still require and validate the
operator's full navigation URL instead of silently assuming that instance.
Resolve the API base, exact OTLP traces endpoint, project, and Log stream
independently. The renderer reports an endpoint candidate but keeps the
runtime value in GALILEO_OTLP_TRACES_ENDPOINT until tenant validation.
Choose One Galileo Source
| Requirement |
Mode |
Galileo receives |
Privacy consequence |
| No application change; model/latency/token metadata |
server-fanout |
Lemonade's native LLM root spans |
With Lemonade hide flags on, input/output/thinking stay [REDACTED]; content evaluators are limited. |
| Agent, workflow, tool, session, async, or selective evaluation context |
client-fanout |
Caller-side OpenInference/OTel spans, Galileo-only by default |
Recommended for actual agent observability. Native Lemonade remains redacted and Splunk-only. |
| Remove Galileo routing without disturbing Splunk |
splunk-only |
Nothing |
Rollback/render cleanup mode. |
Do not send both native and caller-instrumented LLM spans to the same Galileo
Log stream. They have unrelated trace IDs and appear as duplicate observations.
If temporary comparison is required, use separate Log streams and label it.
Direct Lemonade-to-Galileo is technically possible but is not the default: the
server has one OTLP destination, so it displaces Splunk and exposes the
Galileo key to the Lemonade service.
Safety Contract
- Validate the exact Galileo console URL, normalize a copied navigation link
to its origin, and do not assume public Galileo Cloud.
- Use
$galileo-platform-setup for tenant readiness and project/Log stream
lifecycle. Prefer immutable project and Log stream IDs after discovery.
- Require the exact tenant-supported
traces_endpoint. Current Galileo docs
show both /otel/v1/traces for raw OTLP POST and /otel/traces for several
exporter/SDK integrations. Do not derive one from the client type or let the
collector append a path.
- Stock Collector v0.156 follows HTTP redirects and can copy custom headers to
the redirect target. Pin the endpoint to a separately discovered
GALILEO_EXPECTED_ORIGIN, then route only otlp_http/galileo_lemonade
through the literal Collector v0.156 proxy_url field. The proxy must be a
dedicated loopback tinyproxy with FilterDefaultDeny Yes; its only filter
rule is the anchored, regex-escaped exact host derived from that pinned
origin. Production revalidates protected binary/config/filter path identity,
inode, metadata, and SHA-256, then runs bounded credential-free CONNECT
probes that require HTTP 403 for an unlisted host and 2xx for the exact
Galileo host. All Splunk exporters remain direct. Ambient proxy variables
are stripped and are never part of this contract. Install evidence as
root-owned, collector-group-readable mode 0440; a root 0400 file cannot
be read by the non-root collector wrapper.
- Use the current collector component type
otlp_http, not legacy
otlphttp. This skill names its instance otlp_http/galileo_lemonade so it
cannot overwrite another application's Galileo exporter.
- Keep Galileo credentials in a dedicated service-user-owned
0600 file
(root-owned only when the collector runs as root).
Generated YAML contains only ${env:...} placeholders.
- Use
scripts/galileo_bootstrap_transaction.py when a broad bootstrap key
must create or adopt the exact target and mint the project-scoped runtime
key. The bootstrap secret is accepted only from a current-user-owned,
single-link protected file; it is never accepted through argv or the
environment. Keep the private journal and one-time runtime-key output until
the transaction is finalized or rolled back.
- Bootstrap stops at
RUNTIME_KEY_CREATED. Never revoke the old key in that
invocation. Record fresh, exact cutover evidence and run the separate
finalize command with a distinct reviewed unscoped revoker credential only
after host cutover, Galileo API trace/hierarchy and
privacy readback, and unchanged Splunk backend readback all pass. Console UI
review is not inferred from API evidence and is not a revocation gate.
- Use
GALILEO_API_KEY_FILE with the packaged collector runtime wrapper. Do
not source a plaintext key into an interactive shell or store it in the
non-secret collector environment file.
- Pin the collector the wrapper may exec with
GALILEO_COLLECTOR_BINARY and
GALILEO_COLLECTOR_BINARY_SHA256. Exec mode requires both and fails closed
without them, before the API key is read. Exec mode also requires Linux and
refuses to run elsewhere, because the binary and every ancestor directory
must be proven root-owned, link-free, and not group/other-writable.
Recompute the digest after every collector package change.
- Render one complete config from the live base, review its diff, validate with
the exact installed collector binary, back up, then apply transactionally.
- Keep both Lemonade and client OTLP receivers loopback-bound.
- Treat
service.name filtering as classification, not authentication. Use
server-fanout only within a loopback/single-host trust boundary and accept
its shared-receiver replay risk explicitly during production validation.
- Use the persistent Galileo queue for production. The memory queue is an
accepted-loss development option and cannot pass
--production validation.
- Bind every persistent queue to the SHA-256 of its validated endpoint and
selector pair. The queue directory's final component must be that fingerprint;
never reuse, rename, or copy it to a different destination.
- Delete every
galileo.* resource, span, and event attribute immediately
before the Galileo exporter so in-band project, Log stream, experiment, or
dataset fields cannot override the fixed exporter headers.
- Never disable content hiding without explicit approval of every backend that
will receive the affected pipeline.
Workflow
Read reference.md, then load the architecture, application,
or validation reference required by the chosen mode.
Run $lemonade-splunk-otel discovery and confirm the existing Splunk path,
privacy flags, native trace pipeline name, collector config, and receiver.
Run $galileo-platform-setup readiness for the user-confirmed instance.
Put the existing bootstrap key in a protected file without printing it and
identify its exact API-key ID. For read-only inventory, resolve immutable
project and Log stream IDs without creating objects:
python3 skills/galileo-lemonade-instrumentation-setup/scripts/galileo_target_discovery.py \
--api-base "$GALILEO_API_BASE" \
--api-key-file "$GALILEO_API_KEY_FILE" \
--api-key-header Splunk-AO-API-Key
Filter with exact --project-name or --project-id when the tenant has
many projects. Current v2 project/read APIs document
Splunk-AO-API-Key; OTLP ingest separately uses Galileo-API-Key.
Confirm the selected Log stream before rendering.
When the target or project-scoped runtime key must be created, use the
phased bootstrap transaction in
references/runtime-credentials.md.
Existing targets require explicit adoption, preferably by exact IDs. The
default candidate role is annotator; it is accepted only if a live API
probe proves exact project-only visibility and log_data. If that probe
fails, roll back the exact owned key and start a new transaction before
trying editor; do not claim either role is least privilege without the
live permission proof.
State the selected Galileo source and content policy before rendering.
With the confirmed endpoint, expected origin, and exactly one selector pair
set in the protected runtime environment, calculate the non-secret
destination fingerprint. This command emits only the lowercase digest and
does not require or print the API key:
python3 skills/galileo-lemonade-instrumentation-setup/scripts/collector_runtime_wrapper.py \
--print-destination-fingerprint
Record it as GALILEO_DESTINATION_FINGERPRINT, and set
GALILEO_QUEUE_STORAGE_DIRECTORY to a new private directory ending with
that exact digest.
Pin the collector the wrapper may exec. Record the reviewed binary as
GALILEO_COLLECTOR_BINARY and its digest as
GALILEO_COLLECTOR_BINARY_SHA256, read from the installed file itself, not
from a vendor download page:
sha256sum /usr/bin/otelcol
Render from the existing full collector config:
bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh \
--galileo-console-url "$GALILEO_CONSOLE_URL" \
--base /etc/otel/collector/agent_config.yaml \
--output /tmp/lemonade-galileo-agent_config.yaml \
--mode client-fanout \
--routing ids \
--galileo-proxy-url http://127.0.0.1:18888 \
--queue-policy persistent \
--production \
--destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \
--queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY"
Review the diff, then apply four independently journaled layers in dependency
order: create the destination-fingerprinted directory with
transactional_queue_directory.py; install and probe the dedicated
tinyproxy package/config/filter/unit with transactional_proxy_bundle.py;
render protected proxy identity evidence from those installed assets; and
install the routing environment, evidence, wrapper, key, and drop-in with
transactional_runtime_bundle.py. Follow
references/queue-directory-transaction.md,
references/proxy-bundle-transaction.md,
references/runtime-bundle-transaction.md,
after reading the underlying credential contract in
references/runtime-credentials.md.
Do not apply the collector YAML yet. Start the proxy and run the wrapper's
--check; it must prove the protected assets plus credential-free live
allow/deny probes. Then validate the staged YAML statically and with the
installed collector:
Use $lemonade-splunk-otel's value-free config_change_summary.py first;
PyYAML normalizes formatting and comments, so retain the exact source backup.
The production validator inspects service-owned queue files and
collector-group-readable proxy evidence. Run the command under the exact
Collector UID, primary GID, and supplementary groups, while inheriting the
protected systemd environment without copying secret values into argv. Do
not run it as root: root's group set is not proof that the Collector can
read or safely own those assets.
# Execute as the discovered Collector service identity, not as root.
bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh \
--collector-config /tmp/lemonade-galileo-agent_config.yaml \
--mode client-fanout \
--queue-policy persistent \
--production \
--galileo-proxy-url http://127.0.0.1:18888 \
--destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \
--queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY" \
--collector-binary /usr/bin/otelcol
Back up the live collector config and service state, preserving the exact
original command/arguments. After the three prerequisite transactions and
service-identity validation pass, use the baseline skill's SHA-gated
transactional apply helper for the validated YAML and restart,
with /etc/splunk-otel-collector/lemonade-agent-config.yaml as the live
config path pinned by the runtime manifest.
The wrapper validates endpoint/origin, selectors, destination fingerprint,
proxy assets/live probes, and queue before loading the protected key only in
the collector child. Restore the collector YAML transaction immediately if
Splunk regresses, then restore the exact Collector YAML manifest, runtime
manifest, proxy manifest, and queue manifest in that reverse order. Queue
restore is last and quarantines nonempty or uncertain data; do not roll back
only one proxy/runtime file or delete a queue database manually.
For client-fanout, adapt
skills/galileo-lemonade-instrumentation-setup/assets/lemonade_openinference_client.py.
It sends to the dedicated
loopback receiver and keeps Galileo credentials out of the application.
Install its pinned requirements in an isolated environment, then run
lemonade_openinference_client.py --check under the application identity
before sending the real canary.
Send the synthetic canary through the receiver selected by the mode. A
receiver success is only pipeline evidence; continue to Galileo readback.
Follow references/validation.md, including a
real non-sensitive Lemonade request, collector counter deltas, Galileo API
trace/hierarchy readback, privacy assertions, and unchanged Splunk backend
readback. Perform Console trace-shape review only when a signed-in browser
is actually available, and report its absence truthfully.
Build a fresh schema-v2 cutover document from the actual host, Galileo API,
and Splunk backend results. Record it with the transaction's
record-cutover-evidence command. Omit console_review or record only
{"status":"not_observed"}; any signed-in UI review is separate evidence.
In a distinct invocation, run finalize with the same protected bootstrap
file plus a distinct protected unscoped revoker file and its exact key ID.
It revalidates the evidence and runtime key, verifies that the revoker is
neither the old nor runtime key, revokes only the bound old key ID through
that revoker, reconciles full-inventory absence, requires the old key to
return 401 Unauthorized from both /v2/current_user and /v2/token,
rechecks the runtime key, and reaches FINALIZED. Before revocation,
rollback deletes only exact transaction-owned IDs and preserves adopted
targets. The separately documented reconcile-legacy-revocation command
is only for the exact retired, already-started self-delete journal schema;
it performs no DELETE and must never replace the fresh revoker policy.
Rendered Collector Shapes
server-fanout leaves the shared native traces pipeline's exporters
unchanged and adds traces/lemonade_galileo_server. The new branch shares the
reviewed receiver set but fail-closed filters on Lemonade's native
service.name=lemonade-server before the Galileo-only exporter and preserves
the baseline Lemonade deployment/privacy transform. The attribute filter does
not establish provenance; production validation rejects externally bound
receivers.
client-fanout keeps the native traces pipeline Splunk-only and creates:
OpenAI-compatible caller
-> 127.0.0.1:14318/v1/traces
-> traces/lemonade_galileo_client
-> otlp_http/galileo_lemonade
This default avoids duplicate LLM/cost records in both backends and permits a
caller-specific Galileo content policy. --mirror-client-to-native-exporters
is an explicit exception for users who want the richer caller hierarchy in
Splunk too; it requires --allow-client-mirror during validation and an
approved duplicate/content-handling plan.
The renderer inherits only baseline memory_limiter, resource_detection
(or resourcedetection when that is the live distro ID), and batch
processor types. Use repeated --client-processor only after reviewing
the component for source-specific filters, transforms, and content expansion;
validation then requires --allow-custom-client-processors.
Every custom processor must precede the managed client privacy transform. The
privacy transform and Galileo route guard are the final two non-batch
processors, followed only by a terminal batch suffix or direct export;
rendering and validation fail if any processor could mutate spans after them.
The managed resource processor runs after inherited resource detection and
upserts the dedicated client service.name; validation rejects an insert
action that could retain the host application's identity.
Both modes redact error status text; client mode also deletes known
content-bearing span/event attributes—including multimodal message payloads and
message-level function/tool arguments and results—while retaining roles,
models, providers, structural IDs, and content types. It deletes user.id and
arbitrary tags/metadata, but retains opaque session.id for session grouping;
never put personal data in that ID. It also redacts exception messages and
stack traces. Agent/Tool input.value, output.value, and tool call
arguments/results are restored only as the constant [REDACTED] after deletion
so required hierarchy fields remain without source content. Source-side
OpenInference hiding remains mandatory as defense in depth, with GenAI semantic
convention duplication explicitly disabled in the reference client.
Lemonade v10.10's native hide_outputs does not
cover status.message, so removing these transforms can expose an error that
echoes sensitive content.
The renderer strips only an exact recognized prior render before adding the
requested mode, so repeated renders and mode switches are deterministic. It
preserves unrelated components but fails closed on managed drift, foreign
references, every custom Galileo-shaped or Galileo-named exporter/route in all
modes, and any extra pipeline sharing the dedicated client receiver.
Application Choices
- OpenInference + standard OpenAI instrumentation: preferred for sync/async,
streaming, OTel context propagation, collector fan-out, and strong hide
controls. The packaged client demonstrates this path.
- Galileo OpenAI wrapper: smallest synchronous change and supports
OpenAI-compatible model servers, but captures raw input/output by default.
- Galileo OpenAI Agents tracing processor: best when the application actually
uses OpenAI Agents and needs generations, tools, and handoffs.
- Galileo manual logger or
@log: use for custom agent/framework semantics or
explicit redacted fields.
Read references/application-instrumentation.md
before choosing a client library. Content capture is opt-in.
Completion Gate
Report all of the following:
- chosen mode and why the other source is excluded;
- exact Galileo instance/API endpoint and project/Log stream IDs, with secrets
omitted;
- Lemonade version, health, semantics, and privacy flags;
- collector config validation, loopback binds, service health, exact
accepted/failed/refused and sent/send-failed/enqueue-failed deltas,
queue/in-flight state, and unchanged Splunk readback;
- Galileo API readback proving the expected trace, Agent/Tool/LLM hierarchy,
and privacy state. Report signed-in Console confirmation separately when it
was actually observed; otherwise report it as not observed;
- bootstrap transaction phase and sanitized IDs, with
FINALIZED required
only after the separate fresh-evidence revocation gate;
- backup and tested rollback path.
Production completion still requires live Galileo backend readback of the
Agent/Tool/LLM hierarchy. Static validation, collector acceptance, and an empty
or healthy queue do not substitute for that deployment gate.
HTTP 200 alone is insufficient: OTLP responses can contain
partialSuccess.rejectedSpans, and collector counters do not prove backend
storage.
1---2name: galileo-lemonade-instrumentation-setup3description: "Use when adding Galileo OTLP fan-out to a Lemonade collector, capturing agent/workflow/tool traces4 around Lemonade, avoiding duplicate LLM records, or validating privacy-safe Galileo ingestion from an5 AMD Ryzen AI host. Instrument Lemonade Server inference and OpenAI-compatible calling applications for6 Galileo Observe while preserving Splunk OpenTelemetry delivery, privacy policy, rollback, and end-to-end7 readback."8compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."9metadata:10 source: https://skillmd.com/skills/chambear2809/galileo-lemonade-instrumentation-setup11 splunk_cloud_10_5: "not-applicable"12 compatibility_verified: "2026-08-20"13---1415# Galileo Lemonade Instrumentation Setup1617## Prerequisites1819| Tool or access | Purpose | Verify |20|---|---|---|21| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |22| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |23| Credential files for live modes | Keep secrets out of chat | Verify paths only |2425## Workflow Overview2627```text28┌───────────┐ ┌───────────────┐ ┌───────────────┐ ┌─────────────────┐29│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │30└───────────┘ └───────────────┘ └───────────────┘ └─────────────────┘31```3233## When to Activate3435- Adding Galileo OTLP fan-out to a Lemonade collector, capturing agent/workflow/tool traces around Lemonade,36 avoiding duplicate LLM records, or validating privacy-safe Galileo ingestion from an AMD Ryzen AI host.37- Preview and review the galileo lemonade instrumentation setup workflow before any live apply phase.38- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.3940## Scope4142Follow the documented read-only or render-first path whenever it is available.43This skill does not imply permission to mutate live systems. Require explicit44apply flags, protected credentials, and operator review for state changes.4546## Examples4748Inspect the supported setup modes before selecting one:4950```bash51bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh --help52```5354Expected output: usage, supported modes, and required arguments are displayed55without changing the target environment.5657Inspect validation modes before running completion checks:5859```bash60bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh --help61```6263Expected output: offline, live, and completion options are displayed when the64skill supports them; help exits without mutation.6566## Troubleshooting6768| Issue | Cause | Resolution |69|---|---|---|70| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |71| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |72| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |73| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |7475## Purpose7677Use this skill after `$lemonade-splunk-otel` has established a healthy,78privacy-safe Lemonade-to-Splunk trace path. It adds Galileo in one of two79explicit modes and proves delivery by backend readback.8081The tested baseline is Lemonade v10.10 with Splunk OTel Collector v0.156;82Galileo SaaS and Enterprise still require an exact tenant intake endpoint.83Run the tools from an isolated Python environment containing84`requirements-dev.txt` (PyYAML 6.x). Rendering is safe offline; production85validation also requires the exact installed collector binary.8687This skill does **not** treat Lemonade's native inference span as a complete AI88agent trace. Lemonade v10.10 creates one root SERVER span per request, does not89extract `traceparent`, and has no workflow/tool hierarchy.9091## Required Intake9293Ask for and record the exact Galileo instance console URL. A copied console94link may include a navigation route after the host. Validate that exact link95with `--galileo-console-url`, but pass only its reported HTTPS origin to96`$galileo-platform-setup`. For example,97`https://console.example.invalid/tenant-navigation` normalizes to98`https://console.example.invalid/` for endpoint derivation. Treat the99path as navigation context; never infer a project or Log stream from it.100The demo-v2 deployment used by the production example starts at101`https://console.demo-v2.galileocloud.io/`; still require and validate the102operator's full navigation URL instead of silently assuming that instance.103104Resolve the API base, exact OTLP traces endpoint, project, and Log stream105independently. The renderer reports an endpoint candidate but keeps the106runtime value in `GALILEO_OTLP_TRACES_ENDPOINT` until tenant validation.107108## Choose One Galileo Source109110| Requirement | Mode | Galileo receives | Privacy consequence |111|---|---|---|---|112| No application change; model/latency/token metadata | `server-fanout` | Lemonade's native LLM root spans | With Lemonade hide flags on, input/output/thinking stay `[REDACTED]`; content evaluators are limited. |113| Agent, workflow, tool, session, async, or selective evaluation context | `client-fanout` | Caller-side OpenInference/OTel spans, Galileo-only by default | Recommended for actual agent observability. Native Lemonade remains redacted and Splunk-only. |114| Remove Galileo routing without disturbing Splunk | `splunk-only` | Nothing | Rollback/render cleanup mode. |115116Do not send both native and caller-instrumented LLM spans to the same Galileo117Log stream. They have unrelated trace IDs and appear as duplicate observations.118If temporary comparison is required, use separate Log streams and label it.119120Direct Lemonade-to-Galileo is technically possible but is not the default: the121server has one OTLP destination, so it displaces Splunk and exposes the122Galileo key to the Lemonade service.123124## Safety Contract125126- Validate the exact Galileo console URL, normalize a copied navigation link127 to its origin, and do not assume public Galileo Cloud.128- Use `$galileo-platform-setup` for tenant readiness and project/Log stream129 lifecycle. Prefer immutable project and Log stream IDs after discovery.130- Require the exact tenant-supported `traces_endpoint`. Current Galileo docs131 show both `/otel/v1/traces` for raw OTLP POST and `/otel/traces` for several132 exporter/SDK integrations. Do not derive one from the client type or let the133 collector append a path.134- Stock Collector v0.156 follows HTTP redirects and can copy custom headers to135 the redirect target. Pin the endpoint to a separately discovered136 `GALILEO_EXPECTED_ORIGIN`, then route only `otlp_http/galileo_lemonade`137 through the literal Collector v0.156 `proxy_url` field. The proxy must be a138 dedicated loopback tinyproxy with `FilterDefaultDeny Yes`; its only filter139 rule is the anchored, regex-escaped exact host derived from that pinned140 origin. Production revalidates protected binary/config/filter path identity,141 inode, metadata, and SHA-256, then runs bounded credential-free CONNECT142 probes that require HTTP 403 for an unlisted host and 2xx for the exact143 Galileo host. All Splunk exporters remain direct. Ambient proxy variables144 are stripped and are never part of this contract. Install evidence as145 root-owned, collector-group-readable mode `0440`; a root `0400` file cannot146 be read by the non-root collector wrapper.147- Use the current collector component type `otlp_http`, not legacy148 `otlphttp`. This skill names its instance `otlp_http/galileo_lemonade` so it149 cannot overwrite another application's Galileo exporter.150- Keep Galileo credentials in a dedicated service-user-owned `0600` file151 (root-owned only when the collector runs as root).152 Generated YAML contains only `${env:...}` placeholders.153- Use `scripts/galileo_bootstrap_transaction.py` when a broad bootstrap key154 must create or adopt the exact target and mint the project-scoped runtime155 key. The bootstrap secret is accepted only from a current-user-owned,156 single-link protected file; it is never accepted through argv or the157 environment. Keep the private journal and one-time runtime-key output until158 the transaction is finalized or rolled back.159- Bootstrap stops at `RUNTIME_KEY_CREATED`. Never revoke the old key in that160 invocation. Record fresh, exact cutover evidence and run the separate161 `finalize` command with a distinct reviewed unscoped revoker credential only162 after host cutover, Galileo API trace/hierarchy and163 privacy readback, and unchanged Splunk backend readback all pass. Console UI164 review is not inferred from API evidence and is not a revocation gate.165- Use `GALILEO_API_KEY_FILE` with the packaged collector runtime wrapper. Do166 not source a plaintext key into an interactive shell or store it in the167 non-secret collector environment file.168- Pin the collector the wrapper may exec with `GALILEO_COLLECTOR_BINARY` and169 `GALILEO_COLLECTOR_BINARY_SHA256`. Exec mode requires both and fails closed170 without them, before the API key is read. Exec mode also requires Linux and171 refuses to run elsewhere, because the binary and every ancestor directory172 must be proven root-owned, link-free, and not group/other-writable.173 Recompute the digest after every collector package change.174- Render one complete config from the live base, review its diff, validate with175 the exact installed collector binary, back up, then apply transactionally.176- Keep both Lemonade and client OTLP receivers loopback-bound.177- Treat `service.name` filtering as classification, not authentication. Use178 `server-fanout` only within a loopback/single-host trust boundary and accept179 its shared-receiver replay risk explicitly during production validation.180- Use the persistent Galileo queue for production. The memory queue is an181 accepted-loss development option and cannot pass `--production` validation.182- Bind every persistent queue to the SHA-256 of its validated endpoint and183 selector pair. The queue directory's final component must be that fingerprint;184 never reuse, rename, or copy it to a different destination.185- Delete every `galileo.*` resource, span, and event attribute immediately186 before the Galileo exporter so in-band project, Log stream, experiment, or187 dataset fields cannot override the fixed exporter headers.188- Never disable content hiding without explicit approval of every backend that189 will receive the affected pipeline.190191## Workflow1921931. Read [reference.md](reference.md), then load the architecture, application,194 or validation reference required by the chosen mode.1952. Run `$lemonade-splunk-otel` discovery and confirm the existing Splunk path,196 privacy flags, native trace pipeline name, collector config, and receiver.1973. Run `$galileo-platform-setup` readiness for the user-confirmed instance.198 Put the existing bootstrap key in a protected file without printing it and199 identify its exact API-key ID. For read-only inventory, resolve immutable200 project and Log stream IDs without creating objects:201202 ```bash203 python3 skills/galileo-lemonade-instrumentation-setup/scripts/galileo_target_discovery.py \204 --api-base "$GALILEO_API_BASE" \205 --api-key-file "$GALILEO_API_KEY_FILE" \206 --api-key-header Splunk-AO-API-Key207 ```208209 Filter with exact `--project-name` or `--project-id` when the tenant has210 many projects. Current v2 project/read APIs document211 `Splunk-AO-API-Key`; OTLP ingest separately uses `Galileo-API-Key`.212 Confirm the selected Log stream before rendering.213214 When the target or project-scoped runtime key must be created, use the215 phased bootstrap transaction in216 [references/runtime-credentials.md](references/runtime-credentials.md).217 Existing targets require explicit adoption, preferably by exact IDs. The218 default candidate role is `annotator`; it is accepted only if a live API219 probe proves exact project-only visibility and `log_data`. If that probe220 fails, roll back the exact owned key and start a new transaction before221 trying `editor`; do not claim either role is least privilege without the222 live permission proof.2234. State the selected Galileo source and content policy before rendering.2245. With the confirmed endpoint, expected origin, and exactly one selector pair225 set in the protected runtime environment, calculate the non-secret226 destination fingerprint. This command emits only the lowercase digest and227 does not require or print the API key:228229 ```bash230 python3 skills/galileo-lemonade-instrumentation-setup/scripts/collector_runtime_wrapper.py \231 --print-destination-fingerprint232 ```233234 Record it as `GALILEO_DESTINATION_FINGERPRINT`, and set235 `GALILEO_QUEUE_STORAGE_DIRECTORY` to a new private directory ending with236 that exact digest.237238 Pin the collector the wrapper may exec. Record the reviewed binary as239 `GALILEO_COLLECTOR_BINARY` and its digest as240 `GALILEO_COLLECTOR_BINARY_SHA256`, read from the installed file itself, not241 from a vendor download page:242243 ```bash244 sha256sum /usr/bin/otelcol245 ```246247 Render from the existing full collector config:248249 ```bash250 bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh \251 --galileo-console-url "$GALILEO_CONSOLE_URL" \252 --base /etc/otel/collector/agent_config.yaml \253 --output /tmp/lemonade-galileo-agent_config.yaml \254 --mode client-fanout \255 --routing ids \256 --galileo-proxy-url http://127.0.0.1:18888 \257 --queue-policy persistent \258 --production \259 --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \260 --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY"261 ```2622636. Review the diff, then apply four independently journaled layers in dependency264 order: create the destination-fingerprinted directory with265 `transactional_queue_directory.py`; install and probe the dedicated266 tinyproxy package/config/filter/unit with `transactional_proxy_bundle.py`;267 render protected proxy identity evidence from those installed assets; and268 install the routing environment, evidence, wrapper, key, and drop-in with269 `transactional_runtime_bundle.py`. Follow270 [references/queue-directory-transaction.md](references/queue-directory-transaction.md),271 [references/proxy-bundle-transaction.md](references/proxy-bundle-transaction.md),272 [references/runtime-bundle-transaction.md](references/runtime-bundle-transaction.md),273 after reading the underlying credential contract in274 [references/runtime-credentials.md](references/runtime-credentials.md).275 Do not apply the collector YAML yet. Start the proxy and run the wrapper's276 `--check`; it must prove the protected assets plus credential-free live277 allow/deny probes. Then validate the staged YAML statically and with the278 installed collector:279280 Use `$lemonade-splunk-otel`'s value-free `config_change_summary.py` first;281 PyYAML normalizes formatting and comments, so retain the exact source backup.282283 The production validator inspects service-owned queue files and284 collector-group-readable proxy evidence. Run the command under the exact285 Collector UID, primary GID, and supplementary groups, while inheriting the286 protected systemd environment without copying secret values into argv. Do287 not run it as root: root's group set is not proof that the Collector can288 read or safely own those assets.289290 ```bash291 # Execute as the discovered Collector service identity, not as root.292 bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh \293 --collector-config /tmp/lemonade-galileo-agent_config.yaml \294 --mode client-fanout \295 --queue-policy persistent \296 --production \297 --galileo-proxy-url http://127.0.0.1:18888 \298 --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \299 --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY" \300 --collector-binary /usr/bin/otelcol301 ```3023037. Back up the live collector config and service state, preserving the exact304 original command/arguments. After the three prerequisite transactions and305 service-identity validation pass, use the baseline skill's SHA-gated306 transactional apply helper for the validated YAML and restart,307 with `/etc/splunk-otel-collector/lemonade-agent-config.yaml` as the live308 config path pinned by the runtime manifest.309 The wrapper validates endpoint/origin, selectors, destination fingerprint,310 proxy assets/live probes, and queue before loading the protected key only in311 the collector child. Restore the collector YAML transaction immediately if312 Splunk regresses, then restore the exact Collector YAML manifest, runtime313 manifest, proxy manifest, and queue manifest in that reverse order. Queue314 restore is last and quarantines nonempty or uncertain data; do not roll back315 only one proxy/runtime file or delete a queue database manually.3168. For `client-fanout`, adapt317 `skills/galileo-lemonade-instrumentation-setup/assets/lemonade_openinference_client.py`.318 It sends to the dedicated319 loopback receiver and keeps Galileo credentials out of the application.320 Install its pinned requirements in an isolated environment, then run321 `lemonade_openinference_client.py --check` under the application identity322 before sending the real canary.3239. Send the synthetic canary through the receiver selected by the mode. A324 receiver success is only pipeline evidence; continue to Galileo readback.32510. Follow [references/validation.md](references/validation.md), including a326 real non-sensitive Lemonade request, collector counter deltas, Galileo API327 trace/hierarchy readback, privacy assertions, and unchanged Splunk backend328 readback. Perform Console trace-shape review only when a signed-in browser329 is actually available, and report its absence truthfully.33011. Build a fresh schema-v2 cutover document from the actual host, Galileo API,331 and Splunk backend results. Record it with the transaction's332 `record-cutover-evidence` command. Omit `console_review` or record only333 `{"status":"not_observed"}`; any signed-in UI review is separate evidence.33412. In a distinct invocation, run `finalize` with the same protected bootstrap335 file plus a distinct protected unscoped revoker file and its exact key ID.336 It revalidates the evidence and runtime key, verifies that the revoker is337 neither the old nor runtime key, revokes only the bound old key ID through338 that revoker, reconciles full-inventory absence, requires the old key to339 return 401 Unauthorized from both `/v2/current_user` and `/v2/token`,340 rechecks the runtime key, and reaches `FINALIZED`. Before revocation,341 `rollback` deletes only exact transaction-owned IDs and preserves adopted342 targets. The separately documented `reconcile-legacy-revocation` command343 is only for the exact retired, already-started self-delete journal schema;344 it performs no DELETE and must never replace the fresh revoker policy.345346## Rendered Collector Shapes347348`server-fanout` leaves the shared native `traces` pipeline's exporters349unchanged and adds `traces/lemonade_galileo_server`. The new branch shares the350reviewed receiver set but fail-closed filters on Lemonade's native351`service.name=lemonade-server` before the Galileo-only exporter and preserves352the baseline Lemonade deployment/privacy transform. The attribute filter does353not establish provenance; production validation rejects externally bound354receivers.355356`client-fanout` keeps the native `traces` pipeline Splunk-only and creates:357358```text359OpenAI-compatible caller360 -> 127.0.0.1:14318/v1/traces361 -> traces/lemonade_galileo_client362 -> otlp_http/galileo_lemonade363```364365This default avoids duplicate LLM/cost records in both backends and permits a366caller-specific Galileo content policy. `--mirror-client-to-native-exporters`367is an explicit exception for users who want the richer caller hierarchy in368Splunk too; it requires `--allow-client-mirror` during validation and an369approved duplicate/content-handling plan.370371The renderer inherits only baseline `memory_limiter`, `resource_detection`372(or `resourcedetection` when that is the live distro ID), and `batch`373processor types. Use repeated `--client-processor` only after reviewing374the component for source-specific filters, transforms, and content expansion;375validation then requires `--allow-custom-client-processors`.376Every custom processor must precede the managed client privacy transform. The377privacy transform and Galileo route guard are the final two non-batch378processors, followed only by a terminal batch suffix or direct export;379rendering and validation fail if any processor could mutate spans after them.380The managed resource processor runs after inherited resource detection and381upserts the dedicated client `service.name`; validation rejects an `insert`382action that could retain the host application's identity.383384Both modes redact error status text; client mode also deletes known385content-bearing span/event attributes—including multimodal message payloads and386message-level function/tool arguments and results—while retaining roles,387models, providers, structural IDs, and content types. It deletes `user.id` and388arbitrary tags/metadata, but retains opaque `session.id` for session grouping;389never put personal data in that ID. It also redacts exception messages and390stack traces. Agent/Tool `input.value`, `output.value`, and tool call391arguments/results are restored only as the constant `[REDACTED]` after deletion392so required hierarchy fields remain without source content. Source-side393OpenInference hiding remains mandatory as defense in depth, with GenAI semantic394convention duplication explicitly disabled in the reference client.395Lemonade v10.10's native `hide_outputs` does not396cover `status.message`, so removing these transforms can expose an error that397echoes sensitive content.398399The renderer strips only an exact recognized prior render before adding the400requested mode, so repeated renders and mode switches are deterministic. It401preserves unrelated components but fails closed on managed drift, foreign402references, every custom Galileo-shaped or Galileo-named exporter/route in all403modes, and any extra pipeline sharing the dedicated client receiver.404405## Application Choices406407- OpenInference + standard OpenAI instrumentation: preferred for sync/async,408 streaming, OTel context propagation, collector fan-out, and strong hide409 controls. The packaged client demonstrates this path.410- Galileo OpenAI wrapper: smallest synchronous change and supports411 OpenAI-compatible model servers, but captures raw input/output by default.412- Galileo OpenAI Agents tracing processor: best when the application actually413 uses OpenAI Agents and needs generations, tools, and handoffs.414- Galileo manual logger or `@log`: use for custom agent/framework semantics or415 explicit redacted fields.416417Read [references/application-instrumentation.md](references/application-instrumentation.md)418before choosing a client library. Content capture is opt-in.419420## Completion Gate421422Report all of the following:423424- chosen mode and why the other source is excluded;425- exact Galileo instance/API endpoint and project/Log stream IDs, with secrets426 omitted;427- Lemonade version, health, semantics, and privacy flags;428- collector config validation, loopback binds, service health, exact429 accepted/failed/refused and sent/send-failed/enqueue-failed deltas,430 queue/in-flight state, and unchanged Splunk readback;431- Galileo API readback proving the expected trace, Agent/Tool/LLM hierarchy,432 and privacy state. Report signed-in Console confirmation separately when it433 was actually observed; otherwise report it as not observed;434- bootstrap transaction phase and sanitized IDs, with `FINALIZED` required435 only after the separate fresh-evidence revocation gate;436- backup and tested rollback path.437438Production completion still requires live Galileo backend readback of the439Agent/Tool/LLM hierarchy. Static validation, collector acceptance, and an empty440or healthy queue do not substitute for that deployment gate.441442HTTP 200 alone is insufficient: OTLP responses can contain443`partialSuccess.rejectedSpans`, and collector counters do not prove backend444storage.