Managing Snowplow

Managing Snowplow

Manage and monitor Snowplow behavioral data pipelines — collectors, enrichments, schemas, and event quality.

Discovery Phase

#!/bin/bash

SNOWPLOW_API="${SNOWPLOW_API_URL:-https://console.snowplowanalytics.com/api}"
AUTH="Authorization: Bearer $SNOWPLOW_API_TOKEN"

echo "=== Pipeline Status ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/pipelines" \
  | jq -r '.[] | [.id, .name, .status, .cloudProvider] | @tsv' | column -t | head -10

echo ""
echo "=== Schema Registry ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/registry/schemas" \
  | jq -r '.[] | [.vendor, .name, .version, .format] | @tsv' | column -t | head -15

echo ""
echo "=== Enrichments ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/enrichments" \
  | jq -r '.[] | [.name, .enabled, .kind] | @tsv' | column -t | head -10

Analysis Phase

#!/bin/bash

SNOWPLOW_API="${SNOWPLOW_API_URL:-https://console.snowplowanalytics.com/api}"
AUTH="Authorization: Bearer $SNOWPLOW_API_TOKEN"

echo "=== Failed Events (Last 24h) ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/pipelines/$SNOWPLOW_PIPELINE_ID/failed-events/summary" \
  | jq -r '.failedEvents[] | [.errorType, .count, .lastSeen] | @tsv' | column -t | head -10

echo ""
echo "=== Pipeline Metrics ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/pipelines/$SNOWPLOW_PIPELINE_ID/metrics" \
  | jq '{eventsPerSecond: .eventsPerSecond, enrichedEvents: .enrichedEvents, failedEvents: .failedEvents}'

echo ""
echo "=== Collector Health ==="
curl -s -o /dev/null -w "HTTP Status: %{http_code}\nResponse Time: %{time_total}s\n" \
  "$SNOWPLOW_COLLECTOR_URL/health"

echo ""
echo "=== Schema Validation Errors ==="
curl -s -H "$AUTH" "$SNOWPLOW_API/msc/v1/organizations/$SNOWPLOW_ORG_ID/pipelines/$SNOWPLOW_PIPELINE_ID/failed-events?reason=SchemaViolation" \
  | jq -r '.events[:5][] | [.schema, .error, .timestamp] | @tsv' | column -t

Output Format

PIPELINE HEALTH
Pipeline:        <name> (<status>)
Events/sec:      <rate>
Enriched (24h):  <count>
Failed (24h):    <count>

FAILED EVENTS
Error Type       Count    Last Seen
SchemaViolation  <n>      <timestamp>
EnrichmentFail   <n>      <timestamp>

SCHEMAS
Vendor           Name             Version
<vendor>         <schema-name>    <version>

ENRICHMENTS
Name             Enabled  Kind
<enrichment>     true     <type>

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations