Observe

You are an expert at using the Observe CLI (observe) to interact with Observe tenants. The binary is located at ~/go/bin/observe.

Setup & Authentication

Before running any commands, check if the user has a config file:

cat ~/.config/observe.yaml

If not configured, help them log in:

~/go/bin/observe --customerid <CUSTOMER_ID> --site observeinc.com login <EMAIL> --sso

Login methods:

• --sso (recommended): Opens a browser-based approval flow. Best for SSO/SAML users (Okta, Azure AD, Google, PingOne).
• --read-password: Reads password from stdin (no echo). Best for email/password users.
• Direct password on CLI: observe login user@example.com mypassword (less secure, avoid in interactive shells).

The login command saves the auth token to ~/.config/observe.yaml under the active profile (default: default). Use --no-save to prevent saving.

Configuration File

The config file at ~/.config/observe.yaml supports multiple profiles:

profile:
  default:
    customerid: "133742069123"
    site: "observeinc.com"
    authtoken: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  staging:
    customerid: "180316196377"
    site: "observe-staging.com"
    authtoken: "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
    workspace: MyWorkspace

Select a profile with --profile=<name> or the OBSERVE_PROFILE environment variable.

Running OPAL Queries

The primary command for querying data:

~/go/bin/observe query -q '<OPAL_QUERY>' -i '<INPUT_DATASET>'

Key Flags

Input Datasets

Inputs can be specified as:

• Dataset ID: -i '41021818'
• Workspace.Name: -i 'Default.kubernetes/Container Logs'
• Multiple inputs for joins: -i '41021818,right=41012929' or -i 'Default.System,right=Default.network/Flow Logs'

If the default workspace is set in config, you can omit the workspace prefix.

Time Windows

Default: last hour, truncated to full minutes. Specify at most two of: --start-time, --end-time, --relative.

Time formats supported:

• RFC 3339: 2023-04-20T16:20:00Z
• With timezone: 2023-04-20T16:20:00.123-08:00
• Relative: -1d, -4h, -30m
• With snap: -1d@1h (1 day ago, snapped to start of hour)
• Epoch seconds: 1682007600
• Epoch milliseconds: 1682007600000
• Epoch nanoseconds: 1682007600000000000

Query Examples

Simple query:

~/go/bin/observe query -q 'limit 10' -i 'Default.System'

Query with time window and extended output:

~/go/bin/observe query -q 'pick_col timestamp, log | limit 10' -i 'Default.kubernetes/Container Logs' -r 4h -x

Query with join:

~/go/bin/observe query \
    -q 'leftjoin host_ip=@right.dst_ip, source:@right.src_ip' \
    -i '41021818,right=41012929' \
    -r 4h

Query from file with CSV output:

~/go/bin/observe query -f my_query.opal -i 'Default.System' --csv -r 24h

JSON output (useful for piping to jq):

~/go/bin/observe query -q 'limit 5' -i 'Default.System' --json

OPAL Query Patterns

Filtering

filter body ~ /error|fail|exception/i          # regex match (case insensitive)
filter service_name = "payment"                 # exact match
filter pod ~ /payment/                          # partial regex match
filter error = true                             # boolean filter
filter body ~ /error/i and pod ~ /payment/      # combined filters

Aggregation with statsby

statsby count:count(1), group_by(body)                        # count by message
statsby count:count(1), group_by(pod)                         # count by pod
statsby count:count(1), group_by(loyalty_level, error)        # multi-dimensional grouping

Time series with timechart

timechart 1h, count:count(1)                                  # hourly counts
timechart 5m, count:count(1)                                  # 5-minute buckets
filter error = true | timechart 1h, count:count(1)            # error rate over time

Column selection and extraction

pick_col timestamp, body, pod, namespace                      # select specific columns
make_col loyalty_level: string(attributes["app.loyalty.level"])  # extract from JSON
make_col card_type: string(attributes["app.payment.card_type"])  # extract nested attribute

Conditional logic

make_col failure_rate: case(body = "error msg", count, true, 0)

Common Dataset Types

Observe workspaces typically contain these dataset categories:

Kubernetes Logs

• Columns: timestamp, body, stream, cluster, namespace, container, pod, node, attributes, resource_attributes, instrumentation_scope, fields, meta
• Use for: Log search, error analysis, pod-level debugging

Tracing / Spans

• Columns: start_time, end_time, duration, service_name, span_name, error, response_status, status_code, status_message, span_type, attributes, resource_attributes, parent_span_id, span_id, trace_id
• Use for: Distributed trace analysis, service dependency mapping, latency investigation
• IMPORTANT: When using pick_col on span datasets, you MUST always include start_time and end_time columns, or the query will fail with: need to pick 'valid from' column "start_time" / need to pick 'valid to' column "end_time"

Metrics (with metric-sma-for-* companion datasets)

• Use for: Resource utilization, SLI/SLO tracking, capacity planning

Reference Tables

• Reference tables (e.g., Products, Customer Records) are regular datasets that hold enrichment data (product catalog, user info, etc.)
• Use for: Joining with trace/log data to enrich with human-readable names, prices, categories
• Query them like any other dataset: ~/go/bin/observe query -q 'limit 20' -i '<REF_TABLE_ID>' --json -w 0

Investigation Workflows

Error Investigation Playbook

When users report failures, follow this multi-step approach:

1. Get an overview — sample recent records to understand schema:

   ~/go/bin/observe query -q 'limit 5' -i '<DATASET_ID>' -r 1h -x
   

2. Quantify the problem — count errors by message type:

   ~/go/bin/observe query -q 'filter body ~ /error|fail|exception/i | statsby count:count(1), group_by(body)' -i '<LOG_DATASET>' -r 24h
   

3. Identify the blast radius — break down by pod/service/namespace:

   ~/go/bin/observe query -q 'filter body ~ /error/i | statsby count:count(1), group_by(pod)' -i '<LOG_DATASET>' -r 24h
   

4. Check for time patterns — trend errors over time:

   ~/go/bin/observe query -q 'filter body ~ /error/i | timechart 1h, count:count(1)' -i '<LOG_DATASET>' -r 24h
   

5. Inspect full error details — get stack traces and attributes:

   ~/go/bin/observe query -q 'filter body ~ /error/i | pick_col timestamp, body, attributes | limit 5' -i '<LOG_DATASET>' -r 24h --json -w 0
   

6. Correlate with traces — find error spans in the tracing dataset:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and error = true | pick_col start_time, end_time, span_name, attributes, trace_id | limit 10' -i '<SPAN_DATASET>' -r 24h -x -w 0
   

7. Trace a single request end-to-end — follow one trace ID across all services:

   ~/go/bin/observe query -q 'filter trace_id = "<TRACE_ID>" | pick_col start_time, end_time, service_name, span_name, error, duration, attributes' -i '<SPAN_DATASET>' -r 24h -x -w 0
   

8. Cross-reference dimensions — break down errors by attributes (e.g., loyalty level, card type):

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" | make_col attr_val: string(attributes["<KEY>"]) | statsby count:count(1), group_by(attr_val, error)' -i '<SPAN_DATASET>' -r 24h
   

Revenue / Business Impact Analysis Playbook

When asked to quantify the business impact of failures, follow this approach using trace span attributes (e.g., app.payment.amount):

1. Get totals by success vs failure — aggregate revenue and transaction counts:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | statsby total: sum(amount), count: count(1), avg_order: avg(amount), min_order: min(amount), max_order: max(amount), group_by(error)' -i '<SPAN_DATASET>' -r 24h -w 0
   

2. Calculate rates — add hourly burn rate and percentage breakdowns:

   ... | make_col pct_of_total: round(100.0 * count / <TOTAL>.0, 1), hourly_rate: round(total / 24.0, 2)
   

3. Distribution by order size — bucket failed transactions to find where the biggest dollar losses are:

   filter error = true | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | make_col bucket: case(amount < 100, "< $100", amount < 500, "$100-$500", amount < 1000, "$500-$1K", amount < 5000, "$1K-$5K", amount < 10000, "$5K-$10K", true, "$10K+") | statsby lost: sum(amount), count: count(1), group_by(bucket)
   

4. Segment by customer dimension — break down revenue loss by user attributes (loyalty tier, region, etc.):

   filter error = true | make_col segment: string(attributes["<SEGMENT_KEY>"]), amount: float64(attributes["<AMOUNT_KEY>"]) | statsby lost: sum(amount), count: count(1), group_by(segment)
   

5. Break down by product/item — correlate failed traces with product data using the batched trace correlation technique:

   Step 1: Export ALL failed trace IDs and split into batches of ~2,000:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" and error = true | pick_col start_time, end_time, trace_id' -i '<SPAN_DATASET>' -r 24h --json \
     | python3 -c "
   import sys, json
   traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
   print(f'Total unique failed trace IDs: {len(traces)}', file=sys.stderr)
   batch_size = 2000
   for i in range(0, len(traces), batch_size):
       open(f'/tmp/failed_batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
   print(f'Created {(len(traces)+batch_size-1)//batch_size} batches')
   "
   

   Step 2: Run all batches in parallel to find correlated spans (e.g., product lookups):

   NUM_BATCHES=<N>  # from step 1
   for i in $(seq 0 $((NUM_BATCHES-1))); do
     TRACES=$(cat /tmp/failed_batch_${i}.txt)
     ~/go/bin/observe query \
       -q "filter service_name = \"<CATALOG_SERVICE>\" and span_name ~ /GetProduct/ and trace_id ~ /${TRACES}/ | make_col product_name: string(attributes[\"<PRODUCT_NAME_KEY>\"]) | statsby count: count(1), group_by(product_name)" \
       -i '<SPAN_DATASET>' -r 24h --json 2>/dev/null > /tmp/product_batch_${i}.json &
   done
   wait
   

   Step 3: Aggregate all batch results and calculate price-weighted revenue:

   import json, glob
   product_counts = {}
   for f in glob.glob('/tmp/product_batch_*.json'):
       for line in open(f):
           obj = json.loads(line.strip())
           name = obj['product_name']
           product_counts[name] = product_counts.get(name, 0) + int(obj['count'])
   
   # Price-weight using reference table data
   total_weighted = sum(prices[name] * count for name, count in product_counts.items())
   for name, count in product_counts.items():
       pct = (prices[name] * count) / total_weighted
       est_lost = pct * total_lost_revenue
   

   This batched approach is necessary because OPAL's leftjoin cannot pre-filter the right-side input dataset, and a single regex with all trace IDs would exceed CLI argument limits. Batches of ~2,000 trace IDs each work well within limits.

   IMPORTANT: When parsing --json output from statsby, numeric values like count are returned as strings, not integers. Always cast with int() in Python.

6. Present the results — always include:

   • Total lost revenue (absolute dollar amount)
   • Failure rate (% of transactions)
   • Hourly/daily burn rate
   • Projected loss if unresolved (extrapolate to 30 days)
   • Distribution by order size (high-value orders often drive most of the dollar loss)
   • Per-product/item breakdown with unit prices when reference data is available
   • Severity recommendation (P0/P1/P2) based on magnitude

OPAL Tips for Numeric Analysis

• Use float64(attributes["key"]) to extract numeric values from JSON attributes for sum(), avg(), min(), max().
• Use round(value, N) to control decimal precision in results.
• Use case() for bucketing continuous values into discrete ranges.
• When statsby on span datasets produces per-span rows instead of aggregates, ensure you are NOT inadvertently including high-cardinality columns (like trace_id, span_id) — these cause implicit grouping. Use only the group_by() columns you intend.

Cross-Trace Correlation Technique (Batched Parallel Queries)

OPAL's leftjoin with two inputs from the same span dataset cannot pre-filter the right side. This means you cannot directly join "failed payment spans" with "product-catalog spans" on trace_id in a single query.

Workaround — Batched Parallel Queries:

1. Export ALL trace IDs from one query (e.g., failed payment spans) using --json
2. Deduplicate and split into batches of ~2,000 (each batch as a regex alternation id1|id2|id3...)
3. Run all batches in parallel using & and wait
4. Aggregate results from all batch output files

# Step 1: Export all trace IDs and split into batches
~/go/bin/observe query -q '... | pick_col start_time, end_time, trace_id' -i '<DATASET>' -r 24h --json \
  | python3 -c "
import sys, json
traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
batch_size = 2000
for i in range(0, len(traces), batch_size):
    open(f'/tmp/batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
num = (len(traces)+batch_size-1)//batch_size
open('/tmp/num_batches.txt','w').write(str(num))
print(f'{len(traces)} traces -> {num} batches')
"

# Step 2: Run all batches in parallel
NUM=$(cat /tmp/num_batches.txt)
for i in $(seq 0 $((NUM-1))); do
  TRACES=$(cat /tmp/batch_${i}.txt)
  ~/go/bin/observe query \
    -q "filter service_name = \"<SERVICE>\" and trace_id ~ /${TRACES}/ | ..." \
    -i '<DATASET>' -r 24h --json 2>/dev/null > /tmp/result_${i}.json &
done
wait

# Step 3: Aggregate in Python
python3 -c "
import json, glob
totals = {}
for f in glob.glob('/tmp/result_*.json'):
    for line in open(f):
        obj = json.loads(line.strip())
        key = obj['<GROUP_KEY>']
        totals[key] = totals.get(key, 0) + int(obj['count'])
for k, v in sorted(totals.items(), key=lambda x: -x[1]):
    print(f'{k}: {v}')
"

When to use: Correlating data across different services/spans within the same traces (e.g., "which products were in orders that had payment failures"). Use this to cover 100% of traces instead of sampling.

Why batches of ~2,000: Each trace ID is 32 hex chars. A regex alternation of 2,000 IDs is ~66KB, which is within shell argument limits. Batches run in parallel so total wall-clock time is roughly the same as a single query.

Gotcha — --json numeric types: When statsby results are output with --json, numeric aggregation values (like count) are returned as strings, not integers. Always cast with int() or float() when parsing in Python.

Tips for Investigation

• Start broad, then narrow: Begin with high-level counts before drilling into specifics.
• Run queries in parallel: Launch multiple independent queries simultaneously to speed up investigations.
• Use --json -w 0 when you need to see full attribute payloads without truncation.
• Use -x (extended format) for records with many columns or long values.
• Check both logs AND traces: Logs give you error messages and stack traces; traces give you service-to-service call flow, latency, and error propagation paths.
• Steady error rates (consistent per-hour counts) typically indicate a systematic bug; spiky error rates suggest infrastructure or load-related issues.

Listing & Getting Objects

List objects

~/go/bin/observe list <object_type> [substring_filter]

Supported object types: dataset, workspace, user, document, rbacgroup, rbacgroupmember, rbacstatement

Examples:

~/go/bin/observe list dataset
~/go/bin/observe list dataset "Container Logs"
~/go/bin/observe list workspace

Get object details

~/go/bin/observe get <object_type> <object_id>

Returns YAML with config (mutable properties) and state (derived properties) sections.

Example:

~/go/bin/observe get dataset 41042071
~/go/bin/observe get workspace 41042069

Deleting Objects

~/go/bin/observe delete <object_type> <object_id>

Example:

~/go/bin/observe delete document o::1234567890:document:8007654321

Uploading Documents

Upload documents (e.g., prompts for o11y help tool):

~/go/bin/observe upload prompt my-notes.md
~/go/bin/observe upload prompt --as-filename oncall-schedule.md path/to/new/file.md

Prompt documents are indexed for the o11y help tool. Up to 500 prompt documents are supported.

Global Configuration Flags

Guidelines

• Always use the full binary path ~/go/bin/observe when running commands.
• When building queries, prefer --json output for programmatic processing and default table format for human-readable output.
• Use --extended (-x) when inspecting records with long field values (JSON objects, log lines).
• For large result sets, always include a limit in the OPAL query to avoid excessive output.
• When the user asks to "query Observe" or "search logs", use the query command with appropriate OPAL.
• When the user asks to "find a dataset" or "what datasets exist", use the list dataset command.
• If authentication fails, suggest re-running login to refresh the token (tokens expire after ~10 days of inactivity).

Gotchas & Common Pitfalls

• Span datasets require start_time and end_time: When using pick_col on span/trace datasets, you MUST include both start_time and end_time or the query will fail. This is because spans are interval-type data.
• Empty results don't mean no data: If a filter returns nothing, try broadening the time window (-r 24h instead of -r 1h) or relaxing the filter pattern.
• statsby output can be very large: When grouping by high-cardinality fields (like body with unique log lines), the result set can be huge. Add a | topk 20, count or similar to limit.
• Regex filters are case-sensitive by default: Use the i flag for case-insensitive matching: filter body ~ /error/i.
• JSON attribute extraction: Use string(attributes["key"]) with make_col to extract values from JSON columns for grouping and filtering.
• Truncated output: Default column width is 64 chars. Use -w 0 to see full values, or --json for complete attribute payloads.
• Large result files: When output exceeds ~30KB, it gets saved to a file. Use --json with limit to keep output manageable.