Telemetry Querying Skill
Use this skill as the entry point for any investigation, debugging, or data question that may be answered from telemetry data. It helps you decide where the relevant signal lives (metrics, logs, traces, RUM) and tells you which reference files to load before querying.
Loading References
Before querying, load the reference files for the chosen pillar:
| Pillar |
Load these files |
| Logs |
references/dataprime-reference.md + references/logs-querying.md |
| Spans / Traces |
references/dataprime-reference.md + references/spans-querying.md |
| Metrics |
references/promql-guidelines.md + references/metrics-querying.md |
| RUM (frontend) |
references/dataprime-reference.md + references/logs-querying.md + references/rum-querying.md + references/rum-fields.md |
| DataPrime syntax only |
references/dataprime-reference.md |
Safety
All query commands (cx logs, cx spans, cx metrics, cx dataprime, cx search-fields) are read-only and work in --read-only mode. They never modify data and can be run freely without --yes.
Quick Routing Guide
Use this table for obvious cases where one pillar is the clear first choice:
| Question Type |
First Choice |
Fallback |
| UI behavior, page load, frontend errors |
RUM |
Traces (if backend-related) |
| Endpoint latency, throughput, error rates |
Metrics |
Traces (for per-request detail) |
| Service-to-service dependencies, request flow |
Traces |
Logs (for debug output) |
| Specific error messages, stack traces |
Logs |
Traces (for request context) |
| Infrastructure health (CPU, memory, disk) |
Metrics |
- |
| Business events (purchases, signups) |
Depends - see Discovery Workflow |
- |
For ambiguous questions (e.g., "How much money did users spend last week?"), the signal could live in any pillar. Follow the Discovery Workflow below.
Discovery Workflow
When the answer could reside in multiple pillars, run discovery in parallel to find the best source.
Step 1: Search Metrics
Check if a relevant metric exists:
cx metrics search --name '*transaction*'
cx metrics search --name '*payment*'
cx metrics search --name '*revenue*'
cx metrics search --description "total purchase amount"
If a matching metric is found, load references/promql-guidelines.md + references/metrics-querying.md and continue.
Step 2: Search Log and Span Fields
Use semantic field search to find relevant DataPrime paths:
cx search-fields "transaction amount" --dataset logs
cx search-fields "payment total" --dataset spans
cx search-fields "purchase value" --dataset logs --limit 10
If you know a concrete value that should appear in the data but don't know which field holds it, use value search instead. It returns the matching field keys alongside sample values, which also lets you infer the field's type (string, numeric, enum, etc.):
cx search-fields "payment_failed" -s value --dataset logs
cx search-fields "grpc.status.UNAVAILABLE" -s value --dataset spans
cx search-fields "eu-west-1" -s value --dataset all
Requirements: cx search-fields needs a Coralogix API key or OAuth on the active profile. If credentials are missing, prompt the user to run cx profiles add <name>.
If matching fields are found:
- For logs: load
references/dataprime-reference.md + references/logs-querying.md
- For spans: load
references/dataprime-reference.md + references/spans-querying.md
Step 3: Search the Codebase
When discovery results are ambiguous or you need to validate what a metric/field actually represents, search the codebase:
- Look for metric registration code (e.g.,
prometheus.NewCounter, metrics.record)
- Look for log statements that emit the field (e.g.,
logger.info("transaction", ...))
- Look for span attributes (e.g.,
span.setAttribute("purchase.amount", ...))
This confirms the semantic meaning and helps you choose the right pillar.
Step 4: Choose and Query
Based on discovery results, pick the pillar with the clearest signal, load its reference files (see Loading References), then query.
Fallback and Pivoting
If your initial route yields no results, pivot to another pillar.
Example pivot paths:
- Metrics empty → try traces (per-request data) or logs (event records)
- Logs empty → try traces (structured span attributes) or metrics (aggregated counters)
- Traces empty → try logs (text-based debug output)
Do not stop after one failed attempt. Try at least two pillars before concluding the data does not exist.
CLI Commands Reference
| Command |
Purpose |
When to Use |
cx schema |
Output the full command tree as JSON |
Discover all available commands and their flags |
cx metrics search --name <pattern> |
Find metrics by name |
First step for metrics discovery |
cx metrics search --description <text> |
Semantic metric search |
When you know what you want but not the name |
cx search-fields "<text>" --dataset logs |
Find log fields by description |
Discovery for log-based questions |
cx search-fields "<text>" --dataset spans |
Find span fields by description |
Discovery for trace-based questions |
cx search-fields "<value>" -s value --dataset logs |
Find log fields that contain a known value |
When you know a value but not which log field holds it — also reveals field type from the returned values |
cx search-fields "<value>" -s value --dataset spans |
Find span fields that contain a known value |
When you know a value but not which span attribute holds it |
cx search-fields "<value>" -s value --dataset all |
Same, across logs and spans |
When you want to search across both logs and spans at once |
cx spans "filter $l.serviceName == '<service>'" --limit 10 |
Search spans by service |
When investigating a specific service |
cx dataprime list |
List DataPrime commands/functions |
When building log or span queries |
cx dashboards search "<description>" |
Find existing dashboards by natural-language description |
Before creating a new dashboard — check if one already exists |
cx dashboards query-search --description "<text>" |
Find dashboard widgets whose queries cover a topic |
Discover how a topic is already being monitored |
cx dashboards query-search --field "<field-path>" |
Find widgets that reference a specific field |
Reuse existing PromQL/DataPrime patterns for a known field |
Examples
Example 1: Business Question (Ambiguous Source)
Question: "How much money did people spend on the platform last week?"
Approach:
- Search metrics:
cx metrics search --name '*revenue*' and cx metrics search --name '*transaction*'
- Search log fields:
cx search-fields "transaction amount" --dataset logs
- Search span fields:
cx search-fields "payment total" --dataset spans
- If a metric like
payment_total_usd exists, load metrics references and run a range query
- If only logs have the data, load logs references and use DataPrime aggregation
- If traces have
purchase.amount attribute, load spans references
Example 2: Latency Question (Clear First Choice)
Question: "What's the average latency of the checkout route?"
Approach:
- First try metrics:
cx metrics search --name '*checkout*latency*' or cx metrics search --name '*http*duration*'
- If a histogram metric exists, load metrics references and use
histogram_quantile
- If no metric, fall back to traces: load spans references and aggregate span durations
Example 3: Frontend Performance (RUM)
Question: "Why is the dashboard page loading slowly for users?"
Approach:
- This is clearly a RUM question - load
references/rum-querying.md + references/rum-fields.md + references/logs-querying.md + references/dataprime-reference.md
- Query web vitals and page load times
- If RUM shows backend calls are slow, pivot to spans references for the API calls
Example 4: Error Investigation (Logs + Traces)
Question: "Why are users getting 500 errors on the payment endpoint?"
Approach:
- Check error rate metrics → load metrics references
- Search for error logs → load logs references
- Get traces for failed requests → load spans references
- Cross-reference: find trace IDs in logs, then fetch full traces for root cause
Beyond Investigation
Not every question is answered by querying data. If the user's intent is operational rather than investigative, route to the appropriate workflow skill:
| User Intent |
Route To |
| Reducing costs, checking usage, TCO policies |
cx-cost-optimization |
| Triaging a case, who got paged, case timeline |
cx-cases |
| SLO status, error budget, service-level targets |
cx-slos |
| Setting up monitoring, webhooks, notifications |
cx-observability-setup |
| Configuring parsing rules, enrichments, E2M |
cx-data-pipeline |
| Access audit, API keys, user management |
cx-platform-admin |
| Creating or managing dashboards |
cx-dashboards |
| Finding or searching existing dashboards |
cx-search-dashboard |
Key Principles
- Load references before querying: check the Loading References table first
- Discover before querying: always run search/discovery to find the right source
- Parallel discovery: for ambiguous questions, search metrics, logs, and spans concurrently
- Validate with code: when unsure what a metric or field represents, check the codebase
- Pivot on failure: if one pillar is empty, try another before giving up
1---2name: cx-telemetry-querying3description: |4 Use this skill for any question involving telemetry data: "investigate an issue",5 "debug a problem", "find out why something is slow", "check error rates",6 "analyze user behavior", "understand a production incident",7 "query telemetry data", "look at logs", "search logs", "find errors",8 "find stack traces", "filter by severity",9 "check traces", "examine spans", "investigate request latency",10 "debug service-to-service calls", "look up a trace ID",11 "analyze RUM data", "check frontend performance", "frontend errors",12 "Core Web Vitals", "JavaScript exceptions",13 "query metrics", "check CPU usage", "run a PromQL query",14 "check error rate", "look up a metric", "check memory usage",15 "how do I write a DataPrime query", "DataPrime syntax",16 or wants to answer questions using observability data from logs, metrics, traces, RUM, or APM.17metadata:18 source: https://skillmd.com/skills/coralogix/cx-telemetry-querying19 version: "0.2.0"20---2122# Telemetry Querying Skill2324Use this skill as the entry point for any investigation, debugging, or data question that may be answered from telemetry data. It helps you decide **where** the relevant signal lives (metrics, logs, traces, RUM) and tells you **which reference files to load** before querying.2526## Loading References2728Before querying, load the reference files for the chosen pillar:2930| Pillar | Load these files |31|---|---|32| Logs | `references/dataprime-reference.md` + `references/logs-querying.md` |33| Spans / Traces | `references/dataprime-reference.md` + `references/spans-querying.md` |34| Metrics | `references/promql-guidelines.md` + `references/metrics-querying.md` |35| RUM (frontend) | `references/dataprime-reference.md` + `references/logs-querying.md` + `references/rum-querying.md` + `references/rum-fields.md` |36| DataPrime syntax only | `references/dataprime-reference.md` |3738---3940## Safety4142All query commands (`cx logs`, `cx spans`, `cx metrics`, `cx dataprime`, `cx search-fields`) are read-only and work in `--read-only` mode. They never modify data and can be run freely without `--yes`.4344---4546## Quick Routing Guide4748Use this table for obvious cases where one pillar is the clear first choice:4950| Question Type | First Choice | Fallback |51|---|---|---|52| UI behavior, page load, frontend errors | RUM | Traces (if backend-related) |53| Endpoint latency, throughput, error rates | Metrics | Traces (for per-request detail) |54| Service-to-service dependencies, request flow | Traces | Logs (for debug output) |55| Specific error messages, stack traces | Logs | Traces (for request context) |56| Infrastructure health (CPU, memory, disk) | Metrics | - |57| Business events (purchases, signups) | Depends - see Discovery Workflow | - |5859For **ambiguous questions** (e.g., "How much money did users spend last week?"), the signal could live in any pillar. Follow the Discovery Workflow below.6061---6263## Discovery Workflow6465When the answer could reside in multiple pillars, run discovery in parallel to find the best source.6667### Step 1: Search Metrics6869Check if a relevant metric exists:7071```bash72cx metrics search --name '*transaction*'73cx metrics search --name '*payment*'74cx metrics search --name '*revenue*'75cx metrics search --description "total purchase amount"76```7778If a matching metric is found, load `references/promql-guidelines.md` + `references/metrics-querying.md` and continue.7980### Step 2: Search Log and Span Fields8182Use semantic field search to find relevant DataPrime paths:8384```bash85cx search-fields "transaction amount" --dataset logs86cx search-fields "payment total" --dataset spans87cx search-fields "purchase value" --dataset logs --limit 1088```8990If you know a concrete value that should appear in the data but don't know which field holds it, use value search instead. It returns the matching field keys alongside sample values, which also lets you infer the field's type (string, numeric, enum, etc.):9192```bash93cx search-fields "payment_failed" -s value --dataset logs94cx search-fields "grpc.status.UNAVAILABLE" -s value --dataset spans95cx search-fields "eu-west-1" -s value --dataset all96```9798**Requirements:** `cx search-fields` needs a Coralogix API key or OAuth on the active profile. If credentials are missing, prompt the user to run `cx profiles add <name>`.99100If matching fields are found:101- For **logs**: load `references/dataprime-reference.md` + `references/logs-querying.md`102- For **spans**: load `references/dataprime-reference.md` + `references/spans-querying.md`103104### Step 3: Search the Codebase105106When discovery results are ambiguous or you need to validate what a metric/field actually represents, search the codebase:107108- Look for metric registration code (e.g., `prometheus.NewCounter`, `metrics.record`)109- Look for log statements that emit the field (e.g., `logger.info("transaction", ...)`)110- Look for span attributes (e.g., `span.setAttribute("purchase.amount", ...)`)111112This confirms the semantic meaning and helps you choose the right pillar.113114### Step 4: Choose and Query115116Based on discovery results, pick the pillar with the clearest signal, load its reference files (see [Loading References](#loading-references)), then query.117118---119120## Fallback and Pivoting121122**If your initial route yields no results, pivot to another pillar.**123124Example pivot paths:125- Metrics empty → try traces (per-request data) or logs (event records)126- Logs empty → try traces (structured span attributes) or metrics (aggregated counters)127- Traces empty → try logs (text-based debug output)128129Do not stop after one failed attempt. Try at least two pillars before concluding the data does not exist.130131---132133## CLI Commands Reference134135| Command | Purpose | When to Use |136|---|---|---|137| `cx schema` | Output the full command tree as JSON | Discover all available commands and their flags |138| `cx metrics search --name <pattern>` | Find metrics by name | First step for metrics discovery |139| `cx metrics search --description <text>` | Semantic metric search | When you know what you want but not the name |140| `cx search-fields "<text>" --dataset logs` | Find log fields by description | Discovery for log-based questions |141| `cx search-fields "<text>" --dataset spans` | Find span fields by description | Discovery for trace-based questions |142| `cx search-fields "<value>" -s value --dataset logs` | Find log fields that contain a known value | When you know a value but not which log field holds it — also reveals field type from the returned values |143| `cx search-fields "<value>" -s value --dataset spans` | Find span fields that contain a known value | When you know a value but not which span attribute holds it |144| `cx search-fields "<value>" -s value --dataset all` | Same, across logs and spans | When you want to search across both logs and spans at once |145| `cx spans "filter $l.serviceName == '<service>'" --limit 10` | Search spans by service | When investigating a specific service |146| `cx dataprime list` | List DataPrime commands/functions | When building log or span queries |147| `cx dashboards search "<description>"` | Find existing dashboards by natural-language description | Before creating a new dashboard — check if one already exists |148| `cx dashboards query-search --description "<text>"` | Find dashboard widgets whose queries cover a topic | Discover how a topic is already being monitored |149| `cx dashboards query-search --field "<field-path>"` | Find widgets that reference a specific field | Reuse existing PromQL/DataPrime patterns for a known field |150151---152153## Examples154155### Example 1: Business Question (Ambiguous Source)156157**Question:** "How much money did people spend on the platform last week?"158159**Approach:**1601. Search metrics: `cx metrics search --name '*revenue*'` and `cx metrics search --name '*transaction*'`1612. Search log fields: `cx search-fields "transaction amount" --dataset logs`1623. Search span fields: `cx search-fields "payment total" --dataset spans`1634. If a metric like `payment_total_usd` exists, load metrics references and run a range query1645. If only logs have the data, load logs references and use DataPrime aggregation1656. If traces have `purchase.amount` attribute, load spans references166167### Example 2: Latency Question (Clear First Choice)168169**Question:** "What's the average latency of the checkout route?"170171**Approach:**1721. First try metrics: `cx metrics search --name '*checkout*latency*'` or `cx metrics search --name '*http*duration*'`1732. If a histogram metric exists, load metrics references and use `histogram_quantile`1743. If no metric, fall back to traces: load spans references and aggregate span durations175176### Example 3: Frontend Performance (RUM)177178**Question:** "Why is the dashboard page loading slowly for users?"179180**Approach:**1811. This is clearly a RUM question - load `references/rum-querying.md` + `references/rum-fields.md` + `references/logs-querying.md` + `references/dataprime-reference.md`1822. Query web vitals and page load times1833. If RUM shows backend calls are slow, pivot to spans references for the API calls184185### Example 4: Error Investigation (Logs + Traces)186187**Question:** "Why are users getting 500 errors on the payment endpoint?"188189**Approach:**1901. Check error rate metrics → load metrics references1912. Search for error logs → load logs references1923. Get traces for failed requests → load spans references1934. Cross-reference: find trace IDs in logs, then fetch full traces for root cause194195---196197## Beyond Investigation198199Not every question is answered by querying data. If the user's intent is operational rather than investigative, route to the appropriate workflow skill:200201| User Intent | Route To |202|---|---|203| Reducing costs, checking usage, TCO policies | `cx-cost-optimization` |204| Triaging a case, who got paged, case timeline | `cx-cases` |205| SLO status, error budget, service-level targets | `cx-slos` |206| Setting up monitoring, webhooks, notifications | `cx-observability-setup` |207| Configuring parsing rules, enrichments, E2M | `cx-data-pipeline` |208| Access audit, API keys, user management | `cx-platform-admin` |209| Creating or managing dashboards | `cx-dashboards` |210| Finding or searching existing dashboards | `cx-search-dashboard` |211212---213214215## Key Principles216217- **Load references before querying**: check the [Loading References](#loading-references) table first218- **Discover before querying**: always run search/discovery to find the right source219- **Parallel discovery**: for ambiguous questions, search metrics, logs, and spans concurrently220- **Validate with code**: when unsure what a metric or field represents, check the codebase221- **Pivot on failure**: if one pillar is empty, try another before giving up