Loki Logs
Query the production logs from Loki, the live log store. Loki itself is not
publicly reachable; the script asks Grafana (grafana.letsrevel.io) to proxy
LogQL to its Loki datasource using a service-account token.
Tool
.venv/bin/python scripts/loki_logs.py [SERVICE] [filters] [--since DUR] [-n LIMIT]
SERVICE is one of web, celery_default, beat, telegram (omit to search all).
Default window is the last 1h, newest first, 100 lines. Run --help for the full flag list.
Use the venv interpreter — the script imports python-decouple.
Auth: reads GRAFANA_TOKEN via decouple (from the project .env or the environment).
If it errors with "GRAFANA_TOKEN is not set", it isn't configured yet — tell the user; do not guess.
What you can filter on
| Kind |
Flags |
LogQL |
| Stream labels |
SERVICE, --level error (repeatable), --env production |
indexed, cheap |
| Line content |
--grep TEXT, --exclude TEXT, --regex RE (all repeatable) |
substring / regex |
| Request metadata |
--trace-id, --request-id, --user-id, --method, --path, --status-code, --ip, --user-agent |
structlog fields |
| Raw escape hatch |
--query '{service_name="web"} | status_code="500"' |
any LogQL |
Logs are structlog JSON; the displayed line is the event message, with a
metadata line beneath it (suppress with --no-meta, get raw JSON with --json).
Recipes
# Errors in the API in the last hour
.venv/bin/python scripts/loki_logs.py web --level error
# Follow one request end-to-end across every service
.venv/bin/python scripts/loki_logs.py --request-id <uuid> --since 6h --forward
# Celery failures mentioning a traceback, last day
.venv/bin/python scripts/loki_logs.py celery_default --since 1d --grep Traceback
# All 5xx responses, last 2h
.venv/bin/python scripts/loki_logs.py web --status-code 500 --since 2h
# Everything a user triggered
.venv/bin/python scripts/loki_logs.py --user-id <uuid> --since 12h
# Everything from one client IP (e.g. chasing a scraper/429 burst)
.venv/bin/python scripts/loki_logs.py web --ip <ip> --since 6h
# Discover what labels/values exist
.venv/bin/python scripts/loki_logs.py --labels
.venv/bin/python scripts/loki_logs.py --label-values service_name
Workflow when debugging from a symptom
- Start broad:
web --level error --since <window> to find the failure.
- Grab the
trace_id/request_id from the metadata line.
- Re-query by that id with
--forward to read the full request in order,
spanning web → celery_default if work was dispatched.
- Use
--print-query to inspect/borrow the LogQL; use --json when you need
exact timestamps or fields the pretty output omits.
Notes
- Retention is 30 days; queries older than that return nothing.
- A query needs at least one matcher — with no
SERVICE/--level the script
defaults to {service_name=~".+"} (all app streams).
- Hitting
--limit prints a stderr hint; narrow --since or raise -n.
--ip/--user-agent only match lines ingested after the Alloy config that
promotes ip_address/user_agent; older lines won't match those filters.- Read-only. The token is Viewer-scoped; this cannot modify anything.
1---2name: loki-logs3description: Use when investigating production behaviour from logs — a 500/error in prod, a failing or stuck Celery task, tracing one request/trace_id/user across services, or confirming a deploy's runtime effect. Pulls real Loki logs from the live stack (web, celery_default, beat, telegram) via Grafana.4---56# Loki Logs78Query the **production** logs from Loki, the live log store. Loki itself is not9publicly reachable; the script asks Grafana (`grafana.letsrevel.io`) to proxy10LogQL to its Loki datasource using a service-account token.1112## Tool1314```bash15.venv/bin/python scripts/loki_logs.py [SERVICE] [filters] [--since DUR] [-n LIMIT]16```1718`SERVICE` is one of `web`, `celery_default`, `beat`, `telegram` (omit to search all).19Default window is the **last 1h**, newest first, 100 lines. Run `--help` for the full flag list.20Use the venv interpreter — the script imports `python-decouple`.2122**Auth:** reads `GRAFANA_TOKEN` via decouple (from the project `.env` or the environment).23If it errors with "GRAFANA_TOKEN is not set", it isn't configured yet — tell the user; do not guess.2425## What you can filter on2627| Kind | Flags | LogQL |28|------|-------|-------|29| Stream labels | `SERVICE`, `--level error` (repeatable), `--env production` | indexed, cheap |30| Line content | `--grep TEXT`, `--exclude TEXT`, `--regex RE` (all repeatable) | substring / regex |31| Request metadata | `--trace-id`, `--request-id`, `--user-id`, `--method`, `--path`, `--status-code`, `--ip`, `--user-agent` | structlog fields |32| Raw escape hatch | `--query '{service_name="web"} \| status_code="500"'` | any LogQL |3334Logs are structlog JSON; the displayed line is the `event` message, with a35metadata line beneath it (suppress with `--no-meta`, get raw JSON with `--json`).3637## Recipes3839```bash40# Errors in the API in the last hour41.venv/bin/python scripts/loki_logs.py web --level error4243# Follow one request end-to-end across every service44.venv/bin/python scripts/loki_logs.py --request-id <uuid> --since 6h --forward4546# Celery failures mentioning a traceback, last day47.venv/bin/python scripts/loki_logs.py celery_default --since 1d --grep Traceback4849# All 5xx responses, last 2h50.venv/bin/python scripts/loki_logs.py web --status-code 500 --since 2h5152# Everything a user triggered53.venv/bin/python scripts/loki_logs.py --user-id <uuid> --since 12h5455# Everything from one client IP (e.g. chasing a scraper/429 burst)56.venv/bin/python scripts/loki_logs.py web --ip <ip> --since 6h5758# Discover what labels/values exist59.venv/bin/python scripts/loki_logs.py --labels60.venv/bin/python scripts/loki_logs.py --label-values service_name61```6263## Workflow when debugging from a symptom64651. Start broad: `web --level error --since <window>` to find the failure.662. Grab the `trace_id`/`request_id` from the metadata line.673. Re-query by that id with `--forward` to read the full request in order,68 spanning `web` → `celery_default` if work was dispatched.694. Use `--print-query` to inspect/borrow the LogQL; use `--json` when you need70 exact timestamps or fields the pretty output omits.7172## Notes7374- Retention is **30 days**; queries older than that return nothing.75- A query needs at least one matcher — with no `SERVICE`/`--level` the script76 defaults to `{service_name=~".+"}` (all app streams).77- Hitting `--limit` prints a stderr hint; narrow `--since` or raise `-n`.78- `--ip`/`--user-agent` only match lines ingested after the Alloy config that79 promotes `ip_address`/`user_agent`; older lines won't match those filters.80- Read-only. The token is Viewer-scoped; this cannot modify anything.
letsrevel