# New Indicator > The canonical Radon pattern for shipping a new market indicator end to end (ingestion job, Turso storage, API route, regime chart tab, tests, systemd timer, CI, production verify). Derived from the BREADTH (21c13715, f067491e, 770d4674) and FINRA margin-debt (c5f4b877, b2f452fb, 189fcad8) builds. Read before adding any indicator; /indicator orchestrates a parallel swarm on top of this pattern. - Skill: `joemccann/new-indicator` (Agent Skill) - Install (CLI): `npx skillmds@latest add joemccann/new-indicator` - Raw SKILL.md: https://api.skillmd.com/api/skills/joemccann/new-indicator/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Integrations & APIs - Author: joemccann (https://skillmd.com/u/joemccann) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/joemccann/new-indicator --- # New market indicator — the Radon pattern An indicator is a **vertical slice** with seven parts. All seven ship together; the reference implementations are the FINRA margin-debt tab (external HTTP source, daily timer) and the NYSE breadth tab (IB + StockCharts, 5-min timer). Copy margin-debt for scheduled external-source indicators; copy breadth for IB-fed intraday collectors. ## 0. File checklist Create: | File | Modeled on | |---|---| | `scripts/clients/_client.py` (only if the source needs retry/conditional-GET logic) | `scripts/clients/finra_client.py` | | `scripts/fetch_.py` or `scripts/_scan.py` | `scripts/fetch_margin_debt.py` / `scripts/breadth_scan.py` | | `scripts/db/migrations/00NN_.sql` (next free number, 4-digit, lex-ordered) | `0027_margin_debt_history.sql`, `0029_rv_ratio.sql` | | `scripts/tests/test_.py` | `scripts/tests/test_margin_debt.py` | | `web/app/api//route.ts` | `web/app/api/margin-debt/route.ts` | | `web/lib/.ts` (pure helpers + types) | `web/lib/marginDebt.ts` | | `web/lib/use.ts` | `web/lib/useMarginDebt.ts` | | `web/components/Panel.tsx` | `web/components/MarginDebtPanel.tsx` | | `web/app/regime//page.tsx` (time-series regime only; name-ranking scanners use `/scanner?mode=`) | `web/app/regime/margin/page.tsx` / `ScannerModeTabs` | | `web/tests/-api.test.ts` | `web/tests/margin-debt-api.test.ts` | | `web/tests/-panel.test.tsx` | `web/tests/margin-debt-panel.test.tsx` | | `web/e2e/-tab.spec.ts` | `web/e2e/margin-debt-tab.spec.ts` | | `cloud/services/radon-.service` + `.timer` | `radon-margin-debt.{service,timer}`, `radon-bpi.timer` | Modify (lockstep pins — miss one and a test fails, which is the point): - `scripts/db/writer.py` — `upsert__rows(...)` (+ reuse `upsert_scan_snapshot` / `record_service_health`) - IA: time-series market-state charts (CRI, GEX, breadth, margin) go on `/regime/`. Name-ranking scanners (LEAP, GARCH, cheap-wing vol cone) go on `/scanner?mode=` next to Flow / Discover. Do not park a name scanner on Regime. - Regime only: `web/lib/regimeRail.ts` first (the `RegimeTab` union, the `REGIME_RAIL_GROUPS` entry that places the tab in its rail group, and `REGIME_TAB_LABEL`), then `web/components/RegimePanel.tsx` (`MOBILE_TAB_LABEL` if the label needs a short form, the `tabFromPathname` regex, the mobile chip bar's inline `RegimeTab[]` array, plus the `if (activeTab === "")` dispatch branch) - Regime only: `web/tests/regime-tab-routes.test.tsx` — add `["", "app/regime//page.tsx"]` to the `describe.each` table + a render/navigation case - Scanner: `ScannerModeTabs` + `WorkspaceSections` `ScannerMode` + `?mode=` parse + panel branch + `scanner-mode-tabs` tests - `web/lib/serviceHealthWindows.ts` — staleness window entry (kebab-case service name) - `web/tests/service-health-windows.test.ts` — the `expected` set is **exhaustive**; add the new service - `web/lib/refreshSchedule.ts` — `export const _REFRESH: UtcSchedule = { cadence, hourUtc, minuteUtc[, weekdayUtc] }` mirroring the timer's `OnCalendar` line (this is what the panel's countdown reads) - `web/tests/refresh-schedule.test.ts` — add an `it(" mirrors radon-.timer")` case; the test parses the unit file, so a timer edit without the constant fails CI - `scripts/watchdog/services.py` — same window for the Python watchdog + the daily-bucket check list - `cloud/scripts/setup-vps.sh` — append both units to the `SERVICE_FILES` array (line ~34-97), AND add both unit hashes to `cloud/config/installed-units.sha256` in the same commit: since PR #73 the deploy's `radon-deploy-root install-units` verb installs every manifest-pinned unit and `enable --now`s new timers, so no root SSH is owed. A unit missing from the manifest is never installed. - `cloud/tests/test_systemd_services.py` — add both units to the canonical set ## 1. Ingestion job (`scripts/`) - Python 3.13, stdlib-preferred parsing (margin-debt parses xlsx with `zipfile` + `ElementTree`). Composed-method style: small pure functions (`parse_*`, `merge_*`, `compute_*`, `build_output`) so pytest covers them without network. - **Honest User-Agent** (`radon/2.0`). Never impersonate a browser; FINRA's Cloudflare *blocks* browser UAs and serves plain clients. - **Conditional GET fast path**: persist the source `Last-Modified` in the JSON payload; send `If-Modified-Since`; on 304 skip parse + row upserts and only refresh the snapshot + heartbeat with a new `scan_time` ("source unchanged (304); refreshing snapshot only"). - **Empty-payload guard**: never cache or mirror an empty result (`persist_result` refuses; protects last-good cache — cf. feedback_dont_cache_empty_results). - Timestamps: tz-aware UTC ISO `scan_time`; `ZoneInfo("America/New_York")` for session logic, never hardcoded offsets. - Writes, in order, every cycle: 1. `writer.ensure_no_replica_for_writers()` 2. `writer.upsert__rows(series, recorded_at=scan_time)` — **only when rows changed** 3. `writer.upsert_scan_snapshot("", scan_time, payload)` — every cycle 4. `writer.record_service_health("", "ok", finished_at=scan_time)` — every cycle, or error rows latch (feedback_service_health_heartbeat) 5. JSON fallback `data/.json` (atomic write) — fallback only; **Turso is the source of truth** (Data Persistence rule: host-local files are ephemeral on the VPS) - Service name is **kebab-case everywhere** (`margin-debt`, not `margin_debt` — the underscore variant once wrote a stray `service_health` row that had to be deleted by hand on the VPS). - CLI: `--json` prints the payload to stdout; progress/summary to **stderr** (subprocess contract). - If IB is involved: bounded awaits (`asyncio.wait_for`), range-based client IDs registered in `scripts/CLAUDE.md`, `snapshot=True` must pass `""` genericTickList. ## 2. Storage (Turso-first) - Migration `scripts/db/migrations/00NN_.sql`: `CREATE TABLE IF NOT EXISTS _history (...)` with a natural PK (`date` or `(symbol, date)`), a `recorded_at TEXT NOT NULL`, a `DESC` index, and the trailing `INSERT OR IGNORE INTO schema_migrations (version, applied_at) VALUES (NN, datetime('now'));` - Applied automatically: `scripts/db/migrate.py` runs as `radon-api` `ExecStartPre` on deploy (no per-migration registration). Local: `bun run db:migrate`. - Writer: idempotent per-key `INSERT ... ON CONFLICT(date) DO UPDATE`. Latest-snapshot reads go through `scan_snapshots (service, scan_time, payload)` unless the payload is per-symbol (then a dedicated `*_snapshots` table like rv-ratio/bpi). - **Verify the row lands in Turso in production before calling the task done.** `data/*.json` is a stale fallback, never verification evidence. ## 3. API route (`web/app/api//route.ts`) - `export const dynamic = "force-dynamic"; export const runtime = "nodejs";` — GET only unless there is a real on-demand trigger (breadth's POST proxies FastAPI with cooldown + single-flight). - `export const radonCapability = "read";` (GET market data). On-demand POST scans use a method map with `read.spawn`. Unclassified routes fail `web/tests/assistant-catalog-pin.test.ts`. - Chat `list_apis` / `call_api` derive from these pins plus `scripts/api/assistant_catalog.py`. Do not edit a handwritten OPERATIONS seed. A new FastAPI pin is enough for the assistant to call it; a Next-only route appears automatically and fails `web/tests/assistant-catalog-freshness.test.ts` until `web/lib/assistant/nextLoaders.ts` gains a static import for that route id. - Read through **`dbFirstRead`** (`web/lib/dbFirstRead.ts`): `fromDb` selects the latest `scan_snapshots` row (`WHERE service = '' ORDER BY scan_time DESC LIMIT 1`), `fromDisk` reads `data/.json`, and the helper serves whichever content timestamp is fresher. - `MAX_AGE_MS` derives from the real cadence with slack (5-min timer → 30min), commented with the reasoning: "older than X means the writer is down." A session-gated writer (Mon-Fri timer, RTH-only scanner) must NOT get a private budget: take `getFreshnessWindowMs("", "closed")` from the shared catalog (`web/lib/serviceHealthWindows.ts`, mirrored in `scripts/watchdog/services.py`) so the route and the watchdog agree. A daily Mon-Fri timer with a private 48h budget showed Friday's healthy snapshot as an outage every Sunday (vol-cone, `e7323e4e`; dispersion R-450 before it). - **Missing contract**: absent data is `HTTP 200` + a frozen `{ missing: true, scan_time: null, series: [], ... }` object — never a 4xx (feedback_http_status_for_real_errors). - `setCacheResponseHeaders(response, { maxAgeSeconds, staleWhileRevalidateSeconds, requestId, cacheState: "HIT", tags: [""] })` scaled to cadence (daily → 300/3600). - Hook `web/lib/use.ts`: `useSyncHook({ endpoint, interval, hasPost, extractTimestamp: d => d.scan_time })`. Poll interval matches cadence (hourly for a daily series); `0` pauses. ## 4. Chart tab (`web/components/Panel.tsx`) - Gate order: `SpectralLoader` (`label="Loading series"`) while `(loading || syncing) && !data` → `SectionEmptyState` on `missing:true` → content. - History chart: **`CriHistoryChart`** with up to two series; SPX overlay = left axis, `chartSeriesColor("primary")`, `scaleType: "log"` for multi-decade price; the indicator on the right axis. Title `UPPERCASE`, `xTickFormat` for non-session x domains. - Range: `HistoryRangeChips` + `web/lib/historyRange.ts` presets and **`BrushMinimap`** (`values`, `range`, `onRangeChange`, `onCustom`, `testIdPrefix="-brush"`). Long monthly series default to `All`; session charts default per `defaultPresetForLength`. - Strip: `RegimeStrip`/`RegimeStripCell` desktop, `useViewport()` → `MetricCell` grid mobile. - **Brand tokens only** — `var(--token)` + `color-mix(...)`, never raw hex/rgba; 4px max radius; `InfoTooltip` explains the signal and thresholds; **no em dashes in copy**. - **Freshness copy is derived, never asserted**: header clock renders `lastSync` (= payload `scan_time`); a `SOURCE UPDATED` cell renders the upstream `Last-Modified`/data date. Never write "Refreshes daily/5m" unless it names the actual timer cadence — and grep the repo for existing instances before shipping cadence copy (UI Copy rule). Empty-state copy may reference "the refresh timer" only if that timer exists. - **Next-update countdown is mandatory.** Every indicator panel mounts `_REFRESH} asOf={data.data_date ?? current.date} testId="-freshness-rail" asOfTestId="-strip-asof" />` (`web/components/FreshnessRail.tsx`) directly under the strip, exactly as `IvRankPanel.tsx` does. It renders `As of ` plus a live `Next sample` countdown, and it derives both from `web/lib/refreshSchedule.ts` (`nextRefreshUtc`) and `web/lib/freshnessRail.ts` (`computeFreshnessRail`, `formatCountdown`, `WRITER_GRACE_MS`) — never from a cadence string in the panel. `behind` / `overdue` states come from the ET session calendar, so a Friday-evening reading is "current" over the weekend and only turns amber once the timer slot for the missing session has passed plus the writer grace. An indicator without a timer (on-demand scans) has no schedule to countdown and skips the rail; say so in the spec. - Chart-system spec: panels inherit `surface.paddingPx: 16` / `radiusPx: 4` from `web/lib/chart-system-spec.json` (pinned by og-chart tests). ## 5. Tests (red/green TDD — write these first) - **pytest** (`scripts/tests/test_.py`): checked-in real fixture artifacts (a captured upstream file) parsed at import; expected values derived by inspecting fixtures, never mental arithmetic; **window-relative dates** for anything freshness-related (hardcoded dates rot in CI); migration executed into in-memory sqlite to pin schema + version + upsert idempotency; conditional-GET stub client asserting the 304 path heartbeats without row upserts; monkeypatch `_write_db_cache`/writer — `db.client.get_db()` and `hrana_http` **refuse real connections under `PYTEST_CURRENT_TEST`** by design. - **vitest API** (`@vitest-environment node`): mock `@/lib/db` with a real in-memory `@libsql/client` seeded with the actual table so SQL executes; assert Turso-beats-older-disk, disk fallback, exact `missing:true` object at 200, no cross-service snapshot leak, `route.dynamic === "force-dynamic"`. - **vitest panel** (`@vitest-environment jsdom`): stub `ResizeObserver`; `vi.mock` the hook; factory fixtures (`buildSeries(n)`, `hookState()`); assert loader label, empty state, strip values, chart title, chips/toggles, a NaN guard (`no contains "NaN"`), and the freshness rail: `[data-testid="-freshness-rail"]` renders, `-strip-asof` shows the payload date, and the `Next sample` cell shows a countdown (use `vi.useFakeTimers()` + `vi.setSystemTime()` pinned one hour before the slot and assert the formatted `formatCountdown` value, as `web/tests/freshness-rail-render.test.tsx` does). - **Playwright** (`web/e2e/-tab.spec.ts`): `page.route` mocks for `**/api/` + the ambient routes (`portfolio`, `orders`, `ib-status`), abort `**/api/prices`; assert active tab, rendered paths, brush visible, missing-state copy. Localhost dev server needs no login (`RADON_AUTHLESS_TEST=1` + dev-mode localhost bypass in `web/middleware.ts`). - Run from **repo root** (cwd drift produces bogus failures): full vitest is `bunx vitest run --config vitest.config.ts`; pytest is `python -m pytest scripts/tests scripts/api/tests scripts/trade_blotter` + `python -m pytest cloud/tests -q`. ## 6. Scheduling and deploy - Units in `cloud/services/`: `radon-.service` (`Type=oneshot`, `User=radon`, `WorkingDirectory=/home/radon/radon`, `EnvironmentFile=/etc/radon/env`, `Environment=RADON_DB_NO_REPLICA=1`, `ExecStart=/home/radon/radon/.venv/bin/python /home/radon/radon/scripts/fetch_.py`, `TimeoutStartSec` sized to the job, journald out/err) + `radon-.timer` (`OnCalendar=... UTC`, `Persistent=true`, `RandomizedDelaySec`, `WantedBy=timers.target`). Comment the OnCalendar choice. - Register in `setup-vps.sh` `SERVICE_FILES`, `cloud/tests/test_systemd_services.py`, and `cloud/config/installed-units.sha256` (sha256 of each unit file). The deploy's `install-units` verb then installs the pair root-owned and enables the timer on the next green deploy; verify on the host with `systemctl list-timers 'radon-*'`. - Prod env note: units call the venv python **directly** — the `run_*.sh` wrapper fallback ladder resolves the wrong python on the VPS (feedback_scan_wrapper_fallback_picks_system_python). ## 7. CI gates and shipping CI (`.github/workflows/ci.yml`) gates every push to main, then auto-deploys on green: 1. gitleaks secret scan 2. `bunx vitest run --config vitest.config.ts --coverage` — full config, coverage ratchet lines 75 / functions 78 / branches 65 (`web/app/api/**` is inside coverage `include` — an untested route drops the ratchet) 3. `python -m pytest scripts/tests scripts/api/tests scripts/trade_blotter --cov=scripts --cov=api --cov-branch --cov-fail-under=64` + `python -m pytest cloud/tests -q` (validates your systemd units) 4. Perimeter smoke (next build + curl auth asserts) 5. Deploy to Hetzner (no manual approval; never push while a deploy is in flight — cancelled builds have corrupted production) Playwright is **not** in CI — run it locally and attach the evidence. Ship checklist: one focused commit (stage files explicitly, never `git add -A`), push once, `gh run watch` to green, then verify **production**: latest `scan_snapshots`/history rows in Turso, `curl` the prod API route (authenticated perimeter: anon 401/404 is the perimeter working, not an outage), and a browser screenshot of the live tab. ## 8. Gotchas that cost real debugging time - Breadth's `serviceHealthWindows` category/copy drifted from reality twice (says "on-demand" + "IB gateway" though a 5-min timer + StockCharts now drive it). When the data source or cadence changes, sweep every copy string that mentions it. - The `regime-tab-routes` and `service-health-windows` tests are deliberate lockstep pins — update them with the feature, in the same commit. - In-memory libsql test schemas can drift from the real migration (breadth's fixture omits a PK column). Seed test tables from the migration file when practical. - Divergence/threshold semantics: use strict inequalities and pin the boundary in a test. - A brand-new timer's first `service_health` row may be absent until first fire — no-row-ever = dormant, don't page (feedback_watchdog_dormant_no_row).