# Rask Services Fleet

> Gateway (:8888) routing across the rask backend fleet — which service owns an `/api/*` prefix, plus ports and `RASK_*_URL` overrides for compute, controlplane, ingest and the explorer trio (viewer/search/annotator). Use when an `/api/*` call returns `404 no upstream`, `502 upstream unreachable` or a 403; when adding or moving an endpoint or a gateway route row; when changing a port, `RASK_API_PREFIX` or a `RASK_*_URL`; or when reading `scripts/dev-micro.sh`.

- Skill: `ai-riksarkivet/rask-services-fleet` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ai-riksarkivet/rask-services-fleet`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ai-riksarkivet/rask-services-fleet/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: ai-riksarkivet (https://skillmd.com/u/ai-riksarkivet)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ai-riksarkivet/rask-services-fleet

---


# rask services fleet (gateway + per-domain backends)

The day-to-day backend map. The **gateway** on `:8888` is a stateless reverse proxy that path-routes `/api/*` to per-domain services. The old `viewer` monolith is gone; the batches/orchestrator plane died at P7a; and the **R6/R20 media wave (2026-07-28) retired core-api, search-api, and volumes-api** — the S3 object browser now lives in the **explorer viewer** (`/api/explorer/object*`), and lines/EAD FTS re-land as catalog-governed Lance tables behind `/api/explorer/search` (docs/architecture/lance-ns-merge.md). `scripts/dev-micro.sh` is the source of truth for the process list + ports.

⚠️ **The frontend's dev proxy is per-zone and inconsistent** — there is no single "the SPA targets :8888":

| Zones | `/api` proxies to |
|---|---|
| `compute`, `studio`, `models` | `VIEWER_BACKEND` → `:8888`, the gateway |
| `home`, `lakehouse` | `LANCE_BACKEND` → **`:8001`**, the lineage service — and nothing in `dev-micro.sh` serves `:8001` |
| `explorer`, `annotator` | no `/api` proxy at all; they reach `:8101`/`:8102`/`:8103` through their own BFF |

So a `/api/*` call that works in `compute` can 404 or hang in `lakehouse`. See `rask-frontend` for the matching SSR base-URL split.

For FastAPI app/router/lifespan idioms see `fastapi`. This skill is *only* the topology + invariants.

## When to use

- Adding or moving an endpoint — pick the owning service and confirm the gateway prefix routes to it.
- Debugging a `404 no upstream` or `502 upstream unreachable` seen through the SPA.
- Changing a port or pointing the gateway at a remote backend via `RASK_*_URL`.
- Reading/editing `scripts/dev-micro.sh` or wiring a new service into the fleet.

## Fixed port map + env overrides

`scripts/dev-micro.sh` exports `*_PORT` defaults; the gateway reads `RASK_*_URL` (localhost defaults below) so you can point it at remote/containerized backends without touching code.

| Service | Port | Gateway override env | Lifespan builds |
|---|---|---|---|
| **gateway** | 8888 | — (it *is* the proxy) | `httpx.AsyncClient` + route table only |
| **compute** | 8804 | `RASK_COMPUTE_URL` | dashboard httpx client + Ray Job SDK client |
| **controlplane** | 8820 | `RASK_CONTROLPLANE_URL` | k8s client (read-only Project CRs for the home picker) |
| **explorer viewer** | 8101 | `RASK_EXPLORER_VIEWER_URL` | lazy `DatasetRegistry`; the S3 object browser builds its client **per store** from the catalog's storage registry (`RASK_STORES`) — a store declaring a `secret` gets those creds from the Dapr secret store, fail-closed and `lru_cache`d, never the process env (the old env-only `s3_client()` read the external raw tier against the warehouse and listed it as empty). FGA-gated since #90 — see invariant 10 |
| **explorer search** | 8102 | `RASK_EXPLORER_SEARCH_URL` | descriptor-driven Lance search |
| **annotator** | 8103 | `RASK_EXPLORER_ANNOTATOR_URL` | annotations plane |
| **ingest** | 8830 | `RASK_INGEST_URL` | the pre-bronze acquisition plane (control API + workers + the lander) — **`dev-micro.sh` does NOT start it**, so `/api/ingest/*` answers `502 upstream unreachable` against the local fleet |
| **flows** | 8840 | `RASK_FLOWS_URL` | the studio flow-builder's server half (`/api/flows/{catalog,validate,runs}`): an httpx client + an in-memory run store, and a Dapr `WorkflowRuntime` that starts **only** when `DAPR_GRPC_PORT` is set (no sidecar → the inline lane, logged once). `dev-micro.sh` DOES start it. Its row is prefix-interpolated (`f"{prefix}/flows"`), not a literal `/api/flows` — the ingest `/v1` lesson applied rather than restated |
| **notifications** | 8850 | `RASK_NOTIFICATIONS_URL` | the per-subject inbox behind the bell (`/api/notifications/inbox{,/unread,/seen,/dismiss}`): one Dapr **InboxActor** per subject holding claim-check pointers, fed by **two** ingresses: a `lineage.events.v1` subscription on its own pubsub component, and — because the ingest service, Ray TRAIN and every external OpenLineage producer emit over HTTP only and never reach the topic — a `bindings.cron` tick (`notifications-reconcile-cron`, `@every 30s`) that walks lineage's durable `GET /events` down from a persisted cursor. **Lineage self-prunes that feed inline on every ingest**, and the prune cannot consult this cursor — it lives in notifications' Dapr state store, which lineage is not scoped to — so `GET /events` reports `oldest_seq` (the floor it still retains) and the reconciler compares it against its own mark. A mark below that floor means rows were deleted before this lane read them: `ReconcileResult.gapped`, an ERROR (`lineage_feed_pruned_below_cursor`) and the `notifications.feed.gaps` counter. Distinct from `truncated`, which is the walk running out of PAGES — the pruned case exits through the *success* door (`next_cursor: None` reads as "caught up") and was invisible before. That cron route is **root-mounted**, not under the api prefix: Dapr delivers an input binding to `POST /<component name>` at the pod root, so the Component name, `RASK_NOTIFICATIONS_BINDING_NAME` and the served path are one string (all three rendered from `services.notifications.reconcileBindingName`, and pinned together by `tests/unit/test_invariants.py`). Row is prefix-interpolated like flows'. `dev-micro.sh` DOES start it, but **with no sidecar** — `require_actor_plane` then answers every inbox route `503` with the reason rather than 500ing, so the badge is honestly empty, not broken. In-cluster it needs four separate values entries or it is silently useless: `notifications` in `stateStore.scopes` (no actor state store → actor hosting disabled → a healthy pod with a permanently empty bell), `daprIngest: true` (no `APP_API_TOKEN` → the bus handler's `assert_app_token_configured` **crash-loops the pod**), its row in `dapr-resiliency.yaml` (no sidecar retry, no dead-lettering), and `env.RASK_LINEAGE_SERVICE_IDENTITY` (the ONE declaration `services.yaml` scans to build lineage's `LINEAGE_SERVICE_SUBJECTS` allowlist *and* the claim the reconciler sends — omit it and every feed tick 401s, which reads as a credential fault rather than a service never admitted). Admission is only half: the feed is **governed**, so a subject that is allowlisted but granted nothing gets a reconciler that runs cleanly and reconciles nothing |

The gateway also carries the lakehouse rows (`/api/catalog`, `/api/lineage`, `/api/produce`, `/api/train`) plus the **ingest** row `/api/ingest` → the ingest plane (`RASK_INGEST_URL`, `:8830`) — see `gateway/__init__.py::_routes()`. Two traps in that row. It rewrites to **`/api`, not `/v1`**: the ingest module's own docstrings say `/v1/ingests`, which is the ROUTER's path *before* `make_service_app` prepends `settings.api_prefix`, and the `/v1` version that shipped 404'd every call through the gateway. And the `/api/ingest-iiif` row is **GONE** (corrected 2026-08-09 — this skill described it as a live deprecated sibling long after A12 removed it). A12 deleted the medallion route it pointed at, so keeping the row made it **502 rather than 404** — the worse failure of the two, because it names a backend as broken instead of the path as absent. The ORDERING PROPERTY it demonstrated is still load-bearing and still tested: `_pick_route` requires `path == prefix or path.startswith(prefix + "/")`, so a `/api/ingest-iiif` row could never have matched the `/api/ingest` row anyway — the next character is `-`, not `/`. `services/gateway/tests/test_routing.py` pins that.

## Load-bearing invariants

1. **No fleet service owns relational state.** The batches table + Alembic lineage were deleted at P7a; the only databases left are the chart-managed lineage (AGE) and OpenFGA stores, owned by the lance services. Never add a DB engine to a fleet lifespan.
2. **Each service builds only its own `app.state` subset in its own lifespan.** The compute service opens only the dashboard/job clients. Don't widen a lifespan to grab resources the service doesn't use.
3. **Longest-prefix-first routing, NO catch-all.** `gateway/__init__.py::_routes()` returns prefixes most-specific-first; `_pick_route` returns the first whose `path == prefix or path.startswith(prefix + "/")`. Order: the deep explorer rows (`/api/explorer/search`, `/api/explorer/annotations`) before `/api/explorer`, then the lakehouse rows, then `/api/ingest`, `/api/train`, `{prefix}/ray`, `{prefix}/projects`, `{prefix}/flows`, `{prefix}/notifications`, `/api/serve`. **There is no bare `/api` row since R6/R20** — an unmatched `/api/*` 404s with `no upstream`. A new public prefix needs its own route row.
4. **`/api/serve` and `/api/ray` both go to the compute service** (the URL namespace names the Ray cluster, not the service — R22: the SERVICE is `compute` on every surface — uv member, import, k8s/dapr/image — while the public paths stay `/api/ray` + `/api/serve`), but for different reasons: domain routers mount under `RASK_API_PREFIX` (`/api/v1`), while its `proxy_router` mounts at the **root** (no prefix) so `/api/serve/*` reaches the Ray Serve status API. Routers vs proxy_router is the `make_service_app` distinction.
5. **502 contract.** On `httpx.RequestError` (upstream not started / crashed / wrong port) the gateway raises `HTTPException(502, "upstream ... unreachable")` — a clean 502, never a 500 traceback. An unmatched path is a `404 no upstream`. Use the 502 to tell "backend down" from "wrong route."
6. **Hop-by-hop headers are stripped both ways** (`_HOP_BY_HOP`: connection, keep-alive, te, trailers, transfer-encoding, upgrade, host, proxy-*) per RFC 7230 §6.1. Responses stream back via `StreamingResponse(aiter_raw(), background=aclose)`. Don't re-add `Host`/`Transfer-Encoding`.
7. **Merged `/docs`.** The gateway intercepts `{prefix}/openapi.json` and `{prefix}/docs` itself: `_merged_openapi` fans out to every distinct upstream's `openapi.json` and merges `paths`+`components.schemas` into one spec, **skipping unreachable backends** (logged, not fatal). So the gateway's `/docs` shows the whole fleet.
8. **The storage browser's chain is BFF-shaped:** lakehouse zone `/lakehouse/api/explorer/*` (SvelteKit route) → gateway `/api/explorer/*` → viewer `/api/*` (`/api/explorer/objects` → `/api/objects`). Dev needs the viewer running (`dev-micro.sh` starts it); in-cluster it needs `explorer.enabled` + the viewer's rustfs netpol allowlist entry — **and, with `auth.enabled`, a bearer whose subject holds `can_browse_storage` on `RASK_FGA_ROOT_OBJECT`** (owner/estate tier, deliberately not per-store: the shipped default stores come from `DEFAULT_STORES` in code and would never get tuples). `chart/templates/explorer.yaml` sets `RASK_OIDC_*` + `RASK_FGA_*` for **all three** explorer services — it was `if and (eq $name "annotator") auth.enabled`, so the viewer streamed page images and browsed S3 wide open on an auth-enabled estate; the vars change behaviour only where a route declares an auth dependency, so `search` is unaffected. The lakehouse proxy forwards the signed-in user's bearer but does not `requireSession`, so an anonymous browse arrives credential-less and is denied at the viewer, not at the BFF. Dev stays open — `RASK_FGA_ENABLED` unset ⇒ the checker is permissive by construction.
9. **Paths are canonicalized before matching.** `_normalize_path` (`gateway/__init__.py`) collapses `.`, `..`, and duplicate slashes, preserving a trailing slash — so `..`/`//` variants can neither dodge the 403 blocklist nor slip past a longer prefix into a shorter one. This replaced nginx's `merge_slashes` + URI normalization.
10. **A 403 has two possible authors, and only one of them is the gateway.** The gateway's own 403 is `lineage_sidecar_guard` (`gateway/__init__.py`), which prefix-matches the **normalized, case-folded** path against `_lineage_sidecar_only_routes()` and returns `403 {"detail": "sidecar-only lineage route: <route>"}` before the `/api/lineage` proxy runs — the nginx `lance.lineageSidecarOnlyRoutes` blocklist rewritten in Python, with the services' own app-api-token check still the load-bearing guard. Every OTHER 403 through the gateway is a **proxied FGA denial**: since #90 the viewer gates `/api/datasets` + `/api/pages` on `can_get_metadata`, `/api/page` (image bytes) on `can_read_data`, and `/api/object{,s,/download}` on `can_browse_storage` against `RASK_FGA_ROOT_OBJECT` (`viewer/api/security.py`), and the annotator gates its task plane. `Authorization` is not hop-by-hop, so the gateway forwards the bearer the BFF attached untouched and the service is what verifies it. Read the `detail` to tell them apart: the sidecar guard names a *route*; an FGA denial reads `<subject> lacks <relation> on <object>`.
11. **The annotator mounts two planes, and the gateway publishes ONE ROW of one of them.** Edge-reachability is the gateway table's call, never the path's shape: the only annotator row is `/api/explorer/annotations` → `/api/annotations`, so `/api/assist` and `/api/jobs` carry the `/api` prefix and are **in-cluster only** (`jobs.py` states this correctly). `/projects` and `/tasks` are the actor plane — no gateway row, deliberately, because the annotator zone's SSR calls them directly in-cluster on `ANNOTATOR_PROJECTS_API` (`frontend/microfrontends/annotator/src/lib/server/doors.ts`). `require_actor_plane` is attached to the `/tasks` router alone (`tasks.py:84`), so an unregistered actor plane is a 503 there and a 500 on `/projects`. Do not "unify" the prefixes: `/api/projects` already belongs to the **controlplane** (invariant 3's `{prefix}/projects` row), and publishing the actor plane at the edge is a separate decision a prefix move would not make. `services/annotator/tests/test_route_prefixes_are_declared_in_one_place.py` pins the mounted set and refuses a third shape.


## Gotchas

- **`RASK_API_PREFIX`'s code default is `/api/v1`, and nothing uses it.** Every deployment sets `/api` (`chart/values.yaml` under `config:`; `scripts/dev-micro.sh`; `.env.example`). Leave it unset and `/api/ray` and `/api/projects` silently move to `/api/v1/...` — off the paths every frontend client hardcodes. Gateway routing is built from the same value, off `GatewaySettings` (`services/gateway/src/gateway/config.py`) — a pydantic-settings model with `env_file=".env"`, so it reads the same `.env` the services do; keep them in sync. It used to be sixteen raw `os.environ.get()` reads with a `load_dotenv()` in front of only some of them, which is why `RASK_DOCS` in a `.env` was silently ignored (FLEET-ENV-SCATTER).
- **`scripts/dev-micro.sh` deliberately does NOT bash-source `.env`.** Each service loads it via `python-dotenv` so JSON-list settings like `RASK_CORS_ORIGINS=["..."]` parse correctly; bash sourcing strips the quotes. Export only vars *not* in `.env`.