# Cf Adahandle Resolver Mainnet Integration Test Full Sync

> QA Automation Engineer — drives a full mainnet integration test of cf-adahandle-resolver on a remote host via SSH. Provisions a fresh Postgres + API stack with docker compose, syncing the chain from the first ADA Handle mint (slot 47931310) to tip, monitors window-relative progress from the DB cursor, then asserts the run is clean (zero block-parse errors through the Conway era / Van Rossem boundary) and that the REST API resolves handles consistently. Use when the user wants to validate a branch end-to-end against mainnet on a remote box (typed `/cf-adahandle-resolver-mainnet-integration-test-full-sync` or asked for a "mainnet full-sync regression run" / "hard-fork readiness sync test").

- Skill: `cardano-foundation/cf-adahandle-resolver-mainnet-integration-test-full-sync` (Agent Skill, multi-file: 14 files)
- Install (CLI): `npx skillmds@latest add cardano-foundation/cf-adahandle-resolver-mainnet-integration-test-full-sync`
- Raw SKILL.md: https://api.skillmd.com/api/skills/cardano-foundation/cf-adahandle-resolver-mainnet-integration-test-full-sync/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: cardano-foundation (https://skillmd.com/u/cardano-foundation)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/cardano-foundation/cf-adahandle-resolver-mainnet-integration-test-full-sync

---


# Mainnet Full-Sync Integration Test (Remote, via SSH)

You are acting as a **QA Automation Engineer**. The user has a remote Linux box reachable over SSH that will:

1. Pull a fresh checkout of `cf-adahandle-resolver` on a target branch.
2. Spin up a clean Postgres + API stack via the project's `docker-compose.yml`, syncing mainnet from the first ADA Handle minting block (**slot 47931310**) up to tip.
3. As the indexer advances, it parses every block on the way to tip — **including the Conway era and the Van Rossem (protocol v11) boundary**. Parsing the whole window without error *is* the core regression: this is what proves the project's yaci-store/yaci-core versions are hard-fork ready.

Your job is to drive this remotely. You forward the remote API (port 9095) and Postgres (port 5432) to localhost via SSH so the regression spot-checks run on your local machine against the live remote stack.

## What "pass" means here

Unlike a fixture-based API test, the regression signal is the **full sync itself**:

1. **Sync reaches tip** — the DB cursor catches up to the live mainnet tip (window progress ≈ 100%), with the cursor era at **Conway** (Van Rossem is intra-Conway).
2. **Zero genuine block-parse errors** across the whole run — no `BlockParseRuntimeException` / "Block parsing error" whose root cause is a decode failure. (Errors whose root cause is "Database is already closed" / "context closed" are teardown/shutdown noise, not parse failures — see below.)
3. **REST spot-checks pass** — two complementary checks:
   - **Golden handles** (`golden-handles.txt`): a fixed set of commonly-known mainnet handles (`cardano`, `hosky`, `minswap`, `bigpey`, `charles`, …) must all resolve and round-trip. Deterministic and reviewable.
   - **Random DB sample**: for a sample of handles drawn from the synced DB, the API's forward lookup matches the stored row and the reverse lookup round-trips.

## Critical rules

- **Never blow away an existing instance silently.** If the remote already has a stack running, the Postgres volume populated, or the target ports occupied, **STOP** and report. Ask whether to clean up or abort. Don't `docker compose down -v` on your own.
- **Fresh sync means empty data.** A clean run requires the `adahandle-db` Postgres volume to be empty. If it already has data, surface it and let the user decide (wipe vs. resume).
- **Don't run the REST spot-checks too early.** They need the `ada_handle` table populated and the sync at (or near) tip. Running them mid-sync produces meaningless results.
- **Full sync is long.** From slot 47931310 to tip is multiple hours on a typical box. Set expectations before kicking off, and prefer long fallback waits (1200–1800s) when polling progress.
- **The build under test is THIS branch's source.** `start-remote.sh` builds the API image from the checked-out Dockerfile — not a published `:latest`.

## Required configuration

Load config from (in priority order):

1. CLI args passed to the skill: `--ssh-host`, `--ssh-user`, `--remote-dir`, `--branch`, `--env-file`.
2. `./.claude/skills/cf-adahandle-resolver-mainnet-integration-test-full-sync/config.env` (gitignored — not committed).
3. Defaults.

| Variable | Default | Required? |
|---|---|---|
| `REMOTE_SSH_HOST` | none | **yes** (host or ssh_config alias) |
| `REMOTE_SSH_USER` | none | no — leave empty when `REMOTE_SSH_HOST` is an ssh_config alias whose `User` directive already sets the login |
| `REMOTE_PROJECT_DIR` | `$HOME/git/cf-adahandle-resolver` | no |
| `BRANCH` | `main` | no |
| `GIT_REPO_URL` | `https://github.com/cardano-foundation/cf-adahandle-resolver.git` | no |
| `ENV_FILE` | `.env` (mainnet) | no |
| `LOCAL_API_PORT` | `9095` | no |
| `LOCAL_DB_PORT` | `5432` | no |
| `SAMPLE_N` | `10` | no — number of handles sampled for REST spot-checks |

If `REMOTE_SSH_HOST` is missing, refuse to start and tell the user to either supply it as an arg or fill in `config.env`. **Never invent or default a hostname.**

A template lives at `config.env.example`. Tell the user to `cp config.env.example config.env` and fill it in; the real `config.env` is gitignored.

## Procedure

You orchestrate via the helper scripts under `./scripts/`. They are intentionally small and atomic so each step is auditable. Source `scripts/load-config.sh` first to populate environment variables, then call the steps in order. All scripts are idempotent and safe to re-run.

### Step 0 — Confirm intent with the user

Before any remote action, echo back the resolved config (host, user, remote dir, branch, env file) and ask the user to confirm. A fresh sync is destructive to any existing data on the remote.

### Step 1 — Preflight check (read-only on remote)

Run `scripts/preflight-remote.sh`. It SSHs in and checks: a running compose stack in `$REMOTE_PROJECT_DIR`; whether ports 9095/5432 are already bound; that the Docker daemon is reachable; and whether `$REMOTE_PROJECT_DIR/.git` exists (informational). If anything is occupied → **STOP**, print findings, and ask:

> "Found an existing instance / occupied ports on `$REMOTE_SSH_HOST`. Options: (a) abort, (b) stop & wipe the existing stack (`docker compose down -v` in `$REMOTE_PROJECT_DIR`) then continue. What do you want me to do?"

Only proceed with explicit approval.

### Step 2 — Prepare repo on remote

Run `scripts/prepare-remote.sh`. Clones or `git fetch` + `reset --hard` + `clean -fdx` + `checkout $BRANCH` + `pull --ff-only`. Prints the resolved commit SHA so the user can verify what's under test.

### Step 3 — Start the docker compose stack

Run `scripts/start-remote.sh`. Confirms `$ENV_FILE` is present, builds the `api` image from source (`--pull` to refresh the base image), `docker compose up -d`, and prints the running containers. If the `adahandle-db` volume already has data, it does **not** auto-wipe — it surfaces this and asks.

### Step 4 — Open SSH tunnels

Run `scripts/open-tunnels.sh`: backgrounds an SSH connection with `-L $LOCAL_API_PORT:localhost:9095` and `-L $LOCAL_DB_PORT:localhost:5432`, stores the PID in `tunnels.pid`, and sanity-checks `http://localhost:$LOCAL_API_PORT/actuator/health`.

### Step 5 — Monitor sync to tip

**Preferred:** run `scripts/monitor-sync-progress.sh 1200 --until-tip` (or background it under a `Monitor`). It reads the chain cursor straight from Postgres and compares against the live mainnet tip from Koios — **window-relative** progress (relative to the slot-47931310 start), correct throughout the run. It exits 0 once window progress ≥ 99.95%. For a one-shot spot check use `scripts/check-sync-progress.sh`.

**Do NOT trust the API's `syncPercentage`** if you look at `/actuator/health` — yaci-store computes it from Byron genesis, so with a mid-chain start slot it reads misleadingly high from the start. The DB-cursor metric (window-relative) is the source of truth.

From slot 47931310 to tip is several hours. Use `ScheduleWakeup` with a long delay (1200–3600s) while waiting, and warn the user the loop runs for a long time. Print progress + ETA periodically.

### Step 6 — Check for parse errors (the hard-fork regression)

Once at tip, run `scripts/check-parse-errors.sh`. It scans the `api` container logs for genuine block-parse failures, **excluding** the benign shutdown-cascade lines (root cause "Database is already closed" / "context closed already" / "Scheduled to stop"). It also reports the cursor era.

- **0 genuine parse errors + cursor era = Conway at tip** → the build parsed the full window including the Van Rossem boundary. This is the headline pass.
- Any genuine parse error → **report it immediately** with the offending block number/slot and the root-cause line; this is a real hard-fork-readiness failure.

### Step 7 — REST spot-checks

Two scripts, both against the tunneled API:

**`scripts/run-golden-handles.sh`** — the deterministic golden check. Reads commonly-known mainnet handles from `golden-handles.txt` and for each asserts:
- **Forward**: `GET /api/v1/addresses/by-ada-handle/{handle}` → HTTP 200, `paymentAddress` well-formed (`addr1…`), `stakeAddress` absent (enterprise address) or well-formed (`stake1…`).
- **Reverse**: `GET /api/v1/ada-handles/by-payment-address/{paymentAddress}` → the handle is in the returned list. (Reverse uses the payment address — always present — so it works for enterprise-address handles too.)

Golden handles are **names only** (no hardcoded addresses): handles are transferable NFTs, so pinning addresses would rot. The check is therefore drift-proof — it asserts these famous handles resolve and round-trip after a full sync. Run at tip (a handle minted after a mid-sync cursor won't exist yet).

**`scripts/run-rest-spotchecks.sh`** — randomized coverage. Samples `SAMPLE_N` handles from the synced `ada_handle` table (JSON-encoded so arbitrary UTF-8 names parse safely; empty names skipped), then asserts forward lookup equals the stored row and reverse-by-stake-address round-trips. A DB-vs-API consistency check, so no false failures from drift. (For known-address assertions, set `EXPECTED_HANDLES` in `config.env` — optional.)

If any golden handle fails after a clean full sync, that's a real regression — report it.

### Step 8 — Report

Summarize: branch + commit tested; time spent (sync, total); final cursor (slot/block/era) vs tip; parse-error count (and first offenders if any); spot-check pass/fail counts (and first failures). Remind the user the stack is **still running** on the remote and ask whether to `docker compose down -v` + close tunnels, or leave it up for inspection.

## Handling failures

- **SSH connection lost mid-run.** Re-open tunnels and resume monitoring. Don't restart the stack.
- **Stack restarts mid-sync.** `STORE_SYNCAUTOSTART=true`, so the indexer resumes from the persisted cursor automatically. Keep monitoring.
- **Stuck health/no progress.** Pull `docker compose logs --tail=200 api` and `... db` remotely; surface the first ERROR line.

### Known issue: sync wedges on a dead relay IP ("Scheduled to stop")

Same root cause as filed upstream for yaci/yaci-store: the Cardano node hostname fronts a load-balanced pool of A-records; yaci-core's TCP client resolves to a **single** IP with no failover, and if it pins a now-unreachable member it loops forever.

**Symptom:** cursor stops advancing (`check-sync-progress.sh` shows no movement); API logs spam `io.netty.channel.ConnectTimeoutException: connection timed out ... :3001`; `/actuator/health` shows the connection out of service / `Scheduled to stop`.

**Mitigation (already configured):** `.env` sets `JAVA_TOOL_OPTIONS=... -Dnetworkaddress.cache.ttl=0 ...` so the JVM re-resolves DNS on every attempt and tends to pick a healthy IP.

**Recovery if it still wedges:** `ssh <host> 'cd <remote-dir> && docker compose --env-file <env> restart api'`. A fresh JVM re-resolves DNS and usually picks a healthy IP, then catches up the gap quickly. The Postgres cursor persists across the restart, so no re-sync.

## Files this skill owns

```
.claude/skills/cf-adahandle-resolver-mainnet-integration-test-full-sync/
├── SKILL.md                     # this file
├── config.env.example           # template, committed
├── config.env                   # actual config, gitignored
├── golden-handles.txt           # commonly-known mainnet handles for the golden check
├── tunnels.pid                  # PID of background SSH tunnel (runtime, gitignored)
└── scripts/
    ├── load-config.sh           # config loader (source first)
    ├── preflight-remote.sh      # remote occupancy check (read-only)
    ├── prepare-remote.sh        # clone / reset / checkout $BRANCH
    ├── start-remote.sh          # docker compose build + up (image from source)
    ├── open-tunnels.sh          # SSH port-forward 9095 + 5432
    ├── close-tunnels.sh         # kill background SSH tunnel
    ├── check-sync-progress.sh   # one-shot window-relative progress (DB cursor vs Koios tip)
    ├── monitor-sync-progress.sh # looped progress + ETA; --until-tip exits at ~100%
    ├── check-parse-errors.sh    # scan api logs for genuine block-parse failures + report era
    ├── run-golden-handles.sh    # golden check: known handles resolve + round-trip (drift-proof)
    └── run-rest-spotchecks.sh   # DB-vs-API consistency + reverse round-trip on sampled handles
```