Airgap Vetting

airgap-vetting

Rapidly vet a candidate OSS product for air-gap readiness before adoption. Answers eight questions and assigns one of four grades. The organizing insight from the research behind this skill: "documents an offline flag" ≠ "works offline" — several tools fall back to downloading even with their own offline flag set, and the decisive distinction (air-gap-native vs proxy-in-disguise) is only provable by denying egress after activation and watching whether the product keeps working.

Arguments (parse from $ARGUMENTS):

• target (first positional, required): a product name (meilisearch), a local source checkout, a git URL, or a container image ref. Resolve whatever else is needed in Phase 0.
• --dynamic: run the Phase-2 dynamic harness. Without it the vet is static-only and the verdict says so.
• --repo PATH / --image REF: explicit source checkout / image when the positional target is ambiguous.
• --stakes low|high (default low): high mandates the dynamic pass for any candidate that survives static; low allows a static-only provisional grade. Enforcement lives in the gate section.

The eight questions:

Read each reference file at its question — they carry the exact hostnames, env vars, grep patterns, and known counter-examples. Do not answer a question from general knowledge when the reference has a table for it: the tables encode verified, sourced fingerprints, including several cases where docs and behavior disagree.

Phase 0: Scope the target

1. Check prior art. Read references/known-products.md. If the candidate (or its direct dependency) already has a row, say so up front — the row may answer the whole request, or at least pin expectations.
2. Resolve the target. Get a local source checkout (clone if needed; prefer an existing clone under ~/projects/github.com/<org>/<repo>) and, when a container distribution exists, the image ref. Record versions: git tag/commit and image tag@digest. The verdict is version-specific — pin what was actually vetted.
3. Identify the deployment shape (CLI, server, K8s chart/operator, desktop app). This decides which static checks apply (Helm image enumeration only for charts; browser-console check only for web UIs).
4. Locate the docs' own air-gap story. Search the repo/docs for pages titled "air-gap", "offline", "disconnected", "self-hosted". Their existence is a green flag (Q8) and their claims become hypotheses for the static pass to check — never conclusions.

Phase 1: Static pass (always)

Work through Questions 1–7 in order. For each, record in working state: answer (yes/no/partial/unknown), evidence[] (exact grep hit with file:line, hostname, or doc URL), and flags[] (red/green signals for grading). Every claim in the final report carries evidence — a verdict row without a grep hit or citation is a guess, and guesses get unknown, not no.

Q1 — Telemetry. Read references/telemetry.md §Fingerprints. Run BOTH grep layers (SDK packages AND hostnames) — one alone misses what the other catches — plus, when an image is in scope, every §Container-image check (binary strings, ENV, entrypoints, registry-host CNAME).

Q2 — Opt-out semantics. Read references/telemetry.md §Opt-out. Grep for kill-switch-shaped tokens (telemetry-adjacent substrings, not an exact-name list — the mere presence of opt-out handling is evidence telemetry exists), then check all six pitfalls against the code that actually reads the variable. Verify polarity in source, not docs.

Q3 — Proxy-in-disguise. Read references/downloads-and-proxy.md §Proxy. Repo topology, hardcoded base URLs, BYOK-routing docs, pricing-page "self-hosted = Enterprise" tells, mandatory-login issues.

Q4 — Runtime downloads. Same file, §Downloads. Build-time vs runtime is the organizing principle. For Helm charts, enumerate images from rendered helm template output, never values.yaml alone; runtime-computed image names are an automatic no-go.

Q5 — Custom CA. Read references/ca-trust.md. Identify the TLS stack per ecosystem, check additive-vs-replacement semantics, and hunt the two hard blockers plus the verification-off-instead-of-CA anti-pattern.

Q6 — Content egress & offline degradation. Read references/content-egress.md. Inventory every feature that sends user content, with kill switches, plus CDN-loaded frontend assets; run the identity and inbound sub-checks at the tail of that file. Output is a per-feature table: feature → destination → default on/off → kill switch → what's lost offline.

Q7 — Sustainment. Read references/sustainment.md. How do upgrades and content feeds (vuln DBs, rules, models) cross the gap, and what does the product do when a feed is stale — fail-closed, warn, or silently report green? Find the age-validation code (or its absence). Count the gap crossings an upgrade path requires.

Signature-verification and time sub-checks (feed Q4/Q5/Q7): if the product or its install path verifies signatures (cosign, Notation, GPG) or carries an expiring license, read references/verification-time.md and run its greps — provenance verification is itself a common hidden egress.

Static-pass greps run offline; only Phase 0 target resolution and optional gh issue-searches need network. When vetting from inside a gap, skip the gh checks and note them as not-run in the report.

Gate: static → dynamic

Run the dynamic pass when any of:

• --dynamic was passed, or --stakes high and the candidate survived static;
• Q3 is unresolved (docs market "offline" but static found license-ping or activation endpoints) — the post-activation egress-deny test is the mandated tie-breaker between possible-with-mirror and proxy-in-disguise; do not guess this boundary statically;
• Q2 found opt-out flags whose actual behavior can't be confirmed in source (compiled binary, obfuscated build);
• Q5 hinges on runtime behavior (does the binary honor HTTPS_PROXY + custom CA?).

If the gate fires but --dynamic was not passed and stakes are low, emit the static-only report with grade suffix (provisional — dynamic pass not run) and list exactly which dynamic steps would settle which questions.

If stakes are high but the dynamic pass cannot run (no Docker, no disposable host), do NOT emit a final grade — that is the whole point of high. Report the static findings, set grade_provisional: true with the grade left as the static best-guess, and state plainly that a final verdict requires running the listed dynamic steps on a disposable host.

Phase 2: Dynamic pass (gated)

Read references/dynamic-harness.md and run all eight steps of its ladder on a disposable host/VM — isolation, deny+log, passive enumeration, mitmproxy CA injection (settles Q5 and inventories Q1), post-activation egress-deny (the Q3 tie-breaker), browser-console under deny (Q6), stale-feed (Q7), faketime skew. Capture every step in two windows — first-run AND 10–30 min steady-state; timer-based heartbeats don't show up in a 30-second capture.

Phase 3: Grade (Question 8)

Assign exactly one grade. When evidence splits across grades, the worse grade wins — the cost of over-promising air-gap readiness (a dead product inside the gap) far exceeds the cost of extra mirroring work.

air-gap-native — complete published artifact enumeration (image list / relatedImages / SBOM); digest-pinnable references; ONE global registry override covering ALL images including init-containers and operator-pulled ones; license validated locally via cryptographic signature with zero callbacks ever; every outbound feature listed with a documented kill switch or local-mirror path; custom-CA injection works; offline docs artifact; installer completes with egress fully denied; content feeds published as mirrorable artifacts (OCI/HTTPS) with override flags and non-silent staleness handling; a published feature-degradation table.

possible-with-mirror — all artifacts enumerable and mirrorable (possibly with effort); registry overrides exist but per-image rather than global; offline license path exists but may be approval-gated; some features degrade offline in a documented way; requires standing mirror infrastructure and periodic content sync. Also the cap for: default-on content-egress features that DO have a self-hosted path, and security feeds that go silently stale (state the silent-stale warning explicitly in the verdict).

proxy-in-disguise — vendor markets "offline/disconnected" but operation requires allowlisted egress to specific endpoints: license-ping servers, telemetry gateways, activation callbacks, update checks that gate functionality, or login that requires the vendor's cloud with no offline workaround. Works behind an HTTP proxy, dies behind a true gap. Decisive objective test: does the product survive a permanent egress-deny applied after activation? Note: air-gap sold only as a negotiated license tier is a proxy-in-disguise trigger even when technically mirrorable.

no-go — hardcoded URLs with no override; image/artifact names computed at runtime (defeats all mirroring); install scripts that fetch mid-run with no offline flag; license enforcement that shuts the product down absent connectivity with no offline tier; no way to enumerate what to mirror; TLS certificate pinning or compiled-in-roots-only on endpoints that must be mirrored.

Phase 4: Output

4a. Write ./AIRGAP-VETTING.json

{
  "vetting_completed": true,
  "candidate": {
    "name": "...",
    "version": "git tag/commit and/or image@digest actually vetted",
    "source": "repo URL", "image": "ref or null",
    "deployment_shape": "cli|server|k8s-chart|k8s-operator|desktop"
  },
  "mode": "static-only|static+dynamic",
  "stakes": "low|high",
  "grade": "air-gap-native|possible-with-mirror|proxy-in-disguise|no-go",
  "grade_provisional": false,
  "questions": [
    {
      "id": 1,
      "question": "Sends telemetry to internet?",
      "answer": "yes|no|partial|unknown",
      "evidence": ["exact grep hit with file:line, hostname, or captured destination"],
      "flags": ["red: ...", "green: ..."],
      "kill_switches": ["ENV_VAR=value (polarity verified in source at file:line)"],
      "dynamic_confirmed": false
    }
  ],
  "egress_inventory": ["hostname — purpose — first-seen (static grep | dns capture | mitm)"],
  "mirror_list": ["everything that must cross the gap: images@digest, feeds, models, docs"],
  "sustainment": {
    "feeds": [{"feed": "...", "mechanism": "...", "staleness_behavior": "fail-closed|warn|silent"}],
    "upgrade_gap_crossings": "packages/artifacts per upgrade hop",
    "sync_cadence_required": "..."
  },
  "not_run": ["checks skipped and why"]
}

Every question appears exactly once, unknown answers stay unknown, and not_run lists what a follow-up should do. Do not print this JSON to the terminal; write to file only.

4b. Write ./AIRGAP-VETTING.md

Reviewer-facing, in this order: title + one-line verdict; the grade with its two or three decisive pieces of evidence; the eight-question table (question | answer | strongest evidence); the per-feature content-egress table (Q6); the mirror list; the sustainment summary; Not run / next steps. Keep it under ~120 lines — the JSON carries the full detail.

4c. Append to the known-products log

Append one row to ./AIRGAP-KNOWN-PRODUCTS.md in the working directory (create with header if absent, same columns as references/known-products.md). If the skill directory itself is writable (in-repo development, not an installed plugin), ALSO append the row to references/known-products.md so the skill compounds across runs.

4d. Terminal summary

Under ~10 lines: grade (+ provisional marker), the single most decisive finding, count of red/green flags per question, where the reports were written, and — if static-only — the one dynamic test most worth running.

Maintenance

Smoke tests (expected grades for three known-products rows), design rationale, composition with sibling skills, and provenance live in references/maintenance.md. Re-verify the version-gated facts periodically — cosign v2 vs v3 flags, DCT shutdown 2026-12-08, and the dated known-products rows all drift.