Verifier Meshsync

Verifying MeshSync at runtime

MeshSync is a Kubernetes agent: it watches the cluster with dynamic informers and publishes resource events to a NATS broker (or to a file). The surface is the running agent against a real cluster - a change is verified by making the changed code path execute and reading what the agent logs and writes, not by running its unit tests (CI does that).

The whole game is: get the agent running against a local cluster, drive the cluster so the changed code path fires, and read the evidence out of the debug log. This skill encodes the parts that are fiddly to rediscover each time.

Quick start

SKILL=.claude/skills/verifier-meshsync/scripts/verify-env.sh
$SKILL up      # pin an isolated kubeconfig to a reachable local cluster + install CRD/CR
$SKILL run     # build the binary and launch it in file mode with DEBUG logging
sleep 10       # let it connect and finish initial discovery
tail -f "$($SKILL logs)"   # watch it; drive events in another shell (see below)
$SKILL down    # stop meshsync, remove the CRD/CR/namespace it added

verify-env.sh only automates the deterministic setup. Driving the surface and choosing what to observe is your job - it depends on what changed.

Why each setup step matters

Read these before trusting the harness; they are the non-obvious traps.

• Don't trust the default kubeconfig context. In practice it often points at an unreachable remote cluster, and mesherykube.New(nil) will silently try to use it. up probes local contexts (docker-desktop, minikube, kind, orbstack, ...) for /readyz and writes a minified, isolated kubeconfig so the binary cannot wander off. Pass MESHSYNC_VERIFY_CONTEXT=<name> to force one.
• WatchCRDs only runs when the MeshSync CR is present. useCRDFlag (in pkg/lib/meshsync/meshsync.go) is true only when the meshsyncs.meshery.io CRD and the meshery-meshsync CR (namespace meshery) exist. Without them the CRD-watch goroutine never starts and you will "verify" nothing. up installs both (meshery-operator CRDs + integration-tests/infrastructure/meshsync.yaml).
• Use file output mode, not broker mode. --output file runs the full discovery pipeline (informers, WatchCRDs, Run) without needing a NATS broker. ListenToRequests is the only broker-only goroutine, and you rarely need it. The written snapshot doubles as proof discovery worked end-to-end.
• DEBUG=true is required to see the discovery/resync debug lines (see main.go: it maps DEBUG=true|1 to logrus debug level).

Drive the surface

Pick the smallest cluster action that makes the changed code run. Examples:

For a MODIFIED burst use a changing annotation value each time (a repeated identical value is a no-op and emits no watch event).

Observe: signal lines

Grep the log for the lines that mark the behavior. These are current as of this writing - grep the source to confirm they still exist before relying on them, since log strings drift:

Count them and reconcile against what you drove - a bare "it logged the right line" is weaker than "I drove N events and got exactly the N signals I expected, and the counts add up". Use a log mark so you only look at lines after an action:

LOG="$($SKILL logs)"; MARK=$(wc -l < "$LOG")
# ... drive events, then wait past the 5s resync debounce ...
tail -n +$((MARK+1)) "$LOG" | grep -E 'Resyncing informer|did not change discovery|dynamic shared informer factory' \
  | sed -E 's/.*msg="//; s/".*//' | sort | uniq -c

Also sanity-check the whole run: grep -iE 'level=error|panic:' "$LOG".

Reconcile and clean up

Reconcile the totals so the picture is airtight, e.g. "resync-triggers = startup ADDs + my ADD + my DELETE; modified-skips = my MODIFIED burst + the trailing status/finalizer MODIFIEDs k8s emits on create/delete". Then $SKILL down and confirm the process is gone (pgrep -f meshsync-bin).

Worked example: the CRD re-list storm fix

Verifying that CRD MODIFIED events no longer trigger a full informer resync (the cert-manager cainjector caBundle storm):

1. up + run, wait ~10s. Startup lists existing CRDs as ADDED; genuinely new ones (e.g. brokers, meshsyncs) each drive one resync, coalesced by the 5s debounce into one Creating new dynamic shared informer factory.
2. Fire 5 MODIFIED events: for i in 1 2 3 4 5; do kubectl annotate crd brokers.meshery.io storm=$i-$RANDOM --overwrite; sleep 1; done, wait 8s. Expect 5x did not change discovery config; skipping informer resync, 0 resyncs, 0 rebuilds.
3. kubectl apply a dummy CRD -> 1 resync + 1 rebuild; its trailing status MODIFIEDs are skipped. kubectl delete it -> 1 resync + 1 rebuild; finalizer MODIFIEDs skipped.
4. Confirm the snapshot has resources (grep -c 'kind:' "$WORKDIR/snapshot.yaml") and the log has no errors.

Runtime-observable paths NOT covered by this harness: idempotent re-list on watch re-establishment and exponential backoff both need a watch close / API-server failure you can't force cheaply - lean on the unit tests for those and say so.

Report

Follow the /verify report format: verdict, claim, method, numbered steps each with what you did to the running agent and what it logged, an evidence sample (the grepped signal lines), and findings. Never report PASS from tests or a clean build - only from the agent actually doing the thing at its surface.