Verify MongoDB Percona Migration
When to use
Use this skill after migrate-mongodb-to-percona, or standalone to
health-check a Percona-backed installation.
Hard gate for step 2: the restore is a mutating operation. Run it ONLY when
this skill is part of a migration run started by the readiness and migration
skills AND the operator has confirmed, in this conversation, the specific
archive file to restore. In standalone health-check mode, steps 2 and 3 are
always skipped: never restore an archive into an installation as part of a
health check.
Inputs
NVSENTINEL_NAMESPACE (default nvsentinel)
DATA_PATH (restore is the default, clean is the opt-out) and
ARCHIVE (restore path only), carried over from the migration skill
Setup
export NVSENTINEL_NAMESPACE="nvsentinel"
Steps
Run the gates:
scripts/mongodb-migration/verify.sh
Five gates: psmdb resource ready, initialization job Complete, mongod
pods ready, MONGODB_URI pointing at mongodb-rs0, and EVERY deployed
datastore consumer (health-events-analyzer, fault-quarantine,
node-drainer, fault-remediation) logging Successfully pinged. All five
must pass; the script fails gate five when no consumer exists at all,
because connectivity cannot be confirmed. On the restore path this gate
matters doubly: fault-quarantine and health-events-analyzer only see
restored events through their live change streams, so every consumer
must be healthy BEFORE the restore runs. Timeouts are tunable
via VERIFY_CR_TIMEOUT, VERIFY_JOB_TIMEOUT, VERIFY_POD_TIMEOUT,
VERIFY_PING_RETRIES. A consumer stuck in CrashLoopBackOff with old
restart counts may just be in backoff from the bring-up window; delete
the pod to skip the backoff before diagnosing.
Restore (only within a migration run on the preserve path, with the
archive confirmed by the operator; skipped on the clean path and always
skipped in standalone health-check mode):
All gates from step 1 must have passed first, so every consumer is
running and streaming when the restored documents arrive.
scripts/mongodb-migration/migrate-data.sh restore <ARCHIVE>
The script exits 3 (refused) if any deployed consumer is not ready; that
means a consumer went unready since the gates ran, so go back to step 1
rather than overriding. The consumers stay up during the restore on
purpose: fault-quarantine and health-events-analyzer process the restored
events through their live change streams as the inserts happen.
Then restart the datastore consumers, so node-drainer and
fault-remediation also re-query for unfinished events on startup:
for D in health-events-analyzer fault-quarantine node-drainer fault-remediation; do
kubectl get deploy "$D" -n "$NVSENTINEL_NAMESPACE" >/dev/null 2>&1 && \
kubectl rollout restart deploy "$D" -n "$NVSENTINEL_NAMESPACE"
done
The loop restarts only the deployments that exist in this installation.
Rollout restarts are not spec changes, so they create no GitOps drift.
Restore continuity checks (restore path only):
- Quarantined nodes are still cordoned and their
quarantineHealthEvent
annotations reference documents that exist in the new datastore.
- node-drainer logs show NO repeating
unexpected number of events for node ... lines (that loop means an
annotation references a missing document).
ResumeTokens contains only freshly written tokens (the restore never
carries the old ones).
Aftermath. Walk the operator through what to expect either way:
- Restore path (default): the event exporter has no resume token, so it
re-exports every restored event to its sink; warn the downstream
owners about duplicates. Restored quarantine state is latent until
the next event or cold start touches the node. There is NO CSP
maintenance replay, because the restored maintenance events preserve
the watermark.
- Clean path: the CSP health monitor re-ingests provider maintenance
events still visible in the provider API (its watermark lived in the
wiped database); expect one polling cycle of replay. One-time faults
(GPU XIDs) are not re-detected; the operator must work through the
quarantined-node list recorded during readiness. Persistent faults
are re-detected on the next monitoring cycle.
Output
Report to the operator:
- the verify.sh verdict table
- restore performed or skipped, and the continuity check results
- the aftermath items that apply to this installation
Next skill to run
None; the migration is complete when all gates pass and, on the restore
path, the continuity checks hold. If a gate fails, the failure-branch table
in migrate-mongodb-to-percona and the runbook troubleshooting table are
the recovery references.
References
| Topic |
Reference |
| Runbook (source of truth) |
docs/runbooks/mongodb-bitnami-to-percona-migration.md |
| Scripts |
scripts/mongodb-migration/ |
| Migration skill |
migrate-mongodb-to-percona |
1---2name: verify-mongodb-percona-migration3description: >-4 Post-migration verification for the NVSentinel Percona MongoDB backend:5 waits on the five install gates, performs the optional data restore and6 consumer restarts, and walks the operator through the aftermath7 expectations. Use after migrate-mongodb-to-percona completes, or on its8 own to health-check an existing Percona-backed installation.9maturity: experimental10lifecycle: evergreen11api-version: nvsentinel.skills/v112allowed-tools: Bash(kubectl *), Bash(helm *), Bash(scripts/mongodb-migration/*), Read, Grep13metadata:14 source: https://skillmd.com/skills/nvidia/verify-mongodb-percona-migration15---1617# Verify MongoDB Percona Migration1819## When to use2021Use this skill **after** `migrate-mongodb-to-percona`, or standalone to22health-check a Percona-backed installation.2324Hard gate for step 2: the restore is a mutating operation. Run it ONLY when25this skill is part of a migration run started by the readiness and migration26skills AND the operator has confirmed, in this conversation, the specific27archive file to restore. In standalone health-check mode, steps 2 and 3 are28always skipped: never restore an archive into an installation as part of a29health check.3031## Inputs3233- `NVSENTINEL_NAMESPACE` (default `nvsentinel`)34- `DATA_PATH` (`restore` is the default, `clean` is the opt-out) and35 `ARCHIVE` (restore path only), carried over from the migration skill3637## Setup3839```bash40export NVSENTINEL_NAMESPACE="nvsentinel"41```4243## Steps44451. **Run the gates:**4647 ```bash48 scripts/mongodb-migration/verify.sh49 ```5051 Five gates: psmdb resource `ready`, initialization job `Complete`, mongod52 pods ready, `MONGODB_URI` pointing at `mongodb-rs0`, and EVERY deployed53 datastore consumer (health-events-analyzer, fault-quarantine,54 node-drainer, fault-remediation) logging `Successfully pinged`. All five55 must pass; the script fails gate five when no consumer exists at all,56 because connectivity cannot be confirmed. On the restore path this gate57 matters doubly: fault-quarantine and health-events-analyzer only see58 restored events through their live change streams, so every consumer59 must be healthy BEFORE the restore runs. Timeouts are tunable60 via `VERIFY_CR_TIMEOUT`, `VERIFY_JOB_TIMEOUT`, `VERIFY_POD_TIMEOUT`,61 `VERIFY_PING_RETRIES`. A consumer stuck in `CrashLoopBackOff` with old62 restart counts may just be in backoff from the bring-up window; delete63 the pod to skip the backoff before diagnosing.64652. **Restore (only within a migration run on the preserve path, with the66 archive confirmed by the operator; skipped on the clean path and always67 skipped in standalone health-check mode):**6869 All gates from step 1 must have passed first, so every consumer is70 running and streaming when the restored documents arrive.7172 ```bash73 scripts/mongodb-migration/migrate-data.sh restore <ARCHIVE>74 ```7576 The script exits 3 (refused) if any deployed consumer is not ready; that77 means a consumer went unready since the gates ran, so go back to step 178 rather than overriding. The consumers stay up during the restore on79 purpose: fault-quarantine and health-events-analyzer process the restored80 events through their live change streams as the inserts happen.8182 Then restart the datastore consumers, so node-drainer and83 fault-remediation also re-query for unfinished events on startup:8485 ```bash86 for D in health-events-analyzer fault-quarantine node-drainer fault-remediation; do87 kubectl get deploy "$D" -n "$NVSENTINEL_NAMESPACE" >/dev/null 2>&1 && \88 kubectl rollout restart deploy "$D" -n "$NVSENTINEL_NAMESPACE"89 done90 ```9192 The loop restarts only the deployments that exist in this installation.93 Rollout restarts are not spec changes, so they create no GitOps drift.94953. **Restore continuity checks (restore path only):**96 - Quarantined nodes are still cordoned and their `quarantineHealthEvent`97 annotations reference documents that exist in the new datastore.98 - node-drainer logs show NO repeating99 `unexpected number of events for node ...` lines (that loop means an100 annotation references a missing document).101 - `ResumeTokens` contains only freshly written tokens (the restore never102 carries the old ones).1031044. **Aftermath.** Walk the operator through what to expect either way:105 - Restore path (default): the event exporter has no resume token, so it106 re-exports every restored event to its sink; warn the downstream107 owners about duplicates. Restored quarantine state is latent until108 the next event or cold start touches the node. There is NO CSP109 maintenance replay, because the restored maintenance events preserve110 the watermark.111 - Clean path: the CSP health monitor re-ingests provider maintenance112 events still visible in the provider API (its watermark lived in the113 wiped database); expect one polling cycle of replay. One-time faults114 (GPU XIDs) are not re-detected; the operator must work through the115 quarantined-node list recorded during readiness. Persistent faults116 are re-detected on the next monitoring cycle.117118## Output119120Report to the operator:121122- the verify.sh verdict table123- restore performed or skipped, and the continuity check results124- the aftermath items that apply to this installation125126## Next skill to run127128None; the migration is complete when all gates pass and, on the restore129path, the continuity checks hold. If a gate fails, the failure-branch table130in `migrate-mongodb-to-percona` and the runbook troubleshooting table are131the recovery references.132133## References134135| Topic | Reference |136|-------|-----------|137| Runbook (source of truth) | `docs/runbooks/mongodb-bitnami-to-percona-migration.md` |138| Scripts | `scripts/mongodb-migration/` |139| Migration skill | [migrate-mongodb-to-percona](../migrate-mongodb-to-percona/SKILL.md) |