Vp Deploy Test

Validated Pattern Deployment Tester Skill

When to Use

Purpose: Use this skill to build confidence that the Validated Pattern deployment is sound — that installation succeeded, all Applications converged, secrets are flowing, and jobs completed — before handing the environment to students or running workshop-tester. Passing all phases means an operator can run this pattern with limited or no issues.

• After pattern.sh make install or VP Operator installation completes and you want to verify the pattern is fully deployed
• ArgoCD Applications were created but you're not sure all are Healthy/Synced
• Secrets are not reaching application pods (ESO or Vault issues)
• Imperative jobs in the jobs/ directory have not run or are failing
• Before running student-readiness or handing the environment to students
• After fixing a vp-refactor audit finding and re-deploying

Do NOT use this skill to initialize a new pattern from scratch — use the patternizer skill instead. Do NOT use this skill to audit pattern files without a live deployment — use the vp-refactor skill instead. To check an already-running deployment without re-installing, use the vp-deploy-validator skill instead.

Instructions

This skill defines a four-phase process. Work through the phases in order. Do not proceed to the next phase if the current phase has unresolved failures.

Gotchas

• values-secret.yaml must NEVER be committed to git — it contains real credentials for Vault/ESO
• ArgoCD Application sync can take 10+ minutes after initial install — do not flag as failed prematurely
• The VP Operator must be installed BEFORE running make install — check with oc get csv -n openshift-operators | grep patterns
• Imperative jobs run as CronJobs, not regular Jobs — check with oc get cronjob not oc get job
• Secrets delivery via ESO requires the ExternalSecret CR to exist AND the SecretStore to be healthy — check both

Required Input

Before starting, collect:

Phase 1 — Pre-flight

Verify the cluster and pattern repo are ready before installing.

If values-secret.yaml is tracked by git, stop immediately:

CRITICAL: values-secret.yaml is committed to git.
This file contains secrets and must never be committed.
→ Remove it from git history before proceeding:
  git rm --cached values-secret.yaml
  echo "values-secret.yaml" >> .gitignore
  git commit -m "remove secrets from git"

If any other pre-flight check fails, stop and provide the corrective action. Do not proceed to Phase 2.

Phase 2 — Install

Run the pattern installation and capture the result.

CLI install:

Before running pattern.sh make install directly, check whether a deploy.sh script already exists in the pattern repo root:

ls deploy.sh

If deploy.sh exists: run it instead — it includes the values-secret.yaml pre-check required for non-interactive install validation:

./deploy.sh

If deploy.sh does not exist: run pattern.sh make install directly, then recommend generating the script using the patternizer skill's "Independent Deployment Scripts" section:

cd ~/my-pattern
./pattern.sh make install

VP Operator install:

Confirm the operator has picked up the GitOps repo:

oc get pattern -n openshift-operators

Expected: a Pattern CR exists and shows the target GitOps repo URL.

Capture:

• Exit code for CLI install (0 = success, non-zero = failure)
• Whether the ArgoCD ApplicationSet was created
• Whether any interactive prompt appeared in the output

oc get applicationset -n openshift-gitops

Non-interactive install check (CLI method only):

While streaming or reviewing the pattern.sh output, watch for any of these patterns indicating user interaction was required:

• A line containing [y/N], [Y/n], (yes/no), Enter password:, Password:, or Select with a numbered choice
• The process pausing without output for more than 30 seconds (TTY wait)
• A read prompt from a shell script
• An oc login prompt requesting credentials

If any interactive prompt is detected:

Deployment Test — Phase 2: SUBMISSION_BLOCKING — Non-interactive install failure
  Pattern:   <pattern-name>
  Prompt:    <exact prompt text>

  A Validated Pattern must deploy fully unattended via pattern.sh or the VP
  Operator to be accepted at any VP tier. This is a submission-blocking issue.

  Action: Locate and remove all interactive prompts from:
  - pattern.sh and all Makefiles it calls
  - Any shell scripts called during install
  - values files (no prompts for missing values — all must have defaults
    or be provided via values-secret.yaml)
  Common causes:
  - `read` calls in install helpers without a default
  - oc login without pre-configured credentials
  - Helm `--set` prompts for required but unset values
  - `confirm.sh` style scripts

After fixing: re-run this phase to verify a clean unattended install.

Record this as SUBMISSION_BLOCKING and stop. Do not proceed to Phase 3 until the install is confirmed non-interactive.

If install fails (non-zero exit, no interactive prompt):

Deployment Test — Phase 2: Install FAILED
  Pattern:   <pattern-name>
  Method:    <cli|operator>
  Exit code: <N>
  Error:     <last error line>

Next step: activate vp-refactor to audit the pattern structure.
  Common causes:
  - main.clusterGroupName in values-global.yaml does not match a values-<name>.yaml filename
  - values-<cluster>.yaml missing clusterGroup.applications or namespaces blocks
  - pattern-metadata.yaml missing or malformed
  - OpenShift GitOps not installed before pattern.sh make install

Stop and report. Do not proceed until the user has resolved the failure and re-installed.

Phase 3 — ArgoCD Convergence

After the ApplicationSet is created, poll until all ArgoCD Applications reach Healthy/Synced. VP deployments are async — convergence may take several minutes.

# List all Applications created by this pattern
oc get applications -n openshift-gitops

# Check for any not Healthy/Synced
oc get applications -n openshift-gitops \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.health.status}{"\t"}{.status.sync.status}{"\n"}{end}'

Poll every 30 seconds, up to 10 minutes. Report status at each interval.

Convergence success criteria:

• All Applications show health: Healthy and sync: Synced

If convergence fails (timeout or Degraded/OutOfSync):

For each failing Application, collect:

oc describe application <app-name> -n openshift-gitops

Classify the failure:

Deployment Test — Phase 3: ArgoCD Convergence FAILED
  Application:  <app-name>
  Health:       Degraded
  Sync:         OutOfSync
  Reason:       <ArgoCD error message>

Next step: activate vp-refactor to audit values files and chart structure.

Phase 3 Convergence Report

Deployment Test — Phase 3: ArgoCD Convergence
──────────────────────────────────────────────────────
 Application                  Health      Sync        Notes
 <pattern>-hub                Healthy     Synced      —
 <pattern>-acm                Healthy     Synced      —
 <pattern>-app1               Degraded    OutOfSync   Helm render error: missing ingress.host
──────────────────────────────────────────────────────
 Converged: <N>/<N> Applications

Phase 4 — Secrets and Jobs Verification

After convergence, verify that secrets are flowing and imperative jobs have run.

4a. Secrets delivery (Vault + ESO)

Check ExternalSecret resources are in Ready state:

oc get externalsecrets -A

Expected: all ExternalSecrets show READY=True and STATUS=SecretSynced.

If any ExternalSecret shows SecretSyncedError:

oc describe externalsecret <name> -n <namespace>

Common causes:

• Vault is not initialized or unsealed
• ESO SecretStore credentials are incorrect
• The secret path in Vault does not match the ExternalSecret spec

4b. Imperative jobs

If the pattern has a jobs/ directory, check that CronJobs have completed at least one successful run:

oc get cronjobs -n <imperative-namespace>
oc get jobs -n <imperative-namespace>

Expected: at least one Job per CronJob with COMPLETIONS: 1/1.

If jobs are not running or failing:

• Verify the CronJob schedule is not never (patterns sometimes use @once for bootstrap jobs)
• Check job logs: oc logs job/<job-name> -n <namespace>

4c. Student-readiness hand-off

After secrets and jobs checks pass, activate the student-readiness skill:

Phase 4 checks passed. Activating student-readiness for environment validation.
→ Use the student-readiness skill with cluster: <api-url>

If student-readiness fails:

student-readiness reported failures. Likely causes:
  - An Application converged but its workload pods are CrashLooping
  - A secret was not delivered to the expected namespace
  → Escalate to vp-refactor to audit the values files and chart structure

Phase 4 Report

Deployment Test — Phase 4: Secrets and Jobs
──────────────────────────────────────────────────────
 Check                        Status   Notes
 ExternalSecret: <name>       PASS     SecretSynced
 ExternalSecret: <name>       FAIL     SecretSyncedError: Vault path not found
 CronJob: <name>              PASS     1/1 completed
 student-readiness            PASS     —
──────────────────────────────────────────────────────

After all Phase 4 checks pass: the deployment is confirmed sound. The next step in the confidence chain is workshop-tester:

All deployment tests passed.
→ Activate workshop-tester to validate module exercises against this environment.
  This is the next step in building operator confidence before going live.

Final Summary Report

Validated Pattern Deployment Test — Complete
══════════════════════════════════════════════════════
 Pattern:  <pattern-name>
 Cluster:  <api-url>
 Method:   <cli|operator>

 Phase 1 — Pre-flight          PASS
 Phase 2 — Install             PASS
 Phase 3 — ArgoCD Convergence  PASS / FAIL
   Applications converged      <N>/<N>
 Phase 4 — Secrets and Jobs    PASS / FAIL
   ExternalSecrets             <N>/<N> Ready
   CronJobs                    <N>/<N> completed
   student-readiness           PASS / FAIL
══════════════════════════════════════════════════════
 Overall: READY FOR STUDENTS / NEEDS ATTENTION

 Confidence: HIGH — all phases passed. An operator can run this pattern
             with limited or no issues.
             Next step: run workshop-tester to validate module exercises.

Remediation Plan

When the overall result is NEEDS ATTENTION, generate an ordered, prioritized remediation plan from the actual failures found. Create each item as a session todo so progress is tracked.

Prioritize in this order:

Present the plan as an ordered list containing only the items that actually failed:

Deployment Remediation Plan — <pattern-name>
──────────────────────────────────────────────────────
Priority  #  Finding                                Action
BLOCKING  1  hub app SyncFailed (Helm render error) Activate vp-refactor → Audit Area 3
HIGH      2  ExternalSecret db-creds SecretSyncedError  Activate vp-refactor → Audit Area 4
MEDIUM    3  CronJob bootstrap-job not completing   Check job logs, vp-refactor Audit Area 8
──────────────────────────────────────────────────────
Created 3 todos. Resolve in order — BLOCKING items first.

After presenting the plan, ask:

"Would you like to start with item 1 now? (y/n)"

If yes, activate the indicated skill immediately.

Re-test After Fixes

After the developer resolves one or more items from the remediation plan, re-run only the phases that contained failures rather than a full reinstall:

Re-running failed phases only:
  Phase 3 (ArgoCD convergence) — re-checking Application health after Helm fix
  Phase 4 (Secrets and jobs) — re-checking ExternalSecrets and CronJobs

Skipping Phase 1 (pre-flight passed) and Phase 2 (install succeeded).

For ArgoCD re-checks, trigger a manual sync first rather than re-installing:

argocd app sync <app-name> --server <argocd-url>
# or
oc patch application <app-name> -n openshift-gitops \
  --type merge -p '{"operation":{"sync":{}}}'

Compare results against the previous run and show which items were resolved:

Re-test Results — <pattern-name>
──────────────────────────────────────────────────────
 Item  Finding                         Previous   Now
 1     hub app SyncFailed              FAIL    →  PASS  ✓ resolved
 2     db-creds SecretSyncedError      FAIL    →  FAIL  still failing
 3     bootstrap-job not completing    FAIL    →  PASS  ✓ resolved
──────────────────────────────────────────────────────
 Resolved: 2/3   Remaining: 1

Repeat until all items resolve, then present the full Final Summary Report as READY FOR STUDENTS.

Escalation

• Install failure or ApplicationSet not created → Use the vp-refactor skill to audit values files, chart structure, and pattern-metadata.yaml
• ArgoCD Application Degraded/OutOfSync → Use the vp-refactor skill, Audit Area 3 (Charts Directory) and Audit Area 1 (Values File Completeness)
• ExternalSecret errors (Vault/ESO) → Use the vp-refactor skill, Audit Area 4 (Secrets Management)
• Imperative jobs failing → Use the vp-refactor skill, Audit Area 8 (Imperative Jobs)
• student-readiness failures after successful convergence → Use the student-readiness skill's troubleshooting decision tree; if the root cause is in the pattern, escalate to vp-refactor