Agnosticd Deploy Test

AgnosticD Deployment Tester Skill

When to Use

Purpose: Use this skill to build confidence that the AgnosticD deployment is sound — that provisioning succeeded, workloads are running, data is flowing, and the lifecycle works — before handing the environment to students or running workshop-tester. Passing all phases means an operator can run this workload with limited or no issues.

• After agd provision completes and you want to verify the deployment is fully healthy
• Workloads appear to have deployed but you're not sure all are running correctly
• agnosticd_user_info data is missing from the RHDP catalog or Showroom
• The stop/start lifecycle has never been tested for this config
• Before running student-readiness or handing the environment to students
• After fixing an agnosticd-refactor audit finding and re-deploying

Do NOT use this skill to set up AgnosticD v2 from scratch — use the agnosticd skill instead. Do NOT use this skill to audit config files without a live deployment — use the agnosticd-refactor skill instead.

Instructions

This skill defines a four-phase process. Work through the phases in order. Do not proceed to the next phase if the current phase has unresolved failures.

When you encounter an [RQ-*] tag, see references/research-questions.md for the open question and its context.

Gotchas

• agd provision can take 30+ minutes — do not treat long-running provisioning as a failure
• The agnosticd_user_info callback populates Showroom attributes — if lab guide data is missing, check the callback output first, not Showroom config
• agd status returns the last-known state, not a live health check — always run agd provision with --check for a real status
• Stop/start lifecycle only works on cloud instances (AWS/Azure/GCP), not on pre-existing shared clusters
• Teardown is irreversible — always confirm the GUID before running agd destroy

Required Input

Before starting, collect:

Phase 1 — Pre-flight

Verify the local environment is ready before running any agd commands.

If any pre-flight check fails, stop and provide the corrective action. Do not proceed to Phase 2.

Phase 2 — Provision

Before running agd provision directly, check whether a deploy.sh script already exists in the vars directory:

ls agnosticd-v2-vars/<config-name>/deploy.sh

If deploy.sh exists: run it instead of the raw command — it captures the correct GUID, config, account, and student count:

cd agnosticd-v2-vars/<config-name>
NUM_STUDENTS=<N> PARALLEL=false ./deploy.sh

If deploy.sh does not exist: run agd provision directly, then recommend generating the script using the agnosticd skill's "Independent Deployment Scripts" section:

cd ~/Development/agnosticd-v2
./bin/agd provision --config <config-name> --vars <vars-file>

Capture:

• Exit code (0 = success, non-zero = failure)
• The GUID assigned to this deployment
• Any ERROR or FAILED lines in the Ansible output

If provisioning fails:

Deployment Test — Phase 2: Provision FAILED
  Config:    <config-name>
  GUID:      <guid>
  Exit code: <N>
  Error:     <last ERROR line from output>

Next step: activate agnosticd-refactor to audit the config structure.
  Common causes:
  - Missing required variables in the vars file
  - Workload role not found (check workloads: list in vars)
  - Cloud credentials not present or expired in secrets.yml
  - Playbook file missing from ansible/configs/<config-name>/

Stop and report. Do not proceed until the user has resolved the failure and re-provisioned.

If provisioning succeeds: proceed to Phase 3.

Phase 3 — Post-deploy Validation

With the GUID confirmed, verify that the deployment is functionally complete.

3a. Workload completion check

Scan the provision output for each workload role listed in the vars file. Confirm each role completed with no FAILED tasks:

grep -E "(FAILED|ERROR|fatal)" agnosticd-v2-output/<guid>/*/ansible.log

Report any failed workloads:

Workload failures found:
  ocp4_workload_showroom    FAILED  — task: Deploy Showroom Helm release
  ocp4_workload_cert_manager PASS

3b. agnosticd_user_info verification

Check that agnosticd_user_info data was written to the output:

grep -r "agnosticd_user_info" agnosticd-v2-output/<guid>/

Expected: at least one entry showing the user data keys (e.g., openshift_console_url, openshift_cluster_admin_password, Showroom URL if deployed).

If no agnosticd_user_info output is found, note it as a finding but do not stop — record it in the Phase 3 report.

[RQ-4] — exact output format and key list for agnosticd_user_info is an open research question.

3c. Student-readiness hand-off

After workload and data checks, activate the student-readiness skill to verify the deployed environment from the student's perspective:

Phase 3 checks passed. Activating student-readiness for environment validation.
→ Use the student-readiness skill with GUID: <guid>

If student-readiness fails:

student-readiness reported failures. Likely causes:
  - A workload role partially deployed (check 3a findings)
  - agnosticd_user_info is missing required keys for Showroom attribute injection
  → Escalate to agnosticd-refactor: use the agnosticd-refactor skill to audit the config

Phase 3 Report

Deployment Test — Phase 3: Post-deploy Validation
──────────────────────────────────────────────────────
 Config:    <config-name>
 GUID:      <guid>
 Provider:  <cloud-provider>

 Check                        Status   Notes
 Provision exit code          PASS     —
 Workload: <role-1>           PASS     —
 Workload: <role-2>           FAIL     FAILED task: Deploy Showroom Helm release
 agnosticd_user_info output   PASS     Keys: console_url, admin_password, showroom_url
 student-readiness            PASS     —
──────────────────────────────────────────────────────
 Overall: PASS / FAIL

Phase 4 — Lifecycle Test

[RQ-5] — exact playbook names, locations, and return values for lifecycle operations is an open research question.

Test the stop/start/status lifecycle operations. This phase is optional — skip if the user confirmed lifecycle test: no in Required Input, or if Phase 3 failed.

# Stop
./bin/agd stop --config <config-name> --vars <vars-file>
# Verify: cluster/VMs are stopped, no running EC2 instances for this GUID

# Start
./bin/agd start --config <config-name> --vars <vars-file>
# Verify: cluster/VMs restart successfully, routes accessible again

# Status
./bin/agd status --config <config-name> --vars <vars-file>
# Verify: reports correct state

For each operation, capture exit code and any ERROR lines.

Current partial guidance:

The stop/start lifecycle is required for RHDP cost management — environments that cannot be stopped and resumed will not pass RHDP review. Stop should power down all cloud VMs without destroying the cluster state. Start should bring everything back to a running state. Status should report the current power state.

Phase 4 Report

Deployment Test — Phase 4: Lifecycle Test
──────────────────────────────────────────────────────
 Operation   Exit Code   Status   Notes
 agd stop    0           PASS     —
 agd start   0           PASS     —
 agd status  0           PASS     Reported: running
──────────────────────────────────────────────────────
 Lifecycle: PASS / FAIL

After lifecycle passes: the deployment is confirmed sound. The next step in the confidence chain is workshop-tester:

All deployment tests passed.
→ Activate workshop-tester to validate module exercises against this environment.
  This is the next step in building operator confidence before going live.

Final Summary Report

AgnosticD Deployment Test — Complete
══════════════════════════════════════════════════════
 Config:    <config-name>    GUID: <guid>
 Provider:  <cloud-provider>

 Phase 1 — Pre-flight          PASS
 Phase 2 — Provision           PASS
 Phase 3 — Post-deploy         PASS / FAIL
   Workloads                   <N>/<N> passed
   agnosticd_user_info         PASS / MISSING
   student-readiness           PASS / FAIL
 Phase 4 — Lifecycle           PASS / FAIL / SKIPPED
══════════════════════════════════════════════════════
 Overall: READY FOR STUDENTS / NEEDS ATTENTION

 Confidence: HIGH — all phases passed. An operator can run this workload
             with limited or no issues.
             Next step: run workshop-tester to validate module exercises.

Remediation Plan

When the overall result is NEEDS ATTENTION, generate an ordered, prioritized remediation plan from the actual failures found. Create each item as a session todo so progress is tracked.

Prioritize in this order:

Present the plan as an ordered list containing only the items that actually failed:

Deployment Remediation Plan — <config-name> / <guid>
──────────────────────────────────────────────────────
Priority  #  Finding                           Action
BLOCKING  1  ocp4_workload_showroom FAILED     Activate agnosticd-refactor → Audit Area 3
HIGH      2  agnosticd_user_info missing       Activate agnosticd-refactor → Audit Area 4
MEDIUM    3  agd stop failed (exit 1)          Check stop.yml in ansible/configs/<name>/
──────────────────────────────────────────────────────
Created 3 todos. Resolve in order — BLOCKING items first.

After presenting the plan, ask:

"Would you like to start with item 1 now? (y/n)"

If yes, activate the indicated skill immediately.

Re-test After Fixes

After the developer resolves one or more items from the remediation plan, re-run only the phases that contained failures rather than a full end-to-end test:

Re-running failed phases only:
  Phase 3 (Post-deploy validation) — re-checking workloads and agnosticd_user_info
  Phase 4 (Lifecycle) — re-testing agd stop/start/status

Skipping Phase 1 (pre-flight passed) and Phase 2 (provision succeeded).

Compare results against the previous run and show which items were resolved:

Re-test Results — <config-name> / <guid>
──────────────────────────────────────────────────────
 Item  Finding                        Previous   Now
 1     ocp4_workload_showroom         FAILED  →  PASS  ✓ resolved
 2     agnosticd_user_info            MISSING →  PASS  ✓ resolved
 3     agd stop                       FAILED  →  FAIL  still failing
──────────────────────────────────────────────────────
 Resolved: 2/3   Remaining: 1

Repeat until all items resolve, then present the full Final Summary Report as READY FOR STUDENTS.

Escalation

• Provision failure or workload failure → Use the agnosticd-refactor skill to audit the config structure, required variables, and workload role layout
• student-readiness failures after successful provision → Use the student-readiness skill's troubleshooting decision tree; if the root cause is in the config, escalate to agnosticd-refactor
• Lifecycle failures (stop/start/status) → Check that stop/start playbooks exist in ansible/configs/<config-name>/ and that the cloud provider credentials are current
• agnosticd_user_info data missing → The workload role is not calling agnosticd_user_info correctly; use agnosticd-refactor Audit Area 4 once RQ-4 is resolved