Test Rule

Test Rule

Run Automatus tests for a ComplianceAsCode security rule.

Rule ID: $ARGUMENTS

Tool Strategy

This skill uses mcp__content-agent__* tools when available (preferred — deterministic, structured results). When the MCP server is not configured, fall back to filesystem-based alternatives noted as Fallback in each step. See .claude/skills/shared/mcp_fallbacks.md for detailed fallback procedures. The skill must complete successfully either way.

Phase 1: Validate Rule Exists

1. Find the rule using mcp__content-agent__get_rule_details with rule_id=$ARGUMENTS:

   • This returns the full rule metadata including template info, CCE identifiers, remediation types, and file location.
   • If the rule is not found, also use mcp__content-agent__search_rules with query=$ARGUMENTS to check for similar rule IDs.
   • Fallback: Use Glob to find **/$ARGUMENTS/rule.yml, then read the file to extract metadata. For similar rule search, use Grep to search for $ARGUMENTS across rule.yml files.

2. If rule not found:

   • Use mcp__content-agent__list_templates to check if it's a template name instead. Fallback: Run ls shared/templates/ to check for template names.
   • Inform user and exit if not found

3. From the rule details, determine:

   • Is it templated? (has template: key)
   • What products have CCE identifiers? (determines applicable products)
   • What remediation types are available?

Phase 2: Determine Test Scope

1. Identify testable products from CCE identifiers:

   • Only rhel8, rhel9, rhel10 are supported by Automatus
   • Extract from rule.yml identifiers: cce@rhel8, cce@rhel9, cce@rhel10

2. Ask user for testing scope using AskUserQuestion:

   Question: "Which products do you want to test?"

   Options (show only products with CCE identifiers):

   • rhel10 (if CCE exists)
   • rhel9 (if CCE exists)
   • rhel8 (if CCE exists)
   • All applicable products

   Enable multi-select.

3. Ask for remediation types using AskUserQuestion:

   Question: "Which remediation types do you want to test?"

   Options:

   • Bash only - Test Bash remediation (faster)
   • Ansible only - Test Ansible remediation
   • Both Bash and Ansible (Recommended) - Test both remediation types

Important: Sandbox Requirements

All virsh and automatus.py commands require dangerouslyDisableSandbox: true on the Bash tool. These commands use libvirt unix sockets and SSH connections to VMs, which are blocked by the default sandbox. Set this flag on every Bash call that runs virsh or automatus.py.

Phase 3: Determine Libvirt Connection and VM Names

3.1 Detect Libvirt Connection URI

The libvirt connection URI depends on the user's VM setup. Ask the user using AskUserQuestion:

Question: "Which libvirt connection URI does your VM setup use?"

Options:

• qemu:///session (Recommended) - User session VMs, no root required
• qemu:///system - System-wide VMs, may require root/sudo

Store the selected URI as <libvirt_uri> for all subsequent commands.

3.2 Discover VM Names

VM names often differ from product names (e.g., rhel9-test, ssg-rhel9, rhel-9.4). Discover available VMs:

virsh -c <libvirt_uri> list --all 2>/dev/null

For each selected product, ask the user to confirm the VM name:

Question: "Which VM should be used for testing <product>? Available VMs are listed above."

Options (populated from virsh list --all output, filtered to relevant entries):

• Matching VMs from the list
• Allow user to type a custom name

Store the mapping of product → VM name for Phase 5.

Note on sudo: When using qemu:///system, some operations (starting VMs, creating snapshots) may require sudo. When using qemu:///session, sudo is not needed. Adjust commands accordingly.

Phase 4: Verify Prerequisites

1. Check for existing datastreams: Use mcp__content-agent__list_built_products to see which products have been built. Fallback: Run ls build/ssg-*-ds.xml 2>/dev/null to list built datastreams.

2. For each selected product, check if datastream exists and get details: Use mcp__content-agent__get_datastream_info with product=<product> to verify the datastream exists and get its details. Fallback: Run ls -la build/ssg-<product>-ds.xml to check if the datastream exists.

3. Build datastreams if needed:

   • If datastream doesn't exist or is older than rule.yml modifications
   • Ask user: "Datastream for is missing/outdated. Build it now?"

   ./build_product --rule-id $ARGUMENTS <product>
   

4. Verify VMs are available and running (CRITICAL — Automatus requires the VM to be running):

   virsh -c <libvirt_uri> list --all 2>/dev/null | grep -E "rhel[0-9]+"
   

   • Inform user of available VMs
   • If required VM not found, provide guidance on VM setup
   • Check that the required VM is in "running" state. If the VM exists but is shut off, start it before proceeding:

     # For qemu:///system, prefix with sudo if needed
     virsh -c <libvirt_uri> start <vm_name>
     

   • Wait a moment after starting the VM and verify it is running:

     virsh -c <libvirt_uri> list --all 2>/dev/null | grep <vm_name>
     

   • Do NOT proceed to Phase 5 until the VM is confirmed running. Automatus will fail if the VM is not started.

5. Verify VM has a snapshot (CRITICAL — Automatus reverts to a snapshot between test scenarios):

   virsh -c <libvirt_uri> snapshot-list <vm_name>
   

   • Automatus needs at least one snapshot to restore clean state between test scenarios.
   • If no snapshots exist, inform the user and suggest creating one:

     # For qemu:///system, prefix with sudo if needed
     virsh -c <libvirt_uri> snapshot-create-as <vm_name> clean
     

   • Do NOT proceed to Phase 5 if no snapshot exists. Automatus will fail or leave the VM in a dirty state.

Phase 5: Run Automatus Tests

For Rule Testing

Run tests for each selected product and remediation type, using the <libvirt_uri> and <vm_name> determined in Phase 3:

Bash remediation:

cd tests
./automatus.py rule --libvirt <libvirt_uri> <vm_name> \
    --datastream ../build/ssg-<product>-ds.xml \
    $ARGUMENTS

Ansible remediation:

cd tests
./automatus.py rule --libvirt <libvirt_uri> <vm_name> \
    --datastream ../build/ssg-<product>-ds.xml \
    --remediate-using ansible \
    $ARGUMENTS

For Template Testing

If testing a template instead of a rule:

cd tests
./automatus.py template --libvirt <libvirt_uri> <vm_name> \
    --datastream ../build/ssg-<product>-ds.xml \
    $ARGUMENTS

Phase 6: Monitor Test Execution

1. Tests run in foreground - output is streamed

2. Test phases to expect:

   • Initial scan (check if rule is evaluated)
   • Remediation application
   • Final scan (verify remediation worked)

3. For each test scenario (e.g., correct.pass.sh, wrong.fail.sh):

   • .pass.sh scenarios: Should pass on initial scan
   • .fail.sh scenarios: Should fail initially, then pass after remediation

Phase 7: Analyze Results

1. Find results directory:

   ls -td tests/logs/rule-custom-* 2>/dev/null | head -1
   

2. Read results.json:

   cat tests/logs/rule-custom-*/results.json
   

3. Parse and summarize results:

   Scenario	Initial Scan	Remediation	Final Scan	Status
   wrong.fail	FAIL	APPLIED	PASS	OK
   correct.pass	PASS	SKIPPED	N/A	OK

4. Check for common issues:

   • Initial scan passed on .fail scenario: Test scenario didn't properly set up non-compliant state
   • Remediation failed: Bash/Ansible script has errors
   • Final scan failed after remediation: Remediation incomplete or incorrect
   • Not applicable: Platform check excluded the rule

Phase 8: Report Results

Success Report

Test Results for $ARGUMENTS
==============================

Product: rhel9
Remediation: Bash

Scenarios:
  - wrong.fail.sh: PASSED (fail -> remediate -> pass)
  - correct.pass.sh: PASSED (already compliant)

Overall: ALL TESTS PASSED

---

Product: rhel9
Remediation: Ansible

Scenarios:
  - wrong.fail.sh: PASSED (fail -> remediate -> pass)
  - correct.pass.sh: PASSED (already compliant)

Overall: ALL TESTS PASSED

==============================
Summary: 4/4 tests passed across all products and remediation types

Failure Report

Test Results for $ARGUMENTS
==============================

Product: rhel9
Remediation: Bash

Scenarios:
  - wrong.fail.sh: FAILED
    - Initial scan: FAIL (expected)
    - Remediation: APPLIED
    - Final scan: FAIL (unexpected!)
    - Issue: Remediation did not properly configure the setting

  - correct.pass.sh: PASSED

Log files: tests/logs/rule-custom-2024-01-15-1423/

Suggested debugging:
1. Check remediation script: find linux_os/guide -path "*/$ARGUMENTS/bash/*"
2. Review test scenario: find linux_os/guide -path "*/$ARGUMENTS/tests/*"
3. SSH into VM to inspect: virsh console rhel9

==============================
Summary: 1/2 tests failed

Phase 9: Provide Next Steps

Based on results:

If all tests passed:

• "Rule is ready. Use /build-product <product> to build and /run-tests to validate."
• "Consider testing on additional products if applicable."

If tests failed:

• Provide specific debugging guidance based on failure type
• Suggest files to check/modify
• Offer to help fix the issues

If no tests exist (non-templated rule without tests):

• "This rule has no test scenarios. Create tests in <rule_dir>/tests/"
• "Required: At least one .pass.sh and one .fail.sh scenario"

Troubleshooting

Common Issues

1. VM not found:

   Error: VM '<vm_name>' not found
   

   • Ensure VM exists: virsh -c <libvirt_uri> list --all
   • Check connection URI — re-run Phase 3 if needed

2. Datastream not found:

   Error: build/ssg-<product>-ds.xml not found
   

   • Build with: ./build_product --rule-id $ARGUMENTS <product>

3. Permission denied:

   • If using qemu:///system, may need sudo
   • Switch to qemu:///session for user VMs

4. Rule not in datastream:

   • Verify rule is in at least one profile
   • Rebuild datastream after adding to profile

5. Test scenario errors:

   • Check bash syntax in test files
   • Verify required packages are specified: # packages = <pkg>

6. No snapshot available:

   Error: No snapshot found for domain
   

   • Automatus needs a snapshot to revert between test scenarios
   • Create one: virsh -c <libvirt_uri> snapshot-create-as <vm_name> clean (prefix with sudo for qemu:///system)

Debug Mode (Manual Only)

Note: Debug mode requires interactive input and must be run manually by the user, not through this skill.

When tests fail, suggest the user run with --debug in their own terminal:

cd tests
./automatus.py rule --libvirt <libvirt_uri> <vm_name> \
    --datastream ../build/ssg-<product>-ds.xml \
    --debug \
    $ARGUMENTS

How --debug works:

1. When an error occurs (e.g., remediation applied but rule still fails), Automatus pauses and keeps the VM running
2. User can SSH into the VM to inspect system state:

   ssh root@<vm-ip>   # No password required on test VMs
   

3. Press Enter in the Automatus terminal to continue to the next test, or Ctrl+C to abort