Terraform Provider Tests

Terraform Provider Tests

Analyze test coverage and improve Terraform provider acceptance tests using modern patterns from terraform-plugin-testing v1.13.3+.

Summary & Next Steps

When invoked, I will:

1. Run gap analysis: python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md
2. Generate timestamped report in ./ai_reports/
3. Provide succinct summary with:
   • Overall grade (A/B/C)
   • Top 3 findings
   • Recommended next action

Priority levels:

• P1 (Critical): Missing drift/import tests, heavy legacy usage (>20 calls)
• P2 (Important): Missing idempotency, moderate legacy (5-20 calls)
• P3 (Cleanup): Light legacy (<5 calls)

Communication style: Be succinct. Provide summary + single recommended next step.

Quick Start

Common Usage Scenarios

Analyze test modernization gaps:

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md

Analyze a specific test file: "Analyze resource_example_test.go for coverage gaps"

Verify compilation after changes:

./.claude/skills/terraform-provider-tests/verify_compilation.sh ./internal/provider/

Validate all examples:

export BCM_ENDPOINT="https://..." BCM_USERNAME="..." BCM_PASSWORD="..."
./.claude/skills/terraform-provider-tests/test-examples.sh

Recommended Naming Convention

All gap analysis reports should use the tf_provider_tests_* naming pattern in the ./ai_reports/ directory:

Timestamp format: $(date +%Y%m%d_%H%M%S) generates YYYYMMDD_HHMMSS

Workflow Decision Tree

Starting a modernization project? → Begin with Gap Analysis

Have gap analysis report? → Proceed to Pattern Application

Made code changes? → Run Verification

All changes complete? → Complete Testing

Need to validate examples? → Run Example Testing

Phase 1: Gap Analysis (Automated)

Tool: scripts/analyze_gap.py

Automatically scan test files and generate comprehensive gap analysis report.

Usage

python3 scripts/analyze_gap.py <test_directory> [--output report.md]

Example (recommended naming pattern):

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap_$(date +%Y%m%d_%H%M%S).md

Simple filename (for one-time analysis):

python3 scripts/analyze_gap.py ./internal/provider/ --output ./ai_reports/tf_provider_tests_gap.md

What It Detects

• ✅ Legacy Check: resource.TestCheckResourceAttr() patterns
• ✅ Missing drift detection tests
• ✅ Missing import tests
• ✅ Missing idempotency checks
• ✅ Modern pattern adoption statistics
• ✅ ID consistency tracking issues
• ✅ Prioritized recommendations

Output

Markdown report with:

• Codebase statistics (line counts by file type, test-to-impl ratio)
• Executive summary with overall grade (A/B/C)
• File-by-file analysis with status
• Prioritized recommendations (High/Medium/Low)
• Modern pattern quick reference

Review the report to understand:

• Which files need the most work
• What patterns are missing
• Overall modernization progress

Phase 2: Prioritization

Focus on high-impact changes first:

Priority 1 (Critical) ⚠️

• Missing drift detection tests
• Missing import tests
• Heavy legacy usage (>20 calls per file)

Priority 2 (Important) 📋

• Missing idempotency checks
• Moderate legacy usage (5-20 calls)
• Mixed patterns (legacy + modern)

Priority 3 (Cleanup) 📝

• Light legacy usage (<5 calls)
• Documentation improvements

Phase 3: Pattern Application (Guided)

For each file, apply patterns in this order:

Step 1: Add Missing Tests

Missing drift detection? → See references/pattern_templates.md → "Drift Detection Test"

Missing import test? → See references/pattern_templates.md → "Import Test Step"

Missing idempotency checks? → See references/pattern_templates.md → "Idempotency Verification"

Step 2: Convert Legacy to Modern

Replace Check: resource.ComposeAggregateTestCheckFunc() with ConfigStateChecks.

Before:

Check: resource.ComposeAggregateTestCheckFunc(
    resource.TestCheckResourceAttr("example_resource.test", "name", "expected"),
),

After:

ConfigStateChecks: []statecheck.StateCheck{
    statecheck.ExpectKnownValue(
        "example_resource.test",
        tfjsonpath.New("name"),
        knownvalue.StringExact("expected"),
    ),
},

Type mapping → See references/pattern_templates.md

Step 3: Add Required Imports

import (
    "github.com/hashicorp/terraform-plugin-testing/helper/resource"
    "github.com/hashicorp/terraform-plugin-testing/plancheck"
    "github.com/hashicorp/terraform-plugin-testing/statecheck"
    "github.com/hashicorp/terraform-plugin-testing/knownvalue"
    "github.com/hashicorp/terraform-plugin-testing/tfjsonpath"
    "github.com/hashicorp/terraform-plugin-testing/compare"
)

For drift tests, also add: "context", "encoding/json", "time"

Phase 4: Verification (Automated)

Tool: scripts/verify_compilation.sh

Quick compilation check without running tests.

Usage

./.claude/skills/terraform-provider-tests/scripts/verify_compilation.sh <test_directory>

Example:

./.claude/skills/terraform-provider-tests/scripts/verify_compilation.sh ./internal/provider/

What It Validates

• ✅ Go syntax correctness
• ✅ No compilation errors
• ✅ Import completeness
• ✅ Statistics (file count, test count)

If compilation fails:

1. Review error messages
2. Check for missing imports
3. Verify block closures
4. Re-run after fixes

Phase 5: Testing

Quick Compile Check

go test -c ./internal/provider/ -o /tmp/provider_tests

Parallel Test Execution (Recommended)

Tool: scripts/run_tests_parallel.sh

Run acceptance tests concurrently per file for faster execution.

Usage:

# Run all acceptance tests with 15 concurrent files
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh

# Run only resource tests with higher concurrency
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --resources-only -c 8

# Run only data source tests
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --data-sources-only

# Run tests matching specific pattern
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh -p "TestAccCMPartSoftwareImage"

# Run tests from specific file
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh -f resource_cmpart_softwareimage_test.go

# Verbose output with detailed test logs
./.claude/skills/terraform-provider-tests/scripts/run_tests_parallel.sh --verbose

Options:

• -d, --dir DIR - Test directory (default: ./internal/provider)
• -p, --pattern PATTERN - Test pattern to match (default: TestAcc)
• -c, --concurrency N - Max concurrent test files (default: 4)
• -t, --timeout DURATION - Timeout per test file (default: 30m)
• -f, --file FILE - Run only tests from specific file
• --resources-only - Run only resource tests
• --data-sources-only - Run only data source tests
• --verbose - Show detailed test output
• --no-color - Disable colored output

Benefits:

• ⚡ Faster execution (4x-8x speedup with proper concurrency)
• 📊 Per-file progress tracking
• 🎯 Aggregated summary with pass/fail counts
• 🔍 Automatic failure highlighting

Single Test (Sequential)

TF_ACC=1 go test -v -timeout 30m ./internal/provider/ -run "^TestAccResource_Specific$"

Full Suite (Sequential)

TF_ACC=1 go test -v -timeout 120m ./internal/provider/

Phase 6: Example Testing (Automated)

Tool: scripts/test-examples.sh

Validate all Terraform examples in the examples/ directory by building the provider, executing examples, and cleaning up test resources.

Usage

# Run all examples (requires BCM credentials)
export BCM_ENDPOINT="https://172.21.15.254:8081"
export BCM_USERNAME="root"
export BCM_PASSWORD="your-password"
./scripts/test-examples.sh

# Quick validation with existing provider build
SKIP_BUILD=true ./scripts/test-examples.sh

# Test only data sources (parallel, ~10s)
./scripts/test-examples.sh --data-sources-only

# Test only resources (sequential, ~19s)
./scripts/test-examples.sh --resources-only

# Debug a failing test
./scripts/test-examples.sh --verbose --no-cleanup

# Cleanup orphaned resources from failed tests
./scripts/test-examples.sh --cleanup-only

What It Validates

• ✅ Examples compile and initialize successfully
• ✅ Provider configuration is correct
• ✅ Schema validation passes
• ✅ Plan generation works (all examples)
• ✅ Apply/destroy cycle succeeds (test-citest examples only)
• ✅ Resources are properly cleaned up

Execution Strategy

Data Sources (parallel):

• Run up to 4 examples concurrently (configurable with PARALLEL_LIMIT)
• Fast validation (init → validate → plan)
• No actual resources created

Resources (sequential):

• Run one at a time (state-modifying operations)
• Full lifecycle testing for test-citest examples (init → validate → plan → apply → destroy)
• Plan-only validation for documentation examples

Test Phases

1. Environment validation - Verify BCM credentials
2. Provider build - Compile provider binary (skippable with SKIP_BUILD=true)
3. Example discovery - Find all .tf files in examples/data-sources/*/ and examples/resources/*/
4. Example testing - Execute terraform commands on each example
5. Cleanup - Remove test resources (citest-* prefix) with retry logic

Environment Variables

Required:

• BCM_ENDPOINT - BCM API endpoint
• BCM_USERNAME - BCM authentication username
• BCM_PASSWORD - BCM authentication password

Optional:

• PROVIDER_VERSION - Provider version (default: 0.1.0)
• SKIP_BUILD - Skip build phase (default: false)
• PARALLEL_LIMIT - Max parallel data source tests (default: 4)
• CLEANUP_RETRIES - Max cleanup retry attempts (default: 4)
• VERBOSE - Enable verbose logging (default: false)

Exit Codes

• 0 - All tests passed, cleanup successful
• 1 - One or more tests failed
• 2 - Configuration error (missing env vars)
• 3 - Provider build failed
• 130 - Interrupted by user (Ctrl+C)

Best Practices

Example Naming:

• Use citest- prefix for resources that need cleanup
• Examples in test-citest/ directories undergo full apply/destroy
• Other examples are validated with plan-only

Provider Configuration:

• Examples should NOT include provider blocks
• Script automatically injects provider config with environment variables
• Use insecure_skip_verify = true for self-signed certs

Resource Cleanup:

• Script automatically cleans up resources with citest- prefix
• Uses exponential backoff retry for cleanup failures
• Run --cleanup-only to manually cleanup orphaned resources

When to Use Example Testing

Run example testing when:

• Adding new examples to examples/ directory
• Modifying provider schema or behavior
• Before releasing a new provider version
• Debugging example-specific issues
• Validating documentation accuracy

Integration with CI/CD

Example testing complements acceptance tests:

Acceptance Tests (make testacc):

• Full CRUD operation validation
• Import and drift detection
• Comprehensive error handling
• Run on every commit

Example Tests (./scripts/test-examples.sh):

• Documentation accuracy validation
• End-user workflow verification
• Multi-example compatibility
• Run before releases

Completion Criteria

A fully modernized test file has:

• ✅ Zero legacy Check blocks
• ✅ All state assertions use statecheck.ExpectKnownValue()
• ✅ Idempotency checks after Create and Update
• ✅ Import test with ImportStateVerify
• ✅ Drift detection test (resources only)
• ✅ ID consistency tracking with CompareValue
• ✅ All tests compile and pass
• ✅ All examples validate successfully

ID Consistency Tracking

The analyzer detects inconsistent usage of the .id property across test steps.

Why It Matters

Resource IDs should remain stable across:

• Initial creation
• Import operations
• Update operations

Inconsistent ID handling can indicate:

• Resource recreation instead of in-place updates
• Import state mismatches
• State management bugs

What the Analyzer Detects

Correct Pattern

func TestAccResource_Complete(t *testing.T) {
    // Initialize ID tracker BEFORE Steps
    compareID := statecheck.CompareValue(compare.ValuesSame())

    resource.Test(t, resource.TestCase{
        Steps: []resource.TestStep{
            // Step 1: Create - track ID
            {
                Config: testAccResourceConfig(name),
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
            // Step 2: Import - track ID
            {
                ResourceName:      "example_resource.test",
                ImportState:       true,
                ImportStateVerify: true,
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
            // Step 3: Update - track ID
            {
                Config: testAccResourceConfig(name, "updated"),
                ConfigStateChecks: []statecheck.StateCheck{
                    compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")),
                },
            },
        },
    })
}

Common Anti-Patterns

Anti-pattern 1: Legacy ID checks

// ❌ Bad - uses legacy pattern
Check: resource.ComposeAggregateTestCheckFunc(
    resource.TestCheckResourceAttrSet("example_resource.test", "id"),
),

Anti-pattern 2: Inconsistent tracking

// ❌ Bad - only tracks ID in Create step, not Import/Update
Steps: []resource.TestStep{
    {
        Config: testAccConfig(name),
        ConfigStateChecks: []statecheck.StateCheck{
            compareID.AddStateValue("example_resource.test", tfjsonpath.New("id")), // ✅
        },
    },
    {
        ResourceName: "example_resource.test",
        ImportState:  true,
        // ❌ Missing: compareID.AddStateValue
    },
    {
        Config: testAccConfig(name, "updated"),
        // ❌ Missing: compareID.AddStateValue
    },
}

Anti-pattern 3: No tracking at all

// ❌ Bad - multiple steps with no ID consistency tracking
compareID := statecheck.CompareValue(compare.ValuesSame()) // Declared but never used!

Common Pitfalls

Missing imports → Add all required imports from Phase 3, Step 3

Wrong knownvalue matcher → Match types correctly: String→StringExact, Bool→Bool, Int64→Int64Exact

Duplicate validation → Remove Check block, keep only ConfigStateChecks

References

This skill includes comprehensive reference documentation:

references/workflow.md

Complete step-by-step modernization workflow with detailed guidance for each phase, common pitfalls, and completion criteria.

When to read: For detailed phase-by-phase instructions.

references/pattern_templates.md

Ready-to-use code templates for all modern patterns:

• Legacy to modern conversion examples
• Idempotency verification
• Import test steps
• Drift detection tests (complete template)
• ID consistency tracking
• knownvalue type matchers
• Complete test examples

When to read: When applying specific patterns to code.

references/hashicorp_official.md

Consolidated HashiCorp official documentation:

• TestCase and TestStep structure
• State checks (statecheck package)
• Plan checks (plancheck package)
• Value comparers (compare package)
• Import mode testing
• Official testing patterns

When to read: For authoritative information on terraform-plugin-testing features.

Time Estimates

• Simple file (1-2 tests, <10 legacy checks): 15-30 minutes
• Medium file (3-5 tests, 10-30 legacy checks): 30-60 minutes
• Complex file (>5 tests, >30 legacy checks): 1-2 hours

Total project (7 resource + 7 data source files): 4-8 hours across multiple sessions.

Hybrid Approach

This skill uses a hybrid approach:

Automation for:

• ✅ Gap analysis (find legacy patterns, missing tests)
• ✅ Compilation verification (syntax checking)

Guided assistance for:

• ✅ Code changes (apply patterns with context)
• ✅ Complex operations (drift tests, API integration)
• ✅ Decision making (prioritization, test design)

This balance maintains control while automating tedious tasks.