Azure Pipelines Validator

Azure Pipelines Validator

Comprehensive toolkit for validating, linting, testing, and securing Azure DevOps Pipeline configurations (azure-pipelines.yml, azure-pipelines.yaml files). Use this skill when working with Azure Pipelines, validating pipeline syntax, debugging configuration issues, implementing best practices, or performing security audits.

When to Use This Skill

Use the azure-pipelines-validator skill in the following scenarios:

• ✅ Working with azure-pipelines.yml or azure-pipelines.yaml files
• ✅ Validating Azure Pipelines YAML syntax and structure
• ✅ Debugging pipeline configuration errors
• ✅ Implementing Azure Pipelines best practices
• ✅ Performing security audits on pipeline configurations
• ✅ Checking for hardcoded secrets or credentials
• ✅ Optimizing pipeline performance (caching, parallelization)
• ✅ Ensuring compliance with security standards
• ✅ Code review of Azure DevOps CI/CD configurations
• ✅ Migrating or refactoring pipeline configurations

Features

0. YAML Linting (Optional)

• ✅ YAML formatting validation with yamllint
• ✅ Indentation checking (2-space standard)
• ✅ Line length validation
• ✅ Trailing spaces detection
• ✅ Custom Azure Pipelines YAML rules
• ✅ Automatic venv management (no manual install required)

1. Syntax Validation

• ✅ YAML syntax checking
• ✅ Azure Pipelines schema validation
• ✅ Required fields verification
• ✅ Stages/Jobs/Steps hierarchy validation
• ✅ Task format validation (TaskName@version)
• ✅ Pool/agent specification validation
• ✅ Deployment job strategy validation
• ✅ Trigger and PR configuration validation
• ✅ Resource definitions validation
• ✅ Variable and parameter declarations
• ✅ Dependency validation (dependsOn)

2. Best Practices Checking

• ✅ displayName usage for readability
• ✅ Task version pinning (specific @N not @0)
• ✅ Pool vmImage specific versions (not 'latest')
• ✅ Cache usage for package managers
• ✅ Timeout configuration for long-running jobs
• ✅ Deployment job conditions
• ✅ Artifact retention settings
• ✅ Parallel execution opportunities
• ✅ Template usage recommendations
• ✅ Variable group organization
• ✅ Deployment strategy best practices

3. Security Scanning

• ✅ Hardcoded secrets and credentials detection
• ✅ API keys and tokens in variables
• ✅ Task version security
• ✅ Container image security (:latest tags)
• ✅ Dangerous script patterns (curl | bash, eval)
• ✅ Service connection security
• ✅ Secret exposure in logs
• ✅ Checkout security settings
• ✅ Variable security (isSecret flag)
• ✅ Azure credential hardcoding
• ✅ SSL/TLS verification bypasses

Usage

Basic Validation

To validate an Azure Pipelines configuration file:

bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh <file-path>

Example:

bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml

This runs all four validation layers: 0. YAML lint (yamllint) - optional, auto-installed in venv if needed

1. Syntax validation
2. Best practices check
3. Security scan

Validation Options

# Run only syntax validation
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --syntax-only

# Run only best practices check
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices

# Run only security scan
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only

# Skip YAML linting (yamllint)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --skip-yaml-lint

# Skip best practices check
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-best-practices

# Skip security scan
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --no-security

# Strict mode (fail on warnings)
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict

Individual Validators

You can also run individual validation scripts:

# Syntax validation
python3 scripts/validate_syntax.py azure-pipelines.yml

# Best practices check
python3 scripts/check_best_practices.py azure-pipelines.yml

# Security scan
python3 scripts/check_security.py azure-pipelines.yml

Output Example

════════════════════════════════════════════════════════════════════════════════
  Azure Pipelines Validator
════════════════════════════════════════════════════════════════════════════════

File: azure-pipelines.yml

[1/3] Running syntax validation...

✓ Syntax validation passed

[2/3] Running best practices check...

SUGGESTIONS (2):
──────────────────────────────────────────────────────────────────────────────
  INFO: Line 15: Job 'BuildJob' should have displayName for better readability [missing-displayname]
  💡 Suggestion: Add 'displayName: "Your Job Description"' to job 'BuildJob'

  WARNING: Line 25: Task 'Npm@1' in job 'BuildJob' could benefit from caching [missing-cache]
  💡 Suggestion: Add Cache@2 task to cache dependencies and speed up builds

ℹ  Best practices check completed with suggestions

[3/3] Running security scan...

MEDIUM SEVERITY (1):
──────────────────────────────────────────────────────────────────────────────
  MEDIUM: Line 8: Container 'linux' uses ':latest' tag [container-latest-tag]
  🔒 Remediation: Pin container images to specific versions or SHA digests

✓ Security scan passed

════════════════════════════════════════════════════════════════════════════════
  Validation Summary
════════════════════════════════════════════════════════════════════════════════

Syntax Validation:      PASSED
Best Practices:         WARNINGS
Security Scan:          PASSED

════════════════════════════════════════════════════════════════════════════════

✓ All validation checks passed

Common Validation Scenarios

Scenario 1: Validating a New Pipeline

# Validate syntax and structure
bash scripts/validate_azure_pipelines.sh new-pipeline.yml

Scenario 2: Security Audit Before Merge

# Run security scan only with strict mode
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --security-only --strict

Scenario 3: Pipeline Optimization

# Check for best practices and optimization opportunities
bash scripts/validate_azure_pipelines.sh azure-pipelines.yml --best-practices

Scenario 4: CI/CD Integration

# In your Azure Pipeline
trigger:
  branches:
    include:
    - main

pool:
  vmImage: 'ubuntu-22.04'

steps:
- script: |
    pip3 install PyYAML
    bash .claude/skills/azure-pipelines-validator/scripts/validate_azure_pipelines.sh azure-pipelines.yml --strict
  displayName: 'Validate Pipeline Configuration'

Integration with Claude Code

When Claude Code invokes this skill, it will:

1. Auto-detect Azure Pipelines files - Run the validator without arguments to auto-detect azure-pipelines*.yml files in the current directory (up to 3 levels deep)
2. Run validation when you ask to validate, check, or review Azure Pipelines configurations
3. Provide actionable feedback with line numbers and suggestions
4. Stage-aware condition checking - Recognizes when parent stages have conditions, avoiding false positives on deployment jobs
5. Deduplicated findings - Reports each security issue once, even if detected by multiple patterns

Example prompts:

• "Validate my Azure Pipeline"
• "Check this azure-pipelines.yml for security issues"
• "Review my pipeline configuration for best practices"
• "Why is my Azure Pipeline failing?"
• "Optimize my Azure DevOps pipeline"

When to Use Context7/WebSearch for Documentation

The validation scripts provide static analysis. For dynamic documentation lookup, manually use these tools when you need:

• Task version information: "What's the latest version of AzureWebApp task?"
• Task input parameters: "What inputs does Docker@2 support?"
• Feature documentation: "How do I configure deployment environments in Azure Pipelines?"
• Troubleshooting: "Why does my AzureCLI@2 task fail with error X?"

How to fetch documentation:

# Use Context7 MCP for structured docs
mcp__context7__resolve-library-id("azure-pipelines")
mcp__context7__get-library-docs(context7CompatibleLibraryID, topic="deployment")

# Or use WebSearch/WebFetch for Microsoft Learn docs
WebSearch("Azure Pipelines Docker@2 task documentation 2025")
WebFetch("https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/docker-v2")

Note: Documentation lookup is a manual action - the validator scripts focus on static analysis and do not automatically fetch external documentation.

Validation Rules

Syntax Rules

• yaml-syntax: Valid YAML formatting
• yaml-invalid-root: Root must be a dictionary
• invalid-hierarchy: Cannot mix stages/jobs/steps at root level
• task-invalid-format: Tasks must follow TaskName@version format
• pool-invalid: Pool must specify name or vmImage
• stage-missing-jobs: Stages must define jobs
• job-missing-steps: Regular jobs must define steps
• deployment-missing-strategy: Deployment jobs must define strategy
• variable-invalid-name: Variables should use valid naming

Best Practice Rules

• missing-displayname: Stages/jobs should have displayName
• task-version-zero: Tasks should not use @0 version (except whitelisted tasks where @0 is the only version: GoTool, NodeTool, UsePythonVersion, KubernetesManifest, DockerCompose, HelmInstaller, HelmDeploy)
• task-missing-version: Tasks must specify version
• pool-latest-image: Avoid 'latest' in vmImage
• missing-cache: Package installations should use caching
• missing-timeout: Deployment jobs should specify timeout
• missing-deployment-condition: Production deployments should have conditions
• parallel-opportunity: Test jobs could use parallelization
• template-opportunity: Duplicate job patterns could use templates
• many-inline-variables: Consider using variable groups

Security Rules

• hardcoded-password: Hardcoded passwords detected
• hardcoded-api-key: Hardcoded API keys detected
• hardcoded-secret: Hardcoded secrets/tokens detected
• hardcoded-aws-credentials: AWS credentials hardcoded
• hardcoded-azure-ids: Azure subscription/tenant IDs hardcoded
• curl-pipe-shell: Dangerous curl | bash pattern
• eval-command: Eval command usage with variables
• chmod-777: Overly permissive file permissions
• insecure-ssl: SSL/TLS verification disabled
• secret-in-logs: Potential secret exposure in logs
• container-latest-tag: Container using :latest tag
• task-no-version: Task missing version (security risk)
• hardcoded-service-connection: Service connection IDs hardcoded
• checkout-no-clean: Checkout without clean
• variable-not-secret: Sensitive variable not marked as secret

Requirements

• Python 3.7+
• PyYAML and yamllint: Auto-installed in venv if not available systemwide
• Bash: For running the orchestrator script

No manual installation required! The validator uses automatic venv management:

• If PyYAML or yamllint are available system-wide, they'll be used
• Otherwise, a persistent .venv is created and packages are auto-installed
• The venv is reused across runs for optimal performance

To manually install dependencies system-wide (optional):

pip3 install PyYAML yamllint

Documentation

Comprehensive documentation is included in the docs/ directory:

• azure-pipelines-reference.md: Complete Azure Pipelines YAML syntax reference with examples

Examples

Example Azure Pipelines configurations are provided in the examples/ directory:

• basic-pipeline.yml: Simple CI pipeline with build and test stages
• docker-build.yml: Docker build and push workflow
• deployment-pipeline.yml: Multi-environment deployment with approval gates
• multi-platform.yml: Multi-platform build matrix
• template-example.yml: Pipeline using reusable templates

Test the skill with examples:

bash scripts/validate_azure_pipelines.sh examples/basic-pipeline.yml

Fetching Latest Documentation

When encountering specific Azure Pipelines tasks, resources, or version requirements, you can manually use the following tools to get up-to-date information:

1. Use Context7 MCP to fetch version-aware Azure Pipelines documentation
2. Use WebSearch to find latest Azure DevOps documentation
3. Use WebFetch to retrieve specific documentation pages from learn.microsoft.com

Note: These tools are not automatically invoked by the validation scripts. Use them manually when you need to look up specific Azure Pipelines tasks, features, or troubleshoot validation errors.

Extending the Skill

Adding Custom Validation Rules

Add custom rules to the validation scripts:

1. Syntax rules: Edit scripts/validate_syntax.py
2. Best practice rules: Edit scripts/check_best_practices.py
3. Security rules: Edit scripts/check_security.py

Custom Rule Example

# In check_best_practices.py
def _check_custom_rule(self):
    """Check for custom organization rule"""
    for job in self._get_all_jobs():
        job_name = job.get('job') or job.get('deployment')

        # Your custom validation logic
        if 'tags' not in pool:
            self.issues.append(BestPracticeIssue(
                'warning',
                self._get_line(job_name),
                f"Job '{job_name}' should specify agent tags",
                'custom-missing-tags',
                "Add 'tags' to pool to select appropriate agents"
            ))

Troubleshooting

Python Module Not Found

# Install PyYAML
pip3 install PyYAML

# Or with homebrew Python
python3 -m pip install PyYAML

Permission Denied

# Make scripts executable
chmod +x scripts/*.sh scripts/*.py

Validation Errors

Check the documentation:

• Review docs/azure-pipelines-reference.md for syntax reference
• Consult Azure Pipelines documentation at https://learn.microsoft.com/en-us/azure/devops/pipelines/

Version History

v1.0.0 (2025-01-24)

• Initial release
• Syntax validation with comprehensive Azure Pipelines schema checking
• Best practices validation with 10+ rules
• Security scanning with 20+ security checks
• Comprehensive documentation and examples
• Integration with Context7 for latest Azure DevOps docs

Contributing

To improve this skill:

1. Add new validation rules to appropriate scripts
2. Update documentation with new patterns
3. Add example configurations
4. Test with real-world Azure Pipelines files

License

This skill is part of the DevOps Skills collection.

Support

For issues, questions, or contributions:

• Check documentation in docs/ directory
• Review examples in examples/ directory
• Consult Azure Pipelines documentation: https://learn.microsoft.com/en-us/azure/devops/pipelines/

Remember: This skill validates Azure Pipelines configurations but does not execute pipelines. Use Azure DevOps Pipeline validation or Azure CLI for testing actual pipeline execution.