Quality Smoke Testing

Smoke Testing

Purpose

Validate that the most critical functionality of an application works after a deployment, build, or environment change. Smoke tests provide fast, reliable feedback that the system is stable enough for further testing or production release. This skill covers smoke suite design, CI/CD gate integration, rollback automation, and environment-specific strategies.

Agent Protocol

Trigger

User mentions smoke test, BVT, build verification, deployment health check, canary test, sanity check, or asks "is the build stable?"

Input Context

• Deployment target (staging, production, canary)
• Changes included in the build
• Critical user journeys for the application
• Health endpoint specifications
• CI/CD pipeline stage and rollback capabilities

Output Artifact

• Smoke test suite definition with pass/fail thresholds
• Deployment health report
• Rollback recommendation if critical failures found
• CI/CD pipeline configuration with smoke stage

Response Format

Structured report with:

1. Build/version under test and environment details
2. Test results per smoke scenario (pass/fail/skip)
3. Overall verdict: PASS / FAIL / PARTIAL
4. Rollback recommendation and trigger if applicable
5. Execution time and threshold compliance

Completion Criteria

• All smoke tests executed within time budget
• Results recorded with evidence (logs, screenshots)
• Verdict communicated to deployment pipeline
• Rollback triggered automatically if pass rate < threshold

Workflow

1. Define smoke suite: Identify critical-path tests (infrastructure health, core services, critical user journeys). Keep under 25 tests
2. Classify criticality: Mark tests as critical (100% pass required) or non-critical (95% threshold). Critical failures trigger rollback
3. Configure thresholds: Set critical threshold to 100%, non-critical to 95%. Infrastructure failures stop the pipeline
4. Execute on deploy: Run automatically as first CI/CD stage after deployment. Complete in under 5 minutes
5. Evaluate results: Critical failure → automatic rollback. Non-critical failure below 95% → block promotion. All pass → continue
6. Monitor post-deployment: Continue health checks for 15-30 minutes after deployment. Auto-rollback on sustained degradation
7. Maintain suite: Review smoke tests monthly. Add tests for new critical paths. Remove obsolete tests. Verify determinism

Architecture / Decision Trees

Test Inclusion Decision Tree

Is this a critical business path?
├── YES → Can the system function without it?
│   ├── NO → Include as critical smoke test
│   └── YES → Include as non-critical smoke test
└── NO → Move to regression suite

Can it run in under 30 seconds?
├── YES → Keep in smoke suite
└── NO → Can it be split/optimized?
    ├── YES → Optimize and keep
    └── NO → Move to regression suite

Is it 100% deterministic?
├── YES → Keep in smoke suite
└── NO → Fix flakiness or remove

Environment Configuration Decision Tree

Deployment target?
├── Staging → Full smoke suite (write operations OK)
│   ├── Threshold: 100% critical, 95% non-critical
│   └── Rollback: Automatic
├── Canary → Core smoke (read-only + limited write)
│   ├── Threshold: 100% all tests
│   └── Rollback: Canary drain
└── Production → Read-only smoke suite
    ├── Threshold: 100% all tests
    └── Rollback: Automatic full rollback

Common Pitfalls

1. Suite bloat: More than 25 tests becomes slow. Prune aggressively — smoke tests are gates, not comprehensive validators
2. Non-deterministic tests: Zero tolerance for flakiness. Fix or remove immediately
3. Complex test data: Smoke tests requiring complex setup are fragile. Use defaults or existing seed data
4. Environment-specific failures: Tests passing in staging but failing in production have configuration issues
5. No rollback automation: Smoke failures must trigger automatic rollback, not manual review
6. Write operations in production: Production smoke tests must be read-only — never modify production data
7. Stale credentials: Expired smoke test credentials cause false failures. Rotate regularly
8. Overlapping coverage: Multiple tests verifying the same path waste time. Deduplicate
9. Missing production verification: Deploying to production without running smoke tests is risky
10. No monitoring integration: Smoke test results must feed into metrics and alerting systems

Best Practices

1. Keep smoke suite small: 10-25 tests, under 5 minutes, focused on critical paths
2. Every smoke test must be 100% deterministic — non-deterministic tests erode trust
3. Classify tests as critical (100% pass required) or non-critical (95% threshold)
4. Run full smoke suite in staging, read-only smoke in production
5. Connect smoke results to automatic rollback and alerting systems
6. Use dedicated smoke test service accounts with limited permissions
7. Review and update smoke suite monthly to reflect current critical paths
8. Run smoke tests in parallel for speed, but each test must be independent
9. Use infrastructure health checks + core service checks + critical user journeys for complete coverage
10. Continue monitoring after deployment — initial smoke pass doesn't guarantee long-term health

Compared With

Performance Considerations

• Target: 5 minutes maximum for the entire smoke suite
• Each test: under 30 seconds. If slower, optimize or move to regression
• Use parallel execution for concurrent tests
• Reuse HTTP connections and auth tokens across tests in the same suite
• Pre-warm containers and connections before smoke test execution
• Allocate 15-20% buffer time for network latency and intermittent slowdowns
• Monitor execution time trend — growing suite execution time indicates bloat

Smoke Test Examples

Playwright — Critical Path Smoke Tests

// e2e/smoke/critical-paths.spec.ts
import { test, expect } from "@playwright/test";

test.describe("Smoke: Infrastructure", () => {
  test("homepage returns 200", async ({ page }) => {
    const response = await page.goto("/");
    expect(response?.status()).toBe(200);
  });

  test("health endpoint is healthy", async ({ request }) => {
    const response = await request.get("/api/health");
    expect(response.ok()).toBeTruthy();
    const body = await response.json();
    expect(body.status).toBe("healthy");
  });
});

test.describe("Smoke: Authentication", () => {
  test("login page loads with form elements", async ({ page }) => {
    await page.goto("/login");
    await expect(page.getByTestId("email-input")).toBeVisible();
    await expect(page.getByTestId("password-input")).toBeVisible();
    await expect(page.getByTestId("submit-button")).toBeVisible();
  });

  test("login with valid credentials succeeds", async ({ page }) => {
    await page.goto("/login");
    await page.getByTestId("email-input").fill("smoke@example.com");
    await page.getByTestId("password-input").fill(process.env.SMOKE_PASSWORD!);
    await page.getByTestId("submit-button").click();
    await expect(page).toHaveURL(/dashboard/);
  });
});

test.describe("Smoke: Core Business Flow", () => {
  test("product listing page loads", async ({ page }) => {
    await page.goto("/products");
    await expect(page.getByTestId("product-grid")).toBeVisible();
    await expect(page.getByTestId("product-card")).toHaveCount({ gte: 1 });
  });

  test("search returns results", async ({ page }) => {
    await page.goto("/products");
    await page.getByTestId("search-input").fill("wireless");
    await page.getByTestId("search-submit").click();
    await expect(page.getByTestId("search-results")).toBeVisible();
  });
});

Python — API Health Smoke Tests

# smoke/test_api_health.py
import requests
import sys

SMOKE_TESTS = [
    {"name": "homepage", "url": "/", "expected_status": 200},
    {"name": "health endpoint", "url": "/api/health", "expected_status": 200},
    {"name": "login page", "url": "/login", "expected_status": 200},
    {"name": "products API", "url": "/api/products", "expected_status": 200},
]

def test_all():
    base_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:3000"
    failures = []
    for test in SMOKE_TESTS:
        try:
            resp = requests.get(f"{base_url}{test['url']}", timeout=10)
            if resp.status_code != test["expected_status"]:
                failures.append(f"FAIL: {test['name']} - expected {test['expected_status']}, got {resp.status_code}")
            else:
                print(f"PASS: {test['name']}")
        except Exception as e:
            failures.append(f"FAIL: {test['name']} - {e}")
    
    if failures:
        print("\n".join(failures))
        sys.exit(1)
    print(f"\nAll {len(SMOKE_TESTS)} smoke tests passed")

if __name__ == "__main__":
    test_all()

CI/CD Pipeline Integration

GitHub Actions — Smoke Test Stage with Rollback

name: Deploy with Smoke Tests
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      status: ${{ steps.smoke.outputs.status }}
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        run: ./deploy.sh staging
      - name: Smoke tests
        id: smoke
        run: |
          npx playwright test --grep @smoke --reporter=json > smoke-results.json
          if grep -q '"status":"failed"' smoke-results.json; then
            echo "status=failed" >> $GITHUB_OUTPUT
          else
            echo "status=passed" >> $GITHUB_OUTPUT
          fi
      - name: Rollback on smoke failure
        if: steps.smoke.outputs.status == 'failed'
        run: ./rollback.sh staging
  promote:
    needs: deploy
    if: needs.deploy.outputs.status == 'passed'
    runs-on: ubuntu-latest
    steps:
      - name: Promote to production
        run: ./promote.sh staging production

ArgoCD Health Check Integration

# application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/example/app
  syncPolicy:
    automated:
      selfHeal: true
      allowEmpty: false
    retry:
      limit: 3
  health:
    - test: "smoke-test"
      commands:
        - "curl -f http://app:3000/api/health"
        - "curl -f http://app:3000/api/products | jq '. | length > 0'"

Smoke Test Anti-Patterns

Anti-Pattern: Suite Bloat

Smoke suites that exceed 25 tests become slow and lose their purpose as a fast health check. Prune aggressively. Smoke tests are gates, not comprehensive validators. Move detailed tests to regression.

Anti-Pattern: Non-Deterministic Tests

Smoke tests must be 100% deterministic. Zero tolerance for flakiness. If a smoke test fails intermittently, fix it immediately or remove it from the smoke suite. Flaky smoke tests erode trust in the entire deployment pipeline.

Anti-Pattern: Complex Test Data Setup

Smoke tests requiring complex data setup (API seeding, database migrations, multiple test accounts) are fragile and slow. Smoke tests should work with minimal or default data. Use existing seed data.

Anti-Pattern: Write Operations in Production

Production smoke tests that create test orders, register test users, or modify data cause data pollution and may trigger real downstream effects. Production smoke tests must be read-only. Use GET requests only.

Anti-Pattern: No Automatic Rollback

Smoke test failures that only notify humans instead of triggering automatic rollback introduce delay in incident response. Critical smoke test failures must trigger automatic rollback. Test the rollback automation quarterly.

Anti-Pattern: Ignoring Infrastructure Health

Starting functional smoke tests before verifying infrastructure is healthy wastes time debugging false failures. Infrastructure checks (health endpoints, database connectivity, cache connectivity) must run first. If infrastructure is down, skip functional tests and fail fast.

Smoke Testing Maturity Model

Canary Testing Strategy

canary_test:
  deployment:
    strategy: "10% canary → 50% → 100% rollout"
    interval: "10 minutes between stages"
  health_metrics:
    - "Error rate: < 1% (compare to baseline)"
    - "P95 latency: < 120% of baseline"
    - "CPU utilization: < 80%"
  smoke_tests:
    pre_rollout:
      - "Health endpoint returns 200"
      - "Authentication works"
      - "Core read API returns data"
      - "Core write API succeeds"
    during_rollout:
      - "Monitor same metrics in real-time"
      - "Auto-abort canary if any metric exceeds threshold"

Smoke Test Metrics Dashboard

smoke_metrics:
  reliability:
    pass_rate_last_30_days: 99.7%
    consecutive_passes: 187
    last_failure: "2026-05-12 (DB connection pool exhausted)"
  performance:
    avg_execution_time: "2m 43s"
    p95_execution_time: "4m 12s"
    max_execution_time: "5m 01s"
  coverage:
    infrastructure_checks: 4
    auth_checks: 3
    core_business_checks: 8
    total_tests: 15
  rollback:
    auto_rollbacks_triggered: 2
    avg_rollback_time: "45s"
    rollback_success_rate: 100%

Rules

1. Smoke tests must complete in under 5 minutes — 300 seconds total maximum
2. Every deployment must pass smoke tests before reaching users — no exceptions
3. Smoke tests must be 100% deterministic — zero tolerance for flakiness. Fix immediate removal
4. A single critical smoke test failure triggers automatic rollback or pipeline block
5. Smoke suite must be reviewed and updated every sprint to reflect current critical paths
6. Smoke tests must not require complex test data setup — use defaults or existing seed data
7. Production smoke tests must be read-only — never create, update, or delete production data
8. Each smoke test must be independent — no shared state or ordering dependencies
9. Infrastructure tests must run first — if infrastructure is down, skip functional tests
10. Smoke test results must be recorded with evidence (response times, status codes, logs)
11. Smoke test credentials must be rotated every 90 days and stored in secrets management
12. Non-critical failure rate below 95% blocks promotion but does not trigger rollback
13. Post-deployment health monitoring must continue for at least 15 minutes after smoke pass
14. Smoke tests must be version-controlled alongside application code and pipeline config
15. No more than 25 tests in the smoke suite — exceeding this requires pruning
16. Rollback from smoke failure must be tested quarterly to verify automation works
17. Smoke tests for staging may include write operations; production smoke is read-only
18. Each smoke test must produce a clear pass/fail signal — no ambiguous results
19. Canary deployments require smoke tests to pass before traffic shifting
20. Smoke test results must feed into deployment health dashboards automatically

References

• references/bvt-strategy.md — Build Verification Testing (BVT)
• references/canary-testing.md — Canary Testing
• references/deployment-health.md — Deployment Health Checks
• references/health-check-patterns.md — Health Check Patterns
• references/smoke-automation.md — Smoke Test Automation
• references/smoke-testing-advanced.md — Smoke Testing Advanced Topics
• references/smoke-testing-architecture.md — Smoke Testing Architecture and System Design
• references/smoke-testing-fundamentals.md — Smoke Testing Fundamentals
• references/smoke-testing-strategies.md — Smoke Testing Strategies
• references/smoke-testing-strategy.md — Smoke Testing Strategy and Decision Frameworks

Handoff

After smoke testing, hand off to:

• quality-regression-testing — if smoke passes, run targeted regression on affected areas
• quality-acceptance-testing — if smoke passes and UAT is scheduled
• quality-e2e-testing — for deeper end-to-end verification of critical journeys
• quality-load-testing — if smoke reveals performance concerns

Implementation Patterns

Observer Pattern for Event Handling

` interface EventObserver { onEvent(event: T): Promise; }

class EventBus { private observers: Set<EventObserver> = new Set(); subscribe(observer: EventObserver): void { this.observers.add(observer); } unsubscribe(observer: EventObserver): void { this.observers.delete(observer); } async emit(event: T): Promise { const results = Array.from(this.observers).map(o => o.onEvent(event)); await Promise.allSettled(results); } } `

Configuration-Driven Approach

config: defaults: timeout: 30s retryCount: 3 overrides: production: timeout: 60s retryCount: 5 development: timeout: 300s retryCount: 1

Production Considerations

Deployment Checklist

• Configuration validated against schema before startup
• Health check endpoints registered and monitored
• Graceful shutdown with draining period (30s timeout)
• Resource limits configured (CPU, memory, file descriptors)
• Log level set appropriate for environment
• Metrics endpoint secured and exposed
• Rate limiting configured per-tier
• TLS certificates valid and auto-renewing
• Database migrations run as separate deployment step
• Feature flags ready for gradual rollout

Monitoring and Alerting

Anti-Patterns

Performance Optimization

Caching Strategy

Cache hierarchy: L1 (in-memory local) → L2 (distributed Redis/Memcached) → L3 (CDN/Edge). Cache invalidation: TTL-based (simple, stale), event-based (complex, fresh), write-through (consistent, higher write latency), write-behind (fast writes, eventual consistency).

Resource Pooling

• Database connections: Pool of reusable connections (HikariCP, pgBouncer)
• HTTP connections: Keep-alive + connection pooling for external calls
• Thread pool: Bounded thread pools for async task execution

Profiling Methodology

1. Establish baseline with production traffic profile
2. Profile CPU with sampling profiler (pprof, perf, async-profiler)
3. Profile memory with heap dumps and allocation tracking
4. Profile I/O with strace/perf trace for syscall analysis
5. Profile latency with distributed tracing (OpenTelemetry)
6. Identify bottleneck, formulate hypothesis, implement fix
7. Re-profile to verify improvement, repeat

Security Considerations

Threat Modeling (STRIDE)

• Spoofing: Identity validation, authentication
• Tampering: Integrity checks, digital signatures
• Repudiation: Audit logs, non-repudiation
• Information disclosure: Encryption, access control
• Denial of service: Rate limiting, resource quotas
• Elevation of privilege: Principle of least privilege

Supply Chain Security

• Dependency scanning: Snyk, Dependabot, Trivy
• SBOM generation: CycloneDX or SPDX format
• Signed commits: GPG or SSH commit signing
• Artifact verification: Checksum validation, signature verification

Secrets Management

• Secrets never in code — always in secrets manager (Vault, AWS Secrets Manager)
• Rotation policy: Rotate database credentials every 90 days
• Access audit: Log every secrets access, alert on anomalies
• Encryption at rest and in transit for all secrets
• Principle of least privilege: each service gets only its own secrets

Rules

• Default-deny security posture — allow only explicitly required access.
• All inputs validated, all outputs encoded, all errors handled.
• Defend in depth — multiple layers of security controls.
• Fail securely — errors default to safe behavior.
• Log security-relevant events for audit and investigation.
• Keep dependencies updated — automate vulnerability scanning.
• Design for observability from day one, not as an afterthought.
• Document all architectural decisions with rationale.
• Review code for security, performance, and correctness before merging.

Architecture Decision Trees

Smoke Test Selection

Smoke Test Categories

• Health check → Service responds, returns 200 OK
• Database connectivity → Read/write works, migrations applied
• Auth flow → Login/logout/token refresh works
• Critical transaction → Core business operation succeeds
• External dependency → Integrated services reachable