Automated Testing

Automated Testing in CI/CD

Purpose

Define how to automate tests (unit, integration, system/HIL where feasible) in CI/CD pipelines for medical devices with gating and evidence capture.

When to Apply

• Setting up or updating test stages in CI/CD.

Requirements (testable)

1. Scope: Include unit, integration, and where possible system/HIL or simulators; prioritize safety-related modules. Rationale: coverage of risk areas.
2. Gating: Fail pipeline on test failures; no ignored failures for safety code. Rationale: prevent regressions.
3. Artifacts: Store test results (JUnit/XML), logs, coverage, and evidence; retain per release. Rationale: auditability.
4. Flake Management: Detect and address flaky tests; quarantine with issue filed and timeline; do not ignore. Rationale: trustworthiness.
5. Hardware Tests: If hardware required, provide conditional stages and fallbacks (sim); schedule regular hardware runs (nightly). Rationale: practicality with coverage.

Recommended Practices

• Tag tests by type/priority; run critical ones on every commit.
• Use retries sparingly and only for known flaky, with tracking.
• Parallelize to keep feedback fast.
• Use containerized or pinned environments.

Patterns

CI job snippet:

test:
  script: [ "ctest --output-on-failure" ]
  artifacts:
    when: always
    paths: [ "build/test-results.xml", "coverage/" ]

Flake tracking entry:

test_id: TEST-INT-210
status: flaky
issue: QA-1234
expires: 2026-02-01

Anti-Patterns (risks)

• Allowing tests to fail without blocking or tracking -> risk: regressions.
• No artifacts/logs retained -> risk: unverifiable results.
• Silent retries masking real failures -> risk: shipping defects.

Verification Checklist

• Unit/integration (and sim/HIL if available) tests run in CI.
• Failures gate merges; no ignored failures for safety code.
• Results/logs/coverage stored as artifacts per run/release.
• Flaky tests tracked with issues and timelines.
• Hardware-dependent tests scheduled; sim fallback provided.

Traceability

• Link CI test runs to test IDs (TEST-###) and requirements; store artifacts with release metadata.

References

• IEC 62304 verification expectations.
• FDA/MDR submission evidence needs.

Changelog

• 1.0.0 (2026-01-04): Initial automated testing skill with gating and artifact capture.

Audit History

• 2026-01-04: Audit performed. Verified:
  • IEC 62304 verification expectations correctly referenced
  • CI/CD patterns follow industry best practices for regulated environments