Testability Scoring

Testability Scoring

Browser engine

Uses the qe-browser fleet skill (.claude/skills/qe-browser/) as the primary browser engine. Vibium is installed by aqe init. The legacy scripts/run-assessment.sh + Playwright path remains as a fallback when a team already has a Playwright test suite configured, but new runs should prefer:

vibium go "$TARGET_URL"
vibium wait load
vibium a11y-tree --json > /tmp/testability/tree.json
vibium eval --stdin --json <<'EOF' > /tmp/testability/signals.json
JSON.stringify({
  headings: document.querySelectorAll('h1,h2,h3,h4,h5,h6').length,
  testIds: document.querySelectorAll('[data-testid]').length,
  forms: document.querySelectorAll('form').length,
  ariaLabels: document.querySelectorAll('[aria-label]').length,
  links: document.querySelectorAll('a[href]').length,
});
EOF
node .claude/skills/qe-browser/scripts/assert.js --checks '[
  {"kind": "no_console_errors"},
  {"kind": "no_failed_requests"}
]'

Quick Assessment:

# Run assessment on any URL
TEST_URL='https://example.com/' npx playwright test tests/testability-scoring/testability-scoring.spec.js --project=chromium --workers=1

# Or use shell script wrapper
.claude/skills/testability-scoring/scripts/run-assessment.sh https://example.com/

The 10 Principles at a Glance:

Grade Scale:

• A (90-100): Excellent testability
• B (80-89): Good testability
• C (70-79): Adequate testability
• D (60-69): Below average
• F (0-59): Poor testability

Quick Reference Card

Running Assessments

Principle Details

High Weight (15% each)

Medium Weight (10% each)

Low Weight (5% each)

Assessment Workflow

1. Navigate to URL → 2. Collect Metrics → 3. Score Principles
                                              ↓
4. Generate JSON ← 5. Calculate Grades ← 6. Apply Weights
         ↓
7. Generate HTML Report with Radar Chart
         ↓
8. Open in Browser (auto-opens)

Output Files

tests/reports/
├── testability-results-<timestamp>.json  # Raw data
├── testability-report-<timestamp>.html   # Visual report
└── latest.json                           # Symlink

Integration Examples

CI/CD Integration

# GitHub Actions
- name: Testability Assessment
  run: |
    timeout 180 .claude/skills/testability-scoring/scripts/run-assessment.sh ${{ env.APP_URL }}

- name: Upload Reports
  uses: actions/upload-artifact@v3
  with:
    name: testability-reports
    path: tests/reports/testability-*.html

QX Partner Integration

// Combine testability with QX analysis
const qxAnalysis = await Task("QX Analysis", {
  target: 'https://example.com',
  integrateTestability: true
}, "qx-partner");

// Returns combined insights:
// - QX Score: 78/100
// - Testability Integration: Observability 72/100
// - Combined Insight: Low observability may mask UX issues

Programmatic Usage

import { runTestabilityAssessment } from './testability';

const results = await runTestabilityAssessment('https://example.com');
console.log(`Overall: ${results.overallScore}/100 (${results.grade})`);
console.log('Recommendations:', results.recommendations);

Agent Integration

// Run testability assessment
const assessment = await Task("Testability Assessment", {
  url: 'https://example.com',
  generateReport: true,
  openBrowser: true
}, "qe-quality-analyzer");

// Use with QX Partner for holistic analysis
const qxReport = await Task("Full QX Analysis", {
  target: 'https://example.com',
  integrateTestability: true,
  detectOracleProblems: true
}, "qx-partner");

Vibium Integration (Optional)

Overview

Vibium browser automation can be used alongside Playwright for enhanced testability assessment. While Playwright remains the primary engine, Vibium offers complementary capabilities for certain metrics.

Installation:

claude mcp add vibium -- npx -y vibium

Vibium-Enhanced Metrics

When to Use Vibium

✅ USE Vibium for:

• Element stability metrics (auto-wait duration analysis)
• Visual consistency checks (screenshot comparison)
• MCP-native AI agent integration
• Lightweight Docker images (400MB vs 1.2GB)

❌ USE Playwright for:

• Console error detection (Vibium V1 lacks console API)
• Network performance metrics (BiDi network APIs coming in V2)
• Comprehensive browser coverage (Firefox, Safari)
• Production-proven stability (Vibium V1 released Dec 2024)

Hybrid Assessment Example

// Testability assessment using both engines
const assessment = {
  // Playwright: Comprehensive metrics
  playwright: await runPlaywrightAssessment(url),

  // Vibium: Stability metrics
  vibium: {
    elementStability: await measureAutoWaitDuration(url),
    visualConsistency: await compareScreenshots(url),
    accessibilityAttributes: await extractARIALabels(url)
  }
};

// Enhanced Observability Score
const observability =
  (assessment.playwright.consoleErrors * 0.6) +
  (assessment.vibium.elementStability * 0.4);

Vibium MCP Tools for Testability

// 1. Element Stability Measurement
const browser = await browser_launch();
await browser_navigate({ url });
const startTime = Date.now();
const element = await browser_find({ selector: ".critical-element" });
const autoWaitDuration = Date.now() - startTime;
// Lower duration = better stability

// 2. Visual Consistency Check
const screenshot1 = await browser_screenshot();
await browser_navigate({ url }); // Reload
const screenshot2 = await browser_screenshot();
const visualDiff = compareImages(screenshot1.png, screenshot2.png);
// Lower diff = better stability

// 3. Accessibility Attribute Extraction
const elements = await browser_find({ selector: "button, a, input" });
const ariaLabels = elements.map(el => el.attributes["aria-label"]);
const semanticScore = (ariaLabels.filter(Boolean).length / elements.length) * 100;

Migration Strategy

Current (V2.2): Hybrid approach

• Playwright: Primary engine for all 10 principles
• Vibium: Optional enhancement for stability metrics

Future (V3.0): When Vibium V2 ships

• Evaluate Vibium as primary engine if:
  • Console/Network APIs available
  • Production stability proven
  • Community adoption increases

Agent Coordination Hints

Memory Namespace

aqe/testability/
├── assessments/*       - Assessment results by URL
├── historical/*        - Historical scores for trend analysis
├── recommendations/*   - Improvement recommendations
├── integration/*       - QX integration data
└── vibium/*           - Vibium-specific metrics (optional)

Fleet Coordination

const testabilityFleet = await FleetManager.coordinate({
  strategy: 'testability-assessment',
  agents: [
    'qe-quality-analyzer',  // Primary assessment
    'qx-partner',           // UX integration
    'qe-visual-tester'      // Visual validation
  ],
  topology: 'sequential'
});

Common Issues & Solutions

Related Skills

• accessibility-testing - WCAG compliance (overlaps with Explainability)
• visual-testing-advanced - UI consistency
• performance-testing - Load time metrics

Credits & References

Framework Origin

• Heuristics for Software Testability by James Bach and Michael Bolton
• Available at: https://www.satisfice.com/download/heuristics-of-software-testability

Implementation

• Based on https://github.com/fndlalit/testability-scorer (contributed by @fndlalit)
• Playwright v1.49.0+ with AI capabilities (primary engine)
• Vibium v1.0+ with MCP integration (optional enhancement)
• Chart.js for radar visualizations

Vibium Resources

• GitHub: https://github.com/VibiumDev/vibium
• MCP Integration: claude mcp add vibium -- npx -y vibium
• Created by Jason Huggins (creator of Selenium/Appium)

Remember

Testability is an investment, not an afterthought.

Good testability:

• Reduces debugging time
• Enables faster feedback loops
• Makes defects easier to find
• Supports continuous testing

Low scores = High risk. Prioritize improvements by weight × impact.