Paddleocr UI Test

PaddleOCR UI Testing

OCR screenshot text extraction + Playwright Accessibility Tree cross-validation, with 6 levels of UI defect detection.

Prerequisites

• Environment variables: PADDLEOCR_API_KEY or SILICONFLOW_API_KEY (required)
• Python packages: openai, playwright, Pillow
• Browser: playwright install chromium

Control Knobs

1. Test Scope (--levels)

Default: L1,L3. If the user asks for a "full check", enable L1,L2,L3,L4,L5.

2. Page Type (--profile)

Profiles automatically set levels, viewport, wait_ms, and rule overrides.

3. Rule Tuning (rules/*.json)

Each rule file controls the detection behavior for one level. The agent may adjust them as needed.

L1: rules/text-consistency.json

• default_strategy: exact (strict) / substring (default) / fuzzy (tolerant)
• match_strategies.fuzzy.threshold: fuzzy match threshold, default 0.8
• ignore_patterns: text patterns to ignore, such as versions or hash strings

L2: rules/layout-anomaly.json

• overflow.enabled: whether to detect overflow
• element_overlap.enabled + iou_threshold: overlap detection toggle and IoU threshold
• touch_target_size.enabled + min_width_px/min_height_px: minimum touch target size
• full_page_text.width_threshold/height_threshold: thresholds for full-page text detection

L3: rules/dom-ocr-crossval.json

• fuzzy_match.enabled + threshold: fuzzy match toggle and threshold (0.6 means match, 0.7 means warning)
• count_mismatch.delta_threshold: tolerated ratio for count differences, default 0.3
• ignore_patterns: text patterns to ignore

L4: rules/accessibility.json

• missing_alt.enabled: image alt checks (enabled by default)
• missing_label.enabled: interactive element label checks (disabled by default, enabled in the form profile)
• canvas_rendered_text.enabled: detect text visible in OCR but invisible in A11y
• emoji_as_icon.enabled: detect emoji used as icons

L5: rules/i18n.json

• languages: regex patterns for each language (zh/en/ja/ko)
• common_false_positives: terms that should not be falsely flagged, such as OK, API, URL

L6: rules/dynamic-content.json

• state_transitions: predefined state transition patterns (loading→content, content→error)
• max_tracked_changes: maximum number of tracked changes, default 5

4. Specific Expectations (--config)

When the user has explicit text expectations, generate a config JSON:

{
  "expected_texts": {
    "page_title": "Login Page",
    "username_label": "Username",
    "submit_button": "Login"
  },
  "expected_language": "en",
  "ignore_texts": ["Powered by"]
}

5. Regression Comparison (--baseline / --baseline-file)

• --baseline: save the current run result as the baseline (for first-time testing or when the user says "save baseline")
• --baseline-file baseline.json: compare against a historical baseline to detect removed/added text, layout drift, and count changes
• The baseline file is automatically saved as baseline.json in the output directory

Supporting Parameters

• --annotate: mark issue regions on the screenshot; recommended when failures occur
• --actions: action sequence for L6 dynamic testing, such as "click(#btn);wait(2000);screenshot"
• --output: output directory, default ./test-results

Output Artifacts

Integration

Input Contract

The minimum required input for this skill is --url. Upstream skills do not need to understand this skill's config format; the agent is responsible for converting inputs.

Added lightweight flexible input modes while preserving the original behavior:

• --input-mode url (default): original mode, the skill captures screenshot + a11y data by itself
• --input-mode artifacts --artifacts-dir <dir>: consume page artifacts exported upstream
• --input-mode mcp --input-json <file>: consume an MCP payload (v1 currently supports path-based fields only)

Example MCP payload v1:

{
  "source": "playwright-mcp",
  "url": "https://example.com",
  "viewport": "1280x720",
  "screenshot_path": "./artifacts/screenshot.png",
  "a11y_tree_path": "./artifacts/a11y_tree.json",
  "dom_path": "./artifacts/dom.html"
}

Upstream Output Adaptation

Collaboration Mode Selection

Choose the collaboration mode automatically based on user intent:

dev-browser Session Reuse

When collaborating with dev-browser, avoid launching a duplicate browser session:

1. After dev-browser finishes navigation/interactions, export:
   • Screenshot: page.screenshot()
   • HTML: page.content()
   • A11y Tree: page.evaluate(A11Y_TREE_SCRIPT)
2. This skill consumes those artifacts directly without reloading the page
3. Benefits: preserve session/cookies and reduce API calls

Recommended downstream integration for Playwright MCP / UI test generation MCP:

1. The upstream MCP exports screenshot + a11y tree + DOM into local artifacts
2. This skill consumes them directly with --input-mode artifacts or --input-mode mcp
3. This skill outputs report.json / annotated.png for downstream agents or MCP tools

Note: L6 --actions is only executed in url mode in the current lightweight version.

Output Contract

Artifacts produced by this skill can be consumed by downstream skills:

Collaboration Flow With Other Skills

ui-ux-pro-max  →  Define design intent (color / typography / copy / accessibility requirements)
     ↓
dogfood        →  Explore the real page (find issues / unexpected behavior)
     ↓
paddleocr-ui-test →  Validate and guard (turn findings into automated regression checks)

• dev-browser is the execution engine: all skills can use it for page navigation and interaction
• ui-ux-pro-max defines the ideal state: what the page should look like
• dogfood is the discovery layer: what is actually wrong on the page
• This skill is the validation layer: turn findings into repeatable automated checks