# Awt

> AI-powered E2E web app testing with self-healing DevQA loop. Generate test scenarios from URLs or natural language, execute with Playwright, auto-fix failures. Supports YAML scenarios, visual matching (OpenCV + OCR), pattern learning. Use for QA, bug detection, regression testing.

- Skill: `ksgisang/awt` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add ksgisang/awt`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ksgisang/awt/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ksgisang (https://skillmd.com/u/ksgisang)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/ksgisang/awt

---


# AWT — Eyes and Hands for Your AI Coding Tool

AWT (AI Watch Tester) gives your AI coding tool the ability to **see and interact with web applications**. Your AI designs the test strategy; AWT executes it with a real browser — clicking, typing, taking screenshots, and reporting back.

## When to Use This Skill

- User wants to **test a web application** (E2E, QA, regression)
- User says **"test it"**, **"check if it works"**, **"verify the login"**
- User needs to **detect bugs** after code changes
- User wants **automated regression testing**
- E2E test failed and user wants to **find the root cause in source code**

## When NOT to Use

- Unit testing or API-only testing (AWT is for UI/E2E)
- Performance/load testing (use k6, Artillery)
- Mobile native app testing (web and desktop only)

## ⛔ CRITICAL RULES — READ BEFORE DOING ANYTHING

1. **NEVER use `-y` or `--auto-approve`** — the user MUST approve before any test runs.
2. **NEVER use `aat devqa`** — it runs the entire pipeline without user checkpoints. Always use the 4-step workflow below.
3. **NEVER run a test without showing the scenario to the user first** and getting explicit approval (e.g. "진행해", "go ahead", "yes", "run it").
4. **NEVER auto-fix code or scenarios without user permission** — always report failures and ask the user what to do.
5. **NEVER set `headless: true`** — users MUST see the browser.
6. **NEVER guess element names** — always scan first and use real data from `scan_result.json`.

---

## ★ MANDATORY 4-STEP WORKFLOW ★

When the user asks to test anything, you MUST follow these 4 steps **in order**. Do NOT skip any step. Do NOT combine steps.

### STEP 1: SCAN — Analyze the target site

```bash
aat scan --url <URL>
```

After scanning, **read `.aat/scan_result.json`** and present a summary to the user:

```
"I scanned http://localhost:3000 and found 83 interactive elements:
 - 5 input fields (email, password, search, ...)
 - 12 buttons (Login, Sign Up, Submit, ...)
 - 15 links (Home, About, Dashboard, ...)
 
 Should I create a test scenario based on these elements?"
```

**⏸️ WAIT for user response before proceeding to Step 2.**

### STEP 2: GENERATE + PRESENT — Create scenario and show it

Write a YAML scenario using the exact data from `scan_result.json`. Then **show the full scenario to the user**:

```
"Here is the test scenario I created (5 steps):

 1. 🌐 Navigate to http://localhost:3000
 2. 🖱  Type 'test@example.com' into email field
 3. 🖱  Type 'password123' into password field  
 4. 🖱  Click 'Login' button (critical)
 5. ✅ Verify URL contains '/dashboard'

 Should I run this test? Or would you like me to modify anything?"
```

**⏸️ WAIT for user approval before proceeding to Step 3.**
- If user says "modify X" → edit the scenario and show it again
- If user says "go ahead" / "진행해" / "yes" → proceed to Step 3
- If user says "cancel" → stop entirely

### STEP 3: EXECUTE — Run the approved scenario

```bash
aat run --skill-mode --fast scenarios/<scenario>.yaml
```

Monitor the output. **If any step fails:**

1. **STOP immediately** — do not continue to the next scenario
2. **Read the failure details** from the AWT output
3. **Read `ACTUAL_CAUSE` first** if the `=== AWT SKILL DEVQA ===` block has one — see below
4. **Read the SCREENSHOT** if provided in the same block
5. **Report to the user:**

```
"Test failed at Step 3 (Click Login button):
 - Error: Element 'Login' not found in DOM
 - Possible cause: The button text might be 'Sign In' instead of 'Login'
 - Screenshot: .aat/screenshots/step_003_fail.png
 
 Should I:
 (a) Fix the scenario (change 'Login' to 'Sign In') and re-test?
 (b) Fix the source code instead?
 (c) Skip this step and continue?"
```

#### `ERROR` vs `ACTUAL_CAUSE` — do not mix them up

On a critical failure the block can carry two lines:

```
ERROR: Dashboard must load after login
ACTUAL_CAUSE: Bounced to the login page: https://app.example.com/login?next=/dashboard
```

`ERROR` is the scenario author's description of what *should* have happened. It is
a label written before the run, not a finding. `ACTUAL_CAUSE` is what the browser
actually did. **Diagnose from `ACTUAL_CAUSE`**; treating `ERROR` as the cause
produces exactly the wrong diagnosis (here: "the dashboard is broken", when the
truth is that the session was never established).

When `ACTUAL_CAUSE` is absent, `ERROR` already holds the real reason.

#### `STATUS: WARNINGS` — a step ran but changed nothing

Exit code 3, with a `=== AWT SKILL VERIFY ===` block listing each warned step.
The click was performed and did not raise, but not one pixel moved, which almost
always means it landed next to its target rather than on it. **Do not report
such a run as passed.** Read the screenshot for the warned step, check the
target it was given, and tell the user what you found. A warning left unread is
how a missed click surfaces three steps later as "the product is broken".

**⏸️ WAIT for user instruction before fixing or re-running.**

### STEP 4: REPORT — Summarize results

When all steps pass, report to the user:

```
"✅ Test complete: 5/5 steps passed (37.8 seconds)
 
 All navigation links and login flow are working correctly."
```

If the user wants something to keep or to hand to someone else, add
`--report pdf` to the run. It writes `reports/<scenario id>/report.pdf`, with
every step listed and the screenshots of failed and warned steps embedded in the
file, and prints the path. Do not add it by default: most runs are read in the
terminal and a report nobody asked for is a file nobody opens.

Add `--report-screenshots all` when the user wants to see the steps that worked
— a hand-off or a record of a clean run. With the default only failed and warned
steps bring their screenshots, so a report of a run where nothing went wrong
contains no images at all.

---

## Auto-Setup

```bash
which aat || (pip install aat-devqa && playwright install chromium)
```

---

## Scenario YAML Reference

### Writing Scenarios from scan_result.json

**Read `.aat/scan_result.json`** and write YAML using the exact element data:

```yaml
id: "SC-001"
name: "Login Flow"
steps:
  - step: 1
    action: navigate
    value: "http://localhost:3000/login"
    description: "Go to login page"

  # Use EXACT coordinates/selectors from scan_result.json
  - step: 2
    action: find_and_type
    target:
      selector: "#email"           # from scan_result.json
      text: "Email"
    value: "test@example.com"
    description: "Enter email"

  - step: 3
    action: find_and_click
    target:
      text: "Login"                # from scan_result.json
    region: main                   # exclude nav panel
    critical: true                 # stop if login fails
    description: "Click login button"

  - step: 4
    action: assert_url
    value: "/dashboard"
    on_fail: stop
    message: "Login failed — not redirected to dashboard"
    description: "Verify redirect to dashboard"
```

**Rules:**
- Use `selector` from scan data when available (most reliable)
- Use `text` as fallback (OCR-based)
- Use `region: main` for all clicks (avoid nav panel False Positives)
- Mark login/auth steps as `critical: true`
- Add `assert_url` after navigation-triggering clicks

<!-- BEGIN GENERATED: actions -->

### Actions (27 types)

| Category | Actions |
|----------|---------|
| Navigation | `navigate`, `go_back`, `refresh` |
| Find + mouse | `find_and_click`, `find_and_double_click`, `find_and_right_click` |
| Find + keyboard | `find_and_type`, `find_and_clear` |
| Direct | `click_at`, `type_text`, `press_key`, `key_combo` |
| Assert | `assert`, `assert_text`, `assert_screen_changed`, `assert_url` |
| Session | `save_session`, `load_session` |
| Input | `upload_file`, `select_option` |
| Control flow | `if_visible`, `include` |
| Query | `find`, `get_text` |
| Utility | `wait`, `screenshot`, `scroll` |

See `references/scenario-schema.md` for what each action needs.

<!-- END GENERATED: actions -->

### Key Step Options

```yaml
- action: find_and_click
  target:
    text: "Submit"               # OCR text
    selector: "#submit-btn"      # CSS selector (highest priority)
  region: main                   # top/bottom/left/right/center/main/full
  critical: true                 # stop test on failure
  on_fail: stop                  # same as critical
  method: auto                   # auto/semantics/template/ocr/vision
  match_index: 0                 # 0=first match, -1=last
  change_threshold: 0.05         # for critical auto-verification
  learn: false                   # do not remember this position (see below)
  description: "Click submit"
```

**`learn: false` — targets whose position follows the content.** AWT remembers
where a target was found and reuses that position on later runs when the step
gives no `selector`. That helps for fixed elements such as login fields, and
misfires for anything that moves with the content: buttons inside a modal whose
height follows the text, choice overlays drawn on a question image, rows in a
list. Mark those `learn: false`. Writing a `selector` is not a substitute — the
selector wins over the remembered position, but a differently named target is
still remembered under its new name.

### assert_url (login/navigation verification)

```yaml
- action: assert_url
  value: "/dashboard"
  on_fail: stop
  message: "Login failed"
  description: "Verify redirect"
```

### Access-control tests: `expect_login_redirect`

AWT normally treats "we ended up on a login page" as a hard stop, because for most
scenarios it means the session died. An access-control test wants the opposite:
being bounced to `/login` **is** the passing outcome. Mark those steps:

```yaml
- step: 2
  action: navigate
  value: "{{url}}/admin"
  expect_login_redirect: true     # landing on /login here is expected
  description: "Open an admin page while signed out"

- step: 3
  action: assert_url
  value: "/login"
  expect_login_redirect: true
  description: "Verify the guest was sent to the login page"
```

Set it per step. The same field exists at scenario level, but it switches the
detection off for every step in the file, so use it only when the whole scenario
is about access control. Omitting the field keeps the default behaviour: a step
that does not say it expects a login redirect still hard-stops on one.

### Session Reuse

```yaml
# Save after login
- action: save_session
  name: "my_app_login"

# Load in next run (24h expiry)
- action: load_session
  name: "my_app_login"
  max_age_min: 15              # fail here if the saved session is older
```

The log says how old the loaded session is ("saved 21 minutes ago").
`max_age_min` fails at the load step instead: a service that ends idle sessions
kills a saved one long before the file looks stale, and without the cap that
shows up several steps later as a login failure, sending everyone to the
credentials for nothing.

### Region Parameter

| Region | Area | Use When |
|--------|------|----------|
| `full` | Entire screen (default) | General use |
| `main` | Right 80% | **Always use for clicks** (avoids nav panel) |
| `top` | Top 30% | Header elements |
| `bottom` | Bottom 30% | Footer, floating buttons |
| `center` | Central 60%x60% | Modal dialogs |

---

## Flutter CanvasKit Support

AWT automatically detects Flutter CanvasKit and activates Semantics:

1. After `navigate`, clicks `flt-semantics-placeholder` (3 retries, 3s each)
2. Reads `flt-semantics[aria-label]` for element coordinates
3. Falls back to OCR if Semantics unavailable

**Matching priority on Flutter:**
CSS selector → Flutter Semantics → Playwright text → OCR → Vision AI

**Flutter-specific rules:**
- Always use `region: main` (Canvas OCR picks up nav text)
- Use `verify: true` on `type_text` (Canvas input may not render)
- Add `assert_screen_changed` after clicks
- Use `method: semantics` to force Semantics lookup

---

## Source Code Root Cause Analysis

When a test fails, trace to the source code:

1. **Read test output** — which step, what error, what URL
2. **Search codebase** — grep for the expected text, URL route, component
3. **Read the component** — find why it doesn't render/redirect/respond
4. **Propose a fix to the user** — show a concrete diff, ask for approval
5. **After user approves** — apply fix, re-build, re-scan, re-test

**Key principle:** NEVER auto-apply fixes. Always show the diff and ask.

---

## CLI Commands

| Command | Description |
|---------|-------------|
| `aat scan --url URL` | Scan page, collect elements to scan_result.json |
| `aat run --skill-mode PATH` | Execute with structured output for AI |
| `aat run --skill-mode --fast PATH` | Execute in fast DOM-only mode (Next.js, React, etc.) |
| `aat run --debug PATH` | Execute with OCR candidate debug logs |
| `aat run --report pdf PATH` | Execute and write a PDF report per scenario (failure screenshots embedded) |
| `aat doctor` | Check environment |
| `aat setup` | Configure AI + Vision providers |
| `aat validate PATH` | Validate YAML scenarios |
| `aat cost` | View AI API costs |
| `aat learned list` | Show remembered coordinates and learned patterns |
| `aat learn reset "NAME"` | Forget the coordinates remembered for one target (`--all` for every one) |

### Key Flags

```
--skill-mode    Structured output for AI assistants
--fast          DOM-only matching (skip Vision/OCR — fastest for standard web apps)
--debug         Show OCR candidates and matcher details
--strict        Treat skipped steps as failures
--learn         Record healed steps for pattern learning
--no-learn      Neither use nor update remembered coordinates in this run
--slow-mo N     Slow down actions by N ms
--report FMT    Write a report per scenario: pdf | markdown (omit for none)
--report-screenshots WHICH   Steps the PDF illustrates: failures | all | none
                             ('all' pictures the steps that worked)
```

### ⛔ Banned Flags (NEVER use these)

```
-y / --auto-approve    Bypasses user approval — NEVER USE
```

---

## Best Practices

1. **Always scan before writing scenarios** — never guess element names
2. **Use `region: main`** on all `find_and_click` — prevents nav panel False Positives
3. **Mark auth steps `critical: true`** — no point testing after login fails
4. **Add `assert_url` after form submits** — verify navigation happened
5. **Use `save_session`/`load_session`** — skip login on repeated runs
6. **Read screenshots on failure** — the SCREENSHOT path is a real PNG file
7. **Mark moving targets `learn: false`** — modal buttons, choice overlays on an
   image, list rows: their position follows the content, so remembering it makes
   the next run click empty space
7. **One fix per retry** — change only the failing step
8. **Always ask user before fixing** — never auto-modify code or scenarios

## AI Providers (for Vision AI Tier 3)

| Provider | Vision | Cost | Setup |
|----------|--------|------|-------|
| Gemini Flash | Yes | Free | `aat setup` → Gemini |
| Claude | Yes | Medium | `aat setup` → Claude |
| GPT-4o | Yes | Higher | `aat setup` → OpenAI |

Vision AI is optional — Tier 1 (template) + Tier 2 (OCR) are free and work without API keys.

## Reference Files

- `references/scenario-schema.md` — Full YAML schema
- `references/cli-reference.md` — Complete CLI reference
- `references/config-reference.md` — All configuration options
- `templates/scenario-template.yaml` — Blank scenario template
- `templates/config-template.yaml` — Default config template