Sap Testing

SAP Testing — Overview

Non-negotiable execution gate

Every required phase step and artifact is a prerequisite for execution. The abapfs_run_playwright_tests tool verifies the full workflow state and will reject the run if any required item was skipped, deferred, left missing, stale, or unverified. Complete every requirement at the point specified by the relevant phase workflow; you cannot defer it and repair the omission at run time.

Why

This is production SAP for a real business. Every bug that ships costs money — wrong invoice, wrong stock movement, wrong customer charge, blown financial close. Test suites here are the last gate before code touches accounts, orders, or inventory. The goal is NOT to finish; the goal is to catch every bug before it ships. Cutting corners because the code is long, the sandbox is thin, or the task is tedious is exactly how a million-dollar defect reaches production. Speed is worthless if you miss the bug.

The available SAP Testing tools drive Playwright against SAP WebGUI. Specs use the bundled SapSession/SapArtifacts runtime through @sap-testing/runtime, and runs produce .docx evidence. There is no terminal, no npm, no .env — every action below is either a tool call or a normal file edit inside the configured test folder.

Where things live — two different places, on purpose

Everything for one program lives together under tests/<PROGRAM>/ inside the test folder:

<test folder>/
├── tsconfig.json                (managed automatically by SAP Testing — never hand-edit)
├── recordings/                  User-assisted Playwright reference recordings; never runnable specs
└── tests/
    └── <PROGRAM_NAME>/          One folder per ABAP object under test
        ├── test-cases/          Phases 1–4 output — _flow.md + _units.md + _findings.md (P1) + _screens.md (P2) + TC-XXX.md + _index.md + _index.docx (P3) + TC-XXX.data.md (P4)
        ├── test-scripts/        Phase 6 output — TC-XXX.spec.ts (real @playwright/test specs)
        └── test-results/        Phase 5 (data.json) + Phase 7 (evidence) output — segregated per connectionId
            ├── <CONNECTION-ID>/              (e.g. DEV-100)
            │   └── <TC-ID>/                 per TC: screenshots, manifest.json, data.json, fixtures/
            └── <PROGRAM>-<CONNECTION-ID>-report.docx   ONE aggregated docx per (program, connectionId)

Rules:

• All artifacts for a program are siblings under tests/<PROGRAM>/. Never scatter TC-001.md, TC-001.spec.ts, and its screenshots.
• Raw user recordings live under recordings/*.recording.ts, outside tests/. They are temporary reference evidence, not phase outputs or runnable specs.
• Program folder name = the name the user asked for (report / tcode / class), uppercase, no spaces. When the user names a TRANSACTION, keep the folder as the tcode — do NOT rename it to the program the tcode runs. Phase 1 resolves the tcode to its executable object and records BOTH in _flow.md frontmatter (target: = the tcode/name, resolvedProgram: = the report/class actually analysed), so the mapping is explicit and no later phase re-derives it.
• The per-connection results folder is the connectionId in UPPERCASE (test-results/<CONNECTION-ID>/, e.g. DEV-100). The framework derives this folder by uppercasing the connectionId at run time, so any hand-written cache/results path MUST use the uppercase form — a lowercase folder is silently not found on a case-sensitive filesystem (Linux/macOS).
• Every .spec.ts you write imports from the fixed specifier @sap-testing/runtime — never a relative path (there is no helpers/ folder in the test folder to point at). See build-scripts and helpers-reference for the full import convention and method reference.

Skills, tools, and agents are three different things — know which is which

Backticked names in these docs refer to one of three kinds of thing, and you interact with each differently. When a doc names one, it should say the kind ("the X skill/tool/agent"); if it doesn't, use this list to tell them apart:

• Skills (you LOAD/READ them — they are operating procedures): analyze-and-plan, explore-ui, design-cases, define-data, prepare-data, build-scripts, run-scripts, sap-webgui, sap-webgui-recording, helpers-reference, anst-guide. "follow X" / "load X" always means a skill.
• Tools (you CALL them — they take inputs and return a result): the 10 SAP Testing tools listed below (abapfs_get_test_folder, abapfs_get_sap_webgui_url, abapfs_run_playwright_tests, abapfs_build_test_index, abapfs_build_test_index_docx, abapfs_split_test_cases, abapfs_verify_test_data_usage, abapfs_check_test_data, abapfs_build_evidence_report, abapfs_analyze_anst_enhancements) plus the ABAP research tools (abapfs_search_objects, abapfs_get_object_source, abapfs_run_sql_query, abapfs_get_sql_syntax, abapfs_get_connected_systems, abapfs_download_object, …).
• Agents (you DELEGATE to them — they run as separate subagents with their own context): sap-source-download, sap-code-grep, sap-enhancement-research, sap-findings-reviewer, sap-screens-reviewer, sap-test-plan-reviewer, sap-data-scout, sap-task-helper, anst-enhancement-analyser.

If a tool you need isn't in your active toolset, search for it by name before giving up (see "Tools can be hidden" below).

The 10 available SAP Testing tools

Everything else — reading ABAP source, live SAP UI exploration, writing markdown/TypeScript files — uses tools you already have (your built-in file tools, and your built-in browser tool for live exploration; see analyze-and-plan for source reading and explore-ui for live exploration).

The 7 phases (one artifact each, in dependency order)

Test creation is deliberately split into seven standalone phases so no single step is rushed. Each phase produces ONE kind of artifact and hands off to the next:

Phases → skills

These are separate skills, not sub-agents — they activate automatically when relevant. You (the active Copilot agent) do the work directly, following whichever skill applies. Delegate only specific, bounded jobs to dedicated agents instead of doing them yourself — each is deliberately small and narrowly scoped:

• sap-code-grep — mechanical grep/count over the program's own source (message counts, branch counts, AUTHORITY-CHECK, flow-control exits), using an ACTUAL Grep over the snapshot (real line numbers, no eyeballing). See analyze-and-plan Step 5.2.
• sap-source-download — synchronously discovers recursive includes and downloads one complete source snapshot. See analyze-and-plan Step 1.
• sap-enhancement-research — finds and thoroughly explains customer enhancements on the standard call surface. See analyze-and-plan Step 5.2; launch it and sap-code-grep together in one parallel batch.
• sap-findings-reviewer — the adversarial Phase 1 gate: reads the source itself and re-checks _findings.md/_flow.md/_units.md for fabricated line numbers, missed MESSAGE/branch/auth statements, un-analysed value-transformation/default logic, wrong date/number formats, and an under-counted target minimum. See analyze-and-plan Step 6.1. Phase 1 does not hand off until it returns PASS.
• sap-screens-reviewer — the adversarial Phase 2 gate: statically checks _screens.md describes the live web-GUI (accessible names, initial states, dialogs, ALV) and is NOT an ABAP selection-screen/source description (no ABAP names, MODIF IDs, source snippets, message classes). Catches a _screens.md derived from source instead of observed. See explore-ui Step 9.1. Phase 2 does not hand off until it returns PASS.
• anst-enhancement-analyser — consumes an ANST xlsx after anst-guide, calls abapfs_analyze_anst_enhancements, and researches every classified object.
• sap-test-plan-reviewer — the adversarial check that your test-case count and category coverage actually meet the minimum _findings.md derives, before you build the index. See design-cases Step 3. abapfs_build_test_index will not run until this agent has returned PASS.
• sap-data-scout — resolves ad-hoc data requirements from a live SAP system (find 5 articles listed at both sites, find open POs in company code 1000, etc.). Invoke in parallel, one instance per distinct requirement. Handles abapfs_get_sql_syntax + abapfs_run_sql_query + spot-validation internally. See prepare-data Step 2.
• sap-task-helper — a generic one-shot helper for a caller-defined tedious or high-volume task. Use only with a bounded objective, explicit skills to read, artifact paths, disjoint allowed writes, and a compact output contract. Multiple instances may run sequentially or in parallel; each starts with zero context and must not handle user decisions, live exploration, destructive actions, or phase orchestration.

Use these dedicated agents for their bounded tasks so large enumerations remain complete and verifiable instead of being shortened when context grows.

Every subagent is ephemeral and one-shot. A delegated agent runs with its own fresh context, cannot see your conversation, cannot ask you a follow-up, and returns exactly ONE response — there is no back-and-forth. Two consequences: (1) give it everything it needs UP FRONT (all inputs its contract lists); a missing input means it must reject and you re-invoke, costing a full round. (2) Trust its single response to be complete — a good gate agent reports every gap at once, and a good worker reports every value plus any deviation it made, precisely because it knows it gets no second message. If an agent's rejection is vague ("input invalid"), that's an agent bug to raise with the user, because the ephemeral contract requires it to name the exact missing/invalid thing and the fix.

Delegation discipline — pass inputs, not methods. When you delegate to a subagent, pass ONLY the inputs its contract specifies (IDs, paths, the objective, e.g. for sap-source-download: <TEST_FOLDER>, <PROGRAM>, connectionId, object name, object type). Do NOT tell it HOW to do its job — which tools to use, to read the source with abapfs_get_object_source, to "create the files," to skip a step. Every subagent already knows its procedure and has its own guardrails, and over-instructing can OVERRIDE those guardrails and make it do the wrong thing: telling sap-source-download to "read the code and write the files" makes it fabricate the compliance snapshot by hand instead of downloading it with abapfs_download_object — the exact opposite of its purpose. Give it the objective and the inputs; let it choose the method. If you believe a subagent's method is wrong, that's a skill bug to raise with the user — never something to "fix" by adding how-to at the call site.

One phase per chat is supported

Each phase skill is a standalone operating procedure. A fresh chat must be able to continue from artifacts on disk without the prior chat's context. Starting a new chat for each phase is therefore valid and often preferable:

1. analyze-and-plan reads the source and writes _findings.md.
2. explore-ui explores the live UI and writes _screens.md.
3. design-cases writes the reviewed TC-XXX.md files and the index.
4. define-data writes each TC-XXX.data.md requirement spec.
5. prepare-data resolves those specs into per-system data.json caches.
6. build-scripts converts approved cases to TC-XXX.spec.ts.
7. run-scripts validates prerequisites, executes, and produces evidence.

The user should only need to name the phase plus the program or TC-ID. Do not ask them to repeat facts already recorded in _findings.md, _index.md, _screens.md, TC frontmatter, or per-system result folders. Do not rely on conversation memory either: disk artifacts are the handoff contract.

The phases are mostly linear, with ONE expected loop. A seeded data requirement (Phase 4) points at a spec that only exists after Phase 6, so it cannot be resolved on the first Phase 5 pass — Phase 5 marks it deferred-until-phase-6, Phase 6 writes the seeding spec, then Phase 5 runs a SECOND time to resolve just the deferred seeded keys before Phase 7. So the order for programs with seeded preconditions is 4 → 5 → 6 → 5 → 7. This is expected, not rework; each phase's handoff names any deferred seeded key so the next chat knows a second prepare pass is owed.

Tools can be hidden. The editor may not surface every SAP Testing tool until it is searched for, and smaller models often fail to find them and improvise instead. At the start of any phase, if a tool that phase needs (always abapfs_get_test_folder; also abapfs_run_playwright_tests for Phase 7, abapfs_build_test_index/abapfs_split_test_cases for Phase 3–4, etc.) is not available, SEARCH your available tools for it by name before proceeding. Never substitute a terminal command or a fabricated result for a tool you couldn't find — say which tool is missing.

Universal rules

1. ALWAYS call abapfs_get_test_folder before any other SAP Testing action. Treat its returned absolute path as <TEST_FOLDER> for the whole chat. Never infer it from the workspace, current directory, a prior chat, or a path mentioned in an old artifact. If unset, STOP and ask the user to run "ABAP FS: Enable SAP UI Testing Features". If it is not open in the workspace, ask the user to add it before using normal file tools.
2. Then establish the target system. Call abapfs_get_connected_systems and pick the target connectionId. If ambiguous, ASK. The connectionId determines the WebGUI URL, ABAP tool calls, and test-results/<connectionId>/.
3. Rediscover context from disk in every chat. Confirm the program and TC-ID from the user's request and artifacts under <TEST_FOLDER>/tests/. If the request is ambiguous and multiple candidates exist, ask; never silently pick one or depend on prior-chat memory.
4. Every phase enforces its input gate before work. Missing or inconsistent upstream artifacts are blockers. Stop the current workflow and follow analyze-and-plan for test-plan problems, prepare-data for data problems, build-scripts for spec problems, or run-scripts for execution problems. Never create plausible replacements from memory.
5. No login logic in a spec, ever. abapfs_run_playwright_tests signs the browser in itself, before any spec runs, using a SAP reentrance ticket minted from the ABAP FS connection — so a spec never needs to. Never write credentials, a logon step, or an auth workaround into a .spec.ts, .md, or .data.md. If a run still lands on a logon screen, that is a real failure to report (see run-scripts), not something to work around.
6. Never bypass the runtime. All specs use SapSession/SapArtifacts from @sap-testing/runtime. A missing capability is a real limitation — see helpers-reference for what to do (you cannot add a helper method yourself).
7. No CSS-class selectors, no ref numbers, no positional guessing. Only role + accessible name, scoped to a container — enforced inside the runtime; keep it in mind for any sap.raw() escape-hatch code.
8. English only. SAP language must be en in every configured system URL.
9. Test case, data spec, screens map, and spec are 1:1. For every TC-ID: test-cases/TC-001.md ↔ test-cases/TC-001.data.md (if data needed) ↔ test-scripts/TC-001.spec.ts. The screen map is shared per program: test-cases/_screens.md.
10. Data resolution is a separate phase. Specs never hardcode material numbers, plants, or any transactional data; data is resolved per system, not baked in.
11. Don't invent SAP behavior. Confirm against the source / ABAP tools before generating. Read-only clicks during exploration are fine; destructive actions (post, submit, delete, send, approve, release, reverse, reprocess, and any Execute in an update/live/production mode of a write-capable report) MUST be user-approved first. If a selection screen has a Test/Simulate vs Update radio, exploration MUST select the test/simulate side before Execute — see the explore-ui Step 5 destructive-mode gate.
12. Never hand-edit tsconfig.json in the test folder — SAP Testing manages it automatically and will overwrite your changes.
13. Frontmatter lives at the very top of the file, always. Any artifact that carries YAML frontmatter (TC-XXX.md, TC-XXX.data.md, _screens.md, _flow.md, _units.md) must have it as a single ----delimited block that is the FIRST thing in the file — never wrapped in a ```yaml/```markdown code fence, never under a heading, never as prose. The tools parse ONLY a leading --- block; misplaced frontmatter is silently ignored, which for a .data.md means resolveTestData reads zero keys and every data.<key> resolves to undefined.
14. Load a phase's companion skills at the START, before committing to an approach. Each phase names the skills it depends on. Load them up front — not after you're already stuck — so you know your options and escape hatches BEFORE you pick a method. Committing to a wrong approach and only discovering the right tool/skill afterwards is how a whole phase gets redone.
15. When blocked, ASK — never shortcut. A blocker (a control you can't drive, a screen you can't reach, data you don't have, a tool that seems to fail or time out, genuine ambiguity) is a signal to STOP and either ask the user or use the escape hatch the relevant skill documents. It is NEVER license to take a shortcut that violates a skill rule — deriving _screens.md from source, guessing a locator, fabricating data, or skipping a gate/reviewer. A wrong-but-fast artifact is worse than a paused one: it looks done and ships an untested path. Also verify a suspected failure before reacting — a SAP round-trip/splash can look like a hang; re-confirm the tool actually failed before concluding it did.
16. ABAP FS is the ONLY preparation/research data channel — SE16N is evidence, never a fallback. Source reads, test-data discovery, baselines, joins, and authoritative post-run SQL use ABAP FS ADT tools. A generated spec MUST additionally use sap.se16n() when its case declares verification: sql|mixed, solely to prove the resulting state is visible to business users and capture screenshots. Never scrape SE16N to prepare/cache test data or work around an ABAP FS failure.
17. ABAP FS HTTP 401/403/5xx = connection problem, not a permanent restriction. When any ABAP FS tool returns 401, 403, or 5xx, ABAP FS almost always can't reach the target SAP system — usually because the SAP session expired. Tell the user, briefly: "ABAP FS can't reach <connectionId> (HTTP 401/403). Please check the ABAP FS connection and, if needed, reload VS Code to re-establish it — then retry." Do NOT switch to SE16N-via-browser, do NOT fabricate values, do NOT retry with a different auth trick. A WebGUI page showing a logon screen is a separate, browser-side problem — during a abapfs_run_playwright_tests run it means auto-login failed (see run-scripts); during live exploration, ask the user to log in in that browser window. Neither means ABAP FS is disconnected.
18. abapfs_get_object_source with objectType: "TABL" may return an empty body (2–4 lines, no fields) when the table has no DDL source form on this system — common for Z-tables delivered by transport to non-dev tiers. The tool is working; the source just isn't there. Fall back to a zero-row SELECT * FROM <TABLE> WHERE <any-key> = '<impossible-value>' via abapfs_run_sql_query — the response includes full column metadata (name + ABAP type) even when 0 rows come back. Never guess field names. Applies to analyze-and-plan, define-data, prepare-data, and any phase that inspects a Z-table's structure.