Golem Skill Harness

Golem Skill Test Harness

The harness in golem-skills/tests/harness/ runs coding agents against current-schema YAML scenarios. Treat src/executor.ts, src/assertions.ts, src/driver/, src/workspace.ts, and src/services.ts as the source of truth; do not add legacy fields or compatibility shims to new scenarios.

Skills and generated documentation

Catalog skills live in golem-skills/skills/{common,rust,ts,effect,scala,moonbit}/<skill>/SKILL.md. Golem templates install common plus language-specific skills in the canonical .agents/skills/ tree. Harness drivers also seed bootstrap skills there; Claude receives a .claude -> .agents symlink rather than a second catalog copy.

After changing catalog skills:

cargo build -p golem
cargo make generate-docs-skills

The CLI embeds the catalog. The docs command updates the generated How-To index (docs/src/content/next/how-to-guides.mdx), category landing pages, _meta.js files, and individual guide pages. If target/release/golem exists, the harness chooses it before target/debug/golem; rebuild the selected profile or remove the stale binary.

Prerequisites and limitations

• Node.js/npm and one requested agent runtime: Amp SDK credentials, claude, opencode, codex, or gemini.
• GOLEM_PATH must identify the workspace root. If absent, detection walks upward from the process current directory looking for both sdks/rust/golem-rust and sdks/ts/packages.
• Set GOLEM_TARGET_DIR to the directory containing the built golem executable when Cargo uses a redirected target directory. Otherwise the harness searches <GOLEM_PATH>/target/release then target/debug.
• The harness prepends that directory to PATH and starts a clean server. GOLEM_ROUTER_PORT overrides scenario router ports (default 9881); conflicting scenario ports require an explicit override. The CLI and agent drivers inherit the matching GOLEM_BUILTIN_LOCAL_URL. Set GOLEM_CUSTOM_REQUEST_PORT and GOLEM_MCP_PORT to isolate HTTP/MCP ports too. Use {{custom_request_port}} and {{mcp_port}} in scenario prompts and URLs.
• Docker is required for scenario prerequisite services (postgres, mysql, ignite, openai-mock).
• Linux skill activation fallback uses inotifywait when available. macOS deliberately starts no fswatch process. Both platforms can use atime snapshots; filesystems or mounts that do not update atime reliably can miss reads. Amp, Claude, OpenCode, and Gemini primarily report native skill-tool events; Codex falls back to filesystem tracking.
• Language toolchains must match the scenario. Rust commonly needs wasm32-wasip2; TypeScript needs pnpm and built SDK/template artifacts; MoonBit needs moon and wasm-tools.
• Effect needs its built SDK and all three template WASMs under sdks/effect; use the co-located package, not the published 1.5 SDK.
• Scala runs need Java 17, sbt, synchronized WIT, the three generated guest-runtime role WASMs under sdks/scala/{sbt/src/main/resources,mill/resources}/golem/wasm/, and the Scala SDK published locally for the versions used by templates. The Scala plugin/template build combines those resources into its application-local .generated/agent_guest.wasm; there is no generated sdks/scala/agent_guest.wasm. Follow .github/actions/run-skill-harness/action.yml for the authoritative CI setup (including the TS SDK artifacts Scala currently consumes).

Install and fast checks

cd golem-skills/tests/harness
npm install
npm run build          # lint + tsc
npm run format:check
npm test               # tests compiled into dist first

Repository policy forbids unit tests from spawning external tools or compilers. Keep schema, assertion, watcher, and driver parsing logic in harness unit tests; validate real agent CLIs, Golem, Docker services, and language compilers through harness scenarios/CI.

Run scenarios

All relative CLI paths are resolved from the process current directory, not from the location of run.ts. Normally run from golem-skills/tests/harness/:

npx tsx src/run.ts --dry-run --scenario <name>
npx tsx src/run.ts --agent claude-code --language rust --scenario <name>

Current options:

--workspace is a root override, not an exact reusable scenario directory. Workspaces are <root>/<run-uuid>/<scenario>/<language>/ (retries add subdirectories) and are retained. --resume-from still creates a new run hierarchy; it skips earlier scenario steps rather than reopening an old workspace.

The server is restarted between scenarios and retries with the same clean data directory. The harness refuses a pre-existing healthy server on the selected router port.

Current scenario schema

name: example # required
languageAgnostic: false # optional; once per agent on first selected language
settings:
  timeout_per_subprompt: 1800
  golem_server:
    router_port: 9881 # overridden by GOLEM_ROUTER_PORT
    custom_request_port: 9006 # exported to scenario commands
  cleanup: true # accepted; unique workspaces make cleanup unnecessary
prerequisites:
  env: { EXTRA_VAR: "value" }
  services: [postgres] # postgres|mysql|ignite|openai-mock
skip_if: { agent: codex, language: ts, os: linux }
steps:
  - id: create
    create_project: { name: test-app }
    verify:
      build: true
      deploy: true
      expectedFiles: [test-app/golem.yaml]
finally: [] # same step schema; best-effort cleanup

Every step has exactly one action:

• prompt: string or language map. Supports expectedSkills, allowedExtraSkills, strictSkillMatch, and continueSession.
• create_project: { name, presets? }; presets may be language-specific.
• shell: { command, args?, cwd? }, with args optionally language-specific. It executes directly, not through a shell.
• invoke / invoke_json: { agent, method, args? }; all three fields may be language-specific.
• trigger: fire-and-forget form of invocation.
• create_agent: { name, env?, config? }; delete_agent: { name }.
• http: { url, method?, headers?, body? }; methods include GET, POST, PUT, DELETE, PATCH, OPTIONS.
• get_agent_type: { name }; list_agent_types: {}.
• check_file: { path }, resolved from the discovered Golem app directory.
• mcp_call: { url, method, params? }, using Streamable HTTP session initialization.
• sleep: seconds.

Common optional step fields are id, timeout, expect, retry: { attempts, delay }, only_if, skip_if, and verify: { build?, deploy?, expectedFiles? }. allowedExtraSkills or strictSkillMatch requires expectedSkills. Without an explicit extra-skill restriction, additional activations do not fail the step.

Language-conditional values use { ts, effect, rust, scala, moonbit } maps. Supported conditional fields include prompts, skill lists, verify, expect, project presets, shell args, HTTP body, and invocation/trigger agent, method and args. A missing selected-language entry resolves to absent and may fail later validation/execution; include every language the scenario runs. The default Effect agent is Counter, not CounterAgent.

Use source-language method names: snake_case for Rust/MoonBit and camelCase for TypeScript/Scala. Use language-specific argument syntax where composite values differ.

Assertions

expect supports:

• process/file output: exit_code, stdout_contains, stdout_not_contains, stdout_matches
• HTTP/MCP: status, body_contains, body_matches, header_contains
• JSON bodies: body_json
• unwrapped invoke_json results: result_json

body_json and result_json are arrays of { path, equals?, equals_unordered?, contains? }. equals_unordered compares top-level arrays without order. Regexes are JavaScript RegExp, validated by dry-run; avoid PCRE-only syntax.

Variables and paths

Text substitution recognizes {{workspace}}, {{scenario}}, {{agent}}, {{language}}, {{custom_request_port}}, {{mcp_port}}, plus service variables {{postgres_url}}, {{mysql_url}}, {{ignite_url}}, and {{openai_mock_url}} when started. Substitution applies only to fields implemented in substituteStepVariables; do not assume arbitrary YAML strings are expanded.

shell.cwd is relative to the scenario workspace. Most Golem actions discover golem.yaml at the workspace root or one immediate child and run from that app directory. check_file.path is app-relative. expectedFiles verification is workspace-relative, matching existing scenarios (which normally include the project directory).

Skill activation and sessions

Native driver events are preferred when available; otherwise watcher events and changed atimes are combined. Activations accumulate across follow-ups. The first prompt and any continueSession: false prompt reset the activation session. Because activation detection proves that a skill was opened/reported, not that its advice was followed, pair expectedSkills with build, deploy, file, invocation, HTTP, or JSON assertions.

Reports and CI

Normal runs write per-scenario JSON (<agent>-<language>-<scenario>.json), summary.json, and report.html; --ctrf adds CTRF JSON, and GITHUB_STEP_SUMMARY receives a summary. Merge mode writes merged-summary.json, scenario-reports.json, and report.html.

• .github/workflows/ci.yaml runs harness build, formatting, and unit tests when golem-skills/** changes.
• .github/workflows/skill-harness.yaml is manually dispatched (scheduled runs are disabled), runs provider/language matrices, merges artifacts, and publishes the report.
• .github/actions/run-skill-harness/action.yml defines integration prerequisites and executes the harness from an isolated /tmp/harness-run while GOLEM_PATH points at the checkout.

Debugging workflow

1. Run --dry-run --scenario <name> to catch current-schema and regex errors.
2. Run one agent/language/scenario with explicit timeouts; inspect the retained workspace and per-step JSON.
3. Distinguish step timeout from idle timeout. Whole-scenario retry occurs only for idle timeout; per-step retry handles any step failure.
4. For missed activation, check native tool events first, then .agents/skills, atime behavior, and Linux inotifywait. Do not restore stale .claude/skills duplication.
5. For stale skill behavior, confirm which release/debug golem was selected and rebuild it.
6. For Golem failures, verify the selected ports are free and inspect server output. For service failures, inspect Docker availability/container logs.
7. For Scala, reproduce the composite action's Java/sbt/WIT/base-image/local-publish preparation before blaming the scenario.