Scrapbot Testing

Scrapbot Testing

Use this skill whenever Scrapbot must build, load projects, analyze generated Luau types, execute ECS systems, render through current backends, or produce a visually verifiable artifact.

Default Verification

Run the normal suite first:

mise test
git diff --check

mise test currently builds the CLI, checks src/scrapbot, runs all Odin package tests with -all-packages, checks the CLI version, validates examples/minimal, and runs the null backend.

In a multi-agent workflow, individual agents run the narrow checks for their owned files. The integration owner runs mise test, generated-example checks, documentation builds, and any required WGPU verification against the combined branch.

For narrower loops, select packages by ownership:

odin build src/scrapbot_cli -out:bin/scrapbot
odin check src/scrapbot -no-entry-point
odin test src/scrapbot/resources
odin test src/scrapbot/ecs
odin test src/scrapbot/render
bin/scrapbot help run

mise build writes the optimized executable to bin/scrapbot. mise build-dev writes the fast-compiling executable to bin/scrapbot-dev; after a development build, run bin/scrapbot-dev, not a potentially stale bin/scrapbot. mise scrapbot -- [args...] builds and runs the optimized executable in one task. Before a framegrab, confirm that the executable you launch is the one produced by the preceding build.

Use mise test for src/scrapbot/script or the full package tree because Luau tests require the native linker flags from mise.toml.

External Development Fixtures

Run mise setup for a complete contributor checkout. It installs pinned tools, initializes Git submodules, downloads external development fixtures, and configures the tracked hooks. mise setup-assets installs only the external fixtures; mise check-assets verifies their byte lengths and SHA-256 hashes without network access.

The manifest and license notes live under tests/fixtures/external/; downloaded bytes live under its ignored downloads/ directory. Keep the ordinary mise test suite independent of these heavyweight or license-constrained files. An explicit integration task may require them, but it must fail with a direct mise setup-assets instruction when they are absent. Never commit downloaded fixtures or copy them into distributable examples, project packages, or releases.

For real-world glTF importer work, use the pinned Khronos Damaged Helmet fixture:

mise test-gltf
mise test-gltf-gpu

test-gltf imports and checks the model plus its product metadata. test-gltf-gpu additionally produces a bounded headless WGPU framegrab in the platform temporary directory. Run mise setup-assets first if the fixture check tells you it is absent.

Use examples/gltf-showcase for persistent interactive or editor testing of a representative imported model. Its ignored assets/DamagedHelmet.glb placement is maintained by mise setup-assets; do not commit or redistribute it.

Use the pinned Khronos Sponza bundle and its separate CC0 Kloppenheim 01 Pure Sky HDRI when a change needs a substantially larger external-file glTF graph, many generated materials/textures, imported outdoor IBL, model-root shadow inheritance, screen-space ambient-occlusion contacts, or a real architectural clustered-lighting workload:

mise test-sponza
mise test-sponza-gpu

These are explicit heavyweight lanes, not ordinary mise test dependencies. test-sponza deletes and freshly rebuilds only the ignored examples/sponza/.scrapbot cache, then asserts the pinned import shape. test-sponza-gpu also requires 103 renderables/draw batches, clustered point lights, shadow visibility, and writes a bounded framegrab in the platform temporary directory. The source bundle and its license are installed into ignored examples/sponza/assets/ state by mise setup-assets; never commit or redistribute them.

Structured Diagnostics

Prefer --json for agent-driven CLI checks:

bin/scrapbot check examples/minimal --json
bin/scrapbot build examples/minimal --json
bin/scrapbot run examples/minimal --backend null --headless --no-hot-reload --frames 1 --json

JSON mode emits one versioned document on stdout. Use ok, diagnostic code, and documented result fields for assertions and branching. Treat diagnostic messages as human-readable context; do not match their exact text. Check schema_version before consuming the envelope, and fall back to human output only when the command has no structured mode.

Keep run bounded with --frames. Structured success confirms command and runtime behavior, but renderer changes still require the WGPU smoke or framegrab checks described below.

For CPU/RAM growth investigations, request the opt-in structured runtime report:

bin/scrapbot run examples/ecs-showcase --backend null --headless --no-hot-reload --frames 10000 --runtime-stats --json

Compare allocated-slot fields across runtime_stats.early_storage, late_storage, and final_storage; compare allocator_early_bytes with allocator_late_bytes; check allocator_final_bytes; and inspect the early/late frame timing ratio. Ignore live_entities when the workload intentionally oscillates. These are engine-owned signals; direct Luau, SDL, WGPU, driver, GPU, and OS allocations require separate tooling.

Bounded Render Profiles

Use scrapbot profile for renderer performance investigations and before/after evidence:

bin/scrapbot profile examples/sponza \
  --warmup 60 \
  --frames 240 \
  --resolution 1920x1080 \
  --capture-range 100:104 \
  --out /tmp/scrapbot-sponza-profile \
  --json

The command writes:

• profile.json: raw per-frame CPU, GPU-pass, resolution, viewport, frame-local counter_deltas, structured pass workload, and complete renderer snapshots plus median, p95, and maximum summaries.
• overview.png: the final measurement-pass frame.
• frames/frame-NNNNNN.png: an optional narrow lossless sequence from a fresh replay.

Follow these rules:

1. Compare the same executable, machine, project state, resolution, culling mode, editor state, warmup, and frame count.
2. Check physical dimensions, pixel density, viewport, and shaded pixels before interpreting GPU time. HiDPI cost follows physical pixels.
3. Compare cpu_active_ms for active engine work. It intentionally excludes profiler readback waits and presentation idle.
4. Compare GPU distributions only across rows where gpu_timing_valid is true.
5. Inspect raw rows and counters around p95/worst spikes. A summary alone cannot distinguish upload churn, visibility changes, or one expensive pass.
6. Read the matching workload entry before changing a pass. Target dimensions, pass count, workgroups/invocations, draws/instances, and fixed sample budgets distinguish pixel scaling from repeated dispatch, geometry pressure, or shader complexity.
7. Keep --capture-range tight. Captures run in a second replay so pixel readback does not alter telemetry.
8. Use --ui-script with --editor when the workload is an editor interaction. The same script is replayed in both passes.
9. Treat results as same-machine evidence. Do not establish portable absolute thresholds from one adapter.

Summarize one report or compare two compatible reports with:

mise profile-analyze -- /tmp/before/profile.json
mise profile-analyze -- /tmp/before/profile.json /tmp/after/profile.json

The comparator checks backend, adapter, dimensions, density, viewport, and shaded pixels before reporting pass and counter deltas. Exit status 2 means the reports are not comparable.

Separate fixed overhead from pixel-scaled work with:

mise profile-sweep -- examples/sponza \
  --binary bin/scrapbot \
  --warmup 60 \
  --frames 240 \
  --out /tmp/sponza-resolution-sweep

The default matrix is 960×540, 1280×720, and 1920×1080. Repeat --resolution WIDTHxHEIGHT to use an explicit bounded matrix. Read sweep.json, not the aligned human table, from automation.

Use examples/minimal to prove the pipeline and artifact contract. Use examples/ecs-showcase for retained instances and ordinary engine features. Use examples/sponza only after mise setup-assets when the question requires a representative architectural workload. Use examples/virtual-wilds after asset setup when the question requires real photogrammetry, generated cluster hierarchies, bounded virtual-geometry residency, or streaming under camera motion.

Temporal Render Verification

A terminal frame can hide errors that appear only while state changes. Treat consecutive capture as required—not optional—for geometry streaming, residency changes, LOD transitions, temporal history, animation, hot reload, camera cuts, and any reported flicker, z-fighting, or transient topology corruption.

Capture at least five consecutive relevant frames from one deterministic replay:

bin/scrapbot profile <project> \
  --warmup 60 \
  --frames 120 \
  --capture-range 60:67 \
  --out /tmp/scrapbot-temporal-check \
  --json

Then:

1. Inspect every captured PNG in order, not only overview.png or the last frame.
2. Read the same frame rows in profile.json, including render state and counter_deltas, so a visible discontinuity can be tied to an upload, eviction, rebuild, camera cut, or history reset.
3. Capture the real moving workload first. If the cause remains ambiguous, repeat with a fixed camera and the most discriminating view: depth for topology/z-fighting, world normals for vertex or winding errors, LOD for object-level selection, or Virtual Geometry for the resident cluster frontier.
4. For a static camera, treat unexplained pixel changes as a failure. Expected refinement must remain crack-free and respect its projected-error bound; page admission is not permission for popping triangles, invalid indices, or depth-scale changes.
5. Preserve the smallest failing consecutive range and report its artifact directory. Do not call temporal rendering correct from process success, nonblank pixels, or one visually plausible endpoint.

When the issue is purely static and no renderer state can change after the tested frame, one framegrab remains sufficient.

Use the reusable sequence gate when the scenario should remain in the suite:

node tools/test_render_sequence.mjs \
  --project examples/virtual-wilds \
  --warmup 120 \
  --frames 120 \
  --capture-range 56:63 \
  --stable-frontier \
  --out /tmp/scrapbot-virtual-wilds-sequence

--stable-frontier requires one settled projected-error policy, no feedback overflow or page-read failure, no active virtual-geometry admission transition, and no group upload or eviction inside the captured range. The runner preserves the profile, every PNG, and sequence-manifest.json.

Use --require-transition for a transition-focused sequence. It requires at least one active virtual-geometry admission handoff, a visible blended cluster inside the range, and a settled zero-transition endpoint. Do not combine it with --stable-frontier. Pass --golden-dir and --minimum-psnr for a platform-qualified image baseline; do not use exact pixel equality across GPU vendors.

Historical GPU benchmark bundles

Use the benchmark orchestrator when a renderer change needs trend evidence across representative workloads and pixel counts:

mise setup-assets
mise gpu-benchmarks -- --out /tmp/scrapbot-gpu-benchmarks

It profiles minimal, ecs-showcase, and sponza at 960×540, 1280×720, and 1920×1080. Pass --without-sponza for a smaller local run without the external fixture.

Compare complete bundles with:

mise gpu-benchmark-compare -- \
  /tmp/baseline \
  /tmp/candidate \
  /tmp/comparison

Read summary.md and comparison/comparison.md for orientation. Use manifest.json, sweep.json, profile.json, and comparison.json for automation or diagnosis.

The scheduled/manual GPU Benchmarks workflow retains the current Metal bundle and compares it with the previous successful same-architecture artifact. Never copy deltas across incompatible adapters, dimensions, density, viewports, shaded pixels, timestamp support, or culling modes. Never turn one runner's timings into portable hard thresholds. A suspicious history point is a prompt for a controlled same-machine before/after reproduction.

Choose An Example

• Use examples/minimal for fast CLI, project loading, scheduling, Luau/Odin integration, null backend, and basic WGPU smoke tests.
• Use examples/ecs-showcase for geometry, materials, render reconciliation, batching, lighting, lifecycle-heavy ECS behavior, and visual renderer changes.
• Use examples/sponza for explicit heavyweight external-file glTF, generated-material, architectural-shadow, and clustered-lighting integration after mise setup-assets.
• Use examples/virtual-wilds for explicit heavyweight photogrammetry import, generated LOD/cluster products, virtual-geometry page demand, and temporal streaming after mise setup-assets. Use mise test-virtual-wilds-gpu for its bounded consecutive-frame profile and stable-frontier assertion.
• Use examples/ui-showcase for retained ECS UI hierarchy, box-model layout, horizontal/vertical stacks, smooth clipped scroll areas, SDF-rounded backgrounds, pointer-styled buttons, MTSDF text, and overlay/framegrab changes. Use --ui-script for deterministic headless hover, active, focus, typing, scrolling, and editor workflows; do not rely on OS pointer automation.

Validate an example with:

bin/scrapbot check examples/minimal
bin/scrapbot check examples/ecs-showcase
bin/scrapbot check examples/ui-showcase
bin/scrapbot run examples/minimal --backend null --headless --no-hot-reload --frames 1

scrapbot check also regenerates .scrapbot/types/scrapbot.d.luau. After changing Luau APIs or component schemas, run it for every affected example and inspect the generated output. It is ignored engine state; do not hand-edit or commit it.

WGPU Window Smoke

Windowed WGPU opens an SDL3 window and may require graphics-service approval on macOS:

bin/scrapbot run examples/minimal --backend wgpu --window --frames 3

Use --frames for automated smoke checks so the command returns. Without --frames, windowed mode runs until the window closes.

Headless WGPU Framegrabs

Headless WGPU requests a surface-free native adapter and renders the same resource-backed ECS path into an offscreen texture. A run without --framegrab performs no pixel readback; adding --framegrab reads the final frame and writes a PNG. This path creates neither SDL state nor an OS presentation surface, but the host or sandbox must still expose a native graphics adapter:

After changing instance storage, batching, culling, shadows, depth/Hi-Z, geometry LODs, indirect drawing, GPU timestamps/counters, postprocessing, UI geometry retention, or WGPU bind layouts, run mise test-gpu. It exercises a greater-than-64-batch scene with more than 256 instances, proves real Hi-Z rejection, validates asynchronous timing/visibility diagnostics and retained UI uploads, and compares adjacent phases of a static TAA view with a PSNR floor. Compute/CPU-reference captures must otherwise match exactly or differ only by a bounded number of one-value 8-bit channels. The gate also loads a UUID-backed scrapbot.geometry_lod fixture and requires one visible instance in each of LOD 0, 1, and 2.

Use the smaller artifact-preserving acceptance gate for CI qualification and renderer orientation:

mise test-gpu-offscreen -- --out /tmp/scrapbot-gpu-offscreen

It renders examples/minimal, compute- and CPU-culled variants of tests/fixtures/gpu-driven, and examples/pbr-materials. The gate requires nonblank PNGs, bounded compute/reference agreement, coherent structured renderer diagnostics, and the PBR visual contract. It does not impose absolute GPU timing limits.

The output directory is the debugging contract:

• manifest.json summarizes the host, outcome, GPU timings, workload counters, and artifact names.
• Each case keeps its complete Scrapbot JSON envelope, stderr log, and 1:1 PNG.
• A failed process still writes a failed manifest and preserves every artifact produced before the failure.

CI must upload this directory with if: always(). Read the manifest first, then inspect the smallest relevant PNG or stderr log. Do not rerun a failed gate blindly when its artifacts already identify adapter acquisition, structured-contract, pixel-variance, reference-diff, or PBR-contract failure.

The same task imports and renders tests/fixtures/gltf-materials, whose back-facing triangle uses a partially transparent base-color texture, glTF MASK, an explicit cutoff, and doubleSided. Preserve this fixture when changing imported material products, material uniforms/bind groups, raster culling, depth prepasses, or shadow pipelines.

bin/scrapbot run examples/minimal --backend wgpu --headless --frames 20 --json
bin/scrapbot run examples/minimal --backend wgpu --headless --frames 2 --framegrab /tmp/scrapbot-framegrab.png
bin/scrapbot run examples/ecs-showcase --backend wgpu --headless --frames 20 --framegrab /tmp/scrapbot-showcase.png
bin/scrapbot run examples/ui-showcase --backend wgpu --headless --frames 2 --framegrab /tmp/scrapbot-ui.png
bin/scrapbot run examples/ecs-showcase --backend wgpu --editor --headless --frames 20 --framegrab /tmp/scrapbot-editor.png
bin/scrapbot run examples/ecs-showcase --backend wgpu --editor --headless \
  --ui-script tests/fixtures/ui/component-picker.json \
  --ui-dump /tmp/scrapbot-component-picker-tree.json \
  --framegrab /tmp/scrapbot-component-picker.png

Use tests/fixtures/ui/component-picker.json when changing component authoring, live component mutation, registry-driven discovery, namespace grouping, panel title actions, or component-menu rendering. It proves running-mode attachment/removal, Stop-time disposal, stopped-mode membership changes, picker hover feedback, and reusable title-icon button targeting through target.part = "panel_action".

Use tests/fixtures/ui/playback-authoring.json when changing Play, Step, Stop, authoring baselines, dirty-state preservation, or world replacement. It creates an unsaved scene entity, runs a complete Play/Stop round trip, and asserts that the authored entity remains visible.

Use tests/fixtures/ui/authoring-history.json when changing Undo, Redo, Save, Revert, savepoint-based dirty state, or disk-scene restoration. It edits a scene Transform, traverses away from and back to the clean history cursor, reverts from disk, asserts the original value, and captures the complete transport group.

Use tests/fixtures/ui/resource-inspector.json when changing project-resource loading, material UUID resolution, inline resource inspection, or the resource-reference picker. It stops simulation, selects an authored material entity, opens the ECS-built resource menu, asserts known authored materials, and captures the menu tightly.

Use tests/fixtures/ui/resource-browser.json when changing the editor resource browser, resource selection, identity fields, usage reporting, or dedicated resource inspection. It stops simulation, semantically selects the resource-row occurrence of Icosphere, asserts its editable Name and Path fields, and captures the Resources panel tightly. Pair it with the structural lifecycle unit test when changing create, duplicate, rename, move, delete, Find Usage, or Undo/Redo.

Use tests/fixtures/ui/resource-to-entity-selection.json when changing editor selection ownership or inspector routing. It selects an authored resource, then the same-named scene entity, and proves that the entity inspector replaces the resource inspector.

Use tests/fixtures/ui/resource-inputs.json when changing inline resource input bindings or playback behavior. It edits one material channel through typing and another through whole-control scrubbing while simulation is running, waits through inspector refreshes, and asserts both live values.

Use tests/fixtures/ui/asset-imports.json, asset-import-inspector.json, and material-preview.json when changing embedded ECS viewports, pooled render targets, resource-preview resolution/caching, imported Texture/Model presentation, or Material preview scenes. Pair the semantic dumps with target-cropped framegrabs and inspect the structured WGPU ui_viewport_* counters. The Model fixture proves drag-orbit, the Texture fixture crosses explicit reimport, and the Material fixture proves isolated preview geometry plus Reset.

Use tests/fixtures/ui/editor-shortcuts.json when changing editor visibility, workspace, or transport command shortcuts. It drives the same explicit Play/Pause editor actions used by the platform, proves that closing a paused shell resumes playback before reopening pauses it again, and crosses both sidebar toggles. Pair it with the physical-key mapping and state-machine unit tests for command-modified transport keys, F5–F8 aliases, Stop, Step, and world replacement.

Use tests/fixtures/ui/scene-transitions.json against examples/scene-transitions when changing scene discovery, scrapbot.change_scene, world replacement, project UI lifecycle, or transition resource reuse. It clicks through A→B→A, asserts scene-specific project UI after each replacement, and captures the restored first-scene card. Pair it with scene_transition_test.odin for repeated bounded-memory swaps and failed-candidate rollback.

Use tests/fixtures/ui/editor-overlay-lifecycle.json when changing editor visibility, scene-camera overlays, retained UI vertex buffers, or UI render-pass encoding. It selects the project camera so dynamic editor-world overlay vertices exist, closes the editor, and waits through the zero-overlay transition. The WGPU run must finish without encoding a draw that lacks a bound vertex buffer.

The same 60-frame lifecycle replay is the render-membership churn guard for examples/ecs-showcase: its fountain spawns and despawns instances inside existing batches while native and Luau systems mutate transforms. Keep its structured assertion that draw-database growth remains bounded. Pair renderer extraction changes with the ECS test proving unchanged frames visit zero render instances and one dirty mutation visits exactly one.

Use tests/fixtures/ui/playback-warning.json when changing playback-mode chrome or disposable-edit messaging. It captures the complete editor root during playback so the top-bar tint, viewport frame, and status treatment can be reviewed together. Pair it with the transport unit test, which crosses transport states and asserts the exact status copy plus both playback and stopped style tokens.

Use tests/fixtures/ui/system-profiler.json when changing system registration, provenance, timing publication, or the editor profiler. It asserts the complete engine profiler topology in the ECS UI and captures the Systems panel.

Use tests/fixtures/ui/scene-hierarchy.json when changing Transform parenting, scene-tree composition, disclosure icons, hierarchy indentation, or reusable list/tree drag-and-drop. It performs center reparenting, cross-parent edge insertion, and a transformless-source reparent before capturing the complete Scene panel. Pair it with parser cycle/missing-parent tests, ECS world-transform resolution tests, the generic list drop/lander-line test, and editor parent/order Undo/Redo coverage.

Use tests/fixtures/ui/entity-clipboard.json when changing scene Copy/Cut/Paste, clipboard ownership, subtree remapping, destination validation, or structural history. It pastes a multi-selection, cuts the new roots, undoes the complete cut, and captures the Scene panel. Pair it with the unit contracts for hierarchy preservation, explicit-root selection, atomic Undo/Redo, missing-resource rejection, and play-mode Cut/Paste transfer. tests/fixtures/ui/entity-actions.json is the companion playback structural-order replay: authored Duplicate followed by Delete must stage two operations and survive Stop as two authoring-history entries.

Use tests/fixtures/ui/ui-performance.json for repeatable editor-UI performance comparisons. It selects the Sun inspector, sustains a numeric scrub across 30 input frames, and captures the Systems panel after the rolling profiler publishes. Pair it with --ui-dump, then read __scrapbot_editor_system_time_2 from the dump for the scrapbot.ui average. Treat timings as same-machine before/after evidence; the deterministic hierarchy and no-refresh-during-scrub tests enforce the underlying retained traversal contract in CI.

Use tests/fixtures/ui/clustered-lights-paused.json when changing Hi-Z occlusion, editor viewport visibility, or paused-frame rendering. It pauses the animated Cluster Cathedral, waits for the retained depth pyramid to become reusable, and captures the world viewport. The mise test-gpu gate asserts that unsafe large near-field architectural bounds bypass Hi-Z while eligible hidden instances are still rejected.

Use tests/fixtures/ui/clustered-lights-stopped.json when changing authored/runtime entity ownership or stopped-world presentation. It proves the authored cathedral architecture remains rendered and editable after Stop while its disposable runtime lights are absent.

Use tests/fixtures/ui/clustered-lights-restart.json when changing Stop/Play world replacement or one-shot runtime scene construction. It stops and restarts Cluster Cathedral without reloading Luau, then proves its authored architecture survives and its runtime lights reappear. Runtime builders that must repeat after Stop should derive their guard from fresh ECS world state, such as ScrapbotTime.frame_index, rather than closure state retained by the Luau VM.

The retained UI tests instrument layout and paint only under ODIN_TEST. Keep their large-tree node and edge visit assertions intact: they are the deterministic CI guard against quadratic child discovery and redundant paint traversal, while the paint-only mutation check proves cached commands invalidate without forcing layout and the value-mutation matrix ensures already-attached component updates do not trigger structural synchronization. Renderer diagnostics must also prove independent project/editor/overlay rebuild counters: a stable domain stays at zero while another changes, and stable frames never add uploads. Do not replace these contracts with a wall-clock threshold.

Framegrabs are losslessly compressed and preserve 1:1 pixels. The complete frame remains 1280×720. When a visual question concerns one control, label, gizmo, or panel, request a top-left-origin crop instead of passing the entire frame through image inspection:

bin/scrapbot run examples/ui-showcase --backend wgpu --headless --frames 2 \
  --framegrab /tmp/scrapbot-ui-panel.png \
  --framegrab-region 40,40,560,600

On macOS, this does not initialize SDL or WindowServer. A managed sandbox may still hide every Metal adapter; the structured failure then reports that no suitable graphics adapter exists. Do not claim GPU verification from a null-backend fallback, and do not add WGPU commands to the default mise test until its workers expose a supported adapter.

Verify the generated artifact:

file /tmp/scrapbot-framegrab.png
xxd -l 16 /tmp/scrapbot-framegrab.png

Expected basics:

• file reports PNG image data, 1280 x 720, 8-bit/color RGBA.
• xxd starts with 8950 4e47 0d0a 1a0a.
• Visual inspection matches the selected fixture and changed behavior.

Use view_image only when the conclusion depends on visual inspection. Start with its default/high detail and one overview image per visual checkpoint. Do not request original detail for a complete frame by default. For a named pixel-level concern, generate the tightest useful 1:1 framegrab region and inspect that region at original detail. Reuse an existing artifact when the rendered state has not changed, and avoid emitting several near-identical screenshots in one turn.

For interactive UI bugs, prefer a semantic script plus --ui-dump. A script is a versioned JSON object with an actions array. Actions may click, hover, scroll, type, drag by delta_x/delta_y or toward a semantic destination, send a key, wait, expect, or select a capture target. Targets match UUID, entity name, or visible text and may choose a zero-based occurrence; the driver automatically reveals targets through clipped ancestor scroll areas. Use destination drags for reusable lists, trees, and other target-oriented gestures so layout changes do not invalidate pixel offsets. A target capture overrides full-frame output unless an explicit --framegrab-region was supplied. Inspect driver_action_index, driver_action, and driver_target in a failure dump before changing code.

Run tests/fixtures/ui/reflected-inspector.json against examples/ecs-showcase to verify that a runtime type-inspected Camera field absent from the public registry field list still appears, reaches project ECS state, marks stopped authoring dirty, and restores exactly through Undo. Extend its unit companion in src/scrapbot/ui/ui_test.odin when adding a new reflected field shape or specialized picker.

Use tests/fixtures/ui/typed-component-inspector.json when changing custom field types, registry editor metadata, native component schemas, or generic numeric inspector bindings. It scrolls both nested sidebars semantically, selects the native fountain emitter, asserts separate scalar settings, scrubs an opt-in Number field, and captures the exact custom-component panel. Pair it with scene parsing, Luau typed write-back, native extension compilation, and the non-draggable numeric-input unit test.

When reproducing or verifying an editor/UI interaction bug, read references/ui-diagnostics.md and follow its replay-dump-capture loop. Do not claim a visual fix from layout arithmetic or a successful process exit alone: assert the semantic state and inspect the smallest relevant 1:1 PNG.

For renderer changes, inspect the artifact for:

• Nonblank output, expected framing, and coherent depth ordering.
• Geometry topology, face winding, normals, and transforms.
• Material colors, lighting contrast, and point/directional light contributions.
• Stable layout and expected entity visibility across multiple frames.
• Complete editor chrome and a clipped live project viewport that fills all available center space when --editor is used.
• The invariant that a lit material with no ambient, directional, or point contribution renders black.
• Expected batching/resource sharing when frame statistics are relevant.

What Counts As Tested

For ordinary code/docs changes:

• mise test
• git diff --check

For renderer, geometry, material, light, camera, or render-ECS changes, also run a relevant WGPU smoke:

• Window path: --backend wgpu --window --frames 3
• Headless artifact path: --backend wgpu --headless --frames 2 --framegrab /tmp/scrapbot-framegrab.png

For framegrab or PNG writer changes, run:

• odin test src/scrapbot/render
• the headless framegrab command
• file/xxd checks
• visual inspection with view_image

For generated Luau API or schema changes, run:

• mise build
• bin/scrapbot check for every affected example
• mise luau-workspace-types
• mise test
• review of generated declaration diffs

For public ECS UI component changes, run all three example checks and verify TOML parsing, Luau query/partial mutation, native payload round-tripping, lifecycle cleanup, and editor composition. Use the scrapbot-ui-development skill for the ownership checklist.

For documentation changes, build the website from its own directory:

cd docs-website
pnpm run build

For lifecycle or suspected CPU/RAM growth changes, also run:

mise test-soak

The soak parses Scrapbot's JSON result, compares allocated storage across early/late/final checkpoints, allows at most 64 KiB of engine-allocator growth and post-teardown retention, and uses a generous 1.5x late/early frame-cost threshold. Its frame count and thresholds can be steered through SCRAPBOT_SOAK_FRAMES, SCRAPBOT_SOAK_MAX_ALLOCATOR_GROWTH, SCRAPBOT_SOAK_MAX_FINAL_ALLOCATOR_BYTES, and SCRAPBOT_SOAK_MAX_CPU_GROWTH.

mise test-sanitize runs the full Odin package tree under AddressSanitizer on Linux. It is explicitly skipped on macOS because the current Odin and Apple sanitizer runtimes are incompatible.

World Integrity And Lifecycle Reproduction

Call ecs.validate_world_integrity after the mutation under investigation when diagnosing entity, component-index, UUID-map, free-slot, active-set, dirty-queue, editor-back-reference, UI-hierarchy, or resource-handle corruption. Include ecs.format_world_integrity_failure in the assertion so the failing entity and related slot remain visible.

Tests validate every deferred command flush and playback restoration automatically. To enable the same checks in another diagnostic build, pass -define:SCRAPBOT_WORLD_INTEGRITY_CHECKS=true; do not enable it for ordinary release frames because it intentionally scans structural ownership.

scrapbot.test_seeded_editor_lifecycle_preserves_world_integrity runs 600 deterministic authoring and playback operations from seed 0x5c4a9b71d203e8f6. Preserve the seed and failing step in new assertions. When fixing a lifecycle failure, strengthen the validator or sequence at the ownership boundary rather than adding a late array guard.

Scene Persistence Reproduction

When changing Save, Revert, Undo/Redo savepoints, scene serialization, dirty candidate tracking, or authored/runtime provenance, preserve the contracts in src/scrapbot/scene_persistence_test.odin:

• test_scene_persistence_scales_candidate_work_and_preserves_value_blocks uses a 512-entity fixture and fixed seed 0x7a914e2dc6b8035f. Keep candidate-work assertions based on unique dirty UUIDs, never wall-clock duration or whole-scene entity counts.
• test_scene_persistence_write_failure_never_changes_original and test_scene_persistence_rejects_invalid_generated_toml_before_write prove the last valid file survives both writer failure and invalid generated content.
• test_scene_persistence_structural_roundtrip_covers_every_scene_component is the schema-drift guard. Extend it whenever a scene component or serialized field is added.
• test_scene_persistence_savepoints_roundtrip_through_undo_redo_and_revert crosses the editor history clean cursor and disk boundary. Keep the checked-in authoring-history.json semantic fixture as the rendered editor companion.

For exact-text assertions, change only the intended UUID block, require a byte-identical second Save, preserve untouched comments/formatting, and assert that runtime/editor entities never reach disk. Use save_scene_world_with_writer only inside package tests to inject a failure; production callers must keep the atomic writer.

Project Save Transaction Reproduction

When one Save can touch both scene and resource files, preserve the transaction contracts in src/scrapbot/project/save_transaction_test.odin and src/scrapbot/project_save_test.odin:

• Prepare and parse every candidate before the first destination changes; validate scene resource references against the complete candidate project state.
• Inject an ordinary error at every pre-commit filesystem checkpoint and require every original file plus all transaction markers, stages, and backups to be restored exactly.
• Inject a crash on both sides of the committed marker. Project recovery must roll backward before it and preserve the complete new file set after it.
• Exercise resource create, move, and delete explicitly. A move is one Delete plus one create-only Write; rollback restores the old path and removes the new path, committed recovery keeps only the new path, and create-only writes must reject existing destinations.
• Reload a jointly changed scene and resource registry, compare the authored values, and require a byte-identical repeated Save.

Do not weaken project atomicity to per-file atomic renames. Production editor and hot-reload Save paths must use save_project_world; save_project_materials and save_scene_world remain narrow wrappers for package-level compatibility and focused tests.

Notes For Future Agents

• Prefer mise test over reconstructing the suite manually.
• Prefer versioned --json output over parsing human-readable CLI output.
• Keep GPU commands out of the default suite while they require GUI/window-system approval.
• Keep mise test-gpu-offscreen separate from the default suite because it requires a real adapter; its pure manifest/result validation tests belong in mise test.
• Use /tmp for generated framegrabs and temporary test artifacts unless the user asks to keep them.
• Prefer structured diagnostics plus file/size checks before loading image pixels into the conversation.
• Preserve agent choice: use a full frame for composition, a 1:1 region for detail, and never downsample the only verification artifact.
• If a WGPU command fails in the sandbox with SDL display, XPC, or window-system errors, rerun it with approval rather than changing renderer code.
• Use enough frames to expose animated behavior, but keep automated runs bounded with --frames.
• When changing test expectations, update README.md, FDRs, TODO, or this skill if the testing contract changed.