Crabpot Perf Metrics

Purpose

Turn Crabpot/OpenClaw perf artifacts into a clear read: what changed, what is real signal, what is stale/noisy, and what action is worth taking.

When to use

• Comparing openclaw/crabpot branches such as main, crab-beta, or crab-development.
• Reading reports/crabpot-dashboard-data.json, reports/crabpot-import-loop-profile.json, reports/crabpot-runtime-profile.json, README dashboard metrics, or GitHub Actions report artifacts.
• The user asks about perf, runtime, CPU, RSS, import loop, OpenClaw import/activate, dashboard deltas, or whether a Crabpot performance change is meaningful.

Inputs

• Target repo or local checkout, usually openclaw/crabpot.
• Branches, refs, PRs, or artifact URLs to compare.
• Optional target OpenClaw labels/SHAs if the comparison crosses openclaw@latest, beta, or openclaw/openclaw@main.

Workflow

1. Establish freshness before interpreting numbers.
   • Resolve branch heads and report timestamps.
   • Compare dashboard JSON with raw profile JSON on the same branch.
   • If one branch is stale, say so first and avoid strong conclusions.
   • Prefer raw JSON over README snippets when they disagree.
2. Separate the metric families.
   • import-loop-profile: cheap cold capture loop against a tiny fixture. Good for harness/import drift, weak for production runtime claims.
   • runtime-profile: macro command timings across Crabpot analysis commands. Better trend signal for real suite cost.
   • OpenClaw lifecycle: import plus activate phases. If openClawLifecycleCount is 0, the branch dashboard does not include this signal.
3. Interpret import-loop metrics conservatively.
   • Use p50WallMs, p95WallMs, p50PluginWallDeltaMs, p95PluginWallDeltaMs, maxPluginPeakRssDeltaMb, and maxPluginCpuDeltaMsEstimate.
   • Treat 3-run p95 as "worst of three", not a stable tail latency estimate.
   • A small wall-time bump with flat plugin RSS/CPU usually means jitter or module-resolution overhead, not a memory/CPU regression.
   • Baseline-adjusted plugin deltas are more useful than raw wall/RSS/CPU values.
4. Interpret runtime profile metrics as the stronger trend.
   • Use summary.p50WallMs, summary.p95WallMs, summary.maxPeakRssMb, summary.maxCpuMsEstimate, and summary.maxHarnessHeapDeltaMb.
   • Inspect per-command medians for the source of change: fixture-inspection, compat-report-registry, contract-capture, synthetic-probe-plan, cold-import-readiness, workspace-plan, platform-probes, and import-loop-profile.
   • Broadly uniform slowdowns across target-aware commands usually point at target OpenClaw surface parsing or registry work, not one plugin fixture.
   • If RSS is flat and CPU moves only slightly, call it a wall-clock/runtime overhead change, not a memory leak.
5. Account for surface changes.
   • Different fixture counts, entrypoint counts, OpenClaw labels, or commit SHAs make the comparison partly apples-to-oranges.
   • New fixtures can raise issue/probe counts while leaving perf healthy.
   • OpenClaw main vs npm latest can improve compatibility while adding modest analysis overhead.
6. Give the user a direct read.
   • Start with the conclusion.
   • Prefer bullets and a short summary over tables unless the user asks for a table.
   • Include exact deltas and classify them: real regression, modest slowdown, noise, stale data, or missing signal.
   • Recommend one next action: refresh dashboard, add a dedicated lifecycle lane, investigate a specific command, or ignore as noise.

Useful Commands

Use gh if ghx is unavailable.

Fetch current branch artifacts from GitHub:

gh api /repos/openclaw/crabpot/contents/reports/crabpot-dashboard-data.json?ref=crab-development

Prefer a small Node or jq extraction over manual README scraping. Pull these files when available:

• reports/crabpot-dashboard-data.json
• reports/crabpot-import-loop-profile.json
• reports/crabpot-runtime-profile.json
• reports/crabpot-ci-summary.json

Outputs

• Concise perf summary in bullets.
• Freshness caveats when needed.
• Important deltas with interpretation.
• Clear owner/action recommendation, especially when lifecycle import+activate is missing from dashboard artifacts.