Run Guardian Prod Benchmarks

Run Guardian Prod Benchmarks

Read the current source of truth at the start of every task:

• benchmarks/prod-server/README.md
• scripts/run-prod-benchmark-ecs.sh
• benchmarks/prod-server/profiles/*.toml
• benchmarks/prod-server/src/config.rs
• benchmarks/prod-server/src/cleanup.rs
• benchmarks/prod-server/reports/20260408-prod-benchmark-report.md

If the task involves changing the live GUARDIAN deployment or verifying the active AWS shape before benchmarking, also use deploy-guardian-aws.

Trust these sources in this order:

1. scripts/run-prod-benchmark-ecs.sh for the real execution and teardown flow
2. benchmarks/prod-server/profiles/*.toml for supported scenarios
3. benchmarks/prod-server/src/*.rs for benchmark behavior and cleanup semantics
4. benchmarks/prod-server/reports/20260408-prod-benchmark-report.md for the reporting style and reference numbers

Scope

This skill is only for the current distributed ECS benchmark path:

• gRPC only
• prod GUARDIAN endpoint
• ECS/Fargate worker clients in us-east-1
• benchmark-owned logical cleanup through ECS exec
• ECDSA burst
• ECDSA mixed 1 push_delta : 4 get_state
• Falcon mixed 1 push_delta : 4 get_state
• Falcon+ECDSA mixed 1 push_delta : 4 get_state

Do not revive or recommend:

• local single-process benchmark runs
• local direct-SQL cleanup
• file-backed seed caches
• deleted smoke or baseline profiles

Workflow

1. Confirm Target Readiness

Before the first benchmark in a session:

1. Verify the live endpoint and benchmark assumptions.
2. If the deployment shape matters, inspect it first with deploy-guardian-aws.
3. Verify public health at minimum:

   curl -fsS https://guardian.openzeppelin.com/
   curl -fsS https://guardian.openzeppelin.com/pubkey
   

4. If reproducing the April 2026 reference runs, confirm the deployment is still close to:
   • 1 ECS server task
   • ARM64
   • 2 vCPU / 4 GB
   • RDS Proxy enabled

2. Run Local Preflight

Run the toolchain and auth checks before any benchmark run:

set -a && source .env && set +a
aws sts get-caller-identity
docker info
command -v jq
command -v session-manager-plugin

Then run the benchmark preflight for the intended profile:

cargo run --manifest-path benchmarks/prod-server/Cargo.toml -- \
  preflight --profile benchmarks/prod-server/profiles/<profile>.toml

If preflight fails on AWS auth, refresh SSO before continuing.

3. Choose the Scenario

Read references/scenario-matrix.md when selecting or explaining a run.

Use these profiles:

• ecdsa-burst-scale.toml
• ecdsa-mixed-burst-scale.toml
• falcon-mixed-burst-scale.toml
• falcon-ecdsa-mixed-burst-scale.toml

Default worker count for the reference runs is 16.

4. Execute the Distributed ECS Run

Use the repository script, not ad hoc ECS commands:

./scripts/run-prod-benchmark-ecs.sh \
  --profile benchmarks/prod-server/profiles/<profile>.toml \
  --workers 16

Important rules:

• Let the script build the benchmark-runner image unless you are intentionally reusing an existing image with --image-uri.
• Only use --no-cleanup for diagnosis. If you use it, run cleanup immediately afterward.
• Only use --keep-image or --keep-task-definition when you explicitly need post-run inspection.
• Treat missing worker artifacts or 502 shard failures as benchmark caveats that must be reported.

The script already:

• builds and pushes the benchmark image
• creates a temporary task definition
• launches ephemeral Fargate workers
• collects worker artifacts from CloudWatch logs
• aggregates results locally
• runs ECS-exec SQL purge
• tears down temporary ECS and ECR resources on exit

5. Validate Run Outputs and Cleanup

After aggregation, inspect the generated artifacts in benchmarks/prod-server/reports/<run-id>/:

• run-report.json
• summary.md
• cleanup-manifest.json
• canonicalization-samples.json (when canonicalization.sample_rate > 0)

Confirm all of these before finishing:

1. cleanup-manifest.json shows complete
2. the report has all, ecdsa, and/or falcon scopes as expected
3. the run notes any missing shard artifacts
4. the public endpoint still passes / and /pubkey

6. Interpret the Results

Read references/reporting.md before writing conclusions.

Use these rules:

• push_delta/s is the main metric
• for canonicalization A/B comparisons, the accepted→canonical wait in the canonicalization report section is the main metric instead
• mixed runs should report both push_delta/s and get_state/s
• latency should always include at least p50, p95, p99, and max
• if failures are mostly state_conflict, the account pool shape is capping admitted writes, not necessarily the server
• if a run recovered from fewer than all shards, call that out explicitly
• treat 500 TPS as a sizing reference for required GUARDIAN tasks, not as a requirement that one task must absorb the entire network

For comparison against the benchmark we already ran, read references/reference-results.md.

Output Shape

When asked to run or summarize benchmarks, report:

• the exact profile used
• the number and size of ECS worker tasks
• the server shape under test
• the number of users/accounts
• signer distribution
• throughput results
• latency results
• cleanup status
• any shard-loss, conflict, or measurement caveats
• the implied GUARDIAN task count for a 500 TPS reference target when relevant

When writing a benchmark report, follow the same structure as benchmarks/prod-server/reports/20260408-prod-benchmark-report.md:

1. goal
2. setup
3. reference capacity target
4. results summary
5. results table
6. latency section
7. interpretation