# Benchmark > Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote cloud instances, then produce a performance + memory report with charts and a raw-data appendix. Use this whenever the user runs /benchmark, or asks to benchmark / load-test / measure throughput, QPS, latency, or memory efficiency of Dragonfly or another Redis-compatible server, to compare Dragonfly vs Valkey or Redis, or to generate a benchmark report. Trigger it even when the user just describes the goal ("see how fast Dragonfly is on this box", "compare memory per key vs Valkey") without saying the word "benchmark". - Skill: `dragonflydb/benchmark` (Agent Skill, multi-file: 24 files) - Install (CLI): `npx skillmds@latest add dragonflydb/benchmark` - Raw SKILL.md: https://api.skillmd.com/api/skills/dragonflydb/benchmark/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: dragonflydb (https://skillmd.com/u/dragonflydb) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/dragonflydb/benchmark --- # Benchmark Dragonfly Drive a benchmark of Dragonfly (optionally side-by-side with Valkey or Redis) on local or remote instances, collect throughput / latency / memory numbers, and assemble a report that mirrors the team's published format: a narrative TLDR, per-instance metric tables, charts, and an appendix with the exact commands and raw output so anyone can reproduce it. The whole point of the report is **trust**: a reader should be able to see the headline numbers, understand the conditions they were measured under, and re-run the exact commands themselves. Everything below serves that goal. ## When you start If the user passed a free-form spec in `$ARGUMENTS`, parse what you can from it (IPs, use-cases, comparison targets) and only ask about what's missing. The work splits into clear phases — walk through them in order, but stay flexible: the user may already have servers running, or may only want the report half (handed raw output to format). Read what they're actually asking for. Phases: (1) gather spec → (2) prepare instances → (3) network tuning → (4) start monitoring → (5) run workloads → (6) extract metrics, charts, report. The two reference files carry the operational detail — read them before acting, not from memory: - `references/running-benchmarks.md` — how to prep instances, start servers, drive `dfly_bench` and `memtier_benchmark`, run Valkey/Redis for comparison, apply network tuning, and measure memory. **Read this before Phase 2.** - `references/report-template.md` — the exact report structure, with an example. **Read this before Phase 5.** Bundled `scripts/` (reuse these instead of re-improvising the same loops — they encode the safety discipline learned the hard way): - `bench_server.sh start|stop|status [maxmem] [port]` — launch a server tracking its PID; **stop kills by PID and verifies the process is gone + port free** (tmux kill-session does not reap it). - `monitor_fill.sh [session] [abort_pct]` — the live RSS guard: polls `used_memory_rss`, prints a line per sample, aborts the write if RSS crosses the threshold (default 92%). Its output is also the input to `plot_fill.py`. - `capture_result.sh [label]` — saves the run summary + INFO memory + **OS-level `free` snapshot** (`host_memory.txt`) and exits non-zero if there were any eviction errors (enforces the zero-error rule). The OS snapshot matters: `used_memory_rss` and host free can diverge significantly — e.g. with `MIMALLOC_ALLOW_LARGE_OS_PAGES=1` (default), mimalloc holds large-page arenas that the OS counts as used but INFO doesn't fully reflect, making the machine appear near-OOM while RSS looks fine. - `collect_metrics.py` / `make_charts.py` — metrics + charts from dfly_bench JSON (GET/SET workloads). - `plot_fill.py` — memory-fill chart from `monitor_fill.sh` output, for `--command` workloads (e.g. SETEX) that produce no JSON time series. Every benchmark output directory must include a `commands_audit.log` artifact with the exact command lines used for setup, server launch, writes, reads, captures, diagnostics, and chart/report generation. Raw data without the commands that produced it is not reproducible. **Run-control scripts are mandatory for benchmark phases.** For Phase 5 runs, use the bundled scripts as the default API: - start/stop/status servers with `scripts/bench_server.sh`; - monitor long write fills with `scripts/monitor_fill.sh`; - capture write results and enforce the zero-error rule with `scripts/capture_result.sh`; - generate metrics/charts with `scripts/collect_metrics.py`, `scripts/make_charts.py`, and `scripts/plot_fill.py`. Inline SSH is fine for inspection, package setup, one-off diagnostics, and bounded foreground probes, but not as a replacement for the run-control scripts in the main benchmark path. If a script is not usable for a run, document the reason in `commands_audit.log` and in the report appendix before using a manual equivalent. ## Phase 1 — Gather the benchmark spec (interactive) Use `AskUserQuestion` to pin down anything not already given. You need: 1. **Where it runs.** Local, or remote instances? For remote, get the **server** host/IP and the **client** host/IP (the load generator should run on a separate machine from the server — co-locating them steals CPU and corrupts the numbers). Get the SSH user / key if not obvious. On cloud instances, **always use the private/internal IP** (VPC-internal) for both SSH and as the load generator's `-h` target — the public IP routes through the internet gateway, adding latency and egress cost, and the result measures the gateway rather than the server. 2. **Use-cases.** Default menu: simple strings (SETEX writes + GET reads), pipelined / peak-throughput, memory efficiency. Always also ask whether there is a **specific focus** (a particular command, data type, key distribution, value size, or scenario) — the canned use-cases are a starting point, not a cage. 3. **Comparison targets.** Dragonfly only, or also Valkey / Redis? Which versions? 4. **Instance specs** (for the report's conditions section): instance type, CPU count, OS/kernel. Read these directly from the box at the start of Phase 2 rather than asking the user. 5. **Load parameters** if the user has opinions: value size (`-d`), key space (`--key_maximum`), connections (`-c`), pipeline depth, test duration. Pick sensible defaults (see the reference) and tell the user what you chose. **Sizing the key count is the consequential decision** — see the reference's "Sizing the load" section. The dataset must fill a large fraction of instance RAM (benchmarking a near-empty instance is wrong) but stay clear of OOM (~60–75% of RAM, headroom to spare). `--key_maximum` bounds the footprint, so compute it from instance RAM ÷ estimated bytes-per-entry, calibrating with a short pilot write if unsure. Summarize the resolved spec back to the user in a short block before running anything, so a wrong assumption gets caught before machines spin. ## Phase 2 — Prepare the instances - Record instance specs (`nproc`, `uname -r`, cloud metadata) — these go in the report's conditions section. Use the **private IP** when connecting between instances (see Phase 1). - A remote instance usually already has `~/projects/dragonfly`. If not, clone it there. Build a **release** binary (`build-opt/dragonfly`, `build-opt/dfly_bench`) unless the user wants a published release downloaded instead — follow their instruction on build-vs-download. - For comparison targets, pull the Valkey/Redis docker image on the server box. - Verify everything launches and is reachable from the client box (a quick `redis-cli -h ping`) before committing to a long run. Run each step individually over SSH — do not chain launch, polling, capture, and the next phase into one command. If a tool call is interrupted, remote work keeps running but goes uncaptured. Start the server; start the write; monitor it; capture; start reads — one bounded step at a time. ## Phase 3 — Network tuning (optional, remote only) High-throughput numbers on cloud hardware depend heavily on IRQ/SMP affinity tuning. If the user wants maximal numbers, apply the tuning described in the reference and **record exactly what you did** — it belongs in the appendix and materially affects the results. If skipped, say so in the report; untuned and tuned numbers are not comparable. ## Phase 4 — Start the monitoring stack (server machine) Start the local Prometheus + Grafana stack on the **server** machine so you get continuous, fine-grained time-series (server CPU, memory, command rates, host metrics) spanning the whole benchmark — richer than the load generator's own output and the source of the most convincing report charts. ```bash ssh 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose up -d' # when benchmarking Valkey/Redis, also bring up the redis exporter: ssh 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose --profile redis up -d' ``` Prometheus scrapes at 1s. **Always run servers on port 6380** (Dragonfly, Valkey, and Redis alike). The bundled `prometheus.yml` ships pointing at `6379`, so before bringing the stack up, fix the scrape targets: ```bash sed -i 's/:6379/:6380/g' tools/local/monitoring/prometheus/prometheus.yml ``` (This is a workaround for a misconfigured default — if you have repo write access, commit the corrected file once so future runs start clean.) Restart Prometheus if it was already running, then verify targets are `UP` at `http://:9090/targets` before starting the load — a silently-down scrape job means no data for that window. Start monitoring **before** the workloads and leave it running across all runs, so a single continuous window covers everything. See the reference's monitoring section for what to read from Grafana (dashboards are pre-provisioned) and how to pull panels into the report. Record the benchmark start/end timestamps — they let you crop the Grafana window to exactly the run. ## Phase 5 — Run the workloads For each (use-case × datastore) combination, see the reference for the precise commands. The shape is always: **write → memory snapshot → read**. Use the private IP as the load generator's `-h` target. Pass `--json_out_file` to `dfly_bench` (and `--json-out-file` to memtier) when the workload supports it — charts and headline numbers come from these files. `dfly_bench --command` workloads (e.g. `SETEX __key__ 10000 __data__`) do not produce JSON; save full stdout and chart the `monitor_fill.sh` output instead. Drive the run with the bundled scripts: `bench_server.sh start`, write in a client tmux session, `monitor_fill.sh` alongside (live RSS guard), then `capture_result.sh` to save results. `capture_result.sh` enforces the **zero-error rule** — any eviction/OOM error response makes the run invalid; discard it, cut `--key_maximum` ~10–15%, and re-run. See the preamble and the reference's "Sizing the load" section for the full rule. **Prefill writes must use `--pipeline 30`.** Without pipelining, the client stalls waiting for each reply and becomes the bottleneck before the server is saturated — especially at high key counts. Use `--pipeline 30` for all write (fill) phases. Drop to `--pipeline 1` (or omit it) only for latency-honest read benchmarks where you want honest per-request numbers. Initialize `commands_audit.log` at the start and append each command as it runs (see the preamble for the requirement). ## Phase 6 — Extract metrics, build charts, assemble the report 1. **Headline metrics + memory.** Run the bundled extractor over the JSON files to pull QPS / P99 and compute bytes-per-entry (both `used_memory`- and `used_memory_rss`-based, per the user's choice to report both): ```bash python3 .claude/skills/benchmark/scripts/collect_metrics.py \ --bench-json [--bench-json ...] \ --info --dbsize \ --label "Dragonfly m7g.2xlarge" ``` It prints a metrics block (and JSON) you can drop straight into the tables. 2. **Charts.** Generate QPS-over-time PNGs from the time series (this is what visually shows Dragonfly's smooth throughput vs a competitor's degradation): ```bash python3 .claude/skills/benchmark/scripts/make_charts.py \ --series "Dragonfly=df_write.json" --series "Valkey=valkey_write.json" \ --metric qps --title "Write throughput, m7g.2xlarge" \ --out charts/write_m7g2xl.png ``` Embed the PNGs in the report with relative paths. **For `--command` workloads (e.g. SETEX) there is no dfly_bench JSON** — use `plot_fill.py` instead to chart memory-fill over time from the `monitor_fill.sh` output: ```bash python3 .claude/skills/benchmark/scripts/plot_fill.py \ --series "prebuilt=prebuilt/poll.txt" --series "release=release/poll.txt" \ --interval 10 --ram-gib 30 --out charts/memory_fill.png ``` 3. **Report.** Follow `references/report-template.md` exactly. Write a TLDR with the 3–5 genuinely important takeaways (scaling behavior, memory efficiency, any hardware surprise), then per-instance tables, then the appendix. The TLDR is interpretation, not a restatement of the tables — say what the numbers *mean*. Only claim what the data supports. Save the report where the user asks (default: `benchmark--.md` in the repo root) and tell them the path plus a one-line summary of the headline result. ## Cleanup If you started servers, the monitoring stack, or comparison containers on remote instances, offer to stop them (`docker compose down` in the monitoring dir). If you applied network tuning that changes machine state, tell the user what would need reverting. Don't tear anything down without asking — they may want to inspect or re-run.