Experiments writing
Iron rule
Every number printed in the paper (text, table, figure, caption, footnote,
abstract) has an EVIDENCE.md row: ID | number as printed | command | log path | commit | machine | date | runs/median. Tables are generated from the data files
behind those rows (e.g. with the crbench table generator), not typed. Put
% EV:<ID> comments next to hand-placed numbers so a script can cross-check.
Procedure
- Freeze the data. Ask
experimenter for the final json/csv and the EVIDENCE
rows. Anything measured after the freeze requires a new row and a note in
DECISIONS.md.
- Decide the scope sentence for every range ("on the 9 parameter sets at ≥128-bit
estimated security"). Scope comes from the parameter/security table, generated.
- Write the section following
paper-playbook/sections/10_experiments.md.
- Generate tables (below). Mark the best cell per column (bold, optional light
colour); two-level headers with
\multicolumn; captions define every abbreviation
and symbol.
- Write the fairness statements (below) that apply.
- Write boundary/limitation remarks (below).
- Cross-check script:
crbench audit-tex paper/main.tex EVIDENCE.md (lib/bench),
or extract every number from the PDF text (pdftotext -layout) and match it to
EVIDENCE rows; unmatched numbers are errors unless they are parameters or
citations. Record the result in SUBMISSION.md.
Table set
| table |
contents |
notes |
| Parameters |
one row per set: all scheme parameters, derived values, security estimate(s) |
\resizebox{\linewidth}{!} if wide; security column mandatory |
| Main results |
per set: each configuration's time for the changed stage, total time, overheads (capacity, memory), ratio |
ratio column states "vs " in the header |
| Ablation |
each component alone vs composed; failure counts/timeouts for variants that do not work |
strongest evidence that the combination is necessary |
| Mechanism check |
model-predicted vs measured operation counts |
supports a cost model claim |
| Comparison with published work |
Method, Ref, per-stage times, total, ratio; "measured by us" vs "as reported" marker; threads; failure probability; key/communication sizes |
incomparable rows: symbol + "—" in ratio column + caption explanation |
| Client-side / communication |
client cost, key sizes, setup cost |
reviewers of transciphering/PSI/MPC ask for it |
Layout: \scriptsize, \arraystretch ≈ 0.92, \tabcolsep ≈ 2.6pt; tables at [t];
table captions above tables, figure captions below figures.
Fair-comparison statements (use the ones that apply)
- Same pipeline: "All ratios are against run in our pipeline with the
same <radix/parameters/library commit>; only differs."
- Measured vs claimed: "Rerunning the artifact of [X] on our machine reproduces
their claimed a–b× as c–d×; we report both."
- General caution before cross-paper tables: "Such comparisons must be read with
care because of differences in libraries, hardware and failure probabilities; we
mark numbers taken from the original papers with ◇."
- Metric orientation remark: "Our goal is throughput; for latency-critical
applications remains preferable."
- Parallelism remark: "All our numbers are single-threaded. Evaluating
independent instances in parallel helps every scheme equally, so single-thread
numbers compare designs, not hardware."
- Security-level parity: "Both our parameters and the baseline's are estimated
at λ bits with ; where the estimator has moved since the baseline
was published, we re-estimate both."
- Baseline strength: "We use the fastest configuration of for the
baseline (), not the default one." (Reviewers and shepherds check this.)
Boundary / limitation statements
- Exclusions: "The k instances below the claimed security level are excluded from the
headline ranges and flagged in Table ."
- Overheads: "The composed evaluator consumes 1.2–1.5× [ill.] the baseline's
memory/capacity."
- Regime where we lose: "For t ≥ 7 [ill.], [X] is faster because ...; the regimes do
not overlap." Say it before a reviewer does.
- Not applicable: "Our method requires ; for instances violating it the
implementation falls back to the baseline (log shows '')."
- Measurement noise: "Absolute times vary with machine load across sessions; all
ratios come from interleaved back-to-back runs in one session."
Measurement hygiene to report (and to follow)
- Interleave A/B runs (ABAB...) in the same session; report median and IQR/CI.
- No concurrent heavy jobs; use a bench lock; pin threads.
- Never mix numbers from different sessions in one ratio; if an earlier number is not
reproducible, retract it explicitly (and say so in the rebuttal if already quoted).
- Decryption/output verification for every run.
Output
paper/sections/experiments.tex, generated tables in paper/tables/, the
cross-check report in SUBMISSION.md.
1---2name: experiments-writing3description: Use when turning benchmark results into the evaluation section. Enforces that every printed number traces to an EVIDENCE.md row, prescribes table designs (parameters, main results, ablation, comparison), fair-comparison statements, and boundary/limitation statements.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/experiments-writing6---78# Experiments writing910## Iron rule1112**Every number printed in the paper (text, table, figure, caption, footnote,13abstract) has an `EVIDENCE.md` row**: `ID | number as printed | command | log path |14commit | machine | date | runs/median`. Tables are generated from the data files15behind those rows (e.g. with the `crbench` table generator), not typed. Put16`% EV:<ID>` comments next to hand-placed numbers so a script can cross-check.1718## Procedure19201. **Freeze the data.** Ask `experimenter` for the final json/csv and the EVIDENCE21 rows. Anything measured after the freeze requires a new row and a note in22 `DECISIONS.md`.232. **Decide the scope sentence** for every range ("on the 9 parameter sets at ≥128-bit24 estimated security"). Scope comes from the parameter/security table, generated.253. **Write the section** following `paper-playbook/sections/10_experiments.md`.264. **Generate tables** (below). Mark the best cell per column (bold, optional light27 colour); two-level headers with `\multicolumn`; captions define every abbreviation28 and symbol.295. **Write the fairness statements** (below) that apply.306. **Write boundary/limitation remarks** (below).317. **Cross-check script**: `crbench audit-tex paper/main.tex EVIDENCE.md` (lib/bench),32 or extract every number from the PDF text (`pdftotext -layout`) and match it to33 EVIDENCE rows; unmatched numbers are errors unless they are parameters or34 citations. Record the result in `SUBMISSION.md`.3536## Table set3738| table | contents | notes |39|---|---|---|40| Parameters | one row per set: all scheme parameters, derived values, security estimate(s) | `\resizebox{\linewidth}{!}` if wide; security column mandatory |41| Main results | per set: each configuration's time for the changed stage, total time, overheads (capacity, memory), ratio | ratio column states "vs <baseline>" in the header |42| Ablation | each component alone vs composed; failure counts/timeouts for variants that do not work | strongest evidence that the combination is necessary |43| Mechanism check | model-predicted vs measured operation counts | supports a cost model claim |44| Comparison with published work | Method, Ref, per-stage times, total, ratio; "measured by us" vs "as reported" marker; threads; failure probability; key/communication sizes | incomparable rows: symbol + "—" in ratio column + caption explanation |45| Client-side / communication | client cost, key sizes, setup cost | reviewers of transciphering/PSI/MPC ask for it |4647Layout: `\scriptsize`, `\arraystretch` ≈ 0.92, `\tabcolsep` ≈ 2.6pt; tables at `[t]`;48table captions above tables, figure captions below figures.4950## Fair-comparison statements (use the ones that apply)5152- **Same pipeline:** "All ratios are against <baseline> run in our pipeline with the53 same <radix/parameters/library commit>; only <stage> differs."54- **Measured vs claimed:** "Rerunning the artifact of [X] on our machine reproduces55 their claimed a–b× as c–d×; we report both."56- **General caution before cross-paper tables:** "Such comparisons must be read with57 care because of differences in libraries, hardware and failure probabilities; we58 mark numbers taken from the original papers with ◇."59- **Metric orientation remark:** "Our goal is throughput; for latency-critical60 applications <other scheme> remains preferable."61- **Parallelism remark:** "All our numbers are single-threaded. Evaluating62 independent instances in parallel helps every scheme equally, so single-thread63 numbers compare designs, not hardware."64- **Security-level parity:** "Both our parameters and the baseline's are estimated65 at λ bits with <estimator commit>; where the estimator has moved since the baseline66 was published, we re-estimate both."67- **Baseline strength:** "We use the fastest configuration of <library> for the68 baseline (<flags>), not the default one." (Reviewers and shepherds check this.)6970## Boundary / limitation statements7172- Exclusions: "The k instances below the claimed security level are excluded from the73 headline ranges and flagged in Table <n>."74- Overheads: "The composed evaluator consumes 1.2–1.5× [ill.] the baseline's75 memory/capacity."76- Regime where we lose: "For t ≥ 7 [ill.], [X] is faster because ...; the regimes do77 not overlap." Say it before a reviewer does.78- Not applicable: "Our method requires <condition>; for instances violating it the79 implementation falls back to the baseline (log shows '<message>')."80- Measurement noise: "Absolute times vary with machine load across sessions; all81 ratios come from interleaved back-to-back runs in one session."8283## Measurement hygiene to report (and to follow)8485- Interleave A/B runs (ABAB...) in the same session; report median and IQR/CI.86- No concurrent heavy jobs; use a bench lock; pin threads.87- Never mix numbers from different sessions in one ratio; if an earlier number is not88 reproducible, retract it explicitly (and say so in the rebuttal if already quoted).89- Decryption/output verification for every run.9091## Output9293`paper/sections/experiments.tex`, generated tables in `paper/tables/`, the94cross-check report in `SUBMISSION.md`.