log-to-evidence
Rule
No number enters the paper without an EVIDENCE row. Each row records the number
as printed (after rounding), the exact command, the log path, the commit, the machine,
the date, and runs/statistic. Derived numbers (ratios, sums, percentages) get their own
row with derived: <formula over IDs>.
Procedure
- Parse the logs with a registered parser:
crbench parse results/x/new/r003.log --parser criterion, or
crbench stats results/x/run.json --baseline base.
For a new log format, register a parser. A regex plus a test is enough: see
lib/bench/crbench/parsers.py, regex_parser.
- Recompute the number from the raw logs. Do not copy it from an earlier summary.
Keep the recomputation script under
scripts/ so it can run again.
- Decide the printed form: significant digits, unit, rounding rule. Use the same
rule everywhere. Document whether you round or ceil: a
round() vs ceil()
mismatch between two tables produces off-by-one discrepancies that reviewers catch.
- Add the row:
crbench evidence add EVIDENCE.md --printed '0.61 s' \
--command 'crbench run -a base=... -a new=... -n 7 --threads 1' \
--log results/2026-01-01-toy-ab/run.json --runs '7 / median' --target ./src
crbench evidence add EVIDENCE.md --printed '2.02×' --command 'derived: E1/E2' --log -
--commit auto records the target repository's SHA and a +dirty flag;
--machine auto records a host hash.
- Generate the table from the same
run.json, with evidence IDs:
crbench table results/x/run.json --latex --evidence new=E2 new:speedup=E3.
In LaTeX each cell carries \evid{E3}. Add \providecommand{\evid}[1]{} to the
preamble so the markers are invisible in print, or render them as superscripts in
drafts.
- Audit before every paper build:
crbench evidence check EVIDENCE.md --root . # rows complete, logs exist
crbench audit-tex paper/main.tex EVIDENCE.md # each measurement-like number traced
Fix every untraced and mismatch finding. Use % crbench:ignore only for numbers
that are not claims (a year, a section count), and prefer rewording.
Row examples (synthetic)
| ID | number/fact as printed in paper | command | log path | commit | machine | date | runs/median |
|---|---|---|---|---|---|---|---|
| E1 | 1.23 s | crbench run -a base='./b/bench --set toy' ... -n 7 --threads 1 | results/2026-01-01-toy-ab/run.json | 1a2b3c4d5e6f | host:3f9a0c1d2e4b | 2026-01-01 | 7 / median |
| E2 | 0.61 s | (same run, arm new) | results/2026-01-01-toy-ab/run.json | 1a2b3c4d5e6f | host:3f9a0c1d2e4b | 2026-01-01 | 7 / median |
| E3 | 2.02× | derived: E1/E2 (CI [1.95, 2.08], paired bootstrap) | - | - | - | 2026-01-01 | - |
| E4 | ~~2.40×~~ | withdrawn: not reproducible in a same-session rerun (see E3) | results/2025-12-30-toy-ab/run.json | ... | ... | 2026-01-01 | - |
| E5 | 2^128.4 | sage -python est.py toy128 (MATZOV, estimator commit abcdef0) | results/est/toy128.log | abcdef0 | host:3f9a0c1d2e4b | 2026-01-01 | 1 |
| E6 | 7 results, zero sorry | lake build && scripts_check_no_sorry.sh | results/lean/build.log | 7a8b9c0 | host:3f9a0c1d2e4b | 2026-01-01 | 1 |
Pitfalls
- Mixed-configuration rows. Each table row must come from one configuration. A
best-time column and a best-op-count column taken from different sweep points must
not share a row. If you report both "fastest" and "fewest operations", use two
sentences and two rows.
- "Best over the sweep" must really be the best, for both arms. Recompute it.
- Consistency. The same claim printed in the abstract, introduction, evaluation and
appendix must cite the same ID.
- Negative results (timeouts, noise-budget failures, unsupported configurations)
also get rows. The paper may cite them as boundaries.
- Contaminated sessions. If a later same-session rerun contradicts a row, withdraw
the row visibly, with a reason. Do not overwrite it.
1---2name: log-to-evidence3description: Turn raw logs into EVIDENCE.md rows and LaTeX/Markdown tables so that every number in the paper traces to a command, a log, a commit and a machine. Use after every benchmark or estimation campaign, and before every paper build, where `crbench audit-tex` must pass.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/log-to-evidence6---78# log-to-evidence910## Rule1112**No number enters the paper without an EVIDENCE row.** Each row records the number13*as printed* (after rounding), the exact command, the log path, the commit, the machine,14the date, and runs/statistic. Derived numbers (ratios, sums, percentages) get their own15row with `derived: <formula over IDs>`.1617## Procedure18191. **Parse** the logs with a registered parser:20 `crbench parse results/x/new/r003.log --parser criterion`, or21 `crbench stats results/x/run.json --baseline base`.22 For a new log format, register a parser. A regex plus a test is enough: see23 `lib/bench/crbench/parsers.py`, `regex_parser`.242. **Recompute** the number from the raw logs. Do not copy it from an earlier summary.25 Keep the recomputation script under `scripts/` so it can run again.263. **Decide the printed form**: significant digits, unit, rounding rule. Use the same27 rule everywhere. Document whether you round or ceil: a `round()` vs `ceil()`28 mismatch between two tables produces off-by-one discrepancies that reviewers catch.294. **Add the row**:30 ```bash31 crbench evidence add EVIDENCE.md --printed '0.61 s' \32 --command 'crbench run -a base=... -a new=... -n 7 --threads 1' \33 --log results/2026-01-01-toy-ab/run.json --runs '7 / median' --target ./src34 crbench evidence add EVIDENCE.md --printed '2.02×' --command 'derived: E1/E2' --log -35 ```36 `--commit auto` records the target repository's SHA and a `+dirty` flag;37 `--machine auto` records a host hash.385. **Generate the table** from the same `run.json`, with evidence IDs:39 `crbench table results/x/run.json --latex --evidence new=E2 new:speedup=E3`.40 In LaTeX each cell carries `\evid{E3}`. Add `\providecommand{\evid}[1]{}` to the41 preamble so the markers are invisible in print, or render them as superscripts in42 drafts.436. **Audit** before every paper build:44 ```bash45 crbench evidence check EVIDENCE.md --root . # rows complete, logs exist46 crbench audit-tex paper/main.tex EVIDENCE.md # each measurement-like number traced47 ```48 Fix every `untraced` and `mismatch` finding. Use `% crbench:ignore` only for numbers49 that are not claims (a year, a section count), and prefer rewording.5051## Row examples (synthetic)5253```markdown54| ID | number/fact as printed in paper | command | log path | commit | machine | date | runs/median |55|---|---|---|---|---|---|---|---|56| E1 | 1.23 s | crbench run -a base='./b/bench --set toy' ... -n 7 --threads 1 | results/2026-01-01-toy-ab/run.json | 1a2b3c4d5e6f | host:3f9a0c1d2e4b | 2026-01-01 | 7 / median |57| E2 | 0.61 s | (same run, arm new) | results/2026-01-01-toy-ab/run.json | 1a2b3c4d5e6f | host:3f9a0c1d2e4b | 2026-01-01 | 7 / median |58| E3 | 2.02× | derived: E1/E2 (CI [1.95, 2.08], paired bootstrap) | - | - | - | 2026-01-01 | - |59| E4 | ~~2.40×~~ | withdrawn: not reproducible in a same-session rerun (see E3) | results/2025-12-30-toy-ab/run.json | ... | ... | 2026-01-01 | - |60| E5 | 2^128.4 | sage -python est.py toy128 (MATZOV, estimator commit abcdef0) | results/est/toy128.log | abcdef0 | host:3f9a0c1d2e4b | 2026-01-01 | 1 |61| E6 | 7 results, zero sorry | lake build && scripts_check_no_sorry.sh | results/lean/build.log | 7a8b9c0 | host:3f9a0c1d2e4b | 2026-01-01 | 1 |62```6364## Pitfalls6566- **Mixed-configuration rows.** Each table row must come from *one* configuration. A67 best-time column and a best-op-count column taken from different sweep points must68 not share a row. If you report both "fastest" and "fewest operations", use two69 sentences and two rows.70- **"Best over the sweep"** must really be the best, for both arms. Recompute it.71- **Consistency.** The same claim printed in the abstract, introduction, evaluation and72 appendix must cite the same ID.73- **Negative results** (timeouts, noise-budget failures, unsupported configurations)74 also get rows. The paper may cite them as boundaries.75- **Contaminated sessions.** If a later same-session rerun contradicts a row, withdraw76 the row visibly, with a reason. Do not overwrite it.