Identify the real strongest baseline, fetch it, pin an exact commit, build it with a recorded recipe, verify correctness, and record everything in baselines/MANIFEST.md with a fairness checklist. Use at P4 before any comparison is measured, and again whenever a reviewer or falsifier questions a baseline.
1. Identify the real baseline (before fetching anything)
The paper you are improving on is not automatically the baseline. Ask these
questions:
Is its construction a re-derivation of earlier work? Search for the core idea
under other names.
Has a follow-up already improved on it? Check citing papers and ePrint from the
last 24 months.
Is a different library much faster for the same task? For example, an
implementation in another language or with a different backend.
The strongest fair comparison beats the most convenient one. Write the decision in
DECISIONS.md: "baseline = X because ..., rejected Y because ...".
Record the best reproducible local number as the bar to clear. The paper's
single-thread number may understate what the baseline achieves with trivial,
standard optimisations such as parallelisation. If a trivial engineering change
makes the baseline much faster, that faster version is the bar.
2. Fetch and pin
references/baselines/fetch.sh "$PROJECT/baselines" openfhe lattice-estimator
# -> baselines/PINNED.tsv with full SHAs
Pin a full commit SHA. A tag or branch name is not enough, because tags can move
and branches always do.
Research artifacts often ship as patches to a library rather than as a standalone
program. Pin both: the upstream SHA and the patched fork's SHA or patch hash.
Never commit third-party source trees into the research repo. Commit only the
MANIFEST row and the fetch command.
3. Build recipe (exact, copy-pasteable)
Record the exact configure and build commands, compiler and version, flags,
dependency versions (NTL/GMP, Rust toolchain, Go version), and environment variables
needed at runtime.
Runtime linkage is a classic trap. If several versions of a shared library are
installed, the binary may silently load the wrong one. Set LD_LIBRARY_PATH
explicitly, then check with ldd. Keep one build directory per configuration:
build-release-native/, build-release-portable/. Never benchmark from a build
directory whose provenance you cannot state. Stale builds from an earlier
configuration have produced wrong numbers many times. crbench stale BIN SRC flags
sources newer than the binary.
If a configuration misbehaves only at one optimisation level, record which level
was used for each number. An example is a code-generation problem at -O2 that
disappears at -O0, or the reverse.
4. Verify before timing
The library's own tests pass.
End-to-end correctness on your parameter set: decrypt equals expected, the
signature verifies, the proof verifies, PSI output equals the true intersection.
Print every slot, not a sample.
The runtime parameters are printed and diffed against the paper you compare
with (ring dimension, moduli, gadget base, key distribution, failure probability,
security level). A library preset named "128-bit" can use different concrete
values from those in a paper citing it. If the paper's speed-up used different
parameters from the library default, the denominator changed.
Record the failure modes. If a configuration is infeasible, document the
boundary with the exact error message, for example "parameter X too small to
hold Y", "unsupported ring shape" or "Decrypting with too much noise". Make the
harness catch the exception and print FAILED (<reason>) instead of dumping core.
5. MANIFEST row (the project's baselines/MANIFEST.md)
| name | url | commit (full SHA) | tag | build recipe | compiler + flags | threads | bench command | parser | role | fairness notes |
|---|---|---|---|---|---|---|---|---|---|---|
| toylib | https://example.org/toylib.git | 0123456789abcdef0123456789abcdef01234567 | v1.2.0 | `cmake -B b -DCMAKE_BUILD_TYPE=Release && cmake --build b -j4` | gcc 12.3, -O3 -march=native | 1 | `b/bench --set toy128` | gbench | strongest public implementation of X | preset toy128 has n=..., log q=... (paper Table 3 quotes different n); we use the preset on both arms |
Add a "Practical build priority" list: which baseline to reproduce first, and why.
6. Fairness checklist (sign off before handing to experimenter)
Same parameters and the same security estimate (see skills/param-estimation).
Same threads. Report single-thread as well as parallel.
Same compiler, flags and SIMD or accelerator backend.
Same machine, same session, interleaved rounds.
Same work, with the same phases timed. Offline and online are reported separately.
Every cited baseline number has been re-measured locally. Cited and local numbers
are never mixed in one ratio.
The strongest known baseline is included, not only the one the target paper used.
Build freshness is verified, and runtime library linkage is verified with ldd.
Hand-off
The row is complete, every checklist box is ticked or has a written exception, and a
single-arm reference run (crbench run -a base=..., at least 3 repeats) is logged.
Next agent: experimenter.
1---2name: baseline-pin3description: Identify the real strongest baseline, fetch it, pin an exact commit, build it with a recorded recipe, verify correctness, and record everything in baselines/MANIFEST.md with a fairness checklist. Use at P4 before any comparison is measured, and again whenever a reviewer or falsifier questions a baseline.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/baseline-pin6---78# baseline-pin910## 1. Identify the *real* baseline (before fetching anything)1112- The paper you are improving on is **not automatically** the baseline. Ask these13 questions:14 - Is its construction a re-derivation of earlier work? Search for the core idea15 under other names.16 - Has a follow-up already improved on it? Check citing papers and ePrint from the17 last 24 months.18 - Is a different library much faster for the same task? For example, an19 implementation in another language or with a different backend.20- The strongest fair comparison beats the most convenient one. Write the decision in21 DECISIONS.md: "baseline = X because ..., rejected Y because ...".22- Record the **best reproducible local number** as the bar to clear. The paper's23 single-thread number may understate what the baseline achieves with trivial,24 standard optimisations such as parallelisation. If a trivial engineering change25 makes the baseline much faster, that faster version is the bar.2627## 2. Fetch and pin2829```bash30references/baselines/fetch.sh "$PROJECT/baselines" openfhe lattice-estimator31# -> baselines/PINNED.tsv with full SHAs32```3334- Pin a **full commit SHA**. A tag or branch name is not enough, because tags can move35 and branches always do.36- Research artifacts often ship as **patches to a library** rather than as a standalone37 program. Pin both: the upstream SHA and the patched fork's SHA or patch hash.38- Never commit third-party source trees into the research repo. Commit only the39 MANIFEST row and the fetch command.4041## 3. Build recipe (exact, copy-pasteable)4243Record the exact configure and build commands, compiler **and version**, flags,44dependency versions (NTL/GMP, Rust toolchain, Go version), and environment variables45needed at *runtime*.4647Runtime linkage is a classic trap. If several versions of a shared library are48installed, the binary may silently load the wrong one. Set `LD_LIBRARY_PATH`49explicitly, then check with `ldd`. Keep one build directory per configuration:50`build-release-native/`, `build-release-portable/`. **Never benchmark from a build51directory whose provenance you cannot state.** Stale builds from an earlier52configuration have produced wrong numbers many times. `crbench stale BIN SRC` flags53sources newer than the binary.5455If a configuration misbehaves only at one optimisation level, record which level56was used for each number. An example is a code-generation problem at `-O2` that57disappears at `-O0`, or the reverse.5859## 4. Verify before timing6061- [ ] The library's own tests pass.62- [ ] End-to-end correctness on your parameter set: decrypt equals expected, the63 signature verifies, the proof verifies, PSI output equals the true intersection.64 Print every slot, not a sample.65- [ ] The **runtime parameters are printed and diffed** against the paper you compare66 with (ring dimension, moduli, gadget base, key distribution, failure probability,67 security level). A library preset named "128-bit" can use different concrete68 values from those in a paper citing it. If the paper's speed-up used different69 parameters from the library default, the denominator changed.70- [ ] Record the failure modes. If a configuration is infeasible, document the71 *boundary* with the exact error message, for example "parameter X too small to72 hold Y", "unsupported ring shape" or "Decrypting with too much noise". Make the73 harness catch the exception and print `FAILED (<reason>)` instead of dumping core.7475## 5. MANIFEST row (the project's `baselines/MANIFEST.md`)7677```markdown78| name | url | commit (full SHA) | tag | build recipe | compiler + flags | threads | bench command | parser | role | fairness notes |79|---|---|---|---|---|---|---|---|---|---|---|80| toylib | https://example.org/toylib.git | 0123456789abcdef0123456789abcdef01234567 | v1.2.0 | `cmake -B b -DCMAKE_BUILD_TYPE=Release && cmake --build b -j4` | gcc 12.3, -O3 -march=native | 1 | `b/bench --set toy128` | gbench | strongest public implementation of X | preset toy128 has n=..., log q=... (paper Table 3 quotes different n); we use the preset on both arms |81```8283Add a "Practical build priority" list: which baseline to reproduce first, and why.8485## 6. Fairness checklist (sign off before handing to `experimenter`)8687- [ ] Same parameters and the same security estimate (see `skills/param-estimation`).88- [ ] Same threads. Report single-thread as well as parallel.89- [ ] Same compiler, flags and SIMD or accelerator backend.90- [ ] Same machine, same session, interleaved rounds.91- [ ] Same work, with the same phases timed. Offline and online are reported separately.92- [ ] Every cited baseline number has been re-measured locally. Cited and local numbers93 are never mixed in one ratio.94- [ ] The strongest known baseline is included, not only the one the target paper used.95- [ ] Build freshness is verified, and runtime library linkage is verified with `ldd`.9697## Hand-off9899The row is complete, every checklist box is ticked or has a written exception, and a100single-arm reference run (`crbench run -a base=...`, at least 3 repeats) is logged.101Next agent: `experimenter`.
Run npx get-skill add jiayaoqijia/baseline-pin in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Identify the real strongest baseline, fetch it, pin an exact commit, build it with a recorded recipe, verify correctness, and record everything in baselines/MANIFEST.md with a fairness checklist. Use at P4 before any comparison is measured, and again whenever a reviewer or falsifier questions a baseline. It is listed under Coding & Dev Tools on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
jiayaoqijia (@jiayaoqijia) published this skill. Their other Agent Skills are listed on their SkillMD profile.