benchflow-ai Operate BenchFlow trajectory uploads
Use the public contribution route for ordinary users. Treat --direct as a
trusted-operator escape hatch, not as an equivalent public-path test.
Read references/cli-contract.md before explaining non-default modes, troubleshooting a failure, reviewing a generated manifest, or claiming an end-to-end production result. The reference is the complete behavior contract for the current CLI.
Choose the matching CLI build
Use the released CLI when validating the published user experience:
uv tool install --python 3.12 --upgrade benchflow
bench traj upload
Use the repository checkout when validating unreleased PR behavior:
uv sync --extra dev --locked
uv run bench traj upload
Do not substitute the installed bench binary for uv run bench while testing
unreleased code. Record bench --version or the exact Git SHA so the tested
artifact is unambiguous.
Run the requested flow
Prefer the guided flow when a person wants to inspect and confirm the capture:
bench traj upload
The CLI asks for the path, renders a report from a locally redacted staging
copy, asks for missing GitHub and email metadata (after trying gh / git
inference), and defaults the upload confirmation to No.
Use the fully specified form for scripts or an intentional no-prompt upload:
bench traj upload <PATH> --github-id <GITHUB_ID> --email <EMAIL>
Providing all three required inputs makes the command non-interactive: it still
renders the report, then uploads without a confirmation prompt. If any one is
missing, the session is interactive and asks only for missing values before a
final confirmation. Identity that resolves through gh / git inference also
skips the confirmation; without a TTY, unresolved identity fails with the
one-line --github-id / --email fallback instead of hanging on a prompt.
Use a dry run before a real upload when testing new files or CLI changes:
bench traj upload <PATH> \
--github-id <GITHUB_ID> \
--email <EMAIL> \
--dry-run
A dry run validates, redacts, reports, and creates the temporary manifest, but never prompts for confirmation and never makes a network request.
Inspect before uploading
Verify these invariants in the rendered report:
Total steps = Thinking steps + Tool-call steps + Human steps.- Human steps are real user messages. Tool results, status or metadata records,
empty records, and invented placeholders such as
Assistant responseare not trajectory steps. - Each preview row shows up to the first 100 words of a meaningful, already
redacted step.
--preview-stepsaccepts 0 through 20 and defaults to 5. - Every detected secret value is replaced locally with
<XXX-benchflow-key-values-XXX>. Valid JSONL containing secrets is accepted after masking; the original source files remain unchanged. - File count, byte size, creation time, primary file, format, step counts, masked-value count, and preview are plausible for the selected capture.
Verify the result at the right boundary
For a dry run, report only local validation. For a real public upload, verify
that the trusted validator promoted the digest to
sources/community/<digest>/, with manifest.json present last and bound to
the uploaded artifacts. A client success message or quarantine write alone is
not production end-to-end proof.
Report whether the capture was uploaded, cancelled, already present, rejected, or only dry-run validated. Never expose contributor email, detected secret values, signed upload URLs, credentials, or internal service endpoints.