BullshitBench Technical Guide

BullshitBench Technical Guide

This guide is for maintainers and contributors working on benchmark operations and data publishing.

Pipeline Overview

The end-to-end flow is:

1. collect
2. grade-panel
3. publish_latest_to_viewer.sh (updates data/latest)

Run all three steps with:

./scripts/run_end_to_end.sh

Serve locally after publish:

./scripts/run_end_to_end.sh --serve --port 8877

Then open http://localhost:8877/viewer/index.html.

Publish Existing Run Artifacts

Use this when you already have run outputs and only want to refresh data/latest:

./scripts/publish_latest_to_viewer.sh \
  --responses-file <path/to/responses.jsonl> \
  --collection-stats <path/to/collection_stats.json> \
  --panel-summary <path/to/panel_summary.json> \
  --aggregate-summary <path/to/aggregate_summary.json> \
  --aggregate-rows <path/to/aggregate.jsonl>

The publish step strips local-machine path fields from public artifacts.

Launch-Date Metadata Pipeline

Build model launch-date inventory/buckets and export review/candidate/canonical launch datasets:

./scripts/model_launch_pipeline.py run

This writes:

• data/model_metadata/tested_models_inventory.csv
• data/model_metadata/model_buckets.csv
• data/model_metadata/model_launch_sources.csv (template if missing)
• data/model_metadata/model_launch_collection.csv
• data/model_metadata/model_launch_judged.csv
• data/model_metadata/model_launch_attempts.csv
• data/model_metadata/model_launch_dates_review.csv
• data/model_metadata/model_launch_dates_candidates.csv
• data/model_metadata/model_launch_dates.csv (canonical accepted rows)

Publishing also exports:

• data/latest/model_launch_dates.csv
• data/latest/leaderboard_with_launch.csv

Current Config Notes

• Main config: config.json
• Question set: questions.json
• Recent published snapshot includes openai/gpt-5.2-codex and openai/gpt-5.3-codex variants across reasoning levels (low, high, xhigh) in data/latest/leaderboard.csv.

Repository Layout

• scripts/openrouter_benchmark.py: core CLI (collect, grade, grade-panel, aggregate, report)
• scripts/run_end_to_end.sh: one-command pipeline runner
• scripts/publish_latest_to_viewer.sh: publish run outputs into data/latest
• scripts/cleanup_generated_outputs.sh: remove generated local artifacts
• scripts/model_launch_pipeline.py: launch-date collection/judging pipeline
• viewer/index.html: canonical interactive viewer
• data/latest/*: canonical published dataset used by the viewer
• runs/*: local run history

Published Dataset Files

data/latest contains:

• responses.jsonl
• collection_stats.json
• panel_summary.json
• aggregate_summary.json
• aggregate.jsonl
• leaderboard.csv
• leaderboard_with_launch.csv
• model_launch_dates.csv
• manifest.json

Environment

Required:

• OPENROUTER_API_KEY

Optional:

• OPENROUTER_REFERER
• OPENROUTER_APP_NAME