Sst LLM Judge Ranker

LLM-judge ranker

Order N artifacts by quality on a stated dimension. The comparator is "the LLM judges A vs B against this goal," done pairwise.

Project contract

• Output dir: <project>/data/sst-llm-judge-ranker/<run-id>/ where <run-id> is either user-supplied or auto-generated as <utc>_<short-uuid>. Inside:
  • metadata.jsonl — append-only log of every artifact registered.
  • ranking_state/<artifact-set>.ranklist.json — the current ranked list (just IDs, ordered best→worst).
• Tools required: harness's Read (for candidate files). No web access; ranking is pure analysis.
• Input shapes:
  • Direct compare: two file paths + an optional goal → returns winner + rationale.
  • Batch rank: a directory of candidate files + a goal → produces an initial ranklist.
  • Incremental insert: a new candidate path + the existing ranklist file → places the new candidate at its correct position with binary-search comparisons.

Operating principles

• One comparator function. Every comparison answers the same question in the same form: "Given goal G, which of A vs B better accomplishes it? Reply A | B | Equal, then a one-paragraph justification." Don't drift the rubric mid-run.
• Goal-anchored. Every comparison cites the same goal. If the goal changes between runs, the ranklist needs to be rebuilt from scratch — don't mix.
• Cap the list. max_ranklist_size (default 10) caps how many artifacts the list retains. After insertion, prune the worst.
• Atomic writes. Update the ranklist file via temp + rename so partial writes never corrupt it.
• Don't re-rank what's already in the list. Each artifact is registered once; if the same ID surfaces again, skip.

Comparator: LLM judge

Every pairwise call is the same shape:

Comparison goal: <goal — e.g. "comprehensiveness of the research on quantum computing">

Artifact A:
<contents of A>

Artifact B:
<contents of B>

Decide which one better accomplishes the goal. Answer in this exact form:

WINNER: A | B | Equal
RATIONALE: <one paragraph, naming the specific evidence (sections,
  examples, completeness gaps) that drove the decision>

Parse the response to extract the winner letter and the rationale string.

Mode 1: direct compare (two files)

Inputs: file_a, file_b, goal (optional; if absent, default to "general quality")

1. Read both files.
2. Run the comparator once.
3. Return (winner, rationale). Save a comparison record to <run-dir>/comparisons.jsonl if a run-id is in scope.

Mode 2: batch rank (initial ordering)

Inputs: candidates_dir, goal

1. List candidate files (each becomes an artifact ID — use the basename).
2. Append each to metadata.jsonl as {artifact_id, relative_path, registered_at}.
3. Process candidates in arrival order. For each new candidate, run Mode 3 (incremental insert) against the current list (initially empty).

The batch result is the same ranklist file; the difference from Mode 3 is that Mode 2 starts from empty and processes every file once.

Mode 3: incremental insert (one new candidate into existing ranklist)

This is the core algorithm — adaptive insertion sort with binary-search bisection.

Inputs: new_candidate_path, ranklist_path, goal, max_ranklist_size

1. Read the current list. If empty: insert the new candidate at index 0; done.
2. Compare vs top. Run the comparator on (new, current[0]).
   • WINNER=A → new is the best. Insert at index 0. Prune to max_ranklist_size (drop worst). Done.
   • WINNER=Equal → insert at index 1 (just below top). Prune. Done.
   • WINNER=B → new is worse than top. Continue to step 3.
3. Compare vs bottom (only if list has > 1 item). Comparator on (new, current[-1]).
   • WINNER=B → new is worse than the worst. Append to the end. If list is at cap, the append is a no-op (just discard the new candidate, no insert).
   • WINNER=A or Equal → new belongs somewhere in the middle. Continue to step 4.
4. Binary-search the middle. low = 1, high = len(list) - 2. Standard bisection:
   • mid = (low + high) // 2; comparator on (new, current[mid]).
   • WINNER=A or Equal → new is at most mid; record insertion_point = mid; high = mid - 1.
   • WINNER=B → new is below mid; low = mid + 1.
   • Loop until low > high. Insert at insertion_point.
5. Prune to max_ranklist_size (drop worst).
6. Atomically write the updated ranklist (write to <path>.tmp; os.replace).

Worst-case comparator calls per insert: 2 + log2(N). For a list of 10, that's ~5 LLM calls.

4. Save the result

For Mode 1 (direct compare), report:

WINNER: <A|B|Equal>
RATIONALE: <one paragraph>

For Modes 2 and 3, report the path to the updated ranklist plus the new top 3:

Ranklist: <path>
Top 3:
  1. <id> — <one-line preview>
  2. <id> — ...
  3. <id> — ...

Hard rules

• Never modify a candidate's content. Ranking is observation only.
• Never compare across goals. A ranklist built for goal G is invalid for goal G'. Rebuild.
• Never silently overwrite the ranklist file with a corrupted version. Always temp + rename.
• Don't pad rationales. A short, sharp justification with cited evidence beats a long one.
• Don't refuse to compare based on topic. If the artifacts are evaluable on the stated goal, evaluate them.