# Research Preference

> Compare and rank candidate ML research ideas, experiment plans, or code branches before expensive evaluation. Use when the user asks which experiment to run next, which branch deserves a limited GPU or time budget, how prior runs change the choice, or what low-cost pilot would resolve a close decision. Best for two or more frozen candidates sharing a measurable objective; direct testing, ideation, literature review, and result summarization remain separate workflows.

- Skill: `leo-lilinxiao/research-preference` (Agent Skill, multi-file: 15 files)
- Install (CLI): `npx skillmds@latest add leo-lilinxiao/research-preference`
- Raw SKILL.md: https://api.skillmd.com/api/skills/leo-lilinxiao/research-preference/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- License: MIT
- Author: leo-lilinxiao (https://skillmd.com/u/leo-lilinxiao)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/leo-lilinxiao/research-preference

---


# Research Preference

Allocate the next unit of research compute to the candidate with the strongest evidence-adjusted branch value. Make a relative, traceable choice before full execution.

The instruction-only path works in Agent Skills-compatible clients. The bundled validation and aggregation CLI uses Python 3.10 or newer and the standard library.

## Workflow

### 1. Frame the allocation

Extract these fields from the request and available artifacts:

- two or more candidate IDs;
- shared objective, metric direction, and hard constraints;
- scarce budget and the cost of one full evaluation;
- prior attempts visible at decision time;
- the action requested: analyze, design a pilot, run a pilot, or select for full evaluation.

Use a cheap definitive measurement immediately when it fits inside the budget. Keep multi-objective choices provisional until the user supplies the tradeoff or weights.

### 2. Freeze candidates

Snapshot every candidate before judging:

- assign neutral, stable IDs;
- preserve the submitted plan, code, config, or content hash;
- represent each candidate with the same fields and comparable detail;
- separate candidate generation from the selection pass.

For three or more candidates, persistent decisions, or repeatable evaluation, create a decision packet using [`assets/decision-packet.schema.json`](assets/decision-packet.schema.json). Start from [`assets/example-decision.json`](assets/example-decision.json) when useful.

Resolve this skill's root from the location of this `SKILL.md`, then validate the packet:

```bash
python3 <skill-root>/scripts/research_preference.py validate decision.json
```

Treat validation errors as input work to resolve before comparison.

### 3. Build the evidence ledger

Classify every decision-relevant claim as one of:

1. `measurement` — measured under a declared protocol;
2. `artifact` — verified code, config, log, or output;
3. `history` — outcome from a related earlier branch;
4. `static-analysis` — feasibility or correctness evidence;
5. `assumption` — an explicit prior awaiting measurement.

Check metric definition, data split, seed policy, preprocessing, compute budget, and evaluation stage. Mark each item `comparable: true` only when it can support a direct candidate contrast. Record candidate, harness, dependency, data, and infrastructure failures by their actual source.

### 4. Choose the evidence mode

- **Preference mode — default:** compare plans, code, artifacts, and prior results. Use it when the evidence already supports a useful choice.
- **Pilot mode — escalation:** gather one low-cost observation when a specific uncertainty could reverse the preference. Read [`references/pilot-experiments.md`](references/pilot-experiments.md) before designing or running it.

An analysis request produces a pilot proposal. An execution request supports an authorized, in-scope pilot run.

### 5. Judge pairwise

Establish the eligible pool from the hard constraints. Keep fundamental task mismatch distinct from local repair effort.

Compare each decisive pair on:

- task and metric fit;
- quality and comparability of prior evidence;
- correctness and feasibility;
- realistic upside over the next few iterations;
- repair effort and clarity of the upgrade path;
- novelty relative to earlier branches;
- full-run cost, failure risk, and opportunity cost.

State the preference as `A`, `B`, or `tie`, followed by a concise evidence-grounded rationale and the observation that would reverse it. Use categorical confidence: `high`, `medium`, or `low`.

For a two-candidate decision, repeat the decisive comparison with labels and presentation order reversed. For larger pools or close choices, read [`references/preference-judging.md`](references/preference-judging.md), write one JSON object per judgment following [`assets/comparison.schema.json`](assets/comparison.schema.json), and aggregate the judgments:

```bash
python3 <skill-root>/scripts/research_preference.py validate \
  decision.json --comparisons comparisons.jsonl

python3 <skill-root>/scripts/research_preference.py aggregate \
  decision.json comparisons.jsonl --require-complete \
  --output preference-summary.json
```

The aggregator exposes pairwise coverage, Copeland ranking, a Condorcet winner, presentation-order conflicts, and preference cycles. Use these diagnostics as decision evidence; ground the final choice in the underlying rationale and sources.

Interpret CLI exit codes as `0` for success, `2` for input or usage repair, and `3` for missing pairwise coverage under `--require-complete`. Read [`references/cli.md`](references/cli.md) for the complete command, output-safety, and presentation-order contract.

### 6. Decide the next spend

Choose one outcome:

- `selected` — evidence supports the next full evaluation;
- `provisional` — one candidate leads while a reversal condition remains open;
- `tie` — candidates remain indistinguishable under current evidence;
- `direct-evaluation` — the definitive test is already the efficient next step;
- `blocked` — the decision frame or evidence pipeline needs repair.

Use [`references/decision-record.md`](references/decision-record.md) when the user requests a durable Markdown or JSON record. Read [`references/foundations.md`](references/foundations.md) when explaining the method or adapting it to a new domain.

## Response contract

Return this compact structure:

```markdown
## Decision
<status and selected candidate>

## Why
<decisive pairwise evidence and comparability>

## Candidate state
<eligible candidates, fundamental issues, and repairable work>

## Reversal condition
<single observation most likely to change the preference>

## Next spend
<full evaluation, one pilot, evidence repair, or stop>

## Confidence
<high | medium | low, tied to evidence quality>
```

## Completion check

- Candidate snapshots remained fixed throughout selection.
- Objective, metric direction, constraints, and budget are explicit.
- Evidence links resolve to visible artifacts or clearly labeled assumptions.
- Comparability and failure provenance are explicit.
- Decisive comparisons remain stable after order reversal, or confidence reflects the sensitivity.
- The next action stays within the user's requested scope and budget.
- The conclusion is scoped to the declared objective, evidence, and decision boundary.