Frontier Harness Research — Comparative Research Process
Purpose: Before implementing a feature, compare relevant frontier harnesses and the original/upstream implementation, then establish design decision rationale for GEODE application.
When to apply: The user requests a comparison, or an external contract or
architectural choice needs evidence. Ordinary fixes using an established
local pattern do not require a multi-system survey.
Research Sources
| # |
System |
Type |
Core Pattern Areas |
GEODE Skill Reference |
| 1 |
Claude Code |
CLI agent |
Permission model, Hook, Memory, Skill, Context management, UI |
current official documentation |
| 2 |
Codex |
Coding agent |
Sandbox execution, PR workflow, code review, multi-file editing |
current official documentation and source |
| 3 |
OpenClaw |
Chat agent |
Gateway, Session Key, Binding, Lane Queue, Plugin, Failover, 4-tier automation |
openclaw-patterns |
| 4 |
autoresearch |
Autonomous experiment loop |
Constraint-based design, ratchet, Context Budget, program.md, Simplicity Selection |
karpathy-patterns |
| 5 |
Prime Agent |
RLM-native coding/research harness |
Persistent REPL, programmable context, recursive subagents, continual harness state, native-harness evaluation |
official source and technical report |
| 6 |
Original / upstream implementation |
Native authority |
Task and scorer semantics, prompts, assets, workspace assumptions, reference results, reproduction path |
pinned upstream source and primary docs |
Research Process
Step 1: Topic Definition
Define the feature to implement in one line and extract related keywords.
Example:
Topic: "Model Failover automation"
Keywords: failover, fallback, retry, circuit breaker, model switching
Step 2: Relevant-Source Pattern Exploration
Explore only systems relevant to the decision. Read applicable local skills for
source routing, then verify external claims against current primary docs or
pinned source. Recalled knowledge supplies search terms, not verification;
report unavailable evidence without inventing a capability.
2a. Claude Code Pattern Exploration
| Exploration Area |
Checkpoints |
| Permission Model |
allowlist/denylist, auto-approve, fallback after denial |
| Hook System |
pre/post tool hooks, settings.json-based automation |
| Memory |
CLAUDE.md, project memory, auto-memory |
| Skill System |
skill discovery, trigger keywords, 4-tier priority |
| Context Management |
sliding window, compression, token management |
| UI Patterns |
status line, progress indicators, error display |
| Safety |
HITL tiers, bash safety, dangerous tool gates |
2b. Codex Pattern Exploration
| Exploration Area |
Checkpoints |
| Sandbox Execution |
Isolated environment, filesystem restrictions, network restrictions |
| TDD Loop |
test-first, red-green-refactor, automated verification |
| PR Workflow |
Branch creation, change summary, review request |
| Multi-file Editing |
Dependency tracking, consistency maintenance, refactoring scope |
| Task Decomposition |
Complex task breakdown, sequential/parallel determination |
2c. OpenClaw Pattern Exploration (see openclaw-patterns skill)
| Exploration Area |
Checkpoints |
| Gateway + Agent dual system |
Control plane vs execution plane separation |
| Session Key hierarchy |
agent:{id}:{context} format session isolation |
| Binding routing |
Most-Specific Wins, static rules, hot reload |
| Lane Queue |
Session/Global/Subagent Lane concurrency control |
| Sub-agent Spawn+Announce |
Isolated execution, automatic result injection |
| 4-tier automation |
Heartbeat, Cron, Internal Hooks, Gateway Hooks |
| Plugin architecture |
Channel/Tool/Skill/Hook — 4 extension points |
| Policy Chain |
6-layer tool access control |
| Failover |
Auth Rotation, Thinking Fallback, Context Overflow, Model Failover |
| Operational patterns |
Coalescing, Atomic Store, Run Log, Hot Reload, Stuck Detection |
2d. autoresearch Pattern Exploration (see karpathy-patterns skill)
| Exploration Area |
Checkpoints |
| P1 Constraint-based design |
Define "what cannot be done" first |
| P2 Single-file constraint |
Minimize modification surface area |
| P3 Fixed time budget |
Limit by wall clock, not steps |
| P4 Ratchet mechanism |
Keep only improvements, auto-revert on degradation |
| P5 Git as State Machine |
Commit=experiment, reset=discard |
| P6 Context Budget |
Redirect + selective extraction |
| P7 program.md |
Agent behavior change = instruction document modification |
| P10 Simplicity Selection |
Code deletion improvement > code addition improvement |
2e. Prime Agent Pattern Exploration
| Exploration Area |
Checkpoints |
| Recursive Language Model |
Persistent REPL, programmatic context access, subagent calls |
| Continual Harness |
Prompt, subagent, skill, and memory CRUD from trajectory evidence |
| Long-horizon operation |
Goal, heartbeat, autonomous continuation, explicit budgets and gates |
| Evaluation discipline |
Same-model harness comparison, native-harness baseline, token and score accounting |
| Failure analysis |
Reward hacking, verifier boundary, harness/model co-adaptation |
2f. Original / Upstream Implementation Exploration
| Exploration Area |
Checkpoints |
| Native authority |
Pinned repository/version, official docs, canonical entry point |
| Benchmark invariants |
Prompt, task data, assets, workspace, tools, scorer/verifier, stop conditions |
| Reference evidence |
Official result, run configuration, seeds/repeats, metric denominator |
| Adapter parity |
Same agent/model/config on original and adapted paths; deviations documented |
| Provenance |
Original source and adapter code remain distinguishable and auditable |
Step 3: GAP Analysis
Compare exploration results against GEODE's current state to identify GAPs.
Output format:
| # | Pattern | Source | GEODE Status | GAP | Priority |
|---|---------|--------|-------------|-----|----------|
| 1 | Model Failover | OpenClaw | ⚠️ Definition only | No auto-switching logic | P1 |
| 2 | Circuit Breaker | Codex | ✗ None | No blocking on consecutive failures | P1 |
| 3 | Retry Budget | autoresearch P3 | ⚠️ Partial | No time-based limit | P2 |
Step 4: Design Decisions
Select items to implement from GAP analysis results and document design decision rationale.
Decision Criteria:
| Criterion |
Application |
| Pattern appears in several systems |
Compare the failure it solves and whether GEODE has that failure; prevalence alone does not require adoption |
| Pattern appears in one system |
Evaluate the same consumer, failure, and verification evidence; source count alone does not reject it |
| Over-engineering risk |
→ Apply Karpathy P10, implement minimally |
| Conflicts with existing GEODE patterns |
→ Existing pattern takes priority, gradual transition |
| Benchmark integration or platform adapter |
→ Preserve original semantics and require parity evidence before equivalence claims |
Step 5: Plan Document Writing
Record the decision in the existing task plan or PR. Use a docs/plans/ document
when durable research detail warrants one; do not create a second plan merely
to satisfy this skill.
# Plan: [Feature Name]
## Frontier Research Summary
| System | Related Pattern | Adoption | Rationale |
|--------|----------------|----------|-----------|
| Claude Code | ... | Adopt/Adapt/Reject | ... |
| Codex | ... | Adopt/Adapt/Reject | ... |
| OpenClaw | ... | Adopt/Adapt/Reject | ... |
| autoresearch | ... | Adopt/Adapt/Reject | ... |
| Prime Agent | ... | Adopt/Adapt/Reject | ... |
| Original / upstream | ... | Preserve/Adapt/N/A | ... |
## Design Decisions
...
## Implementation Phases
...
Research Checklist
For the selected research scope, verify:
Notes
- Ground the affected decision before implementation. If later evidence changes it, revise the plan and verify the affected behavior before proceeding.
- Not every frontier system is relevant. Mark irrelevant systems "N/A"; the original/upstream source is mandatory whenever one exists.
- Always read skill files first if they exist. The
openclaw-patterns and karpathy-patterns skills already contain distilled patterns, preventing redundant exploration.
- Prevent over-research: Stop once the decision is supported or the missing evidence is identified. Respect the task's time and cost budget.
1---2name: frontier-harness-research3description: Research process for comparing frontier harnesses (Claude Code, Codex, OpenClaw, autoresearch, Prime Agent) with the original/upstream implementation and performing GAP analysis before feature implementation. Triggered by "research" ("리서치"), "gap", "frontier", "harness", "case study" ("사례 조사"), "pattern exploration" ("패턴 탐색"), "comparative analysis" ("비교 분석") keywords.4---56# Frontier Harness Research — Comparative Research Process78> **Purpose**: Before implementing a feature, compare relevant frontier harnesses and the original/upstream implementation, then establish design decision rationale for GEODE application.9> **When to apply**: The user requests a comparison, or an external contract or10> architectural choice needs evidence. Ordinary fixes using an established11> local pattern do not require a multi-system survey.1213## Research Sources1415| # | System | Type | Core Pattern Areas | GEODE Skill Reference |16|---|--------|------|-------------------|----------------------|17| 1 | **Claude Code** | CLI agent | Permission model, Hook, Memory, Skill, Context management, UI | current official documentation |18| 2 | **Codex** | Coding agent | Sandbox execution, PR workflow, code review, multi-file editing | current official documentation and source |19| 3 | **OpenClaw** | Chat agent | Gateway, Session Key, Binding, Lane Queue, Plugin, Failover, 4-tier automation | `openclaw-patterns` |20| 4 | **autoresearch** | Autonomous experiment loop | Constraint-based design, ratchet, Context Budget, program.md, Simplicity Selection | `karpathy-patterns` |21| 5 | **Prime Agent** | RLM-native coding/research harness | Persistent REPL, programmable context, recursive subagents, continual harness state, native-harness evaluation | official source and technical report |22| 6 | **Original / upstream implementation** | Native authority | Task and scorer semantics, prompts, assets, workspace assumptions, reference results, reproduction path | pinned upstream source and primary docs |2324## Research Process2526### Step 1: Topic Definition2728Define the feature to implement in one line and extract related keywords.2930```31Example:32 Topic: "Model Failover automation"33 Keywords: failover, fallback, retry, circuit breaker, model switching34```3536### Step 2: Relevant-Source Pattern Exploration3738Explore only systems relevant to the decision. Read applicable local skills for39source routing, then verify external claims against current primary docs or40pinned source. Recalled knowledge supplies search terms, not verification;41report unavailable evidence without inventing a capability.4243#### 2a. Claude Code Pattern Exploration4445| Exploration Area | Checkpoints |46|-----------------|-------------|47| Permission Model | allowlist/denylist, auto-approve, fallback after denial |48| Hook System | pre/post tool hooks, settings.json-based automation |49| Memory | CLAUDE.md, project memory, auto-memory |50| Skill System | skill discovery, trigger keywords, 4-tier priority |51| Context Management | sliding window, compression, token management |52| UI Patterns | status line, progress indicators, error display |53| Safety | HITL tiers, bash safety, dangerous tool gates |5455#### 2b. Codex Pattern Exploration5657| Exploration Area | Checkpoints |58|-----------------|-------------|59| Sandbox Execution | Isolated environment, filesystem restrictions, network restrictions |60| TDD Loop | test-first, red-green-refactor, automated verification |61| PR Workflow | Branch creation, change summary, review request |62| Multi-file Editing | Dependency tracking, consistency maintenance, refactoring scope |63| Task Decomposition | Complex task breakdown, sequential/parallel determination |6465#### 2c. OpenClaw Pattern Exploration (see `openclaw-patterns` skill)6667| Exploration Area | Checkpoints |68|-----------------|-------------|69| Gateway + Agent dual system | Control plane vs execution plane separation |70| Session Key hierarchy | `agent:{id}:{context}` format session isolation |71| Binding routing | Most-Specific Wins, static rules, hot reload |72| Lane Queue | Session/Global/Subagent Lane concurrency control |73| Sub-agent Spawn+Announce | Isolated execution, automatic result injection |74| 4-tier automation | Heartbeat, Cron, Internal Hooks, Gateway Hooks |75| Plugin architecture | Channel/Tool/Skill/Hook — 4 extension points |76| Policy Chain | 6-layer tool access control |77| Failover | Auth Rotation, Thinking Fallback, Context Overflow, Model Failover |78| Operational patterns | Coalescing, Atomic Store, Run Log, Hot Reload, Stuck Detection |7980#### 2d. autoresearch Pattern Exploration (see `karpathy-patterns` skill)8182| Exploration Area | Checkpoints |83|-----------------|-------------|84| P1 Constraint-based design | Define "what cannot be done" first |85| P2 Single-file constraint | Minimize modification surface area |86| P3 Fixed time budget | Limit by wall clock, not steps |87| P4 Ratchet mechanism | Keep only improvements, auto-revert on degradation |88| P5 Git as State Machine | Commit=experiment, reset=discard |89| P6 Context Budget | Redirect + selective extraction |90| P7 program.md | Agent behavior change = instruction document modification |91| P10 Simplicity Selection | Code deletion improvement > code addition improvement |9293#### 2e. Prime Agent Pattern Exploration9495| Exploration Area | Checkpoints |96|-----------------|-------------|97| Recursive Language Model | Persistent REPL, programmatic context access, subagent calls |98| Continual Harness | Prompt, subagent, skill, and memory CRUD from trajectory evidence |99| Long-horizon operation | Goal, heartbeat, autonomous continuation, explicit budgets and gates |100| Evaluation discipline | Same-model harness comparison, native-harness baseline, token and score accounting |101| Failure analysis | Reward hacking, verifier boundary, harness/model co-adaptation |102103#### 2f. Original / Upstream Implementation Exploration104105| Exploration Area | Checkpoints |106|-----------------|-------------|107| Native authority | Pinned repository/version, official docs, canonical entry point |108| Benchmark invariants | Prompt, task data, assets, workspace, tools, scorer/verifier, stop conditions |109| Reference evidence | Official result, run configuration, seeds/repeats, metric denominator |110| Adapter parity | Same agent/model/config on original and adapted paths; deviations documented |111| Provenance | Original source and adapter code remain distinguishable and auditable |112113### Step 3: GAP Analysis114115Compare exploration results against GEODE's current state to identify GAPs.116117```118Output format:119120| # | Pattern | Source | GEODE Status | GAP | Priority |121|---|---------|--------|-------------|-----|----------|122| 1 | Model Failover | OpenClaw | ⚠️ Definition only | No auto-switching logic | P1 |123| 2 | Circuit Breaker | Codex | ✗ None | No blocking on consecutive failures | P1 |124| 3 | Retry Budget | autoresearch P3 | ⚠️ Partial | No time-based limit | P2 |125```126127### Step 4: Design Decisions128129Select items to implement from GAP analysis results and document design decision rationale.130131**Decision Criteria:**132133| Criterion | Application |134|-----------|-------------|135| Pattern appears in several systems | Compare the failure it solves and whether GEODE has that failure; prevalence alone does not require adoption |136| Pattern appears in one system | Evaluate the same consumer, failure, and verification evidence; source count alone does not reject it |137| Over-engineering risk | → Apply Karpathy P10, implement minimally |138| Conflicts with existing GEODE patterns | → Existing pattern takes priority, gradual transition |139| Benchmark integration or platform adapter | → Preserve original semantics and require parity evidence before equivalence claims |140141### Step 5: Plan Document Writing142143Record the decision in the existing task plan or PR. Use a `docs/plans/` document144when durable research detail warrants one; do not create a second plan merely145to satisfy this skill.146147```markdown148# Plan: [Feature Name]149150## Frontier Research Summary151152| System | Related Pattern | Adoption | Rationale |153|--------|----------------|----------|-----------|154| Claude Code | ... | Adopt/Adapt/Reject | ... |155| Codex | ... | Adopt/Adapt/Reject | ... |156| OpenClaw | ... | Adopt/Adapt/Reject | ... |157| autoresearch | ... | Adopt/Adapt/Reject | ... |158| Prime Agent | ... | Adopt/Adapt/Reject | ... |159| Original / upstream | ... | Preserve/Adapt/N/A | ... |160161## Design Decisions162...163164## Implementation Phases165...166```167168## Research Checklist169170For the selected research scope, verify:171172- [ ] Topic keywords defined173- [ ] Relevant systems selected and primary evidence cited; irrelevant systems omitted174- [ ] Original/upstream source pinned and native invariants checked175- [ ] Adapter parity requirement recorded for benchmark integration or platform adapter176- [ ] GAP analysis table written177- [ ] Design decision rationale documented178- [ ] Decision and limitations recorded in the existing task artifact179180## Notes181182- **Ground the affected decision before implementation.** If later evidence changes it, revise the plan and verify the affected behavior before proceeding.183- **Not every frontier system is relevant.** Mark irrelevant systems "N/A"; the original/upstream source is mandatory whenever one exists.184- **Always read skill files first if they exist.** The `openclaw-patterns` and `karpathy-patterns` skills already contain distilled patterns, preventing redundant exploration.185- **Prevent over-research**: Stop once the decision is supported or the missing evidence is identified. Respect the task's time and cost budget.
mangowhoiscloud