Codex Autoresearch Skill

Codex Autoresearch Skill

Skill by ara.so — Codex Skills collection.

What is Codex Autoresearch?

Codex Autoresearch is an autonomous goal-driven experimentation system for Codex that continuously cycles through: modify code → verify result → retain (if improved) or discard (if worse) → repeat indefinitely. Tell Codex what you want to improve, walk away, and come back to a log of experiments and a better codebase.

Key capabilities:

• Autonomous iteration loops (foreground or background)
• Git-based experiment tracking with automatic revert on failure
• Dual-gate verification (did it improve? did anything break?)
• Escalating retry strategies (REFINE → PIVOT → Web search → Stop)
• Cross-run learning from past experiments
• Multiple modes: loop, plan, debug, fix, security, ship, exec

Installation

Recommended: Skill Installer

# In Codex
$skill-installer install https://github.com/leo-lilinxiao/codex-autoresearch

Then restart Codex.

Manual Installation

# Clone to Codex skills directory
cd ~/.config/codex/skills
git clone https://github.com/leo-lilinxiao/codex-autoresearch

# Or create a symlink
ln -s /path/to/codex-autoresearch ~/.config/codex/skills/codex-autoresearch

Prerequisites for Full Functionality

For background runs and unattended execution:

# Start Codex with Goals, hooks, and Full Access enabled
codex --enable goals --enable hooks --dangerously-bypass-approvals-and-sandbox

For foreground-only usage: Standard Codex workspace permissions are sufficient.

Core Command

$codex-autoresearch

This is the single entry point. Codex will infer the mode and configuration from your natural language goal.

Basic Usage Patterns

1. Improve Test Coverage

You:   $codex-autoresearch
       I want to improve test coverage to at least 80%

Codex: Current coverage: 62% (measured via pytest --cov)
       Target: ≥80%
       Verify: pytest --cov --cov-report=term
       Guard: pytest passes
       Results directory: ./autoresearch-results/
       Run mode: foreground or background?

You:   Foreground, go.

Codex: Starting iteration loop — baseline: 62%

What happens:

• Codex identifies untested files/functions
• Writes new tests or improves existing ones
• Commits each change
• Runs coverage check
• Keeps if coverage increases, reverts if it decreases or tests fail
• Repeats until target reached or interrupted

2. Eliminate TypeScript any Types

You:   $codex-autoresearch
       Get rid of all the any types in my TypeScript code

Codex: Found 47 `any` occurrences in src/**/*.ts
       Metric: `any` count (current: 47), direction: lower
       Verify: grep -r "any" src | wc -l
       Guard: npm run type-check (tsc --noEmit)
       Run mode?

You:   Background, run overnight.

Codex: Starting background run — baseline: 47. Iterating.

Background mode:

• Runs in detached process
• Logs to autoresearch-results/results.tsv
• Can be stopped/resumed
• Check progress: tail -f autoresearch-results/results.tsv

3. Fix Failing Tests

You:   $codex-autoresearch
       Fix the 12 failing tests

Codex: Detected 12 test failures via pytest
       Strategy: Repair one failure at a time
       Verify: pytest (exit code 0 = success)
       Mode?

You:   Foreground.

Codex: Iteration 1: test_auth_token_expiry
       Root cause: Clock skew in mock time...

4. Optimize Performance

You:   $codex-autoresearch
       Reduce API latency below 100ms

Codex: Current p95 latency: 247ms (from load test logs)
       Target: <100ms
       Verify: python benchmark.py --p95
       Guard: pytest integration tests
       Mode?

You:   Foreground.

5. Security Audit

You:   $codex-autoresearch
       Is this code secure?

Codex: Mode: security audit
       Framework: STRIDE + OWASP Top 10
       Scope: src/**/*.py (detected Flask app)
       Findings will be logged with code evidence.
       Run mode?

You:   Foreground.

Codex: Finding 1/7: SQL Injection risk in user_search()
       Evidence: Line 42, raw string interpolation...

Modes Reference

Codex infers the mode from your natural language, but understanding them helps you craft better prompts.

Configuration

You typically don't need to write config. Codex infers from your repo and goal.

But if you want explicit control, Codex will show you the inferred config and let you adjust before starting:

Codex: Inferred configuration:
       goal: "eliminate any types"
       scope: "src/**/*.ts"
       metric: any_count
       current_value: 47
       direction: lower
       verify_cmd: "grep -r 'any' src | wc -l"
       guard_cmd: "npm run type-check"
       
       Adjust anything?

You:   Change scope to include test files too.

Codex: Updated scope: "{src,test}/**/*.ts"
       Current value: 63 (including tests)
       Proceed?

Explicit Config (Advanced)

For CI/CD or scripted use, you can provide a JSON config:

{
  "goal": "Reduce bundle size",
  "metric": "bundle_kb",
  "current_value": 487,
  "target_value": 300,
  "direction": "lower",
  "verify_cmd": "npm run build && du -k dist/bundle.js | cut -f1",
  "guard_cmd": "npm test",
  "scope": "src/**/*.{ts,tsx}",
  "max_iterations": 50
}

# Save to file, then:
codex exec -f autoresearch_config.json

Results and State Files

All runs create an autoresearch-results/ directory in your workspace root:

autoresearch-results/
├── results.tsv          # Full experiment log (audit trail)
├── state.json           # Resume state (last consistent checkpoint)
├── lessons.json         # Cross-run learning (what worked/failed)
└── sessions/
    └── 2026-05-16_14-23-01/
        ├── experiment_1_keep.diff
        ├── experiment_2_discard.diff
        └── ...

Reading the Results Log

# View all experiments
cat autoresearch-results/results.tsv

# Watch live (during background run)
tail -f autoresearch-results/results.tsv

# Filter successful improvements
grep "keep" autoresearch-results/results.tsv

Example log:

iteration	commit	metric	delta	status	description
0	a1b2c3d	47	0	baseline	initial any count
1	b2c3d4e	41	-6	keep	replace any in auth module
2	-	49	+8	discard	generic wrapper introduced new anys
3	d4e5f6g	38	-3	keep	type-narrow API response handlers
4	e5f6g7h	38	0	discard	refactor had no effect
5	f6g7h8i	35	-3	keep	infer types from JSON schema

Escalation Strategy (When Stuck)

The loop doesn't blindly retry. It escalates:

One success resets all counters.

Example:

Iteration 12: discard (3rd consecutive failure)
→ REFINE: Try smaller type changes, one file at a time

Iteration 17: discard (5th consecutive failure)
→ PIVOT: Switch from manual typing to codegen from OpenAPI spec

Iteration 23: discard (2nd PIVOT without progress)
→ Web search: "TypeScript eliminate any types best practices"
  Found: Use strictNullChecks + noImplicitAny...

Iteration 25: keep
→ Counters reset. Back to normal iteration.

Stopping and Resuming

Stop a Foreground Run

You:   (interrupt Codex)

Codex: Autoresearch interrupted.
       State saved to autoresearch-results/state.json
       Resume with: $codex-autoresearch

Stop a Background Run

You:   $codex-autoresearch

Codex: Background run detected (PID 12345).
       Status: 15 iterations, metric improved 47→35
       
You:   Stop it.

Codex: Stopping background run...
       Final state saved. Resume anytime with $codex-autoresearch.

Resume

$codex-autoresearch

Codex automatically detects state.json and offers to resume:

Codex: Found interrupted session:
       Goal: eliminate any types
       Progress: 47→35 (12 iterations)
       Last commit: f6g7h8i
       Resume from iteration 13?

You:   Yes, continue.

Dual-Gate Verification

Every iteration runs two checks:

1. Verify — Did the metric improve?
2. Guard — Did anything break?

# Pseudocode of each iteration
git checkout -b experiment_N
modify_code()
git commit -m "experiment N: {hypothesis}"

verify_result = run(verify_cmd)
guard_result = run(guard_cmd)

if verify_result.improved and guard_result.passed:
    git merge experiment_N
    log("keep")
else:
    git reset --hard HEAD~1
    log("discard")

Example:

• Verify: pytest --cov (did coverage increase?)
• Guard: pytest (did all tests still pass?)

A change that increases coverage but breaks tests is discarded.

CI/CD Mode (exec)

For automation pipelines, use exec mode:

# Non-interactive, JSON output, exit codes
codex exec -f config.json --max-iterations 20 --timeout 3600

Exit codes:

• 0 — Target reached
• 1 — Max iterations reached without target
• 2 — Error or guard failure

Example config.json:

{
  "goal": "Reduce lint warnings to zero",
  "metric": "lint_warnings",
  "current_value": 34,
  "target_value": 0,
  "direction": "lower",
  "verify_cmd": "npm run lint -- --format json | jq '.length'",
  "guard_cmd": "npm test",
  "max_iterations": 30
}

Output (JSON):

{
  "status": "success",
  "iterations": 18,
  "initial_value": 34,
  "final_value": 0,
  "improvements": 12,
  "discards": 6,
  "elapsed_seconds": 1847
}

Advanced Features

Parallel Experiments (Git Worktrees)

Test up to 3 hypotheses simultaneously:

You:   $codex-autoresearch
       Try multiple approaches to reduce latency

Codex: Parallel mode enabled (3 worktrees).
       Hypothesis A: Database query optimization
       Hypothesis B: Response caching
       Hypothesis C: Async I/O refactor
       
       Testing in parallel...
       
       Results:
       A: -23ms (keep)
       B: -67ms (keep)  ← Best
       C: +12ms (discard)
       
       Merging B, continuing from new baseline.

Cross-Run Learning

The lessons.json file accumulates knowledge:

{
  "successful_patterns": [
    {
      "goal": "reduce any types",
      "approach": "infer from JSON schema",
      "success_rate": 0.83,
      "avg_improvement": 4.2
    }
  ],
  "failed_patterns": [
    {
      "goal": "reduce any types",
      "approach": "generic type wrappers",
      "failure_rate": 0.91,
      "reason": "introduced more anys downstream"
    }
  ]
}

Future runs bias toward proven approaches and away from known failures.

Session Hooks (Auto-Persistence)

Hooks keep Codex on track across session boundaries:

# Auto-installed with skill
~/.config/codex/hooks/post-session.sh

On every Codex session end:

1. Saves current state to state.json
2. Commits results log to git (if repo)
3. Backs up lessons learned

Manual hook setup (if needed):

chmod +x ~/.config/codex/skills/codex-autoresearch/hooks/post-session.sh
ln -s ~/.config/codex/skills/codex-autoresearch/hooks/post-session.sh \
      ~/.config/codex/hooks/post-session.sh

Real-World Examples

Python: Improve Type Coverage

# Before autoresearch
def process_data(data):  # No types
    result = []
    for item in data:
        result.append(item['value'] * 2)
    return result

# After 8 iterations
from typing import List, Dict, Any

def process_data(data: List[Dict[str, Any]]) -> List[float]:
    result: List[float] = []
    for item in data:
        result.append(float(item['value']) * 2)
    return result

Command:

$codex-autoresearch
Improve type hints coverage to 90% with mypy strict mode

JavaScript: Reduce Bundle Size

// Before (487kb)
import _ from 'lodash';
import moment from 'moment';
import * as utils from './utils';

// After 12 iterations (312kb)
import { debounce, throttle } from 'lodash-es';  // Tree-shakeable
import { formatDate } from 'date-fns/formatDate';  // Targeted import
import { parseJSON, validateEmail } from './utils';  // Explicit imports

Command:

$codex-autoresearch
Reduce production bundle size below 350kb

Rust: Eliminate Clippy Warnings

// Before (23 clippy warnings)
fn calculate(x: i32, y: i32) -> i32 {
    let mut result = 0;
    for i in 0..x {
        result = result + y;  // clippy: use += instead
    }
    result
}

// After 5 iterations (0 warnings)
fn calculate(x: i32, y: i32) -> i32 {
    x * y  // Direct multiplication, clippy-clean
}

Command:

$codex-autoresearch
Eliminate all clippy warnings with default lints

Troubleshooting

"Background run failed to start"

Cause: Codex not started with --dangerously-bypass-approvals-and-sandbox

Fix:

# Restart Codex with Full Access
codex --enable goals --enable hooks --dangerously-bypass-approvals-and-sandbox

Or use foreground mode instead.

"Verify command failed"

Cause: The verify command isn't executable or returns unexpected format

Fix:

You:   Test the verify command manually.

Codex: Running: grep -r "any" src | wc -l
       Output: "      47"  ← Extra spaces

You:   Adjust to: grep -r "any" src | wc -l | xargs

Codex: Updated verify_cmd. Baseline: 47 (clean integer).

"All experiments discarded, no progress"

Cause: Guard too strict (e.g., flaky tests) or goal unachievable

Check results log:

grep "discard" autoresearch-results/results.tsv

Common reasons:

• Guard fails even on unchanged code (flaky tests)
• Metric can't be improved with current tooling
• Scope too broad (try narrowing)

Fix:

You:   Run guard manually: npm test

Codex: Tests fail intermittently (test_cache_timeout).
       Recommendation: Fix flaky test or exclude from guard.

You:   Exclude that test from guard.

Codex: Updated guard: npm test -- --ignore test_cache_timeout

"State file corrupted"

Cause: Interrupted during JSON write

Fix:

# Restore from git (if committed)
git restore autoresearch-results/state.json

# Or start fresh (loses resume state, keeps logs)
rm autoresearch-results/state.json
$codex-autoresearch

"Infinite loop, no termination"

Cause: No target value set, or metric unstable

Fix:

You:   Set max_iterations to 50 and stop.

Codex: Updated config: max_iterations = 50
       Will stop after 50 iterations regardless of target.

Or add explicit target:

You:   Target is 80% coverage, stop when reached.

Codex: Updated target_value: 80
       Will stop when coverage ≥80%.

Best Practices

1. Start with Small, Measurable Goals

❌ "Make the code better" ✅ "Reduce ESLint warnings from 42 to 0"

2. Verify Your Verify Command First

# Before starting autoresearch, confirm the metric works
pytest --cov --cov-report=term | grep TOTAL
# Should output a parseable percentage

3. Use Foreground for New Goals

Run foreground first to watch the loop and verify behavior. Switch to background once confident.

4. Let Codex Infer, Then Adjust

Don't write config upfront. Let Codex propose, then refine:

Codex: Proposed verify: npm run test:coverage
You:   Change to: npm run test:coverage -- --json
Codex: Updated. Baseline: 62%

5. Check Results Log After Each Run

tail -20 autoresearch-results/results.tsv

Understand what worked and what didn't. This informs your next goal.

6. Use Git Strategically

Autoresearch commits every experiment. Your git log becomes the audit trail:

git log --oneline --grep="autoresearch"

To squash experiments into clean commits after the run:

git rebase -i HEAD~20  # Interactive rebase last 20 autoresearch commits

Environment Variables

Autoresearch respects standard tool configs via environment:

# Example: Use specific Python for pytest
export PYTHON=/usr/bin/python3.11
$codex-autoresearch

# Example: Increase test timeout
export PYTEST_TIMEOUT=300
$codex-autoresearch

No secrets needed — autoresearch runs local tools, no external API calls.

Integration with Other Codex Skills

Combine autoresearch with other skills:

You:   $code-review
       Review the autoresearch improvements from last night.

Codex: Reviewing 12 commits in autoresearch-results/sessions/...
       
       Summary:
       - 8 type improvements: Good, no regressions detected
       - 3 test additions: Coverage gaps filled correctly
       - 1 refactor: Extracted helper, maintains behavior
       
       Recommendation: Merge to main.

You:   $codex-autoresearch
       Optimize performance
       
       Then:
       $benchmark
       Compare before/after with flamegraphs

Limitations

1. Requires git — All experiments are git-based (commit/revert cycle)
2. Local tools only — Verify and guard must be executable commands in your environment
3. No multi-repo (yet) — Operates within a single workspace root
4. Deterministic metrics work best — Flaky metrics lead to false discards

Getting Help

If autoresearch behaves unexpectedly:

1. Check the results log: cat autoresearch-results/results.tsv
2. Review state file: cat autoresearch-results/state.json
3. Run verify manually: Test your verify command outside autoresearch
4. Ask Codex to explain: $codex-autoresearch then "explain the last 5 iterations"

Summary

Codex Autoresearch is a single-command autonomous improvement loop:

$codex-autoresearch

You describe the goal in natural language. Codex infers the config, confirms with you, then iterates:

modify → commit → verify → keep or discard → repeat

Foreground for interactive runs. Background for overnight. Results logged, state resumable, lessons learned across runs.

Most common workflow:

$codex-autoresearch
I want to [measurable goal]
→ Codex proposes config
→ You confirm or adjust
→ Choose foreground/background
→ Walk away or watch
→ Review results.tsv
→ Merge improvements

That's it. Autonomous iteration with human-in-the-loop goal-setting.