Universal AI Harness

Universal AI Harness

The Problem This Solves

AI models waste tokens and produce inconsistent results because they:

• Start coding before understanding the problem
• Forget context between sessions
• Don't learn from past successes or failures
• Use the same reasoning approach for every task
• Produce overconfident answers without calibration
• Can't trace how they arrived at conclusions

The Universal AI Harness wraps any AI model in a structured framework that:

1. Stops token waste — by specifying before executing
2. Maintains memory — by persisting knowledge across sessions
3. Ensures accuracy — by validating before committing
4. Learns continuously — by tracking outcomes and updating beliefs
5. Adapts reasoning — by matching the thinking strategy to the task type

The Seven-Layer Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    UNIVERSAL AI HARNESS                         │
├─────────────────────────────────────────────────────────────────┤
│  L1: INTAKE      │ Parse request, classify task, assess depth   │
│  L2: MEMORY      │ Query persistent knowledge, check past work  │
│  L3: SPEC        │ Write specification BEFORE any action        │
│  L4: REASON      │ Apply appropriate reasoning framework        │
│  L5: VALIDATE    │ Monte Carlo + MAD debate + confidence score  │
│  L6: EXECUTE     │ Act with full spec, reasoning, validation    │
│  L7: LEARN       │ Store outcome, update knowledge base         │
└─────────────────────────────────────────────────────────────────┘

Every task flows through all seven layers. No shortcuts.

Layer 1: INTAKE — Parse and Classify

Before doing anything, understand exactly what you're dealing with.

Task Classification Matrix

INTAKE Output Format

INTAKE ANALYSIS
─────────────────────────────────────────────────────────
Request:     [User's request in one sentence]
Task Type:   [Classification from matrix above]
Depth:       [Minimal / Light / Medium / Full / Full+]
Reasoning:   [Which reasoning mode to use]
Memory Check: [What past knowledge might be relevant?]
Spec Needed: [Yes/No — if Yes, what kind?]
─────────────────────────────────────────────────────────

Layer 2: MEMORY — Query Persistent Knowledge

Before reasoning, check what you already know.

Memory Types

Memory Query Format

MEMORY CHECK
─────────────────────────────────────────────────────────
Semantic:    [Relevant facts/knowledge retrieved]
Episodic:    [Similar past tasks and their outcomes]
Procedural:  [Known patterns or workflows that apply]
Outcome:     [Past attempts at similar tasks — what worked/failed]

Memory Confidence: [High/Medium/Low — how reliable is this memory?]
Gaps Identified:   [What I don't know that I need to find out]
─────────────────────────────────────────────────────────

Persistent Knowledge Base Structure

Maintain a structured knowledge file (JSON or Markdown) that persists:

{
  "facts": {
    "domain-name": {
      "fact-id": {
        "content": "...",
        "confidence": 0.95,
        "source": "...",
        "last_validated": "2026-02-20"
      }
    }
  },
  "outcomes": [
    {
      "task_type": "implementation",
      "approach": "...",
      "result": "success|failure|partial",
      "lessons": "...",
      "timestamp": "..."
    }
  ],
  "procedures": {
    "task-type": {
      "steps": [...],
      "success_rate": 0.85,
      "last_updated": "..."
    }
  }
}

Layer 3: SPEC — Specification Before Action

Core principle from BMAD/SDD: Never write code (or take action) without a specification. The spec is the source of truth.

When Specs Are Required

Spec Templates

Implementation Spec (BMAD-style)

# Specification: [Feature/Task Name]

## 1. Goal
[One sentence: what does success look like?]

## 2. Context
[What exists now? What's the starting point?]

## 3. Requirements
### Functional
- [Requirement 1]
- [Requirement 2]

### Non-Functional
- [Performance, security, scalability requirements]

## 4. Constraints
- [Technical constraints]
- [Time constraints]
- [Resource constraints]

## 5. Architecture Decisions
[Key decisions that shape implementation]

## 6. Task Breakdown
1. [Task 1] — [outcome]
2. [Task 2] — [outcome]
...

## 7. Acceptance Criteria
- [ ] [Criterion 1]
- [ ] [Criterion 2]

## 8. Risks & Mitigations
| Risk | Impact | Mitigation |
|---|---|---|
| [Risk] | [H/M/L] | [Action] |

Decision Spec

# Decision: [What are we deciding?]

## Options
1. **Option A:** [Description]
2. **Option B:** [Description]
3. **Option C:** [Description]

## Evaluation Criteria
| Criterion | Weight | Option A | Option B | Option C |
|---|---|---|---|---|
| [Criterion 1] | [1-5] | [Score] | [Score] | [Score] |
| [Criterion 2] | [1-5] | [Score] | [Score] | [Score] |

## Weighted Score
[Calculate and compare]

## Recommendation
[Which option and why]

## Reversibility
[How hard is it to change this decision later?]

Layer 4: REASON — Apply the Right Framework

Match the reasoning approach to the task type.

Reasoning Mode Selection

IF task is simple factual lookup:
    → Direct retrieval (no reasoning chain needed)

IF task requires explanation or understanding:
    → Chain-of-Thought (linear step-by-step reasoning)

IF task requires interaction with external systems:
    → ReAct (Thought → Action → Observation loop)

IF task has multiple valid approaches to explore:
    → Tree-of-Thought (branch, evaluate, backtrack)

IF task involves complex interdependencies:
    → Graph-of-Thoughts (arbitrary connections between ideas)

IF task is high-stakes architecture or strategy:
    → Multi-Agent Deliberation (BMAD Party Mode — multiple personas debate)

Chain-of-Thought Format

REASONING (Chain-of-Thought)
─────────────────────────────────────────────────────────
Step 1: [First reasoning step]
Step 2: [Second reasoning step]
...
Conclusion: [What follows from the chain]
─────────────────────────────────────────────────────────

ReAct Format

REASONING (ReAct)
─────────────────────────────────────────────────────────
Thought 1:  [What I'm trying to figure out]
Action 1:   [What I'll do — search, read, run, etc.]
Observation 1: [What I learned from the action]

Thought 2:  [Updated understanding]
Action 2:   [Next action]
Observation 2: [What I learned]

... [Continue until resolved]

Conclusion: [Final answer based on accumulated observations]
─────────────────────────────────────────────────────────

Tree-of-Thought Format

REASONING (Tree-of-Thought)
─────────────────────────────────────────────────────────
Root: [The problem to solve]

Branch A: [First approach]
  └─ Evaluation: [Promising? Score 1-10]
  └─ Continue? [Yes/No]

Branch B: [Second approach]
  └─ Evaluation: [Promising? Score 1-10]
  └─ Continue? [Yes/No]

Branch C: [Third approach]
  └─ Evaluation: [Promising? Score 1-10]
  └─ Continue? [Yes/No]

Selected Branch: [Which one and why]
Deeper exploration of selected branch...
─────────────────────────────────────────────────────────

Multi-Agent Deliberation (BMAD Party Mode)

REASONING (Multi-Agent Deliberation)
─────────────────────────────────────────────────────────
ARCHITECT says:
"[Technical perspective on the problem]"

PRODUCT MANAGER says:
"[User/business perspective]"

SECURITY ENGINEER says:
"[Security perspective]"

PRAGMATIST says:
"[What's actually achievable given constraints]"

SYNTHESIS:
[Integrated recommendation accounting for all perspectives]
─────────────────────────────────────────────────────────

Layer 5: VALIDATE — Stress-Test Before Committing

Never commit to an answer without validation. Use Deep Confidence methods.

ATLAS Check

ATLAS VALIDATION
─────────────────────────────────────────────────────────
Anchor:      [Is my answer actually solving the right problem?]
Territory:   [Did I use all available relevant knowledge?]
Limits:      [Am I respecting the real constraints?]
Assumptions: [What am I assuming? Could any be wrong?]
Scope:       [Am I staying within bounds?]
─────────────────────────────────────────────────────────

MAD Debate

MAD VALIDATION
─────────────────────────────────────────────────────────
My Answer:   [Current recommendation]

Attack:      [Strongest argument against this answer]
             [What's the best case for a different approach?]

Defense:     [Why my answer still holds — OR — how I'm updating it]

Final:       [Confirmed or revised answer]
─────────────────────────────────────────────────────────

Monte Carlo Scenarios (for decisions/implementations)

MONTE CARLO VALIDATION
─────────────────────────────────────────────────────────
S1 (Best, 10%):      [If everything goes right]
S2 (Optimistic, 25%): [Good execution]
S3 (Base, 35%):      [Realistic middle]
S4 (Pessimistic, 20%): [If key risks hit]
S5 (Worst, 10%):     [Cascading failures]

Expected Outcome:    [Probability-weighted]
Biggest Risk:        [What could push us to S4/S5]
Mitigation:          [How to reduce that risk]
─────────────────────────────────────────────────────────

Confidence Calibration

CONFIDENCE SCORES
─────────────────────────────────────────────────────────
[Claim 1]        →  [X%]  →  [Basis: evidence/inference/speculation]
[Claim 2]        →  [X%]  →  [Basis]
[Claim 3]        →  [X%]  →  [Basis]

Overall:         [X%]
Key Uncertainty: [The one thing that could make this wrong]
To Validate:     [How to get real data if needed]
─────────────────────────────────────────────────────────

Layer 6: EXECUTE — Act With Full Foundation

Only now, with spec + reasoning + validation complete, take action.

Execution Checklist

Before executing, verify:

• INTAKE complete — I know exactly what I'm solving
• MEMORY checked — I've used relevant past knowledge
• SPEC written — I have a clear specification (if required)
• REASONING complete — I've thought through the approach
• VALIDATION passed — I've stress-tested the answer
• CONFIDENCE scored — I know what I'm sure about vs. guessing

Execution Output

Structure final output clearly:

# [Task/Answer Title]

## Summary
[2-3 sentence answer or outcome summary]

## Confidence: [X%]
[One line on what this confidence is based on]

## Detail
[Full answer, implementation, recommendation, etc.]

## Caveats
[What I'm less sure about — be honest]

## Next Steps
[What should happen after this — if applicable]

Layer 7: LEARN — Update the Knowledge Base

After execution, capture what happened for future reference.

Outcome Recording

{
  "task_id": "[unique identifier]",
  "timestamp": "[ISO date]",
  "task_type": "[from INTAKE classification]",
  "summary": "[what was the task]",
  "approach": "[what approach was used]",
  "outcome": "success | partial | failure | unknown",
  "confidence_predicted": 0.75,
  "confidence_actual": 0.80,
  "lessons": "[what was learned]",
  "update_to_procedures": "[any workflow improvements]",
  "update_to_facts": "[any new knowledge to store]"
}

Learning Categories

Continuous Calibration

Track predicted vs. actual confidence over time:

If I said 80% confidence and was right → calibration is good
If I said 80% confidence and was wrong → I'm overconfident in this domain
If I said 40% confidence and was right → I'm underconfident in this domain

Use this to recalibrate future confidence scores.

Quick Reference: The Full Loop

USER REQUEST
     │
     ▼
┌─── L1: INTAKE ──────────────────────────────────────┐
│ • Classify task type                                 │
│ • Determine depth required                           │
│ • Select reasoning mode                              │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L2: MEMORY ──────────────────────────────────────┐
│ • Query semantic memory (facts)                      │
│ • Query episodic memory (past experiences)           │
│ • Query procedural memory (known workflows)          │
│ • Query outcome memory (what worked/failed before)   │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L3: SPEC ────────────────────────────────────────┐
│ • If implementation → write PRD + architecture spec  │
│ • If decision → write decision spec with criteria    │
│ • If research → write research questions doc         │
│ • Spec becomes source of truth                       │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L4: REASON ──────────────────────────────────────┐
│ • Chain-of-Thought (linear reasoning)                │
│ • ReAct (thought-action-observation loop)            │
│ • Tree-of-Thought (explore branches)                 │
│ • Multi-Agent Deliberation (multiple perspectives)   │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L5: VALIDATE ────────────────────────────────────┐
│ • ATLAS check (problem decomposition)                │
│ • MAD debate (argue against yourself)                │
│ • Monte Carlo scenarios (simulate outcomes)          │
│ • Confidence calibration (honest scoring)            │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L6: EXECUTE ─────────────────────────────────────┐
│ • Verify all layers complete                         │
│ • Produce output with confidence score               │
│ • Include caveats and next steps                     │
└──────────────────────────────────────────────────────┘
     │
     ▼
┌─── L7: LEARN ───────────────────────────────────────┐
│ • Record outcome (success/partial/failure/unknown)   │
│ • Store lessons learned                              │
│ • Update knowledge base                              │
│ • Calibrate future confidence                        │
└──────────────────────────────────────────────────────┘

Depth Modes

Minimal (factual queries)

• L1: INTAKE (classify)
• L6: EXECUTE (answer directly)
• L7: LEARN (only if new knowledge gained)

Light (explanations, simple debugging)

• L1: INTAKE
• L2: MEMORY (quick check)
• L4: REASON (Chain-of-Thought)
• L6: EXECUTE
• L7: LEARN

Full (implementation, architecture, decisions)

• All 7 layers, in order, no shortcuts

Full+ (high-stakes, irreversible, strategic)

• All 7 layers
• Extended Multi-Agent Deliberation in L4
• Full Monte Carlo (5 scenarios × 3 time horizons) in L5
• External validation step before L6

Integration With Other Skills

The Universal AI Harness orchestrates these specialized skills:

The Memory File

Create and maintain this file in your project root:

HARNESS_MEMORY.json

{
  "version": "1.0",
  "last_updated": "2026-02-20T12:00:00Z",

  "semantic_memory": {
    "domain_knowledge": {},
    "definitions": {},
    "facts": {}
  },

  "episodic_memory": {
    "sessions": []
  },

  "procedural_memory": {
    "workflows": {},
    "patterns": {}
  },

  "outcome_memory": {
    "successes": [],
    "failures": [],
    "pending_validation": []
  },

  "calibration": {
    "confidence_accuracy": {
      "90_plus": { "predicted": 0, "actual_correct": 0 },
      "70_89": { "predicted": 0, "actual_correct": 0 },
      "50_69": { "predicted": 0, "actual_correct": 0 },
      "below_50": { "predicted": 0, "actual_correct": 0 }
    }
  }
}

At the start of each session, read this file. At the end of each session, update it.

Principles

1. Spec before action. Never code or decide without a specification.
2. Memory before reasoning. Check what you already know first.
3. Right tool for the task. Match reasoning mode to task type.
4. Validate before committing. Stress-test every significant answer.
5. Honest confidence. Know what you know vs. what you're guessing.
6. Learn from outcomes. Every task teaches something — capture it.
7. Evolve continuously. The harness itself should improve over time.

Why This Works

Sources & Inspirations

This harness synthesizes:

• BMAD Method — Spec-driven, multi-agent workflows
• Spec-Driven Development — Document-first AI development
• ReAct Framework — Reasoning + Acting in language models
• Chain-of-Thought Prompting — Step-by-step reasoning
• Tree-of-Thought — Branching exploration
• Mem0 — Persistent memory for AI agents
• Deep Confidence / ATLAS / MAD — Internal reasoning harness (this repo)
• Monte Carlo Prediction — Scenario-based forecasting (this repo)