Hitl Maturity And Confidence Routing

HITL Maturity & Confidence Routing

Purpose

Moves "a human checks it" from a vague promise to an assessable, designed architecture. Most organizations describe their human oversight in aspirational terms ("we have a human in the loop") without ever checking which of four very different maturity levels that claim actually corresponds to — and the gap between the claimed level and the real one is usually invisible until something goes wrong. This skill classifies the current state, designs the routing logic that should replace it, and names who is accountable for keeping it calibrated.

Anchored in research

• Su Belagodu (Managing Partner, Intellectus Advisors; former product leadership at Pegasystems, IBM, Wipro) — a four-level HITL maturity model and a confidence-score routing table, presented in her public "Humans + AI = Impact" HITL series. Her general focus on HITL maturity and override-rate diagnostics is independently confirmed; the exact wording of quotes attributed to her in the source material for this skill could not be independently verified word-for-word — treat the structure of the model as reliably sourced and any verbatim quote as illustrative paraphrase, not a confirmed direct quote.
• Databricks engineering blog, "What is Human-in-the-Loop (HITL)?" — used as an independent, verifiable industry anchor for the general HITL maturity concept, so this skill doesn't rest on a single named source.
• The human-in/on/outside-the-loop model already used in this repo (see ../../../specialisation-packs/ai-native-startup-design/skills/closed-loop-process-and-human-oversight-design/SKILL.md) as this skill's simpler predecessor layer — this skill is the deeper, operational version for a process that's already live.

Method

1. Classify the process's current state against four maturity levels — be honest about where it actually is, not where the team believes it is:

   • Level 0 — Smoke and Mirrors (Automation First). A human is nominally "in the loop" but rarely exercises real judgment. Diagnostic sign: the override rate is below 5%, which usually means reviewers have stopped exercising critical judgment and are rubber-stamping AI output — a dangerous form of automation bias, not evidence the AI is simply that good.
   • Level 1 — Reactive (Human-on-the-Loop). A human monitors and intervenes only after an error has already happened. This is exhausting "babysitting automation" that catches mistakes but doesn't systematically improve the process.
   • Level 2 — Intentionally Designed HITL. Routing is explicit and rule-based: confidence scores determine whether a case is automated, sent for assisted validation, or escalated to a human — see the routing table in step 3.
   • Level 3 — Collaborative Intelligence (AI-in-the-Loop). The relationship inverts: the AI is now the assistant. It surfaces rich context, comparable cases, confidence scores, and its own reasoning transparency; the human makes the final call with full information. This is the target state for high-stakes, expertise-dependent workflows — not a universal goal for every process (see "What this skill does NOT do").

2. Don't take a claimed level at face value — check it against usage data. A team that describes itself as Level 2 or 3 but whose override rate sits under 5% is very likely still operating as Level 0 in practice. Cross-check the claimed level against ../hitl-override-metrics-and-feedback-audit/SKILL.md's override-rate measurement before accepting a self-reported maturity level.

3. Design the confidence-score routing table for the specific process being designed, using this as a starting structure — adjust the thresholds to the process's actual risk profile, don't treat 80/60 as a universal constant:

   Confidence score	Routing decision	Human's role
   80–100%	Automated	Process runs unattended; random-sample audits catch drift
   60–79%	Assisted validation	AI proposes, human confirms before it proceeds
   Below 60%, or flagged high-stakes regardless of score	Escalate to human	Human decides; the system must supply reasoning transparency (what the AI considered, and why it's uncertain), not just the raw score

4. Name a single accountable calibration owner — a "Chief Calibration Officer" role in spirit, not necessarily title: one named person or small team responsible for reviewing where the thresholds and routing rules are drifting, and for balancing experimentation, risk management, and the development of the humans doing the reviewing. Don't leave calibration as "everyone's job," which in practice means no one's.

5. Design the one-click override as a hard requirement, not an afterthought. If undoing or correcting an AI suggestion takes more than one deliberate action, experts will route around the system rather than through it, and the override-rate metric in step 6 stops being trustworthy (people stop bothering to record disagreement they can't act on cheaply).

6. Track the override rate as an ongoing health signal, not a one-time design check — hand this off to ../hitl-override-metrics-and-feedback-audit/SKILL.md for the full audit methodology. As a first-pass rule: below ~5% signals automation bias risk (investigate before trusting it); above ~30% signals the model isn't yet fit for the task at this confidence threshold.

7. Watch for the "Clippy 2.0" anti-pattern. Don't over-personify the AI (a name, a face, chatty first-person language) in ways that make its real capability and confidence level harder for the human to judge accurately. A collaborator the human trusts appropriately needs to look exactly as reliable as it is — no more comforting, no less.

8. Produce a structured output: current maturity level (with the evidence for it, not just the claim), target level and why that target — not necessarily Level 3 — is the right one for this process, the confidence-score routing table, the named calibration owner, and the override-rate baseline to track going forward.

What this skill does NOT do

• Doesn't assume Level 3 (Collaborative Intelligence) is always the goal — a low-stakes, high-volume, well-understood task may be entirely appropriate at Level 2's rule-based routing, or even a well-audited Level 0-adjacent automation once its accuracy is genuinely proven, not assumed.
• Doesn't implement the technical confidence-scoring model itself (the underlying classifier/model that produces the 0–100% number) — that's a data-science and engineering task; this skill designs what happens with the score once it exists.
• Doesn't replace ../../../specialisation-packs/ai-native-startup-design/skills/closed-loop-process-and-human-oversight-design/SKILL.md's simpler three-tier model for early-stage product design, before a process has real usage data to calibrate against.
• Doesn't perform the regulatory/ethics risk check — see ../../../ai-strategy-and-governance/skills/responsible-ai-and-governance-check/SKILL.md for whether human-oversight obligations are legally required, not just operationally wise.

Refinement notes

Areas to keep deepening with real practice:

• your own calibrated thresholds by task type — the 80/60 split above is a starting structure, not a validated universal rule
• concrete examples of a claimed maturity level that didn't survive an override-rate check, and what that revealed
• your own template for naming and scoping the calibration-owner role in a client organization (into ../../references/)

Once this section is filled in and validated in practice, update the maturity field in skills_index.json to draft, validated, or canonical (see ../../../meta/maturity_levels.md). Don't add new fields to the frontmatter — name and description are the only ones allowed (see ../../../meta/frontmatter_schema.md).

Continue from here

• Simpler predecessor model: ../../../specialisation-packs/ai-native-startup-design/skills/closed-loop-process-and-human-oversight-design/SKILL.md
• Applied to content specifically: ../../../ai-strategy-and-governance/skills/ai-output-curation-and-quality-control/SKILL.md
• Next in this pack (specify how the AI behaves within the routing this skill designs): ../ai-behavioral-specification-design/SKILL.md
• Next in this pack (verify the routing is actually working): ../hitl-override-metrics-and-feedback-audit/SKILL.md
• Related skill in this pack (draw the hard boundary the routing table must never cross): ../expert-agency-and-apprenticeship-protection/SKILL.md
• Related skill in another pack: ../../../ai-strategy-and-governance/skills/responsible-ai-and-governance-check/SKILL.md
• This pack's shared guardrails: ../../CLAUDE.md

References

• ../../references/hitl-partnership-heuristics-research.md — full sourcing and grounding-strength notes for this pack
• ../../references/ — the pack's shared background material
• ../../CLAUDE.md — the pack's shared guardrails