# Human In The Loop AI

> Design AI systems with human oversight checkpoints, approval workflows, and escalation mechanisms. Covers when to require human intervention, how to build approval queues, confidence thresholds, and compliance-aware AI decision making.

- Skill: `sandeeprdy1729/human-in-the-loop-ai` (Agent Skill)
- Install (CLI): `npx skillmds add sandeeprdy1729/human-in-the-loop-ai`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sandeeprdy1729/human-in-the-loop-ai/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: Apache 2.0
- Author: Sandeeprdy1729 (https://skillmd.com/u/sandeeprdy1729)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/sandeeprdy1729/human-in-the-loop-ai

---


# Human-in-the-Loop AI

## Overview

Human-in-the-Loop (HITL) AI systems incorporate human judgment at critical decision points within automated workflows. As AI agents gain the ability to take real-world actions — sending emails, processing payments, modifying records — HITL patterns become essential for safety, compliance, and trust.

## When to Use This Skill

- Building AI agents that take irreversible actions (payments, deletions, communications)
- Implementing compliance requirements for regulated industries (finance, healthcare, legal)
- Designing confidence-based escalation from AI to human reviewers
- Creating approval queues for AI-generated content or decisions
- Building feedback loops for continuous AI improvement

## Core Concepts

### HITL Decision Framework

```
┌─────────────────────────────────────────────────┐
│                  AI Action                       │
├─────────────────────────────────────────────────┤
│                                                  │
│  Low Risk + High Confidence ──► Auto-execute     │
│  Low Risk + Low Confidence  ──► Auto + Log       │
│  High Risk + High Confidence──► Request Approval │
│  High Risk + Low Confidence ──► Escalate + Block │
│                                                  │
└─────────────────────────────────────────────────┘
```

### Action Classification Matrix

| Action Type | Risk Level | HITL Required? | Example |
|-------------|-----------|----------------|---------|
| Read data | None | No | Query database, fetch API |
| Format/transform | None | No | Generate report, summarize |
| Send notification | Low | Optional | Slack message, email alert |
| Modify record | Medium | Recommended | Update CRM, edit document |
| Financial transaction | High | Required | Process payment, refund |
| External comms (customer) | High | Required | Send client email |
| Delete/destroy | Critical | Required | Delete records, cancel subscription |
| Access escalation | Critical | Required | Grant permissions |

### Approval Flow Architecture

```
User Request
     │
     ▼
┌──────────┐     ┌──────────────┐     ┌──────────┐
│ AI Agent │────►│ Risk Assessor│────►│ Executor │
│ (Plans)  │     │ (Classifies) │     │ (Acts)   │
└──────────┘     └──────┬───────┘     └──────────┘
                        │                    ▲
                   High Risk?                │
                        │Yes                 │
                        ▼                    │
                 ┌─────────────┐     ┌──────┴──────┐
                 │  Approval   │────►│  Decision   │
                 │   Queue     │     │  (Approve/  │
                 │             │     │   Reject)   │
                 └─────────────┘     └─────────────┘
```

## Implementation Guide

### Confidence Thresholds

```python
class ConfidenceRouter:
    """Route AI decisions based on confidence and risk."""

    THRESHOLDS = {
        "auto_execute": 0.95,   # High confidence, auto-proceed
        "soft_review": 0.80,    # Execute but flag for review
        "approval_required": 0.60,  # Queue for human approval
        "block": 0.0,           # Below 0.60, block entirely
    }

    def route(self, action, confidence, risk_level):
        if risk_level == "critical":
            return "approval_required"  # Always require approval

        if confidence >= self.THRESHOLDS["auto_execute"]:
            return "auto_execute"
        elif confidence >= self.THRESHOLDS["soft_review"]:
            return "soft_review" if risk_level == "low" else "approval_required"
        elif confidence >= self.THRESHOLDS["approval_required"]:
            return "approval_required"
        else:
            return "block"
```

### Approval Queue System

```typescript
interface ApprovalRequest {
  id: string;
  agentId: string;
  action: string;
  details: Record<string, unknown>;
  riskLevel: "low" | "medium" | "high" | "critical";
  confidence: number;
  createdAt: string;
  expiresAt: string;
  status: "pending" | "approved" | "rejected" | "expired";
}

class ApprovalQueue {
  async submit(request: Omit<ApprovalRequest, "id" | "status">): Promise<ApprovalRequest> {
    const approval: ApprovalRequest = {
      ...request,
      id: crypto.randomUUID(),
      status: "pending",
    };
    await this.store.save(approval);
    await this.notify(approval);  // Slack, email, dashboard
    return approval;
  }

  async waitForDecision(id: string, timeoutMs = 300000): Promise<Decision> {
    const deadline = Date.now() + timeoutMs;
    while (Date.now() < deadline) {
      const approval = await this.store.get(id);
      if (approval.status !== "pending") {
        return {
          approved: approval.status === "approved",
          approver: approval.reviewedBy,
          reason: approval.reviewNote,
        };
      }
      await sleep(1000);
    }
    // Auto-expire if no response
    await this.store.update(id, { status: "expired" });
    return { approved: false, reason: "Approval request expired" };
  }
}
```

### Feedback Loop Integration

```python
class HITLFeedbackLoop:
    """Capture human decisions to improve future AI confidence."""

    def record_decision(self, request_id, ai_recommendation, human_decision):
        self.feedback_store.append({
            "request_id": request_id,
            "ai_said": ai_recommendation,
            "human_said": human_decision,
            "agreed": ai_recommendation == human_decision,
            "timestamp": datetime.utcnow().isoformat(),
        })

    def get_accuracy_stats(self, action_type, window_days=30):
        """Track how often AI recommendations match human decisions."""
        recent = self.feedback_store.query(
            action_type=action_type,
            since=datetime.utcnow() - timedelta(days=window_days),
        )
        agreed = sum(1 for r in recent if r["agreed"])
        return {
            "total_reviews": len(recent),
            "agreement_rate": agreed / len(recent) if recent else 0,
            "can_auto_approve": (agreed / len(recent)) > 0.95 if len(recent) > 50 else False,
        }
```

## Best Practices

1. **Default to requiring approval** for new action types — relax as trust builds
2. **Set timeouts on approval requests** — stale approvals should expire, not block forever
3. **Log everything** — every approval, rejection, and auto-execution for audit trails
4. **Provide context to reviewers** — show what the AI saw and why it made its recommendation
5. **Build escalation paths** — if a reviewer is unavailable, route to backup
6. **Never bypass HITL for compliance-required actions** — even if confidence is 100%
7. **Make approvals async** — don't block users waiting for approvals; notify when decided

## Resources

- [Anthropic: Building Safe AI Systems](https://docs.anthropic.com/en/docs/)
- [NIST AI Risk Management Framework](https://www.nist.gov/artificial-intelligence)
- [EU AI Act Compliance Requirements](https://artificialintelligenceact.eu/)

## Changelog

| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2026-03-31 | Initial documentation |