Human-in-the-Loop
Human-in-the-loop design defines when, where, and how humans intervene in automated workflows. Too little human involvement and the system makes dangerous mistakes. Too much and you've just built an expensive notification system.
Intervention Types
- Approval gates: The system pauses and waits for human approval before proceeding
- Review checkpoints: The system presents results for human review but can continue if no objection
- Correction opportunities: The system shows its work and the human can edit before it's finalised
- Override controls: The human can stop, redirect, or undo the system's actions at any time
- Monitoring dashboards: The human passively observes the system and intervenes only when needed
When to Require Human Intervention
- High stakes: Actions that are expensive, irreversible, or affect many people
- Low confidence: The system is uncertain about the right action
- Novel situations: The input or context is outside the system's training distribution
- Ethical judgments: Decisions that require moral reasoning or value trade-offs
- Legal requirements: Regulatory or compliance requirements mandate human review
- User request: The user explicitly asks for human involvement
Designing Intervention Points
For each intervention point:
- Trigger: What causes the intervention? (confidence threshold, stakes level, user request, policy requirement)
- Presentation: What does the human see? (summary, full context, recommendations, options)
- Time constraint: How quickly must the human respond? What happens if they don't?
- Decision options: What can the human do? (approve, reject, edit, escalate, defer)
- Feedback integration: How does the human's decision feed back into the system?
Avoiding Human Bottlenecks
Human intervention is expensive and slow. Design to minimise unnecessary intervention:
- Graduated autonomy: Start with more human oversight, reduce as the system proves reliable
- Batch review: Group similar decisions for efficient human processing
- Smart routing: Send interventions to the right human based on expertise and availability
- Default actions: If the human doesn't respond within a time window, take a safe default action
- Learning from interventions: Use human decisions to improve the system so it needs less intervention over time
Design Artefacts
- Intervention point map (where in the workflow, what triggers each)
- Intervention interface designs (what the human sees and can do)
- Time constraint specifications
- Graduated autonomy plans
- Intervention analytics specifications (tracking how often, why, and what humans decide)
1---2name: human-in-the-loop3description: Designing intervention points where humans review, approve, or redirect agent work.4---5# Human-in-the-Loop
6Human-in-the-loop design defines when, where, and how humans intervene in automated workflows. Too little human involvement and the system makes dangerous mistakes. Too much and you've just built an expensive notification system.
7## Intervention Types
8- **Approval gates**: The system pauses and waits for human approval before proceeding
9- **Review checkpoints**: The system presents results for human review but can continue if no objection
10- **Correction opportunities**: The system shows its work and the human can edit before it's finalised
11- **Override controls**: The human can stop, redirect, or undo the system's actions at any time
12- **Monitoring dashboards**: The human passively observes the system and intervenes only when needed
13## When to Require Human Intervention
14- **High stakes**: Actions that are expensive, irreversible, or affect many people
15- **Low confidence**: The system is uncertain about the right action
16- **Novel situations**: The input or context is outside the system's training distribution
17- **Ethical judgments**: Decisions that require moral reasoning or value trade-offs
18- **Legal requirements**: Regulatory or compliance requirements mandate human review
19- **User request**: The user explicitly asks for human involvement
20## Designing Intervention Points
21For each intervention point:
22- **Trigger**: What causes the intervention? (confidence threshold, stakes level, user request, policy requirement)
23- **Presentation**: What does the human see? (summary, full context, recommendations, options)
24- **Time constraint**: How quickly must the human respond? What happens if they don't?
25- **Decision options**: What can the human do? (approve, reject, edit, escalate, defer)
26- **Feedback integration**: How does the human's decision feed back into the system?
27## Avoiding Human Bottlenecks
28Human intervention is expensive and slow. Design to minimise unnecessary intervention:
29- **Graduated autonomy**: Start with more human oversight, reduce as the system proves reliable
30- **Batch review**: Group similar decisions for efficient human processing
31- **Smart routing**: Send interventions to the right human based on expertise and availability
32- **Default actions**: If the human doesn't respond within a time window, take a safe default action
33- **Learning from interventions**: Use human decisions to improve the system so it needs less intervention over time
34## Design Artefacts
35- Intervention point map (where in the workflow, what triggers each)
36- Intervention interface designs (what the human sees and can do)
37- Time constraint specifications
38- Graduated autonomy plans
39- Intervention analytics specifications (tracking how often, why, and what humans decide)
gabrielmoreira