Vision Analytics

Vision Analytics Skill

Use this skill for planned advanced visibility metrics and analytics (mesh/visibility/gaze). As of 2025-12-13 this is docs/config scaffolding only.

When to Use

• Need detailed visibility breakdown (not just visible/not visible)
• Analyzing gaze patterns in an episode
• Debugging visibility fraction calculations
• Understanding regional face occlusion
• Screen time breakdown by visibility type

Sub-agents

Key Skills

compute_visibility_labels() (planned)

Compute per-frame visibility metrics (not implemented yet).

# Planned module: FEATURES/vision_analytics/src/visibility.py
# from FEATURES.vision_analytics.src.visibility import compute_visibility

# Planned API:
# result = compute_visibility(mesh=face_mesh_result, face_bbox=bbox, frame_shape=(1080, 1920))

generate_screen_time_breakdown() (planned)

Generate face vs body timeline for an identity (not implemented yet).

# Planned module: FEATURES/vision_analytics/src/timeline.py
# from FEATURES.vision_analytics.src.timeline import generate_screen_time_breakdown

# Planned API:
# breakdown = generate_screen_time_breakdown(identity=identity, face_segments=face_timeline, body_segments=body_timeline)

Config Reference

File: config/pipeline/analytics.yaml

Visibility Fraction

Visibility fraction indicates how much of the face is visible:

Regional Breakdown

Face regions tracked:

Gaze Direction

Coarse gaze categories:

Common Issues

Low visibility on frontals

Cause: Mesh extraction failing or thresholds too strict

Check: mesh_confidence in diagnostic

Fix: Lower detection confidence:

face_mesh:
  confidence:
    min_detection: 0.4  # default is 0.5

Gaze always "center"

Cause: Iris landmarks not detected

Check: use_iris setting and iris landmark presence

Fix: Ensure iris refinement enabled:

gaze:
  use_iris: true

Mesh not running on faces

Cause: Faces not classified as close-ups

Check: Face area vs closeup_threshold

Fix: Lower threshold or increase sample rate:

face_mesh:
  execution:
    closeup_threshold: 0.03  # default is 0.05
    sample_rate: 15  # default is 30

Diagnostic Output

{
  "face_id": "F_42_100",
  "visibility": {
    "overall_fraction": 0.85,
    "left_eye_visible": 1.0,
    "right_eye_visible": 0.7,
    "nose_visible": 1.0,
    "mouth_visible": 0.9,
    "forehead_visible": 0.8,
    "chin_visible": 0.75,
    "occluded_regions": ["right_eye"],
    "occlusion_type": "partial"
  },
  "gaze": {
    "horizontal": "center",
    "vertical": "center",
    "yaw_degrees": 5.2,
    "pitch_degrees": -3.1,
    "looking_at_camera": true,
    "confidence": 0.82
  },
  "mesh": {
    "landmarks_count": 468,
    "confidence": 0.91
  }
}

Key Files

Related Skills

• face-alignment - Alignment quality
• body-tracking - Body visibility