Mediapipe Tracking

MediaPipe Tracking

When to Use This Skill

• Tracking human body pose in real-time video
• Detecting and tracking hand gestures or finger positions
• Building face filters, expression analysis, or gaze tracking
• Needing 3D depth information (Z-axis) for body landmarks
• Creating fitness apps, sign language recognition, or virtual avatars

Decision Framework

Which MediaPipe Solution?

What body part do you need to track?
├── Full body pose (standing, sitting, exercising)
│   └── ✅ Pose Estimation — 33 landmarks, 3D (x, y, z)
│
├── Hands and fingers
│   └── ✅ Hand Tracking — 21 landmarks per hand
│
├── Face details (lips, eyes, eyebrows, jawline)
│   └── ✅ Face Mesh — 468 landmarks, 3D
│
└── Multiple tasks simultaneously
    └── ✅ Holistic — combines Pose + Hands + Face

MediaPipe vs YOLO Pose Estimation

Rule of thumb: Need detailed single-person anatomy (hands, face, depth)? → MediaPipe. Need to detect multiple people quickly? → YOLO Pose.

Landmark Counts Reference

Integration Guide

Pose Estimation Setup

import cv2
import mediapipe as mp

mp_pose = mp.solutions.pose
mp_draw = mp.solutions.drawing_utils

cap = cv2.VideoCapture(0)
with mp_pose.Pose(min_detection_confidence=0.5,
                   min_tracking_confidence=0.5) as pose:
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        
        # MediaPipe expects RGB
        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        results = pose.process(rgb)
        
        if results.pose_landmarks:
            mp_draw.draw_landmarks(
                frame, results.pose_landmarks,
                mp_pose.POSE_CONNECTIONS)
        
        cv2.imshow('Pose', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

cap.release()
cv2.destroyAllWindows()

Hand Tracking Setup

mp_hands = mp.solutions.hands

with mp_hands.Hands(max_num_hands=2,
                     min_detection_confidence=0.7) as hands:
    # Same loop pattern as pose
    results = hands.process(rgb)
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            mp_draw.draw_landmarks(
                frame, hand_landmarks,
                mp_hands.HAND_CONNECTIONS)

Face Mesh Setup

mp_face = mp.solutions.face_mesh

with mp_face.FaceMesh(max_num_faces=1,
                       min_detection_confidence=0.5) as face_mesh:
    results = face_mesh.process(rgb)
    if results.multi_face_landmarks:
        for face_landmarks in results.multi_face_landmarks:
            mp_draw.draw_landmarks(
                frame, face_landmarks,
                mp_face.FACEMESH_TESSELATION)

Critical Gotchas

1. MediaPipe Expects RGB, Not BGR

MediaPipe processes RGB images. OpenCV captures BGR. Always convert:

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = pose.process(rgb)

2. Landmark Coordinates Are Normalized (0-1)

Landmark .x and .y are normalized to image dimensions. To get pixel coordinates:

h, w, _ = frame.shape
lm = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER]
pixel_x, pixel_y = int(lm.x * w), int(lm.y * h)

3. Z-Depth Is Relative, Not Absolute

The .z coordinate represents depth relative to the hip center (Pose) or wrist (Hand). It is NOT world-scale distance in meters.

4. Single Person Limitation

MediaPipe Pose processes one person per frame. For multi-person scenarios, use YOLO Pose or run MediaPipe per detected person crop.

5. Use the Context Manager

Always use with statement for proper resource cleanup:

# CORRECT
with mp_pose.Pose() as pose:
    ...

# AVOID
pose = mp_pose.Pose()
# resources may leak

Application Ideas