Video Intelligence

Video Intelligence — Autonomous Video Editing

AI-powered pipeline that watches videos like a human, finds the best moments, and edits automatically.

Tool Stack

Environment Variables Needed

export HUME_API_KEY="your-hume-api-key"
export TWELVE_LABS_API_KEY="your-twelve-labs-api-key"

The Autonomous Pipeline

Input: Raw video file
         ↓
Step 1: PySceneDetect → Find all scene boundaries
         ↓
Step 2: Whisper → Transcribe audio with timestamps
         ↓
Step 3: Hume AI → Analyze emotion peaks (excitement, anger, surprise)
         ↓
Step 4: Twelve Labs → Index video, find key moments by description
         ↓
Step 5: Claude → Score moments, select best hooks (HSP + BEAR)
         ↓
Step 6: FFmpeg → Cut clips at selected timestamps
         ↓
Step 7: MediaPipe → Track faces, calculate crop coordinates
         ↓
Step 8: FFmpeg → Apply 9:16 crop with face tracking
         ↓
Step 9: Remotion → Add subtitles (emotion-styled)
         ↓
Output: Finished clips ready to post

Quick Commands

Detect Scenes

scenedetect -i video.mp4 detect-content list-scenes

Find Emotion Peaks (Hume)

from hume import HumeClient
from hume.models.config import ProsodyConfig

client = HumeClient(api_key=os.environ["HUME_API_KEY"])

# Analyze audio for emotion
job = client.expression_measurement.batch.start_inference_job(
    files=["audio.wav"],
    models=[ProsodyConfig()]
)

# Get results - find peaks of excitement, surprise, anger
results = client.expression_measurement.batch.get_job_predictions(job.job_id)

Search Video Content (Twelve Labs)

from twelvelabs import TwelveLabsClient

client = TwelveLabsClient(api_key=os.environ["TWELVE_LABS_API_KEY"])

# Create index and upload video
index = client.index.create(name="my-videos", engines=["marengo2.6"])
task = client.task.create(index_id=index.id, file="video.mp4")

# Search for moments
results = client.search.query(
    index_id=index.id,
    query="when the speaker gets excited and raises voice",
    search_options=["visual", "audio"]
)

# Returns timestamps of matching moments
for clip in results.data:
    print(f"{clip.start} - {clip.end}: {clip.score}")

Auto-Reframe with Face Tracking

import cv2
import mediapipe as mp

mp_face = mp.solutions.face_detection
face_detection = mp_face.FaceDetection(min_detection_confidence=0.5)

def get_face_center(frame):
    """Get center coordinates of detected face."""
    results = face_detection.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    if results.detections:
        bbox = results.detections[0].location_data.relative_bounding_box
        center_x = bbox.xmin + bbox.width / 2
        center_y = bbox.ymin + bbox.height / 2
        return center_x, center_y
    return 0.5, 0.5  # Default to center

def calculate_crop(frame_width, frame_height, face_center_x):
    """Calculate 9:16 crop coordinates centered on face."""
    target_ratio = 9 / 16
    crop_width = int(frame_height * target_ratio)

    # Center crop on face
    center_pixel = int(face_center_x * frame_width)
    x_start = max(0, center_pixel - crop_width // 2)
    x_end = min(frame_width, x_start + crop_width)

    # Adjust if we hit edges
    if x_end - x_start < crop_width:
        x_start = max(0, x_end - crop_width)

    return x_start, x_end

Full Autonomous Edit Command

# Run complete pipeline
python3 ~/.claude/skills/video-intelligence/auto_edit.py \
    --input video.mp4 \
    --output clips/ \
    --target-duration 60 \
    --style vertical

Integration with NRVES Pipeline

This skill enhances the existing NRVES pipeline:

What Gets Automated

API Costs (Estimates)

Setup Checklist

• PySceneDetect installed
• OpenCV installed
• MediaPipe installed
• Hume AI API key set
• Twelve Labs API key set
• Test on sample video

Created: 2026-02-06 Layer: Machine Status: Ready (pending API keys)