Photo Content Recognition Curation Expert

Photo Content Recognition & Curation Expert

Expert in photo content analysis and intelligent curation. Combines classical computer vision with modern deep learning for comprehensive photo analysis.

Decision Points

PHOTO LIBRARY SIZE?
├─ <1K photos → Single-pass processing
│   ├─ dHash for duplicates (fastest)
│   └─ Agglomerative clustering (known thresholds)
├─ 1K-10K photos → Optimized pipeline
│   ├─ pHash → DINOHash hybrid
│   └─ HDBSCAN if variance in cluster sizes >3x
└─ >10K photos → Batch processing
    ├─ GPU batching (batch_size=32)
    └─ Incremental updates only

CLUSTERING ALGORITHM CHOICE?
├─ Face variance known? → Agglomerative (threshold=0.4, 0.6)
├─ Unknown distribution? → HDBSCAN (auto-threshold)
└─ Real-time additions? → Incremental clustering

DUPLICATE DETECTION ROBUSTNESS?
├─ Exact duplicates only? → dHash (Hamming ≤3)
├─ Brightness/contrast edits? → pHash (Hamming ≤5)
├─ Heavy crops/compression? → DINOHash (Hamming ≤10)
└─ Production system? → Hybrid (pHash→DINOHash)

BURST PHOTO SELECTION?
├─ Face quality available? → Weight faces 35%, sharpness 30%
├─ No faces detected? → Weight sharpness 50%, aesthetics 30%
└─ Action photos? → Weight sharpness 60%, exposure 25%

CONTENT FILTERING CONFIDENCE?
├─ Screenshot confidence >0.6? → Filter out
├─ NSFW confidence >0.8? → Filter out  
├─ Face confidence <0.9? → Exclude from clustering
└─ Pet confidence <0.7? → Manual review

Failure Modes

1. Too Many Face Clusters (Symptom: 100+ clusters for 1000 photos)

Root Cause: Clustering threshold too strict, splitting same person into multiple clusters Detection Rule: If clusters/unique_people ratio > 2.0, threshold too conservative Fix:

• Increase distance threshold from 0.4 to 0.6
• Run second-pass merging with relaxed threshold
• Verify with manual spot-check of 5 random cluster pairs

2. False Duplicate Detection (Symptom: Different people flagged as duplicates)

Root Cause: Perceptual hash threshold too loose, faces in similar poses triggering matches Detection Rule: If duplicate groups contain faces with cosine distance >0.7, hash threshold too aggressive Fix:

• Tighten Hamming threshold from 10 to 5 bits
• Add face verification step: extract face embeddings from "duplicates"
• If face similarity <0.5, not duplicates

3. Burst Selection Missing Best Frame (Symptom: Blurry photos selected over sharp ones)

Root Cause: Face quality override masking sharpness issues when faces poorly detected Detection Rule: If selected burst frame Laplacian variance <500 while other frames >1000 Fix:

• Cap face quality bonus at 20% total score
• Add face detection confidence check (>0.9 required for face bonus)
• Fallback to pure sharpness if no high-confidence faces

4. Screenshot False Negatives (Symptom: UI screenshots in final curated set)

Root Cause: Single-signal detection missing edge cases (dark mode, custom UI) Detection Rule: If photos contain perfect rectangles + high text density but confidence <0.6 Fix:

• Lower combined threshold from 0.6 to 0.5 for multi-signal hits
• Add aspect ratio check for common device screens (16:9, 19.5:9, 4:3)
• OCR confidence boost for text >20% coverage

5. Memory Explosion on Large Libraries (Symptom: >16GB RAM usage on 10K photos)

Root Cause: Loading all images into memory simultaneously instead of streaming Detection Rule: If memory usage >2GB per 1000 photos, batch size too large Fix:

• Reduce batch size from 64 to 16 images
• Implement lazy loading with generators
• Cache only embeddings/hashes, not raw images

Worked Examples

Example 1: 500-Photo Family Library Curation

Scenario: Family vacation photos (500 images), many bursts and similar poses, want top 100 for sharing.

Step 1 - Library Analysis:

• Photo count: 500 → Use pHash→DINOHash hybrid
• Face variance unknown → HDBSCAN clustering
• Mix of portraits/landscapes → Multi-criteria burst selection

Step 2 - Duplicate Detection:

# Stage 1: Fast pHash filtering
pHash_duplicates = find_duplicates_phash(photos, threshold=5)
# Found 45 potential duplicate groups

# Stage 2: DINOHash verification  
verified_duplicates = []
for group in pHash_duplicates:
    dino_hashes = [compute_dinohash(img) for img in group]
    if hamming_distance(dino_hashes[0], dino_hashes[1]) <= 8:
        verified_duplicates.append(group)
# Verified 38 true duplicate groups

Step 3 - Face Clustering Decision:

• Extract face embeddings: 234 faces detected
• Check cluster size variance: max_size/min_size = 4.2 (>3x threshold)
• Choose HDBSCAN over agglomerative

Step 4 - Burst Selection Trade-offs:

• Detected 12 burst sequences (3-8 photos each)
• Burst #3: 5 photos of kids playing
  • Frame 2: Highest sharpness (1850 Laplacian) but no faces detected
  • Frame 4: Lower sharpness (1200) but 2 clear faces, both smiling
  • Decision: Frame 4 selected (face quality 35% weight > sharpness 30%)

Final Output: 387 unique photos → 100 curated (top aesthetic + diversity)

Example 2: Burst Selection Edge Case - Blurry + Exposed Face

Scenario: Wedding ceremony burst of 15 shots, bride's face overexposed in sharpest frames.

Challenge: Best technical shot (frame 8) has bride's face blown out by direct sunlight.

Analysis Process:

# Frame 8: Sharpness winner but face quality fail
frame_8_metrics = {
    'sharpness': 2100,      # Highest in burst
    'face_quality': 0.2,    # Overexposed face detection
    'aesthetics': 0.75,     # Good composition 
    'exposure': 0.3         # Highlights clipped
}

# Frame 11: Balanced option
frame_11_metrics = {
    'sharpness': 1650,      # Still acceptable  
    'face_quality': 0.9,    # Clear face, good exposure
    'aesthetics': 0.8,      # Slightly better angle
    'exposure': 0.85        # Well exposed
}

# Weighted scoring:
frame_8_score = 2100*0.3 + 0.2*0.35 + 0.75*0.2 + 0.3*0.05 = 630 + 0.07 + 0.15 + 0.015 = 630.235
frame_11_score = 1650*0.3 + 0.9*0.35 + 0.8*0.2 + 0.85*0.05 = 495 + 0.315 + 0.16 + 0.0425 = 495.518

Expert Insight: Frame 11 selected despite lower sharpness because face quality penalty (-80%) outweighs sharpness advantage. Novice would pick frame 8 based on technical metrics alone.

Quality Gates

Photo Indexing Complete:

• All EXIF metadata extracted and cached
• Perceptual hashes computed for 100% of images
• Face embeddings extracted (confidence >0.9 only)
• CLIP embeddings generated for semantic search
• Index file saved and can be reloaded without errors

Duplicate Detection Complete:

• Hamming distance threshold validated (manual check 10 random pairs)
• Duplicate groups reviewed (no false positives in sample)
• Performance acceptable (<2 min per 1000 photos)
• Original files preserved (only flagging, not deleting)

Face Clustering Complete:

• Cluster count reasonable (max 1 cluster per 50 photos)
• No singleton clusters with confidence >0.95 faces
• Manual verification: 5 random clusters contain same person
• Noise properly classified (face confidence <0.9)

Burst Selection Complete:

• All burst sequences identified (timestamp gap <0.5s)
• One photo selected per burst (highest weighted score)
• Face quality bonus only applied when face confidence >0.9
• Manual spot-check: 3 burst selections look correct

Content Filtering Complete:

• Screenshot detection tested on known UI captures (>95% accuracy)
• NSFW filter validated on test set (no false positives on family photos)
• Pet detection covers common animals (cat, dog, horse, bird)
• Confidence thresholds produce <5% manual review rate

NOT-FOR Boundaries

This skill should NOT be used for:

• GPS-based location clustering → Use event-detection-temporal-intelligence-expert instead

  • Reason: Requires temporal event modeling and GPS clustering algorithms

• Color palette extraction and harmony → Use color-theory-palette-harmony-expert instead

  • Reason: Needs color theory knowledge and palette composition rules

• Semantic image-text matching → Use clip-aware-embeddings instead

  • Reason: Requires cross-modal understanding and text-image alignment

• Video analysis or frame extraction → Use video processing specialist instead

  • Reason: Different pipeline for temporal sequences and video codecs

• Advanced photo editing → Use image editing specialist instead

  • Reason: Focus is on analysis/curation, not pixel-level manipulation

• 3D scene reconstruction → Use computer vision specialist instead

  • Reason: Requires stereo vision and depth estimation techniques

Delegation Rules:

• If task involves time/location → event-detection-temporal-intelligence-expert
• If task involves colors/aesthetics → color-theory-palette-harmony-expert
• If task involves text matching → clip-aware-embeddings
• If task involves editing → Image editing specialist