Vision Tagger
macOS-native image analysis using Apple's Vision framework. All processing is local — no cloud APIs, no API keys needed.
Requirements
- macOS 12+ (Monterey or later)
- Xcode Command Line Tools
- Python 3 with Pillow
Setup (one-time)
# Install Xcode CLI tools if needed
xcode-select --install
# Install Pillow
pip3 install Pillow
# Compile the Swift binary
cd scripts/
swiftc -O -o image_tagger image_tagger.swift
Usage
Analyze image → JSON
./scripts/image_tagger /path/to/photo.jpg
Output includes:
faces — bounding boxes, roll/yaw/pitch, landmarks (eyes, nose, mouth)bodies — 18 skeleton joints with confidence scoreshands — 21 joints per hand (left/right)text — OCR results with bounding boxeslabels — scene classification (desk, outdoor, clothing, etc.)barcodes — QR codes, UPC, etc.saliency — attention and objectness regions
Annotate image with boxes
python3 scripts/annotate_image.py photo.jpg output.jpg
Draws colored boxes:
- 🟢 Green: faces
- 🟠 Orange: body skeleton
- 🟣 Magenta: hands
- 🔵 Cyan: text regions
- 🟡 Yellow: rectangles/objects
- Scene labels at bottom
Python integration
import subprocess, json
def analyze(path):
r = subprocess.run(['./scripts/image_tagger', path], capture_output=True, text=True)
return json.loads(r.stdout[r.stdout.find('{'):])
tags = analyze('photo.jpg')
print(tags['labels']) # [{'label': 'desk', 'confidence': 0.85}, ...]
print(tags['faces']) # [{'bbox': {...}, 'confidence': 0.99, 'yaw': 5.2}]
Example JSON Output
{
"dimensions": {"width": 1920, "height": 1080},
"faces": [{"bbox": {"x": 0.3, "y": 0.4, "width": 0.15, "height": 0.2}, "confidence": 0.99, "roll": -2, "yaw": 5}],
"bodies": [{"joints": {"head_joint": {"x": 0.5, "y": 0.7, "confidence": 0.9}, "left_shoulder": {...}}, "confidence": 1}],
"hands": [{"chirality": "left", "joints": {"VNHLKWRI": {"x": 0.4, "y": 0.3, "confidence": 0.85}}}],
"text": [{"text": "HELLO", "confidence": 0.95, "bbox": {...}}],
"labels": [{"label": "outdoor", "confidence": 0.88}, {"label": "sky", "confidence": 0.75}],
"saliency": {"attentionBased": [{"x": 0.2, "y": 0.1, "width": 0.6, "height": 0.8}]}
}
Detection Capabilities
| Feature |
Details |
| Faces |
Bounding box, confidence, roll/yaw/pitch angles, 76-point landmarks |
| Bodies |
18 joints: head, neck, shoulders, elbows, wrists, hips, knees, ankles |
| Hands |
21 joints per hand, left/right chirality |
| Text (OCR) |
Recognized text with confidence and bounding boxes |
| Labels |
1000+ scene/object categories (clothing, furniture, outdoor, etc.) |
| Barcodes |
QR, UPC, EAN, Code128, PDF417, Aztec, DataMatrix |
| Saliency |
Attention-based and objectness-based regions |
Use Cases
- Photo tagging — Auto-tag photos with detected objects/scenes
- Posture monitoring — Track face/body position for ergonomics
- Document scanning — Extract text from images
- Security — Detect people in camera feeds
- Accessibility — Describe image contents
1---2name: vision-tagger3description: Tag and annotate images using Apple Vision framework (macOS only). Detects faces, bodies, hands, text (OCR), barcodes, objects, scene labels, and saliency regions. Use for image analysis, photo tagging, posture monitoring, or any task requiring computer vision on images.4---5
6# Vision Tagger
7
8macOS-native image analysis using Apple's Vision framework. All processing is local — no cloud APIs, no API keys needed.
9
10## Requirements
11
12- macOS 12+ (Monterey or later)
13- Xcode Command Line Tools
14- Python 3 with Pillow
15
16## Setup (one-time)
17
18```bash
19# Install Xcode CLI tools if needed
20xcode-select --install
21
22# Install Pillow
23pip3 install Pillow
24
25# Compile the Swift binary
26cd scripts/
27swiftc -O -o image_tagger image_tagger.swift
28```
29
30## Usage
31
32### Analyze image → JSON
33
34```bash
35./scripts/image_tagger /path/to/photo.jpg
36```
37
38Output includes:
39- `faces` — bounding boxes, roll/yaw/pitch, landmarks (eyes, nose, mouth)
40- `bodies` — 18 skeleton joints with confidence scores
41- `hands` — 21 joints per hand (left/right)
42- `text` — OCR results with bounding boxes
43- `labels` — scene classification (desk, outdoor, clothing, etc.)
44- `barcodes` — QR codes, UPC, etc.
45- `saliency` — attention and objectness regions
46
47### Annotate image with boxes
48
49```bash
50python3 scripts/annotate_image.py photo.jpg output.jpg
51```
52
53Draws colored boxes:
54- 🟢 Green: faces
55- 🟠 Orange: body skeleton
56- 🟣 Magenta: hands
57- 🔵 Cyan: text regions
58- 🟡 Yellow: rectangles/objects
59- Scene labels at bottom
60
61### Python integration
62
63```python
64import subprocess, json
65
66def analyze(path):
67 r = subprocess.run(['./scripts/image_tagger', path], capture_output=True, text=True)
68 return json.loads(r.stdout[r.stdout.find('{'):])
69
70tags = analyze('photo.jpg')
71print(tags['labels']) # [{'label': 'desk', 'confidence': 0.85}, ...]
72print(tags['faces']) # [{'bbox': {...}, 'confidence': 0.99, 'yaw': 5.2}]
73```
74
75## Example JSON Output
76
77```json
78{
79 "dimensions": {"width": 1920, "height": 1080},
80 "faces": [{"bbox": {"x": 0.3, "y": 0.4, "width": 0.15, "height": 0.2}, "confidence": 0.99, "roll": -2, "yaw": 5}],
81 "bodies": [{"joints": {"head_joint": {"x": 0.5, "y": 0.7, "confidence": 0.9}, "left_shoulder": {...}}, "confidence": 1}],
82 "hands": [{"chirality": "left", "joints": {"VNHLKWRI": {"x": 0.4, "y": 0.3, "confidence": 0.85}}}],
83 "text": [{"text": "HELLO", "confidence": 0.95, "bbox": {...}}],
84 "labels": [{"label": "outdoor", "confidence": 0.88}, {"label": "sky", "confidence": 0.75}],
85 "saliency": {"attentionBased": [{"x": 0.2, "y": 0.1, "width": 0.6, "height": 0.8}]}
86}
87```
88
89## Detection Capabilities
90
91| Feature | Details |
92|---------|---------|
93| Faces | Bounding box, confidence, roll/yaw/pitch angles, 76-point landmarks |
94| Bodies | 18 joints: head, neck, shoulders, elbows, wrists, hips, knees, ankles |
95| Hands | 21 joints per hand, left/right chirality |
96| Text (OCR) | Recognized text with confidence and bounding boxes |
97| Labels | 1000+ scene/object categories (clothing, furniture, outdoor, etc.) |
98| Barcodes | QR, UPC, EAN, Code128, PDF417, Aztec, DataMatrix |
99| Saliency | Attention-based and objectness-based regions |
100
101## Use Cases
102
103- **Photo tagging** — Auto-tag photos with detected objects/scenes
104- **Posture monitoring** — Track face/body position for ergonomics
105- **Document scanning** — Extract text from images
106- **Security** — Detect people in camera feeds
107- **Accessibility** — Describe image contents
johnalbertini14-glitch