sharpai HomeSafe-Bench — Indoor Safety Hazard Detection
VLM benchmark evaluating indoor home safety hazard detection from static camera frames. Inspired by HomeSafeBench (arXiv 2509.23690), adapted for fixed indoor security cameras with AI-generated fixture images.
40 tests across 5 hazard categories:
| Category | Tests | Example Hazards |
|---|---|---|
| 🔥 Fire / Smoke | 8 | Stove smoke, candle near curtain, iron left face-down |
| ⚡ Electrical | 8 | Overloaded power strip, exposed wire, frayed cord |
| 🚶 Trip / Fall | 8 | Toys on staircase, wet floor, loose rug edge |
| 👶 Child Safety | 8 | Open cabinet with chemicals, uncovered outlet, blind cord |
| 📦 Falling Objects | 8 | Heavy items on high shelf, leaning bookshelf, loose ceiling fan |
Setup
Requires npm install. This skill has a package.json with dependencies (e.g. openai SDK). The deployment agent should check for package.json and run npm install in the skill directory before launching.
Entry script: scripts/run-benchmark.cjs
Verification
node scripts/run-benchmark.cjs --help
Quick Start
As an Aegis Skill (automatic)
When spawned by Aegis, all configuration is injected via environment variables. The benchmark discovers your VLM server automatically, generates an HTML report, and opens it when complete.
Standalone
# Run all 40 tests
node scripts/run-benchmark.cjs --vlm http://localhost:5405
# Quick mode (2 tests per category = 10 total)
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --mode quick
# Skip report auto-open
node scripts/run-benchmark.cjs --vlm http://localhost:5405 --no-open
Configuration
Environment Variables (set by Aegis)
| Variable | Default | Description |
|---|---|---|
AEGIS_VLM_URL |
(required) | VLM server base URL |
AEGIS_VLM_MODEL |
— | Loaded VLM model ID |
AEGIS_SKILL_ID |
— | Skill identifier (enables skill mode) |
AEGIS_SKILL_PARAMS |
{} |
JSON params from skill config |
Note: URLs should be base URLs (e.g.
http://localhost:5405). The benchmark appends/v1/chat/completionsautomatically.
User Configuration (config.yaml)
| Parameter | Type | Default | Description |
|---|---|---|---|
mode |
select | full |
Which mode: full (40 tests) or quick (10 tests — 2 per category) |
noOpen |
boolean | false |
Skip auto-opening the HTML report in browser |
CLI Arguments (standalone fallback)
| Argument | Default | Description |
|---|---|---|
--vlm URL |
(required) | VLM server base URL |
--mode MODE |
full |
Test mode: full or quick |
--out DIR |
~/.aegis-ai/homesafe-benchmarks |
Results directory |
--no-open |
— | Don't auto-open report in browser |
Protocol
Aegis → Skill (env vars)
AEGIS_VLM_URL=http://localhost:5405
AEGIS_SKILL_ID=homesafe-bench
AEGIS_SKILL_PARAMS={}
Skill → Aegis (stdout, JSON lines)
{"event": "ready", "vlm": "SmolVLM-500M", "system": "Apple M3"}
{"event": "suite_start", "suite": "🔥 Fire / Smoke"}
{"event": "test_result", "suite": "...", "test": "...", "status": "pass", "timeMs": 4500}
{"event": "suite_end", "suite": "...", "passed": 7, "failed": 1}
{"event": "complete", "passed": 36, "total": 40, "timeMs": 180000, "reportPath": "/path/to/report.html"}
Human-readable output goes to stderr (visible in Aegis console tab).
Citation
This benchmark is inspired by:
HomeSafeBench: Towards Measuring the Proficiency of Home Safety for Embodied AI Agents arXiv:2509.23690
Unlike the academic benchmark (embodied agent + navigation in simulated 3D environments), our version uses static indoor camera frames — matching real-world indoor security camera deployment (fixed wall/ceiling mount). All fixture images are AI-generated consistent with DeepCamera's privacy-first approach.
Requirements
- Node.js ≥ 18
npm install(foropenaiSDK dependency)- Running VLM server (llama-server with vision model, or OpenAI-compatible VLM endpoint)