Sensing Track

Sensing Event History

Quick Start

The primary data source is the flow events JSONL at /root/local/flow_events_YYYY-MM-DD.jsonl. Each file covers one calendar day (7-day retention, no size-rotation mid-day). Use Bash + jq to query it.

Important: Always use the absolute path /root/local/ — the read tool cannot access files outside the workspace, so use exec (Bash) for all JSONL queries.

Persistent camera snapshots are stored under /var/lib/hal/snapshots/sensing_<prefix>/<ms>.jpg (72h TTL, 50 MB cap) — one subdir per event kind:

Reference these when the user asks what happened visually.

JSONL format

Each line is a JSON object:

{"kind":"enter","node":"sensing_input","ts":1712345678.123,"seq":42,"trace_id":"run-abc","data":{"type":"presence.enter","message":"Person detected — new: friend (gray); faces in frame: 1 (gray)\n[snapshot: /var/lib/hal/snapshots/sensing_face/1712345678123.jpg]"},"version":"1.2.3"}
{"kind":"exit","node":"sensing_input","ts":1712345678.456,"seq":43,"trace_id":"run-abc","duration_ms":332,"data":{"path":"agent","run_id":"run-abc"},"version":"1.2.3"}

Key fields:

• node — filter on "sensing_input" for sensing events
• kind — "enter" = event received, "exit" = event processed (with duration_ms)
• data.type — event type: presence.enter, presence.leave, motion, motion.activity, sound, light.level, voice, voice_command, emotion.detected, speech_emotion.detected
• data.message — natural-language description; may contain [snapshot: /var/lib/hal/snapshots/sensing_<prefix>/<ms>.jpg]
• data.path — in exit records: "agent" (forwarded), "local" (handled locally), or has "error" key (failed/dropped)
• ts — Unix timestamp (seconds with fractional ms)
• trace_id — correlates enter/exit and links to agent turn

Tools

Bash — jq, cat, date arithmetic. No writes.

Query recipes

Timezone — always set before date arithmetic

export TZ=$(cat /etc/timezone)

All sensing events in a time range

export TZ=$(cat /etc/timezone)
DATE="$(date +%Y-%m-%d)"
FROM_TS=$(date -d "$DATE 22:00:00" +%s)
TO_TS=$(date -d "$DATE 23:59:59" +%s)
jq -c 'select(.node=="sensing_input" and .kind=="enter" and .ts >= '"$FROM_TS"' and .ts <= '"$TO_TS"')' \
  "/root/local/flow_events_${DATE}.jsonl"

Events of a specific type in the last N hours

Use "motion" for raw motion, "motion.activity" for activity analysis (HAL-categorised — bucket names drink/break/celebrate and raw Kinetics sedentary labels like using computer, writing, reading). Most queries want both:

For any relative range, select every local calendar day's file intersecting that range, including yesterday when crossing midnight. Run this range setup and the chosen query in the same Bash call; do not assume variables survive between tool calls. GNU date -d is available on the device.

export TZ=$(cat /etc/timezone)
SINCE=$(date -d "1 hour ago" +%s)
UNTIL=$(date +%s)
first_day=$(date -d "@$SINCE" +%F)
last_day=$(date -d "@$UNTIL" +%F)
range_files=()
range_day=$first_day
while [[ "$range_day" < "$last_day" || "$range_day" == "$last_day" ]]; do
  range_file="/root/local/flow_events_${range_day}.jsonl"
  if [ -r "$range_file" ]; then
    range_files+=("$range_file")
  else
    printf 'History unavailable for %s\n' "$range_day" >&2
  fi
  range_day=$(date -d "$range_day +1 day" +%F)
done
if [ "${#range_files[@]}" -gt 0 ]; then
  jq -c --argjson since "$SINCE" --argjson until "$UNTIL" \
    'select(.node=="sensing_input" and .kind=="enter" and .ts >= $since and .ts <= $until and (.data.type=="motion" or .data.type=="motion.activity"))' \
    "${range_files[@]}"
fi

Any activity in the last N minutes

Use the same range setup above with SINCE=$(date -d "30 minutes ago" +%s) and remove the type predicate from the jq filter. Keep both time bounds and all range_files; even 30 minutes can span two local dates.

Presence events only (who came by)

Names in messages are lowercase (friend (gray)). Use test() with "i" flag for case-insensitive search:

TODAY=$(date +%Y-%m-%d)
# All presence events
jq -c 'select(.node=="sensing_input" and .kind=="enter" and (.data.type=="presence.enter" or .data.type=="presence.leave"))' \
  "/root/local/flow_events_${TODAY}.jsonl"
# Search for a specific person (case-insensitive)
jq -c 'select(.node=="sensing_input" and .kind=="enter" and .data.type=="presence.enter" and (.data.message | test("gray";"i")))' \
  "/root/local/flow_events_${TODAY}.jsonl"

Events spanning multiple days

export TZ=$(cat /etc/timezone)
YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
TODAY=$(date +%Y-%m-%d)
cat "/root/local/flow_events_${YESTERDAY}.jsonl" "/root/local/flow_events_${TODAY}.jsonl" \
  | jq -c 'select(.node=="sensing_input" and .kind=="enter" and .ts >= '"$FROM_TS"' and .ts <= '"$TO_TS"')'

Dropped events (agent was busy)

TODAY=$(date +%Y-%m-%d)
jq -c 'select(.node=="sensing_input" and .kind=="exit" and .data.error != null)' \
  "/root/local/flow_events_${TODAY}.jsonl"

List snapshots (72h TTL — older files may be purged)

Snapshots are bucketed into sensing_face/ (presence), sensing_motion_activity/, sensing_emotion/. Recurse into subdirs:

# Most-recent snapshots across all categories
find /var/lib/hal/snapshots -type f -name '*.jpg' -printf '%T@ %p\n' | sort -rn | head -20 | cut -d' ' -f2-

# Only a specific category
ls -lt /var/lib/hal/snapshots/sensing_motion_activity/ | head -20

Pose buckets (posture history)

Posture snapshots are NOT in /var/lib/hal/snapshots/ — they live in tmp under a per-window bucket layout at /tmp/hal-sensing-snapshots/sensing_pose/buckets/<bucket_id>/. A bucket only exists when a tumbling window closed with bad posture (bad_ratio >= POSE_BAD_RATIO). Kept buckets survive ~2 days (POSE_BUCKET_KEEP_S); windows that didn't fire a nudge are deleted immediately, so the buckets you can see are by definition "bad posture" sessions.

Each kept bucket contains:

• <sample_ts>_<score>.jpg — annotated frame per sample (skeleton overlay + RULA score)
• bucket.json — manifest:
  • bucket_id, window_start_ts, window_end_ts, kept: true
  • summary — same shape as the [posture_summary:] block on motion.activity (bad_ratio, dominant_region, samples, …)
  • samples[] — {ts, score, risk_level, filename, left, right} (per-side RULA body_scores + angles)
  • worst_snapshots[] — pre-selected worst filenames (the ones the device auto-attaches to /dm on posture nudges)

# List recent buckets (newest first)
ls -lt /tmp/hal-sensing-snapshots/sensing_pose/buckets/ | head -10

# Read a specific bucket's manifest
jq . /tmp/hal-sensing-snapshots/sensing_pose/buckets/1779259742/bucket.json

# Buckets that closed in the last 2 hours
find /tmp/hal-sensing-snapshots/sensing_pose/buckets -maxdepth 1 -type d -mmin -120 -name '[0-9]*' | sort

# Worst-frame paths from the latest kept bucket
LATEST=$(ls -t /tmp/hal-sensing-snapshots/sensing_pose/buckets/ | head -1)
jq -r '.worst_snapshots[]' "/tmp/hal-sensing-snapshots/sensing_pose/buckets/${LATEST}/bucket.json" \
  | sed "s|^|/tmp/hal-sensing-snapshots/sensing_pose/buckets/${LATEST}/|"

# Today's bad-posture sessions — bucket id == window_start unix-seconds
TODAY_START=$(date -d "today 00:00" +%s)
for b in /tmp/hal-sensing-snapshots/sensing_pose/buckets/*/bucket.json; do
  jq --arg start "$TODAY_START" 'select((.window_start_ts | floor) >= ($start | tonumber)) | {bucket_id, dominant: .summary.dominant_region, bad_ratio: .summary.bad_ratio, started: .window_start_ts}' "$b"
done

Note: motion.activity event messages contain [pose_bucket: <id>] and [pose_worst: <fn1>,<fn2>,...] markers — parse these out of data.message when you need to map a sensing_input record to its bucket. Markers are present whenever a posture nudge folded in.

Fallback: system log

For detailed debugging or when you need Go-side log context (errors, warnings, lifecycle details), fall back to ${OS_LOG:-/var/log/os-server.log}:

LOG="${OS_LOG:-/var/log/os-server.log}"
sed 's/\x1b\[[0-9;]*m//g' "$LOG" | grep "sensing event received"

The system log uses lumberjack rotation (1 MB cap, 3 backups) — it may miss data during high traffic. Use it only when JSONL doesn't have enough detail, or when investigating bugs.

Mood history

A dedicated mood history log tracks user mood per user. Only the user's emotional state is logged — not system events or device emotions. Each user's mood data lives in their own directory.

Read API:

# Current user's mood history (auto-detects who's present)
curl -s "http://127.0.0.1:5000/api/openclaw/mood-history?date=$(date +%Y-%m-%d)&last=100"

# Specific user's mood history
curl -s "http://127.0.0.1:5000/api/openclaw/mood-history?user=gray&date=$(date +%Y-%m-%d)&last=100"

Write: Follow the Mood skill to log user mood from camera or conversation.

{"ts":1776138500,"seq":1,"hour":10,"mood":"happy","source":"camera","trigger":"laughing"}
{"ts":1776139200,"seq":2,"hour":10,"mood":"stressed","source":"conversation","trigger":"user said feeling overwhelmed"}

Storage: /root/local/users/{name}/mood/YYYY-MM-DD.jsonl (30-day retention).

Device sleep history

Your own sleep — when you went to sleep and woke up, not the user's. Use it for "how many times have you slept?", "when do you usually go to sleep?", "did you sleep while I was out?".

Storage: /root/local/device/sleep/YYYY-MM-DD.jsonl (30-day retention). One line per transition, appended:

{"ts":1758000000.12,"local":"2026-09-16T22:00:00+07:00","tz":"Asia/Ho_Chi_Minh","date":"2026-09-16","hour":22,"event":"sleep","emotion":"sleepy","source":"api"}
{"ts":1758021600.45,"local":"2026-09-17T06:00:00+07:00","tz":"Asia/Ho_Chi_Minh","date":"2026-09-17","hour":6,"event":"wake","emotion":"stretching","source":"button"}

• local — the device's own wall-clock with its UTC offset. Say times to the user from this field, never by converting ts yourself.
• tz — the zone that offset came from. Empty means the device could not resolve its zone and the time is naive: report it as approximate rather than quoting it exactly.
• ts — epoch seconds. Use it for ordering and for durations; it is the only field safe to subtract, because the user can change the zone between two rows.
• event — sleep or wake. Count sleep rows; a sleepy re-sent to an already sleeping device writes nothing, so every row is a real transition.
• source — who caused it: button / touch / MPR121 (someone did it by hand), api (your own [HW:/emotion:sleepy] marker, or the web UI).
• emotion — sleepy going in; stretching or greeting coming out.

export TZ=$(cat /etc/timezone)
# Times slept today
jq -c 'select(.event=="sleep")' "/root/local/device/sleep/$(date +%Y-%m-%d).jsonl" | wc -l

# Times slept over everything retained
cat /root/local/device/sleep/*.jsonl | jq -c 'select(.event=="sleep")' | wc -l

# Usual bedtime hour
cat /root/local/device/sleep/*.jsonl \
  | jq -r 'select(.event=="sleep") | .hour' | sort -n | uniq -c | sort -rn | head -3

# Recent transitions with readable local times
cat /root/local/device/sleep/*.jsonl | jq -c '{local,event,source}' | tail -6

# How long each sleep lasted — pair every sleep with the wake that follows it.
# Durations come from `ts`, the times shown come from `local`.
cat /root/local/device/sleep/*.jsonl | jq -s -r '
  def shown: .local // ((.ts | strftime("%Y-%m-%dT%H:%M:%SZ")) + " (UTC)");
  sort_by(.ts)
  | reduce .[] as $r ({open:null, out:[]};
      if $r.event == "sleep" then .open = $r
      elif .open then .out += [{from: (.open | shown), to: ($r | shown),
                                mins: (($r.ts - .open.ts) / 60 | floor)}] | .open = null
      else . end)
  | .out[] | "\(.from) → \(.to)  (\(.mins) min)"'

Two gaps to report honestly rather than paper over:

• A trailing sleep with no wake after it means the device is still asleep, or was powered off while sleeping. The pairing drops it instead of inventing an end.
• Rows written before local existed have no local time. The shown fallback above prints those in UTC, labelled — do not present a UTC time as the user's local time.

Do not confuse this with action:"sleep" in the wellbeing log — that is the user telling you they are going to bed (see the Habit skill). This file is about your own body.

This file is the only complete record. Flow events carry hw_emotion rows for sleeps your own marker fired, but the button never reaches os-server, so counting from flow_events_*.jsonl silently undercounts and misses exactly the times a person put you to sleep by hand. Count from here.

Nothing before the journal existed can be recovered — it was never written down anywhere. If the files start mid-window, say how far back you can actually see rather than reporting a total as if it were lifetime.

Rules

• Never write to any log file — they are owned by the system.
• Answer conversationally — translate results into natural language. Never dump raw JSON to the user.
• Handle empty results — only when the required files were readable and parsing succeeded, say "I didn't detect any [type] events in that window." Missing, unreadable, expired or malformed history means incomplete evidence; report the gap instead of claiming no activity.
• Mention dropped events when relevant — check exit records with data.error for events the agent missed. Mention it: "There was motion at 10:45 PM but I was mid-conversation and missed it."
• Resolve relative times — translate "last hour", "this morning", "while I was away" into concrete Unix timestamps using date -d before filtering.
• Span multiple days — for questions covering more than today, cat multiple JSONL files together.
• Parse the message field for who/what details — the new: segment carries friend (gray), friend (chloe), stranger (stranger_1); already present: gray (friend) means gray was in frame when someone else arrived; faces in frame: N (...) is the box count of that snapshot; other events: Large movement detected, etc.
• Reference snapshots — when the user asks "what did you see?", extract the [snapshot: ...] path from the message. Path format is /var/lib/hal/snapshots/sensing_<prefix>/<ms>.jpg (category subdir per event kind). Snapshots have 72h TTL — check the file exists before referencing (test -f <path>).
• Posture history — for questions about the user's posture ("how was I sitting this morning?", "show me my worst posture today"), scan /tmp/hal-sensing-snapshots/sensing_pose/buckets/. Only sessions that crossed the bad-ratio threshold survive here, so the bucket list itself answers "when did my posture get bad today?". Read each bucket's bucket.json for summary.dominant_region and summary.bad_ratio, then reference worst_snapshots[] for representative frames.

Examples

Input: "Have you seen anybody between 10pm and midnight?" Action: Resolve which evening the user means, then query presence.enter from 22:00 on that date up to (but not including) 00:00 on the following date, selecting the files for the resolved interval. 12pm means noon, not midnight; clarify an ambiguous request rather than silently changing it. Response: "Yes — I detected a stranger at 10:03 PM and again at 10:07 PM." or "No one came by between 10 PM and midnight."

Input: "Is there any motion in the last hour?" Action: Query data.type in ["motion", "motion.activity"] with SINCE=$(date -d "1 hour ago" +%s). Response: "Yes, I detected large movement 3 times — at 9:29, 9:59, and 10:12." or "No motion in the last hour."

Input: "What happened while I was away?" Action: Ask the user when they left, or find the last presence.leave and query all events after that timestamp. Response: "After around 3 PM — I saw motion at 4:30 PM and again at 5:15 PM. No one was identified though. I have snapshots from those moments if you want to see."

Input: "How was my posture today?" Action: List today's pose buckets and aggregate summary.dominant_region + summary.bad_ratio from each bucket.json. Response: "You had 3 bad-posture sessions today: a neck-flexion one at 10:14 AM (77% bad), another neck stretch at 1:39 PM (100% bad), and one trunk lean at 3:20 PM (62% bad). The worst frames are in the bucket dirs if you want me to pull one up."