# Scan Pipeline

> Read before modifying the scan pipeline (server/scan/session_events.py, server/scan/device_handling.py). Covers process_scan()'s call order and why it's load-bearing, the CurrentScan/Events/Sessions/DevicesView relationships, how a session actually closes (there is no close function), and the FIELD_SPECS field-write authority mechanism. Use this when touching device presence, connect/disconnect events, or session/timeline behavior.

- Skill: `netalertx/scan-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add netalertx/scan-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/netalertx/scan-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: netalertx (https://skillmd.com/u/netalertx)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/netalertx/scan-pipeline

---


# Scan Pipeline & Device Presence Lifecycle

## Scope

Covers what happens after a plugin's rows land in `CurrentScan`: presence computation, event generation, session/timeline derivation. Not plugin authoring (manifest, data contract, settings) — see `plugin-development` and `docs/PLUGINS_DEV*.md`. Not the general `*Source` attribution system or SQLite audit triggers — see `database-patterns`; the field-authority section below is the scan-pipeline-local half of that system.

## Core tables and views

- **`CurrentScan`** — ephemeral scratch table. `process_plugin_events()` populates it for any plugin whose `config.json` declares `mapped_to_table`; `process_scan()` deletes all rows at the end of every cycle. A value written to a `CurrentScan` row is not readable in a later cycle — the row is gone by then. See Gotcha 2.
- **`Devices`** — persistent identity + state table.
- **`Events`** — persistent, append-only log of state-transition events (`New Device`, `Connected`, `Down Reconnected`, `Device Down`, `Disconnected`, `IP Changed`). This is the audit trail; `Sessions` is derived from it, not the reverse.
- **`Sessions`** — fully wiped and rebuilt every cycle from `Convert_Events_to_Sessions` (below), not incrementally updated. Treat it as a materialized query result, not a live connection state machine.
- **`Online_History`** — one row per scan cycle, feeds the dashboard's online/offline graph. A rollup of `devPresentLastScan`/`devAlertDown`/`devIsSleeping` counts on `DevicesView` — no state of its own.

## Key views

- **`LatestDeviceScan`** (`server/db/db_upgrade.py`) — `Devices` LEFT JOIN'd to the most recent `CurrentScan` row per `(scanMac, scanSourcePlugin)` pair, via `ROW_NUMBER() OVER (PARTITION BY scanMac, scanSourcePlugin ...)`. `update_devices_data_from_scan()` loops over `DISTINCT scanSourcePlugin` and re-queries this view once per plugin: when two plugins report the same device in one cycle, each contribution is evaluated separately, per field, through the authority mechanism below — they are not merged into one row first.
- **`LatestEventsPerMAC`** — most recent Event per MAC, joined to `Devices` and `CurrentScan`. The "New Connections" query in `insert_events()` uses it to decide whether a device was previously down (→ `Down Reconnected`) or new (→ `Connected`).
- **`Convert_Events_to_Sessions`** — defines "is this device's session still open." There is no `close_session()` function anywhere in this codebase. A session closes as an emergent property: `pair_sessions_events()` sets `evePairEventRowid` on a `New Device`/`Connected`/`Down Reconnected` Event to point at the next `Disconnected`/`Device Down` Event for that MAC; this view sets `sesStillConnected = 1` exactly when that pairing is `NULL`. To close a session, insert the right `Events` row — never mutate `Sessions` directly (the one exception is `create_new_devices()`'s reconnect-insert, in the call order below).
- **`DevicesView`** — adds computed `devIsSleeping`/`devFlapping`/`devStatus` on top of `Devices`. The UI and `insertOnlineHistory()` read presence from this, not the raw `Devices` table.

## `process_scan()` call order (`server/scan/session_events.py`) — order is load-bearing

1. `save_own_device()`, `exclude_ignored_devices()`
2. `insert_events(db)` — runs before presence updates for this cycle. The Down/Disconnected/Connected queries need the *previous* cycle's `devPresentLastScan` to detect a transition. If this ran after the presence update, every query would see the new value and the edge-triggered design would break — firing never, or every cycle.
3. `create_new_devices(db)` — runs before presence updates so a brand-new device gets a `New Device` event, not a `Connected` event (it has no `Devices` row yet for step 2's queries to match). Also has a raw `INSERT INTO Sessions ... sesStillConnected = 1` for existing devices with no open session — the one place outside the `Events`-derived path that writes `Sessions` directly.
4. `update_devices_data_from_scan(db)` — field-level updates for existing devices; see the authority mechanism below.
5. `update_sync_hub_node`, `update_devLastConnection_from_CurrentScan`
6. `update_presence_from_CurrentScan(db)` — sets `devPresentLastScan` from `CurrentScan` for this cycle (step 2 reads this as "previous" on the *next* cycle).
7. `update_devPresentLastScan_based_on_nics(db)` — NIC/parent-child presence aggregation; can override step 6 for parent devices.
8. `update_devPresentLastScan_based_on_force_status(db)` — the user's manual `devForceStatus` override; runs last, wins over everything above.
9. `update_vendors_from_mac`, `update_ipv4_ipv6`, `update_icons_and_types`
10. `pair_sessions_events(db)` — pairs `Events` rows as described above.
11. `create_sessions_snapshot(db)` — `DELETE FROM Sessions; INSERT INTO Sessions SELECT * FROM Convert_Events_to_Sessions`. `Sessions` reflects step 10's pairing from here.
12. `insertOnlineHistory(db)` — dashboard graph rollup.
13. `skip_repeated_notifications(db)`
14. `DELETE FROM CurrentScan` — the table's entire lifetime is one call to `process_scan()`.

## Field-write authority for scan-derived updates

`update_devices_data_from_scan()` (`server/scan/device_handling.py`) does not overwrite fields from whichever plugin ran most recently. Each trackable field is declared once in `FIELD_SPECS` (`scan_col`, `source_col`, a `priority` list of plugin prefixes, optional `allow_override_if_changed`). `can_overwrite_field()` uses that plus `get_plugin_authoritative_settings()` (a plugin's own authority-override setting, if any) to decide, per field per row, whether this plugin's value may replace what's there. The paired `<field>Source` column (`devNameSource`, `devLastIPSource`, etc.) records who currently owns the field. `devMac` is never a target of these updates — it's the join key, not a tracked field — so no scan-derived update can alter a device's identity, only its attributes.

This is the scan-pipeline-local half of a bigger attribution system — see `database-patterns` for `FIELD_SOURCE_MAP`/`server/db/authoritative_handler.py`, the full `*Source` model, and the SQLite triggers that consume it for audit logging. Read both before touching anything that writes a `*Source` column.

## Gotchas

1. **A "presence" check exists in more than one place.** A per-row signal meaning "don't count this as a live sighting" (e.g. `scanPresence`) has to reach every query that independently re-derives "is this MAC currently present" from `CurrentScan`. `current_scan_presence_condition()` (`server/scan/presence.py`) centralizes that check for five sites: `update_presence_from_CurrentScan()` (both statements), `update_devLastConnection_from_CurrentScan()`, and three of `insert_events()`'s four queries (both `Device Down` variants, `Disconnected`). Two sites can't use it: the "New Connections" query and the raw `Sessions` insert in `create_new_devices()` need the actual `scanLastIP`/`scanVendor` value off the presence-asserting row via `MIN()`/`GROUP BY`, not just a boolean. Check any new presence-adjacent query against both patterns — a bare helper call isn't always enough.
2. **`CurrentScan` is deleted at the end of every cycle — a per-row flag on it can't express a decision that needs to survive to a cycle where the row is gone.** Anything that fires because a row is *missing* (`Device Down`, `Disconnected`) can't read a flag that lived on that row. A per-row plugin signal that needs to affect behavior beyond its own cycle has to persist onto the `Devices` row at creation time (e.g. seeding `devAlertDown`/`devAlertEvents` from the row's flag instead of the global `NEWDEV_*` defaults), not ride on the ephemeral table.
3. **`CurrentScan` is not small, and it's indexed on `scanMac`.** Real production users run 10,000+ devices; with one row per contributing plugin (see `LatestDeviceScan` above), a single cycle's `CurrentScan` is routinely 20,000-50,000+ rows. `idx_currentscan_scanmac` (`server/db/db_upgrade.py:ensure_CurrentScan()`, mirrored in `server/db/schema/app.sql`) covers every `scanMac`-keyed lookup in this file. `ensure_CurrentScan()`'s `DROP TABLE`/`CREATE TABLE` runs once, at app startup (`DB.initDB()`, `server/__main__.py`) — don't confuse this with the per-cycle `DELETE FROM CurrentScan` in point 1, which clears rows but leaves the table and its index in place.
4. **`server/plugins/sync/sync.py` bypasses this pipeline on purpose, twice — a permanent exception, not a bug.** It fires its own direct `INSERT OR IGNORE INTO Events (... 'New Device' ...)` for newly-seen synced devices (hardcoded `evePendingAlertEmail = 1`, no `scanNotificationMode` awareness), and in `carbon-copy` mode its own raw `Devices` UPSERT via `ON CONFLICT(devMac) DO UPDATE` — both skip `create_new_devices()`/`update_devices_data_from_scan()`/`can_overwrite_field()` (`sync.py`'s own comments: "Node is fully authoritative in this mode"). It's a normal `mapped_to_table: CurrentScan` plugin for its presence contribution, so `IMPORT_ON`/`scanPresence` apply to it like any other plugin — but its two direct-write paths ignore `scanNotificationMode = 'quiet'` or `scanCreatesDevice = 0`. Don't assume every `Events`/`Devices` write goes through the generic pipeline — `sync.py` doesn't.
5. **A blank/null-equivalent `scanMac` can create a phantom `Devices` row.** `create_new_devices()`'s two creation-path queries filter `scanMac NOT IN (NULL_EQUIVALENTS_SQL)` (`server/scan/device_handling.py`, `const.NULL_EQUIVALENTS_SQL`) as a backstop, because `scanCreatesDevice` defaults to `1` — any plugin reporting a row with no real MAC, without setting `scanCreatesDevice = 0` itself, would otherwise create a `devMac = ''` device, and every other blank-MAC row from every other plugin would then silently write onto it. The filter doesn't replace `scanCreatesDevice = 0` as the correct thing for a plugin to set on such rows; it keeps a MAC-less row inert when some other plugin forgets to. Check any new creation-adjacent query against blank `scanMac` too.
6. **`app.sql` is not dead code.** `install/production-filesystem/entrypoint.d/25-first-run-db.sh` pipes it into `sqlite3` to bootstrap a brand-new database on first install; `scripts/db_cleanup/regenerate-database.sh` uses it too. `CurrentScan`, `Parameters`, and `Settings` are safe from drift: each has a dedicated `ensure_X()` function (`server/db/db_upgrade.py`) that drops and recreates the table on every startup, superseding whatever `app.sql` bootstrapped. `Plugins_Language_Strings` gets the same treatment inside the shared `ensure_plugins_tables()`. `AppEvents` gets its own drop/recreate via `AppEvent_obj.__init__()` (`server/workflows/app_events.py`), independent of `db_upgrade.py`. `Devices` has no drop/recreate, but `server/database.py` has 18 explicit `ensure_column()` calls that backfill any column missing from an older `app.sql` snapshot on every startup. `Events`, `Sessions`, and `Notifications` get the same backfill via `ensure_table_columns()` (`server/db/db_upgrade.py`), driven by one Python column-list constant per table (`server/db/schema_columns.py`) that's diffed against `app.sql` in CI (`test/db/test_schema_drift_guard.py`). `AppEvents`/`Notifications` each also have a second schema-definition surface — their own inline `CREATE TABLE IF NOT EXISTS` in `server/workflows/app_events.py`/`server/models/notification_instance.py` — kept in sync by the same drift-check test. Check any new query here with `EXPLAIN QUERY PLAN` at a realistic row count rather than assuming it's fine because it resembles an existing one — a correlated subquery re-evaluated per row (an accidental self-join) is the pattern most likely to look reasonable while actually being quadratic at this scale.

## When to read this vs. other docs/skills

- Writing or reviewing a plugin's `config.json`/data contract → `plugin-development`, `docs/PLUGINS_DEV*.md`. This skill covers what happens *after* a plugin's rows land in `CurrentScan`, not the authoring contract.
- Devices-table write paths, `*Source` attribution, audit/history logging, SQLite triggers → `database-patterns`.
- Implementing a change here → read the actual function in `server/scan/session_events.py`/`server/scan/device_handling.py` first; this skill's line numbers are a map, not a guarantee, and drift as the code moves.