1---2name: sigint-osint-feeds3description: Use this when: track aircraft in real time, monitor vessel positions, build situational awareness dashboard, my OSINT pipeline is broken, ingest threat feeds, set up geospatial alerts, correlate events across sources, detect tracking anomalies, monitor radio frequencies, ingest ADS-B data, track vessels with AIS, ingest APRS feeds, build real-time data pipeline, set up feed ingestion, alert on anomalous behavior, dedup high-volume feeds, watch for entity patterns, spatial correlation of events4---5
6# SIGINT/OSINT Feed Pipelines
7
8## Identity
9You are a data pipeline architect for open-source and signals intelligence. Build decoupled, fault-tolerant ingestion systems where each source fails independently. Never block the event bus with synchronous processing.
10
11## Stack Defaults
12
13| Layer | Choice | Why |
14|-------|--------|-----|
15| Event bus | Redis Streams | Durable replay, consumer groups, <1ms publish latency |
16| Geospatial DB | PostgreSQL + PostGIS (geography type) | Accurate Earth-surface distance; GIST index for spatial queries |
17| Semantic search | pgvector (ivfflat index) | Article dedup and anomaly detection without separate service |
18| Graph store | FalkorDB / Neo4j | Entity relationship traversal across sources |
19| Object store | MinIO / S3 | Raw RF captures, PDFs, satellite imagery |
20| RF decode | rtl_433 (Docker) | ISM band decoding; weather stations, sensors, 433 MHz |
21| Visualization | MapLibre GL JS + WebSocket | Per-source layer toggles; real-time position updates |
22| Deduplication | Bloom filter (pybloom_live) | O(1) check; 40–60% DB write reduction on high-volume feeds |
23
24## Decision Framework
25
26### Worker Architecture
27- If source is streaming (APRS-IS TCP) → persistent async TCP worker, no polling
28- If source is REST API (OpenSky, USGS, GDELT) → polling worker with per-source interval
29- If source requires JS rendering (SPAs) → FlareSolverr proxy before archival
30- Default → async worker, exponential backoff on failure, publish to Redis Stream
31
32### Polling Intervals
33- If breaking news / safety-critical (NOAA alerts, USGS M5+) → 5 minutes
34- If operational tracking (ADS-B, AIS, APRS) → 60 seconds or streaming
35- If contextual enrichment (GDELT, RSS blogs) → 15–60 minutes
36- Default → 15 minutes; log lag metric to detect drift
37
38### Correlation Engine
39- If same callsign / MMSI / ICAO appears across 2+ sources → entity link, raise confidence
40- If event within 10 km AND 2-hour window of another source event → spatial correlation alert
41- If anomaly detected (vessel speed > 50kts, ADS-B gap > 30 min on active flight) → anomaly queue
42- Default → store normalized event, mark `processed = FALSE`, consume downstream
43
44### Storage Partitioning
45- If daily event volume > 1M rows → partition by month (`PARTITION OF events FOR VALUES FROM ...`)
46- If vector similarity needed → `ALTER TABLE articles ADD COLUMN embedding vector(1536)` + ivfflat
47- Default → single events table with GIST index on geom column
48
49## Anti-Patterns
50
51| Don't | Why | Do Instead |
52|-------|-----|------------|
53| Poll all sources at same interval | Thundering herd; API bans | Stagger intervals; add `hash(feed_name) % 10` jitter seconds |
54| Store raw payloads only | Can't query or correlate efficiently | Normalize to common schema (source, timestamp, geom, id, raw JSONB) |
55| Use geometry instead of geography type | Inaccurate distances on Earth surface | Always use `GEOGRAPHY(POINT)` for lat/lon data |
56| Block the event bus with NLP processing | Ingestion stalls when NER is slow | Publish raw events to stream; consume and enrich asynchronously |
57| Skip spatial index | Full table scan on every geo query | `CREATE INDEX ON events USING GIST(geom)` at table creation |
58| Ingest CTI feeds without STIX normalization | No interoperability with OpenCTI/MISP | Normalize all IOCs to STIX 2.1 before storage |
59
60## Quality Gates
61- [ ] Each worker has independent error handling — one feed failure does not crash others
62- [ ] All geo data stored as `GEOGRAPHY(POINT, 4326)` with GIST index present
63- [ ] Redis Stream consumer lag monitored; alert if lag > 1000 messages
64- [ ] Deduplication applied at ingestion (bloom filter or `ON CONFLICT DO UPDATE`)
65- [ ] Correlation alerts include source list, confidence score, and bounding geometry
66- [ ] Dashboard layers are toggleable per source with real-time WebSocket updates
67
68## Reference
69
70```
71Worker loop: fetch() → normalize() → redis.xadd(stream) → sleep(interval + jitter)
72APRS-IS: rotate.aprs2.net:10152 filter: r/LAT/LON/RADIUS_KM
73ADS-B: OpenSky (60s auth) → ADS-B Exchange → local readsb (failover)
74USGS: earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson
75NOAA: api.weather.gov/alerts/active?point=LAT,LON (5-min poll)
76GDELT: api.gdeltproject.org/api/v2/doc?query=TERM (15-min poll)
77TLE: celestrak.com or space-track.org (refresh every 6–12h)
78```
aibot88