Amey-Thakur Search indexing pipeline
The index is a derived copy, which means it can and will drift from the source. A pipeline is judged by how quickly it reflects a change, how it recovers from being wrong, and whether a full rebuild is a routine operation or a crisis.
Method
- State the staleness budget. Seconds for a chat search, minutes for a catalog, hours for an archive. It decides whether you stream changes or batch them (see batch-vs-streaming).
- Drive updates from change events, not polling. Capturing writes as they happen keeps the index close to the source without scanning everything repeatedly (see change-data-capture).
- Make indexing idempotent and ordered per document. Retries are normal, and an out-of-order update that resurrects an old version is a bug users see as ghost results (see idempotency).
- Support full rebuild as a routine path. Analyzer and mapping changes require reindexing, so build into a new index and swap an alias rather than mutating in place.
- Handle deletes explicitly. A deleted record that lingers in the index is worse than a stale one, especially where permissions are involved.
- Denormalise at index time. Joins do not exist at query time, so whatever the result must display or filter on belongs in the document.
- Monitor lag and document counts. Divergence between source and index counts is the earliest signal that the pipeline is quietly broken.
Boundaries
- The index is a copy; it is never the system of record and should be rebuildable from source at any time.
- Reindexing large corpora costs real time and capacity, so plan the window rather than discovering it.
- Access control that changes frequently is awkward to denormalise and may need filtering at query time instead.