Elasticsearch File Ingest

Elasticsearch File Ingest

Stream-based ingestion and transformation of large data files (NDJSON, CSV, Parquet, Arrow IPC) into Elasticsearch.

Setup

Install dependencies and configure environment:

npm install

export ELASTICSEARCH_URL="https://elasticsearch:9200"
export ELASTICSEARCH_API_KEY="<your-api-key>"

Test connection:

node scripts/ingest.js test

Usage

Ingest a JSON file

node scripts/ingest.js ingest --file /path/to/data.json --target my-index

Ingest CSV

node scripts/ingest.js ingest --file /path/to/users.csv --source-format csv --target users

Ingest Parquet

node scripts/ingest.js ingest --file /path/to/users.parquet --source-format parquet --target users

Ingest with transformation

node scripts/ingest.js ingest --file /path/to/data.json --target my-index --transform transform.js

Infer mappings from CSV

node scripts/ingest.js ingest --file /path/to/users.csv --infer-mappings --target users

Command Reference

Required Options

--target <index>         # Target index name

Source Options (choose one)

--file <path>            # Source file (supports wildcards)
--stdin                  # Read NDJSON/CSV from stdin

Index Configuration

--mappings <file.json>          # Mappings file
--infer-mappings                # Infer mappings/pipeline from file
--delete-index                  # Delete target index if exists
--pipeline <name>               # Ingest pipeline name

Processing

--transform <file.js>    # Transform function
--source-format <fmt>    # ndjson|csv|parquet|arrow (default: ndjson)
--csv-options <file>     # CSV parser options (JSON file)

Transform Functions

export default function transform(doc) {
  return {
    ...doc,
    full_name: `${doc.first_name} ${doc.last_name}`,
    timestamp: new Date().toISOString(),
  };
}

Return null to skip a document. Return an array to split into multiple documents.

Guidelines

• Test first: Always run node scripts/ingest.js test before ingesting.
• Never combine --infer-mappings with --source-format.
• Use --source-format csv with --mappings for known field types.
• Never echo, print, or log credential environment variables.