Elasticsearch File Ingest
Load local data files into Elasticsearch by converting them to bulk NDJSON, creating an index with the right mappings
when types matter, bulk-indexing documents, and verifying the outcome.
Environment Configuration
This skill executes Elasticsearch operations through the elastic CLI. If the
elastic CLI is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,
GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document
maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API
directly.
Scope
This skill covers file → index loading through POST /_bulk. It does not use Logstash, Filebeat, Elastic Agent,
Node.js ingest tools, or other sidecar pipelines. For copying documents between existing indices, use index-to-index
reindex instead of re-parsing source files.
Supported source shapes:
| Source shape |
Example |
Bulk requirement |
| CSV with header row |
id,name,age,... then data rows |
Parse header into field names; emit one action line + one JSON object per data row |
| JSON array file |
[{"a":1},{"a":2}] |
Split into per-document lines — never bulk-load the raw array as a single document |
| NDJSON / JSON Lines |
one JSON object per line |
Optionally add action lines if missing; otherwise ready for bulk |
Parquet, Arrow, and other binary columnar formats are out of scope unless the user converts them to CSV or JSON first.
Process
Confirm connectivity. Call GET /. If the call fails, stop and resolve CLI configuration before reading files or
mutating cluster state.
Inspect the source file and classify its shape. Open the file (or sample the first lines) and decide:
- CSV — first line is a comma-separated header; subsequent lines are records. Count data rows (exclude the
header) — you will report this count after load.
- JSON array — file starts with
[ and contains an array of objects. Count array elements — each element becomes
one indexed document, not one.
- NDJSON — one JSON value per line; lines alternate action metadata and document source, or each line is a
document that still needs a preceding action line.
The decision: pick the conversion path from NDJSON Bulk Format. Never send
raw CSV text or a raw JSON array body to POST /_bulk.
Choose the target index name. Use the name the user supplied, or propose a lowercase name derived from the file.
Index names must be lowercase, cannot contain spaces or /, and should not start with -, _, or +.
Decide whether an explicit mapping is required. Call GET /{index}/_mapping if the index may already exist.
Create an explicit mapping before bulk loading when:
- CSV columns include numbers, dates, or booleans that must be queryable as typed fields (not plain text).
- The user asks for usable column types or aggregation-friendly fields.
- A prior load indexed everything as
text/keyword strings and must be corrected.
When every field can remain string-like and the user did not specify types, dynamic mapping on first bulk ingest may
suffice — but prefer explicit mappings for CSV unless the user explicitly accepts all-string typing.
Read Mapping Design for Ingest for type choices. When the index exists with wrong
types, ask the user before calling DELETE /{index} and recreating it.
Create the index when needed. When step 4 requires explicit types (or the index does not exist), call
PUT /{index} with a mappings block before bulk loading. Do not rely on dynamic mapping to infer long,
date, or boolean from CSV string cells — dynamic mapping often maps ambiguous strings to text with a .keyword
sub-field.
Convert the file to bulk NDJSON. Write a temporary NDJSON file where each document occupies two lines:
- Line 1 — action metadata, e.g.
{"index":{"_index":"<index>"}} (add "_id" only when the user requires stable
IDs).
- Line 2 — document JSON with correctly typed values (numbers as JSON numbers, booleans as
true/false, dates as
ISO-8601 strings such as 2023-01-15).
For CSV, map the header row to JSON field names and convert cell values to the JSON types that match the mapping from
step 5. For JSON arrays, iterate each array element and emit the action line + object line pair. See worked examples
in NDJSON Bulk Format.
Bulk index the documents. Call POST /_bulk with the NDJSON file produced in step 6. Inspect the response: if
errors is true, read per-item error objects, fix mapping or document issues, and retry failed items after
remediation. Do not assume success from a zero exit code alone.
Verify the outcome. Always confirm the load — never report counts from file inspection alone.
- Call
GET /{index}/_count and compare to the expected row/element count from step 2.
- When typed columns matter, call
GET /{index}/_mapping and confirm fields such as age are numeric (long /
integer), dates are date, and booleans are boolean — not text.
Report the verified document count and, when relevant, the confirmed field types. If count or mapping checks fail,
see Troubleshooting.
Guidelines
- Bulk only. All file loads go through
POST /_bulk with NDJSON action lines — not single-document PUT loops for
batch files, not ingest pipelines as a substitute for client-side CSV parsing, and not posting the untouched source
file.
- JSON arrays must be split. A four-element array bulk-loaded as one document yields count
1; the correct load
yields count 4.
- CSV header is schema. The first CSV row names fields; each remaining row is one document. A file with one header
plus five data rows must report count
5 after ingest.
- Type coercion happens in the document JSON. CSV cells arrive as strings; when mappings declare
long, date, or
boolean, emit JSON numbers, ISO date strings, and boolean literals in the bulk body — do not rely on Elasticsearch
to infer types from quoted CSV strings after dynamic mapping chose text.
- Prefer explicit mappings for typed CSV. Creating the index with
PUT /{index} first prevents silent all-text
indexing that breaks range queries and aggregations.
- Idempotent re-loads. When reloading into an existing index, ask the user before deleting data. Duplicate bulk
index actions append new documents unless _id is specified.
Examples
CSV with typed columns
Source (users.csv — header + 5 data rows):
id,name,age,signup_date,active
1,Ada Lovelace,36,2023-01-15,true
Create the index with explicit types, convert rows to NDJSON (five action+document pairs for five data rows), bulk load,
then verify count 5 and mapping types. Full walkthrough:
Mapping Design for Ingest and
NDJSON Bulk Format.
JSON array file
Source (events.json):
[
{ "event_id": "e-1", "type": "login", "user_id": 1, "value": 12.5 },
{ "event_id": "e-2", "type": "logout", "user_id": 1, "value": 0.0 }
]
Convert to four bulk line pairs for four array elements (not one pair for the whole array). Verify GET /{index}/_count
returns 4. See NDJSON Bulk Format.
NDJSON already prepared
When the file alternates action lines and document lines, validate the format and pass it directly to POST /_bulk
after confirming the target index and mappings.
When Not to Use
- Continuous or streaming ingestion — use Elastic Agent or Beats to tail logs and metrics.
- Complex enrichment pipelines — design server-side ingest pipelines separately; this skill still converts files to
bulk NDJSON client-side before load.
- Index-to-index copy or mapping migration — reindex between indices instead of exporting to files.
- Very large binary columnar files — convert to CSV or JSON offline first, then follow this skill.
References
- NDJSON Bulk Format — CSV and JSON-array conversion, action-line syntax, batch
sizing
- Mapping Design for Ingest — explicit mappings for CSV types, eval-style schemas
- Troubleshooting — wrong counts, text-typed numerics, bulk item errors
Operations
| HTTP API (shorthand) |
elastic CLI command |
GET / |
elastic es info |
PUT /{index} |
elastic es indices create --index '<index>' --mappings '<json>' |
DELETE /{index} |
elastic es indices delete --index '<index>' |
POST /_bulk |
elastic es bulk --index '<index>' --input-file '<ndjson-path>' |
GET /{index}/_count |
elastic es count --index '<index>' |
GET /{index}/_mapping |
elastic es indices get-mapping --index '<index>' |
1---
2name: elasticsearch-ingest
3description: >
4 Load CSV and JSON files into Elasticsearch indices using the bulk API and explicit
5 mappings when field types matter. Use when batch-importing local files, converting
6 CSV rows or JSON arrays to NDJSON bulk format, or verifying document counts and
7 mappings after ingest — not for Logstash pipelines, Beats, custom scripts, or index-to-index
8 reindex.
9metadata:
10 source: https://skillmd.com/skills/gabrielmoreira/elasticsearch-ingest
11 author: elastic
12 version: 0.1.0
13 universal: true
14compatibility: Elasticsearch 8.x or 9.x, self-managed, Elastic Cloud Hosted, or Elastic
15 Cloud Serverless; uses the bulk API available on all deployment types. Requires
16 the `elastic` CLI ≥ 0.2 with `stack es` support.
17---
18
19# Elasticsearch File Ingest
20
21Load local data files into Elasticsearch by converting them to bulk NDJSON, creating an index with the right mappings
22when types matter, bulk-indexing documents, and verifying the outcome.
23
24<!-- begin-partial: preamble -->
25
26## Environment Configuration
27
28This skill executes Elasticsearch operations through the `elastic` CLI. If the
29[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
30not guess credentials, call the HTTP API directly, or attempt other workarounds.
31
32This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
33`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
34maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
35directly.
36
37<!-- end-partial: preamble -->
38
39## Scope
40
41This skill covers **file → index** loading through `POST /_bulk`. It does not use Logstash, Filebeat, Elastic Agent,
42Node.js ingest tools, or other sidecar pipelines. For copying documents between existing indices, use index-to-index
43reindex instead of re-parsing source files.
44
45Supported source shapes:
46
47| Source shape | Example | Bulk requirement |
48| ------------------- | -------------------------------- | ---------------------------------------------------------------------------------- |
49| CSV with header row | `id,name,age,...` then data rows | Parse header into field names; emit one action line + one JSON object per data row |
50| JSON array file | `[{"a":1},{"a":2}]` | Split into per-document lines — never bulk-load the raw array as a single document |
51| NDJSON / JSON Lines | one JSON object per line | Optionally add action lines if missing; otherwise ready for bulk |
52
53Parquet, Arrow, and other binary columnar formats are out of scope unless the user converts them to CSV or JSON first.
54
55## Process
56
571. **Confirm connectivity.** Call `GET /`. If the call fails, stop and resolve CLI configuration before reading files or
58 mutating cluster state.
59
602. **Inspect the source file and classify its shape.** Open the file (or sample the first lines) and decide:
61 - **CSV** — first line is a comma-separated header; subsequent lines are records. Count data rows (exclude the
62 header) — you will report this count after load.
63 - **JSON array** — file starts with `[` and contains an array of objects. Count array elements — each element becomes
64 one indexed document, not one.
65 - **NDJSON** — one JSON value per line; lines alternate action metadata and document source, or each line is a
66 document that still needs a preceding action line.
67
68 The decision: pick the conversion path from [NDJSON Bulk Format](references/ndjson-bulk-format.md). **Never** send
69 raw CSV text or a raw JSON array body to `POST /_bulk`.
70
713. **Choose the target index name.** Use the name the user supplied, or propose a lowercase name derived from the file.
72 Index names must be lowercase, cannot contain spaces or `/`, and should not start with `-`, `_`, or `+`.
73
744. **Decide whether an explicit mapping is required.** Call `GET /{index}/_mapping` if the index may already exist.
75
76 Create an explicit mapping **before** bulk loading when:
77 - CSV columns include numbers, dates, or booleans that must be queryable as typed fields (not plain text).
78 - The user asks for usable column types or aggregation-friendly fields.
79 - A prior load indexed everything as `text`/`keyword` strings and must be corrected.
80
81 When every field can remain string-like and the user did not specify types, dynamic mapping on first bulk ingest may
82 suffice — but prefer explicit mappings for CSV unless the user explicitly accepts all-string typing.
83
84 Read [Mapping Design for Ingest](references/mapping-design.md) for type choices. When the index exists with wrong
85 types, ask the user before calling `DELETE /{index}` and recreating it.
86
875. **Create the index when needed.** When step 4 requires explicit types (or the index does not exist), call
88 `PUT /{index}` with a `mappings` block **before** bulk loading. Do not rely on dynamic mapping to infer `long`,
89 `date`, or `boolean` from CSV string cells — dynamic mapping often maps ambiguous strings to `text` with a `.keyword`
90 sub-field.
91
926. **Convert the file to bulk NDJSON.** Write a temporary NDJSON file where **each document occupies two lines**:
93 - Line 1 — action metadata, e.g. `{"index":{"_index":"<index>"}}` (add `"_id"` only when the user requires stable
94 IDs).
95 - Line 2 — document JSON with correctly typed values (numbers as JSON numbers, booleans as `true`/`false`, dates as
96 ISO-8601 strings such as `2023-01-15`).
97
98 For CSV, map the header row to JSON field names and convert cell values to the JSON types that match the mapping from
99 step 5. For JSON arrays, iterate each array element and emit the action line + object line pair. See worked examples
100 in [NDJSON Bulk Format](references/ndjson-bulk-format.md).
101
1027. **Bulk index the documents.** Call `POST /_bulk` with the NDJSON file produced in step 6. Inspect the response: if
103 `errors` is `true`, read per-item `error` objects, fix mapping or document issues, and retry failed items after
104 remediation. Do not assume success from a zero exit code alone.
105
1068. **Verify the outcome.** Always confirm the load — never report counts from file inspection alone.
107 - Call `GET /{index}/_count` and compare to the expected row/element count from step 2.
108 - When typed columns matter, call `GET /{index}/_mapping` and confirm fields such as `age` are numeric (`long` /
109 `integer`), dates are `date`, and booleans are `boolean` — not `text`.
110
111 Report the verified document count and, when relevant, the confirmed field types. If count or mapping checks fail,
112 see [Troubleshooting](references/troubleshooting.md).
113
114## Guidelines
115
116- **Bulk only.** All file loads go through `POST /_bulk` with NDJSON action lines — not single-document `PUT` loops for
117 batch files, not ingest pipelines as a substitute for client-side CSV parsing, and not posting the untouched source
118 file.
119- **JSON arrays must be split.** A four-element array bulk-loaded as one document yields count `1`; the correct load
120 yields count `4`.
121- **CSV header is schema.** The first CSV row names fields; each remaining row is one document. A file with one header
122 plus five data rows must report count `5` after ingest.
123- **Type coercion happens in the document JSON.** CSV cells arrive as strings; when mappings declare `long`, `date`, or
124 `boolean`, emit JSON numbers, ISO date strings, and boolean literals in the bulk body — do not rely on Elasticsearch
125 to infer types from quoted CSV strings after dynamic mapping chose `text`.
126- **Prefer explicit mappings for typed CSV.** Creating the index with `PUT /{index}` first prevents silent all-text
127 indexing that breaks range queries and aggregations.
128- **Idempotent re-loads.** When reloading into an existing index, ask the user before deleting data. Duplicate bulk
129 `index` actions append new documents unless `_id` is specified.
130
131## Examples
132
133### CSV with typed columns
134
135Source (`users.csv` — header + 5 data rows):
136
137```csv
138id,name,age,signup_date,active
1391,Ada Lovelace,36,2023-01-15,true
140```
141
142Create the index with explicit types, convert rows to NDJSON (five action+document pairs for five data rows), bulk load,
143then verify count `5` and mapping types. Full walkthrough:
144[Mapping Design for Ingest](references/mapping-design.md#csv-with-mixed-types) and
145[NDJSON Bulk Format](references/ndjson-bulk-format.md#csv-to-bulk-ndjson).
146
147### JSON array file
148
149Source (`events.json`):
150
151```json
152[
153 { "event_id": "e-1", "type": "login", "user_id": 1, "value": 12.5 },
154 { "event_id": "e-2", "type": "logout", "user_id": 1, "value": 0.0 }
155]
156```
157
158Convert to four bulk line pairs for four array elements (not one pair for the whole array). Verify `GET /{index}/_count`
159returns `4`. See [NDJSON Bulk Format](references/ndjson-bulk-format.md#json-array-to-bulk-ndjson).
160
161### NDJSON already prepared
162
163When the file alternates action lines and document lines, validate the format and pass it directly to `POST /_bulk`
164after confirming the target index and mappings.
165
166## When Not to Use
167
168- **Continuous or streaming ingestion** — use Elastic Agent or Beats to tail logs and metrics.
169- **Complex enrichment pipelines** — design server-side ingest pipelines separately; this skill still converts files to
170 bulk NDJSON client-side before load.
171- **Index-to-index copy or mapping migration** — reindex between indices instead of exporting to files.
172- **Very large binary columnar files** — convert to CSV or JSON offline first, then follow this skill.
173
174## References
175
176- [NDJSON Bulk Format](references/ndjson-bulk-format.md) — CSV and JSON-array conversion, action-line syntax, batch
177 sizing
178- [Mapping Design for Ingest](references/mapping-design.md) — explicit mappings for CSV types, eval-style schemas
179- [Troubleshooting](references/troubleshooting.md) — wrong counts, text-typed numerics, bulk item errors
180
181## Operations
182
183| HTTP API (shorthand) | `elastic` CLI command |
184| ----------------------- | ----------------------------------------------------------------- |
185| `GET /` | `elastic es info` |
186| `PUT /{index}` | `elastic es indices create --index '<index>' --mappings '<json>'` |
187| `DELETE /{index}` | `elastic es indices delete --index '<index>'` |
188| `POST /_bulk` | `elastic es bulk --index '<index>' --input-file '<ndjson-path>'` |
189| `GET /{index}/_count` | `elastic es count --index '<index>'` |
190| `GET /{index}/_mapping` | `elastic es indices get-mapping --index '<index>'` |