Elasticsearch Index Design
Design explicit index mappings from access patterns, review existing mappings for type and storage mistakes, and apply
corrections through a new index plus reindex when field types must change.
Environment Configuration
This skill executes Elasticsearch operations through the elastic CLI. If the
elastic CLI is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,
GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document
maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API
directly.
Process
Gather access patterns per field. Before choosing types, list how each field is used. For every field capture:
- Search — full-text match, phrase, relevance scoring?
- Filter — exact term, terms set, prefix?
- Aggregate — terms, cardinality, histogram, stats?
- Sort — ascending/d descending in result sets?
- Retrieve only — returned in
_source but never queried?
The decision: classify each field into one primary access pattern (search, exact, numeric metric, date, boolean,
structured object, or retrieve-only). Missing access-pattern data is a blocker — ask the user rather than guessing.
Call GET / to confirm connectivity; when reviewing an existing index, call GET /{index}/_mapping to ground the
discussion in the current mapping.
Choose field types from access patterns. Map each field to the minimal type set that satisfies its pattern. Read
Field Type Decisions and
Multi-Field Patterns before proposing mappings.
Key judgments:
| Pattern |
Mapping |
| Full-text search only |
text (no keyword sub-field) |
| Filter / agg / sort only |
keyword (not text) |
| Full-text search and sort or aggregation |
text with fields.keyword multi-field |
| Decimal price or metric |
double, float, or scaled_float — not text or integer |
| Timestamp |
date |
| True/false flag |
boolean |
| Free-form key/value map with many distinct keys |
flattened — not dynamic object |
Multi-field rule: When a field must be searchable and sortable/aggregatable (e.g. product name), map it as
text with a keyword sub-field — search on name, sort and aggregate on name.keyword. Mapping as only text or
only keyword is wrong for that combined pattern.
Explicit mapping rule: For new indices, always define mappings explicitly with PUT /{index}. Do not rely on
dynamic mapping for production indices — the first document can lock in wrong types (strings as text, ambiguous
numbers as keyword).
Index settings: Set deliberate number_of_shards and number_of_replicas in the same PUT /{index} request
when the deployment allows it (Self-Managed / Elastic Cloud Hosted). On Serverless, omit shard and replica counts
(Elastic manages them); still supply explicit mappings. State chosen values or document that defaults apply.
Example — products index optimized for search plus sort/agg on name:
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
},
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": { "type": "keyword", "ignore_above": 256 }
}
},
"price": { "type": "double" },
"created": { "type": "date" },
"in_stock": { "type": "boolean" }
}
}
}
Create with PUT /products passing the settings and mappings blocks. Verify with GET /products/_mapping.
Guard against mapping explosion and storage bloat. On high-volume indices, type mistakes multiply cost. Read
Mapping Explosion and Storage Bloat and apply these review checks:
- Analyzed-but-not-searched fields — Fields used only for filter and aggregation (
url, HTTP status_code,
tags, IDs) must be keyword, not text. text wastes space; aggregations on text require fielddata or a
.keyword sub-field that should not exist if the field is not searched.
message.keyword without ignore_above — A keyword sub-field on a large full-text body indexes the entire raw
string as one term. Flag this anti-pattern; remove the sub-field when only full-text search is needed, or add
ignore_above when a bounded exact-match sub-field is truly required.
- Dynamic free-form objects —
object with "dynamic": true on user-supplied key/value data with thousands of
distinct keys causes mapping explosion. Recommend flattened (or strict dynamic / allowlist strategy).
doc_values: false — On fields retrieved in hits but never sorted, aggregated, or filtered (e.g. display-only
session_id), set "doc_values": false on keyword to save disk at scale.
scaled_float — For metrics with bounded precision (e.g. response_time_ms), prefer scaled_float with an
appropriate scaling_factor over plain float/double when storage dominates.
Prefer "dynamic": "strict" on the root mapping unless unknown fields are an explicit requirement.
Apply design: create new index and reindex when types change. Elasticsearch cannot change an existing field's
type in place. When review finds wrong types (text→keyword, object→flattened, float→scaled_float, doc_values changes
on existing fields), state clearly that fixes require a new index and reindex — not a mapping update on the
live index.
Workflow for correcting an existing high-volume index such as events:
- Design the corrected mapping on a new index name (e.g.
events-v2) incorporating all fixes from steps 2–3.
- Create the destination with
PUT /events-v2 and the full corrected mappings (and settings where
applicable).
- Copy documents with
POST /_reindex — for large indices use wait_for_completion=false and track the task.
Source: { "index": "events" }, destination: { "index": "events-v2" }.
- Verify with
GET /events-v2/_count (compare to source count) and GET /events-v2/_mapping (confirm types).
- Cut over reads and writes (index alias swap or application config) after validation.
Example corrected excerpt for the events review pattern:
{
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"event_id": { "type": "keyword" },
"session_id": { "type": "keyword", "doc_values": false },
"url": { "type": "keyword" },
"status_code": { "type": "keyword" },
"response_time_ms": { "type": "scaled_float", "scaling_factor": 100 },
"tags": { "type": "keyword" },
"message": { "type": "text" },
"labels": { "type": "flattened" }
}
}
}
Do not attempt in-place mapping fixes for these type changes — they are rejected or leave data inconsistent. For
greenfield indices, a single PUT /{index} before first ingest avoids reindex entirely.
Review checklist
When the user supplies a mapping JSON and usage notes, walk this checklist in order:
- Match each field's type to its stated access pattern (see step 2).
- Flag
text on filter/agg-only fields; flag missing multi-fields where search and sort/agg share one logical field.
- Flag
message.keyword (or similar) without ignore_above on large analyzed text.
- Flag dynamic
object on high-cardinality free-form maps; recommend flattened.
- Propose retrieve-only and numeric storage optimizations (
doc_values: false, scaled_float).
- State that type changes require a new index and
POST /_reindex, then show the corrected mapping and reindex plan.
Examples
"Users search product names and also sort and aggregate on them" — one logical field, two access patterns, so use a
text field with a keyword multi-field:
{
"mappings": {
"properties": {
"product_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }
}
}
}
"A status field is only ever filtered and aggregated, never full-text searched" — use keyword, not text:
{ "mappings": { "properties": { "status": { "type": "keyword" } } } }
"Free-form labels object with unbounded keys" — avoid mapping explosion with flattened:
{ "mappings": { "properties": { "labels": { "type": "flattened" } } } }
Guidelines
- Minimal mapping — Map only what access patterns require; every sub-field and analyzed form adds indexed data.
- Never guess access patterns — Wrong type choice is expensive to fix at scale.
- Verify after create — Always confirm with
GET /{index}/_mapping; use GET /{index}/_count after reindex.
- Cross-skill boundary — Copying documents between indices is
POST /_reindex (see the reindex skill for slicing,
throttling, and task tracking). Loading files into a new index is bulk ingest, not index design.
Reference material
- Field Type Decisions — access-pattern-to-type table and common mistakes
- Multi-Field Patterns — text+keyword,
ignore_above, anti-patterns
- Mapping Explosion and Storage Bloat —
flattened, doc_values, dynamic objects
Operations
| HTTP API (shorthand) |
elastic CLI command |
GET / |
elastic es info |
GET /{index}/_mapping |
elastic es indices get-mapping --index '<index>' |
PUT /{index} |
elastic es indices create --index '<index>' --mappings '<json>' --settings '<json>' |
POST /_reindex |
elastic es reindex --source '<json>' --dest '<json>' |
POST /_reindex?wait_for_completion=false |
elastic es reindex --wait-for-completion false --source '<json>' --dest '<json>' |
GET /{index}/_count |
elastic es count --index '<index>' |
1---
2name: elasticsearch-index-design
3description: >
4 Design and review Elasticsearch index mappings for stated access patterns: correct
5 field types, text+keyword multi-fields, doc_values tuning, mapping-explosion avoidance,
6 and explicit shard settings. Use when creating a new index, reviewing a mapping
7 for storage or query performance, fixing wrong field types, or when the user asks
8 which type to use for search, filter, sort, or aggregation on a field.
9metadata:
10 source: https://skillmd.com/skills/gabrielmoreira/elasticsearch-index-design
11 author: elastic
12 version: 0.1.0
13 universal: true
14compatibility: Elasticsearch 8.x or 9.x, self-managed, Elastic Cloud Hosted, or Elastic
15 Cloud Serverless; explicit shard and replica settings apply to self-managed and
16 Elastic Cloud Hosted only (managed internally on Serverless). Requires the `elastic`
17 CLI ≥ 0.2 with `stack es` support.
18---
19
20# Elasticsearch Index Design
21
22Design explicit index mappings from access patterns, review existing mappings for type and storage mistakes, and apply
23corrections through a new index plus reindex when field types must change.
24
25<!-- begin-partial: preamble -->
26
27## Environment Configuration
28
29This skill executes Elasticsearch operations through the `elastic` CLI. If the
30[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
31not guess credentials, call the HTTP API directly, or attempt other workarounds.
32
33This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
34`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
35maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
36directly.
37
38<!-- end-partial: preamble -->
39
40## Process
41
421. **Gather access patterns per field.** Before choosing types, list how each field is used. For every field capture:
43 - **Search** — full-text match, phrase, relevance scoring?
44 - **Filter** — exact term, terms set, prefix?
45 - **Aggregate** — terms, cardinality, histogram, stats?
46 - **Sort** — ascending/d descending in result sets?
47 - **Retrieve only** — returned in `_source` but never queried?
48
49 The decision: classify each field into one primary access pattern (search, exact, numeric metric, date, boolean,
50 structured object, or retrieve-only). Missing access-pattern data is a blocker — ask the user rather than guessing.
51 Call `GET /` to confirm connectivity; when reviewing an existing index, call `GET /{index}/_mapping` to ground the
52 discussion in the current mapping.
53
542. **Choose field types from access patterns.** Map each field to the minimal type set that satisfies its pattern. Read
55 [Field Type Decisions](references/field-type-decisions.md) and
56 [Multi-Field Patterns](references/multi-field-patterns.md) before proposing mappings.
57
58 Key judgments:
59
60 | Pattern | Mapping |
61 | ----------------------------------------------- | ------------------------------------------------------------ |
62 | Full-text search only | `text` (no keyword sub-field) |
63 | Filter / agg / sort only | `keyword` (not `text`) |
64 | Full-text search **and** sort or aggregation | `text` with `fields.keyword` multi-field |
65 | Decimal price or metric | `double`, `float`, or `scaled_float` — not `text` or integer |
66 | Timestamp | `date` |
67 | True/false flag | `boolean` |
68 | Free-form key/value map with many distinct keys | `flattened` — not dynamic `object` |
69
70 **Multi-field rule:** When a field must be searchable **and** sortable/aggregatable (e.g. product `name`), map it as
71 `text` with a `keyword` sub-field — search on `name`, sort and aggregate on `name.keyword`. Mapping as only `text` or
72 only `keyword` is wrong for that combined pattern.
73
74 **Explicit mapping rule:** For new indices, always define mappings explicitly with `PUT /{index}`. Do not rely on
75 dynamic mapping for production indices — the first document can lock in wrong types (strings as `text`, ambiguous
76 numbers as `keyword`).
77
78 **Index settings:** Set deliberate `number_of_shards` and `number_of_replicas` in the same `PUT /{index}` request
79 when the deployment allows it (Self-Managed / Elastic Cloud Hosted). On Serverless, omit shard and replica counts
80 (Elastic manages them); still supply explicit mappings. State chosen values or document that defaults apply.
81
82 Example — `products` index optimized for search plus sort/agg on name:
83
84 ```json
85 {
86 "settings": {
87 "number_of_shards": 1,
88 "number_of_replicas": 1
89 },
90 "mappings": {
91 "properties": {
92 "name": {
93 "type": "text",
94 "fields": {
95 "keyword": { "type": "keyword", "ignore_above": 256 }
96 }
97 },
98 "price": { "type": "double" },
99 "created": { "type": "date" },
100 "in_stock": { "type": "boolean" }
101 }
102 }
103 }
104 ```
105
106 Create with `PUT /products` passing the `settings` and `mappings` blocks. Verify with `GET /products/_mapping`.
107
1083. **Guard against mapping explosion and storage bloat.** On high-volume indices, type mistakes multiply cost. Read
109 [Mapping Explosion and Storage Bloat](references/mapping-explosion.md) and apply these review checks:
110 - **Analyzed-but-not-searched fields** — Fields used only for filter and aggregation (`url`, HTTP `status_code`,
111 `tags`, IDs) must be `keyword`, not `text`. `text` wastes space; aggregations on `text` require fielddata or a
112 `.keyword` sub-field that should not exist if the field is not searched.
113 - **`message.keyword` without `ignore_above`** — A keyword sub-field on a large full-text body indexes the entire raw
114 string as one term. Flag this anti-pattern; remove the sub-field when only full-text search is needed, or add
115 `ignore_above` when a bounded exact-match sub-field is truly required.
116 - **Dynamic free-form objects** — `object` with `"dynamic": true` on user-supplied key/value data with thousands of
117 distinct keys causes **mapping explosion**. Recommend `flattened` (or strict dynamic / allowlist strategy).
118 - **`doc_values: false`** — On fields retrieved in hits but never sorted, aggregated, or filtered (e.g. display-only
119 `session_id`), set `"doc_values": false` on `keyword` to save disk at scale.
120 - **`scaled_float`** — For metrics with bounded precision (e.g. `response_time_ms`), prefer `scaled_float` with an
121 appropriate `scaling_factor` over plain `float`/`double` when storage dominates.
122
123 Prefer `"dynamic": "strict"` on the root mapping unless unknown fields are an explicit requirement.
124
1254. **Apply design: create new index and reindex when types change.** Elasticsearch **cannot** change an existing field's
126 type in place. When review finds wrong types (text→keyword, object→flattened, float→scaled_float, doc_values changes
127 on existing fields), state clearly that fixes require a **new index** and **reindex** — not a mapping update on the
128 live index.
129
130 Workflow for correcting an existing high-volume index such as `events`:
131 1. **Design the corrected mapping** on a new index name (e.g. `events-v2`) incorporating all fixes from steps 2–3.
132 2. **Create the destination** with `PUT /events-v2` and the full corrected `mappings` (and `settings` where
133 applicable).
134 3. **Copy documents** with `POST /_reindex` — for large indices use `wait_for_completion=false` and track the task.
135 Source: `{ "index": "events" }`, destination: `{ "index": "events-v2" }`.
136 4. **Verify** with `GET /events-v2/_count` (compare to source count) and `GET /events-v2/_mapping` (confirm types).
137 5. **Cut over** reads and writes (index alias swap or application config) after validation.
138
139 Example corrected excerpt for the `events` review pattern:
140
141 ```json
142 {
143 "mappings": {
144 "properties": {
145 "@timestamp": { "type": "date" },
146 "event_id": { "type": "keyword" },
147 "session_id": { "type": "keyword", "doc_values": false },
148 "url": { "type": "keyword" },
149 "status_code": { "type": "keyword" },
150 "response_time_ms": { "type": "scaled_float", "scaling_factor": 100 },
151 "tags": { "type": "keyword" },
152 "message": { "type": "text" },
153 "labels": { "type": "flattened" }
154 }
155 }
156 }
157 ```
158
159 Do not attempt in-place mapping fixes for these type changes — they are rejected or leave data inconsistent. For
160 greenfield indices, a single `PUT /{index}` before first ingest avoids reindex entirely.
161
162## Review checklist
163
164When the user supplies a mapping JSON and usage notes, walk this checklist in order:
165
1661. Match each field's type to its stated access pattern (see step 2).
1672. Flag `text` on filter/agg-only fields; flag missing multi-fields where search and sort/agg share one logical field.
1683. Flag `message.keyword` (or similar) without `ignore_above` on large analyzed text.
1694. Flag dynamic `object` on high-cardinality free-form maps; recommend `flattened`.
1705. Propose retrieve-only and numeric storage optimizations (`doc_values: false`, `scaled_float`).
1716. State that type changes require a new index and `POST /_reindex`, then show the corrected mapping and reindex plan.
172
173## Examples
174
175**"Users search product names and also sort and aggregate on them"** — one logical field, two access patterns, so use a
176`text` field with a `keyword` multi-field:
177
178```json
179{
180 "mappings": {
181 "properties": {
182 "product_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }
183 }
184 }
185}
186```
187
188**"A `status` field is only ever filtered and aggregated, never full-text searched"** — use `keyword`, not `text`:
189
190```json
191{ "mappings": { "properties": { "status": { "type": "keyword" } } } }
192```
193
194**"Free-form `labels` object with unbounded keys"** — avoid mapping explosion with `flattened`:
195
196```json
197{ "mappings": { "properties": { "labels": { "type": "flattened" } } } }
198```
199
200## Guidelines
201
202- **Minimal mapping** — Map only what access patterns require; every sub-field and analyzed form adds indexed data.
203- **Never guess access patterns** — Wrong type choice is expensive to fix at scale.
204- **Verify after create** — Always confirm with `GET /{index}/_mapping`; use `GET /{index}/_count` after reindex.
205- **Cross-skill boundary** — Copying documents between indices is `POST /_reindex` (see the reindex skill for slicing,
206 throttling, and task tracking). Loading files into a new index is bulk ingest, not index design.
207
208## Reference material
209
210- [Field Type Decisions](references/field-type-decisions.md) — access-pattern-to-type table and common mistakes
211- [Multi-Field Patterns](references/multi-field-patterns.md) — text+keyword, `ignore_above`, anti-patterns
212- [Mapping Explosion and Storage Bloat](references/mapping-explosion.md) — `flattened`, `doc_values`, dynamic objects
213
214## Operations
215
216| HTTP API (shorthand) | `elastic` CLI command |
217| ------------------------------------------ | ------------------------------------------------------------------------------------- |
218| `GET /` | `elastic es info` |
219| `GET /{index}/_mapping` | `elastic es indices get-mapping --index '<index>'` |
220| `PUT /{index}` | `elastic es indices create --index '<index>' --mappings '<json>' --settings '<json>'` |
221| `POST /_reindex` | `elastic es reindex --source '<json>' --dest '<json>'` |
222| `POST /_reindex?wait_for_completion=false` | `elastic es reindex --wait-for-completion false --source '<json>' --dest '<json>'` |
223| `GET /{index}/_count` | `elastic es count --index '<index>'` |