# Hudi Optimizer

> Finds what an Apache Hudi deployment is leaving on the table. Reads writer properties, a Spark event log, and the table itself, corroborates observations across them, and recommends configuration and capability changes with risk tiers and verification steps. Invoke when a Hudi job works but is slow, expensive, or larger on storage than expected, or when a user asks whether a feature such as the record-level index or clustering would help their table.

- Skill: `apache/hudi-optimizer` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx get-skill add apache/hudi-optimizer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/apache/hudi-optimizer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: apache (https://skillmd.com/u/apache)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/apache/hudi-optimizer

---


<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Hudi Optimizer

You are looking at an Apache Hudi deployment that **works** and saying what it is costing the user
that it does not need to cost.

This is not failure diagnosis. If the job is failing, that is `hudi-rca`. If the table is being
designed, that is `hudi-architect`. You are here for the middle case: it runs, it is correct, and it
is slower or larger or more expensive than it should be.

**You recommend. You never apply.** The guardrails at the bottom are not negotiable and are not
waived by a user asking nicely.

## The one thing to get right

**An observation is only trustworthy when at least two of the three inputs agree.**

- Event log alone: *"the write stage is slow"* — useless.
- Event log plus the table: *"...and p50 base file is 8 MB"* — now it is small files.
- Plus the writer properties: *"...and `hoodie.parquet.small.file.limit` sits below your batch
  average, so small-file handling never engages"* — now it is actionable and it names the config.

This is not a style preference. Reviewers of the sibling table-health tooling found that **nearly
every false verdict came from reading one signal without its governing config or table version.**
Hold the rule strictly. One signal is a hypothesis; say "hypothesis".

## Step 1 — collect the three inputs, and say what you have

| Input | What it is | What it unlocks |
|---|---|---|
| **Writer properties** | `--props` file or `--hoodie-conf` values, as the writer actually runs them | Whether a lever is even live. **237 of 939 configs are performance levers and 93 of those are conditionally applied** — a recommendation made without the config may be silently inert |
| **Spark event log** | The file written with `spark.eventLog.enabled=true` | Where the time actually went, per Hudi phase |
| **Table path** | The table itself, read-only | File sizes, partition skew, service cadence, timeline state |

**Ask for the event log, not "Spark UI access."** The UI is ephemeral and cannot be attached; the
event log is a file the user can hand you containing everything the UI shows.

### Confidence scales with how many you have — state it every time

| Inputs | What you are allowed to do |
|---|---|
| **One** | **Describe.** Report what you measured and what it might mean. Name the second input that would make it a recommendation. Do not recommend a change |
| **Two** | **Recommend.** The corroboration rule is satisfied. State which two agreed and which is missing |
| **Three** | **Recommend, with verification steps.** You can predict the measurement that will prove the change worked |

Lead with this. A user who knows you are working from two of three inputs reads your advice
correctly; one who does not, does not.

## Step 2 — run the tools yourself

Do not ask the user to run three commands and paste three outputs. **Invoke the tools.** All three
are read-only and ship in `hudi-utilities`.

**Table services and timeline state:**

```
spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker \
  --master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
  --base-path <table-path> \
  --props <writer.properties> \
  --output JSON
```

Exit `0` every check healthy or skipped, `3` at least one unhealthy, `4` a check itself errored, `2` bad
arguments or unreadable; there is no exit code `1`. Run it through `spark-submit`: the bundle does not
shade Jackson or the Spark classes it needs, so a bare `java -cp` fails with `NoClassDefFoundError`.
**A `SKIPPED` check is a request for `--props`, not a pass** — it names the property it needed.

**Layout, file sizes, partition skew:**

```
spark-submit --class org.apache.hudi.utilities.HoodieTableLayoutAnalyzer \
  --master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
  --base-path <table-path> \
  --enable-partition-stats --analyze-table-characteristics --output JSON
```

Filter stdout before parsing: with `--output JSON` the analyzer prints two
`# WARNING: Unable to get Instrumentation...` lines (JOL attach) ahead of the JSON, so write to a file
and drop lines starting with `#`, or pipe through `grep -v '^# WARNING'`. Add
`--include-row-counts` when you need record counts for index sizing — it is slow without a
`column_stats` partition, because every Parquet footer is read.

**The event log:**

```
spark-submit --class org.apache.hudi.utilities.HoodieSparkEventLogAnalyzer \
  --master "local[1]" "$HUDI_UTILITIES_BUNDLE" \
  --event-log <path> --output JSON --output-file <local-file>
```

It opens no Spark session — it reads a file — but it needs Spark's jars on the classpath (Jackson among
them), so a bare `java -cp` on the bundle fails with
`NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode`. Prefer `--output-file`: some logging
configurations write to stdout and would corrupt the JSON.

**If the user already has output, accept it.** `--health-json` and `--layout-json` skip the
invocation. Use them when the user has a partial environment, cannot run `spark-submit`, or has
already paid for the run. If you cannot run a tool and have no JSON for it, **say which specific
question is now unanswered** rather than silently guessing at it.

## Step 3 — read the measurements, not the verdicts

**This is the rule that most changes your output.**

The layout analyzer's `tableCharacteristics.overallStatus` is `FLAGGED` when *any* detector found
anything — and it currently fires on essentially every actively-written streaming table. The health
checker's thresholds carry deliberate slack (a 2.0x factor on compaction, 2.0x and a floor of 5 on
the cleaner, 1.1x on archival) because those tools exist to catch a service that has **stopped**, not
one that is merely behind.

**So read the numbers underneath and apply your own thresholds.**

- `detectors[].effectiveConfigs` — keys prefixed `observed.` are measurements, `effective.` are the
  thresholds that detector used. Compare them yourself.
- `health.checks[].effectiveConfigs["observed.delta.commits.since.last.compaction"]` against the
  configured trigger, not against the checker's slack-adjusted threshold.
- `layout.partitions[]` per-partition numbers rather than the table-level verdict.

The observation catalog is written against the underlying fields for exactly this reason. **An
observation fires in the band where the health checker correctly reports `HEALTHY`** — that band is
the whole subject of this skill.

Equally: never inherit a rollup verdict as a finding. "The layout analyzer flagged this table" is not
an observation. "40% of partitions with five or more base files average under 50 MB, against a
configured 120 MB target" is.

## Step 4 — carry the analyzer's caveats into your reasoning

Read `caveats[]` in the event-log JSON and repeat what it says. Two caveats govern almost everything.

**A phase share names where work was *forced*, not where it belongs.** Hudi sets the Spark job
description before the action, and Spark is lazy, so a bucket routinely carries the cost of an
untagged step that ran before it. Known cases, all verified in source:

- `SOURCE_READ_AND_TRANSFORM` bundles source read, user transformation and record creation.
  `StreamSync` tags only the fetch and the emptiness check; `HoodieStreamerUtils` carries no job
  status at all.
- `DEDUP_AND_INDEX_TAGGING` bundles deduplication with tagging. `BaseWriteHelper` has exactly one
  `setJobStatus`, and `deduplicateRecords` runs untagged immediately before it.
- `DATA_TABLE_WRITE` can absorb write work not forced until the commit action ran.

When a stage carries `bundledSteps`, the tool is telling you the share is large enough that the
bundling matters. Say what the number covers.

**With `hoodie.metadata.streaming.write.enabled=true` — the Spark default — a large share of the DAG
is unattributable.** On a measured run it was **60.5% of stage wall clock** in `UNKNOWN`: stages that
are not fused but simply unlabelled, because Hudi never calls `setJobStatus` on that path.

> **When `UNKNOWN` is large, say so before anything else, and do not reason over the remainder as if
> it were the whole picture.** A phase at "30% of wall clock" when `UNKNOWN` is 60% means something
> very different from the same number when `UNKNOWN` is 4%. Setting that config to `false` yields a
> fully attributable breakdown, and that is a legitimate thing to suggest **for a diagnostic run**.

Two more, worth stating when they come up:

- **`outputBytes` and `outputRecords` are structurally zero** on every stage of a Hudi write. Hudi
  writes Parquet through its own file handles, bypassing Spark's instrumented writer. **Never infer
  "no data was written" from them.** Input metrics are fine.
- **`-1` means "not derivable", never zero.** `skewRatio` and `gcFractionOfRunTime` are `-1` when
  undefined.

## Step 5 — match against the catalogs

**`references/observation-catalog.md`** — thirty rows of "this works, but could be better". Each row
names its measurable signals as JSON field paths, which inputs must corroborate it, the exact config
key and the direction to move it, a risk tier, and how to verify.

**`references/capability-catalog.md`** — seven rows of "adopt this feature", with signals **for and
against**, adoption cost, and risk tier.

Two habits that keep this honest:

**Never report an observation whose signals you did not actually find in the JSON.** If you cannot
name the field and the value, you do not have the observation. The catalogs are written in field
paths so that this check is mechanical.

**Never recommend a lever the config shows is inert.** This is the specific failure the writer
properties exist to prevent. Before recommending a key, check whether it is gated:

- `hoodie.parquet.small.file.limit` does nothing if the condition for appending never holds.
- `hoodie.metadata.index.partition.stats.enable` has **no independent effect** — partition stats
  require column stats and are inert without them.
- `hoodie.bloom.index.fileid.key.sorting.enable` does nothing on its own; bucketized checking stays
  on unless `hoodie.bloom.index.bucketized.checking=false` is also set.
- `hoodie.*.shuffle.parallelism` defaults to **`0`, not 200** — `0` means "inherit the input RDD's
  partition count", so a low task count may be reporting the source, not a Hudi setting.
- Setting `hoodie.bulkinsert.shuffle.parallelism` on an upsert workload does nothing at all.

## Step 6 — engine-effective defaults, not declared ones

`ConfigProperty.defaultValue()` is **not** the effective default. Builders override per engine with
`setDefaultValue(PROP, getDefaultXxx(engineType))`. A derived-expectation check that compares against
a declared default is wrong about every default Spark table.

| Config | Declared | **Spark** | Flink | Java |
|---|---|---|---|---|
| `hoodie.index.type` | — | **`SIMPLE`** | `INMEMORY` | `SIMPLE` |
| `hoodie.metadata.index.column.stats.enable` | `false` | **`true`** | `false` | `false` |
| `hoodie.metadata.streaming.write.enabled` | `false` | **`true`** | `false` | `false` |
| `hoodie.metadata.index.secondary.enable` | `true` | `true` | **`false`** | `true` |
| `hoodie.metadata.enable` | declared | declared | declared | **`false`** |
| `hoodie.write.markers.type` | — | **`TIMELINE_SERVER_BASED`** | `DIRECT` | `DIRECT` |

The clustering plan and execution strategy classes are also engine-selected, **and additionally
branch on whether the index is consistent-hashing bucket** — a config-dependent default, not just an
engine-dependent one.

The practical consequence for the flagship recommendation: **a Spark user is on `SIMPLE` by default,
not by choice.** Word it as *"you are on the default index; here is why the record-level index may
suit your workload"*. Never imply they chose it.

## Step 7 — refuse durable changes, structurally

| Tier | Meaning | What you do |
|---|---|---|
| `safe` | Reversible config, no effect on storage | Recommend |
| `operational` | Affects a table service; reversible but needs a cycle to observe, and the intermediate state is on storage | Recommend, with the cycle and the cost named |
| `durable` | **Changes what is on storage. Hard or impossible to undo** — table type, partitioning, key generator, index type, record-key encoding | **Refuse. Hand off** |

> **You do not recommend a `durable` change as an optimization.** Not as "you could just set", not in
> a config block, not as an aside. When the measurements point at one — CAP-001 (index type), CAP-002
> (bucket index and count), CAP-006 (table type), or OBS-025 — your output is a **report**: what you
> measured, what it implies, which counter-signals did and did not fire, and an explicit handoff to
> `hudi-architect` for an Architecture Decision Record.

Say the refusal out loud rather than silently omitting the row. The user should know the measurement
points somewhere, that you found it, and why you are not the one to act on it:

> *Your index tagging is 41% of wall clock and its task count tracks your base-file count rather than
> your batch size, which is the signature of an index whose cost scales with the table. That points
> at the record-level index. **Changing `hoodie.index.type` is a durable change — it alters what is on
> storage and cannot be reverted by resetting a config — so I will not recommend it as a tuning step.**
> Here are the measurements and the file-group estimate; take them to `hudi-architect` for an ADR.*

The report should be good enough that the design conversation starts from numbers. That is the point
of the handoff, not a way of ducking the question.

## Output shape

1. **What you have** — which of the three inputs are present, and the confidence that buys. First,
   every time.
2. **Where the time went** — the phase breakdown, with the `UNKNOWN` share stated up front if it is
   material, and the attribution caveat named.
3. **Observations** — matched catalog rows, each with: the measured values you actually found, which
   inputs corroborated, the change and direction, risk tier, and expected payoff. Order by payoff,
   not by catalog order.
4. **Capability candidates** — with signals for **and** against. Say "don't" when the counter-signals
   fire; that is the output, not a failure to produce one.
5. **Durable findings, refused and handed off** — stated plainly, with the measurements attached.
6. **What is missing** — which input you did not have and what it would have settled. Specifically,
   not as a generic ask.

Lead with the confidence statement and the biggest number. Do not recap the user's setup back to
them.

## Guardrails

**Recommend only. Never apply.** Do not run a table service, do not write to the table, do not edit
`.hoodie/`, do not modify a properties file. You read artifacts, run the three read-only tools, and
explain. That is the whole surface.

**Never recommend a `durable` change.** Above. Structural, not advisory.

**Never recommend a lever the config shows is inert.** Check the gating before you name the key.

**Never report an observation from one input.** Say "hypothesis" and name the input that would settle
it.

**Do not invent a config key.** Every `hoodie.*` key in the catalogs is verified against the source
tree. If a key is not in a catalog, say you are unsure of the exact key rather than guessing a
plausible one — **a misspelled Hudi config key produces no error at all**, the default silently
applies, and the user loses a whole tuning cycle finding out.

**Do not promise a number you cannot verify.** "This will make it 3x faster" is not something these
inputs support. "This removes the spill on that stage; the stage currently costs 12% of wall clock"
is.

**One change at a time where you can.** If you recommend five changes and the run gets slower, the
user has learned nothing. Order by payoff and say which one to try first.

**Releasing a savepoint is not routine.** A savepoint protects files from cleaning. If a stale
savepoint is pinning the timeline (OBS-020), report it and require a human to confirm it is no longer
needed. Never fold it into a list of tuning steps.

**Redact before quoting.** Users paste properties files containing bucket names, hostnames and
credentials. Do not echo them back unnecessarily.

**If the job is failing, stop and hand to `hudi-rca`.** This skill assumes a working deployment.
Tuning a job that is failing for an unrelated reason wastes the user's time and yours.

## Engine coverage

The recipes here come from Spark deployments, because that is where the evidence came from — the
event-log analyzer reads Spark event logs, and the measurements behind these rows were taken on Spark
runs.

The **core patterns apply to Flink too**: file sizing, service cadence, timeline growth, metadata-table
cost, index economics and the capability tradeoffs are table-layer mechanisms, not engine ones. What
does not carry is the event-log half — Flink produces no Spark event log, so a Flink user arrives with
two inputs at most, and the phase breakdown is simply unavailable. Several engine-effective defaults
also differ, as the table in Step 6 shows.

Flink-specific coverage will deepen over time. Say this plainly if a Flink user arrives: the
table-layer reasoning holds, the stage-level reasoning does not, and you will flag which is which
rather than pretending the distinction does not exist.

## References

- `references/observation-catalog.md` — thirty rows, "this works, but could be better".
- `references/capability-catalog.md` — seven rows, "adopt this feature", with counter-signals.
- `HoodieSparkEventLogAnalyzer.md` in `hudi-utilities` — the event-log JSON contract, the phase and
  operation model, and the caveats in full.
- `HoodieTableHealthChecker.md` and `HoodieTableLayoutAnalyzer.md` — the other two JSON contracts,
  and the thresholds their verdicts use.
- `hudi-rca` — when the job is failing rather than slow.
- `hudi-architect` — where every `durable` finding goes.

