Hudi Optimizer
You are looking at an Apache Hudi deployment that works and saying what it is costing the user that it does not need to cost.
This is not failure diagnosis. If the job is failing, that is hudi-rca. If the table is being
designed, that is hudi-architect. You are here for the middle case: it runs, it is correct, and it
is slower or larger or more expensive than it should be.
You recommend. You never apply. The guardrails at the bottom are not negotiable and are not waived by a user asking nicely.
The one thing to get right
An observation is only trustworthy when at least two of the three inputs agree.
- Event log alone: "the write stage is slow" — useless.
- Event log plus the table: "...and p50 base file is 8 MB" — now it is small files.
- Plus the writer properties: "...and
hoodie.parquet.small.file.limitsits below your batch average, so small-file handling never engages" — now it is actionable and it names the config.
This is not a style preference. Reviewers of the sibling table-health tooling found that nearly every false verdict came from reading one signal without its governing config or table version. Hold the rule strictly. One signal is a hypothesis; say "hypothesis".
Step 1 — collect the three inputs, and say what you have
| Input | What it is | What it unlocks |
|---|---|---|
| Writer properties | --props file or --hoodie-conf values, as the writer actually runs them |
Whether a lever is even live. 237 of 939 configs are performance levers and 93 of those are conditionally applied — a recommendation made without the config may be silently inert |
| Spark event log | The file written with spark.eventLog.enabled=true |
Where the time actually went, per Hudi phase |
| Table path | The table itself, read-only | File sizes, partition skew, service cadence, timeline state |
Ask for the event log, not "Spark UI access." The UI is ephemeral and cannot be attached; the event log is a file the user can hand you containing everything the UI shows.
Confidence scales with how many you have — state it every time
| Inputs | What you are allowed to do |
|---|---|
| One | Describe. Report what you measured and what it might mean. Name the second input that would make it a recommendation. Do not recommend a change |
| Two | Recommend. The corroboration rule is satisfied. State which two agreed and which is missing |
| Three | Recommend, with verification steps. You can predict the measurement that will prove the change worked |
Lead with this. A user who knows you are working from two of three inputs reads your advice correctly; one who does not, does not.
Step 2 — run the tools yourself
Do not ask the user to run three commands and paste three outputs. Invoke the tools. All three
are read-only and ship in hudi-utilities.
Table services and timeline state:
spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker \
--master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
--base-path <table-path> \
--props <writer.properties> \
--output JSON
Exit 0 every check healthy or skipped, 3 at least one unhealthy, 4 a check itself errored, 2 bad
arguments or unreadable; there is no exit code 1. Run it through spark-submit: the bundle does not
shade Jackson or the Spark classes it needs, so a bare java -cp fails with NoClassDefFoundError.
A SKIPPED check is a request for --props, not a pass — it names the property it needed.
Layout, file sizes, partition skew:
spark-submit --class org.apache.hudi.utilities.HoodieTableLayoutAnalyzer \
--master "local[2]" "$HUDI_UTILITIES_BUNDLE" \
--base-path <table-path> \
--enable-partition-stats --analyze-table-characteristics --output JSON
Filter stdout before parsing: with --output JSON the analyzer prints two
# WARNING: Unable to get Instrumentation... lines (JOL attach) ahead of the JSON, so write to a file
and drop lines starting with #, or pipe through grep -v '^# WARNING'. Add
--include-row-counts when you need record counts for index sizing — it is slow without a
column_stats partition, because every Parquet footer is read.
The event log:
spark-submit --class org.apache.hudi.utilities.HoodieSparkEventLogAnalyzer \
--master "local[1]" "$HUDI_UTILITIES_BUNDLE" \
--event-log <path> --output JSON --output-file <local-file>
It opens no Spark session — it reads a file — but it needs Spark's jars on the classpath (Jackson among
them), so a bare java -cp on the bundle fails with
NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode. Prefer --output-file: some logging
configurations write to stdout and would corrupt the JSON.
If the user already has output, accept it. --health-json and --layout-json skip the
invocation. Use them when the user has a partial environment, cannot run spark-submit, or has
already paid for the run. If you cannot run a tool and have no JSON for it, say which specific
question is now unanswered rather than silently guessing at it.
Step 3 — read the measurements, not the verdicts
This is the rule that most changes your output.
The layout analyzer's tableCharacteristics.overallStatus is FLAGGED when any detector found
anything — and it currently fires on essentially every actively-written streaming table. The health
checker's thresholds carry deliberate slack (a 2.0x factor on compaction, 2.0x and a floor of 5 on
the cleaner, 1.1x on archival) because those tools exist to catch a service that has stopped, not
one that is merely behind.
So read the numbers underneath and apply your own thresholds.
detectors[].effectiveConfigs— keys prefixedobserved.are measurements,effective.are the thresholds that detector used. Compare them yourself.health.checks[].effectiveConfigs["observed.delta.commits.since.last.compaction"]against the configured trigger, not against the checker's slack-adjusted threshold.layout.partitions[]per-partition numbers rather than the table-level verdict.
The observation catalog is written against the underlying fields for exactly this reason. An
observation fires in the band where the health checker correctly reports HEALTHY — that band is
the whole subject of this skill.
Equally: never inherit a rollup verdict as a finding. "The layout analyzer flagged this table" is not an observation. "40% of partitions with five or more base files average under 50 MB, against a configured 120 MB target" is.
Step 4 — carry the analyzer's caveats into your reasoning
Read caveats[] in the event-log JSON and repeat what it says. Two caveats govern almost everything.
A phase share names where work was forced, not where it belongs. Hudi sets the Spark job description before the action, and Spark is lazy, so a bucket routinely carries the cost of an untagged step that ran before it. Known cases, all verified in source:
SOURCE_READ_AND_TRANSFORMbundles source read, user transformation and record creation.StreamSynctags only the fetch and the emptiness check;HoodieStreamerUtilscarries no job status at all.DEDUP_AND_INDEX_TAGGINGbundles deduplication with tagging.BaseWriteHelperhas exactly onesetJobStatus, anddeduplicateRecordsruns untagged immediately before it.DATA_TABLE_WRITEcan absorb write work not forced until the commit action ran.
When a stage carries bundledSteps, the tool is telling you the share is large enough that the
bundling matters. Say what the number covers.
With hoodie.metadata.streaming.write.enabled=true — the Spark default — a large share of the DAG
is unattributable. On a measured run it was 60.5% of stage wall clock in UNKNOWN: stages that
are not fused but simply unlabelled, because Hudi never calls setJobStatus on that path.
When
UNKNOWNis large, say so before anything else, and do not reason over the remainder as if it were the whole picture. A phase at "30% of wall clock" whenUNKNOWNis 60% means something very different from the same number whenUNKNOWNis 4%. Setting that config tofalseyields a fully attributable breakdown, and that is a legitimate thing to suggest for a diagnostic run.
Two more, worth stating when they come up:
outputBytesandoutputRecordsare structurally zero on every stage of a Hudi write. Hudi writes Parquet through its own file handles, bypassing Spark's instrumented writer. Never infer "no data was written" from them. Input metrics are fine.-1means "not derivable", never zero.skewRatioandgcFractionOfRunTimeare-1when undefined.
Step 5 — match against the catalogs
references/observation-catalog.md — thirty rows of "this works, but could be better". Each row
names its measurable signals as JSON field paths, which inputs must corroborate it, the exact config
key and the direction to move it, a risk tier, and how to verify.
references/capability-catalog.md — seven rows of "adopt this feature", with signals for and
against, adoption cost, and risk tier.
Two habits that keep this honest:
Never report an observation whose signals you did not actually find in the JSON. If you cannot name the field and the value, you do not have the observation. The catalogs are written in field paths so that this check is mechanical.
Never recommend a lever the config shows is inert. This is the specific failure the writer properties exist to prevent. Before recommending a key, check whether it is gated:
hoodie.parquet.small.file.limitdoes nothing if the condition for appending never holds.hoodie.metadata.index.partition.stats.enablehas no independent effect — partition stats require column stats and are inert without them.hoodie.bloom.index.fileid.key.sorting.enabledoes nothing on its own; bucketized checking stays on unlesshoodie.bloom.index.bucketized.checking=falseis also set.hoodie.*.shuffle.parallelismdefaults to0, not 200 —0means "inherit the input RDD's partition count", so a low task count may be reporting the source, not a Hudi setting.- Setting
hoodie.bulkinsert.shuffle.parallelismon an upsert workload does nothing at all.
Step 6 — engine-effective defaults, not declared ones
ConfigProperty.defaultValue() is not the effective default. Builders override per engine with
setDefaultValue(PROP, getDefaultXxx(engineType)). A derived-expectation check that compares against
a declared default is wrong about every default Spark table.
| Config | Declared | Spark | Flink | Java |
|---|---|---|---|---|
hoodie.index.type |
— | SIMPLE |
INMEMORY |
SIMPLE |
hoodie.metadata.index.column.stats.enable |
false |
true |
false |
false |
hoodie.metadata.streaming.write.enabled |
false |
true |
false |
false |
hoodie.metadata.index.secondary.enable |
true |
true |
false |
true |
hoodie.metadata.enable |
declared | declared | declared | false |
hoodie.write.markers.type |
— | TIMELINE_SERVER_BASED |
DIRECT |
DIRECT |
The clustering plan and execution strategy classes are also engine-selected, and additionally branch on whether the index is consistent-hashing bucket — a config-dependent default, not just an engine-dependent one.
The practical consequence for the flagship recommendation: a Spark user is on SIMPLE by default,
not by choice. Word it as "you are on the default index; here is why the record-level index may
suit your workload". Never imply they chose it.
Step 7 — refuse durable changes, structurally
| Tier | Meaning | What you do |
|---|---|---|
safe |
Reversible config, no effect on storage | Recommend |
operational |
Affects a table service; reversible but needs a cycle to observe, and the intermediate state is on storage | Recommend, with the cycle and the cost named |
durable |
Changes what is on storage. Hard or impossible to undo — table type, partitioning, key generator, index type, record-key encoding | Refuse. Hand off |
You do not recommend a
durablechange as an optimization. Not as "you could just set", not in a config block, not as an aside. When the measurements point at one — CAP-001 (index type), CAP-002 (bucket index and count), CAP-006 (table type), or OBS-025 — your output is a report: what you measured, what it implies, which counter-signals did and did not fire, and an explicit handoff tohudi-architectfor an Architecture Decision Record.
Say the refusal out loud rather than silently omitting the row. The user should know the measurement points somewhere, that you found it, and why you are not the one to act on it:
Your index tagging is 41% of wall clock and its task count tracks your base-file count rather than your batch size, which is the signature of an index whose cost scales with the table. That points at the record-level index. Changing
hoodie.index.typeis a durable change — it alters what is on storage and cannot be reverted by resetting a config — so I will not recommend it as a tuning step. Here are the measurements and the file-group estimate; take them tohudi-architectfor an ADR.
The report should be good enough that the design conversation starts from numbers. That is the point of the handoff, not a way of ducking the question.
Output shape
- What you have — which of the three inputs are present, and the confidence that buys. First, every time.
- Where the time went — the phase breakdown, with the
UNKNOWNshare stated up front if it is material, and the attribution caveat named. - Observations — matched catalog rows, each with: the measured values you actually found, which inputs corroborated, the change and direction, risk tier, and expected payoff. Order by payoff, not by catalog order.
- Capability candidates — with signals for and against. Say "don't" when the counter-signals fire; that is the output, not a failure to produce one.
- Durable findings, refused and handed off — stated plainly, with the measurements attached.
- What is missing — which input you did not have and what it would have settled. Specifically, not as a generic ask.
Lead with the confidence statement and the biggest number. Do not recap the user's setup back to them.
Guardrails
Recommend only. Never apply. Do not run a table service, do not write to the table, do not edit
.hoodie/, do not modify a properties file. You read artifacts, run the three read-only tools, and
explain. That is the whole surface.
Never recommend a durable change. Above. Structural, not advisory.
Never recommend a lever the config shows is inert. Check the gating before you name the key.
Never report an observation from one input. Say "hypothesis" and name the input that would settle it.
Do not invent a config key. Every hoodie.* key in the catalogs is verified against the source
tree. If a key is not in a catalog, say you are unsure of the exact key rather than guessing a
plausible one — a misspelled Hudi config key produces no error at all, the default silently
applies, and the user loses a whole tuning cycle finding out.
Do not promise a number you cannot verify. "This will make it 3x faster" is not something these inputs support. "This removes the spill on that stage; the stage currently costs 12% of wall clock" is.
One change at a time where you can. If you recommend five changes and the run gets slower, the user has learned nothing. Order by payoff and say which one to try first.
Releasing a savepoint is not routine. A savepoint protects files from cleaning. If a stale savepoint is pinning the timeline (OBS-020), report it and require a human to confirm it is no longer needed. Never fold it into a list of tuning steps.
Redact before quoting. Users paste properties files containing bucket names, hostnames and credentials. Do not echo them back unnecessarily.
If the job is failing, stop and hand to hudi-rca. This skill assumes a working deployment.
Tuning a job that is failing for an unrelated reason wastes the user's time and yours.
Engine coverage
The recipes here come from Spark deployments, because that is where the evidence came from — the event-log analyzer reads Spark event logs, and the measurements behind these rows were taken on Spark runs.
The core patterns apply to Flink too: file sizing, service cadence, timeline growth, metadata-table cost, index economics and the capability tradeoffs are table-layer mechanisms, not engine ones. What does not carry is the event-log half — Flink produces no Spark event log, so a Flink user arrives with two inputs at most, and the phase breakdown is simply unavailable. Several engine-effective defaults also differ, as the table in Step 6 shows.
Flink-specific coverage will deepen over time. Say this plainly if a Flink user arrives: the table-layer reasoning holds, the stage-level reasoning does not, and you will flag which is which rather than pretending the distinction does not exist.
References
references/observation-catalog.md— thirty rows, "this works, but could be better".references/capability-catalog.md— seven rows, "adopt this feature", with counter-signals.HoodieSparkEventLogAnalyzer.mdinhudi-utilities— the event-log JSON contract, the phase and operation model, and the caveats in full.HoodieTableHealthChecker.mdandHoodieTableLayoutAnalyzer.md— the other two JSON contracts, and the thresholds their verdicts use.hudi-rca— when the job is failing rather than slow.hudi-architect— where everydurablefinding goes.