Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.
The three budgets, the allocator inventory, the pool stack, the accounting gap
docs/source/contributor-guide/ffi.md
Where buffers physically live when a batch crosses the boundary
docs/source/contributor-guide/debugging.md
"Debugging Memory Reservations", for what evidence to ask the author for
docs/source/contributor-guide/tracing.md
jemalloc_allocated against summed reservations, the only measurement of the gap
Read memory_management.md in full before the diff. Most bad memory PRs are not wrong about code,
they are wrong about which of the three budgets they are affecting.
1. Name the Budget
There are three budgets, enforced by three different parties, and a PR that improves one can make
another worse:
Budget
Enforced by
Failure mode
JVM heap
The JVM
OutOfMemoryError, executor exits with code 52
Spark memory pools
Spark bookkeeping
A consumer spills, or the task fails with SparkOutOfMemoryError
Container RSS
OS or cgroup
SIGKILL of the whole executor, exit 137, OOMKilled
The first two are accounting, a running total of voluntarily declared bytes. The third is physical.
Ask which one the PR claims to fix and check that the mechanism actually reaches it. Lowering a
declared reservation does not reduce resident pages. Reducing resident pages does not by itself
prevent a SparkOutOfMemoryError.
Comet's declared reservations are a lower bound on real usage, for structural reasons the guide
enumerates: undeclared allocations in builders and kernels, 64-byte padding and doubling growth,
allocator fragmentation and retained pages, non-Rust allocations, and batches in flight across FFI.
A PR that treats reserved() as the real footprint is built on a false premise.
2. Reservations
Native operators reserve through DataFusion's MemoryConsumer and MemoryReservation API.
try_grow failure turns into spilling, not an error. For a spillable operator,
ResourcesExhausted is the signal to spill and retry. A new operator that propagates it as a
query failure has converted a recoverable condition into a lost task.
Treat a new grow call site as potential unbacked memory.grow cannot fail. When
Spark grants less than asked, the Comet pools record the full amount anyway and carry the
shortfall as overcommit, which Spark does not know about and can hand to another consumer
or task. Nothing caps how far successive grow calls overcommit short of the container
limit. That is acceptable only for memory that already exists and cannot be spilled, such
as a spilled batch read back from disk, so a new call site needs that justification.
Memory the caller is about to allocate belongs behind try_grow.
Every try_grow has a matching shrink, including on the error path. A reservation
leaked on an early return is charged for the life of the task.
In try_grow, a partial grant is released, not kept.acquired < additional must
release and report ResourcesExhausted. Only grow keeps a partial grant, as overcommit.
An operator that buffers without reserving is invisible to the pool. If the PR adds
buffering, ask where the reservation is. "It is only a few batches" is how the accounting gap
grows.
Reservation size matches what is actually held. Check the PR uses buffer sizes read off
the ArrayData, as get_reserved_bytes_for_record_batch and get_record_batch_memory_size
do, rather than a row-count estimate.
3. The Pool Stack
create_memory_pool builds a base pool and wraps it in decorators. In the default off-heap
configuration, from the inside out:
[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
[TaskSharedMemoryPool] <- RAII handle for the per-task registry
[TrackConsumersPool] <- DataFusion, names the top 10 consumers in error messages
[CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
A new decorator forwards everyMemoryPool method to its inner pool. A partial
implementation makes reserved() disagree between levels.
fair_unified holds the consumers that can spill to a share of the task's share. The
task's share is spark.memory.offHeap.size divided by the tasks on the executor running
native plans, which RunningTasks counts while each task holds its task-shared pool, or
pool_size if that is smaller. A consumer that can spill is
refused if what it holds plus the request exceeds what the consumers that cannot spill leave
of the task's share, divided by the registered consumers that can spill, or if the task's
total plus the request exceeds the task's share. A consumer that cannot spill is refused only
if the task's total plus the request exceeds pool_size, which binds before Spark's own
limit only while the deprecated fraction is below 1 / N. With eight running tasks, a 16 GiB
pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
that new_empty(), split() and take() create count against one share. A PR that checks
reservation.size() instead lets an operator with several reservations, such as a sort's
streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
size, which is after it calls try_grow but before it calls shrink. A PR that limits a
consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
#7088: one operator takes the task's share from Spark and a merge or hash join build fails
the task. If the PR changes any of these comparisons it changes when every query spills, so
it needs benchmark evidence, not reasoning.
The consumers that divide the task's share are every consumer that can spill in the
task, across every native plan, because the pool is task-shared.
Task-shared pool lifetime.acquire_task_shared_pool keeps a process-wide
HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>. The returned Arc is the only
lifetime handle, so there is no explicit release to forget and a createPlan that fails
partway cleans up on unwind. A PR that adds an explicit release call, or stores the Arc
somewhere longer-lived, breaks that property. TaskSharedMemoryPool::drop compares pointers
so it only removes an entry that is still its own, which handles the race where an acquire
observes an expired Weak and inserts a replacement first. Do not let that check be
simplified away.
fair_unified and greedy_unified are the only pool types. On-heap mode ignores the
pool-type string and always gets UnboundedMemoryPool: it exists so the Spark SQL test suite
can run against Comet, it accounts for nothing, and it must not be used in production. A PR
re-adding a sized on-heap pool is reintroducing a budget that bounds nothing real (see
issue #6063).
4. The Spark Bridge
CometUnifiedMemoryPool and CometFairMemoryPool call CometTaskMemoryManager.acquireMemory over
JNI, which goes through Spark's ordinary TaskMemoryManager. Two asymmetries matter:
Spark can force its own consumers in the same task to spill to satisfy a Comet request.
Comet cannot be forced to spill by Spark. NativeMemoryConsumer.spill() returns 0, so a
JVM consumer blocked behind native reservations has no way to reclaim them. Native operators
spill only when their own try_grow fails.
If the PR claims Spark-side reclamation works, check spill(). If the PR implements native
reclamation, that is a significant change and needs to say what it does to in-flight operators.
5. Budget and Configuration
CometExecIterator.getMemoryConfig computes the budget once per executor and passes it across JNI
as memory_limit, along with spark.memory.offHeap.size itself as off_heap_size, which
fair_unified divides among the tasks running native plans:
A new memory config goes through getMemoryConfig rather than being read separately in
native code, and a new value crossing JNI is added to createPlan on both sides.
Changing memoryPool.fraction semantics affects every deployment that tuned it as a haircut
for the accounting gap.
On Kubernetes, spark.memory.offHeap.size is part of the pod limit, not headroom on top
of it. A PR whose fix is "raise the off-heap size" is asking for fewer executors per node.
spark.executor.memoryOverhead is the only real slack in the container, and JVM non-heap
usage already consumes much of it.
6. Unaccounted Allocators
Two allocators sit outside Comet's pool entirely, and they are easy to confuse:
CometArrowAllocator (spark/src/main/scala/org/apache/comet/package.scala) is a
process-wide RootAllocator(Long.MaxValue). It is unbounded and no budget sees it. Child
allocators are cut from it for FFI stream export, broadcast coalescing, and
CometSparkToColumnarExec. A PR that adds a child allocator, or makes an existing one hold more,
is adding uncounted container RSS. Say so even if the volume is small.
JVM shuffle pages go through CometShuffleMemoryAllocator.getInstance, which returns
CometUnifiedShuffleMemoryAllocator, an ordinary Spark MemoryConsumer drawing from
spark.memory.offHeap.size. These are arbitrated by Spark against its other consumers. Do
not review them as if they were native allocations.
7. Evidence
Memory changes are hard to unit test, so the review question is what evidence exists rather than
what test was added. Ask for at least one of:
spark.comet.debug.memory=true output from LoggingMemoryPool, showing register, grow, and
shrink with consumer names
A trace comparing jemalloc_allocated against the summed
thread_NNN_comet_memory_reserved values, which is the only way to see the accounting gap
Spill counts from spark.comet.explain.native.enabled=true, before and after
For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills
under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the
ability to spill again during that replay, and only a run at a small off-heap size showed it
(#6254).
For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or
OOMKilled is the cgroup. 52 with java.lang.OutOfMemoryError is JVM heap. A failed task with
SparkOutOfMemoryError and a surviving executor is Spark's pool, the only one of the three that
is recoverable at task level.
spark/src/test/scala/org/apache/spark/CometTaskMemoryManagerSuite.scala and
CometUnboundedShuffleMemoryAllocatorSuite.scala are the existing JVM-side tests. A change to the
bridge or an allocator should extend one of them.
8. Does the PR Make memory_management.md Stale?
This doc states invariants and enumerates known gaps, so it goes stale in ways a file-path check
will not catch. Check:
The Who allocates what table, if the PR adds an allocator or changes what bounds one
The pool stack diagram, if a decorator is added, removed, or reordered
The pool type table, if a pool type is added, removed, or changes how it is sized
The memory_limit formula, if the budget calculation changes
The fair pool description, which spells out the task share, the share and task-total
checks, and their consequences
The Crossing the FFI boundary section, which names the operators that reserve for imported
batches
The Open problems list. If the PR closes one of these gaps, the entry must be removed or
rewritten in the same PR. Leaving a fixed problem listed as open is worse than no doc, because
the next contributor will re-fix it.
The Debugging memory issues table and checklist, if a tool or signal is added or removed
1---2name: review-comet-memory-pr3description: Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.4argument-hint: <pr-number>5metadata:6 source: https://skillmd.com/skills/apache/review-comet-memory-pr7---89<!--10Licensed to the Apache Software Foundation (ASF) under one11or more contributor license agreements. See the NOTICE file12distributed with this work for additional information13regarding copyright ownership. The ASF licenses this file14to you under the Apache License, Version 2.0 (the15"License"); you may not use this file except in compliance16with the License. You may obtain a copy of the License at1718 http://www.apache.org/licenses/LICENSE-2.01920Unless required by applicable law or agreed to in writing,21software distributed under the License is distributed on an22"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY23KIND, either express or implied. See the License for the24specific language governing permissions and limitations25under the License.26-->2728Memory-specific review for Comet PR #$ARGUMENTS.2930**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review31bar, and the output format. This skill only covers memory.3233## Read the Contributor Guide First3435| Doc | What you need from it |36| ---------------------------------------------------- | --------------------------------------------------------------------------------- |37| `docs/source/contributor-guide/memory_management.md` | The three budgets, the allocator inventory, the pool stack, the accounting gap |38| `docs/source/contributor-guide/ffi.md` | Where buffers physically live when a batch crosses the boundary |39| `docs/source/contributor-guide/debugging.md` | "Debugging Memory Reservations", for what evidence to ask the author for |40| `docs/source/contributor-guide/tracing.md` | `jemalloc_allocated` against summed reservations, the only measurement of the gap |4142Read `memory_management.md` in full before the diff. Most bad memory PRs are not wrong about code,43they are wrong about which of the three budgets they are affecting.4445## 1. Name the Budget4647There are three budgets, enforced by three different parties, and a PR that improves one can make48another worse:4950| Budget | Enforced by | Failure mode |51| ------------------ | ----------------- | ----------------------------------------------------------------- |52| JVM heap | The JVM | `OutOfMemoryError`, executor exits with code 52 |53| Spark memory pools | Spark bookkeeping | A consumer spills, or the task fails with `SparkOutOfMemoryError` |54| Container RSS | OS or cgroup | `SIGKILL` of the whole executor, exit 137, `OOMKilled` |5556The first two are accounting, a running total of voluntarily declared bytes. The third is physical.57Ask which one the PR claims to fix and check that the mechanism actually reaches it. Lowering a58declared reservation does not reduce resident pages. Reducing resident pages does not by itself59prevent a `SparkOutOfMemoryError`.6061Comet's declared reservations are a **lower bound** on real usage, for structural reasons the guide62enumerates: undeclared allocations in builders and kernels, 64-byte padding and doubling growth,63allocator fragmentation and retained pages, non-Rust allocations, and batches in flight across FFI.64A PR that treats `reserved()` as the real footprint is built on a false premise.6566## 2. Reservations6768Native operators reserve through DataFusion's `MemoryConsumer` and `MemoryReservation` API.6970- [ ] **`try_grow` failure turns into spilling, not an error.** For a spillable operator,71 `ResourcesExhausted` is the signal to spill and retry. A new operator that propagates it as a72 query failure has converted a recoverable condition into a lost task.73- [ ] **Treat a new `grow` call site as potential unbacked memory.** `grow` cannot fail. When74 Spark grants less than asked, the Comet pools record the full amount anyway and carry the75 shortfall as overcommit, which Spark does not know about and can hand to another consumer76 or task. Nothing caps how far successive `grow` calls overcommit short of the container77 limit. That is acceptable only for memory that already exists and cannot be spilled, such78 as a spilled batch read back from disk, so a new call site needs that justification.79 Memory the caller is about to allocate belongs behind `try_grow`.80- [ ] **Every `try_grow` has a matching `shrink`, including on the error path.** A reservation81 leaked on an early return is charged for the life of the task.82- [ ] **In `try_grow`, a partial grant is released, not kept.** `acquired < additional` must83 release and report `ResourcesExhausted`. Only `grow` keeps a partial grant, as overcommit.84- [ ] **An operator that buffers without reserving is invisible to the pool.** If the PR adds85 buffering, ask where the reservation is. "It is only a few batches" is how the accounting gap86 grows.87- [ ] **Reservation size matches what is actually held.** Check the PR uses buffer sizes read off88 the `ArrayData`, as `get_reserved_bytes_for_record_batch` and `get_record_batch_memory_size`89 do, rather than a row-count estimate.9091## 3. The Pool Stack9293`create_memory_pool` builds a base pool and wraps it in decorators. In the default off-heap94configuration, from the inside out:9596```text97[LoggingMemoryPool] <- only when spark.comet.debug.memory=true98 [TaskSharedMemoryPool] <- RAII handle for the per-task registry99 [TrackConsumersPool] <- DataFusion, names the top 10 consumers in error messages100 [CometFairMemoryPool] <- delegates acquire/release to Spark over JNI101```102103- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial104 implementation makes `reserved()` disagree between levels.105- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The106 task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running107 native plans, which `RunningTasks` counts while each task holds its task-shared pool, or108 `pool_size` if that is smaller. A consumer that can spill is109 refused if what it holds plus the request exceeds what the consumers that cannot spill leave110 of the task's share, divided by the registered consumers that can spill, or if the task's111 total plus the request exceeds the task's share. A consumer that cannot spill is refused only112 if the task's total plus the request exceeds `pool_size`, which binds before Spark's own113 limit only while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB114 pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds115 512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations116 that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks117 `reservation.size()` instead lets an operator with several reservations, such as a sort's118 streaming merge, take other consumers' shares. It also depends on when DataFusion updates the119 size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a120 consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back121 #7088: one operator takes the task's share from Spark and a merge or hash join build fails122 the task. If the PR changes any of these comparisons it changes when every query spills, so123 it needs benchmark evidence, not reasoning.124- [ ] **The consumers that divide the task's share are every consumer that can spill in the125 task**, across every native plan, because the pool is task-shared.126- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide127 `HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. The returned `Arc` is the only128 lifetime handle, so there is no explicit release to forget and a `createPlan` that fails129 partway cleans up on unwind. A PR that adds an explicit release call, or stores the `Arc`130 somewhere longer-lived, breaks that property. `TaskSharedMemoryPool::drop` compares pointers131 so it only removes an entry that is still its own, which handles the race where an `acquire`132 observes an expired `Weak` and inserts a replacement first. Do not let that check be133 simplified away.134- [ ] **`fair_unified` and `greedy_unified` are the only pool types.** On-heap mode ignores the135 pool-type string and always gets `UnboundedMemoryPool`: it exists so the Spark SQL test suite136 can run against Comet, it accounts for nothing, and it must not be used in production. A PR137 re-adding a sized on-heap pool is reintroducing a budget that bounds nothing real (see138 issue #6063).139140## 4. The Spark Bridge141142`CometUnifiedMemoryPool` and `CometFairMemoryPool` call `CometTaskMemoryManager.acquireMemory` over143JNI, which goes through Spark's ordinary `TaskMemoryManager`. Two asymmetries matter:144145- Spark **can** force its own consumers in the same task to spill to satisfy a Comet request.146- Comet **cannot** be forced to spill by Spark. `NativeMemoryConsumer.spill()` returns `0`, so a147 JVM consumer blocked behind native reservations has no way to reclaim them. Native operators148 spill only when their own `try_grow` fails.149150If the PR claims Spark-side reclamation works, check `spill()`. If the PR implements native151reclamation, that is a significant change and needs to say what it does to in-flight operators.152153## 5. Budget and Configuration154155`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI156as `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which157`fair_unified` divides among the tasks running native plans:158159```text160memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction161off_heap_size = spark.memory.offHeap.size162```163164- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in165 native code, and a new value crossing JNI is added to `createPlan` on both sides.166- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut167 for the accounting gap.168- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top169 of it. A PR whose fix is "raise the off-heap size" is asking for fewer executors per node.170 `spark.executor.memoryOverhead` is the only real slack in the container, and JVM non-heap171 usage already consumes much of it.172173## 6. Unaccounted Allocators174175Two allocators sit outside Comet's pool entirely, and they are easy to confuse:176177- **`CometArrowAllocator`** (`spark/src/main/scala/org/apache/comet/package.scala`) is a178 process-wide `RootAllocator(Long.MaxValue)`. It is unbounded and no budget sees it. Child179 allocators are cut from it for FFI stream export, broadcast coalescing, and180 `CometSparkToColumnarExec`. A PR that adds a child allocator, or makes an existing one hold more,181 is adding uncounted container RSS. Say so even if the volume is small.182- **JVM shuffle pages** go through `CometShuffleMemoryAllocator.getInstance`, which returns183 `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer` drawing from184 `spark.memory.offHeap.size`. These **are** arbitrated by Spark against its other consumers. Do185 not review them as if they were native allocations.186187## 7. Evidence188189Memory changes are hard to unit test, so the review question is what evidence exists rather than190what test was added. Ask for at least one of:191192- `spark.comet.debug.memory=true` output from `LoggingMemoryPool`, showing register, grow, and193 shrink with consumer names194- A trace comparing `jemalloc_allocated` against the summed195 `thread_NNN_comet_memory_reserved` values, which is the only way to see the accounting gap196- Spill counts from `spark.comet.explain.native.enabled=true`, before and after197- For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills198 under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the199 ability to spill again during that replay, and only a run at a small off-heap size showed it200 (#6254).201- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or202 `OOMKilled` is the cgroup. 52 with `java.lang.OutOfMemoryError` is JVM heap. A failed task with203 `SparkOutOfMemoryError` and a surviving executor is Spark's pool, the only one of the three that204 is recoverable at task level.205206`spark/src/test/scala/org/apache/spark/CometTaskMemoryManagerSuite.scala` and207`CometUnboundedShuffleMemoryAllocatorSuite.scala` are the existing JVM-side tests. A change to the208bridge or an allocator should extend one of them.209210## 8. Does the PR Make `memory_management.md` Stale?211212This doc states invariants and enumerates known gaps, so it goes stale in ways a file-path check213will not catch. Check:214215- The **Who allocates what** table, if the PR adds an allocator or changes what bounds one216- The **pool stack** diagram, if a decorator is added, removed, or reordered217- The **pool type** table, if a pool type is added, removed, or changes how it is sized218- The `memory_limit` formula, if the budget calculation changes219- The **fair pool** description, which spells out the task share, the share and task-total220 checks, and their consequences221- The **Crossing the FFI boundary** section, which names the operators that reserve for imported222 batches223- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or224 rewritten in the same PR. Leaving a fixed problem listed as open is worse than no doc, because225 the next contributor will re-fix it.226- The **Debugging memory issues** table and checklist, if a tool or signal is added or removed
Run npx get-skill add apache/review-comet-memory-pr in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr. It is listed under Coding & Dev Tools on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
apache (@apache) published this skill. Their other Agent Skills are listed on their SkillMD profile.