MM Large Folios, THP, and Hugetlb

MM Large Folios, THP, and Hugetlb

Large Folio State Tracking

Misunderstanding which flags are per-page vs per-folio leads to bugs where code checks or sets state on the wrong struct page. A common mistake is assuming all flags work like small pages when operating on subpages of a large folio.

The Tracking Level column indicates where the state lives. Per-folio means a single value on the head page applies to the entire folio. Per-page means each subpage carries its own independent value. PTE-level means the state is in the page table entry, not in struct page at all. Mixed means the granularity depends on how the folio is mapped.

Page flag policies control which struct page within a folio carries each flag. Using the wrong page silently reads stale data or corrupts unrelated state. See the "Page flags policies wrt compound pages" comment block in include/linux/page-flags.h:

• PF_HEAD: flag operations redirect to head page (most flags)
• PF_ANY: flag is relevant for head, tail, and small pages
• PF_NO_TAIL: modifications only on head/small pages, reads allowed on tail
• PF_SECOND: flag stored in the first tail page (e.g., PG_has_hwpoisoned, PG_large_rmappable, PG_partially_mapped)

Atomic vs non-atomic page flag operations:

Non-atomic flag operations (__set_bit / __clear_bit, generated by __FOLIO_SET_FLAG / __FOLIO_CLEAR_FLAG in include/linux/page-flags.h) perform a read-modify-write on the entire unsigned long flags word. They are only safe when the caller has exclusive access to the whole flags word -- not merely when access to the individual bit is serialized by a lock. Multiple page flags share the same unsigned long, so a lock that serializes one flag does NOT protect against concurrent modification of a different flag in the same word by code running under a different lock.

This is especially important for PF_SECOND flags (stored on the first tail page): PG_has_hwpoisoned, PG_large_rmappable, PG_partially_mapped, and PG_anon_exclusive (via PF_ANY on tail pages) all share the same flags word and are modified by different subsystems under different locks.

REPORT as bugs: Code that uses non-atomic page flag operations (__folio_set_* / __folio_clear_* / __SetPage* / __ClearPage*) on a page whose flags word can be concurrently modified by another code path, unless the caller holds exclusive access to the entire page (e.g., during allocation before the page is visible, or during final freeing).

Page Cache Reference Counting for Large Folios

__filemap_add_folio() in mm/filemap.c adds folio_nr_pages(folio) extra references. A page-cache folio's refcount = 1 (base) + folio_nr_pages() (page cache) + other holders. Removal must drop all page-cache refs via folio_put_refs(folio, folio_nr_pages(folio)) (see filemap_free_folio() in mm/filemap.c). Using folio_put() (single-ref drop) after __filemap_remove_folio() leaks folio_nr_pages() - 1 refs -- a silent memory leak only visible with large folios.

Large Folio Split Minimum Order

Splitting a file-backed large folio below the mapping's minimum folio order fails with -EINVAL. Callers that assume a successful split always yields order-0 folios will hit warnings, operate on unexpectedly large folios, or mishandle error paths on LBS (large block size) filesystems.

File-backed address spaces can set a minimum folio order via mapping_set_folio_min_order() (see mapping_min_folio_order() in include/linux/pagemap.h). The split infrastructure in __folio_split() in mm/huge_memory.c enforces this: if new_order < min_order, the split returns -EINVAL.

Split API behavior:

• split_huge_page() and split_folio_to_list() always request order-0. They will fail for file-backed folios whose mapping has min order > 0.
• try_folio_split_to_order() takes an explicit new_order parameter. Its documentation states: "Use min_order_for_split() to get the lower bound of @new_order" (see include/linux/huge_mm.h).
• min_order_for_split() in mm/huge_memory.c returns mapping_min_folio_order(folio->mapping) for file-backed folios and 0 for anonymous folios.

Review checklist for folio split callers:

• If code calls split_huge_page() or split_folio_to_list() and then assumes the result is order-0 (e.g., WARN_ON(folio_test_large(folio))), verify it handles -EINVAL from mappings with min order > 0
• If code needs to split to the lowest possible order, it must call min_order_for_split() first and pass that order explicitly to try_folio_split_to_order() or split_huge_page_to_list_to_order()
• After a successful split to non-zero order, the resulting folios are still large (folio_test_large() returns true); code must not assume they are base pages

Large Folio Split Refcount Precondition

Calling split_folio() on a large folio while extra references are held causes it to return -EAGAIN, and if the caller retries in a loop, this creates a livelock that only manifests under contention from multiple tasks operating on the same folio.

split_folio() calls __folio_split() in mm/huge_memory.c, which checks folio_expected_ref_count(folio) != folio_ref_count(folio) - 1. The expected count is computed by folio_expected_ref_count() in include/linux/mm.h, summing references from mapcount, page cache, swap cache, and PG_private. The - 1 accounts for the single caller pin. Any additional folio_get() references -- such as those held by other tasks blocked on folio_lock() -- cause this check to fail.

Dangerous ordering (refcount before lock):

// WRONG: raises refcount, then blocks on lock; other tasks doing
// the same inflate the refcount, causing split_folio() to fail
folio_get(folio);
folio_lock(folio);          // blocks while holding extra ref
err = split_folio(folio);   // fails: refcount too high

Safe ordering (lock before refcount):

// CORRECT: lock first so only one task proceeds; then raise refcount
if (!folio_trylock(folio))
    return -EAGAIN;         // no refcount raised, no livelock
folio_get(folio);
err = split_folio(folio);   // expected refcount matches

See madvise_free_huge_pmd() in mm/huge_memory.c for the correct lock-before-refcount pattern when splitting is needed.

REPORT as bugs: Code paths that call folio_get() then block on folio_lock() before calling split_folio() on large folios, especially in retry loops.

Large Folio i_size Boundary Checks

Mapping file-backed large folios without checking i_size breaks POSIX SIGBUS semantics: accesses within a VMA but beyond i_size rounded up to PAGE_SIZE must generate SIGBUS, but over-mapping a large folio silently serves zero-filled pages for those accesses instead.

Invariants:

• PTEs must not be installed for file pages beyond DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE). Code that computes the number of PTEs to install from folio_nr_pages() or folio_next_index() must clamp against i_size. See filemap_map_pages() in mm/filemap.c which clamps end_pgoff to file_end
• PMD mappings must not be installed when the folio extends past i_size. Both filemap_map_pages() in mm/filemap.c and finish_fault() in mm/memory.c check file_end >= folio_next_index(folio) before allowing PMD mapping via filemap_map_pmd() or do_set_pmd()
• On truncate, if a large folio spanning the new i_size cannot be split, the folio must be fully unmapped so it will be refaulted with PTEs (which respect i_size clamping). See try_folio_split_or_unmap() in mm/truncate.c, which calls try_to_unmap() on split failure

Exception -- shmem/tmpfs: shmem_mapping() returns true for shmem/tmpfs address spaces. These mappings are exempt from the i_size boundary check for PMD mapping and are allowed to map with PMDs across i_size. All three enforcement sites (filemap_map_pages(), finish_fault(), and try_folio_split_or_unmap()) check shmem_mapping() to preserve this behavior.

Large Folio Swapin and Swap Cache Conflicts

Large folio (mTHP) swapin without checking for existing smaller swap cache entries causes unbounded retry loops. swapcache_prepare() fails with -EEXIST when any slot in the range has SWAP_HAS_CACHE set, and callers retrying on -EEXIST loop forever because readahead or concurrent swapin can populate individual order-0 entries persistently.

Before mTHP swapin, use non_swapcache_batch(entry, nr_pages) in mm/swap.h to verify no slot is occupied; fall back to order-0 if result < nr_pages. See can_swapin_thp() in mm/memory.c and shmem_swap_alloc_folio() in mm/shmem.c. Any new large folio swapin path must include this check.

Hugetlb Folio Type Transition Races

Accessing hugetlb-specific folio metadata (such as calling folio_hstate()) after an unlocked folio_test_hugetlb() check causes a NULL pointer dereference when another CPU concurrently clears the hugetlb type and frees the folio.

__update_and_free_hugetlb_folio() in mm/hugetlb.c calls __folio_clear_hugetlb() under hugetlb_lock, which clears the hugetlb page type. After the type is cleared, folio_hstate() (in include/linux/hugetlb.h) calls size_to_hstate(folio_size()) which returns NULL because the folio size no longer matches any registered hstate.

// WRONG: TOCTOU race
if (folio_test_hugetlb(folio)) {
    h = folio_hstate(folio);  // May return NULL if type cleared concurrently
}

// CORRECT: Check and use under the same lock
spin_lock_irq(&hugetlb_lock);
if (folio_test_hugetlb(folio)) {
    h = folio_hstate(folio);
}
spin_unlock_irq(&hugetlb_lock);

An unlocked folio_test_hugetlb() is acceptable as a preliminary fast-path filter (to avoid taking the lock), but the result must not be trusted for subsequent hstate derivation without re-checking under the lock. Code outside mm/hugetlb.c that encounters hugetlb pages through lockless PFN iteration (such as has_unmovable_pages() in mm/page_isolation.c) must use size_to_hstate() with a NULL check instead of folio_hstate().

Hugetlb Fault Path Locking

Lock ordering: hugetlb_fault_mutex -> vma_lock -> i_mmap_rwsem -> folio_lock (see mm/rmap.c comment block). hugetlb_wp() drops the mutex and vma_lock mid-operation (to call unmap_ref_private() which needs vma_lock in write mode). Acquiring folio_lock before this drop inverts the ordering.

Do NOT use filemap_lock_folio() while holding hugetlb_fault_mutex in paths reaching hugetlb_wp(). Use folio_trylock() and bail on failure, waiting after releasing all locks (see need_wait_lock in hugetlb_fault() in mm/hugetlb.c). Prefer testing folio state without locking when possible (e.g., folio_test_anon() does not need the folio lock).

Hugetlb Pool Accounting

The hstate struct (include/linux/hugetlb.h) has four counters (all protected by hugetlb_lock): nr_huge_pages (total), free_huge_pages, surplus_huge_pages (beyond persistent pool), resv_huge_pages (reserved). Available = free_huge_pages - resv_huge_pages (see available_huge_pages()). Each has per-node variants except resv_huge_pages.

Key rules:

• alloc_hugetlb_folio() uses gbl_chg from vma_needs_reservation() to distinguish reserved vs unreserved allocation; only decrements resv_huge_pages when !gbl_chg
• Derived values (persistent_huge_pages() = nr_huge_pages - surplus_huge_pages) combine counters that must be updated in the same hugetlb_lock hold to avoid transient inconsistency
• remove_hugetlb_folio() / add_hugetlb_folio() take bool adjust_surplus; callers must check surplus_huge_pages_node[nid] and pass the result -- hardcoding false silently skips surplus adjustment. Error paths must use the same value as the forward path

Hugetlb PMD Page Table Sharing and Unsharing

When unsharing hugetlb PMD page tables, the freed page table page must go through tlb_remove_table() (not direct free_page()) to synchronize against GUP-fast, which traverses page tables locklessly under local_irq_disable. tlb_remove_table_sync_one() sends an IPI to ensure no concurrent GUP-fast before reuse. struct mmu_gather tracks unsharing via unshared_tables / fully_unshared_tables flags.

Locking: PMD sharing/unsharing requires i_mmap_rwsem in write mode. Fault paths calling huge_pmd_share() hold it for read and retry with write on failure. See huge_pmd_unshare() and huge_pmd_share() in mm/hugetlb.c.

Memory Failure Folio Handling

memory_failure() return values (mm/memory-failure.c): 0 = recovered (no signal needed), -EHWPOISON = already poisoned, -EOPNOTSUPP = filtered by hwpoison_filter(), other negative = recovery failed (process killed). Architecture MCE handlers (kill_me_maybe() in arch/x86/kernel/cpu/mce/core.c) branch on these. Internal helpers whose returns propagate through memory_failure() must return 0 for successful recovery, not -EFAULT.

Large folio hwpoison: PG_hwpoison is per-page (PF_ANY); PG_has_hwpoisoned is per-folio (PF_SECOND) as a fast indicator. Both must stay synchronized. When splitting a poisoned folio, PG_has_hwpoisoned must propagate to the correct sub-folio. Re-check folio_test_large() after acquiring folio lock (concurrent split possible).

HWPoison content access guard: accessing poisoned page content triggers an unrecoverable MCE/panic. Check PageHWPoison(page) per-subpage (to skip individual pages) or folio_contain_hwpoisoned_page(folio) for early-exit before any content access (pages_identical(), copy_page(), kmap_local_page() + read, etc.). REPORT as bugs: content reads in THP split, migration, KSM, or compaction without hwpoison checks.

Quick Checks

• Large folio mapcount field consistency: _large_mapcount, _entire_mapcount, per-page _mapcount, _nr_pages_mapped are updated non-atomically in rmap operations. Reading multiple fields requires folio_lock_large_mapcount() for a consistent snapshot. See __wp_can_reuse_large_anon_folio() in mm/memory.c: optimistic check, lock, recheck, act
• Large folio boundary spanning in PFN iteration: large folios crossing sub-range boundaries are seen in multiple sub-ranges. Without dedup, actions apply multiple times (double pageout, inflated stats). Track last-processed folio or advance by folio_size(). See last_applied in struct damos
• Large folio size preconditions on hwpoison paths: unmap_poisoned_folio() cannot handle large non-hugetlb folios. Callers must check folio_test_large() && !folio_test_hugetlb() and split before calling. memory_failure() handles this via try_to_split_thp_page() first
• Hugetlb page cache insertion protocol: before hugetlb_add_to_page_cache(): zero with folio_zero_user(), mark uptodate with __folio_mark_uptodate(), hold hugetlb_fault_mutex_table[hash]. Not enforced by the function itself. See hugetlbfs_fallocate() in fs/hugetlbfs/inode.c
• Hugetlb HPG flag propagation during demotion: init_new_hugetlb_folio() does NOT propagate HPG flags from folio->private. Allocation-origin flags like HPG_cma control the deallocation path (CMA vs buddy). Code creating new hugetlb folios from an existing one must explicitly copy these flags. See demote_free_hugetlb_folios() in mm/hugetlb.c
• Hugetlb folio rejection in generic folio paths: generic rmap/migration callbacks that only handle PTE-level folios must reject hugetlb early via folio_test_hugetlb(). Hugetlb requires distinct PTE ops, rmap functions, RSS accounting, and locking. Compare try_to_unmap_one()/ try_to_migrate_one() in mm/rmap.c (full hugetlb branches) against new callbacks
• Hugetlb subpool reservation rollback: hugepage_subpool_get_pages() absorbs some pages from rsv_hpages and returns smaller gbl_reserve. Error paths must return chg - gbl_reserve (not chg) to hugepage_subpool_put_pages(), and feed its return to hugetlb_acct_memory(). Over-crediting causes resv_huge_pages underflow
• Deferred split queue for large folios: when partially unmapping a large folio (unmapping some but not all subpages), the folio should be queued for deferred splitting via deferred_split_folio(). Missing this call wastes memory because the partially-mapped folio cannot be reclaimed until fully unmapped. See deferred_split_folio() in mm/huge_memory.c
• try_to_unmap() on PMD-mapped large folios: requires TTU_SPLIT_HUGE_PMD flag, otherwise VM_BUG_ON_FOLIO(!pvmw.pte, folio) fires in try_to_unmap_one(). Wrappers like unmap_poisoned_folio() do NOT set this flag — callers must guarantee not a large folio (except hugetlb)
• pmd_trans_huge() matches both THP and hugetlb PMDs: check is_vm_hugetlb_page(vma) first before THP-specific operations (split_huge_pmd(), pgtable_trans_huge_withdraw()). Hugetlb uses different page table layout, locking, and splitting semantics. Same ambiguity at PUD level
• Memory failure accounting consistency: action_result() updates num_poisoned_pages (global) and memory_failure_stats (per-node) together. Calling num_poisoned_pages_inc() without update_per_node_mf_stats() causes /proc/meminfo vs sysfs inconsistency