linux-toradex.git/mm/page_alloc.h, branch master Linux kernel for Apalis and Colibri modules Merge tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm 2026-10-27T22:23:17+00:00 Linus Torvalds torvalds@linux-foundation.org 2026-10-27T22:23:17+00:00 8a36f53fd5ec3eff01f1b637d42cbdc3821d0493 Pull more MM updates from Andrew Morton: - "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo Stoakes): Replace VM_SPECIAL with helpers describing mapping ownership and behavior. Enforce the mmap flag rules and convert affected drivers, removing ambiguous tests shared by driver mappings and ordinary memory. - "support kselftest on nommu platform" (Hajime Tazaki): Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner to avoid its fork-dependent timeout command and document the build and runtime requirements. - "mm/collapse: separate a collapse from its callers" (Kiryl Shutsemau): Separate huge-page collapse from caller policy. Replace khugepaged identity checks with explicit permissions, and separate scanning from collapse with the mmap_lock drop in one place. - "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo Stoakes): Provide a common helper for testing whether a VMA has anonymous reverse mappings. Document the locking rules, annotate intentional concurrent reads for KCSAN, and convert the core MM callers. - "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups" (David Hildenbrand): Remove SECTION_MARKED_PRESENT in favor of the early-section flag. Simplify sparse memory initialization, section metadata handling and related PFN validity checks. - "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport): Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have been unusable since the NO_BOOTMEM conversion, leaving only the code and data overhead. - "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun Song): Unify the generic device DAX and HugeTLB vmemmap population paths and shared tail-page handling. Remove the duplicate DAX machinery while retaining the powerpc radix implementation. - "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum Yun): Fix huge-page split tests racing with khugepaged recollapse. Use pagemap and page flags instead of VMA-boundary assumptions which fail when glibc has already enabled THP. - "memcg: move memcgid refcount to objcg to unpin dying memcgs" (Bingfang Guo): Let dying memcgs be freed while swapped-out shmem still holds their IDs. Move ID references to reparentable objcgs, with the necessary swap accounting and RCU lifetime changes. - "mm/truncate: fix data loss when truncating straddling large folios" (Zhang Yi): Preserve out-of-range data when truncating straddling large folios. Fix a lookup race and split-failure handling, and handle minimum folio orders for the forthcoming ext4 iomap buffered I/O path. - "mm/damon: introduce damos quota goal target metric complement flag" (SJ Park): Let DAMOS quota goals use a metric's complement, so utilization can also express free-memory goals without adding separate metrics. Add the sysfs interface, documentation and tests. - "arm64/mm: Standardize printing for pgtable entries" (Anshuman Khandual): Use the generic formatter for arm64 page-table entries, ready for future wider entries. Move the safe-PTE-update check out of the header to avoid an include cycle. - "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk): Fix KMSAN metadata leaks on iounmap and failed ioremap allocations. Free backing pages before removing their mappings, and share the teardown with partial-allocation cleanup. - "mm/damon/core: preserve quota state when constructing schemes" (Karl Mehltretter): Preserve running DAMOS quota state while validating parameter updates. Even rejected updates could reset accounting and exceed the quota window; add tests for both accepted and rejected updates. - "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu): Fix vmalloc_dump_obj() lookups for last-page addresses and allocations spanning vmap zones. These allocations were being missed in diagnostic output. - "mm: various (urgent?) fixes" (David Hildenbrand): Fix mmap charge leaks and hugetlb use-after-free, wrong-address unmapping, stuck locks and endless faults. Also preserve reservations on failed mremap and handle stale userfaultfd bits. - "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris Charisis): Fix pointer and metadata confusion during RCU destruction of maple_range_64 nodes, avoiding double frees and invalid accesses. Add a regression test; no in-tree user currently uses this combination. - "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu Li): Fix hugetlb migration and preallocation accounting so /proc/vmstat retains the correct count and memory.numa_stat follows the folio to its new node. - "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park): Fix 32-bit DAMON age overflow which made old regions look young. Share hugetlb folio acquisition code and add a regression test for the previously fixed last-region skipping bug. - "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ Park): Reduce kdamond stack usage and remove unused enums. Add deterministic PSI quota-goal tests which do not depend on the machine's current memory pressure. - "mm/damon: fix the temporal goal tuner's size quota conversion" (Donggeun Yoo): Bound the temporal tuner's quota conversion to prevent overflow and zero effective quotas, which could stop schemes making progress. Add boundary tests. - "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He): Remove redundant GFP_DMA flags from coherent DMA allocation calls. The DMA API already selects the zone from device limits and ignores the caller's flag. - "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum Yun): Propagate child-process failures in the khugepaged tests and prevent background collapse from racing their checks. This removes false passes and intermittent failures. - "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM" (Enze Li): Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds the right maintainers. Also label the closing hugetlb_cgroup.h include guard. Summary of the 65 singletons in this merge: The singletons include a few worthwhile performance fixes. Avoiding unnecessary reverse-map tree updates gave a reported 15.9% improvement in UnixBench's execl test on a 192-core x86 system. A reclaim retry fix stops defrag_mode from spinning under memory.low protection; a reported file-creation stress test ran 5.7 times faster, with far less memory pressure. Narrower hugetlb invalidations also spare KVM guests needless refaults: the reported Redis-in-VM P99 QPS penalty while punching holes fell from 7.09% to 1.45%. Several fixes prevent lost data or corrupted memory. Lazyfree huge-page races with mremap()/munmap() could lose data or access freed memory, and MADV_COLD could lose a concurrent write's dirty bit. Another fix prevents stale migration entries from outliving large folios. On filesystems with blocks smaller than a page, extending a file by 4 GiB or more could expose bytes written past the old EOF through mmap instead of zeroes. That is fixed, along with use-after-free races in PAGEMAP_SCAN during process exit and in memory-cgroup accounting on PREEMPT_RT kernels. Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios, which could wedge other memory operations, and stop mremap() leaking commit charges until allocations fail under strict overcommit. A further fix keeps userfaultfd tracking intact on neighboring mappings. NUMA binding no longer prevents long-term pins when the chosen node has only movable memory, fixing QEMU/VFIO startup and io_uring buffer registration failures. MGLRU also gets a fix for early-boot crashes with memory cgroups disabled at boot, notably in small crash kernels. * tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits) mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root mm/memcontrol: publish null pointer before dropping css reference test_maple_tree: test maple_range_64 node in RCU mode maple_tree: fix maple_range_64 crashes in RCU mode mm/vma: avoid broken uffd merge on mremap tools/testing/shared: catch a failed posix_memalign() mm/damon/core: remove unreachable damos charge_{target,addr}_from mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache riscv: mm: exclude invalid THP PMDs from page table check mm/damon/core: filter out damos address filter-split failed region fs/proc/task_mmu: pin mm_users in do_pagemap_scan() mm/mempolicy: apply bind policies against the scoped allocation zone mm/vmscan: reclaim every zone in per-node proactive reclaim mm/mremap: correctly uncharge accountable VMAs in error path mm/mremap: populate calculated fields in vrm state correctly MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP mm: bdi: reset min_ratio after the sysfs attributes are gone mm/page_vma_mapped: continue past nonmatching huge PMDs selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers ... 
Pull more MM updates from Andrew Morton:

 - "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
   Stoakes):

   Replace VM_SPECIAL with helpers describing mapping ownership and
   behavior. Enforce the mmap flag rules and convert affected drivers,
   removing ambiguous tests shared by driver mappings and ordinary
   memory.

 - "support kselftest on nommu platform" (Hajime Tazaki):

   Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
   to avoid its fork-dependent timeout command and document the build
   and runtime requirements.

 - "mm/collapse: separate a collapse from its callers" (Kiryl
   Shutsemau):

   Separate huge-page collapse from caller policy. Replace khugepaged
   identity checks with explicit permissions, and separate scanning from
   collapse with the mmap_lock drop in one place.

 - "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
   Stoakes):

   Provide a common helper for testing whether a VMA has anonymous
   reverse mappings. Document the locking rules, annotate intentional
   concurrent reads for KCSAN, and convert the core MM callers.

 - "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
   (David Hildenbrand):

   Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
   Simplify sparse memory initialization, section metadata handling and
   related PFN validity checks.

 - "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):

   Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
   been unusable since the NO_BOOTMEM conversion, leaving only the code
   and data overhead.

 - "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
   Song):

   Unify the generic device DAX and HugeTLB vmemmap population paths and
   shared tail-page handling. Remove the duplicate DAX machinery while
   retaining the powerpc radix implementation.

 - "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
   Yun):

   Fix huge-page split tests racing with khugepaged recollapse. Use
   pagemap and page flags instead of VMA-boundary assumptions which fail
   when glibc has already enabled THP.

 - "memcg: move memcgid refcount to objcg to unpin dying memcgs"
   (Bingfang Guo):

   Let dying memcgs be freed while swapped-out shmem still holds their
   IDs. Move ID references to reparentable objcgs, with the necessary
   swap accounting and RCU lifetime changes.

 - "mm/truncate: fix data loss when truncating straddling large folios"
   (Zhang Yi):

   Preserve out-of-range data when truncating straddling large folios.
   Fix a lookup race and split-failure handling, and handle minimum
   folio orders for the forthcoming ext4 iomap buffered I/O path.

 - "mm/damon: introduce damos quota goal target metric complement flag"
   (SJ Park):

   Let DAMOS quota goals use a metric's complement, so utilization can
   also express free-memory goals without adding separate metrics. Add
   the sysfs interface, documentation and tests.

 - "arm64/mm: Standardize printing for pgtable entries" (Anshuman
   Khandual):

   Use the generic formatter for arm64 page-table entries, ready for
   future wider entries. Move the safe-PTE-update check out of the
   header to avoid an include cycle.

 - "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):

   Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
   Free backing pages before removing their mappings, and share the
   teardown with partial-allocation cleanup.

 - "mm/damon/core: preserve quota state when constructing schemes" (Karl
   Mehltretter):

   Preserve running DAMOS quota state while validating parameter
   updates. Even rejected updates could reset accounting and exceed the
   quota window; add tests for both accepted and rejected updates.

 - "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):

   Fix vmalloc_dump_obj() lookups for last-page addresses and
   allocations spanning vmap zones. These allocations were being missed
   in diagnostic output.

 - "mm: various (urgent?) fixes" (David Hildenbrand):

   Fix mmap charge leaks and hugetlb use-after-free, wrong-address
   unmapping, stuck locks and endless faults. Also preserve reservations
   on failed mremap and handle stale userfaultfd bits.

 - "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
   Charisis):

   Fix pointer and metadata confusion during RCU destruction of
   maple_range_64 nodes, avoiding double frees and invalid accesses. Add
   a regression test; no in-tree user currently uses this combination.

 - "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
   Li):

   Fix hugetlb migration and preallocation accounting so /proc/vmstat
   retains the correct count and memory.numa_stat follows the folio to
   its new node.

 - "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):

   Fix 32-bit DAMON age overflow which made old regions look young.
   Share hugetlb folio acquisition code and add a regression test for
   the previously fixed last-region skipping bug.

 - "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
   Park):

   Reduce kdamond stack usage and remove unused enums. Add deterministic
   PSI quota-goal tests which do not depend on the machine's current
   memory pressure.

 - "mm/damon: fix the temporal goal tuner's size quota conversion"
   (Donggeun Yoo):

   Bound the temporal tuner's quota conversion to prevent overflow and
   zero effective quotas, which could stop schemes making progress. Add
   boundary tests.

 - "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):

   Remove redundant GFP_DMA flags from coherent DMA allocation calls.
   The DMA API already selects the zone from device limits and ignores
   the caller's flag.

 - "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
   Yun):

   Propagate child-process failures in the khugepaged tests and prevent
   background collapse from racing their checks. This removes false
   passes and intermittent failures.

 - "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
   (Enze Li):

   Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
   the right maintainers. Also label the closing hugetlb_cgroup.h
   include guard.

Summary of the 65 singletons in this merge:

  The singletons include a few worthwhile performance fixes. Avoiding
  unnecessary reverse-map tree updates gave a reported 15.9% improvement
  in UnixBench's execl test on a 192-core x86 system. A reclaim retry
  fix stops defrag_mode from spinning under memory.low protection; a
  reported file-creation stress test ran 5.7 times faster, with far less
  memory pressure. Narrower hugetlb invalidations also spare KVM guests
  needless refaults: the reported Redis-in-VM P99 QPS penalty while
  punching holes fell from 7.09% to 1.45%.

  Several fixes prevent lost data or corrupted memory. Lazyfree
  huge-page races with mremap()/munmap() could lose data or access freed
  memory, and MADV_COLD could lose a concurrent write's dirty bit.
  Another fix prevents stale migration entries from outliving large
  folios. On filesystems with blocks smaller than a page, extending a
  file by 4 GiB or more could expose bytes written past the old EOF
  through mmap instead of zeroes. That is fixed, along with
  use-after-free races in PAGEMAP_SCAN during process exit and in
  memory-cgroup accounting on PREEMPT_RT kernels.

  Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
  which could wedge other memory operations, and stop mremap() leaking
  commit charges until allocations fail under strict overcommit. A
  further fix keeps userfaultfd tracking intact on neighboring mappings.
  NUMA binding no longer prevents long-term pins when the chosen node
  has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
  registration failures. MGLRU also gets a fix for early-boot crashes
  with memory cgroups disabled at boot, notably in small crash kernels.

* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
  mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
  mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
  mm/memcontrol: publish null pointer before dropping css reference
  test_maple_tree: test maple_range_64 node in RCU mode
  maple_tree: fix maple_range_64 crashes in RCU mode
  mm/vma: avoid broken uffd merge on mremap
  tools/testing/shared: catch a failed posix_memalign()
  mm/damon/core: remove unreachable damos charge_{target,addr}_from
  mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
  riscv: mm: exclude invalid THP PMDs from page table check
  mm/damon/core: filter out damos address filter-split failed region
  fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
  mm/mempolicy: apply bind policies against the scoped allocation zone
  mm/vmscan: reclaim every zone in per-node proactive reclaim
  mm/mremap: correctly uncharge accountable VMAs in error path
  mm/mremap: populate calculated fields in vrm state correctly
  MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
  mm: bdi: reset min_ratio after the sysfs attributes are gone
  mm/page_vma_mapped: continue past nonmatching huge PMDs
  selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
  ...
mm: page_alloc: make defrag_mode retries follow the promoted order 2026-10-24T08:56:29+00:00 Kiryl Shutsemau (Meta) kas@kernel.org 2026-10-13T17:37:20+00:00 4cd78a2dcfbb13f049f841b98763896a3d1d7750 Since commit 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode"), direct reclaim and compaction for non-movable requests under defrag_mode run at pageblock_order, to produce the whole blocks that ALLOC_NOFRAGMENT needs. The retry decisions that follow still use the request order. An order-0 request can therefore retry indefinitely without ever reaching the ALLOC_NOFRAGMENT fallback: - Reclaim at pageblock_order gives up after one pass as soon as a zone looks compaction_ready(), and do_try_to_free_pages() then returns 1 even though nothing was reclaimed. It returns before the retry that would reclaim memory.low-protected cgroups, so when most memory is protected, the pass that did run finds next to nothing. - Compaction at pageblock_order fails or is deferred. - should_reclaim_retry() takes the reported progress as progress for the order-0 request and resets no_progress_loops. The request retries. Order 1-3 requests loop the same way, and should_compact_retry() also checks their pageblock_order compaction result against the request order. On a production host (64G, defrag_mode, memory.low covering most of the workload), 95% of direct reclaim runs were order-9 runs that returned 1 with nothing reclaimed, at up to 60k runs per second. Across ~200M should_reclaim_retry() calls in a day, no_progress_loops never left 0. The spinning allocations were SLUB slab refills for inode and dentry caches. The time spent registers as memory pressure, and pressure-based OOM killing takes down both workloads and system services. For promoted requests: - Reclaim progress does not reset no_progress_loops, as for costly orders. - should_compact_retry() checks the compaction result at the promoted order and does not retry COMPACT_SKIPPED, since the request can fall back. The compaction priority floor and the COMPACT_SUCCESS retry limit stay those of the request order, so a non-costly request still gets its COMPACT_PRIO_SYNC_FULL pass before it falls back. When a promoted request takes the fallback, reset the retry counters, so that the fallback attempt gets a full retry budget before the OOM killer is considered. A request whose reclaim was not promoted spent its budget on the request itself and keeps its counters, as Usama Arif pointed out. __alloc_pages_slowpath() computes the promoted order once per iteration and keeps it in alloc_context for direct reclaim, direct compaction and the two retry helpers, so no callee has to recompute it. Harry Yoo asked for the promoted order to be explicit rather than derived in each callee, and Vlastimil Babka suggested alloc_context over new function parameters. In a VM reproducer (32G, defrag_mode, inode churn under memory.low): before after should_reclaim_retry() calls 63M 293k peak memory pressure (PSI some avg10) 99% 12% File creation runs 5.7x faster. [kas@kernel.org: s/order_nofrag_promoted/order_defrag_promoted/, per hannes] Link: https://lore.kernel.org/atDL-nSFP2KVU7cI@thinkstation Link: https://lore.kernel.org/as5q14OsbK2CmOrh@thinkstation Fixes: 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode") Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Assisted-by: LLM Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: David Hildenbrand <david@kernel.org> Cc: Harry Yoo <harry@kernel.org> Cc: Usama Arif <usama.arif@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Zi Yan <ziy@nvidia.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: <stable@vger.kernel.org> 
Since commit 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm
in defrag_mode"), direct reclaim and compaction for non-movable requests
under defrag_mode run at pageblock_order, to produce the whole blocks that
ALLOC_NOFRAGMENT needs.  The retry decisions that follow still use the
request order.  An order-0 request can therefore retry indefinitely
without ever reaching the ALLOC_NOFRAGMENT fallback:

- Reclaim at pageblock_order gives up after one pass as soon as a zone
  looks compaction_ready(), and do_try_to_free_pages() then returns 1
  even though nothing was reclaimed. It returns before the retry that
  would reclaim memory.low-protected cgroups, so when most memory is
  protected, the pass that did run finds next to nothing.

- Compaction at pageblock_order fails or is deferred.

- should_reclaim_retry() takes the reported progress as progress for
  the order-0 request and resets no_progress_loops. The request
  retries.

Order 1-3 requests loop the same way, and should_compact_retry() also
checks their pageblock_order compaction result against the request order.

On a production host (64G, defrag_mode, memory.low covering most of the
workload), 95% of direct reclaim runs were order-9 runs that returned 1
with nothing reclaimed, at up to 60k runs per second.  Across ~200M
should_reclaim_retry() calls in a day, no_progress_loops never left 0. 
The spinning allocations were SLUB slab refills for inode and dentry
caches.  The time spent registers as memory pressure, and pressure-based
OOM killing takes down both workloads and system services.

For promoted requests:

- Reclaim progress does not reset no_progress_loops, as for costly
  orders.

- should_compact_retry() checks the compaction result at the promoted
  order and does not retry COMPACT_SKIPPED, since the request can fall
  back. The compaction priority floor and the COMPACT_SUCCESS retry
  limit stay those of the request order, so a non-costly request still
  gets its COMPACT_PRIO_SYNC_FULL pass before it falls back.

When a promoted request takes the fallback, reset the retry counters, so
that the fallback attempt gets a full retry budget before the OOM killer
is considered.  A request whose reclaim was not promoted spent its budget
on the request itself and keeps its counters, as Usama Arif pointed out.

__alloc_pages_slowpath() computes the promoted order once per iteration
and keeps it in alloc_context for direct reclaim, direct compaction and
the two retry helpers, so no callee has to recompute it.  Harry Yoo asked
for the promoted order to be explicit rather than derived in each callee,
and Vlastimil Babka suggested alloc_context over new function parameters.

In a VM reproducer (32G, defrag_mode, inode churn under memory.low):

                                        before    after
    should_reclaim_retry() calls           63M     293k
    peak memory pressure (PSI some avg10)  99%      12%

File creation runs 5.7x faster.

[kas@kernel.org: s/order_nofrag_promoted/order_defrag_promoted/, per hannes]
  Link: https://lore.kernel.org/atDL-nSFP2KVU7cI@thinkstation
Link: https://lore.kernel.org/as5q14OsbK2CmOrh@thinkstation
Fixes: 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode")
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Assisted-by: LLM
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
mm/memory_hotplug: optimize zone contiguous check when changing pfn range 2026-09-23T14:21:54+00:00 Yuan Liu yuan1.liu@intel.com 2026-09-20T08:49:46+00:00 554e941ffb5fa25964bd58edad32f3868f1b3ec5 When move_pfn_range_to_zone() or remove_pfn_range_from_zone() updates a zone, set_zone_contiguous() rescans the entire zone pageblock-by-pageblock to rebuild zone->contiguous. For large zones this is a significant cost during memory hotplug and hot-unplug. Add a new zone member, pages_with_online_memmap, that tracks the number of pages within the zone span that have an online memory map, including present pages and memory holes whose memory map has been initialized and for which pfn_to_online_page() succeeds. For early boot memory, pages_with_online_memmap is calculated in memmap_init_zone_range(). PFNs initialized by memmap_init_range() are included in pages_with_online_memmap, and hole PFNs for which pfn_to_online_page() succeeds are also counted in init_unavailable_range(). For hotplugged memory, pages_with_online_memmap is updated through adjust_present_page_count(), which is called during memory online and offline operations. When spanned_pages == pages_with_online_memmap, every PFN in the zone span has a valid memmap entry, so pfn_to_page() can be called for any PFN within the zone span without an additional pfn_valid() check. Note: this counter may temporarily undercount when pages with an online memory map exist outside the current zone span. Such pages are only created during boot, when initializing the memory map of pages that do not fall into any zone span. The undercount itself can only happen after boot, during memory hotplug, when growing the zone to cover such pages and later shrinking it back, which may result in a "too small" value. This is safe: it merely prevents detecting a contiguous zone. Here is an example (page numbers are just for illustration purposes): after boot: [ zone span ] [ zone pages ] spanned=10, initialized=15, online=10 online == spanned -> contiguous growing after hotplug (hotplug 5): [ zone span ] [ zone pages ] [ zone pages ] spanned=30, initialized=20, online=15 online != spanned -> not contiguous shrinking after hotunplug (hotunplug 5 again): [ zone span ] [ zone pages ] spanned=15, initialized=15, online=10 online != spanned -> not contiguous although contiguous The contiguity check using pages_with_online_memmap is stricter than the old pageblock-by-pageblock scan. The old set_zone_contiguous() iterated at pageblock granularity via pageblock_pfn_to_page(), so a zone could be marked contiguous even if a subsection-sized hole existed within a pageblock. The new check requires spanned_pages == pages_with_online_memmap, meaning every PFN in the zone span must satisfy pfn_to_online_page(). The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3]. +----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 3s | 70% | | +------+---------------+--------------+----------------+ | | 512G | 36s | 7s | 81% | +----------------+------+---------------+--------------+----------------+ +----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 11s | 4s | 64% | | +------+---------------+--------------+----------------+ | | 512G | 36s | 9s | 75% | +----------------+------+---------------+--------------+----------------+ [1] Qemu commands to hotplug 256G/512G memory for a VM: object_add memory-backend-ram,id=hotmem0,size=256G/512G,share=on device_add virtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory) [2] Hardware : Intel Icelake server Guest Kernel : v7.3-rc3 Qemu : v9.0.0 Launch VM : qemu-system-x86_64 -accel kvm -cpu host \ -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \ -drive file=./seed.img,format=raw,if=virtio \ -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \ -m 2G,slots=10,maxmem=2052472M \ -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \ -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \ -nographic -machine q35 \ -nic user,hostfwd=tcp::3000-:22 Guest kernel auto-onlines newly added memory blocks: echo online > /sys/devices/system/memory/auto_online_blocks [3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal /proc/meminfo' on Guest reflects that all hotplugged memory is recognized. Reported-by: Nanhai Zou <nanhai.zou@intel.com> Reported-by: Chen Zhang <zhangchen.kidd@jd.com> Tested-by: Yuan Liu <yuan1.liu@intel.com> Reviewed-by: Jason Zeng <jason.zeng@intel.com> Reviewed-by: Chen Yu <yu.c.chen@intel.com> Reviewed-by: Pan Deng <pan.deng@intel.com> Co-developed-by: Tianyou Li <tianyou.li@intel.com> Signed-off-by: Tianyou Li <tianyou.li@intel.com> Signed-off-by: Yuan Liu <yuan1.liu@intel.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Wei Yang <richard.weiyang@gmail.com> Link: https://patch.msgid.link/20260920084946.3266279-3-yuan1.liu@intel.com Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> 
When move_pfn_range_to_zone() or remove_pfn_range_from_zone() updates a
zone, set_zone_contiguous() rescans the entire zone pageblock-by-pageblock
to rebuild zone->contiguous. For large zones this is a significant cost
during memory hotplug and hot-unplug.

Add a new zone member, pages_with_online_memmap, that tracks the
number of pages within the zone span that have an online memory map,
including present pages and memory holes whose memory map has been
initialized and for which pfn_to_online_page() succeeds.

For early boot memory, pages_with_online_memmap is calculated in
memmap_init_zone_range(). PFNs initialized by memmap_init_range() are
included in pages_with_online_memmap, and hole PFNs for which
pfn_to_online_page() succeeds are also counted in
init_unavailable_range(). For hotplugged memory,
pages_with_online_memmap is updated through adjust_present_page_count(),
which is called during memory online and offline operations. When
spanned_pages == pages_with_online_memmap, every PFN in the zone span
has a valid memmap entry, so pfn_to_page() can be called for any PFN
within the zone span without an additional pfn_valid() check.

Note: this counter may temporarily undercount when pages with an
online memory map exist outside the current zone span. Such pages
are only created during boot, when initializing the memory map of
pages that do not fall into any zone span. The undercount itself
can only happen after boot, during memory hotplug, when growing
the zone to cover such pages and later shrinking it back, which
may result in a "too small" value. This is safe: it merely
prevents detecting a contiguous zone.

Here is an example (page numbers are just for illustration purposes):
  after boot:
      [  zone span  ]
      [   zone pages   ]
      spanned=10, initialized=15, online=10
      online == spanned  ->  contiguous

  growing after hotplug (hotplug 5):
      [  zone span                          ]
      [   zone pages   ]   [   zone pages   ]
      spanned=30, initialized=20, online=15
      online != spanned  ->  not contiguous

  shrinking after hotunplug (hotunplug 5 again):
      [  zone span     ]
      [   zone pages   ]
      spanned=15, initialized=15, online=10
      online != spanned  ->  not contiguous although contiguous

The contiguity check using pages_with_online_memmap is stricter than
the old pageblock-by-pageblock scan. The old set_zone_contiguous()
iterated at pageblock granularity via pageblock_pfn_to_page(), so a
zone could be marked contiguous even if a subsection-sized hole
existed within a pageblock. The new check requires
spanned_pages == pages_with_online_memmap, meaning every PFN in the
zone span must satisfy pfn_to_online_page().

The following test cases of memory hotplug for a VM [1], tested in the
environment [2], show that this optimization can significantly reduce the
memory hotplug time [3].

+----------------+------+---------------+--------------+----------------+
|                | Size | Time (before) | Time (after) | Time Reduction |
|                +------+---------------+--------------+----------------+
| Plug Memory    | 256G |      10s      |      3s      |       70%      |
|                +------+---------------+--------------+----------------+
|                | 512G |      36s      |      7s      |       81%      |
+----------------+------+---------------+--------------+----------------+

+----------------+------+---------------+--------------+----------------+
|                | Size | Time (before) | Time (after) | Time Reduction |
|                +------+---------------+--------------+----------------+
| Unplug Memory  | 256G |      11s      |      4s      |       64%      |
|                +------+---------------+--------------+----------------+
|                | 512G |      36s      |      9s      |       75%      |
+----------------+------+---------------+--------------+----------------+

[1] Qemu commands to hotplug 256G/512G memory for a VM:
    object_add memory-backend-ram,id=hotmem0,size=256G/512G,share=on
    device_add virtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1
    qom-set vmem1 requested-size 256G/512G (Plug Memory)
    qom-set vmem1 requested-size 0G (Unplug Memory)

[2] Hardware     : Intel Icelake server
    Guest Kernel : v7.3-rc3
    Qemu         : v9.0.0

    Launch VM    :
    qemu-system-x86_64 -accel kvm -cpu host \
    -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \
    -drive file=./seed.img,format=raw,if=virtio \
    -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \
    -m 2G,slots=10,maxmem=2052472M \
    -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \
    -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \
    -nographic -machine q35 \
    -nic user,hostfwd=tcp::3000-:22

    Guest kernel auto-onlines newly added memory blocks:
    echo online > /sys/devices/system/memory/auto_online_blocks

[3] The time from typing the QEMU commands in [1] to when the output of
    'grep MemTotal /proc/meminfo' on Guest reflects that all hotplugged
    memory is recognized.

Reported-by: Nanhai Zou <nanhai.zou@intel.com>
Reported-by: Chen Zhang <zhangchen.kidd@jd.com>
Tested-by: Yuan Liu <yuan1.liu@intel.com>
Reviewed-by: Jason Zeng <jason.zeng@intel.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Pan Deng <pan.deng@intel.com>
Co-developed-by: Tianyou Li <tianyou.li@intel.com>
Signed-off-by: Tianyou Li <tianyou.li@intel.com>
Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Link: https://patch.msgid.link/20260920084946.3266279-3-yuan1.liu@intel.com
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG 2026-07-31T02:40:44+00:00 Brendan Jackman jackmanb@google.com 2026-07-03T12:31:55+00:00 6372aac4d41fcdab6779b19e44544afed0a39cf0 Now that alloc_pages has an entrypoint that allows passing alloc_flags, we can take advantage of this to start removing GFP flags that are only used for mm-internal stuff. This requires also plumbing the alloc_flags into some more of the allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags arg to go along with its callees, and we now need to pass those flags deeper into the allocator so they can reach the alloc_tag code. While moving the flag definition into page_alloc.h, also update the comment per Hao's suggestion. No functional change intended. Link: https://lore.kernel.org/all/b4916118-3537-4e19-8bc8-1d103dd0d225@linux.dev/ Link: https://lore.kernel.org/20260703-alloc-trylock-v5-15-c87b714e19d3@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Tested-by: Hao Ge <hao.ge@linux.dev> Acked-by: Hao Ge <hao.ge@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> 
Now that alloc_pages has an entrypoint that allows passing alloc_flags, we
can take advantage of this to start removing GFP flags that are only used
for mm-internal stuff.

This requires also plumbing the alloc_flags into some more of the
allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags
arg to go along with its callees, and we now need to pass those flags
deeper into the allocator so they can reach the alloc_tag code.

While moving the flag definition into page_alloc.h, also update the
comment per Hao's suggestion.

No functional change intended.

Link: https://lore.kernel.org/all/b4916118-3537-4e19-8bc8-1d103dd0d225@linux.dev/
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-15-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: move __alloc_pages() to mm/page_alloc.h 2026-07-31T02:40:43+00:00 Brendan Jackman jackmanb@google.com 2026-07-03T12:31:54+00:00 a4f6c0f83d91ae093900eff55c4c4b0971206899 It's no longer used outside of mm/. Since this means __alloc_pages_noprof() is no longer visible from gfp.h, this also means moving the definition of alloc_pages_node_noprof into the .c file. Also remove references to this API from the documentation tree - referring to the specific function name was already questionable but now the function is not even public it definitely seems wrong. Link: https://lore.kernel.org/20260703-alloc-trylock-v5-14-c87b714e19d3@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> 
It's no longer used outside of mm/.

Since this means __alloc_pages_noprof() is no longer visible from gfp.h,
this also means moving the definition of alloc_pages_node_noprof into
the .c file.

Also remove references to this API from the documentation tree -
referring to the specific function name was already questionable but
now the function is not even public it definitely seems wrong.

Link: https://lore.kernel.org/20260703-alloc-trylock-v5-14-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: move some stuff to mm/page_alloc.h 2026-07-31T02:40:41+00:00 Brendan Jackman jackmanb@google.com 2026-07-03T12:31:47+00:00 6f414dbf21782ad8e906b391a429ddd21330e011 Some of this stuff in the public header is only used internally so shrink the scope to avoid silently growing new users. drain_local_pages() is still used from kernel/power/snapshot.c so that needs to stay behind. Link: https://lore.kernel.org/20260703-alloc-trylock-v5-7-c87b714e19d3@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> 
Some of this stuff in the public header is only used internally so shrink
the scope to avoid silently growing new users.

drain_local_pages() is still used from kernel/power/snapshot.c so that
needs to stay behind.

Link: https://lore.kernel.org/20260703-alloc-trylock-v5-7-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/page_alloc: unify __alloc_frozen_pages[_nolock]_noprof() 2026-07-31T02:40:41+00:00 Brendan Jackman jackmanb@google.com 2026-07-03T12:31:45+00:00 2fc4c1d51d878bcd615774d19a97dc29ac0ce970 Currently the core allocator code is controlled by ALLOC_NOLOCK, but the main entry point function is significantly different from the normal __alloc_frozen_pages_nolock(), this is tiring when reading the code. Plumb the ALLOC_NOLOCK control one layer up in the call stack: create an alloc_flags argument to __alloc_frozen_pages_nolock() (which is only exposed to mm/) and then turn the nolock variant into a thin wrapper that just sets that flag (as well as handling NUMA_NO_NODE, similar to how some of the wrappers in gfp.h do). For consistency, set ALLOC_WMARK_MIN explicitly in fastpath_alloc_flags for the new ALLOC_NOLOCK path. This was already "done" silently in __alloc_frozen_pages_nolock_noprof(): ALLOC_WMARK_MIN is 0. Rationale that this doesn't change anything: 1. Simple bits: A bunch of the nolock-specific handling is just moved to the new alloc_order_allowed(), alloc_nolock_allowed() and gfp_nolock. 2. __alloc_frozen_pages_noprof() has some extra logic that wasn't previously in the nolock variant: a. Application of gfp_allowed_mask; this only affects early boot, only flags that affect the slowpath get changed here, and the nolock allocation path isn't allowed to the GFP_BOOT_MASK flags. b. Application of current_gfp_context() - also only affects the slowpath 3. The slowpath itself: this is now just explicitly skipped under !ALLOC_NOLOCK. Ulterior motive: adding an alloc_flags arg to the allocator's mm-internal entrypoint can later be used to do more allocation customisation without needing to create new GFP flags. No functional change intended. Link: https://lore.kernel.org/20260703-alloc-trylock-v5-5-c87b714e19d3@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> 
Currently the core allocator code is controlled by ALLOC_NOLOCK, but the
main entry point function is significantly different from the normal
__alloc_frozen_pages_nolock(), this is tiring when reading the code.

Plumb the ALLOC_NOLOCK control one layer up in the call stack: create an
alloc_flags argument to __alloc_frozen_pages_nolock() (which is only
exposed to mm/) and then turn the nolock variant into a thin wrapper that
just sets that flag (as well as handling NUMA_NO_NODE, similar to how some
of the wrappers in gfp.h do).

For consistency, set ALLOC_WMARK_MIN explicitly in fastpath_alloc_flags
for the new ALLOC_NOLOCK path.  This was already "done" silently in
__alloc_frozen_pages_nolock_noprof(): ALLOC_WMARK_MIN is 0.

Rationale that this doesn't change anything:

1. Simple bits: A bunch of the nolock-specific handling is just moved to
   the new alloc_order_allowed(), alloc_nolock_allowed() and
   gfp_nolock.

2. __alloc_frozen_pages_noprof() has some extra logic that wasn't
   previously in the nolock variant:

   a. Application of gfp_allowed_mask; this only affects early boot,
      only flags that affect the slowpath get changed here, and the
      nolock allocation path isn't allowed to the GFP_BOOT_MASK flags.

   b. Application of current_gfp_context() - also only affects the
      slowpath

3. The slowpath itself: this is now just explicitly skipped under
   !ALLOC_NOLOCK.

Ulterior motive: adding an alloc_flags arg to the allocator's mm-internal
entrypoint can later be used to do more allocation customisation without
needing to create new GFP flags.

No functional change intended.

Link: https://lore.kernel.org/20260703-alloc-trylock-v5-5-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: split out internal page_alloc.h 2026-07-31T02:40:41+00:00 Brendan Jackman jackmanb@google.com 2026-07-03T12:31:44+00:00 d0850de699b0cb7b02225747ae364f68f0bf86e1 internal.h is a bit bloated, seems like time for a page_alloc.h. Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in mm/page_alloc.c?" Only need to include it from ~20 .c files out of ~150 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges! Link: https://lore.kernel.org/20260703-alloc-trylock-v5-4-c87b714e19d3@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Suggested-by: "David Hildenbrand (Arm)" <david@kernel.org> Link: https://lore.kernel.org/all/41e92bab-6882-401a-8de9-154adbdcfb36@kernel.org/ Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> 
internal.h is a bit bloated, seems like time for a page_alloc.h.

Where it wasn't obvious, the heuristic for deciding what goes into this
new header was "does it support/correspond to a definition in
mm/page_alloc.c?"

Only need to include it from ~20 .c files out of ~150 so this does seem
like a genuine reduction in scopes, which is nice. And there's no
circular internal.h<->page_alloc.h dependency, so it seems worthwhile to
split this up before that inevitably emerges!

Link: https://lore.kernel.org/20260703-alloc-trylock-v5-4-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Suggested-by: "David Hildenbrand (Arm)" <david@kernel.org>
Link: https://lore.kernel.org/all/41e92bab-6882-401a-8de9-154adbdcfb36@kernel.org/
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>