linux-toradex.git/include/linux/pgtable.h, branch v7.0-rc6

mm: rmap: support batched checks of the references for large folios

2026-02-12T23:43:00+00:00

Patch series "support batch checking of references and unmapping for large
folios", v6.


Currently, folio_referenced_one() always checks the young flag for each
PTE sequentially, which is inefficient for large folios.  This
inefficiency is especially noticeable when reclaiming clean file-backed
large folios, where folio_referenced() is observed as a significant
performance hotspot.

Moreover, on Arm architecture, which supports contiguous PTEs, there is
already an optimization to clear the young flags for PTEs within a
contiguous range.  However, this is not sufficient.  We can extend this to
perform batched operations for the entire large folio (which might exceed
the contiguous range: CONT_PTE_SIZE).

Similar to folio_referenced_one(), we can also apply batched unmapping for
large file folios to optimize the performance of file folio reclamation. 
By supporting batched checking of the young flags, flushing TLB entries,
and unmapping, I can observed a significant performance improvements in my
performance tests for file folios reclamation.  Please check the
performance data in the commit message of each patch.


This patch (of 5):

Currently, folio_referenced_one() always checks the young flag for each
PTE sequentially, which is inefficient for large folios.  This
inefficiency is especially noticeable when reclaiming clean file-backed
large folios, where folio_referenced() is observed as a significant
performance hotspot.

Moreover, on Arm64 architecture, which supports contiguous PTEs, there is
already an optimization to clear the young flags for PTEs within a
contiguous range.  However, this is not sufficient.  We can extend this to
perform batched operations for the entire large folio (which might exceed
the contiguous range: CONT_PTE_SIZE).

Introduce a new API: clear_flush_young_ptes() to facilitate batched
checking of the young flags and flushing TLB entries, thereby improving
performance during large folio reclamation.  And it will be overridden by
the architecture that implements a more efficient batch operation in the
following patches.

While we are at it, rename ptep_clear_flush_young_notify() to
clear_flush_young_ptes_notify() to indicate that this is a batch
operation.

Link: https://lkml.kernel.org/r/cover.1770645603.git.baolin.wang@linux.alibaba.com
Link: https://lkml.kernel.org/r/12132694536834262062d1fb304f8f8a064b6750.1770645603.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang 
Reviewed-by: Harry Yoo 
Reviewed-by: Ryan Roberts 
Acked-by: David Hildenbrand (Arm) 
Cc: Catalin Marinas 
Cc: Jann Horn 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Rik van Riel 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Cc: Barry Song 
Signed-off-by: Andrew Morton

mm: relocate the page table ceiling and floor definitions

2026-02-12T23:42:53+00:00

Patch series " Remove XA_ZERO from error recovery of dup_mmap()", v3.

It is possible that the dup_mmap() call fails on allocating or setting up
a vma after the maple tree of the oldmm is copied.  Today, that failure
point is marked by inserting an XA_ZERO entry over the failure point so
that the exact location does not need to be communicated through to
exit_mmap().

However, a race exists in the tear down process because the dup_mmap()
drops the mmap lock before exit_mmap() can remove the partially set up vma
tree.  This means that other tasks may get to the mm tree and find the
invalid vma pointer (since it's an XA_ZERO entry), even though the mm is
marked as MMF_OOM_SKIP and MMF_UNSTABLE.

To remove the race fully, the tree must be cleaned up before dropping the
lock.  This is accomplished by extracting the vma cleanup in exit_mmap()
and changing the required functions to pass through the vma search limit. 
Any other tree modifications would require extra cycles which should be
spent on freeing memory.

This does run the risk of increasing the possibility of finding no vmas
(which is already possible!) in code that isn't careful.

The final four patches are to address the excessive argument lists being
passed between the functions.  Using the struct unmap_desc also allows
some special-case code to be removed in favour of the struct setup
differences.


This patch (of 11):

pgtables.h defines a fallback for ceiling and floor of the page tables
within the CONFIG_MMU section.  Moving the definitions to outside the
CONFIG_MMU allows for using them in generic code.

[akpm@linux-foundation.org: remove stray newline, per SeongJae]
Link: https://lkml.kernel.org/r/20260121164946.2093480-1-Liam.Howlett@oracle.com
Link: https://lkml.kernel.org/r/20260121164946.2093480-2-Liam.Howlett@oracle.com
Signed-off-by: Liam R. Howlett 
Suggested-by: Lorenzo Stoakes 
Suggested-by: SeongJae Park 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Chris Li 
Cc: David Hildenbrand 
Cc: Jann Horn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nhat Pham 
Cc: Pedro Falcato 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton

mm/page_table_check: reinstate address parameter in [__]page_table_check_pte_clear()

2026-01-27T04:02:35+00:00

This reverts commit aa232204c468 ("mm/page_table_check: remove unused
parameter in [__]page_table_check_pte_clear").

Reinstate previously unused parameters for the purpose of supporting
powerpc platforms, as many do not encode user/kernel ownership of the page
in the pte, but instead in the address of the access.

[ajd@linux.ibm.com: rebase, fix additional occurrence and loop handling]
Link: https://lkml.kernel.org/r/20251219-pgtable_check_v18rebase-v18-8-755bc151a50b@linux.ibm.com
Signed-off-by: Rohan McLure 
Signed-off-by: Andrew Donnellan 
Reviewed-by: Pasha Tatashin 
Acked-by: Ingo Molnar   # x86
Acked-by: Alexandre Ghiti  # riscv
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alistair Popple 
Cc: Christophe Leroy 
Cc: "Christophe Leroy (CS GROUP)" 
Cc: David Hildenbrand 
Cc: Donet Tom 
Cc: Guo Weikang 
Cc: Jason Gunthorpe 
Cc: Kevin Brodsky 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: Nicholas Miehlbradt 
Cc: Nicholas Piggin 
Cc: Paul Mackerras 
Cc: Qi Zheng 
Cc: "Ritesh Harjani (IBM)" 
Cc: Sweet Tea Dorminy 
Cc: Thomas Huth 
Cc: "Vishal Moola (Oracle)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/page_table_check: reinstate address parameter in [__]page_table_check_pmd_clear()

2026-01-27T04:02:35+00:00

This reverts commit 1831414cd729 ("mm/page_table_check: remove unused
parameter in [__]page_table_check_pmd_clear").

Reinstate previously unused parameters for the purpose of supporting
powerpc platforms, as many do not encode user/kernel ownership of the page
in the pte, but instead in the address of the access.

[ajd@linux.ibm.com: rebase on arm64 changes]
Link: https://lkml.kernel.org/r/20251219-pgtable_check_v18rebase-v18-7-755bc151a50b@linux.ibm.com
Signed-off-by: Rohan McLure 
Signed-off-by: Andrew Donnellan 
Reviewed-by: Pasha Tatashin 
Acked-by: Ingo Molnar   # x86
Acked-by: Alexandre Ghiti  # riscv
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alistair Popple 
Cc: Christophe Leroy 
Cc: "Christophe Leroy (CS GROUP)" 
Cc: David Hildenbrand 
Cc: Donet Tom 
Cc: Guo Weikang 
Cc: Jason Gunthorpe 
Cc: Kevin Brodsky 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: Nicholas Miehlbradt 
Cc: Nicholas Piggin 
Cc: Paul Mackerras 
Cc: Qi Zheng 
Cc: "Ritesh Harjani (IBM)" 
Cc: Sweet Tea Dorminy 
Cc: Thomas Huth 
Cc: "Vishal Moola (Oracle)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/page_table_check: reinstate address parameter in [__]page_table_check_pud_clear()

2026-01-27T04:02:34+00:00

This reverts commit 931c38e16499 ("mm/page_table_check: remove unused
parameter in [__]page_table_check_pud_clear").

Reinstate previously unused parameters for the purpose of supporting
powerpc platforms, as many do not encode user/kernel ownership of the page
in the pte, but instead in the address of the access.

[ajd@linux.ibm.com: rebase on arm64 changes]
Link: https://lkml.kernel.org/r/20251219-pgtable_check_v18rebase-v18-6-755bc151a50b@linux.ibm.com
Signed-off-by: Rohan McLure 
Signed-off-by: Andrew Donnellan 
Reviewed-by: Pasha Tatashin 
Acked-by: Ingo Molnar   # x86
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alexandre Ghiti 
Cc: Alistair Popple 
Cc: Christophe Leroy 
Cc: "Christophe Leroy (CS GROUP)" 
Cc: David Hildenbrand 
Cc: Donet Tom 
Cc: Guo Weikang 
Cc: Jason Gunthorpe 
Cc: Kevin Brodsky 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: Nicholas Miehlbradt 
Cc: Nicholas Piggin 
Cc: Paul Mackerras 
Cc: Qi Zheng 
Cc: "Ritesh Harjani (IBM)" 
Cc: Sweet Tea Dorminy 
Cc: Thomas Huth 
Cc: "Vishal Moola (Oracle)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm/page_table_check: provide addr parameter to page_table_check_ptes_set()

2026-01-27T04:02:34+00:00

To provide support for powerpc platforms, provide an addr parameter to the
__page_table_check_ptes_set() and page_table_check_ptes_set() routines. 
This parameter is needed on some powerpc platforms which do not encode
whether a mapping is for user or kernel in the pte.  On such platforms,
this can be inferred from the addr parameter.

[ajd@linux.ibm.com: rebase on arm64 + riscv changes, update commit message]
Link: https://lkml.kernel.org/r/20251219-pgtable_check_v18rebase-v18-5-755bc151a50b@linux.ibm.com
Signed-off-by: Rohan McLure 
Reviewed-by: Pasha Tatashin 
Acked-by: Alexandre Ghiti  # riscv
Signed-off-by: Andrew Donnellan 
Cc: Alexander Gordeev 
Cc: Alexandre Ghiti 
Cc: Alistair Popple 
Cc: Christophe Leroy 
Cc: "Christophe Leroy (CS GROUP)" 
Cc: David Hildenbrand 
Cc: Donet Tom 
Cc: Guo Weikang 
Cc: Ingo Molnar 
Cc: Jason Gunthorpe 
Cc: Kevin Brodsky 
Cc: Madhavan Srinivasan 
Cc: Magnus Lindholm 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Michael Ellerman 
Cc: Nicholas Miehlbradt 
Cc: Nicholas Piggin 
Cc: Paul Mackerras 
Cc: Qi Zheng 
Cc: "Ritesh Harjani (IBM)" 
Cc: Sweet Tea Dorminy 
Cc: Thomas Huth 
Cc: "Vishal Moola (Oracle)" 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: enable lazy_mmu sections to nest

2026-01-21T03:24:34+00:00

Despite recent efforts to prevent lazy_mmu sections from nesting, it
remains difficult to ensure that it never occurs - and in fact it does
occur on arm64 in certain situations (CONFIG_DEBUG_PAGEALLOC).  Commit
1ef3095b1405 ("arm64/mm: Permit lazy_mmu_mode to be nested") made nesting
tolerable on arm64, but without truly supporting it: the inner call to
leave() disables the batching optimisation before the outer section ends.

This patch actually enables lazy_mmu sections to nest by tracking the
nesting level in task_struct, in a similar fashion to e.g. 
pagefault_{enable,disable}().  This is fully handled by the generic
lazy_mmu helpers that were recently introduced.

lazy_mmu sections were not initially intended to nest, so we need to
clarify the semantics w.r.t.  the arch_*_lazy_mmu_mode() callbacks.  This
patch takes the following approach:

* The outermost calls to lazy_mmu_mode_{enable,disable}() trigger
  calls to arch_{enter,leave}_lazy_mmu_mode() - this is unchanged.

* Nested calls to lazy_mmu_mode_{enable,disable}() are not forwarded
  to the arch via arch_{enter,leave} - lazy MMU remains enabled so
  the assumption is that these callbacks are not relevant. However,
  existing code may rely on a call to disable() to flush any batched
  state, regardless of nesting. arch_flush_lazy_mmu_mode() is
  therefore called in that situation.

A separate interface was recently introduced to temporarily pause the lazy
MMU mode: lazy_mmu_mode_{pause,resume}().  pause() fully exits the mode
*regardless of the nesting level*, and resume() restores the mode at the
same nesting level.

pause()/resume() are themselves allowed to nest, so we actually store two
nesting levels in task_struct: enable_count and pause_count.  A new helper
is_lazy_mmu_mode_active() is introduced to determine whether we are
currently in lazy MMU mode; this will be used in subsequent patches to
replace the various ways arch's currently track whether the mode is
enabled.

In summary (enable/pause represent the values *after* the call):

lazy_mmu_mode_enable()		-> arch_enter()	    enable=1 pause=0
    lazy_mmu_mode_enable()	-> ø		    enable=2 pause=0
	lazy_mmu_mode_pause()	-> arch_leave()     enable=2 pause=1
	lazy_mmu_mode_resume()	-> arch_enter()     enable=2 pause=0
    lazy_mmu_mode_disable()	-> arch_flush()     enable=1 pause=0
lazy_mmu_mode_disable()		-> arch_leave()     enable=0 pause=0

Note: is_lazy_mmu_mode_active() is added to  to allow
arch headers included by  to use it.

Link: https://lkml.kernel.org/r/20251215150323.2218608-10-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: David Hildenbrand (Red Hat) 
Reviewed-by: Yeoreum Yun 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Anshuman Khandual 
Cc: Borislav Betkov 
Cc: Boris Ostrovsky 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: David Hildenbrand 
Cc: David S. Miller 
Cc: David Woodhouse 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Juegren Gross 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Ritesh Harjani (IBM) 
Cc: Ryan Roberts 
Cc: Suren Baghdasaryan 
Cc: Thomas Gleinxer 
Cc: Venkat Rao Bagalkote 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: bail out of lazy_mmu_mode_* in interrupt context

2026-01-21T03:24:34+00:00

The lazy MMU mode cannot be used in interrupt context.  This is documented
in , but isn't consistently handled across architectures.

arm64 ensures that calls to lazy_mmu_mode_* have no effect in interrupt
context, because such calls do occur in certain configurations - see
commit b81c688426a9 ("arm64/mm: Disable barrier batching in interrupt
contexts").  Other architectures do not check this situation, most likely
because it hasn't occurred so far.

Let's handle this in the new generic lazy_mmu layer, in the same fashion
as arm64: bail out of lazy_mmu_mode_* if in_interrupt().  Also remove the
arm64 handling that is now redundant.

Both arm64 and x86/Xen also ensure that any lazy MMU optimisation is
disabled while in interrupt (see queue_pte_barriers() and
xen_get_lazy_mode() respectively).  This will be handled in the generic
layer in a subsequent patch.

Link: https://lkml.kernel.org/r/20251215150323.2218608-9-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: David Hildenbrand (Red Hat) 
Reviewed-by: Anshuman Khandual 
Reviewed-by: Yeoreum Yun 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Borislav Betkov 
Cc: Boris Ostrovsky 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: David Hildenbrand 
Cc: David S. Miller 
Cc: David Woodhouse 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Juegren Gross 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Ritesh Harjani (IBM) 
Cc: Ryan Roberts 
Cc: Suren Baghdasaryan 
Cc: Thomas Gleinxer 
Cc: Venkat Rao Bagalkote 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: introduce generic lazy_mmu helpers

2026-01-21T03:24:33+00:00

The implementation of the lazy MMU mode is currently entirely
arch-specific; core code directly calls arch helpers:
arch_{enter,leave}_lazy_mmu_mode().

We are about to introduce support for nested lazy MMU sections.  As things
stand we'd have to duplicate that logic in every arch implementing
lazy_mmu - adding to a fair amount of logic already duplicated across
lazy_mmu implementations.

This patch therefore introduces a new generic layer that calls the
existing arch_* helpers. Two pair of calls are introduced:

* lazy_mmu_mode_enable() ... lazy_mmu_mode_disable()
    This is the standard case where the mode is enabled for a given
    block of code by surrounding it with enable() and disable()
    calls.

* lazy_mmu_mode_pause() ... lazy_mmu_mode_resume()
    This is for situations where the mode is temporarily disabled
    by first calling pause() and then resume() (e.g. to prevent any
    batching from occurring in a critical section).

The documentation in  will be updated in a subsequent
patch.

No functional change should be introduced at this stage.  The
implementation of enable()/resume() and disable()/pause() is currently
identical, but nesting support will change that.

Most of the call sites have been updated using the following Coccinelle
script:

@@
@@
{
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_enable();
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_disable();
...
}

@@
@@
{
...
- arch_leave_lazy_mmu_mode();
+ lazy_mmu_mode_pause();
...
- arch_enter_lazy_mmu_mode();
+ lazy_mmu_mode_resume();
...
}

A couple of notes regarding x86:

* Xen is currently the only case where explicit handling is required
  for lazy MMU when context-switching. This is purely an
  implementation detail and using the generic lazy_mmu_mode_*
  functions would cause trouble when nesting support is introduced,
  because the generic functions must be called from the current task.
  For that reason we still use arch_leave() and arch_enter() there.

* x86 calls arch_flush_lazy_mmu_mode() unconditionally in a few
  places, but only defines it if PARAVIRT_XXL is selected, and we
  are removing the fallback in . Add a new fallback
  definition to  to keep things building.

Link: https://lkml.kernel.org/r/20251215150323.2218608-8-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: David Hildenbrand 
Reviewed-by: Anshuman Khandual 
Reviewed-by: Yeoreum Yun 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Borislav Betkov 
Cc: Boris Ostrovsky 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: David Hildenbrand (Red Hat) 
Cc: David S. Miller 
Cc: David Woodhouse 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Juegren Gross 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Ritesh Harjani (IBM) 
Cc: Ryan Roberts 
Cc: Suren Baghdasaryan 
Cc: Thomas Gleinxer 
Cc: Venkat Rao Bagalkote 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: introduce CONFIG_ARCH_HAS_LAZY_MMU_MODE

2026-01-21T03:24:33+00:00

Architectures currently opt in for implementing lazy_mmu helpers by
defining __HAVE_ARCH_ENTER_LAZY_MMU_MODE.

In preparation for introducing a generic lazy_mmu layer that will require
storage in task_struct, let's switch to a cleaner approach: instead of
defining a macro, select a CONFIG option.

This patch introduces CONFIG_ARCH_HAS_LAZY_MMU_MODE and has each arch
select it when it implements lazy_mmu helpers. 
__HAVE_ARCH_ENTER_LAZY_MMU_MODE is removed and  relies on
the new CONFIG instead.

On x86, lazy_mmu helpers are only implemented if PARAVIRT_XXL is selected.
This creates some complications in arch/x86/boot/, because a few files
manually undefine PARAVIRT* options.  As a result  does
not define the lazy_mmu helpers, but this breaks the build as
 only defines them if !CONFIG_ARCH_HAS_LAZY_MMU_MODE. 
There does not seem to be a clean way out of this - let's just undefine
that new CONFIG too.

Link: https://lkml.kernel.org/r/20251215150323.2218608-7-kevin.brodsky@arm.com
Signed-off-by: Kevin Brodsky 
Acked-by: David Hildenbrand 
Reviewed-by: Ritesh Harjani (IBM) 
Reviewed-by: Ryan Roberts 
Reviewed-by: Yeoreum Yun 
Acked-by: Andreas Larsson 	[sparc]
Cc: Alexander Gordeev 
Cc: Anshuman Khandual 
Cc: Borislav Betkov 
Cc: Boris Ostrovsky 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: David Hildenbrand (Red Hat) 
Cc: David S. Miller 
Cc: David Woodhouse 
Cc: "H. Peter Anvin" 
Cc: Ingo Molnar 
Cc: Jann Horn 
Cc: Juegren Gross 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Madhavan Srinivasan 
Cc: Michael Ellerman 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Nicholas Piggin 
Cc: Peter Zijlstra 
Cc: Suren Baghdasaryan 
Cc: Thomas Gleinxer 
Cc: Venkat Rao Bagalkote 
Cc: Vlastimil Babka 
Cc: Will Deacon 
Signed-off-by: Andrew Morton