linux-toradex.git/mm/migrate.c, branch v6.19-rc2

memcg: remove __mod_lruvec_state

2025-11-24T23:08:54+00:00

__mod_lruvec_state() is already safe against irqs, so there is no need to
have a separate interface (i.e.  mod_lruvec_state) which wraps calls to it
with irq disabling and reenabling.  Let's rename __mod_lruvec_state() to
mod_lruvec_state().

Link: https://lkml.kernel.org/r/20251110232008.1352063-4-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt 
Reviewed-by: Harry Yoo 
Acked-by: Roman Gushchin 
Acked-by: Vlastimil Babka 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Muchun Song 
Cc: Qi Zheng 
Signed-off-by: Andrew Morton

mm: replace remaining pte_to_swp_entry() with softleaf_from_pte()

2025-11-24T23:08:52+00:00

There are straggler invocations of pte_to_swp_entry() lying around,
replace all of these with the software leaf entry equivalent -
softleaf_from_pte().

With those removed, eliminate pte_to_swp_entry() altogether.

No functional change intended.

Link: https://lkml.kernel.org/r/d8ee5ccefe4c42d7c4fe1a2e46f285ac40421cd3.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vlastimil Babka 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: eliminate further swapops predicates

2025-11-24T23:08:52+00:00

Having converted so much of the code base to software leaf entries, we can
mop up some remaining cases.

We replace is_pfn_swap_entry(), pfn_swap_entry_to_page(),
is_writable_device_private_entry(), is_device_exclusive_entry(),
is_migration_entry(), is_writable_migration_entry(),
is_readable_migration_entry(), swp_offset_pfn() and pfn_swap_entry_folio()
with softleaf equivalents.

No functional change intended.

Link: https://lkml.kernel.org/r/956bc9c031604811c0070d2f4bf2f1373f230213.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vlastimil Babka 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: remove is_hugetlb_entry_[migration, hwpoisoned]()

2025-11-24T23:08:51+00:00

We do not need to have explicit helper functions for these, it adds a
level of confusion and indirection when we can simply use software leaf
entry logic here instead and spell out the special huge_pte_none() case we
must consider.

No functional change intended.

Link: https://lkml.kernel.org/r/0e92d6924d3de88cd014ce1c53e20edc08fc152e.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Vlastimil Babka 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: replace pmd_to_swp_entry() with softleaf_from_pmd()

2025-11-24T23:08:51+00:00

Introduce softleaf_from_pmd() to do the equivalent operation for PMDs that
softleaf_from_pte() fulfils, and cascade changes through code base
accordingly, introducing helpers as necessary.

We are then able to eliminate pmd_to_swp_entry(),
is_pmd_migration_entry(), is_pmd_device_private_entry() and
is_pmd_non_present_folio_entry().

This further establishes the use of leaf operations throughout the code
base and further establishes the foundations for eliminating
is_swap_pmd().

No functional change intended.

[lorenzo.stoakes@oracle.com: check writable, not readable/writable, per Vlastimil]
  Link: https://lkml.kernel.org/r/cd97b6ec-00f9-45a4-9ae0-8f009c212a94@lucifer.local
Link: https://lkml.kernel.org/r/3fb431699639ded8fdc63d2210aa77a38c8891f1.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: SeongJae Park \
Reviewed-by: Vlastimil Babka 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Cc: Zi Yan 
Signed-off-by: Andrew Morton

mm: avoid unnecessary uses of is_swap_pte()

2025-11-24T23:08:50+00:00

There's an established convention in the kernel that we treat PTEs as
containing swap entries (and the unfortunately named non-swap swap
entries) should they be neither empty (i.e.  pte_none() evaluating true)
nor present (i.e.  pte_present() evaluating true).

However, there is some inconsistency in how this is applied, as we also
have the is_swap_pte() helper which explicitly performs this check:

	/* check whether a pte points to a swap entry */
	static inline int is_swap_pte(pte_t pte)
	{
		return !pte_none(pte) && !pte_present(pte);
	}

As this represents a predicate, and it's logical to assume that in order
to establish that a PTE entry can correctly be manipulated as a
swap/non-swap entry, this predicate seems as if it must first be checked.

But we instead, we far more often utilise the established convention of
checking pte_none() / pte_present() before operating on entries as if they
were swap/non-swap.

This patch works towards correcting this inconsistency by removing all
uses of is_swap_pte() where we are already in a position where we perform
pte_none()/pte_present() checks anyway or otherwise it is clearly logical
to do so.

We also take advantage of the fact that pte_swp_uffd_wp() is only set on
swap entries.

Additionally, update comments referencing to is_swap_pte() and
non_swap_entry().

No functional change intended.

Link: https://lkml.kernel.org/r/17fd6d7f46a846517fd455fadd640af47fcd7c55.1762812360.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Zi Yan 
Reviewed-by: Vlastimil Babka 
Cc: Alexander Gordeev 
Cc: Alistair Popple 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Axel Rasmussen 
Cc: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: Byungchul Park 
Cc: Chengming Zhou 
Cc: Chris Li 
Cc: Christian Borntraeger 
Cc: Christian Brauner 
Cc: Claudio Imbrenda 
Cc: David Hildenbrand 
Cc: Dev Jain 
Cc: Gerald Schaefer 
Cc: Gregory Price 
Cc: Heiko Carstens 
Cc: "Huang, Ying" 
Cc: Hugh Dickins 
Cc: Jan Kara 
Cc: Jann Horn 
Cc: Janosch Frank 
Cc: Jason Gunthorpe 
Cc: Joshua Hahn 
Cc: Kairui Song 
Cc: Kemeng Shi 
Cc: Lance Yang 
Cc: Leon Romanovsky 
Cc: Liam Howlett 
Cc: Mathew Brost 
Cc: Matthew Wilcox (Oracle) 
Cc: Miaohe Lin 
Cc: Michal Hocko 
Cc: Mike Rapoport 
Cc: Muchun Song 
Cc: Naoya Horiguchi 
Cc: Nhat Pham 
Cc: Nico Pache 
Cc: Oscar Salvador 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Rakie Kim 
Cc: Rik van Riel 
Cc: Ryan Roberts 
Cc: SeongJae Park 
Cc: Suren Baghdasaryan 
Cc: Sven Schnelle 
Cc: Vasily Gorbik 
Cc: Wei Xu 
Cc: xu xin 
Cc: Yuanchu Xie 
Signed-off-by: Andrew Morton

mm/huge_memory: implement device-private THP splitting

2025-11-24T23:08:48+00:00

Add support for splitting device-private THP folios, enabling fallback
to smaller page sizes when large page allocation or migration fails.

Key changes:
- split_huge_pmd(): Handle device-private PMD entries during splitting
- Preserve RMAP_EXCLUSIVE semantics for anonymous exclusive folios
- Skip RMP_USE_SHARED_ZEROPAGE for device-private entries as they
  don't support shared zero page semantics

Link: https://lkml.kernel.org/r/20251001065707.920170-6-balbirs@nvidia.com
Signed-off-by: Balbir Singh 
Cc: David Hildenbrand 
Cc: Zi Yan 
Cc: Joshua Hahn 
Cc: Rakie Kim 
Cc: Byungchul Park 
Cc: Gregory Price 
Cc: Ying Huang 
Cc: Alistair Popple 
Cc: Oscar Salvador 
Cc: Lorenzo Stoakes 
Cc: Baolin Wang 
Cc: "Liam R. Howlett" 
Cc: Nico Pache 
Cc: Ryan Roberts 
Cc: Dev Jain 
Cc: Barry Song 
Cc: Lyude Paul 
Cc: Danilo Krummrich 
Cc: David Airlie 
Cc: Simona Vetter 
Cc: Ralph Campbell 
Cc: Mika Penttilä 
Cc: Matthew Brost 
Cc: Francois Dugast 
Signed-off-by: Andrew Morton

migrate: optimise alloc_migration_target()

2025-11-20T21:43:57+00:00

Extract the zone number directly from the folio instead of using the
folio's zone number to look up the zone and asking the zone what its
number is.

[ziy@nvidia.com: fix folio_zonenum() return type]
  Link: https://lkml.kernel.org/r/26E8FF35-503E-4F14-98F7-7B4FA25FBD37@nvidia.com
Link: https://lkml.kernel.org/r/20251106201452.2292631-2-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Zi Yan 
Cc: David Hildenbrand 
Cc: Zi Yan 
Cc: Matthew Brost 
Cc: Joshua Hahn 
Cc: Rakie Kim 
Cc: Byungchul Park 
Cc: Gregory Price 
Cc: Ying Huang 
Cc: Alistair Popple 
Signed-off-by: Andrew Morton

mm/migrate, swap: drop usage of folio_index

2025-11-17T01:28:20+00:00

This helper was used when swap cache was mixed with page cache.  Now they
are completely separate from each other, access to the swap cache is all
wrapped by the swap_cache_* helpers, which expect the folio's swap entry
as a parameter.

This helper is no longer used, remove the last redundant user and drop it.

Link: https://lkml.kernel.org/r/20251024-swap-clean-after-swap-table-p1-v2-4-a709469052e7@tencent.com
Signed-off-by: Kairui Song 
Acked-by: Chris Li 
Acked-by: Nhat Pham 
Reviewed-by: Baolin Wang 
Cc: Baoquan He 
Cc: Barry Song 
Cc: David Hildenbrand 
Cc: "Huang, Ying" 
Cc: Kemeng Shi 
Cc: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton

mm: prevent poison consumption when splitting THP

2025-10-15T20:24:34+00:00

When performing memory error injection on a THP (Transparent Huge Page)
mapped to userspace on an x86 server, the kernel panics with the following
trace.  The expected behavior is to terminate the affected process instead
of panicking the kernel, as the x86 Machine Check code can recover from an
in-userspace #MC.

  mce: [Hardware Error]: CPU 0: Machine Check Exception: f Bank 3: bd80000000070134
  mce: [Hardware Error]: RIP 10: {memchr_inv+0x4c/0xf0}
  mce: [Hardware Error]: TSC afff7bbff88a ADDR 1d301b000 MISC 80 PPIN 1e741e77539027db
  mce: [Hardware Error]: PROCESSOR 0:d06d0 TIME 1758093249 SOCKET 0 APIC 0 microcode 80000320
  mce: [Hardware Error]: Run the above through 'mcelog --ascii'
  mce: [Hardware Error]: Machine check: Data load in unrecoverable area of kernel
  Kernel panic - not syncing: Fatal local machine check

The root cause of this panic is that handling a memory failure triggered
by an in-userspace #MC necessitates splitting the THP.  The splitting
process employs a mechanism, implemented in
try_to_map_unused_to_zeropage(), which reads the pages in the THP to
identify zero-filled pages.  However, reading the pages in the THP results
in a second in-kernel #MC, occurring before the initial memory_failure()
completes, ultimately leading to a kernel panic.  See the kernel panic
call trace on the two #MCs.

  First Machine Check occurs // [1]
    memory_failure()         // [2]
      try_to_split_thp_page()
        split_huge_page()
          split_huge_page_to_list_to_order()
            __folio_split()  // [3]
              remap_page()
                remove_migration_ptes()
                  remove_migration_pte()
                    try_to_map_unused_to_zeropage()  // [4]
                      memchr_inv()                   // [5]
                        Second Machine Check occurs  // [6]
                          Kernel panic

[1] Triggered by accessing a hardware-poisoned THP in userspace, which is
    typically recoverable by terminating the affected process.

[2] Call folio_set_has_hwpoisoned() before try_to_split_thp_page().

[3] Pass the RMP_USE_SHARED_ZEROPAGE remap flag to remap_page().

[4] Try to map the unused THP to zeropage.

[5] Re-access pages in the hw-poisoned THP in the kernel.

[6] Triggered in-kernel, leading to a panic kernel.

In Step[2], memory_failure() sets the poisoned flag on the page in the THP
by TestSetPageHWPoison() before calling try_to_split_thp_page().

As suggested by David Hildenbrand, fix this panic by not accessing to the
poisoned page in the THP during zeropage identification, while continuing
to scan unaffected pages in the THP for possible zeropage mapping.  This
prevents a second in-kernel #MC that would cause kernel panic in Step[4].

Thanks to Andrew Zaborowski for his initial work on fixing this issue.

Link: https://lkml.kernel.org/r/20251015064926.1887643-1-qiuxu.zhuo@intel.com
Link: https://lkml.kernel.org/r/20251011075520.320862-1-qiuxu.zhuo@intel.com
Fixes: b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")
Signed-off-by: Qiuxu Zhuo 
Reported-by: Farrah Chen 
Suggested-by: David Hildenbrand 
Acked-by: David Hildenbrand 
Tested-by: Farrah Chen 
Tested-by: Qiuxu Zhuo 
Acked-by: Lance Yang 
Reviewed-by: Wei Yang 
Acked-by: Zi Yan 
Reviewed-by: Miaohe Lin 
Cc: Barry Song 
Cc: Dev Jain 
Cc: Jiaqi Yan 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: "Luck, Tony" 
Cc: Mariano Pache 
Cc: Miaohe Lin 
Cc: Naoya Horiguchi 
Cc: Ryan Roberts 
Cc: 
Signed-off-by: Andrew Morton