linux-toradex.git/mm/page_idle.c, branch v6.16

mm/page_idle: handle device-exclusive entries correctly in page_idle_clear_pte_refs_one()

2025-03-17T05:05:59+00:00

Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we
can return with a device-exclusive entry from page_vma_mapped_walk().

page_idle_clear_pte_refs_one() is not prepared for that, so let's teach it
what to do with these PFN swap PTEs.  Note that device-private entries are
so far not applicable on that path, as page_idle_get_folio() filters out
non-lru folios.

Should we just skip PFN swap PTEs completely?  Possible, but it seems
straight forward to just handle them correctly.

Note that we could currently only run into this case with device-exclusive
entries on THPs.  We still adjust the mapcount on conversion to
device-exclusive; this makes the rmap walk abort early for small folios,
because we'll always have !folio_mapped() with a single device-exclusive
entry.  We'll adjust the mapcount logic once all page_vma_mapped_walk()
users can properly handle device-exclusive entries.

Link: https://lkml.kernel.org/r/20250210193801.781278-14-david@redhat.com
Fixes: b756a3b5e7ea ("mm: device exclusive memory access")
Signed-off-by: David Hildenbrand 
Tested-by: Alistair Popple 
Cc: Alex Shi 
Cc: Danilo Krummrich 
Cc: Dave Airlie 
Cc: Jann Horn 
Cc: Jason Gunthorpe 
Cc: Jerome Glisse 
Cc: John Hubbard 
Cc: Jonathan Corbet 
Cc: Karol Herbst 
Cc: Liam Howlett 
Cc: Lorenzo Stoakes 
Cc: Lyude 
Cc: "Masami Hiramatsu (Google)" 
Cc: Oleg Nesterov 
Cc: Pasha Tatashin 
Cc: Peter Xu 
Cc: Peter Zijlstra (Intel) 
Cc: SeongJae Park 
Cc: Simona Vetter 
Cc: Vlastimil Babka 
Cc: Yanteng Si 
Cc: Barry Song 
Signed-off-by: Andrew Morton

mm/page_idle: constify 'struct bin_attribute'

2025-01-26T04:22:19+00:00

The sysfs core now allows instances of 'struct bin_attribute' to be moved
into read-only memory.  Make use of that to protect them against
accidental or malicious modifications.

Link: https://lkml.kernel.org/r/20241216-sysfs-const-bin_attr-page_idle-v1-1-cc01ecc55196@weissschuh.net
Signed-off-by: Thomas Weißschuh 
Signed-off-by: Andrew Morton

mm: page_idle: convert page idle to use a folio

2023-01-19T01:12:52+00:00

Firstly, make page_idle_get_page() return a folio, also rename it to
page_idle_get_folio(), then, use it to convert page_idle_bitmap_read() and
page_idle_bitmap_write() functions.

Link: https://lkml.kernel.org/r/20221230070849.63358-3-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang 
Reviewed-by: SeongJae Park 
Cc: David Hildenbrand 
Cc: Matthew Wilcox (Oracle) 
Cc: Vishal Moola (Oracle) 
Signed-off-by: Andrew Morton

mm: don't be stuck to rmap lock on reclaim path

2022-05-19T21:08:54+00:00

The rmap locks(i_mmap_rwsem and anon_vma->root->rwsem) could be contended
under memory pressure if processes keep working on their vmas(e.g., fork,
mmap, munmap).  It makes reclaim path stuck.  In our real workload traces,
we see kswapd is waiting the lock for 300ms+(worst case, a sec) and it
makes other processes entering direct reclaim, which were also stuck on
the lock.

This patch makes lru aging path try_lock mode like shink_page_list so the
reclaim context will keep working with next lru pages without being stuck.
if it found the rmap lock contended, it rotates the page back to head of
lru in both active/inactive lrus to make them consistent behavior, which
is basic starting point rather than adding more heristic.

Since this patch introduces a new "contended" field as out-param along
with try_lock in-param in rmap_walk_control, it's not immutable any longer
if the try_lock is set so remove const keywords on rmap related functions.
Since rmap walking is already expensive operation, I doubt the const
would help sizable benefit( And we didn't have it until 5.17).

In a heavy app workload in Android, trace shows following statistics.  It
almost removes rmap lock contention from reclaim path.

Martin Liu reported:

Before:

   max_dur(ms)  min_dur(ms)  max-min(dur)ms  avg_dur(ms)  sum_dur(ms)  count blocked_function
         1632            0            1631   151.542173        31672    209  page_lock_anon_vma_read
          601            0             601   145.544681        28817    198  rmap_walk_file

After:

   max_dur(ms)  min_dur(ms)  max-min(dur)ms  avg_dur(ms)  sum_dur(ms)  count blocked_function
          NaN          NaN              NaN          NaN          NaN    0.0             NaN
            0            0                0     0.127645            1     12  rmap_walk_file

[minchan@kernel.org: add comment, per Matthew]
  Link: https://lkml.kernel.org/r/YnNqeB5tUf6LZ57b@google.com
Link: https://lkml.kernel.org/r/20220510215423.164547-1-minchan@kernel.org
Signed-off-by: Minchan Kim 
Acked-by: Johannes Weiner 
Cc: Suren Baghdasaryan 
Cc: Michal Hocko 
Cc: John Dias 
Cc: Tim Murray 
Cc: Matthew Wilcox 
Cc: Vladimir Davydov 
Cc: Martin Liu 
Cc: Minchan Kim 
Cc: Matthew Wilcox 
Signed-off-by: Andrew Morton

mm/rmap: Constify the rmap_walk_control argument

2022-03-21T17:01:35+00:00

The rmap walking functions do not modify the rmap_walk_control, and
page_idle_clear_pte_refs() takes advantage of that to move construction
of the rmap_walk_control to compile time.  This lets us remove an
unclean cast.

Signed-off-by: Matthew Wilcox (Oracle)

mm/rmap: Convert rmap_walk() to take a folio

2022-03-21T17:01:35+00:00

This ripples all the way through to every calling and called function
from rmap.

Signed-off-by: Matthew Wilcox (Oracle)

mm/rmap: Turn page_referenced() into folio_referenced()

2022-03-21T16:59:03+00:00

Both its callers pass a page which was previously on an LRU list,
so were passing a folio by definition.  Use the type system to enforce
that and remove a few calls to compound_head().

Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Christoph Hellwig

mm/page_idle: Convert page_idle_clear_pte_refs() to use a folio

2022-03-21T16:59:02+00:00

The PG_idle and PG_young bits are ignored if they're set on tail
pages, so ensure we're passing a folio around.

Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Christoph Hellwig

mm: Add DEFINE_PAGE_VMA_WALK and DEFINE_FOLIO_VMA_WALK

2022-03-21T16:59:02+00:00

Instead of declaring a struct page_vma_mapped_walk directly,
use these helpers to allow us to transition to a PFN approach in the
following patches.

Signed-off-by: Matthew Wilcox (Oracle)

mm/idle_page_tracking: make PG_idle reusable

2021-09-08T18:50:24+00:00

PG_idle and PG_young allow the two PTE Accessed bit users, Idle Page
Tracking and the reclaim logic concurrently work while not interfering
with each other.  That is, when they need to clear the Accessed bit, they
set PG_young to represent the previous state of the bit, respectively.
And when they need to read the bit, if the bit is cleared, they further
read the PG_young to know whether the other has cleared the bit meanwhile
or not.

For yet another user of the PTE Accessed bit, we could add another page
flag, or extend the mechanism to use the flags.  For the DAMON usecase,
however, we don't need to do that just yet.  IDLE_PAGE_TRACKING and DAMON
are mutually exclusive, so there's only ever going to be one user of the
current set of flags.

In this commit, we split out the CONFIG options to allow for the use of
PG_young and PG_idle outside of idle page tracking.

In the next commit, DAMON's reference implementation of the virtual memory
address space monitoring primitives will use it.

[sjpark@amazon.de: set PAGE_EXTENSION for non-64BIT]
  Link: https://lkml.kernel.org/r/20210806095153.6444-1-sj38.park@gmail.com
[akpm@linux-foundation.org: tweak Kconfig text]
[sjpark@amazon.de: hide PAGE_IDLE_FLAG from users]
  Link: https://lkml.kernel.org/r/20210813081238.34705-1-sj38.park@gmail.com

Link: https://lkml.kernel.org/r/20210716081449.22187-5-sj38.park@gmail.com
Signed-off-by: SeongJae Park 
Reviewed-by: Shakeel Butt 
Reviewed-by: Fernand Sieber 
Cc: Alexander Shishkin 
Cc: Amit Shah 
Cc: Benjamin Herrenschmidt 
Cc: Brendan Higgins 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: David Woodhouse 
Cc: Fan Du 
Cc: Greg Kroah-Hartman 
Cc: Greg Thelen 
Cc: Ingo Molnar 
Cc: Joe Perches 
Cc: Jonathan Cameron 
Cc: Jonathan Corbet 
Cc: Leonard Foerster 
Cc: Marco Elver 
Cc: Markus Boehme 
Cc: Maximilian Heyne 
Cc: Mel Gorman 
Cc: Minchan Kim 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Cc: Rik van Riel 
Cc: Shuah Khan 
Cc: Steven Rostedt (VMware) 
Cc: Vladimir Davydov 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds