linux-toradex.git/include/linux/mm_inline.h, branch v6.0-rc1

mm/shmem: persist uffd-wp bit across zapping for file-backed

2022-05-13T14:20:10+00:00

File-backed memory is prone to being unmapped at any time.  It means all
information in the pte will be dropped, including the uffd-wp flag.

To persist the uffd-wp flag, we'll use the pte markers.  This patch
teaches the zap code to understand uffd-wp and know when to keep or drop
the uffd-wp bit.

Add a new flag ZAP_FLAG_DROP_MARKER and set it in zap_details when we
don't want to persist such an information, for example, when destroying
the whole vma, or punching a hole in a shmem file.  For the rest cases we
should never drop the uffd-wp bit, or the wr-protect information will get
lost.

The new ZAP_FLAG_DROP_MARKER needs to be put into mm.h rather than
memory.c because it'll be further referenced in hugetlb files later.

Link: https://lkml.kernel.org/r/20220405014847.14295-1-peterx@redhat.com
Signed-off-by: Peter Xu 
Cc: Alistair Popple 
Cc: Andrea Arcangeli 
Cc: Axel Rasmussen 
Cc: David Hildenbrand 
Cc: Hugh Dickins 
Cc: Jerome Glisse 
Cc: "Kirill A . Shutemov" 
Cc: Matthew Wilcox 
Cc: Mike Kravetz 
Cc: Mike Rapoport 
Cc: Nadav Amit 
Signed-off-by: Andrew Morton

Merge tag 'folio-5.18c' of git://git.infradead.org/users/willy/pagecache

2022-03-23T00:03:12+00:00

Pull folio updates from Matthew Wilcox:

 - Rewrite how munlock works to massively reduce the contention on
   i_mmap_rwsem (Hugh Dickins):

     https://lore.kernel.org/linux-mm/8e4356d-9622-a7f0-b2c-f116b5f2efea@google.com/

 - Sort out the page refcount mess for ZONE_DEVICE pages (Christoph
   Hellwig):

     https://lore.kernel.org/linux-mm/20220210072828.2930359-1-hch@lst.de/

 - Convert GUP to use folios and make pincount available for order-1
   pages. (Matthew Wilcox)

 - Convert a few more truncation functions to use folios (Matthew
   Wilcox)

 - Convert page_vma_mapped_walk to use PFNs instead of pages (Matthew
   Wilcox)

 - Convert rmap_walk to use folios (Matthew Wilcox)

 - Convert most of shrink_page_list() to use a folio (Matthew Wilcox)

 - Add support for creating large folios in readahead (Matthew Wilcox)

* tag 'folio-5.18c' of git://git.infradead.org/users/willy/pagecache: (114 commits)
  mm/damon: minor cleanup for damon_pa_young
  selftests/vm/transhuge-stress: Support file-backed PMD folios
  mm/filemap: Support VM_HUGEPAGE for file mappings
  mm/readahead: Switch to page_cache_ra_order
  mm/readahead: Align file mappings for non-DAX
  mm/readahead: Add large folio readahead
  mm: Support arbitrary THP sizes
  mm: Make large folios depend on THP
  mm: Fix READ_ONLY_THP warning
  mm/filemap: Allow large folios to be added to the page cache
  mm: Turn can_split_huge_page() into can_split_folio()
  mm/vmscan: Convert pageout() to take a folio
  mm/vmscan: Turn page_check_references() into folio_check_references()
  mm/vmscan: Account large folios correctly
  mm/vmscan: Optimise shrink_page_list for non-PMD-sized folios
  mm/vmscan: Free non-shmem folios without splitting them
  mm/rmap: Constify the rmap_walk_control argument
  mm/rmap: Convert rmap_walk() to take a folio
  mm: Turn page_anon_vma() into folio_anon_vma()
  mm/rmap: Turn page_lock_anon_vma_read() into folio_lock_anon_vma_read()
  ...

mm: prevent vm_area_struct::anon_name refcount saturation

2022-03-05T19:08:32+00:00

A deep process chain with many vmas could grow really high.  With
default sysctl_max_map_count (64k) and default pid_max (32k) the max
number of vmas in the system is 2147450880 and the refcounter has
headroom of 1073774592 before it reaches REFCOUNT_SATURATED
(3221225472).

Therefore it's unlikely that an anonymous name refcounter will overflow
with these defaults.  Currently the max for pid_max is PID_MAX_LIMIT
(4194304) and for sysctl_max_map_count it's INT_MAX (2147483647).  In
this configuration anon_vma_name refcount overflow becomes theoretically
possible (that still require heavy sharing of that anon_vma_name between
processes).

kref refcounting interface used in anon_vma_name structure will detect a
counter overflow when it reaches REFCOUNT_SATURATED value but will only
generate a warning and freeze the ref counter.  This would lead to the
refcounted object never being freed.  A determined attacker could leak
memory like that but it would be rather expensive and inefficient way to
do so.

To ensure anon_vma_name refcount does not overflow, stop anon_vma_name
sharing when the refcount reaches REFCOUNT_MAX (2147483647), which still
leaves INT_MAX/2 (1073741823) values before the counter reaches
REFCOUNT_SATURATED.  This should provide enough headroom for raising the
refcounts temporarily.

Link: https://lkml.kernel.org/r/20220223153613.835563-2-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Suggested-by: Michal Hocko 
Acked-by: Michal Hocko 
Cc: Alexey Gladkov 
Cc: Chris Hyser 
Cc: Christian Brauner 
Cc: Colin Cross 
Cc: Cyrill Gorcunov 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Davidlohr Bueso 
Cc: "Eric W. Biederman" 
Cc: Johannes Weiner 
Cc: Kees Cook 
Cc: "Kirill A. Shutemov" 
Cc: Matthew Wilcox 
Cc: Peter Collingbourne 
Cc: Sasha Levin 
Cc: Sumit Semwal 
Cc: Vlastimil Babka 
Cc: Xiaofeng Cao 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: refactor vm_area_struct::anon_vma_name usage code

2022-03-05T19:08:32+00:00

Avoid mixing strings and their anon_vma_name referenced pointers by
using struct anon_vma_name whenever possible.  This simplifies the code
and allows easier sharing of anon_vma_name structures when they
represent the same name.

[surenb@google.com: fix comment]

Link: https://lkml.kernel.org/r/20220223153613.835563-1-surenb@google.com
Link: https://lkml.kernel.org/r/20220224231834.1481408-1-surenb@google.com
Signed-off-by: Suren Baghdasaryan 
Suggested-by: Matthew Wilcox 
Suggested-by: Michal Hocko 
Acked-by: Michal Hocko 
Cc: Colin Cross 
Cc: Sumit Semwal 
Cc: Dave Hansen 
Cc: Kees Cook 
Cc: "Kirill A. Shutemov" 
Cc: Vlastimil Babka 
Cc: Johannes Weiner 
Cc: "Eric W. Biederman" 
Cc: Christian Brauner 
Cc: Alexey Gladkov 
Cc: Sasha Levin 
Cc: Chris Hyser 
Cc: Davidlohr Bueso 
Cc: Peter Collingbourne 
Cc: Xiaofeng Cao 
Cc: David Hildenbrand 
Cc: Cyrill Gorcunov 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/munlock: maintain page->mlock_count while unevictable

2022-02-17T16:56:57+00:00

Previous patches have been preparatory: now implement page->mlock_count.
The ordering of the "Unevictable LRU" is of no significance, and there is
no point holding unevictable pages on a list: place page->mlock_count to
overlay page->lru.prev (since page->lru.next is overlaid by compound_head,
which needs to be even so as not to satisfy PageTail - though 2 could be
added instead of 1 for each mlock, if that's ever an improvement).

But it's only safe to rely on or modify page->mlock_count while lruvec
lock is held and page is on unevictable "LRU" - we can save lots of edits
by continuing to pretend that there's an imaginary LRU here (there is an
unevictable count which still needs to be maintained, but not a list).

The mlock_count technique suffers from an unreliability much like with
page_mlock(): while someone else has the page off LRU, not much can
be done.  As before, err on the safe side (behave as if mlock_count 0),
and let try_to_unlock_one() move the page to unevictable if reclaim finds
out later on - a few misplaced pages don't matter, what we want to avoid
is imbalancing reclaim by flooding evictable lists with unevictable pages.

I am not a fan of "if (!isolate_lru_page(page)) putback_lru_page(page);":
if we have taken lruvec lock to get the page off its present list, then
we save everyone trouble (and however many extra atomic ops) by putting
it on its destination list immediately.

Signed-off-by: Hugh Dickins 
Acked-by: Vlastimil Babka 
Signed-off-by: Matthew Wilcox (Oracle)

mm: move tlb_flush_pending inline helpers to mm_inline.h

2022-01-15T14:30:27+00:00

linux/mm_types.h should only define structure definitions, to make it
cheap to include elsewhere.  The atomic_t helper function definitions
are particularly large, so it's better to move the helpers using those
into the existing linux/mm_inline.h and only include that where needed.

As a follow-up, we may want to go through all the indirect includes in
mm_types.h and reduce them as much as possible.

Link: https://lkml.kernel.org/r/20211207125710.2503446-2-arnd@kernel.org
Signed-off-by: Arnd Bergmann 
Cc: Al Viro 
Cc: Stephen Rothwell 
Cc: Suren Baghdasaryan 
Cc: Colin Cross 
Cc: Kees Cook 
Cc: Peter Xu 
Cc: Peter Zijlstra (Intel) 
Cc: Yu Zhao 
Cc: Vlastimil Babka 
Cc: Matthew Wilcox (Oracle) 
Cc: Eric Biederman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: move anon_vma declarations to linux/mm_inline.h

2022-01-15T14:30:27+00:00

The patch to add anonymous vma names causes a build failure in some
configurations:

  include/linux/mm_types.h: In function 'is_same_vma_anon_name':
  include/linux/mm_types.h:924:37: error: implicit declaration of function 'strcmp' [-Werror=implicit-function-declaration]
    924 |         return name && vma_name && !strcmp(name, vma_name);
        |                                     ^~~~~~
  include/linux/mm_types.h:22:1: note: 'strcmp' is defined in header ''; did you forget to '#include '?

This should not really be part of linux/mm_types.h in the first place,
as that header is meant to only contain structure defintions and need a
minimum set of indirect includes itself.

While the header clearly includes more than it should at this point,
let's not make it worse by including string.h as well, which would pull
in the expensive (compile-speed wise) fortify-string logic.

Move the new functions into a separate header that only needs to be
included in a couple of locations.

Link: https://lkml.kernel.org/r/20211207125710.2503446-1-arnd@kernel.org
Fixes: "mm: add a field to store names for private anonymous memory"
Signed-off-by: Arnd Bergmann 
Cc: Al Viro 
Cc: Colin Cross 
Cc: Eric Biederman 
Cc: Kees Cook 
Cc: Matthew Wilcox (Oracle) 
Cc: Peter Xu 
Cc: Peter Zijlstra (Intel) 
Cc: Stephen Rothwell 
Cc: Suren Baghdasaryan 
Cc: Vlastimil Babka 
Cc: Yu Zhao 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/lru: Add folio LRU functions

2021-09-27T13:27:29+00:00

Handle arbitrary-order folios being added to the LRU.  By definition,
all pages being added to the LRU were already head or base pages, but
call page_folio() on them anyway to get the type right and avoid the
buried calls to compound_head().

Saves 783 bytes of kernel text; no functions grow.

Signed-off-by: Matthew Wilcox (Oracle) 
Reviewed-by: Yu Zhao 
Reviewed-by: Christoph Hellwig 
Reviewed-by: David Howells 
Acked-by: Kirill A. Shutemov 
Acked-by: Mike Rapoport 
Acked-by: Vlastimil Babka

include/linux/mm_inline.h: fold __update_lru_size() into its sole caller

2021-02-24T21:38:33+00:00

All other references to the function were removed after commit
a892cb6b977f ("mm/vmscan.c: use update_lru_size() in update_lru_sizes()").

Link: https://lore.kernel.org/linux-mm/20201207220949.830352-10-yuzhao@google.com/
Link: https://lkml.kernel.org/r/20210122220600.906146-10-yuzhao@google.com
Signed-off-by: Yu Zhao 
Reviewed-by: Alex Shi 
Cc: Hugh Dickins 
Cc: Johannes Weiner 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Vladimir Davydov 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

include/linux/mm_inline.h: fold page_lru_base_type() into its sole caller

2021-02-24T21:38:33+00:00

We've removed all other references to this function.

Link: https://lore.kernel.org/linux-mm/20201207220949.830352-9-yuzhao@google.com/
Link: https://lkml.kernel.org/r/20210122220600.906146-9-yuzhao@google.com
Signed-off-by: Yu Zhao 
Reviewed-by: Alex Shi 
Cc: Hugh Dickins 
Cc: Johannes Weiner 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Vladimir Davydov 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds