linux-toradex.git/mm/Makefile, branch v6.14

mm: pgtable: reclaim empty PTE page in madvise(MADV_DONTNEED)

2025-01-14T06:40:48+00:00

Now in order to pursue high performance, applications mostly use some
high-performance user-mode memory allocators, such as jemalloc or
tcmalloc.  These memory allocators use madvise(MADV_DONTNEED or MADV_FREE)
to release physical memory, but neither MADV_DONTNEED nor MADV_FREE will
release page table memory, which may cause huge page table memory usage.

The following are a memory usage snapshot of one process which actually
happened on our server:

        VIRT:  55t
        RES:   590g
        VmPTE: 110g

In this case, most of the page table entries are empty.  For such a PTE
page where all entries are empty, we can actually free it back to the
system for others to use.

As a first step, this commit aims to synchronously free the empty PTE
pages in madvise(MADV_DONTNEED) case.  We will detect and free empty PTE
pages in zap_pte_range(), and will add zap_details.reclaim_pt to exclude
cases other than madvise(MADV_DONTNEED).

Once an empty PTE is detected, we first try to hold the pmd lock within
the pte lock.  If successful, we clear the pmd entry directly (fast path).
Otherwise, we wait until the pte lock is released, then re-hold the pmd
and pte locks and loop PTRS_PER_PTE times to check pte_none() to re-detect
whether the PTE page is empty and free it (slow path).

For other cases such as madvise(MADV_FREE), consider scanning and freeing
empty PTE pages asynchronously in the future.

The following code snippet can show the effect of optimization:

        mmap 50G
        while (1) {
                for (; i < 1024 * 25; i++) {
                        touch 2M memory
                        madvise MADV_DONTNEED 2M
                }
        }

As we can see, the memory usage of VmPTE is reduced:

                        before                          after
VIRT                   50.0 GB                        50.0 GB
RES                     3.1 MB                         3.1 MB
VmPTE                102640 KB                         240 KB

[zhengqi.arch@bytedance.com: fix uninitialized symbol 'ptl']
  Link: https://lkml.kernel.org/r/20241206112348.51570-1-zhengqi.arch@bytedance.com
  Link: https://lore.kernel.org/linux-mm/224e6a4e-43b5-4080-bdd8-b0a6fb2f0853@stanley.mountain/
Link: https://lkml.kernel.org/r/92aba2b319a734913f18ba41e7d86a265f0b84e2.1733305182.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng 
Cc: Andy Lutomirski 
Cc: Catalin Marinas 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jann Horn 
Cc: Lorenzo Stoakes 
Cc: Matthew Wilcox 
Cc: Mel Gorman 
Cc: Muchun Song 
Cc: Peter Xu 
Cc: Peter Zijlstra 
Cc: Will Deacon 
Cc: Zach O'Keefe 
Cc: Dan Carpenter 
Signed-off-by: Andrew Morton

mm: move the page fragment allocator from page_alloc into its own file

2024-11-11T18:56:26+00:00

Inspired by [1], move the page fragment allocator from page_alloc
into its own c file and header file, as we are about to make more
change for it to replace another page_frag implementation in
sock.c

As this patchset is going to replace 'struct page_frag' with
'struct page_frag_cache' in sched.h, including page_frag_cache.h
in sched.h has a compiler error caused by interdependence between
mm_types.h and mm.h for asm-offsets.c, see [2]. So avoid the compiler
error by moving 'struct page_frag_cache' to mm_types_task.h as
suggested by Alexander, see [3].

1. https://lore.kernel.org/all/20230411160902.4134381-3-dhowells@redhat.com/
2. https://lore.kernel.org/all/15623dac-9358-4597-b3ee-3694a5956920@gmail.com/
3. https://lore.kernel.org/all/CAKgT0UdH1yD=LSCXFJ=YM_aiA4OomD-2wXykO42bizaWMt_HOA@mail.gmail.com/
CC: David Howells 
CC: Linux-MM 
Signed-off-by: Yunsheng Lin 
Acked-by: Andrew Morton 
Reviewed-by: Alexander Duyck 
Link: https://patch.msgid.link/20241028115343.3405838-3-linyunsheng@huawei.com
Signed-off-by: Jakub Kicinski

mm: introduce numa_emulation

2024-09-04T04:15:31+00:00

Move numa_emulation code from arch/x86 to mm/numa_emulation.c

This code will be later reused by arch_numa.

No functional changes.

Link: https://lkml.kernel.org/r/20240807064110.1003856-20-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Tested-by: Zi Yan  # for x86_64 and arm64
Reviewed-by: Jonathan Cameron 
Tested-by: Jonathan Cameron  [arm64 + CXL via QEMU]
Acked-by: Dan Williams 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Dave Hansen 
Cc: David Hildenbrand 
Cc: Davidlohr Bueso 
Cc: David S. Miller 
Cc: Greg Kroah-Hartman 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: John Paul Adrian Glaubitz 
Cc: Jonathan Corbet 
Cc: Michael Ellerman 
Cc: Palmer Dabbelt 
Cc: Rafael J. Wysocki 
Cc: Rob Herring (Arm) 
Cc: Samuel Holland 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleixner 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: introduce numa_memblks

2024-09-04T04:15:30+00:00

Move code dealing with numa_memblks from arch/x86 to mm/ and add Kconfig
options to let x86 select it in its Kconfig.

This code will be later reused by arch_numa.

No functional changes.

Link: https://lkml.kernel.org/r/20240807064110.1003856-18-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Tested-by: Zi Yan  # for x86_64 and arm64
Reviewed-by: Jonathan Cameron 
Tested-by: Jonathan Cameron  [arm64 + CXL via QEMU]
Acked-by: Dan Williams 
Acked-by: David Hildenbrand 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Dave Hansen 
Cc: Davidlohr Bueso 
Cc: David S. Miller 
Cc: Greg Kroah-Hartman 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: John Paul Adrian Glaubitz 
Cc: Jonathan Corbet 
Cc: Michael Ellerman 
Cc: Palmer Dabbelt 
Cc: Rafael J. Wysocki 
Cc: Rob Herring (Arm) 
Cc: Samuel Holland 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleixner 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: move kernel/numa.c to mm/

2024-09-04T04:15:26+00:00

Patch series "mm: introduce numa_memblks", v4.

Following the discussion about handling of CXL fixed memory windows on
arm64 [1] I decided to bite the bullet and move numa_memblks from x86 to
the generic code so they will be available on arm64/riscv and maybe on
loongarch sometime later.

While it could be possible to use memblock to describe CXL memory windows,
it currently lacks notion of unpopulated memory ranges and numa_memblks
does implement this.

Another reason to make numa_memblks generic is that both arch_numa (arm64
and riscv) and loongarch use trimmed copy of x86 code although there is no
fundamental reason why the same code cannot be used on all these
platforms.  Having numa_memblks in mm/ will make it's interaction with
ACPI and FDT more consistent and I believe will reduce maintenance burden.

And with generic numa_memblks it is (almost) straightforward to enable
NUMA emulation on arm64 and riscv.

The first 9 commits in this series are cleanups that are not strictly
related to numa_memblks.
Commits 10-16 slightly reorder code in x86 to allow extracting numa_memblks
and NUMA emulation to the generic code.
Commits 17-19 actually move the code from arch/x86/ to mm/ and commits 20-22
does some aftermath cleanups.
Commit 23 updates of_numa_init() to return error of no NUMA nodes were
found in the device tree.
Commit 24 switches arch_numa to numa_memblks.
Commit 25 enables usage of phys_to_target_node() and
memory_add_physaddr_to_nid() with numa_memblks.
Commit 26 moves the description for numa=fake from x86 to admin-guide.

[1] https://lore.kernel.org/all/20240529171236.32002-1-Jonathan.Cameron@huawei.com/


This patch (of 26):

The stub functions in kernel/numa.c belong to mm/ rather than to kernel/

Link: https://lkml.kernel.org/r/20240807064110.1003856-1-rppt@kernel.org
Link: https://lkml.kernel.org/r/20240807064110.1003856-2-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) 
Acked-by: David Hildenbrand 
Reviewed-by: Jonathan Cameron 
Tested-by: Zi Yan  # for x86_64 and arm64
Tested-by: Jonathan Cameron  [arm64 + CXL via QEMU]
Acked-by: Dan Williams 
Cc: Alexander Gordeev 
Cc: Andreas Larsson 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Catalin Marinas 
Cc: Christophe Leroy 
Cc: Dave Hansen 
Cc: Davidlohr Bueso 
Cc: David S. Miller 
Cc: Greg Kroah-Hartman 
Cc: Heiko Carstens 
Cc: Huacai Chen 
Cc: Ingo Molnar 
Cc: Jiaxun Yang 
Cc: John Paul Adrian Glaubitz 
Cc: Jonathan Corbet 
Cc: Michael Ellerman 
Cc: Palmer Dabbelt 
Cc: Rafael J. Wysocki 
Cc: Rob Herring (Arm) 
Cc: Samuel Holland 
Cc: Thomas Bogendoerfer 
Cc: Thomas Gleixner 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

mm: move internal core VMA manipulation functions to own file

2024-09-02T03:25:54+00:00

This patch introduces vma.c and moves internal core VMA manipulation
functions to this file from mmap.c.

This allows us to isolate VMA functionality in a single place such that we
can create userspace testing code that invokes this functionality in an
environment where we can implement simple unit tests of core
functionality.

This patch ensures that core VMA functionality is explicitly marked as
such by its presence in mm/vma.h.

It also places the header includes required by vma.c in vma_internal.h,
which is simply imported by vma.c.  This makes the VMA functionality
testable, as userland testing code can simply stub out functionality as
required.

Link: https://lkml.kernel.org/r/c77a6aafb4c42aaadb8e7271a853658cbdca2e22.1722251717.git.lorenzo.stoakes@oracle.com
Signed-off-by: Lorenzo Stoakes 
Reviewed-by: Vlastimil Babka 
Reviewed-by: Liam R. Howlett 
Cc: Alexander Viro 
Cc: Brendan Higgins 
Cc: Christian Brauner 
Cc: David Gow 
Cc: Eric W. Biederman 
Cc: Jan Kara 
Cc: Kees Cook 
Cc: Matthew Wilcox (Oracle) 
Cc: Rae Moar 
Cc: SeongJae Park 
Cc: Shuah Khan 
Cc: Suren Baghdasaryan 
Cc: Pengfei Xu 
Signed-off-by: Andrew Morton

shmem_quota: build the object file conditionally to the config option

2024-09-02T03:25:45+00:00

Initially I added shmem-quota to obj-y, move it to the correct place and
remove the unneeded full file #ifdef

Link: https://lkml.kernel.org/r/20240717063737.910840-1-cem@kernel.org
Signed-off-by: Carlos Maiolino 
Suggested-by: Aristeu Rozanski 
Reviewed-by: Jan Kara 
Cc: Christian Brauner 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton

mm: memcg: put cgroup v1-specific code under a config option

2024-07-05T01:05:54+00:00

Put legacy cgroup v1 memory controller code under a new CONFIG_MEMCG_V1
config option.  The option is turned off by default.  Nobody except those
who are still using cgroup v1 should turn it on.

If the option is not set, memory controller can still be mounted under
cgroup v1, but none of memcg-specific control files are present.

Please note, that not all cgroup v1's memory controller code is guarded
yet (but most of it), it's a subject for some follow-up work.

Thanks to Michal Hocko for providing a better Kconfig option description.

[roman.gushchin@linux.dev: better config option description provided by Michal]
  Link: https://lkml.kernel.org/r/ZnxXNtvqllc9CDoo@google.com
Link: https://lkml.kernel.org/r/20240625005906.106920-14-roman.gushchin@linux.dev
Signed-off-by: Roman Gushchin 
Acked-by: Michal Hocko 
Acked-by: Shakeel Butt 
Cc: Johannes Weiner 
Cc: Matthew Wilcox (Oracle) 
Cc: Muchun Song 
Signed-off-by: Andrew Morton

mm: memcg: introduce memcontrol-v1.c

2024-07-05T01:05:51+00:00

Patch series "mm: memcg: separate legacy cgroup v1 code and put under
config option", v2.

Cgroups v2 have been around for a while and many users have fully adopted
them, so they never use cgroups v1 features and functionality.  Yet they
have to "pay" for the cgroup v1 support anyway:
1) the kernel binary contains an unused cgroup v1 code,
2) some code paths have additional checks which are not needed,
3) some common structures like task_struct and mem_cgroup contain unused
   cgroup v1-specific members.

Cgroup v1's memory controller has a number of features that are not
supported by cgroup v2 and their implementation is pretty much self
contained.  Most notably, these features are: soft limit reclaim, oom
handling in userspace, complicated event notification system, charge
migration.  Cgroup v1-specific code in memcontrol.c is close to 4k lines
in size and it's intervened with generic and cgroup v2-specific code. 
It's a burden on developers and maintainers.

This patchset aims to solve these problems by:
1) moving cgroup v1-specific memcg code to the new mm/memcontrol-v1.c file,
2) putting definitions shared by memcontrol.c and memcontrol-v1.c into the
   mm/memcontrol-v1.h header,
3) introducing the CONFIG_MEMCG_V1 config option, turned off by default,
4) making memcontrol-v1.c to compile only if CONFIG_MEMCG_V1 is set.

If CONFIG_MEMCG_V1 is not set, cgroup v1 memory controller is still available
for mounting, however no memory-specific control knobs are present.

This patch (of 14):


This patch introduces the mm/memcontrol-v1.c source file which will be
used for all legacy (cgroup v1) memory cgroup code.  It also introduces
mm/memcontrol-v1.h to keep declarations shared between mm/memcontrol.c and
mm/memcontrol-v1.c.

As of now, let's compile it if CONFIG_MEMCG is set, similar to
mm/memcontrol.c.  Later on it can be switched to use a separate config
option, so that the legacy code won't be compiled if not required.

Link: https://lkml.kernel.org/r/20240625005906.106920-1-roman.gushchin@linux.dev
Link: https://lkml.kernel.org/r/20240625005906.106920-2-roman.gushchin@linux.dev
Signed-off-by: Roman Gushchin 
Acked-by: Michal Hocko 
Acked-by: Shakeel Butt 
Cc: Johannes Weiner 
Cc: Matthew Wilcox (Oracle) 
Cc: Muchun Song 
Cc: Roman Gushchin 
Signed-off-by: Andrew Morton

mseal: add mseal syscall

2024-05-24T02:40:26+00:00

The new mseal() is an syscall on 64 bit CPU, and with following signature:

int mseal(void addr, size_t len, unsigned long flags)
addr/len: memory range.
flags: reserved.

mseal() blocks following operations for the given memory range.

1> Unmapping, moving to another location, and shrinking the size,
   via munmap() and mremap(), can leave an empty space, therefore can
   be replaced with a VMA with a new set of attributes.

2> Moving or expanding a different VMA into the current location,
   via mremap().

3> Modifying a VMA via mmap(MAP_FIXED).

4> Size expansion, via mremap(), does not appear to pose any specific
   risks to sealed VMAs. It is included anyway because the use case is
   unclear. In any case, users can rely on merging to expand a sealed VMA.

5> mprotect() and pkey_mprotect().

6> Some destructive madvice() behaviors (e.g. MADV_DONTNEED) for anonymous
   memory, when users don't have write permission to the memory. Those
   behaviors can alter region contents by discarding pages, effectively a
   memset(0) for anonymous memory.

Following input during RFC are incooperated into this patch:

Jann Horn: raising awareness and providing valuable insights on the
destructive madvise operations.
Linus Torvalds: assisting in defining system call signature and scope.
Liam R. Howlett: perf optimization.
Theo de Raadt: sharing the experiences and insight gained from
  implementing mimmutable() in OpenBSD.

Finally, the idea that inspired this patch comes from Stephen Röttger's
work in Chrome V8 CFI.

[jeffxu@chromium.org: add branch prediction hint, per Pedro]
  Link: https://lkml.kernel.org/r/20240423192825.1273679-2-jeffxu@chromium.org
Link: https://lkml.kernel.org/r/20240415163527.626541-3-jeffxu@chromium.org
Signed-off-by: Jeff Xu 
Reviewed-by: Kees Cook 
Reviewed-by: Liam R. Howlett 
Cc: Pedro Falcato 
Cc: Dave Hansen 
Cc: Greg Kroah-Hartman 
Cc: Guenter Roeck 
Cc: Jann Horn 
Cc: Jeff Xu 
Cc: Jonathan Corbet 
Cc: Jorge Lucangeli Obes 
Cc: Linus Torvalds 
Cc: Matthew Wilcox (Oracle) 
Cc: Muhammad Usama Anjum 
Cc: Pedro Falcato 
Cc: Stephen Röttger 
Cc: Suren Baghdasaryan 
Cc: Amer Al Shanawany 
Cc: Javier Carrasco 
Cc: Shuah Khan 
Signed-off-by: Andrew Morton