diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-27 12:23:17 -1000 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-27 12:23:17 -1000 |
| commit | 8a36f53fd5ec3eff01f1b637d42cbdc3821d0493 (patch) | |
| tree | 4813a0736eeb40ee58db667445d6770eaa47ca52 | |
| parent | 7d1f84f1b94e79cb95d974ab3ec54cb3a4d64f5b (diff) | |
| parent | 183895979c441e52dbd651b388553b09eaa35cfa (diff) | |
Merge tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
228 files changed, 4606 insertions, 2415 deletions
diff --git a/.clang-format b/.clang-format index 5ef5743b77c9..395292ab2ac6 100644 --- a/.clang-format +++ b/.clang-format @@ -282,6 +282,7 @@ ForEachMacros: - 'for_each_dpcm_fe' - 'for_each_drhd_unit' - 'for_each_dss_dev' + - 'for_each_early_section_nr' - 'for_each_efi_memory_desc' - 'for_each_efi_memory_desc_in_map' - 'for_each_element' @@ -403,7 +404,6 @@ ForEachMacros: - 'for_each_possible_cpu_wrap' - 'for_each_present_blessed_reg' - 'for_each_present_cpu' - - 'for_each_present_section_nr' - 'for_each_prime_number' - 'for_each_prime_number_from' - 'for_each_probe_cache_entry' diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-damon b/Documentation/ABI/testing/sysfs-kernel-mm-damon index 55df688ea596..4acb2015c2f4 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-damon +++ b/Documentation/ABI/testing/sysfs-kernel-mm-damon @@ -375,6 +375,12 @@ Contact: SJ Park <sj@kernel.org> Description: Writing to and reading from this file sets and gets the quota auto-tuning goal metric. +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/complement +Date: Sep 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing to and reading from this file sets and gets whether to + use the complement value of the quota auto-tuning goal metric. + What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/target_value Date: Nov 2023 Contact: SJ Park <sj@kernel.org> diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst index ba4725544856..3d43954c7046 100644 --- a/Documentation/admin-guide/mm/damon/usage.rst +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -98,7 +98,8 @@ comma (","). │ │ │ │ │ │ │ fail_charge_num,fail_charge_denom │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil │ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals - │ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path + │ │ │ │ │ │ │ │ │ 0/target_metric,complement,target_value, + │ │ │ │ │ │ │ │ │ current_value,nid,path │ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low │ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,damon_target_idx,min,max @@ -492,12 +493,12 @@ number (``N``) to the file creates the number of child directories named ``0`` to ``N-1``. Each directory represents each goal and current achievement. Among the multiple feedback, the best one is used. -Each goal directory contains five files, namely ``target_metric``, -``target_value``, ``current_value``, ``nid``, and ``path``. Users can set and -get the five parameters for the quota auto-tuning goals that specified on the -:ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and -reading from each of the files. Because the kernel does not update -``current_value``, reading it only makes sense when ``target_metric`` is +Each goal directory contains six files, namely ``target_metric``, +``complement``, ``target_value``, ``current_value``, ``nid``, and ``path``. +Users can set and get the six parameters for the quota auto-tuning goals that +specified on the :ref:`design doc <damon_design_damos_quotas_auto_tuning>` by +writing to and reading from each of the files. Because the kernel does not +update ``current_value``, reading it only makes sense when ``target_metric`` is ``user_input``. Note that users should further write ``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond directory <sysfs_kdamond>` to pass the feedback to DAMON. diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 3cc15d800be1..2075f1f14a42 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -373,9 +373,9 @@ contains the following attribute files:: free_hugepages surplus_hugepages -The free\_' and surplus\_' attribute files are read-only. They return the number -of free and surplus [overcommitted] huge pages, respectively, on the parent -node. +The ``free_hugepages`` and ``surplus_hugepages`` attribute files are read-only. +They return the number of free and surplus [overcommitted] huge pages, +respectively, on the parent node. The ``nr_hugepages`` attribute returns the total number of huge pages on the specified node. When this attribute is written, the number of persistent huge diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 5f4a056e4dc2..3cb0138e850c 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -539,6 +539,13 @@ watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size. +By default, the kernel sets this value based on the amount of lowmem +and recalculates it on memory hotplug. When Transparent Hugepage (THP) +support is enabled through global or per-size controls, the kernel may +also raise it to help keep pageblocks free and reduce fragmentation for +THP allocations. Once a value is written to this file, the kernel no +longer adjusts it automatically. + Some minimal amount of memory is needed to satisfy PF_MEMALLOC allocations; if you set this to lower than 1024KB, your system will become subtly broken, and prone to deadlock under high loads. diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 26a536ef4cd6..0772525cbc62 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -268,6 +268,24 @@ section:: .. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress +Build and test on nommu target +============================== + +The nommu selftests detect NOMMU at runtime by checking for the ``MmapCopy`` +entry in ``/proc/meminfo``. Ensure procfs is mounted at ``/proc`` before +building and running the tests. + +:: + + $ make ARCH=um O=build kselftest-all TARGETS=nommu # <= build-only + $ make ARCH=um O=build kselftest-install TARGETS=nommu + $ ./build/kselftest/kselftest_install/run_kselftest.sh -p -c nommu + +The C tests report a skip if ``/proc/meminfo`` is unavailable. On a NOMMU +system, the runner needs that file to avoid wrapping the tests in ``timeout`` +before they can report a skip. Mounting procfs is therefore a prerequisite +for running this target through the kselftest runner. + Contributing new tests ====================== diff --git a/Documentation/filesystems/mmap_prepare.rst b/Documentation/filesystems/mmap_prepare.rst index 82c99c95ad85..b5afbb33e5c8 100644 --- a/Documentation/filesystems/mmap_prepare.rst +++ b/Documentation/filesystems/mmap_prepare.rst @@ -164,5 +164,86 @@ pointer. These are: sufficient entries in the page array to cover the entire range of the described VMA. +* mmap_action_map_discontig_kernel_pages() - Maps a discontiguous range of + `struct page` pointers over the VMA. They must span from the start of the VMA, + but may terminate prior to the end (leaving the remainder unmapped). + **NOTE:** The ``action`` field should never normally be manipulated directly, rather you ought to use one of these helpers. + +Discontiguous Actions +===================== + +Some actions can be performed across discontiguous ranges. + +Map kernel pages +---------------- + +To map kernel pages discontiguously, you must provide hooks using ``struct +discontig_kernel_page_ops``: + +.. code-block:: C + + struct discontig_kernel_page_ops { + int (*init)(void *vm_private_data, void **private); + int (*get)(struct discontig_kernel_page_state *state); + }; + +The ``init`` hook is optional and allows state to be established before the +operation starts, for instance taking a reference count. Nothing is invoked +after the operation, so ``init`` must not leave locks held, and state that must +be released once the mapping goes away should be released in +``vm_ops->close``. + +The ``init`` hook, if provided, is invoked prior to the operation starting. It +may update what is pointed to by ``vm_private_data`` and/or ``private``. If an +error is returned, then the operation is aborted. The ``private`` field can be +reassigned. + +**NOTE:** The operation may sleep between invocations of ``get``, so locks +needed to stabilise state must be taken and released within each hook. + +The ``get`` handler is the key means through which the operation is +executed. The current state of the operation is provided through ``struct +discontig_kernel_page_state``: + +.. code-block:: C + + struct discontig_kernel_page_state { + /* Map state. */ + unsigned long start; /* Start address of VMA. */ + unsigned long end; /* End address of VMA. */ + unsigned long addr; /* The current address to be mapped. */ + pgoff_t pgoff; /* The current pgoff to be mapped. */ + unsigned long nr_pages_mapped; /* The number of pages mapped. */ + unsigned long nr_pages_remain; /* The number of pages remaining. */ + + /* User-defined state. */ + void *vm_private_data; /* VMA private data. */ + void *private; /* Mapping private data. */ + + /* Users should not touch these, use discontig_kernel_map_*() helpers. */ + ... internal fields ... + }; + +With ``private`` being an additional user-controllable state variable, +initialised via ``mmap_action_map_discontig_kernel_pages()``, and +``vm_private_data`` being equal to the ``desc->private_data`` field set in +the ``mmap_prepare()`` hook. + +In the ``get`` hook, the user must choose how to map kernel pages: + +* ``discontig_kernel_map_abort()`` - Call this to abort the operation, whatever + has been mapped so far will be retained, the rest of the mapping will SIGBUS + if accessed. +* ``discontig_kernel_map_page()`` - Maps a single page, correctly handling + compound pages (if the compound page is bigger than the remaining pages in the + VMA, then only those pages that fit will be mapped). For a compound page, the + head page must be passed. +* ``discontig_kernel_map_pages()`` - Map an array of pages of a specified + size. As with a compound page, if more pages are specified than remain in + the VMA then only those that fit will be mapped. + +If an error arises after ``init`` succeeded, the core unmaps the VMA, invoking +``vm_ops->close`` if set, which is therefore the place to release any state +that ``init`` established. diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/design.rst index e82390e77a70..5cd651b1b7aa 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -702,10 +702,10 @@ There are two such tuning algorithms that users can select as they need. that the zero quota is a valid quota, and therefore ``qt_exceeds`` :ref:`stat <damon_design_damos_stat>` will keep increasing in this case. -The goal can be specified with five parameters, namely ``target_metric``, -``target_value``, ``current_value``, ``nid`` and ``path``. The auto-tuning -mechanism tries to make ``current_value`` of ``target_metric`` be same to -``target_value``. +The goal can be specified with six parameters, namely ``target_metric``, +``complement``, ``target_value``, ``current_value``, ``nid`` and ``path``. The +auto-tuning mechanism tries to make ``current_value`` of ``complement``-ed +``target_metric`` be same to ``target_value``. - ``user_input``: User-provided value. Users could use any metric that they has interest in for the value. Use space main workload's latency or @@ -733,6 +733,13 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to - ``hugepage_mem_bp``: Total huge page to total used memory ratio in bp (1/10,000). +``complement`` is a boolean parameter that determines whether to use +complemented value of the target metric. For example, if ``complement`` is set +and target metric is ``active_mem_bp``, it is effectively same to +``inactive_mem_bp``. ``complement`` is no-op when the target metric type is +``user_input``. For the metric type, the user should be able to emit +complemented metric values on their own. + ``nid`` is optionally required for ``node_mem_used_bp``, ``node_mem_free_bp``, ``node_memcg_used_bp``, ``node_memcg_free_bp`` and ``node_eligible_mem_bp`` to point the specific NUMA node. diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 78231995e490..04b649818cc1 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -819,9 +819,17 @@ All this is achieved by the use of per-mm and per-VMA sequence counts, which are used in order to reduce complexity, especially for operations which write-lock multiple VMAs at once. -If the mm sequence count, :c:member:`!mm->mm_lock_seq` is equal to the VMA -sequence count :c:member:`!vma->vm_lock_seq` then the VMA is write-locked. If -they differ, then it is not. +The mm sequence count :c:member:`!mm->mm_lock_seq` is incremented on both mmap +write lock and unlock, so it is odd while the mmap write lock is held. If it is +equal to the VMA sequence count :c:member:`!vma->vm_lock_seq`, and that value +is odd, then the VMA is write-locked. If they differ, then it is not. + +A VMA's sequence count is not updated on unlock, so after 2^31 write lock/unlock +cycles the mm sequence count would wrap and match stale VMA values again. To +prevent this, when the mm sequence count wraps to zero, +:c:func:`!vma_end_write_all` invokes :c:func:`!__mm_reset_vma_seqcounts` to +reset every VMA's sequence count to zero, an even value which can never +indicate a write lock. Each time the mmap write lock is released in :c:func:`!mmap_write_unlock` or :c:func:`!mmap_write_downgrade`, :c:func:`!vma_end_write_all` is invoked which diff --git a/MAINTAINERS b/MAINTAINERS index b1abfcd343ea..801f964899ba 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12337,7 +12337,7 @@ F: Documentation/admin-guide/mm/hugetlbpage.rst F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ -F: include/linux/hugetlb.h +F: include/linux/hugetlb* F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c @@ -17496,9 +17496,6 @@ F: mm/mmzone.c F: mm/pagewalk.c F: mm/pgtable-generic.c F: mm/ptdump.c -F: mm/sparse-vmemmap.c -F: mm/sparse.c -F: mm/sparse.h F: mm/util.c F: mm/vmpressure.c F: mm/vmstat.c @@ -17546,6 +17543,21 @@ F: include/trace/events/ksm.h F: mm/ksm.c F: mm/mm_slot.h +MEMORY MANAGEMENT - MEMORY MAP +M: Muchun Song <muchun.song@linux.dev> +M: Mike Rapoport <rppt@kernel.org> +M: David Hildenbrand <david@kernel.org> +L: linux-mm@kvack.org +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/mm/core.git memmap +F: Documentation/mm/memory-model.rst +F: arch/*/include/asm/sparsemem.h +F: include/asm-generic/memory_model.h +F: mm/sparse-vmemmap.c +F: mm/sparse.c +F: mm/sparse.h +K: (?i:vmemmap) + MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU) M: Andrew Morton <akpm@linux-foundation.org> R: Kairui Song <kasong@tencent.com> @@ -17607,6 +17619,8 @@ F: Documentation/admin-guide/mm/ F: Documentation/core-api/memory-allocation.rst F: Documentation/core-api/mm-api.rst F: Documentation/mm/ +X: Documentation/admin-guide/mm/damon/ +X: Documentation/mm/damon/ F: mm/char-mem.c F: include/linux/cma.h F: include/linux/dmapool.h @@ -17786,6 +17800,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/arch/arm/Kconfig.debug b/arch/arm/Kconfig.debug index 366f162e147d..abcf14f10276 100644 --- a/arch/arm/Kconfig.debug +++ b/arch/arm/Kconfig.debug @@ -17,7 +17,7 @@ config ARM_PTDUMP_DEBUGFS kernel. If in doubt, say "N" -config ARM_DEBUG_WX +config ARM_CHECK_WX bool "Warn on W+X mappings at boot" depends on MMU select ARM_PTDUMP_CORE diff --git a/arch/arm/configs/aspeed_g4_defconfig b/arch/arm/configs/aspeed_g4_defconfig index f86dd4ce7d0d..2c9d5a644ae9 100644 --- a/arch/arm/configs/aspeed_g4_defconfig +++ b/arch/arm/configs/aspeed_g4_defconfig @@ -249,7 +249,7 @@ CONFIG_DEBUG_INFO_REDUCED=y CONFIG_GDB_SCRIPTS=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y CONFIG_SCHED_STACK_END_CHECK=y CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 diff --git a/arch/arm/configs/aspeed_g5_defconfig b/arch/arm/configs/aspeed_g5_defconfig index 45b937419dbd..1327a09e163a 100644 --- a/arch/arm/configs/aspeed_g5_defconfig +++ b/arch/arm/configs/aspeed_g5_defconfig @@ -300,7 +300,7 @@ CONFIG_DEBUG_INFO_REDUCED=y CONFIG_GDB_SCRIPTS=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y CONFIG_SCHED_STACK_END_CHECK=y CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 diff --git a/arch/arm/configs/shmobile_defconfig b/arch/arm/configs/shmobile_defconfig index cb7eddd4eb22..58059f943a9b 100644 --- a/arch/arm/configs/shmobile_defconfig +++ b/arch/arm/configs/shmobile_defconfig @@ -226,4 +226,4 @@ CONFIG_CMA_SIZE_MBYTES=64 CONFIG_PRINTK_TIME=y CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_FS=y -CONFIG_ARM_DEBUG_WX=y +CONFIG_ARM_CHECK_WX=y diff --git a/arch/arm/include/asm/ptdump.h b/arch/arm/include/asm/ptdump.h index 46a4575146ee..3c5245220ecf 100644 --- a/arch/arm/include/asm/ptdump.h +++ b/arch/arm/include/asm/ptdump.h @@ -32,10 +32,10 @@ void ptdump_check_wx(void); #endif /* CONFIG_ARM_PTDUMP_CORE */ -#ifdef CONFIG_ARM_DEBUG_WX -#define arm_debug_checkwx() ptdump_check_wx() +#ifdef CONFIG_ARM_CHECK_WX +#define arm_pgtable_checkwx() ptdump_check_wx() #else -#define arm_debug_checkwx() do { } while (0) +#define arm_pgtable_checkwx() do { } while (0) #endif #endif /* __ASM_PTDUMP_H */ diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c index 0cc1bf04686d..c515faf22eeb 100644 --- a/arch/arm/mm/init.c +++ b/arch/arm/mm/init.c @@ -403,7 +403,7 @@ static int __mark_rodata_ro(void *unused) void mark_rodata_ro(void) { stop_machine(__mark_rodata_ro, NULL, NULL); - arm_debug_checkwx(); + arm_pgtable_checkwx(); } #else diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 292392ded42f..cfef22aee709 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -11,7 +11,7 @@ config ARM64 select ACPI_MCFG if (ACPI && PCI) select ACPI_SPCR_TABLE if ACPI select ACPI_PPTT if ACPI - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_BINFMT_ELF_EXTRA_PHDRS select ARCH_BINFMT_ELF_STATE select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e89ec5f4787b..763c5a411d64 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -387,52 +387,13 @@ static inline pte_t __ptep_get(pte_t *ptep) extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(pteval_t old, pteval_t new); -/* - * PTE bits configuration in the presence of hardware Dirty Bit Management - * (PTE_WRITE == PTE_DBM): - * - * Dirty Writable | PTE_RDONLY PTE_WRITE PTE_DIRTY (sw) - * 0 0 | 1 0 0 - * 0 1 | 1 1 0 - * 1 0 | 1 0 1 - * 1 1 | 0 1 x - * - * When hardware DBM is not present, the software PTE_DIRTY bit is updated via - * the page fault mechanism. Checking the dirty status of a pte becomes: - * - * PTE_DIRTY || (PTE_WRITE && !PTE_RDONLY) - */ - -static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, - pte_t pte) +#ifdef CONFIG_DEBUG_VM +void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte); +#else +static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) { - pte_t old_pte; - - if (!IS_ENABLED(CONFIG_DEBUG_VM)) - return; - - old_pte = __ptep_get(ptep); - - if (!pte_valid(old_pte) || !pte_valid(pte)) - return; - if (mm != current->active_mm && atomic_read(&mm->mm_users) <= 1) - return; - - /* - * Check for potential race with hardware updates of the pte - * (__ptep_set_access_flags safely changes valid ptes without going - * through an invalid entry). - */ - VM_WARN_ONCE(!pte_young(pte), - "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); - VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), - "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); - VM_WARN_ONCE(!pgattr_change_is_safe(pte_val(old_pte), pte_val(pte)), - "%s: unsafe attribute change: 0x%016llx -> 0x%016llx", - __func__, pte_val(old_pte), pte_val(pte)); } +#endif /* CONFIG_DEBUG_VM */ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 745cb86f9e04..5b9ff769729e 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1587,7 +1587,7 @@ transparent_hugepage_adjust(const struct kvm_s2_fault_desc *s2fd, kvm_pfn_t *pfn static int get_vma_page_shift(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return huge_page_shift(hstate_vma(vma)); return PAGE_SHIFT; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index d82118afdf4b..b22736853fcd 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -131,6 +131,7 @@ static inline unsigned long mm_to_pgd_phys(struct mm_struct *mm) */ static void show_pte(unsigned long addr) { + char pxd_str[PTVAL_STR_MAX]; struct mm_struct *mm; pgd_t *pgdp; pgd_t pgd; @@ -160,7 +161,8 @@ static void show_pte(unsigned long addr) pgdp = pgd_offset(mm, addr); pgd = READ_ONCE(*pgdp); - pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd)); + ptval_to_str(pxd_str, pgd_val(pgd)); + pr_alert("[%016lx] pgd=%s", addr, pxd_str); do { p4d_t *p4dp, p4d; @@ -173,19 +175,22 @@ static void show_pte(unsigned long addr) p4dp = p4d_offset_lockless(pgdp, pgd, addr); p4d = READ_ONCE(*p4dp); - pr_cont(", p4d=%016llx", p4d_val(p4d)); + ptval_to_str(pxd_str, p4d_val(p4d)); + pr_cont(", p4d=%s", pxd_str); if (p4d_none(p4d) || p4d_bad(p4d)) break; pudp = pud_offset_lockless(p4dp, p4d, addr); pud = READ_ONCE(*pudp); - pr_cont(", pud=%016llx", pud_val(pud)); + ptval_to_str(pxd_str, pud_val(pud)); + pr_cont(", pud=%s", pxd_str); if (pud_none(pud) || pud_bad(pud)) break; pmdp = pmd_offset_lockless(pudp, pud, addr); pmd = READ_ONCE(*pmdp); - pr_cont(", pmd=%016llx", pmd_val(pmd)); + ptval_to_str(pxd_str, pmd_val(pmd)); + pr_cont(", pmd=%s", pxd_str); if (pmd_none(pmd) || pmd_bad(pmd)) break; @@ -194,7 +199,8 @@ static void show_pte(unsigned long addr) break; pte = __ptep_get(ptep); - pr_cont(", pte=%016llx", pte_val(pte)); + ptval_to_str(pxd_str, pte_val(pte)); + pr_cont(", pte=%s", pxd_str); pte_unmap(ptep); } while(0); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d4384131e10d..140520b351e2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -2393,3 +2393,51 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val) return 0; } #endif + +/* + * PTE bits configuration in the presence of hardware Dirty Bit Management + * (PTE_WRITE == PTE_DBM): + * + * Dirty Writable | PTE_RDONLY PTE_WRITE PTE_DIRTY (sw) + * 0 0 | 1 0 0 + * 0 1 | 1 1 0 + * 1 0 | 1 0 1 + * 1 1 | 0 1 x + * + * When hardware DBM is not present, the software PTE_DIRTY bit is updated via + * the page fault mechanism. Checking the dirty status of a pte becomes: + * + * PTE_DIRTY || (PTE_WRITE && !PTE_RDONLY) + */ +#ifdef CONFIG_DEBUG_VM +void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte) +{ + char pte_str_old[PTVAL_STR_MAX]; + char pte_str[PTVAL_STR_MAX]; + pte_t old_pte; + + old_pte = __ptep_get(ptep); + + if (!pte_valid(old_pte) || !pte_valid(pte)) + return; + if (mm != current->active_mm && atomic_read(&mm->mm_users) <= 1) + return; + + /* + * Check for potential race with hardware updates of the pte + * (__ptep_set_access_flags safely changes valid ptes without going + * through an invalid entry). + */ + ptval_to_str(pte_str, pte_val(pte)); + ptval_to_str(pte_str_old, pte_val(old_pte)); + VM_WARN_ONCE(!pte_young(pte), + "%s: racy access flag clearing: %s -> %s", + __func__, pte_str_old, pte_str); + VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), + "%s: racy dirty state clearing: %s -> %s", + __func__, pte_str_old, pte_str); + VM_WARN_ONCE(!pgattr_change_is_safe(pte_val(old_pte), pte_val(pte)), + "%s: unsafe attribute change: %s -> %s", + __func__, pte_str_old, pte_str); +} +#endif /* CONFIG_DEBUG_VM */ diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1b5ab1963be2..c3b6cd655606 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -130,7 +130,7 @@ config PPC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX if STRICT_KERNEL_RWX + select ARCH_HAS_CHECK_WX if STRICT_KERNEL_RWX select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES select ARCH_HAS_DMA_OPS if PPC64 diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig index e0b7901add11..f40e74456cf7 100644 --- a/arch/powerpc/configs/ppc64_defconfig +++ b/arch/powerpc/configs/ppc64_defconfig @@ -393,7 +393,7 @@ CONFIG_MAGIC_SYSRQ=y CONFIG_PAGE_OWNER=y CONFIG_PAGE_POISONING=y CONFIG_DEBUG_RODATA_TEST=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_DEBUG_STACK_USAGE=y CONFIG_DEBUG_VM=y # CONFIG_DEBUG_VM_PGTABLE is not set diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index a3332f32ffb2..44868b62b14e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1126,7 +1126,10 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in pud_t *pud; pmd_t *pmd; pte_t *pte; + unsigned long pfn = page_to_pfn((struct page *)start); + if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn))) + return vmemmap_populate_compound_pages(pfn, start, end, node, NULL); /* * If altmap is present, Make sure we align the start vmemmap addr * to PAGE_SIZE so that we calculate the correct start_pfn in diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 7de5760164a9..b4603a98224b 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -627,7 +627,7 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd { #ifdef CONFIG_HUGETLB_PAGE /* need the return fix for nohash.c */ - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__local_flush_hugetlb_page(vma, vmaddr); #endif radix__local_flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); @@ -945,7 +945,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, void radix__flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { #ifdef CONFIG_HUGETLB_PAGE - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__flush_hugetlb_page(vma, vmaddr); #endif radix__flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); @@ -1113,7 +1113,7 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, { #ifdef CONFIG_HUGETLB_PAGE - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return radix__flush_hugetlb_tlb_range(vma, start, end); #endif diff --git a/arch/powerpc/mm/nohash/e500_hugetlbpage.c b/arch/powerpc/mm/nohash/e500_hugetlbpage.c index a134d28a0e4d..b87623f04be5 100644 --- a/arch/powerpc/mm/nohash/e500_hugetlbpage.c +++ b/arch/powerpc/mm/nohash/e500_hugetlbpage.c @@ -180,7 +180,7 @@ book3e_hugetlb_preload(struct vm_area_struct *vma, unsigned long ea, pte_t pte) */ void __update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) book3e_hugetlb_preload(vma, address, *ptep); } diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c index 0a650742f3a0..07a2db16c2b1 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb.c @@ -278,7 +278,7 @@ void __flush_tlb_page(struct mm_struct *mm, unsigned long vmaddr, void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { #ifdef CONFIG_HUGETLB_PAGE - if (vma && is_vm_hugetlb_page(vma)) + if (vma && vma_is_hugetlb(vma)) flush_hugetlb_page(vma, vmaddr); #endif diff --git a/arch/powerpc/mm/ptdump/ptdump.c b/arch/powerpc/mm/ptdump/ptdump.c index 0d499aebee72..3451351b756b 100644 --- a/arch/powerpc/mm/ptdump/ptdump.c +++ b/arch/powerpc/mm/ptdump/ptdump.c @@ -191,7 +191,7 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr) if (!pte_write(pte) || !pte_exec(pte)) return; - WARN_ONCE(IS_ENABLED(CONFIG_DEBUG_WX), + WARN_ONCE(IS_ENABLED(CONFIG_CHECK_WX), "powerpc/mm: Found insecure W+X mapping at address %p/%pS\n", (void *)st->start_address, (void *)st->start_address); diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index f35b7b1584ec..6c9d61d83d71 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -29,7 +29,7 @@ config RISCV select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_DELAY_TIMER select ARCH_HAS_ELF_CORE_EFLAGS if BINFMT_ELF && ELF_CORE select ARCH_HAS_FAST_MULTIPLIER diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 4c8fc6845503..d48f90140841 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -989,7 +989,14 @@ static inline bool pte_user_accessible_page(struct mm_struct *mm, unsigned long static inline bool pmd_user_accessible_page(struct mm_struct *mm, unsigned long addr, pmd_t pmd) { - return pmd_leaf(pmd) && pmd_user(pmd); + /* + * page_table_check() must ignore THP split invalidation entries created by + * pmd_mkinvalid(). These retain _PAGE_LEAF so pmd_present()/pmd_leaf() stay + * true during the split, but they no longer describe a user-accessible + * mapping once both _PAGE_PRESENT and _PAGE_PROT_NONE are cleared. + */ + return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) && + (pmd_val(pmd) & _PAGE_LEAF) && pmd_user(pmd); } static inline bool pud_user_accessible_page(struct mm_struct *mm, unsigned long addr, pud_t pud) diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 3e955d808743..371fccaf9df0 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -665,7 +665,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, return -EFAULT; } - is_hugetlb = is_vm_hugetlb_page(vma); + is_hugetlb = vma_is_hugetlb(vma); if (is_hugetlb) vma_pageshift = huge_page_shift(hstate_vma(vma)); else diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 962db300a166..a74a7d5258aa 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -149,7 +149,7 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, { unsigned long stride_size; - if (!is_vm_hugetlb_page(vma)) { + if (!vma_is_hugetlb(vma)) { stride_size = PAGE_SIZE; } else { stride_size = huge_page_size(hstate_vma(vma)); diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 2c46cfc526a9..7138d923d86f 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -92,7 +92,7 @@ config S390 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_OPS if PCI select ARCH_HAS_ELF_RANDOMIZE diff --git a/arch/s390/configs/debug_defconfig b/arch/s390/configs/debug_defconfig index ac30bf5d0b31..d44674b63715 100644 --- a/arch/s390/configs/debug_defconfig +++ b/arch/s390/configs/debug_defconfig @@ -855,7 +855,7 @@ CONFIG_DEBUG_PAGEALLOC=y CONFIG_SLUB_DEBUG_ON=y CONFIG_PAGE_OWNER=y CONFIG_DEBUG_RODATA_TEST=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_PTDUMP_DEBUGFS=y CONFIG_DEBUG_OBJECTS=y CONFIG_DEBUG_OBJECTS_SELFTEST=y diff --git a/arch/s390/configs/defconfig b/arch/s390/configs/defconfig index a02d8bf57297..e00e252d4694 100644 --- a/arch/s390/configs/defconfig +++ b/arch/s390/configs/defconfig @@ -834,7 +834,7 @@ CONFIG_DEBUG_INFO_DWARF4=y CONFIG_GDB_SCRIPTS=y CONFIG_DEBUG_SECTION_MISMATCH=y CONFIG_MAGIC_SYSRQ=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_PTDUMP_DEBUGFS=y CONFIG_DEBUG_MEMORY_INIT=y CONFIG_PANIC_ON_OOPS=y diff --git a/arch/s390/mm/dump_pagetables.c b/arch/s390/mm/dump_pagetables.c index 89badbe72ae7..a23a0bd4d8a8 100644 --- a/arch/s390/mm/dump_pagetables.c +++ b/arch/s390/mm/dump_pagetables.c @@ -86,7 +86,7 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr) */ if (addr == PAGE_SIZE && (nospec_uses_trampoline() || !cpu_has_bear())) return; - WARN_ONCE(IS_ENABLED(CONFIG_DEBUG_WX), + WARN_ONCE(IS_ENABLED(CONFIG_CHECK_WX), "s390/mm: Found insecure W+X mapping at address %pS\n", (void *)st->start_address); st->wx_pages += (addr - st->start_address) / PAGE_SIZE; diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index ff63ffb1dbd2..3f6783b93e67 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -102,7 +102,7 @@ __context_unsafe(/* pte_unmap_unlock() not instrumented */) /* Find the vm address for the guest address */ vma = vma_lookup(mm, vmaddr); - if (!vma || is_vm_hugetlb_page(vma)) + if (!vma || vma_is_hugetlb(vma)) return; /* Get pointer to the page table entry */ @@ -139,7 +139,7 @@ void gmap_helper_discard(struct mm_struct *mm, unsigned long vmaddr, unsigned lo vma = find_vma_intersection(mm, vmaddr, end); if (!vma) return; - if (!is_vm_hugetlb_page(vma)) + if (!vma_is_hugetlb(vma)) zap_vma_range(vma, vmaddr, min(end, vma->vm_end) - vmaddr); vmaddr = vma->vm_end; } @@ -247,7 +247,7 @@ static int __gmap_helper_unshare_zeropages(struct mm_struct *mm) * proof to catch unexpected zeropages in other mappings and * fail. */ - if ((vma->vm_flags & VM_PFNMAP) || is_vm_hugetlb_page(vma)) + if ((vma->vm_flags & VM_PFNMAP) || vma_is_hugetlb(vma)) continue; addr = vma->vm_start; diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index 8eceb80dd83f..5512f446eed1 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -108,9 +108,6 @@ config SYS_SUPPORTS_APM_EMULATION config SYS_SUPPORTS_SMP bool -config SYS_SUPPORTS_NUMA - bool - config STACKTRACE_SUPPORT def_bool y @@ -199,7 +196,6 @@ config CPU_SHX3 bool select DMA_COHERENT select SYS_SUPPORTS_SMP - select SYS_SUPPORTS_NUMA config ARCH_SHMOBILE bool @@ -392,7 +388,6 @@ config CPU_SUBTYPE_SH7723 select CPU_SH4A select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE select SYS_SUPPORTS_SH_CMT select PINCTRL help @@ -403,7 +398,6 @@ config CPU_SUBTYPE_SH7724 select CPU_SH4A select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE select SYS_SUPPORTS_SH_CMT select PINCTRL help @@ -444,8 +438,6 @@ config CPU_SUBTYPE_SH7785 bool "Support SH7785 processor" select CPU_SH4A select CPU_SHX2 - select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select PINCTRL config CPU_SUBTYPE_SH7786 @@ -480,8 +472,6 @@ config CPU_SUBTYPE_SH7722 select CPU_SH4AL_DSP select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select SYS_SUPPORTS_SH_CMT select PINCTRL @@ -490,8 +480,6 @@ config CPU_SUBTYPE_SH7366 select CPU_SH4AL_DSP select CPU_SHX2 select ARCH_SHMOBILE - select ARCH_SPARSEMEM_ENABLE - select SYS_SUPPORTS_NUMA select SYS_SUPPORTS_SH_CMT endchoice diff --git a/arch/sh/configs/migor_defconfig b/arch/sh/configs/migor_defconfig index bde70f3bf857..c2c0600c66a4 100644 --- a/arch/sh/configs/migor_defconfig +++ b/arch/sh/configs/migor_defconfig @@ -9,7 +9,6 @@ CONFIG_MODULES=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7722=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y # CONFIG_MIGRATION is not set CONFIG_SH_MIGOR=y # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/configs/sdk7786_defconfig b/arch/sh/configs/sdk7786_defconfig index d40be415dcb9..a7748fa3dc3e 100644 --- a/arch/sh/configs/sdk7786_defconfig +++ b/arch/sh/configs/sdk7786_defconfig @@ -41,7 +41,6 @@ CONFIG_CPU_SUBTYPE_SH7786=y CONFIG_MEMORY_START=0x40000000 CONFIG_MEMORY_SIZE=0x20000000 CONFIG_PMB=y -CONFIG_NUMA=y CONFIG_HUGETLB_PAGE_SIZE_64MB=y CONFIG_KSM=y CONFIG_SH_STORE_QUEUES=y diff --git a/arch/sh/configs/se7722_defconfig b/arch/sh/configs/se7722_defconfig index 85b8eb013b79..2908611b118e 100644 --- a/arch/sh/configs/se7722_defconfig +++ b/arch/sh/configs/se7722_defconfig @@ -10,7 +10,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7722=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y CONFIG_SH_STORE_QUEUES=y CONFIG_SH_7722_SOLUTION_ENGINE=y # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/configs/shx3_defconfig b/arch/sh/configs/shx3_defconfig index d7496ea9a9fc..249dbd8c7c99 100644 --- a/arch/sh/configs/shx3_defconfig +++ b/arch/sh/configs/shx3_defconfig @@ -29,7 +29,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SHX3=y CONFIG_MEMORY_START=0x0c000000 -CONFIG_NUMA=y CONFIG_PAGE_SIZE_64KB=y CONFIG_SH_STORE_QUEUES=y CONFIG_SH_X3PROTO=y diff --git a/arch/sh/configs/ul2_defconfig b/arch/sh/configs/ul2_defconfig index 00a37944b043..1740a6465280 100644 --- a/arch/sh/configs/ul2_defconfig +++ b/arch/sh/configs/ul2_defconfig @@ -10,7 +10,6 @@ CONFIG_MODULE_UNLOAD=y # CONFIG_BLK_DEV_BSG is not set CONFIG_CPU_SUBTYPE_SH7366=y CONFIG_MEMORY_SIZE=0x01f00000 -CONFIG_NUMA=y # CONFIG_MIGRATION is not set # CONFIG_SH_DSP is not set # CONFIG_SH_TIMER_CMT is not set diff --git a/arch/sh/include/asm/mmu.h b/arch/sh/include/asm/mmu.h index b9c9f91e6616..def7c1edd605 100644 --- a/arch/sh/include/asm/mmu.h +++ b/arch/sh/include/asm/mmu.h @@ -2,6 +2,8 @@ #ifndef __MMU_H #define __MMU_H +#define MAX_PHYSMEM_BITS 32 + /* * Privileged Space Mapping Buffer (PMB) definitions */ diff --git a/arch/sh/include/asm/mmzone.h b/arch/sh/include/asm/mmzone.h deleted file mode 100644 index 63f88b465e39..000000000000 --- a/arch/sh/include/asm/mmzone.h +++ /dev/null @@ -1,42 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __ASM_SH_MMZONE_H -#define __ASM_SH_MMZONE_H - -#ifdef CONFIG_NUMA -#include <linux/numa.h> - -static inline int pfn_to_nid(unsigned long pfn) -{ - int nid; - - for (nid = 0; nid < MAX_NUMNODES; nid++) - if (pfn >= node_start_pfn(nid) && pfn <= node_end_pfn(nid)) - break; - - return nid; -} - -static inline struct pglist_data *pfn_to_pgdat(unsigned long pfn) -{ - return NODE_DATA(pfn_to_nid(pfn)); -} - -/* arch/sh/mm/numa.c */ -void __init setup_bootmem_node(int nid, unsigned long start, unsigned long end); -#else -static inline void -setup_bootmem_node(int nid, unsigned long start, unsigned long end) -{ -} -#endif /* CONFIG_NUMA */ - -/* Platform specific mem init */ -void __init plat_mem_setup(void); - -/* arch/sh/kernel/setup.c */ -void __init __add_active_range(unsigned int nid, unsigned long start_pfn, - unsigned long end_pfn); -/* arch/sh/mm/init.c */ -void __init allocate_pgdat(unsigned int nid); - -#endif /* __ASM_SH_MMZONE_H */ diff --git a/arch/sh/include/asm/setup.h b/arch/sh/include/asm/setup.h index 8488f76b48b4..6762612f456c 100644 --- a/arch/sh/include/asm/setup.h +++ b/arch/sh/include/asm/setup.h @@ -24,4 +24,7 @@ void check_for_initrd(void); void per_cpu_trap_init(void); void sh_fdt_init(phys_addr_t dt_phys); +void __add_active_range(unsigned int nid, unsigned long start_pfn, + unsigned long end_pfn); + #endif /* _SH_SETUP_H */ diff --git a/arch/sh/include/asm/sparsemem.h b/arch/sh/include/asm/sparsemem.h deleted file mode 100644 index 4703cbe23844..000000000000 --- a/arch/sh/include/asm/sparsemem.h +++ /dev/null @@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __ASM_SH_SPARSEMEM_H -#define __ASM_SH_SPARSEMEM_H - -/* - * SECTION_SIZE_BITS 2^N: how big each section will be - * MAX_PHYSMEM_BITS 2^N: how much physical address space we have - */ -#define SECTION_SIZE_BITS 26 -#define MAX_PHYSMEM_BITS 32 - -#endif /* __ASM_SH_SPARSEMEM_H */ diff --git a/arch/sh/include/asm/topology.h b/arch/sh/include/asm/topology.h index 1db470e02456..a3c31754ba5f 100644 --- a/arch/sh/include/asm/topology.h +++ b/arch/sh/include/asm/topology.h @@ -2,19 +2,6 @@ #ifndef _ASM_SH_TOPOLOGY_H #define _ASM_SH_TOPOLOGY_H -#ifdef CONFIG_NUMA - -#define cpu_to_node(cpu) ((void)(cpu),0) - -#define cpumask_of_node(node) ((void)node, cpu_online_mask) - -#define pcibus_to_node(bus) ((void)(bus), -1) -#define cpumask_of_pcibus(bus) (pcibus_to_node(bus) == -1 ? \ - cpu_all_mask : \ - cpumask_of_node(pcibus_to_node(bus))) - -#endif - #define mc_capable() (1) const struct cpumask *cpu_coregroup_mask(int cpu); diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7366.c b/arch/sh/kernel/cpu/sh4a/setup-sh7366.c index 6676beef053e..0a4ceb9785f6 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7366.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7366.c @@ -381,8 +381,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - /* TODO: Register Node 1 */ -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c index 0c6757ef63f4..5e2c24c518b4 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7722.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7722.c @@ -16,7 +16,6 @@ #include <linux/usb/m66592.h> #include <asm/clock.h> -#include <asm/mmzone.h> #include <asm/siu.h> #include <asm/platform_early.h> @@ -658,9 +657,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - /* Register the URAM space as Node 1 */ - setup_bootmem_node(1, 0x055f0000, 0x05610000); -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7723.c b/arch/sh/kernel/cpu/sh4a/setup-sh7723.c index d64d28c4f059..5c24ff407c89 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7723.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7723.c @@ -17,7 +17,6 @@ #include <asm/cacheflush.h> #include <asm/clock.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> #include <cpu/sh7723.h> diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7724.c b/arch/sh/kernel/cpu/sh4a/setup-sh7724.c index ef4b26a4b3d6..9441e4a0f402 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7724.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7724.c @@ -24,7 +24,6 @@ #include <asm/cacheflush.h> #include <asm/suspend.h> #include <asm/clock.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> #include <cpu/dma-register.h> diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c index 2ad19a0c5e04..3750b598d6bd 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7757.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7757.c @@ -21,7 +21,6 @@ #include <cpu/dma-register.h> #include <cpu/sh7757.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> static struct plat_sci_port scif2_platform_data = { @@ -1239,7 +1238,3 @@ void __init plat_irq_setup_pins(int mode) BUG(); } } - -void __init plat_mem_setup(void) -{ -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c index 3b4a414d60a9..13f08c44fb02 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7785.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7785.c @@ -13,7 +13,6 @@ #include <linux/sh_dma.h> #include <linux/sh_timer.h> #include <linux/sh_intc.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> #include <cpu/dma-register.h> @@ -600,9 +599,3 @@ void __init plat_irq_setup_pins(int mode) BUG(); } } - -void __init plat_mem_setup(void) -{ - /* Register the URAM space as Node 1 */ - setup_bootmem_node(1, 0xe55f0000, 0xe5610000); -} diff --git a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c index c048842d8a58..79484c598b83 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-sh7786.c +++ b/arch/sh/kernel/cpu/sh4a/setup-sh7786.c @@ -22,7 +22,6 @@ #include <linux/sh_intc.h> #include <linux/usb/ohci_pdriver.h> #include <cpu/dma-register.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> static struct plat_sci_port scif0_platform_data = { @@ -783,10 +782,6 @@ void __init plat_irq_setup_pins(int mode) } } -void __init plat_mem_setup(void) -{ -} - static int __init sh7786_devices_setup(void) { int ret, irq; diff --git a/arch/sh/kernel/cpu/sh4a/setup-shx3.c b/arch/sh/kernel/cpu/sh4a/setup-shx3.c index 60be8dca54f6..100a91034f2c 100644 --- a/arch/sh/kernel/cpu/sh4a/setup-shx3.c +++ b/arch/sh/kernel/cpu/sh4a/setup-shx3.c @@ -13,7 +13,6 @@ #include <linux/sh_timer.h> #include <linux/sh_intc.h> #include <cpu/shx3.h> -#include <asm/mmzone.h> #include <asm/platform_early.h> /* @@ -377,20 +376,3 @@ void __init plat_irq_setup(void) { register_intc_controller(&intc_desc); } - -void __init plat_mem_setup(void) -{ - unsigned int nid = 1; - - /* Register CPU#0 URAM space as Node 1 */ - setup_bootmem_node(nid++, 0x145f0000, 0x14610000); /* CPU0 */ - -#if 0 - /* XXX: Not yet.. */ - setup_bootmem_node(nid++, 0x14df0000, 0x14e10000); /* CPU1 */ - setup_bootmem_node(nid++, 0x155f0000, 0x15610000); /* CPU2 */ - setup_bootmem_node(nid++, 0x15df0000, 0x15e10000); /* CPU3 */ -#endif - - setup_bootmem_node(nid++, 0x16000000, 0x16020000); /* CSM */ -} diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c index 039a51291002..99f52d888aa3 100644 --- a/arch/sh/kernel/setup.c +++ b/arch/sh/kernel/setup.c @@ -23,7 +23,6 @@ #include <linux/smp.h> #include <linux/err.h> #include <linux/crash_dump.h> -#include <linux/mmzone.h> #include <linux/clk.h> #include <linux/delay.h> #include <linux/platform_device.h> @@ -41,9 +40,7 @@ #include <asm/clock.h> #include <asm/smp.h> #include <asm/mmu_context.h> -#include <asm/mmzone.h> #include <asm/processor.h> -#include <asm/sparsemem.h> #include <asm/platform_early.h> /* @@ -227,13 +224,9 @@ void __init __add_active_range(unsigned int nid, unsigned long start_pfn, /* * Also make sure that there is a PMB mapping that covers this * range before we attempt to activate it, to avoid reset by MMU. - * We can hit this path with NUMA or memory hot-add. */ pmb_bolt_mapping((unsigned long)__va(start), start, end - start, PAGE_KERNEL); - - memblock_set_node(PFN_PHYS(start_pfn), PFN_PHYS(end_pfn - start_pfn), - &memblock.memory, nid); } void __init __weak plat_early_device_setup(void) diff --git a/arch/sh/kernel/topology.c b/arch/sh/kernel/topology.c index 2d2a7509b565..906b7b1d5443 100644 --- a/arch/sh/kernel/topology.c +++ b/arch/sh/kernel/topology.c @@ -57,18 +57,6 @@ static int __init topology_init(void) __func__, i, ret); } -#if defined(CONFIG_NUMA) && !defined(CONFIG_SMP) - /* - * In the UP case, make sure the CPU association is still - * registered under each node. Without this, sysfs fails - * to make the connection between nodes other than node0 - * and cpu0. - */ - for_each_online_node(i) - if (i != numa_node_id()) - register_cpu_under_node(raw_smp_processor_id(), i); -#endif - return 0; } subsys_initcall(topology_init); diff --git a/arch/sh/kernel/vmcore_info.c b/arch/sh/kernel/vmcore_info.c index a244a204a1b1..9f029b4a7e63 100644 --- a/arch/sh/kernel/vmcore_info.c +++ b/arch/sh/kernel/vmcore_info.c @@ -5,10 +5,6 @@ void arch_crash_save_vmcoreinfo(void) { -#ifdef CONFIG_NUMA - VMCOREINFO_SYMBOL(node_data); - VMCOREINFO_LENGTH(node_data, MAX_NUMNODES); -#endif #ifdef CONFIG_X2TLB VMCOREINFO_CONFIG(X2TLB); #endif diff --git a/arch/sh/mm/Kconfig b/arch/sh/mm/Kconfig index 1862411665ab..c7d889a1a9d7 100644 --- a/arch/sh/mm/Kconfig +++ b/arch/sh/mm/Kconfig @@ -112,37 +112,8 @@ config VSYSCALL For systems with an MMU that can afford to give up a page, (the default value) say Y. -config NUMA - bool "Non-Uniform Memory Access (NUMA) Support" - depends on MMU && SYS_SUPPORTS_NUMA - select ARCH_WANT_NUMA_VARIABLE_LOCALITY - default n - help - Some SH systems have many various memories scattered around - the address space, each with varying latencies. This enables - support for these blocks by binding them to nodes and allowing - memory policies to be used for prioritizing and controlling - allocation behaviour. - -config NODES_SHIFT - int - default "3" if CPU_SUBTYPE_SHX3 - default "1" - depends on NUMA - config ARCH_FLATMEM_ENABLE def_bool y - depends on !NUMA - -config ARCH_SPARSEMEM_ENABLE - def_bool y - select SPARSEMEM_STATIC - -config ARCH_SPARSEMEM_DEFAULT - def_bool y - -config ARCH_SELECT_MEMORY_MODEL - def_bool y config IOREMAP_FIXED def_bool y diff --git a/arch/sh/mm/Makefile b/arch/sh/mm/Makefile index f69ddc70b146..7033947955d6 100644 --- a/arch/sh/mm/Makefile +++ b/arch/sh/mm/Makefile @@ -37,7 +37,6 @@ endif obj-$(CONFIG_DEBUG_FS) += $(debugfs-y) obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PMB) += pmb.o -obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_IOREMAP_FIXED) += ioremap_fixed.o obj-$(CONFIG_UNCACHED_MAPPING) += uncached.o obj-$(CONFIG_HAVE_SRAM_POOL) += sram.o diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c index 93921109f4e6..b34a5c304abd 100644 --- a/arch/sh/mm/init.c +++ b/arch/sh/mm/init.c @@ -20,7 +20,6 @@ #include <linux/dma-mapping.h> #include <linux/export.h> #include <asm/mmu_context.h> -#include <asm/mmzone.h> #include <asm/kexec.h> #include <asm/tlb.h> #include <asm/cacheflush.h> @@ -38,11 +37,6 @@ void __init generic_mem_init(void) memblock_add(__MEMORY_START, __MEMORY_SIZE); } -void __init __weak plat_mem_setup(void) -{ - /* Nothing to see here, move along. */ -} - #ifdef CONFIG_MMU static pte_t *__get_pte_phys(unsigned long addr) { @@ -201,20 +195,6 @@ void __init page_table_range_init(unsigned long start, unsigned long end, } #endif /* CONFIG_MMU */ -void __init allocate_pgdat(unsigned int nid) -{ - unsigned long start_pfn, end_pfn; - - get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); - -#ifdef CONFIG_NUMA - alloc_node_data(nid); -#endif - - NODE_DATA(nid)->node_start_pfn = start_pfn; - NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; -} - static void __init do_init_bootmem(void) { unsigned long start_pfn, end_pfn; @@ -224,11 +204,7 @@ static void __init do_init_bootmem(void) for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) __add_active_range(0, start_pfn, end_pfn); - /* All of system RAM sits in node 0 for the non-NUMA case */ - allocate_pgdat(0); node_set_online(0); - - plat_mem_setup(); } static void __init early_reserve_mem(void) diff --git a/arch/sh/mm/numa.c b/arch/sh/mm/numa.c deleted file mode 100644 index 9bc212b5e762..000000000000 --- a/arch/sh/mm/numa.c +++ /dev/null @@ -1,53 +0,0 @@ -/* - * arch/sh/mm/numa.c - Multiple node support for SH machines - * - * Copyright (C) 2007 Paul Mundt - * - * This file is subject to the terms and conditions of the GNU General Public - * License. See the file "COPYING" in the main directory of this archive - * for more details. - */ -#include <linux/module.h> -#include <linux/memblock.h> -#include <linux/mm.h> -#include <linux/numa.h> -#include <linux/pfn.h> -#include <asm/sections.h> - -/* - * On SH machines the conventional approach is to stash system RAM - * in node 0, and other memory blocks in to node 1 and up, ordered by - * latency. Each node's pgdat is node-local at the beginning of the node, - * immediately followed by the node mem map. - */ -void __init setup_bootmem_node(int nid, unsigned long start, unsigned long end) -{ - unsigned long start_pfn, end_pfn; - - /* Don't allow bogus node assignment */ - BUG_ON(nid >= MAX_NUMNODES || nid <= 0); - - start_pfn = PFN_DOWN(start); - end_pfn = PFN_DOWN(end); - - pmb_bolt_mapping((unsigned long)__va(start), start, end - start, - PAGE_KERNEL); - - memblock_add(start, end - start); - - __add_active_range(nid, start_pfn, end_pfn); - - /* Node-local pgdat */ - NODE_DATA(nid) = memblock_alloc_node(sizeof(struct pglist_data), - SMP_CACHE_BYTES, nid); - if (!NODE_DATA(nid)) - panic("%s: Failed to allocate %zu bytes align=0x%x nid=%d\n", - __func__, sizeof(struct pglist_data), SMP_CACHE_BYTES, - nid); - - NODE_DATA(nid)->node_start_pfn = start_pfn; - NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; - - /* It's up */ - node_set_online(nid); -} diff --git a/arch/sh/mm/sram.c b/arch/sh/mm/sram.c index 2d8fa718d55e..208080f072df 100644 --- a/arch/sh/mm/sram.c +++ b/arch/sh/mm/sram.c @@ -16,8 +16,7 @@ * This provides a standard SRAM pool for tiny memories that can be * added either by the CPU or the platform code. Typical SRAM sizes * to be inserted in to the pool will generally be less than the page - * size, with anything more reasonably sized handled as a NUMA memory - * node. + * size. */ struct gen_pool *sram_pool; diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 103db4683b16..9bbccb5d23a8 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -413,7 +413,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, if (mm->context.hugetlb_pte_count || mm->context.thp_pte_count) { unsigned long hugepage_size = PAGE_SIZE; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugepage_size = huge_page_size(hstate_vma(vma)); if (hugepage_size >= PUD_SIZE) { diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a5e29ffd5ddc..58dce0b66179 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -109,7 +109,7 @@ config X86 select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE select ARCH_HAS_SYSCALL_WRAPPER select ARCH_HAS_UBSAN - select ARCH_HAS_DEBUG_WX + select ARCH_HAS_CHECK_WX select ARCH_HAS_ZONE_DMA_SET if EXPERT select ARCH_HAVE_NMI_SAFE_CMPXCHG select ARCH_HAVE_EXTRA_ELF_NOTES diff --git a/arch/x86/configs/x86_64_defconfig b/arch/x86/configs/x86_64_defconfig index 269f7d808be4..e6896aeb77d8 100644 --- a/arch/x86/configs/x86_64_defconfig +++ b/arch/x86/configs/x86_64_defconfig @@ -263,7 +263,7 @@ CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_PRINTK_TIME=y CONFIG_DEBUG_KERNEL=y CONFIG_MAGIC_SYSRQ=y -CONFIG_DEBUG_WX=y +CONFIG_CHECK_WX=y CONFIG_DEBUG_STACK_USAGE=y CONFIG_SCHEDSTATS=y CONFIG_BLK_DEV_IO_TRACE=y diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index c79554dac5f7..000fc6c1ca37 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -41,10 +41,10 @@ void ptdump_walk_user_pgd_level_checkwx(void); #define pgprot_encrypted(prot) __pgprot(cc_mkenc(pgprot_val(prot))) #define pgprot_decrypted(prot) __pgprot(cc_mkdec(pgprot_val(prot))) -#ifdef CONFIG_DEBUG_WX -#define debug_checkwx_user() ptdump_walk_user_pgd_level_checkwx() +#ifdef CONFIG_CHECK_WX +#define pgtable_checkwx_user() ptdump_walk_user_pgd_level_checkwx() #else -#define debug_checkwx_user() do { } while (0) +#define pgtable_checkwx_user() do { } while (0) #endif extern spinlock_t pgd_lock; diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 65a2de82ecd2..0f60c0d076b6 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -715,7 +715,7 @@ static struct vm_area_struct *get_uprobe_trampoline(struct mm_struct *mm, unsign *new_mapping = true; return _install_special_mapping(mm, vaddr, PAGE_SIZE, - VM_READ|VM_EXEC|VM_MAYEXEC|VM_MAYREAD|VM_IO, + VM_READ|VM_EXEC|VM_MAYEXEC|VM_MAYREAD|VM_MIXEDMAP, &tramp_mapping); } diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c index 598f553cc871..31055ee6f1de 100644 --- a/arch/x86/mm/pti.c +++ b/arch/x86/mm/pti.c @@ -688,5 +688,5 @@ void pti_finalize(void) pti_clone_entry_text(true); pti_clone_kernel_text(); - debug_checkwx_user(); + pgtable_checkwx_user(); } diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 28f9503f6a71..1ac12c424200 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -977,7 +977,7 @@ void __init memory_dev_init(void) * block so that it can be covered. */ block_id = ULONG_MAX; - for_each_present_section_nr(0, nr) { + for_each_early_section_nr(0, nr) { if (block_id != ULONG_MAX && memory_block_id(nr) == block_id) continue; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 024402438bd1..ab064c84b092 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1553,68 +1553,70 @@ static void zram_debugfs_destroy(void) debugfs_remove_recursive(zram_debugfs_root); } -static ssize_t read_block_state(struct file *file, char __user *buf, - size_t count, loff_t *ppos) +static void *zram_block_state_start(struct seq_file *seq, loff_t *pos) { - char *kbuf; - unsigned long index; - ssize_t written = 0; - struct zram *zram = file->private_data; + struct zram *zram = seq->private; unsigned long nr_pages; - kbuf = kvmalloc(count, GFP_KERNEL); - if (!kbuf) - return -ENOMEM; - - guard(rwsem_read)(&zram->dev_lock); - if (!init_done(zram)) { - kvfree(kbuf); - return -EINVAL; - } + down_read(&zram->dev_lock); + if (!init_done(zram)) + return ERR_PTR(-EINVAL); nr_pages = zram->disksize >> PAGE_SHIFT; + if (*pos >= nr_pages) + return NULL; - for (index = *ppos; index < nr_pages; index++) { - int copied; + return pos; +} - slot_lock(zram, index); - if (!slot_allocated(zram, index)) - goto next; +static void *zram_block_state_next(struct seq_file *seq, void *v, loff_t *pos) +{ + struct zram *zram = seq->private; + unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; - copied = snprintf(kbuf + written, count, - "%12lu %12u.%06d %c%c%c%c%c%c\n", - index, zram->table[index].attr.ac_time, 0, - test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', - test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', - test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', - test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', - get_slot_comp_priority(zram, index) ? 'r' : '.', - test_slot_flag(zram, index, - ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); - - if (count <= copied) { - slot_unlock(zram, index); - break; - } - written += copied; - count -= copied; -next: - slot_unlock(zram, index); - *ppos += 1; - } + ++*pos; + if (*pos >= nr_pages) + return NULL; + + return pos; +} + +static void zram_block_state_stop(struct seq_file *seq, void *v) +{ + struct zram *zram = seq->private; - if (copy_to_user(buf, kbuf, written)) - written = -EFAULT; - kvfree(kbuf); + up_read(&zram->dev_lock); +} + +static int zram_block_state_show(struct seq_file *seq, void *v) +{ + struct zram *zram = seq->private; + unsigned long index = *(loff_t *)v; + + slot_lock(zram, index); + if (slot_allocated(zram, index)) { + seq_printf(seq, "%12lu %12u.%06d %c%c%c%c%c%c\n", + index, zram->table[index].attr.ac_time, 0, + test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', + test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', + test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', + test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', + get_slot_comp_priority(zram, index) ? 'r' : '.', + test_slot_flag(zram, index, + ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); + } + slot_unlock(zram, index); - return written; + return 0; } -static const struct file_operations proc_zram_block_state_op = { - .open = simple_open, - .read = read_block_state, - .llseek = default_llseek, +static const struct seq_operations zram_block_state_sops = { + .start = zram_block_state_start, + .next = zram_block_state_next, + .stop = zram_block_state_stop, + .show = zram_block_state_show, }; +DEFINE_SEQ_ATTRIBUTE(zram_block_state); static void zram_debugfs_register(struct zram *zram) { @@ -1624,7 +1626,7 @@ static void zram_debugfs_register(struct zram *zram) zram->debugfs_dir = debugfs_create_dir(zram->disk->disk_name, zram_debugfs_root); debugfs_create_file("block_state", 0400, zram->debugfs_dir, - zram, &proc_zram_block_state_op); + zram, &zram_block_state_fops); } static void zram_debugfs_unregister(struct zram *zram) diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c index a37b26881b73..03dbe304e4b8 100644 --- a/drivers/gpu/drm/drm_gpusvm.c +++ b/drivers/gpu/drm/drm_gpusvm.c @@ -9,9 +9,9 @@ #include <linux/dma-mapping.h> #include <linux/export.h> #include <linux/hmm.h> -#include <linux/hugetlb_inline.h> #include <linux/memremap.h> #include <linux/mm_types.h> +#include <linux/mm.h> #include <linux/slab.h> #include <drm/drm_device.h> @@ -1170,8 +1170,7 @@ drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, * limitations. If/when migrate_vma_* add more support, this logic will * have to change. */ - migrate_devmem = ctx->devmem_possible && - vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas); + migrate_devmem = ctx->devmem_possible && vma_is_anonymous(vas); chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier, vas, fault_addr, gpuva_start, diff --git a/drivers/gpu/drm/sti/sti_cursor.c b/drivers/gpu/drm/sti/sti_cursor.c index 699bcc452024..6a3ddb7fc17d 100644 --- a/drivers/gpu/drm/sti/sti_cursor.c +++ b/drivers/gpu/drm/sti/sti_cursor.c @@ -240,7 +240,7 @@ static int sti_cursor_atomic_check(struct drm_plane *drm_plane, cursor->pixmap.base = dma_alloc_wc(cursor->dev, cursor->pixmap.size, &cursor->pixmap.paddr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!cursor->pixmap.base) { DRM_ERROR("Failed to allocate memory for pixmap\n"); return -EINVAL; @@ -380,7 +380,7 @@ struct drm_plane *sti_cursor_create(struct drm_device *drm_dev, /* Allocate clut buffer */ size = 0x100 * sizeof(unsigned short); cursor->clut = dma_alloc_wc(dev, size, &cursor->clut_paddr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!cursor->clut) { DRM_ERROR("Failed to allocate memory for cursor clut\n"); diff --git a/drivers/gpu/drm/sti/sti_hqvdp.c b/drivers/gpu/drm/sti/sti_hqvdp.c index 4787c6f2b8c1..b367f7f84f5f 100644 --- a/drivers/gpu/drm/sti/sti_hqvdp.c +++ b/drivers/gpu/drm/sti/sti_hqvdp.c @@ -860,7 +860,7 @@ static void sti_hqvdp_init(struct sti_hqvdp *hqvdp) size = NB_VDP_CMD * sizeof(struct sti_hqvdp_cmd); hqvdp->hqvdp_cmd = dma_alloc_wc(hqvdp->dev, size, &dma_addr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!hqvdp->hqvdp_cmd) { DRM_ERROR("Failed to allocate memory for VDP cmd\n"); return; diff --git a/drivers/gpu/ipu-v3/ipu-image-convert.c b/drivers/gpu/ipu-v3/ipu-image-convert.c index 29b6d36c9bb6..493be4d4b091 100644 --- a/drivers/gpu/ipu-v3/ipu-image-convert.c +++ b/drivers/gpu/ipu-v3/ipu-image-convert.c @@ -371,7 +371,7 @@ static int alloc_dma_buf(struct ipu_image_convert_priv *priv, { buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(priv->ipu->dev, buf->len, &buf->phys, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) { dev_err(priv->ipu->dev, "failed to alloc dma buffer\n"); return -ENOMEM; diff --git a/drivers/hsi/clients/cmt_speech.c b/drivers/hsi/clients/cmt_speech.c index 7226677ebde7..801697b74d4f 100644 --- a/drivers/hsi/clients/cmt_speech.c +++ b/drivers/hsi/clients/cmt_speech.c @@ -1084,22 +1084,6 @@ static void cs_hsi_stop(struct cs_hsi_iface *hi) kfree(hi); } -static vm_fault_t cs_char_vma_fault(struct vm_fault *vmf) -{ - struct cs_char *csdata = vmf->vma->vm_private_data; - struct page *page; - - page = virt_to_page((void *)csdata->mmap_base); - get_page(page); - vmf->page = page; - - return 0; -} - -static const struct vm_operations_struct cs_char_vm_ops = { - .fault = cs_char_vma_fault, -}; - static int cs_char_fasync(int fd, struct file *file, int on) { struct cs_char *csdata = file->private_data; @@ -1256,18 +1240,19 @@ static long cs_char_ioctl(struct file *file, unsigned int cmd, return r; } -static int cs_char_mmap(struct file *file, struct vm_area_struct *vma) +static int cs_char_mmap_prepare(struct vm_area_desc *desc) { - if (vma->vm_end < vma->vm_start) - return -EINVAL; + struct file *file = desc->file; + struct cs_char *csdata = file->private_data; + struct page **pages = (struct page **)&desc->private_data; - if (vma_pages(vma) != 1) + if (vma_desc_pages(desc) != 1) return -EINVAL; - vm_flags_set(vma, VM_IO | VM_DONTDUMP | VM_DONTEXPAND); - vma->vm_ops = &cs_char_vm_ops; - vma->vm_private_data = file->private_data; + vma_desc_set_flags(desc, VMA_DONTDUMP_BIT, VMA_DONTEXPAND_BIT); + *pages = virt_to_page((void *)csdata->mmap_base); + mmap_action_map_kernel_pages_full(desc, pages); return 0; } @@ -1353,7 +1338,7 @@ static const struct file_operations cs_char_fops = { .write = cs_char_write, .poll = cs_char_poll, .unlocked_ioctl = cs_char_ioctl, - .mmap = cs_char_mmap, + .mmap_prepare = cs_char_mmap_prepare, .open = cs_char_open, .release = cs_char_release, .fasync = cs_char_fasync, diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index 1a36f995c4f6..680364fda856 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -70,7 +70,6 @@ static int set_ctxt_pkey(struct hfi1_ctxtdata *uctxt, unsigned long arg); static int ctxt_reset(struct hfi1_ctxtdata *uctxt); static int manage_rcvq(struct hfi1_ctxtdata *uctxt, u16 subctxt, unsigned long arg); -static vm_fault_t vma_fault(struct vm_fault *vmf); static long hfi1_file_ioctl(struct file *fp, unsigned int cmd, unsigned long arg); @@ -85,10 +84,6 @@ static const struct file_operations hfi1_file_ops = { .llseek = noop_llseek, }; -static const struct vm_operations_struct vm_ops = { - .fault = vma_fault, -}; - /* * Types of memories mapped into user processes' space */ @@ -304,13 +299,13 @@ static ssize_t hfi1_write_iter(struct kiocb *kiocb, struct iov_iter *from) return reqs; } -static inline void mmap_cdbg(u16 ctxt, u8 subctxt, u8 type, u8 mapio, u8 vmf, +static inline void mmap_cdbg(u16 ctxt, u8 subctxt, u8 type, u8 mapio, u8 is_vmalloc, u64 memaddr, void *memvirt, dma_addr_t memdma, ssize_t memlen, struct vm_area_struct *vma) { hfi1_cdbg(PROC, - "%u:%u type:%u io/vf/dma:%d/%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx", - ctxt, subctxt, type, mapio, vmf, !!memdma, + "%u:%u type:%u io/vmalloc/dma:%d/%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx", + ctxt, subctxt, type, mapio, is_vmalloc, !!memdma, memaddr ?: (u64)memvirt, memlen, vma->vm_end - vma->vm_start, vma->vm_flags); } @@ -325,7 +320,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memaddr = 0; void *memvirt = NULL; dma_addr_t memdma = 0; - u8 subctxt, mapio = 0, vmf = 0, type; + u8 subctxt, mapio = 0, is_vmalloc = 0, type; size_t memdmalen = 0; ssize_t memlen = 0; int ret = 0; @@ -348,7 +343,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) /* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning. - */ + */ vma->vm_pgoff = 0; flags = vma->vm_flags; @@ -367,7 +362,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) */ memlen = PAGE_ALIGN(uctxt->sc->credits * PIO_BLOCK_SIZE); flags &= ~VM_MAYREAD; - flags |= VM_DONTCOPY | VM_DONTEXPAND; + flags |= VM_DONTCOPY; vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); mapio = 1; break; @@ -411,6 +406,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memlen = rcvhdrq_size(uctxt); memvirt = uctxt->rcvhdrq; memdma = uctxt->rcvhdrq_dma; + flags |= VM_DONTEXPAND; break; case RCV_EGRBUF: { unsigned long vm_start_save; @@ -432,7 +428,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EPERM; goto done; } - vm_flags_clear(vma, VM_MAYWRITE); + vm_flags_mod(vma, VM_DONTEXPAND, VM_MAYWRITE); /* * Mmap multiple separate allocations into a single vma. From * here, dma_mmap_coherent() calls dma_direct_mmap(), which @@ -448,7 +444,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memvirt = uctxt->egrbufs.buffers[i].addr; memdma = uctxt->egrbufs.buffers[i].dma; vma->vm_end += memlen; - mmap_cdbg(ctxt, subctxt, type, mapio, vmf, memaddr, + mmap_cdbg(ctxt, subctxt, type, mapio, is_vmalloc, memaddr, memvirt, memdma, memlen, vma); ret = dma_mmap_coherent(&dd->pcidev->dev, vma, memvirt, memdma, memlen); @@ -477,7 +473,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) * user registers. */ memlen = PAGE_SIZE; - flags |= VM_DONTCOPY | VM_DONTEXPAND; + flags |= VM_DONTCOPY; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); mapio = 1; break; @@ -486,15 +482,10 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) * Use the page where this context's flags are. User level * knows where it's own bitmap is within the page. */ - memaddr = (unsigned long) - (dd->events + uctxt_offset(uctxt)) & PAGE_MASK; + memvirt = dd->events + uctxt_offset(uctxt); + memvirt = (void *)(((uintptr_t)memvirt) & PAGE_MASK); memlen = PAGE_SIZE; - /* - * v3.7 removes VM_RESERVED but the effect is kept by - * using VM_IO. - */ - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case STATUS: if (flags & VM_WRITE) { @@ -503,7 +494,6 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) } memaddr = kvirt_to_phys((void *)dd->status); memlen = PAGE_SIZE; - flags |= VM_IO | VM_DONTEXPAND; break; case RTAIL: if (!HFI1_CAP_IS_USET(DMA_RTAIL)) { @@ -522,25 +512,23 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) memvirt = (void *)hfi1_rcvhdrtail_kvaddr(uctxt); memdma = uctxt->rcvhdrqtailaddr_dma; flags &= ~VM_MAYWRITE; + flags |= VM_DONTEXPAND; break; case SUBCTXT_UREGS: - memaddr = (u64)uctxt->subctxt_uregbase; + memvirt = uctxt->subctxt_uregbase; memlen = PAGE_SIZE; - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case SUBCTXT_RCV_HDRQ: - memaddr = (u64)uctxt->subctxt_rcvhdr_base; + memvirt = uctxt->subctxt_rcvhdr_base; memlen = rcvhdrq_size(uctxt) * uctxt->subctxt_cnt; - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; case SUBCTXT_EGRBUF: - memaddr = (u64)uctxt->subctxt_rcvegrbuf; + memvirt = uctxt->subctxt_rcvegrbuf; memlen = uctxt->egrbufs.size * uctxt->subctxt_cnt; - flags |= VM_IO | VM_DONTEXPAND; flags &= ~VM_MAYWRITE; - vmf = 1; + is_vmalloc = 1; break; case SDMA_COMP: { struct hfi1_user_sdma_comp_q *cq = fd->cq; @@ -549,10 +537,9 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EFAULT; goto done; } - memaddr = (u64)cq->comps; + memvirt = cq->comps; memlen = PAGE_ALIGN(sizeof(*cq->comps) * cq->nentries); - flags |= VM_IO | VM_DONTEXPAND; - vmf = 1; + is_vmalloc = 1; break; } default: @@ -569,12 +556,10 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) } vm_flags_reset(vma, flags); - mmap_cdbg(ctxt, subctxt, type, mapio, vmf, memaddr, memvirt, memdma, + mmap_cdbg(ctxt, subctxt, type, mapio, is_vmalloc, memaddr, memvirt, memdma, memlen, vma); - if (vmf) { - vma->vm_pgoff = PFN_DOWN(memaddr); - vma->vm_ops = &vm_ops; - ret = 0; + if (is_vmalloc) { + ret = remap_vmalloc_range(vma, memvirt, 0); } else if (memdma) { ret = dma_mmap_coherent(&dd->pcidev->dev, vma, memvirt, memdma, @@ -599,24 +584,6 @@ done: return ret; } -/* - * Local (non-chip) user memory is not mapped right away but as it is - * accessed by the user-level code. - */ -static vm_fault_t vma_fault(struct vm_fault *vmf) -{ - struct page *page; - - page = vmalloc_to_page((void *)(vmf->pgoff << PAGE_SHIFT)); - if (!page) - return VM_FAULT_SIGBUS; - - get_page(page); - vmf->page = page; - - return 0; -} - static __poll_t hfi1_poll(struct file *fp, struct poll_table_struct *pt) { struct hfi1_ctxtdata *uctxt; diff --git a/drivers/media/platform/nxp/imx7-media-csi.c b/drivers/media/platform/nxp/imx7-media-csi.c index 0b2ee4dfdcbb..f06b03967aef 100644 --- a/drivers/media/platform/nxp/imx7-media-csi.c +++ b/drivers/media/platform/nxp/imx7-media-csi.c @@ -466,7 +466,7 @@ static int imx7_csi_alloc_dma_buf(struct imx7_csi *csi, buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(csi->dev, buf->len, &buf->dma_addr, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) return -ENOMEM; diff --git a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c index f45c2aae59ce..fc907d357149 100644 --- a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +++ b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c @@ -773,7 +773,7 @@ static int mxc_isi_video_alloc_discard_buffers(struct mxc_isi_video *video) buf->size = PAGE_ALIGN(video->pix.plane_fmt[i].sizeimage); buf->addr = dma_alloc_coherent(video->pipe->isi->dev, buf->size, - &buf->dma, GFP_DMA | GFP_KERNEL); + &buf->dma, GFP_KERNEL); if (!buf->addr) { mxc_isi_video_free_discard_buffers(video); return -ENOMEM; diff --git a/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c b/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c index c61f3983a713..0f1528710a6a 100644 --- a/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c +++ b/drivers/mtd/nand/raw/gpmi-nand/gpmi-nand.c @@ -1387,7 +1387,7 @@ static int gpmi_alloc_dma_buffer(struct gpmi_nand_data *this) goto error_alloc; this->auxiliary_virt = dma_alloc_coherent(dev, geo->auxiliary_size, - &this->auxiliary_phys, GFP_DMA); + &this->auxiliary_phys, GFP_KERNEL); if (!this->auxiliary_virt) goto error_alloc; diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c index 5408f002e6c0..3f9e08725602 100644 --- a/drivers/scsi/sg.c +++ b/drivers/scsi/sg.c @@ -1212,85 +1212,72 @@ sg_fasync(int fd, struct file *filp, int mode) return fasync_helper(fd, filp, mode, &sfp->async_qp); } -static vm_fault_t -sg_vma_fault(struct vm_fault *vmf) +static int sg_discontig_init(void *vm_private_data, void **private) { - struct vm_area_struct *vma = vmf->vma; - Sg_fd *sfp; - unsigned long offset, len, sa; - Sg_scatter_hold *rsv_schp; - int k, length; - - if ((NULL == vma) || (!(sfp = (Sg_fd *) vma->vm_private_data))) - return VM_FAULT_SIGBUS; - rsv_schp = &sfp->reserve; - offset = vmf->pgoff << PAGE_SHIFT; - if (offset >= rsv_schp->bufflen) - return VM_FAULT_SIGBUS; - SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, - "sg_vma_fault: offset=%lu, scatg=%d\n", - offset, rsv_schp->k_use_sg)); - sa = vma->vm_start; - length = 1 << (PAGE_SHIFT + rsv_schp->page_order); - for (k = 0; k < rsv_schp->k_use_sg && sa < vma->vm_end; k++) { - len = vma->vm_end - sa; - len = (len < length) ? len : length; - if (offset < len) { - struct page *page = rsv_schp->pages[k] + (offset >> PAGE_SHIFT); - get_page(page); /* increment page count */ - vmf->page = page; - return 0; /* success */ - } - sa += len; - offset -= len; + const unsigned long req_sz = (unsigned long)*private; + Sg_fd *sfp = vm_private_data; + Sg_scatter_hold *rsv_schp = &sfp->reserve; + int err = 0; + + mutex_lock(&sfp->f_mutex); + if (req_sz > rsv_schp->bufflen) { + err = -ENOMEM; /* cannot map more than reserved buffer */ + goto out; + } + sfp->mmap_called = 1; /* Prevents changes to buffer size. */ +out: + mutex_unlock(&sfp->f_mutex); + return err; +} + +static int +sg_discontig_get(struct discontig_kernel_page_state *state) +{ + Sg_fd *sfp = state->vm_private_data; + Sg_scatter_hold *rsv_schp = &sfp->reserve; + const unsigned int order = rsv_schp->page_order; + const pgoff_t nr_pages = state->nr_pages_mapped; + + if (nr_pages >= (rsv_schp->bufflen >> PAGE_SHIFT)) { + discontig_kernel_map_abort(state); + return 0; } - return VM_FAULT_SIGBUS; + SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, + "%s: offset=%lu, scatg=%d\n", __func__, + nr_pages << PAGE_SHIFT, rsv_schp->k_use_sg)); + + discontig_kernel_map_page(state, rsv_schp->pages[nr_pages >> order]); + return 0; } -static const struct vm_operations_struct sg_mmap_vm_ops = { - .fault = sg_vma_fault, +static const struct discontig_kernel_page_ops sg_discontig_ops = { + .init = sg_discontig_init, + .get = sg_discontig_get, }; static int -sg_mmap(struct file *filp, struct vm_area_struct *vma) +sg_mmap_prepare(struct vm_area_desc *desc) { - Sg_fd *sfp; - unsigned long req_sz, len, sa; - Sg_scatter_hold *rsv_schp; - int k, length; - int ret = 0; + Sg_fd *sfp = desc->file->private_data; + const unsigned long req_sz = vma_desc_size(desc); - if ((!filp) || (!vma) || (!(sfp = (Sg_fd *) filp->private_data))) + if (!sfp) return -ENXIO; - req_sz = vma->vm_end - vma->vm_start; + SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp, "sg_mmap starting, vm_start=%p, len=%d\n", - (void *) vma->vm_start, (int) req_sz)); - if (vma->vm_pgoff) + (void *) desc->start, (int) req_sz)); + + if (desc->pgoff) return -EINVAL; /* want no offset */ - rsv_schp = &sfp->reserve; - mutex_lock(&sfp->f_mutex); - if (req_sz > rsv_schp->bufflen) { - ret = -ENOMEM; /* cannot map more than reserved buffer */ - goto out; - } - sa = vma->vm_start; - length = 1 << (PAGE_SHIFT + rsv_schp->page_order); - for (k = 0; k < rsv_schp->k_use_sg && sa < vma->vm_end; k++) { - len = vma->vm_end - sa; - len = (len < length) ? len : length; - sa += len; - } + vma_desc_set_flags(desc, VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); + desc->private_data = sfp; - sfp->mmap_called = 1; - vm_flags_set(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); - vma->vm_private_data = sfp; - vma->vm_ops = &sg_mmap_vm_ops; -out: - mutex_unlock(&sfp->f_mutex); - return ret; + mmap_action_map_discontig_kernel_pages(desc, (void *)req_sz, + &sg_discontig_ops); + return 0; } static void @@ -1415,7 +1402,7 @@ static const struct file_operations sg_fops = { .unlocked_ioctl = sg_ioctl, .compat_ioctl = compat_ptr_ioctl, .open = sg_open, - .mmap = sg_mmap, + .mmap_prepare = sg_mmap_prepare, .release = sg_release, .fasync = sg_fasync, }; diff --git a/drivers/spi/spi-atmel.c b/drivers/spi/spi-atmel.c index c107c610dc01..ffc896e4621a 100644 --- a/drivers/spi/spi-atmel.c +++ b/drivers/spi/spi-atmel.c @@ -624,7 +624,7 @@ static int atmel_spi_configure_dma(struct spi_controller *host, if (IS_ENABLED(CONFIG_SOC_SAM_V4_V5)) { as->addr_tx_bbuf = dma_alloc_coherent(dev, SPI_MAX_DMA_XFER, &as->dma_addr_tx_bbuf, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!as->addr_tx_bbuf) { err = -ENOMEM; goto err_release_dma; @@ -632,7 +632,7 @@ static int atmel_spi_configure_dma(struct spi_controller *host, as->addr_rx_bbuf = dma_alloc_coherent(dev, SPI_MAX_DMA_XFER, &as->dma_addr_rx_bbuf, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!as->addr_rx_bbuf) { err = -ENOMEM; goto err_release_dma; diff --git a/drivers/spi/spi-ti-qspi.c b/drivers/spi/spi-ti-qspi.c index 24612e93eba1..091a8b8842f5 100644 --- a/drivers/spi/spi-ti-qspi.c +++ b/drivers/spi/spi-ti-qspi.c @@ -855,7 +855,7 @@ static int ti_qspi_probe(struct platform_device *pdev) qspi->rx_bb_addr = dma_alloc_coherent(qspi->dev, QSPI_DMA_BUFFER_SIZE, &qspi->rx_bb_dma_addr, - GFP_KERNEL | GFP_DMA); + GFP_KERNEL); if (!qspi->rx_bb_addr) { dev_err(qspi->dev, "dma_alloc_coherent failed, using PIO mode\n"); diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c index f119477cac6b..7bdcb88a686b 100644 --- a/drivers/staging/media/imx/imx-media-utils.c +++ b/drivers/staging/media/imx/imx-media-utils.c @@ -588,7 +588,7 @@ int imx_media_alloc_dma_buf(struct device *dev, buf->len = PAGE_ALIGN(size); buf->virt = dma_alloc_coherent(dev, buf->len, &buf->phys, - GFP_DMA | GFP_KERNEL); + GFP_KERNEL); if (!buf->virt) return -ENOMEM; diff --git a/drivers/usb/cdns3/cdns3-gadget.c b/drivers/usb/cdns3/cdns3-gadget.c index 0d272e9c556c..f4dfd0131708 100644 --- a/drivers/usb/cdns3/cdns3-gadget.c +++ b/drivers/usb/cdns3/cdns3-gadget.c @@ -3380,7 +3380,7 @@ static int cdns3_gadget_start(struct cdns *cdns) /* allocate memory for setup packet buffer */ priv_dev->setup_buf = dma_alloc_coherent(priv_dev->sysdev, 8, - &priv_dev->setup_dma, GFP_DMA); + &priv_dev->setup_dma, GFP_KERNEL); if (!priv_dev->setup_buf) { ret = -ENOMEM; goto err2; diff --git a/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c b/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c index 308d3c468ab1..8719f1f86e61 100644 --- a/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c +++ b/drivers/usb/gadget/udc/cdns2/cdns2-gadget.c @@ -2341,7 +2341,7 @@ static int cdns2_gadget_start(struct cdns2_device *pdev) /* Allocate memory for setup packet buffer. */ buf = dma_alloc_coherent(pdev->dev, 8, &pdev->ep0_preq.request.dma, - GFP_DMA); + GFP_KERNEL); pdev->ep0_preq.request.buf = buf; if (!pdev->ep0_preq.request.buf) { diff --git a/drivers/usb/gadget/udc/lpc32xx_udc.c b/drivers/usb/gadget/udc/lpc32xx_udc.c index 35cf44025559..94355d3d930f 100644 --- a/drivers/usb/gadget/udc/lpc32xx_udc.c +++ b/drivers/usb/gadget/udc/lpc32xx_udc.c @@ -3079,7 +3079,7 @@ static int lpc32xx_udc_probe(struct platform_device *pdev) /* Allocate memory for the UDCA */ udc->udca_v_base = dma_alloc_coherent(&pdev->dev, UDCA_BUFF_SIZE, &dma_handle, - (GFP_KERNEL | GFP_DMA)); + GFP_KERNEL); if (!udc->udca_v_base) { dev_err(udc->dev, "error getting UDCA region\n"); retval = -ENOMEM; diff --git a/drivers/usb/mon/mon_bin.c b/drivers/usb/mon/mon_bin.c index 687f6a8981f3..9d00b21a8153 100644 --- a/drivers/usb/mon/mon_bin.c +++ b/drivers/usb/mon/mon_bin.c @@ -1219,6 +1219,15 @@ mon_bin_poll(struct file *file, struct poll_table_struct *wait) return mask; } +static void __mon_bin_vma_open(struct mon_reader_bin *rp) +{ + unsigned long flags; + + spin_lock_irqsave(&rp->b_lock, flags); + rp->mmap_active++; + spin_unlock_irqrestore(&rp->b_lock, flags); +} + /* * open and close: just keep track of how many times the device is * mapped, to use the proper memory allocation function. @@ -1226,64 +1235,79 @@ mon_bin_poll(struct file *file, struct poll_table_struct *wait) static void mon_bin_vma_open(struct vm_area_struct *vma) { struct mon_reader_bin *rp = vma->vm_private_data; - unsigned long flags; - spin_lock_irqsave(&rp->b_lock, flags); - rp->mmap_active++; - spin_unlock_irqrestore(&rp->b_lock, flags); + __mon_bin_vma_open(rp); } -static void mon_bin_vma_close(struct vm_area_struct *vma) +static void __mon_bin_vma_close(struct mon_reader_bin *rp) { unsigned long flags; - struct mon_reader_bin *rp = vma->vm_private_data; spin_lock_irqsave(&rp->b_lock, flags); rp->mmap_active--; spin_unlock_irqrestore(&rp->b_lock, flags); } -/* - * Map ring pages to user space. - */ -static vm_fault_t mon_bin_vma_fault(struct vm_fault *vmf) +static void mon_bin_vma_close(struct vm_area_struct *vma) { - struct mon_reader_bin *rp = vmf->vma->vm_private_data; + struct mon_reader_bin *rp = vma->vm_private_data; + + __mon_bin_vma_close(rp); +} + +static const struct vm_operations_struct mon_bin_vm_ops = { + .open = mon_bin_vma_open, + .close = mon_bin_vma_close, +}; + +static int mon_bin_discontig_init(void *vm_private_data, void **private) +{ + struct mon_reader_bin *rp = vm_private_data; + + /* Dropped by mon_bin_vma_close() on unmap, including on error. */ + __mon_bin_vma_open(rp); + return 0; +} + +static int mon_bin_discontig_get(struct discontig_kernel_page_state *state) +{ + struct mon_reader_bin *rp = state->vm_private_data; unsigned long offset, chunk_idx; - struct page *pageptr; unsigned long flags; spin_lock_irqsave(&rp->b_lock, flags); - offset = vmf->pgoff << PAGE_SHIFT; + + offset = state->pgoff << PAGE_SHIFT; if (offset >= rp->b_size) { spin_unlock_irqrestore(&rp->b_lock, flags); - return VM_FAULT_SIGBUS; + discontig_kernel_map_abort(state); + return 0; } chunk_idx = offset / CHUNK_SIZE; - pageptr = rp->b_vec[chunk_idx].pg; - get_page(pageptr); - vmf->page = pageptr; + discontig_kernel_map_page(state, rp->b_vec[chunk_idx].pg); + spin_unlock_irqrestore(&rp->b_lock, flags); return 0; } -static const struct vm_operations_struct mon_bin_vm_ops = { - .open = mon_bin_vma_open, - .close = mon_bin_vma_close, - .fault = mon_bin_vma_fault, +static const struct discontig_kernel_page_ops mon_discontig_ops = { + .init = mon_bin_discontig_init, + .get = mon_bin_discontig_get, }; -static int mon_bin_mmap(struct file *filp, struct vm_area_struct *vma) +static int mon_bin_mmap_prepare(struct vm_area_desc *desc) { - /* don't do anything here: "fault" will set up page table entries */ - vma->vm_ops = &mon_bin_vm_ops; + const struct file *filp = desc->file; - if (vma->vm_flags & VM_WRITE) + if (vma_desc_test(desc, VMA_WRITE_BIT)) return -EPERM; - vm_flags_mod(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_MAYWRITE); - vma->vm_private_data = filp->private_data; - mon_bin_vma_open(vma); + desc->vm_ops = &mon_bin_vm_ops; + vma_desc_clear_flags(desc, VMA_MAYWRITE_BIT); + vma_desc_set_flags(desc, VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); + desc->private_data = filp->private_data; + + mmap_action_map_discontig_kernel_pages(desc, NULL, &mon_discontig_ops); return 0; } @@ -1298,7 +1322,7 @@ static const struct file_operations mon_fops_binary = { .compat_ioctl = mon_bin_compat_ioctl, #endif .release = mon_bin_release, - .mmap = mon_bin_mmap, + .mmap_prepare = mon_bin_mmap_prepare, }; static int mon_bin_wait_event(struct file *file, struct mon_reader_bin *rp) diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c index fd00b86e1ae6..fb359ecc3966 100644 --- a/drivers/video/fbdev/core/fb_defio.c +++ b/drivers/video/fbdev/core/fb_defio.c @@ -366,13 +366,13 @@ int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma) { vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); + if (WARN_ON_ONCE(!(info->flags & FBINFO_VIRTFB))) + return -EINVAL; if (!try_module_get(THIS_MODULE)) return -EINVAL; vma->vm_ops = &fb_deferred_io_vm_ops; - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP); - if (!(info->flags & FBINFO_VIRTFB)) - vm_flags_set(vma, VM_IO); + vm_flags_set(vma, VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = info->fbdefio_state; fb_deferred_io_state_get(info->fbdefio_state); /* released in vma->vm_ops->close() */ diff --git a/drivers/video/fbdev/fsl-diu-fb.c b/drivers/video/fbdev/fsl-diu-fb.c index b71d15794ce8..d7ed007915c1 100644 --- a/drivers/video/fbdev/fsl-diu-fb.c +++ b/drivers/video/fbdev/fsl-diu-fb.c @@ -1690,7 +1690,7 @@ static int fsl_diu_probe(struct platform_device *pdev) int ret; data = dmam_alloc_coherent(&pdev->dev, sizeof(struct fsl_diu_data), - &dma_addr, GFP_DMA | __GFP_ZERO); + &dma_addr, __GFP_ZERO); if (!data) return -ENOMEM; data->dma_addr = dma_addr; diff --git a/drivers/video/fbdev/ssd1307fb.c b/drivers/video/fbdev/ssd1307fb.c index 8fd96c37ee1a..0254a547182d 100644 --- a/drivers/video/fbdev/ssd1307fb.c +++ b/drivers/video/fbdev/ssd1307fb.c @@ -767,6 +767,8 @@ static int ssd1307fb_probe(struct i2c_client *client) info->fix.smem_start = __pa(vmem); info->fix.smem_len = vmem_size; + info->flags = FBINFO_VIRTFB; + ret = fb_deferred_io_init(info); if (ret) { dev_err(dev, "failed to init fb deferred io: %d\n", ret); diff --git a/fs/coredump.c b/fs/coredump.c index d88ae18ec59b..b25ef2ffbdee 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -1850,7 +1850,7 @@ static unsigned long vma_dump_size(struct vm_area_struct *vma, } /* Hugetlb memory check */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { if ((vma->vm_flags & VM_SHARED) && COREDUMP_MEMORY_TYPE_INCLUDE(memory_types, HUGETLB_SHARED)) goto whole; diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c index b79fa26ce783..cae217ff89ba 100644 --- a/fs/fuse/dax.c +++ b/fs/fuse/dax.c @@ -826,7 +826,7 @@ int fuse_vdax_mmap(struct file *file, struct vm_area_struct *vma) { file_accessed(file); vma->vm_ops = &fuse_dax_vm_ops; - vm_flags_set(vma, VM_MIXEDMAP | VM_HUGEPAGE); + vma_set_flags(vma, VMA_HUGEPAGE_BIT); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); return 0; } diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 6656807ce437..01c4d6f44ebe 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -106,11 +106,10 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) /* * vma address alignment (but not the pgoff alignment) has - * already been checked by prepare_hugepage_range. If you add - * any error returns here, do so after setting VM_HUGETLB, so - * is_vm_hugetlb_page tests below unmap_region go the right - * way when do_mmap unwinds (may be important on powerpc - * and ia64). + * already been checked by hugetlb_get_unmapped_area(). If you + * add any error returns here, do so after setting VM_HUGETLB, + * so vma_is_hugetlb tests below unmap_region go the right + * way when do_mmap unwinds (may be important on powerpc). */ vma_set_flags(vma, VMA_HUGETLB_BIT, VMA_DONTEXPAND_BIT); vma->vm_ops = &hugetlb_vm_ops; @@ -921,6 +920,7 @@ static struct inode *hugetlbfs_get_inode(struct super_block *sb, simple_inode_init_ts(inode); info->resv_map = resv_map; info->seals = F_SEAL_SEAL; + hugetlbfs_pmd_sharing_init(inode); switch (mode & S_IFMT) { default: init_special_inode(inode, mode, dev); diff --git a/fs/proc/page.c b/fs/proc/page.c index f90e1030825e..6aa99e42ec22 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -31,10 +31,9 @@ static inline unsigned long get_max_dump_pfn(void) { #ifdef CONFIG_SPARSEMEM /* - * The memmap of early sections is completely populated and marked - * online even if max_pfn does not fall on a section boundary - - * pfn_to_online_page() will succeed on all pages. Allow inspecting - * these memmaps. + * If max_pfn does not fall on a section boundary, pfn_to_online_page() + * can succeed on PFNs beyond max_pfn within the same section. Allow + * inspection of these memmaps. */ return round_up(max_pfn, PAGES_PER_SECTION); #else diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index ee9fc2fd2b2f..7452b695fdae 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2986,7 +2986,7 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, * hugetlb differs, see pagemap_hugetlb_category(). */ categories = p->cur_vma_category; - if (userfaultfd_wp(vma) && !is_vm_hugetlb_page(vma)) + if (userfaultfd_wp(vma) && !vma_is_hugetlb(vma)) categories |= PAGE_IS_WRITTEN; if (!pagemap_scan_is_interesting_page(categories, p)) @@ -2999,7 +2999,7 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, if (~p->arg.flags & PM_SCAN_WP_MATCHING) return ret; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) err = pagemap_scan_hugetlb_hole_wp(vma, addr, end); else err = uffd_wp_range(vma, addr, end - addr, true); @@ -3120,15 +3120,18 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) size_t n_ranges_out = 0; int ret; + if (!mmget_not_zero(mm)) + return -ESRCH; + ret = pagemap_scan_get_args(&p.arg, uarg); if (ret) - return ret; + goto out_mmput; p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask | p.arg.return_mask; ret = pagemap_scan_init_bounce_buffer(&p); if (ret) - return ret; + goto out_mmput; for (walk_start = p.arg.start; walk_start < p.arg.end; walk_start = p.arg.walk_end) { @@ -3183,6 +3186,8 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) ret = -EFAULT; kfree(p.vec_buf); +out_mmput: + mmput(mm); return ret; } @@ -3441,7 +3446,7 @@ static int show_numa_map(struct seq_file *m, void *v) seq_puts(m, " stack"); } - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) seq_puts(m, " huge"); /* Skip walking pages if gate VMA */ @@ -3470,7 +3475,7 @@ static int show_numa_map(struct seq_file *m, void *v) if (md->swapcache) seq_printf(m, " swapcache=%lu", md->swapcache); - if (md->active < md->pages && !is_vm_hugetlb_page(vma)) + if (md->active < md->pages && !vma_is_hugetlb(vma)) seq_printf(m, " active=%lu", md->active); if (md->writeback) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 9d827076db19..fbe1114e9a44 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -11,9 +11,9 @@ #ifndef _ASM_GENERIC__TLB_H #define _ASM_GENERIC__TLB_H +#include <linux/mm.h> #include <linux/mmu_notifier.h> #include <linux/swap.h> -#include <linux/hugetlb_inline.h> #include <asm/tlbflush.h> #include <asm/cacheflush.h> @@ -438,7 +438,7 @@ tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) * We rely on tlb_end_vma() to issue a flush, such that when we reset * these values the batch is empty. */ - tlb->vma_huge = is_vm_hugetlb_page(vma); + tlb->vma_huge = vma_is_hugetlb(vma); tlb->vma_exec = !!(vma->vm_flags & VM_EXEC); /* diff --git a/include/linux/cache.h b/include/linux/cache.h index e69768f50d53..b6e857b985bc 100644 --- a/include/linux/cache.h +++ b/include/linux/cache.h @@ -89,6 +89,7 @@ */ #ifndef INTERNODE_CACHE_SHIFT #define INTERNODE_CACHE_SHIFT L1_CACHE_SHIFT +#define INTERNODE_CACHE_BYTES (1 << INTERNODE_CACHE_SHIFT) #endif #if !defined(____cacheline_internodealigned_in_smp) diff --git a/include/linux/damon.h b/include/linux/damon.h index 844b175120f0..63050eb2206a 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -117,7 +117,6 @@ struct damon_target { * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions. * @DAMOS_MIGRATE_COLD: Migrate the regions prioritizing colder regions. * @DAMOS_STAT: Do nothing but count the stat. - * @NR_DAMOS_ACTIONS: Total number of DAMOS actions * * The support of each action is up to running &struct damon_operations. * Refer to 'Operation Action' section of Documentation/mm/damon/design.rst for @@ -137,7 +136,6 @@ enum damos_action { DAMOS_MIGRATE_HOT, DAMOS_MIGRATE_COLD, DAMOS_STAT, /* Do nothing but only record the stat */ - NR_DAMOS_ACTIONS, }; /** @@ -154,9 +152,6 @@ enum damos_action { * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a * node in basis points (0-10000). * @DAMOS_QUOTA_HUGEPAGE_MEM_BP: Huge page to total used memory ratio. - * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics. - * - * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported. */ enum damos_quota_goal_metric { DAMOS_QUOTA_USER_INPUT, @@ -169,12 +164,12 @@ enum damos_quota_goal_metric { DAMOS_QUOTA_INACTIVE_MEM_BP, DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP, DAMOS_QUOTA_HUGEPAGE_MEM_BP, - NR_DAMOS_QUOTA_GOAL_METRICS, }; /** * struct damos_quota_goal - DAMOS scheme quota auto-tuning goal. * @metric: Metric to be used for representing the goal. + * @complement: Use the complement of the metric. * @target_value: Target value of @metric to achieve with the tuning. * @current_value: Current value of @metric. * @nid: Node id. @@ -197,6 +192,7 @@ enum damos_quota_goal_metric { */ struct damos_quota_goal { enum damos_quota_goal_metric metric; + bool complement; unsigned long target_value; unsigned long current_value; /* metric-dependent fields */ @@ -313,12 +309,10 @@ struct damos_quota { * * @DAMOS_WMARK_NONE: Ignore the watermarks of the given scheme. * @DAMOS_WMARK_FREE_MEM_RATE: Free memory rate of the system in [0,1000]. - * @NR_DAMOS_WMARK_METRICS: Total number of DAMOS watermark metrics */ enum damos_wmark_metric { DAMOS_WMARK_NONE, DAMOS_WMARK_FREE_MEM_RATE, - NR_DAMOS_WMARK_METRICS, }; /** @@ -1096,7 +1090,7 @@ bool damos_filter_for_ops(enum damos_filter_type type); void damos_destroy_filter(struct damos_filter *f); struct damos_quota_goal *damos_new_quota_goal( - enum damos_quota_goal_metric metric, + enum damos_quota_goal_metric metric, bool complement, unsigned long target_value); void damos_add_quota_goal(struct damos_quota *q, struct damos_quota_goal *g); void damos_destroy_quota_goal(struct damos_quota_goal *goal); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 80a5a03e9cee..5029c7241863 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -7,11 +7,11 @@ #include <linux/mm_types.h> #include <linux/mmdebug.h> #include <linux/fs.h> -#include <linux/hugetlb_inline.h> #include <linux/cgroup.h> #include <linux/page_ref.h> #include <linux/list.h> #include <linux/kref.h> +#include <linux/atomic.h> #include <linux/pgtable.h> #include <linux/gfp.h> #include <linux/userfaultfd_k.h> @@ -252,14 +252,14 @@ extern void __hugetlb_zap_end(struct vm_area_struct *vma, static inline void hugetlb_zap_begin(struct vm_area_struct *vma, unsigned long *start, unsigned long *end) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) __hugetlb_zap_begin(vma, start, end); } static inline void hugetlb_zap_end(struct vm_area_struct *vma, struct zap_details *details) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) __hugetlb_zap_end(vma, details); } @@ -508,6 +508,9 @@ struct hugetlbfs_inode_info { struct inode vfs_inode; struct resv_map *resv_map; unsigned int seals; +#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING + atomic64_t pmd_sharing_count; +#endif }; static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode) @@ -515,6 +518,39 @@ static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode) return container_of(inode, struct hugetlbfs_inode_info, vfs_inode); } +#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING +static inline void hugetlbfs_pmd_sharing_init(struct inode *inode) +{ + atomic64_set(&HUGETLBFS_I(inode)->pmd_sharing_count, 0); +} + +static inline void hugetlbfs_pmd_sharing_inc(struct inode *inode) +{ + atomic64_inc(&HUGETLBFS_I(inode)->pmd_sharing_count); +} + +static inline void hugetlbfs_pmd_sharing_dec(struct inode *inode) +{ + atomic64_dec(&HUGETLBFS_I(inode)->pmd_sharing_count); +} + +static inline bool hugetlbfs_pmd_sharing_active(struct inode *inode) +{ + return atomic64_read(&HUGETLBFS_I(inode)->pmd_sharing_count) != 0; +} +#else +static inline void hugetlbfs_pmd_sharing_init(struct inode *inode) {} + +static inline void hugetlbfs_pmd_sharing_inc(struct inode *inode) {} + +static inline void hugetlbfs_pmd_sharing_dec(struct inode *inode) {} + +static inline bool hugetlbfs_pmd_sharing_active(struct inode *inode) +{ + return false; +} +#endif + extern const struct vm_operations_struct hugetlb_vm_ops; struct file *hugetlb_file_setup(const char *name, size_t size, vma_flags_t acct, int creat_flags, int page_size_log); diff --git a/include/linux/hugetlb_cgroup.h b/include/linux/hugetlb_cgroup.h index 16d72c8c71f6..b4e0958eabf3 100644 --- a/include/linux/hugetlb_cgroup.h +++ b/include/linux/hugetlb_cgroup.h @@ -268,4 +268,4 @@ static inline void hugetlb_cgroup_migrate(struct folio *old_folio, } #endif /* CONFIG_CGROUP_HUGETLB */ -#endif +#endif /* _LINUX_HUGETLB_CGROUP_H */ diff --git a/include/linux/hugetlb_inline.h b/include/linux/hugetlb_inline.h deleted file mode 100644 index 5c29cd3223a1..000000000000 --- a/include/linux/hugetlb_inline.h +++ /dev/null @@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_HUGETLB_INLINE_H -#define _LINUX_HUGETLB_INLINE_H - -#include <linux/mm.h> - -#ifdef CONFIG_HUGETLB_PAGE - -static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) -{ - return vma_flags_test(flags, VMA_HUGETLB_BIT); -} - -#else - -static inline bool is_vma_hugetlb_flags(const vma_flags_t *flags) -{ - return false; -} - -#endif - -static inline bool is_vm_hugetlb_page(const struct vm_area_struct *vma) -{ - return is_vma_hugetlb_flags(&vma->flags); -} - -#endif diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 64c183be8cbf..69e1bcc67c58 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -180,6 +180,7 @@ struct obj_cgroup { struct percpu_ref refcnt; struct mem_cgroup *memcg; atomic_t nr_charged_bytes; + refcount_t private_id_ref; union { struct list_head list; /* protected by objcg_lock */ struct rcu_head rcu; @@ -225,9 +226,6 @@ struct mem_cgroup { /* vmpressure notifications. Written on every reclaim iteration. */ struct vmpressure vmpressure; - /* Written on every swap charge and uncharge. */ - refcount_t private_id_ref; - #ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC /* MEMCG_KMEM for nmi context */ atomic_t kmem_stat; @@ -324,6 +322,9 @@ struct mem_cgroup { unsigned long zswap_max; #endif + /* The objcg holding private memcg ID. */ + struct obj_cgroup *private_id_objcg; + /* Private memcg ID. Used to ID objects that outlive the cgroup */ int private_id; @@ -983,6 +984,9 @@ unsigned long lruvec_page_state_monotonic(const struct lruvec *lruvec, unsigned long lruvec_page_state_local(const struct lruvec *lruvec, enum node_stat_item idx); +void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val); + void mem_cgroup_flush_stats(struct mem_cgroup *memcg); void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg); @@ -1451,6 +1455,11 @@ static inline unsigned long lruvec_page_state_local(const struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } +static inline void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ +} + static inline void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { } @@ -1708,6 +1717,7 @@ static inline u64 mem_cgroup_get_socket_pressure(const struct mem_cgroup *memcg) } #endif +void shrinker_unit_cache_init(void); int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); diff --git a/include/linux/mm.h b/include/linux/mm.h index 691cef038d1e..9f167950926a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -578,14 +578,6 @@ enum { #define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT) /* - * Special vmas that are non-mergable, non-mlock()able. - */ - -#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ - VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) -#define VM_SPECIAL vma_flags_to_legacy(VMA_SPECIAL_FLAGS) - -/* * Physically remapped pages are special. Tell the * rest of the world about it: * IO tells people not to look at these pages @@ -601,9 +593,6 @@ enum { #define VMA_REMAP_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, \ VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT) -/* This mask prevents VMA from being scanned with khugepaged */ -#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) - /* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGE @@ -1614,6 +1603,183 @@ static inline bool vma_is_shared_maywrite(const struct vm_area_struct *vma) } /** + * vma_flags_is_hugetlb() - Do the specified VMA flags indicate that the + * VMA is a hugetlb mapping? + * @flags: The VMA flags to test. + * + * Returns: true if the flags indicate a hugetlb mapping, false otherwise. + */ +static inline bool vma_flags_is_hugetlb(const vma_flags_t *flags) +{ + return IS_ENABLED(CONFIG_HUGETLB_PAGE) && + vma_flags_test(flags, VMA_HUGETLB_BIT); +} + +/** + * vma_is_hugetlb() - Is @vma a hugetlb mapping? + * @vma: The VMA to test. + * + * Returns: true if @vma is a hugetlb mapping, false otherwise. + */ +static inline bool vma_is_hugetlb(const struct vm_area_struct *vma) +{ + return vma_flags_is_hugetlb(&vma->flags); +} + +/** + * vma_flags_is_mm_managed() - Do the specified VMA flags indicate that the + * contents of the VMA are managed by core mm? + * @flags: The VMA flags to test. + * + * Such a mapping is one established by a simple mmap() or brk() with no driver + * or other kernel component involved. + * + * They are rmappable, and core mm services rmap operations including reclaim, + * as well as population, CoW and merge/split (if eligible). + * + * Mappings which fail this test are those where this is not the case - + * e.g. MMIO, user-mapped kernel pages, udmabuf (pfnmap shmem), etc. + * + * Returns: true if the flags indicate an mm-managed mapping. + */ +static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) +{ + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); +} + +/** + * vma_is_mm_managed() - Are the contents of @vma managed by core mm? + * @vma: The VMA to test. + * + * See vma_flags_is_mm_managed() for a description of this property. + * + * Returns: true if the VMA is mm-managed. + */ +static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_managed(&vma->flags); +} + +/** + * vma_flags_is_fixed_mapping() - Do the specified VMA flags indicate that this + * is a fixed mapping that cannot be expanded or merged? + * @flags: The VMA flags to test. + * + * Fixed mappings are those whose size is set at the point of mmap (for + * instance, a mapping of a fixed range of memory established by a driver), + * and thus cannot be expanded or merged. + * + * Returns: true if the flags indicate a fixed mapping. + */ +static inline bool vma_flags_is_fixed_mapping(const vma_flags_t *flags) +{ + /* + * VMA_PFNMAP_BIT should imply VMA_DONTEXPAND_BIT, but some callers set + * only the former. + */ + return vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT); +} + +/** + * vma_is_fixed_mapping() - Is this VMA a fixed mapping that cannot be + * expanded or merged? + * @vma: The VMA to test. + * + * See vma_flags_is_fixed_mapping() for a description of this property. + * + * Returns: true if the VMA maps a fixed mapping. + */ +static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_fixed_mapping(&vma->flags); +} + +/** + * vma_flags_is_mm_backed() - Do the specified VMA flags imply the mapping is + * backed by core mm? + * @flags: The VMA flags to test. + * + * An mm-backed mapping is one managed by core mm (see + * vma_flags_is_mm_managed()) whose contents are established by core mm's own + * fault path rather than a custom ->fault handler. + * + * Mappings not managed by core mm are populated by their owner, and fixed + * mappings may be mm-managed but have their contents established by a custom + * ->fault handler, so neither are mm-backed. hugetlb mappings are fixed, but + * core mm owns them entirely, so they are. + * + * These are the only mappings which can sensibly be merged, locked, dumped or + * be subject to uffd faulting, as all of these assume core mm and core mm + * alone manages their contents. + * + * Returns: true if the flags imply an mm-backed mapping, otherwise false. + */ +static inline bool vma_flags_is_mm_backed(const vma_flags_t *flags) +{ + /* hugetlb is a fixed mapping, but core mm owns it entirely. */ + if (vma_flags_is_hugetlb(flags)) + return true; + /* + * Mappings not managed by core mm are populated by their owner. + * + * Fixed mappings may be mm-managed, but their contents are established + * by a custom ->fault handler, so core mm cannot assume ordinary fault + * semantics over them. + */ + return vma_flags_is_mm_managed(flags) && !vma_flags_is_fixed_mapping(flags); +} + +/** + * vma_is_mm_backed() - Is @vma backed by core mm? + * @vma: The VMA to test. + * + * See vma_flags_is_mm_backed() for details. + * + * Returns: true if the VMA is mm-backed, otherwise false. + */ +static inline bool vma_is_mm_backed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_backed(&vma->flags); +} + +/** + * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be + * merged with another? + * @flags: The VMA flags to test. + * Returns: true if the flags permit merging, false otherwise. + */ +static inline bool vma_flags_can_merge(const vma_flags_t *flags) +{ + /* + * VMA merging assumes that a VMA's flags and fields completely describe + * its state, which is only the case for mm-backed mappings. + * + * Mappings which are not mm-managed may have established state upon + * mapping not embodied in any attribute of the VMA. + * + * Additionally, private (CoW) PFN maps encode the source PFN of the + * range in vma->vm_pgoff, which may otherwise cause spurious merges. + */ + if (!vma_flags_is_mm_backed(flags)) + return false; + /* + * hugetlb mappings contain per-VMA state, so their flags and fields do + * not completely describe them. + */ + return !vma_flags_is_hugetlb(flags); +} + +/** + * vma_can_merge() - Do @vma's flags permit it to be merged with another VMA? + * @vma: The VMA to test. + * Returns: true if the flags permit merging, otherwise false. + */ +static inline bool vma_can_merge(const struct vm_area_struct *vma) +{ + return vma_flags_can_merge(&vma->flags); +} + +/** * vma_kernel_pagesize - Default page size granularity for this VMA. * @vma: The user mapping. * @@ -4164,6 +4330,12 @@ void mapping_rmap_tree_insert_after(struct vm_area_struct *vma, struct address_space *mapping); void mapping_rmap_tree_remove(struct vm_area_struct *vma, struct address_space *mapping); +void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged); +void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged); struct vm_area_struct * mapping_rmap_tree_iter_first(struct address_space *mapping, pgoff_t pgoff_start, pgoff_t pgoff_last); @@ -4181,6 +4353,10 @@ void anon_rmap_tree_insert(struct anon_vma_chain *avc, struct anon_vma *anon_vma); void anon_rmap_tree_remove(struct anon_vma_chain *avc, struct anon_vma *anon_vma); +void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged); +void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged); struct anon_vma_chain * anon_rmap_tree_iter_first(struct anon_vma *anon_vma, pgoff_t pgoff_start, pgoff_t pgoff_last); @@ -4601,7 +4777,7 @@ static inline void mmap_action_simple_ioremap(struct vm_area_desc *desc, * @desc: The VMA descriptor for the VMA requiring kernel pags to be mapped. * @start: The virtual address from which to map them. * @pages: An array of struct page pointers describing the memory to map. - * @nr_pages: The number of entries in the @pages aray. + * @nr_pages: The number of entries in the @pages array. */ static inline void mmap_action_map_kernel_pages(struct vm_area_desc *desc, unsigned long start, struct page **pages, @@ -4609,7 +4785,7 @@ static inline void mmap_action_map_kernel_pages(struct vm_area_desc *desc, { struct mmap_action *action = &desc->action; - action->type = MMAP_MAP_KERNEL_PAGES; + action->type = MMAP_KERNEL_PAGES; action->map_kernel.start = start; action->map_kernel.pages = pages; action->map_kernel.nr_pages = nr_pages; @@ -4633,10 +4809,53 @@ static inline void mmap_action_map_kernel_pages_full(struct vm_area_desc *desc, vma_desc_pages(desc)); } +static inline +void mmap_action_map_discontig_kernel_pages(struct vm_area_desc *desc, + void *init_private, const struct discontig_kernel_page_ops *ops) +{ + struct mmap_action *action = &desc->action; + + action->type = MMAP_DISCONTIG_KERNEL_PAGES; + action->map_kernel_discontig.init_private = init_private; + action->map_kernel_discontig.ops = ops; +} + int mmap_action_prepare(struct vm_area_desc *desc); int mmap_action_complete(struct vm_area_struct *vma, struct mmap_action *action, bool is_compat); +static inline void +discontig_kernel_map_abort(struct discontig_kernel_page_state *state) +{ + state->action = DISCONTIG_KERNEL_PAGE_ABORT; +} + +static inline void +discontig_kernel_map_page(struct discontig_kernel_page_state *state, + struct page *page) +{ + struct folio *folio = page_folio(page); + unsigned long nr_pages = 1; + + if (folio_test_large(folio)) { + VM_WARN_ON_ONCE(page != folio_page(folio, 0)); + nr_pages = folio_nr_pages(folio); + } + + state->action = DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES; + state->__page = page; + state->__nr_pages = min(nr_pages, state->nr_pages_remain); +} + +static inline void +discontig_kernel_map_pages(struct discontig_kernel_page_state *state, + struct page **page_arr, unsigned long nr_pages) +{ + state->action = DISCONTIG_KERNEL_PAGE_MAP_PAGES; + state->__page_arr = page_arr; + state->__nr_pages = min(nr_pages, state->nr_pages_remain); +} + /* Look up the first VMA which exactly match the interval vm_start ... vm_end */ static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm, unsigned long vm_start, unsigned long vm_end) @@ -4754,9 +4973,6 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, int vm_insert_page(struct vm_area_struct *, unsigned long addr, struct page *); int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num); -int map_kernel_pages_prepare(struct vm_area_desc *desc); -int map_kernel_pages_complete(struct vm_area_struct *vma, - struct mmap_action *action); int vm_map_pages(struct vm_area_struct *vma, struct page **pages, unsigned long num); int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0720a4e98286..1ed35d258d2e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -759,7 +759,7 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) /* * While __vma_enter_locked() is working to ensure are no read-locks held on a * VMA (either while acquiring a VMA write lock or marking a VMA detached) we - * set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indiciate to + * set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indicate to * vma_start_read() that the reference count should be left alone. * * See the comment describing vm_refcnt in vm_area_struct for details as to @@ -815,11 +815,45 @@ struct pfnmap_track_ctx { /* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type { - MMAP_NOTHING, /* Mapping is complete, no further action. */ - MMAP_REMAP_PFN, /* Remap PFN range. */ - MMAP_IO_REMAP_PFN, /* I/O remap PFN range. */ - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_MAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_NOTHING, + MMAP_REMAP_PFN, + MMAP_IO_REMAP_PFN, + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ +}; + +enum discontig_kernel_page_action { + DISCONTIG_KERNEL_PAGE_ABORT, + DISCONTIG_KERNEL_PAGE_MAP_PAGES, /* Array of individual pages. */ + DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES, /* 1..N contiguous pages. */ +}; + +struct discontig_kernel_page_state { + /* Map state. */ + const unsigned long start; /* Start address of VMA. */ + const unsigned long end; /* End address of VMA. */ + unsigned long addr; /* The current address to be mapped. */ + pgoff_t pgoff; /* The current pgoff to be mapped. */ + unsigned long nr_pages_mapped; /* The number of pages mapped. */ + unsigned long nr_pages_remain; /* The number of pages remaining. */ + + /* User-defined state. */ + void *vm_private_data; /* VMA private data. */ + void *private; /* Mapping private data. */ + + /* Users should not touch these, use discontig_kernel_map_*() helpers. */ + enum discontig_kernel_page_action action; + union { + struct page *__page; + struct page **__page_arr; + }; + unsigned long __nr_pages; +}; + +struct discontig_kernel_page_ops { + int (*init)(void *vm_private_data, void **private); + int (*get)(struct discontig_kernel_page_state *state); }; /* @@ -844,6 +878,10 @@ struct mmap_action { unsigned long nr_pages; pgoff_t pgoff; } map_kernel; + struct { + void *init_private; + const struct discontig_kernel_page_ops *ops; + } map_kernel_discontig; }; enum mmap_action_type type; @@ -954,15 +992,19 @@ struct vm_area_struct { * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_FLAG is set + * The one exception is __mm_reset_vma_seqcounts(), which stores zero + * (an even value, which can never indicate a write lock) under the + * mmap write lock alone once the mm seqcount has wrapped, so a reader + * holding only vm_refcnt may observe the value change from odd to zero. * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set or vm_refcnt > 1 + * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * - * This sequence counter is explicitly allowed to overflow; sequence - * counter reuse can only lead to occasional unnecessary use of the - * slowpath. + * Assigned the (odd) mm seqcount on write lock and not updated on + * unlock, so a stale value would alias the mm seqcount again 2^31 write + * lock/unlock cycles later. The reset described above prevents this. */ unsigned int vm_lock_seq; @@ -1971,7 +2013,7 @@ enum { /* * MMF_HAS_PINNED: Whether this mm has pinned any pages. This can be either * replaced in the future by mm.pinned_vm when it becomes stable, or grow into - * a counter on its own. We're aggresive on this bit for now: even if the + * a counter on its own. We're aggressive on this bit for now: even if the * pinned pages were unpinned later on, we'll still keep this bit set for the * lifecycle of this mm, just for simplicity. */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 28e3696ce9ff..11b3ffef3966 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -154,7 +154,7 @@ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) #endif if (reset_refcnt) refcount_set(&vma->vm_refcnt, 0); - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; } /* @@ -229,8 +229,9 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) /* * Use only while holding mmap read lock which guarantees that vma lock is not - * contended (nobody can concurrently write-lock the vma). vma_start_read() should - * not be used in such cases because it might fail due to mm_lock_seq overflow. + * contended (nobody can concurrently write-lock the vma). vma_start_read() need + * not be used in such cases, as the mmap lock already excludes VMA writers so + * its sequence number checks are redundant. * This functionality is used to obtain vma read lock and drop the mmap read lock. * * VMA can't be detached while we are holding mmap lock, therefore in practice this @@ -252,8 +253,9 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int /* * Use only while holding mmap read lock which guarantees that vma lock is not - * contended (nobody can concurrently write-lock the vma). vma_start_read() should - * not be used in such cases because it might fail due to mm_lock_seq overflow. + * contended (nobody can concurrently write-lock the vma). vma_start_read() need + * not be used in such cases, as the mmap lock already excludes VMA writers so + * its sequence number checks are redundant. * This functionality is used to obtain vma read lock and drop the mmap read lock. * * VMA can't be detached while we are holding mmap lock, therefore in practice this @@ -273,7 +275,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) vma_refcount_put(vma); } -static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma) +static inline unsigned int __vma_raw_mm_seqnum(const struct vm_area_struct *vma) { const struct mm_struct *mm = vma->vm_mm; @@ -288,7 +290,7 @@ static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma) * * Returns true if write-locked, otherwise false. */ -static inline bool __is_vma_write_locked(struct vm_area_struct *vma) +static inline bool __is_vma_write_locked(const struct vm_area_struct *vma) { /* * current task is holding mmap_write_lock, both vma->vm_lock_seq and @@ -344,7 +346,7 @@ int vma_start_write_killable(struct vm_area_struct *vma) * vma_assert_write_locked() - assert that @vma holds a VMA write lock. * @vma: The VMA to assert. */ -static inline void vma_assert_write_locked(struct vm_area_struct *vma) +static inline void vma_assert_write_locked(const struct vm_area_struct *vma) { if (!IS_ENABLED(CONFIG_MMU)) { mmap_assert_write_locked(vma->vm_mm); @@ -359,7 +361,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) * lock and is not detached. * @vma: The VMA to assert. */ -static inline void vma_assert_locked(struct vm_area_struct *vma) +static inline void vma_assert_locked(const struct vm_area_struct *vma) { unsigned int refcnt; @@ -410,7 +412,7 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) * With lockdep disabled we may sometimes race with other threads acquiring the * mmap read lock simultaneous with our VMA read lock. */ -static inline void vma_assert_stabilised(struct vm_area_struct *vma) +static inline void vma_assert_stabilised(const struct vm_area_struct *vma) { /* * If another thread owns an mmap lock, it may go away at any time, and @@ -445,7 +447,7 @@ static inline void vma_assert_stabilised(struct vm_area_struct *vma) vma_assert_locked(vma); } -static inline bool vma_is_attached(struct vm_area_struct *vma) +static inline bool vma_is_attached(const struct vm_area_struct *vma) { return refcount_read(&vma->vm_refcnt); } @@ -552,6 +554,8 @@ static inline int __must_check mmap_write_lock_killable(struct mm_struct *mm) return ret; } +void __mm_reset_vma_seqcounts(struct mm_struct *mm); + /* * Drop all currently-held per-VMA locks. * This is called from the mmap_lock implementation directly before releasing @@ -564,6 +568,16 @@ static inline void vma_end_write_all(struct mm_struct *mm) { mmap_assert_write_locked(mm); mm_lock_seqcount_end(mm); + + /* + * VMA seqcounts are not updated on unlock, so once the mm seqcount has + * wrapped, any VMA still carrying a sequence number from before the + * wrap would satisfy __is_vma_write_locked() when that number comes + * round again, and readers would not be excluded. Reset every VMA's + * seqcount so that no stale value survives the wrap. + */ + if (unlikely(!mm->mm_lock_seq.sequence)) + __mm_reset_vma_seqcounts(mm); } static inline void mmap_write_unlock(struct mm_struct *mm) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d44b04589f48..ecd1ebd49756 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1114,9 +1114,6 @@ struct zone { #ifdef CONFIG_UNACCEPTED_MEMORY /* Pages to be accepted. All pages on the list are MAX_PAGE_ORDER */ struct list_head unaccepted_pages; - - /* To be called once the last page in the zone is accepted */ - struct work_struct unaccepted_cleanup; #endif /* zone flags, see below */ @@ -2131,7 +2128,6 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) * accommodate SECTION_MAP_LAST_BIT. We use BUILD_BUG_ON() to ensure this. */ enum { - SECTION_MARKED_PRESENT_BIT, SECTION_HAS_MEM_MAP_BIT, SECTION_IS_ONLINE_BIT, SECTION_IS_EARLY_BIT, @@ -2141,7 +2137,6 @@ enum { SECTION_MAP_LAST_BIT, }; -#define SECTION_MARKED_PRESENT BIT(SECTION_MARKED_PRESENT_BIT) #define SECTION_HAS_MEM_MAP BIT(SECTION_HAS_MEM_MAP_BIT) #define SECTION_IS_ONLINE BIT(SECTION_IS_ONLINE_BIT) #define SECTION_IS_EARLY BIT(SECTION_IS_EARLY_BIT) @@ -2158,16 +2153,6 @@ static inline struct page *__section_mem_map_addr(struct mem_section *section) return (struct page *)map; } -static inline int present_section(const struct mem_section *section) -{ - return (section && (section->section_mem_map & SECTION_MARKED_PRESENT)); -} - -static inline int present_section_nr(unsigned long nr) -{ - return present_section(__nr_to_section(nr)); -} - static inline int valid_section(const struct mem_section *section) { return (section && (section->section_mem_map & SECTION_HAS_MEM_MAP)); @@ -2183,6 +2168,11 @@ static inline int valid_section_nr(unsigned long nr) return valid_section(__nr_to_section(nr)); } +static inline int early_section_nr(unsigned long nr) +{ + return early_section(__nr_to_section(nr)); +} + static inline int online_section(const struct mem_section *section) { return (section && (section->section_mem_map & SECTION_IS_ONLINE)); @@ -2227,7 +2217,7 @@ static inline struct mem_section *__pfn_to_section(unsigned long pfn) return __nr_to_section(pfn_to_section_nr(pfn)); } -extern unsigned long __highest_present_section_nr; +extern unsigned long __highest_used_section_nr; static inline int subsection_map_index(unsigned long pfn) { @@ -2326,7 +2316,7 @@ static inline unsigned long first_valid_pfn(unsigned long pfn, unsigned long end rcu_read_lock_sched(); - while (nr <= __highest_present_section_nr && pfn < end_pfn) { + while (nr <= __highest_used_section_nr && pfn < end_pfn) { struct mem_section *ms = __pfn_to_section(pfn); if (valid_section(ms) && @@ -2372,27 +2362,20 @@ static inline unsigned long next_valid_pfn(unsigned long pfn, unsigned long end_ #endif -static inline int pfn_in_present_section(unsigned long pfn) -{ - if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS) - return 0; - return present_section(__pfn_to_section(pfn)); -} - -static inline unsigned long next_present_section_nr(unsigned long section_nr) +static inline unsigned long next_early_section_nr(unsigned long section_nr) { - while (++section_nr <= __highest_present_section_nr) { - if (present_section_nr(section_nr)) + while (++section_nr <= __highest_used_section_nr) { + if (early_section_nr(section_nr)) return section_nr; } return -1; } -#define for_each_present_section_nr(start, section_nr) \ - for (section_nr = next_present_section_nr(start - 1); \ +#define for_each_early_section_nr(start, section_nr) \ + for (section_nr = next_early_section_nr(start - 1); \ section_nr != -1; \ - section_nr = next_present_section_nr(section_nr)) + section_nr = next_early_section_nr(section_nr)) /* * These are _only_ used during initialisation, therefore they @@ -2408,9 +2391,6 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #else #define pfn_to_nid(pfn) (0) #endif - -#else -#define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ /* diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index bcbb0afe1a68..73af18a37367 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,7 +14,6 @@ #include <linux/gfp.h> #include <linux/bitops.h> #include <linux/hardirq.h> /* for in_interrupt() */ -#include <linux/hugetlb_inline.h> struct folio_batch; diff --git a/include/linux/ptdump.h b/include/linux/ptdump.h index 240bd3bff18d..af18d1459b2f 100644 --- a/include/linux/ptdump.h +++ b/include/linux/ptdump.h @@ -31,9 +31,9 @@ bool ptdump_walk_pgd_level_core(struct seq_file *m, void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd); bool ptdump_check_wx(void); -static inline void debug_checkwx(void) +static inline void pgtable_checkwx(void) { - if (IS_ENABLED(CONFIG_DEBUG_WX)) + if (IS_ENABLED(CONFIG_CHECK_WX)) ptdump_check_wx(); } diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0b332770abee..74cca0e3c726 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -888,7 +888,7 @@ struct page_vma_mapped_walk { static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) { /* HugeTLB pte is set to the relevant page table entry without pte_mapped. */ - if (pvmw->pte && !is_vm_hugetlb_page(pvmw->vma)) + if (pvmw->pte && !vma_is_hugetlb(pvmw->vma)) pte_unmap(pvmw->pte); if (pvmw->ptl) spin_unlock(pvmw->ptl); diff --git a/include/linux/swap.h b/include/linux/swap.h index cb434cccd653..4f686709d71f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -395,6 +395,8 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); extern int pin_hibernation_swap_type(dev_t device, sector_t offset); extern void unpin_hibernation_swap_type(int type); +extern int repin_hibernation_swap_type(int old_type, dev_t device, + sector_t offset); extern int find_hibernation_swap_type(dev_t device, sector_t offset); int find_first_swap(dev_t *device); extern unsigned int count_swap_pages(int, int); diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index a4351cffc60c..a14b8a9ffb7b 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -18,7 +18,6 @@ #include <linux/swap.h> #include <linux/leafops.h> #include <asm-generic/pgtable_uffd.h> -#include <linux/hugetlb_inline.h> /* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_MINOR | \ diff --git a/init/Kconfig b/init/Kconfig index b31aa298c4fe..4013781e0a02 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -815,17 +815,10 @@ config CC_STRINGOP_OVERFLOW config ARCH_SUPPORTS_INT128 bool -# For architectures that (ab)use NUMA to represent different memory regions -# all cpu-local but of different latencies, such as SuperH. -# -config ARCH_WANT_NUMA_VARIABLE_LOCALITY - bool - config NUMA_BALANCING bool "Memory placement aware NUMA scheduler" depends on ARCH_SUPPORTS_NUMA_BALANCING depends on ARCH_HAS_PTE_PROTNONE - depends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY depends on SMP && NUMA_MIGRATION && !PREEMPT_RT help This option adds support for automatic NUMA aware memory/task placement. diff --git a/init/main.c b/init/main.c index ceba948b333e..d7bf1ffc5f50 100644 --- a/init/main.c +++ b/init/main.c @@ -1592,7 +1592,7 @@ static void mark_readonly(void) flush_module_init_free_work(); jump_label_init_ro(); mark_rodata_ro(); - debug_checkwx(); + pgtable_checkwx(); rodata_test(); } else if (IS_ENABLED(CONFIG_STRICT_KERNEL_RWX)) { pr_info("Kernel memory protection disabled.\n"); diff --git a/ipc/shm.c b/ipc/shm.c index bb1a721a3e74..6862f681373f 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -936,26 +936,11 @@ static inline unsigned long copy_shminfo_to_user(void __user *buf, struct shminf static void shm_add_rss_swap(struct shmid_kernel *shp, unsigned long *rss_add, unsigned long *swp_add) { - struct inode *inode; + struct inode *inode = file_inode(shp->shm_file); - inode = file_inode(shp->shm_file); - - if (is_file_hugepages(shp->shm_file)) { - struct address_space *mapping = inode->i_mapping; - struct hstate *h = hstate_file(shp->shm_file); - *rss_add += pages_per_huge_page(h) * mapping->nrpages; - } else { -#ifdef CONFIG_SHMEM - struct shmem_inode_info *info = SHMEM_I(inode); - - spin_lock_irq(&info->lock); - *rss_add += inode->i_mapping->nrpages; - *swp_add += info->swapped; - spin_unlock_irq(&info->lock); -#else - *rss_add += inode->i_mapping->nrpages; -#endif - } + *rss_add += READ_ONCE(inode->i_mapping->nrpages); + if (IS_ENABLED(CONFIG_SHMEM) && !is_file_hugepages(shp->shm_file)) + *swp_add += READ_ONCE(SHMEM_I(inode)->swapped); } /* diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 868589f07330..3c61c181a6fa 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -692,8 +692,9 @@ static int arena_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) * clears VM_MAYEXEC. Set VM_DONTEXPAND to avoid potential change * of user_vm_start. Set VM_DONTCOPY to prevent arena VMA from * being copied into the child process on fork. + * This is a kernel page so set VM_MIXEDMAP. */ - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTCOPY); + vm_flags_set(vma, VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTCOPY); vma->vm_ops = &arena_vm_ops; return 0; } diff --git a/kernel/configs/debug.config b/kernel/configs/debug.config index 307c97ac5fa9..ac878669c193 100644 --- a/kernel/configs/debug.config +++ b/kernel/configs/debug.config @@ -50,7 +50,7 @@ CONFIG_DEBUG_NET=y # CONFIG_DEBUG_PAGEALLOC is not set # CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set # CONFIG_DEBUG_RODATA_TEST is not set -# CONFIG_DEBUG_WX is not set +# CONFIG_CHECK_WX is not set # CONFIG_KFENCE is not set # CONFIG_PAGE_POISONING is not set # CONFIG_SLUB_STATS is not set diff --git a/kernel/events/core.c b/kernel/events/core.c index f36a3bbaa33c..457d75ea7142 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -9927,7 +9927,7 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event) if (vma->vm_flags & VM_LOCKED) flags |= MAP_LOCKED; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) flags |= MAP_HUGETLB; if (file) { diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 3192806eebfe..caf444762a27 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1726,8 +1726,8 @@ static int xol_add_vma(struct mm_struct *mm, struct xol_area *area) } vma = _install_special_mapping(mm, area->vaddr, PAGE_SIZE, - VM_EXEC|VM_MAYEXEC|VM_DONTCOPY|VM_IO| - VM_SEALED_SYSMAP, + VM_EXEC|VM_MAYEXEC|VM_DONTCOPY| + VM_MIXEDMAP|VM_SEALED_SYSMAP, &xol_mapping); if (IS_ERR(vma)) { ret = PTR_ERR(vma); diff --git a/kernel/power/user.c b/kernel/power/user.c index d0fcfba7ac23..6e4f40e49319 100644 --- a/kernel/power/user.c +++ b/kernel/power/user.c @@ -218,6 +218,7 @@ static int snapshot_set_swap_area(struct snapshot_data *data, { sector_t offset; dev_t swdev; + int new_type; if (swsusp_swap_in_use()) return -EPERM; @@ -238,19 +239,11 @@ static int snapshot_set_swap_area(struct snapshot_data *data, offset = swap_area.offset; } - /* - * Unpin the swap device if a swap area was already - * set by SNAPSHOT_SET_SWAP_AREA. - */ - unpin_hibernation_swap_type(data->swap); + new_type = repin_hibernation_swap_type(data->swap, swdev, offset); + if (new_type < 0) + return new_type; - /* - * User space encodes device types as two-byte values, - * so we need to recode them - */ - data->swap = pin_hibernation_swap_type(swdev, offset); - if (data->swap < 0) - return swdev ? -ENODEV : -EINVAL; + data->swap = new_type; data->dev = swdev; return 0; } diff --git a/kernel/resource.c b/kernel/resource.c index cfc1a00e86aa..0c315c917fbb 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -1695,10 +1695,15 @@ int devm_request_resource(struct device *dev, struct resource *root, *ptr = new; - conflict = request_resource_conflict(root, new); - if (conflict) { + /* Report a conflict while the lock still keeps it in the tree. */ + write_lock(&resource_lock); + conflict = __request_resource(root, new); + if (conflict) dev_err(dev, "resource collision: %pR conflicts with %s %pR\n", new, conflict->name, conflict); + write_unlock(&resource_lock); + + if (conflict) { devres_free(ptr); return -EBUSY; } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f77e7226693b..cfc982205b98 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -22,7 +22,6 @@ */ #include <linux/energy_model.h> #include <linux/mmap_lock.h> -#include <linux/hugetlb_inline.h> #include <linux/jiffies.h> #include <linux/math.h> #include <linux/mm_api.h> @@ -4364,7 +4363,7 @@ retry_pids: for (; vma; vma = vma_next(&vmi)) { if (!vma_migratable(vma) || !vma_policy_mof(vma) || - is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + vma_is_hugetlb(vma) || !vma_is_mm_managed(vma)) { trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UNSUITABLE); continue; } diff --git a/kernel/vmcore_info.c b/kernel/vmcore_info.c index 5a417f8a922a..7833a36064a8 100644 --- a/kernel/vmcore_info.c +++ b/kernel/vmcore_info.c @@ -182,6 +182,7 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_STRUCT_SIZE(mem_section); VMCOREINFO_OFFSET(mem_section, section_mem_map); VMCOREINFO_NUMBER(SECTION_SIZE_BITS); + VMCOREINFO_NUMBER(SECTION_HAS_MEM_MAP); VMCOREINFO_NUMBER(MAX_PHYSMEM_BITS); #endif VMCOREINFO_STRUCT_SIZE(page); diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 1aba6cced713..0b0036c9f848 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -764,43 +764,6 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, } /* - * mt_clear_meta() - clear the metadata information of a node, if it exists - * @mt: The maple tree - * @mn: The maple node - * @type: The maple node type - */ -static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn, - enum maple_type type) -{ - struct maple_metadata *meta; - unsigned long *pivots; - void __rcu **slots; - void *next; - - switch (type) { - case maple_range_64: - pivots = mn->mr64.pivot; - if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { - slots = mn->mr64.slot; - next = mt_slot_locked(mt, slots, - MAPLE_RANGE64_SLOTS - 1); - if (unlikely((mte_to_node(next) && - mte_node_type(next)))) - return; /* no metadata, could be node */ - } - fallthrough; - case maple_arange_64: - meta = ma_meta(mn, type); - break; - default: - return; - } - - meta->gap = 0; - meta->end = 0; -} - -/* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node * @mt: The maple node type @@ -4789,9 +4752,7 @@ static void mt_free_walk(struct rcu_head *head) type = mte_node_type(enode); slots = ma_slots(mte_to_node(enode), type); - if ((offset < mt_slots[type]) && - rcu_dereference_protected(slots[offset], - lock_is_held(&rcu_callback_map))) + if (offset < mte_to_node(enode)->slot_len) slots = mte_dead_walk(&enode, offset); node = mte_to_node(enode); } while ((node != start) || (node->slot_len < offset)); @@ -4885,8 +4846,6 @@ next: free_leaf: if (free) kfree(node); - else - mt_clear_meta(mt, node, node->type); } /* diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c index b9367c61e8b5..fb40e193a3cb 100644 --- a/lib/test_maple_tree.c +++ b/lib/test_maple_tree.c @@ -3710,6 +3710,36 @@ static noinline void __init alloc_cyclic_testing(struct maple_tree *mt) MT_BUG_ON(mt, ret != 1); } +static noinline void __init check_range64_in_rcu(struct maple_tree *mt) +{ + unsigned long i; + unsigned long nr_entries = 226; /* Build a full maple_range_64 root node */ + + MT_BUG_ON(mt, !mtree_empty(mt)); + mt_init_flags(mt, MT_FLAGS_USE_RCU); + + for (i = 0; i < nr_entries; i++) { + MT_BUG_ON(mt, mtree_test_insert_range(mt, i*10, i*10 + 9, + xa_mk_value(i))); + } + + mtree_destroy(mt); + rcu_barrier(); + + nr_entries = 3166; /* Height 3. Root has 15 children + metadata */ + + MT_BUG_ON(mt, !mtree_empty(mt)); + mt_init_flags(mt, MT_FLAGS_USE_RCU); + + for (i = 0; i < nr_entries; i++) { + MT_BUG_ON(mt, mtree_test_insert_range(mt, i*10, i*10 + 9, + xa_mk_value(i))); + } + + mtree_destroy(mt); + rcu_barrier(); +} + static DEFINE_MTREE(tree); static int __init maple_tree_seed(void) { @@ -3999,6 +4029,8 @@ static int __init maple_tree_seed(void) alloc_cyclic_testing(&tree); mtree_destroy(&tree); + check_range64_in_rcu(&tree); + #if defined(BENCH) skip: diff --git a/mm/Kconfig b/mm/Kconfig index 7179fd209546..afff1b25b543 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1490,6 +1490,47 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_HAS_CHECK_WX + bool + +config CHECK_WX + bool "Warn on W+X mappings at boot" + default STRICT_KERNEL_RWX + depends on ARCH_HAS_CHECK_WX + depends on ARCH_HAS_PTDUMP + depends on MMU + select PTDUMP + help + Generate a warning if any W+X mappings are found at boot. + + This is useful for discovering cases where the kernel is leaving W+X + mappings after applying NX, as such mappings are a security risk. + + Look for a message in dmesg output like this: + + <arch>/mm: Checked W+X mappings: passed, no W+X pages found. + + or like this, if the check failed: + + <arch>/mm: Checked W+X mappings: failed, <N> W+X pages found. + + Note that even if the check fails, your kernel is possibly + still fine, as W+X mappings are not a security hole in + themselves, what they do is that they make the exploitation + of other unfixed kernel bugs easier. + + There is no runtime or memory usage effect of this option + once the kernel has booted up - it's a one time check. + + If in doubt, say "Y". + +config ARCH_HAS_PTDUMP + bool + +config PTDUMP + bool + + source "mm/damon/Kconfig" endmenu diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9eaa25d1cf23..9ae75229c76d 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -180,45 +180,6 @@ config DEBUG_RODATA_TEST help This option enables a testcase for the setting rodata read-only. -config ARCH_HAS_DEBUG_WX - bool - -config DEBUG_WX - bool "Warn on W+X mappings at boot" - depends on ARCH_HAS_DEBUG_WX - depends on ARCH_HAS_PTDUMP - depends on MMU - select PTDUMP - help - Generate a warning if any W+X mappings are found at boot. - - This is useful for discovering cases where the kernel is leaving W+X - mappings after applying NX, as such mappings are a security risk. - - Look for a message in dmesg output like this: - - <arch>/mm: Checked W+X mappings: passed, no W+X pages found. - - or like this, if the check failed: - - <arch>/mm: Checked W+X mappings: failed, <N> W+X pages found. - - Note that even if the check fails, your kernel is possibly - still fine, as W+X mappings are not a security hole in - themselves, what they do is that they make the exploitation - of other unfixed kernel bugs easier. - - There is no runtime or memory usage effect of this option - once the kernel has booted up - it's a one time check. - - If in doubt, say "Y". - -config ARCH_HAS_PTDUMP - bool - -config PTDUMP - bool - config PTDUMP_DEBUGFS bool "Export kernel pagetable layout to userspace via debugfs" depends on DEBUG_KERNEL diff --git a/mm/backing-dev.c b/mm/backing-dev.c index 18e999053bae..fad1495d3b20 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -1162,19 +1162,20 @@ void bdi_unregister(struct backing_dev_info *bdi) wb_shutdown(&bdi->wb); cgwb_bdi_unregister(bdi); - /* - * If this BDI's min ratio has been set, use bdi_set_min_ratio() to - * update the global bdi_min_ratio. - */ - if (bdi->min_ratio) - bdi_set_min_ratio(bdi, 0); - if (bdi->dev) { bdi_debug_unregister(bdi); device_unregister(bdi->dev); bdi->dev = NULL; } + /* + * If this BDI's min ratio has been set, use bdi_set_min_ratio() to + * update the global bdi_min_ratio. After device_unregister(): the + * sysfs min_ratio stores can no longer run and set it again. + */ + if (bdi->min_ratio) + bdi_set_min_ratio(bdi, 0); + if (bdi->owner) { put_device(bdi->owner); bdi->owner = NULL; diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..759ec9cd4541 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include <linux/mm.h> +#include <linux/nodemask.h> +#include <linux/pgtable.h> +#include <linux/types.h> + +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +/* How many PTEs of a window may be missing, swapped out or shared */ +struct collapse_limits { + /* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */ + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; +}; + +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { + /* Limits for a PMD-sized window */ + struct collapse_limits pmd; + + /* + * Limits for a smaller window. Its max_ptes_none is either 0 or + * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the + * window whatever its order; any other value counts as 0. + */ + struct collapse_limits sub_pmd; + + /* Leave clean lazyfree folios to reclaim rather than collapse them */ + bool anon_skip_lazyfree; + + /* Refuse an anonymous range with no sign of use */ + bool anon_require_referenced; + + /* Map the PMD over a file collapse instead of leaving it to a fault */ + bool file_install_pmd; + + /* Write dirty pages back and retry once instead of refusing them */ + bool file_writeback_dirty; + + /* How hard to try for a destination folio */ + gfp_t gfp; + + /* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ + enum tva_type tva_type; +}; + +struct collapse_control { + struct collapse_policy policy; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* + * What a scan found and the collapse after it needs. Live only between + * the two, and read by nobody else. + */ + + /* Anonymous only: orders to try, referenced and swap PTE counts */ + unsigned long scan_orders; + int scan_referenced; + int scan_unmapped; + + /* + * File only. A file collapse works on the page cache and never sees a + * VMA, so the scan takes a reference to the file while it still has + * one, and the collapse gives it back. + */ + struct file *scan_file; + pgoff_t scan_pgoff; +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/compaction.c b/mm/compaction.c index 4994e200bbec..f1b2060eb201 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -216,7 +216,7 @@ static unsigned long skip_offline_sections(unsigned long start_pfn) if (online_section_nr(start_nr)) return 0; - while (++start_nr <= __highest_present_section_nr) { + while (++start_nr <= __highest_used_section_nr) { if (online_section_nr(start_nr)) return section_nr_to_pfn(start_nr); } diff --git a/mm/damon/core.c b/mm/damon/core.c index 733025b36745..eb1fcdabc007 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -691,7 +691,7 @@ void damos_destroy_filter(struct damos_filter *f) } struct damos_quota_goal *damos_new_quota_goal( - enum damos_quota_goal_metric metric, + enum damos_quota_goal_metric metric, bool complement, unsigned long target_value) { struct damos_quota_goal *goal; @@ -700,6 +700,7 @@ struct damos_quota_goal *damos_new_quota_goal( if (!goal) return NULL; goal->metric = metric; + goal->complement = complement; goal->target_value = target_value; if (metric == DAMOS_QUOTA_SOME_MEM_PSI_US) goal->last_psi_total = U64_MAX; @@ -734,7 +735,7 @@ static bool damos_quota_goals_empty(struct damos_quota *q) } /* initialize fields of @quota that normally API users wouldn't set */ -static struct damos_quota *damos_quota_init(struct damos_quota *quota) +static void damos_quota_init(struct damos_quota *quota) { quota->esz = 0; quota->total_charged_sz = 0; @@ -744,7 +745,6 @@ static struct damos_quota *damos_quota_init(struct damos_quota *quota) quota->charge_target_from = NULL; quota->charge_addr_from = 0; quota->esz_bp = 0; - return quota; } struct damos *damon_new_scheme(struct damos_access_pattern *pattern, @@ -776,7 +776,8 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, scheme->last_applied = NULL; INIT_LIST_HEAD(&scheme->list); - scheme->quota = *(damos_quota_init(quota)); + scheme->quota = *quota; + damos_quota_init(&scheme->quota); /* quota.goals should be separately set by caller */ INIT_LIST_HEAD(&scheme->quota.goals); @@ -1224,6 +1225,7 @@ static int damos_commit_quota_goal( if (!src->target_value) return -EINVAL; dst->metric = src->metric; + dst->complement = src->complement; dst->target_value = src->target_value; if (dst->metric == DAMOS_QUOTA_USER_INPUT) dst->current_value = src->current_value; @@ -1261,8 +1263,8 @@ int damos_commit_quota_goals(struct damos_quota *dst, struct damos_quota *src) damos_for_each_quota_goal_safe(src_goal, next, src) { if (j++ < i) continue; - new_goal = damos_new_quota_goal( - src_goal->metric, src_goal->target_value); + new_goal = damos_new_quota_goal(src_goal->metric, + src_goal->complement, src_goal->target_value); if (!new_goal) return -ENOMEM; err = damos_commit_quota_goal(new_goal, src_goal); @@ -2549,13 +2551,14 @@ static void damos_update_stat(struct damos *s, static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos_filter *filter, - unsigned long min_region_sz) + unsigned long min_region_sz, int *err) { bool matched = false; struct damon_target *ti; int target_idx = 0; unsigned long start, end, wsum; + *err = 0; switch (filter->type) { case DAMOS_FILTER_TYPE_TARGET: damon_for_each_target(ti, ctx) { @@ -2581,12 +2584,13 @@ static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, } /* start before the range and overlap */ if (r->ar.start < start) { - damon_split_region_at(t, r, start - r->ar.start); + *err = damon_split_region_at(t, r, + start - r->ar.start); matched = false; break; } /* start inside the range */ - damon_split_region_at(t, r, end - r->ar.start); + *err = damon_split_region_at(t, r, end - r->ar.start); matched = true; break; case DAMOS_FILTER_TYPE_PROBE_HITS_WSUM: @@ -2605,10 +2609,16 @@ static bool damos_core_filter_out(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos *s) { struct damos_filter *filter; + bool matching; + int err; s->core_filters_allowed = false; damos_for_each_core_filter(filter, s) { - if (damos_filter_match(ctx, t, r, filter, ctx->min_region_sz)) { + matching = damos_filter_match(ctx, t, r, filter, + ctx->min_region_sz, &err); + if (err) + return true; + if (matching) { if (filter->allow) s->core_filters_allowed = true; return !filter->allow; @@ -2960,10 +2970,18 @@ static void damos_set_psi_current_val(u64 now_psi_total, struct damos_quota_goal *goal, struct damos *s) { u64 last_psi_total = goal->last_psi_total; + unsigned long val; goal->last_psi_total = now_psi_total; if (last_psi_total != U64_MAX) { - goal->current_value = now_psi_total - last_psi_total; + val = now_psi_total - last_psi_total; + if (goal->complement) { + if (val < s->quota.reset_interval * 1000) + val = s->quota.reset_interval * 1000 - val; + else + val = 0; + } + goal->current_value = val; return; } /* uninitialized last_psi_total; make no effect this round */ @@ -3255,6 +3273,21 @@ static void damos_set_quota_goal_current_value(struct damon_ctx *c, default: break; } + if (!goal->complement) + return; + + /* update current_value to complemented value */ + + /* for user_input, users set complemented value on their own */ + if (goal->metric == DAMOS_QUOTA_USER_INPUT) + return; + /* damos_set_psi_current_val() handles complement flag itself */ + if (goal->metric == DAMOS_QUOTA_SOME_MEM_PSI_US) + return; + if (goal->current_value < 10000) + goal->current_value = 10000 - goal->current_value; + else + goal->current_value = 0; } /* Return the highest score since it makes schemes least aggressive */ @@ -3291,7 +3324,7 @@ static void damos_goal_tune_esz_bp_temporal(struct damon_ctx *c, if (score >= 10000) quota->esz_bp = 0; - else if (quota->sz) + else if (quota->sz && quota->sz <= ULONG_MAX / 10000) quota->esz_bp = quota->sz * 10000; else quota->esz_bp = ULONG_MAX; @@ -3350,6 +3383,26 @@ static void damos_trace_esz(struct damon_ctx *c, struct damos *s, trace_damos_esz(cidx, sidx, quota->esz); } +static void damos_fix_stale_charge_pointers(struct damon_ctx *c, + struct damos *s) +{ + struct damos_quota *quota; + struct damon_target *t; + + quota = &s->quota; + if (!quota->charge_target_from) + return; + damon_for_each_target(t, c) { + if (t == quota->charge_target_from) { + if (damon_nr_regions(t) > 0) + return; + break; + } + } + quota->charge_target_from = NULL; + quota->charge_addr_from = 0; +} + static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) { struct damos_quota *quota = &s->quota; @@ -3385,6 +3438,8 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) damos_trace_esz(c, s, quota); } + damos_fix_stale_charge_pointers(c, s); + if (damos_quota_is_full(quota, c->min_region_sz)) return; if (!c->ops.get_scheme_score) @@ -3433,7 +3488,7 @@ static void damos_trace_stat(struct damon_ctx *c, struct damos *s) trace_call__damos_stat_after_apply_interval(cidx, sidx, &s->stat); } -static void kdamond_apply_schemes(struct damon_ctx *c) +static noinline_for_stack void kdamond_apply_schemes(struct damon_ctx *c) { struct damon_target *t; struct damos *s; @@ -3589,8 +3644,9 @@ set_prev_continue: * while DAMON is running. For such a case, repeat merging until the limit is * met while increasing @threshold up to possible maximum level. */ -static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold, - unsigned long sz_limit) +static noinline_for_stack void kdamond_merge_regions(struct damon_ctx *c, + unsigned int threshold, + unsigned long sz_limit) { struct damon_target *t; unsigned int nr_regions; @@ -3734,7 +3790,7 @@ static void damon_split_some_regions(struct damon_ctx *ctx, * split was unnecessarily made, later 'kdamond_merge_regions()' will revert * it. */ -static void kdamond_split_regions(struct damon_ctx *ctx) +static noinline_for_stack void kdamond_split_regions(struct damon_ctx *ctx) { struct damon_target *t; unsigned long nr_regions = 0; diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c index 273efa3c913e..64e086985eb5 100644 --- a/mm/damon/lru_sort.c +++ b/mm/damon/lru_sort.c @@ -233,12 +233,13 @@ static int damon_lru_sort_add_quota_goals(struct damos *hot_scheme, if (!active_mem_bp) return 0; - goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, active_mem_bp); + goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, false, + active_mem_bp); if (!goal) return -ENOMEM; damos_add_quota_goal(&hot_scheme->quota, goal); /* aim 0.2 % goal conflict, to keep little ping pong */ - goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP, + goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP, false, 10000 - active_mem_bp + 2); if (!goal) return -ENOMEM; diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 77366f42b3e5..2f3bf86a221b 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -140,9 +140,11 @@ void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, { bool referenced = false; pte_t entry = huge_ptep_get(mm, addr, pte); - struct folio *folio = pfn_folio(pte_pfn(entry)); + struct folio *folio; - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + return; referenced = damon_hugetlb_ptep_mkold(pte, mm, vma, addr, &entry); if (mmu_notifier_clear_young(mm, addr, @@ -164,7 +166,7 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, struct damos *s) { int freq_subscore; - unsigned int age_in_sec; + u64 age_in_sec; int age_in_log, age_subscore; unsigned int freq_weight = s->quota.weight_nr_accesses; unsigned int age_weight = s->quota.weight_age; @@ -174,7 +176,8 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, DAMON_MAX_SUBSCORE, damon_nr_samples_per_aggr(&c->attrs)); - age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000; + age_in_sec = div_u64((u64)r->age * c->attrs.aggr_interval, + USEC_PER_SEC); if (age_in_sec) age_in_log = min_t(int, ilog2(age_in_sec) + 1, DAMON_MAX_AGE_IN_LOG); diff --git a/mm/damon/reclaim.c b/mm/damon/reclaim.c index 42a2c9cb1343..014b0779ea6d 100644 --- a/mm/damon/reclaim.c +++ b/mm/damon/reclaim.c @@ -233,7 +233,7 @@ static int damon_reclaim_apply_parameters(void) damon_set_schemes(param_ctx, &scheme, 1); if (quota_mem_pressure_us) { - goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US, + goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US, false, quota_mem_pressure_us); if (!goal) goto out; @@ -241,7 +241,8 @@ static int damon_reclaim_apply_parameters(void) } if (quota_autotune_feedback) { - goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000); + goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, false, + 10000); if (!goal) goto out; goal->current_value = quota_autotune_feedback; diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index bfb6f0bc3f21..8f083611741f 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -1220,6 +1220,7 @@ static const struct kobj_type damon_sysfs_watermarks_ktype = { struct damos_sysfs_quota_goal { struct kobject kobj; enum damos_quota_goal_metric metric; + bool complement; unsigned long target_value; unsigned long current_value; int nid; @@ -1316,6 +1317,30 @@ static ssize_t target_metric_store(struct kobject *kobj, return -EINVAL; } +static ssize_t complement_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct damos_sysfs_quota_goal *goal = container_of(kobj, + struct damos_sysfs_quota_goal, kobj); + + return sysfs_emit(buf, "%c\n", goal->complement ? 'Y' : 'N'); +} + +static ssize_t complement_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t count) +{ + struct damos_sysfs_quota_goal *goal = container_of(kobj, + struct damos_sysfs_quota_goal, kobj); + bool complement; + int err = kstrtobool(buf, &complement); + + if (err) + return err; + + goal->complement = complement; + return count; +} + static ssize_t target_value_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -1424,6 +1449,9 @@ static void damos_sysfs_quota_goal_release(struct kobject *kobj) static struct kobj_attribute damos_sysfs_quota_goal_target_metric_attr = __ATTR_RW_MODE(target_metric, 0600); +static struct kobj_attribute damos_sysfs_quota_goal_complement_attr = + __ATTR_RW_MODE(complement, 0600); + static struct kobj_attribute damos_sysfs_quota_goal_target_value_attr = __ATTR_RW_MODE(target_value, 0600); @@ -1438,6 +1466,7 @@ static struct kobj_attribute damos_sysfs_quota_goal_path_attr = static struct attribute *damos_sysfs_quota_goal_attrs[] = { &damos_sysfs_quota_goal_target_metric_attr.attr, + &damos_sysfs_quota_goal_complement_attr.attr, &damos_sysfs_quota_goal_target_value_attr.attr, &damos_sysfs_quota_goal_current_value_attr.attr, &damos_sysfs_quota_goal_nid_attr.attr, @@ -2870,6 +2899,7 @@ static int damos_sysfs_add_quota_score( continue; goal = damos_new_quota_goal(sysfs_goal->metric, + sysfs_goal->complement, sysfs_goal->target_value); if (!goal) return -ENOMEM; diff --git a/mm/damon/tests/core-kunit.h b/mm/damon/tests/core-kunit.h index df84d9cc7d20..43e2f950f8ff 100644 --- a/mm/damon/tests/core-kunit.h +++ b/mm/damon/tests/core-kunit.h @@ -812,6 +812,48 @@ static void damos_test_new_filter(struct kunit *test) damos_destroy_filter(filter); } +static void damos_test_new_scheme_keeps_src_quota(struct kunit *test) +{ + struct damos_access_pattern pattern = {}; + struct damon_target target = {}; + struct damos_quota quota = { + .sz = SZ_64K, + .esz = 123, + .esz_bp = 456, + .total_charged_sz = 789, + .total_charged_ns = 1011, + .charged_sz = 12, + .charged_from = 13, + .charge_target_from = &target, + .charge_addr_from = 14, + }; + struct damos_watermarks wmarks = {}; + struct damos *s; + + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) + kunit_skip(test, "scheme alloc fail"); + KUNIT_EXPECT_EQ(test, s->quota.sz, (unsigned long)SZ_64K); + KUNIT_EXPECT_EQ(test, s->quota.esz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.esz_bp, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_sz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_ns, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_sz, 0ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_from, 0ul); + KUNIT_EXPECT_PTR_EQ(test, s->quota.charge_target_from, NULL); + KUNIT_EXPECT_EQ(test, s->quota.charge_addr_from, 0ul); + KUNIT_EXPECT_EQ(test, quota.esz, 123ul); + KUNIT_EXPECT_EQ(test, quota.esz_bp, 456ul); + KUNIT_EXPECT_EQ(test, quota.total_charged_sz, 789ul); + KUNIT_EXPECT_EQ(test, quota.total_charged_ns, 1011ul); + KUNIT_EXPECT_EQ(test, quota.charged_sz, 12ul); + KUNIT_EXPECT_EQ(test, quota.charged_from, 13ul); + KUNIT_EXPECT_PTR_EQ(test, quota.charge_target_from, &target); + KUNIT_EXPECT_EQ(test, quota.charge_addr_from, 14ul); + damon_destroy_scheme(s); +} + static void damos_test_commit_quota_goal_for(struct kunit *test, struct damos_quota_goal *dst, struct damos_quota_goal *src) @@ -819,6 +861,7 @@ static void damos_test_commit_quota_goal_for(struct kunit *test, damos_commit_quota_goal(dst, src); KUNIT_EXPECT_EQ(test, dst->metric, src->metric); + KUNIT_EXPECT_EQ(test, dst->complement, src->complement); KUNIT_EXPECT_EQ(test, dst->target_value, src->target_value); if (src->metric == DAMOS_QUOTA_USER_INPUT) KUNIT_EXPECT_EQ(test, dst->current_value, src->current_value); @@ -862,6 +905,7 @@ static void damos_test_commit_quota_goal(struct kunit *test) damos_test_commit_quota_goal_for(test, &dst, &(struct damos_quota_goal){ .metric = DAMOS_QUOTA_USER_INPUT, + .complement = true, .target_value = 789, .current_value = 12}); damos_test_commit_quota_goal_for(test, &dst, @@ -910,6 +954,48 @@ static void damos_test_commit_quota_goal(struct kunit *test) }); } +static void damos_test_set_psi_current_val(struct kunit *test) +{ + struct damos s = { + .quota.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST, + }; + struct damos_quota_goal goal = { + .metric = DAMOS_QUOTA_SOME_MEM_PSI_US, + .target_value = 100, + .last_psi_total = U64_MAX, + }; + + /* uninitialized last_psi_total keeps the consist tuner quota */ + damos_set_psi_current_val(1000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 100ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 1000ull); + + /* initialized last_psi_total gives the delta */ + damos_set_psi_current_val(1030, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 30ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 1030ull); + + /* temporal tuner keeps a zero quota */ + s.quota.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL; + s.quota.esz = 0; + goal.last_psi_total = U64_MAX; + damos_set_psi_current_val(2000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 100ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 2000ull); + + /* temporal tuner keeps a non-zero quota */ + s.quota.esz = SZ_64K; + goal.last_psi_total = U64_MAX; + damos_set_psi_current_val(3000, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 0ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 3000ull); + + /* temporal tuner uses the measured PSI delta */ + damos_set_psi_current_val(3250, &goal, &s); + KUNIT_EXPECT_EQ(test, goal.current_value, 250ul); + KUNIT_EXPECT_EQ(test, goal.last_psi_total, 3250ull); +} + static void damos_test_commit_quota_goals_for(struct kunit *test, struct damos_quota_goal *dst_goals, int nr_dst_goals, struct damos_quota_goal *src_goals, int nr_src_goals) @@ -930,6 +1016,7 @@ static void damos_test_commit_quota_goals_for(struct kunit *test, * Make it kfree()-able. */ goal = damos_new_quota_goal(dst_goals[i].metric, + dst_goals[i].complement, dst_goals[i].target_value); if (!goal) goto out; @@ -1572,6 +1659,69 @@ static void damon_test_commit_ctx(struct kunit *test) damon_destroy_ctx(dst); } +static void damon_test_commit_ctx_keeps_quota_for(struct kunit *test, + unsigned long min_region_sz, int expected_err) +{ + struct damos_access_pattern pattern = {}; + struct damos_quota quota = {.sz = SZ_64K}; + struct damos_watermarks wmarks = {}; + struct damon_ctx *src, *dst; + struct damon_target *target; + struct damos *s; + + dst = damon_new_ctx(); + if (!dst) + kunit_skip(test, "dst alloc fail"); + target = damon_new_target(); + if (!target) { + damon_destroy_ctx(dst); + kunit_skip(test, "target alloc fail"); + } + damon_add_target(dst, target); + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) { + damon_destroy_ctx(dst); + kunit_skip(test, "scheme alloc fail"); + } + damon_add_scheme(dst, s); + + /* Copy the parameters before populating dst's runtime quota state. */ + src = damon_new_test_ctx(dst); + if (!src) { + damon_destroy_ctx(dst); + kunit_skip(test, "src alloc fail"); + } + src->min_region_sz = min_region_sz; + s->quota.esz = 123; + s->quota.esz_bp = 456; + s->quota.total_charged_sz = 789; + s->quota.total_charged_ns = 1011; + s->quota.charged_sz = 12; + s->quota.charged_from = 13; + s->quota.charge_target_from = target; + s->quota.charge_addr_from = 14; + + KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), expected_err); + KUNIT_EXPECT_EQ(test, s->quota.esz, 123ul); + KUNIT_EXPECT_EQ(test, s->quota.esz_bp, 456ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_sz, 789ul); + KUNIT_EXPECT_EQ(test, s->quota.total_charged_ns, 1011ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_sz, 12ul); + KUNIT_EXPECT_EQ(test, s->quota.charged_from, 13ul); + KUNIT_EXPECT_PTR_EQ(test, s->quota.charge_target_from, target); + KUNIT_EXPECT_EQ(test, s->quota.charge_addr_from, 14ul); + damon_destroy_ctx(src); + damon_destroy_ctx(dst); +} + +static void damon_test_commit_ctx_keeps_quota(struct kunit *test) +{ + /* Only power of two min_region_sz is allowed. */ + damon_test_commit_ctx_keeps_quota_for(test, 4096, 0); + damon_test_commit_ctx_keeps_quota_for(test, 4095, -EINVAL); +} + static void damon_test_valid_probe_params(struct kunit *test) { struct damon_ctx *ctx; @@ -1633,6 +1783,7 @@ static void damos_test_filter_out(struct kunit *test) struct damon_target *t; struct damon_region *r, *r2; struct damos_filter *f; + int err; f = damos_new_filter(DAMOS_FILTER_TYPE_ADDR, true, false); if (!f) @@ -1653,27 +1804,27 @@ static void damos_test_filter_out(struct kunit *test) damon_add_region(r, t); /* region in the range */ - KUNIT_EXPECT_TRUE(test, damos_filter_match(NULL, t, r, f, 1)); + KUNIT_EXPECT_TRUE(test, damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region before the range */ r->ar.start = 1; r->ar.end = 2; KUNIT_EXPECT_FALSE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region after the range */ r->ar.start = 6; r->ar.end = 8; KUNIT_EXPECT_FALSE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1); /* region started before the range */ r->ar.start = 1; r->ar.end = 4; - KUNIT_EXPECT_FALSE(test, damos_filter_match(NULL, t, r, f, 1)); + KUNIT_EXPECT_FALSE(test, damos_filter_match(NULL, t, r, f, 1, &err)); /* filter should have split the region */ KUNIT_EXPECT_EQ(test, r->ar.start, 1); KUNIT_EXPECT_EQ(test, r->ar.end, 2); @@ -1689,7 +1840,7 @@ static void damos_test_filter_out(struct kunit *test) r->ar.start = 2; r->ar.end = 8; KUNIT_EXPECT_TRUE(test, - damos_filter_match(NULL, t, r, f, 1)); + damos_filter_match(NULL, t, r, f, 1, &err)); /* filter should have split the region */ KUNIT_EXPECT_EQ(test, r->ar.start, 2); KUNIT_EXPECT_EQ(test, r->ar.end, 6); @@ -2242,6 +2393,94 @@ static void damon_test_rand(struct kunit *test) } } +static void damos_test_esz_goal_temporal(struct kunit *test) +{ + struct damos_access_pattern pattern = {}; + struct damos_watermarks wmarks = {}; + struct damos_quota quota = { + .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL, + }; + struct damos_quota_goal *goal; + struct damon_ctx *ctx; + struct damos *s; + + ctx = damon_new_ctx(); + KUNIT_ASSERT_NOT_NULL(test, ctx); + + s = damon_new_scheme(&pattern, DAMOS_STAT, 0, "a, &wmarks, + NUMA_NO_NODE); + if (!s) { + damon_destroy_ctx(ctx); + kunit_skip(test, "scheme alloc fail"); + } + damon_add_scheme(ctx, s); + + goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, false, 10000); + if (!goal) { + damon_destroy_ctx(ctx); + kunit_skip(test, "quota goal alloc fail"); + } + goal->current_value = 0; + damos_add_quota_goal(&s->quota, goal); + + /* The largest size quota the basis-point conversion can hold. */ + s->quota.sz = ULONG_MAX / 10000; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + /* Any larger one saturates instead of wrapping. */ + s->quota.sz = ULONG_MAX / 10000 + 1; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + s->quota.sz = ULONG_MAX; + damos_set_effective_quota(ctx, s); + KUNIT_EXPECT_EQ(test, s->quota.esz, ULONG_MAX / 10000); + + damon_destroy_ctx(ctx); +} + +static void damon_test_last_region_quota_reset(struct kunit *test) +{ + struct damos s = {0}; + struct damon_target *t; + struct damon_region *r1, *r2; + unsigned long min_region_sz = 1; + bool skipped; + + t = damon_new_target(); + if (!t) + kunit_skip(test, "target alloc fail"); + + r1 = damon_new_region(0, 100); + if (!r1) { + damon_free_target(t); + kunit_skip(test, "region 1 alloc fail"); + } + damon_add_region(r1, t); + + r2 = damon_new_region(100, 200); + if (!r2) { + damon_free_target(t); + kunit_skip(test, "region 2 alloc fail"); + } + damon_add_region(r2, t); + + s.quota.charge_target_from = t; + s.quota.charge_addr_from = r1->ar.end; + + skipped = damos_skip_charged_region(t, r2, &s, min_region_sz); + + /* 'r2' is not processed, it should not skip */ + KUNIT_EXPECT_EQ(test, skipped, false); + + /* 'r2' is last region, it should reset charge_{target,addr}_from */ + KUNIT_EXPECT_PTR_EQ(test, s.quota.charge_target_from, NULL); + KUNIT_EXPECT_EQ(test, s.quota.charge_addr_from, 0); + + damon_free_target(t); +} + static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_target), KUNIT_CASE(damon_test_regions), @@ -2259,7 +2498,9 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_mvsum), KUNIT_CASE(damon_test_nr_accesses_mvsum), KUNIT_CASE(damos_test_new_filter), + KUNIT_CASE(damos_test_new_scheme_keeps_src_quota), KUNIT_CASE(damos_test_commit_quota_goal), + KUNIT_CASE(damos_test_set_psi_current_val), KUNIT_CASE(damos_test_commit_quota_goals), KUNIT_CASE(damos_test_commit_quota), KUNIT_CASE(damos_test_commit_dests), @@ -2270,6 +2511,7 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_commit_filter), KUNIT_CASE(damon_test_commit_probes), KUNIT_CASE(damon_test_commit_ctx), + KUNIT_CASE(damon_test_commit_ctx_keeps_quota), KUNIT_CASE(damon_test_valid_probe_params), KUNIT_CASE(damos_test_filter_out), KUNIT_CASE(damos_test_apply_scheme_filtered_sz), @@ -2281,6 +2523,8 @@ static struct kunit_case damon_test_cases[] = { KUNIT_CASE(damon_test_is_last_region), KUNIT_CASE(damon_test_walk_control_obsolete), KUNIT_CASE(damon_test_rand), + KUNIT_CASE(damos_test_esz_goal_temporal), + KUNIT_CASE(damon_test_last_region_quota_reset), {}, }; diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index d5dde97b3cd0..b549496ea8e2 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -411,8 +411,9 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, if (!pte_present(entry)) goto out; - folio = pfn_folio(pte_pfn(entry)); - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + goto out; if (pte_young(entry) || !folio_test_idle(folio) || mmu_notifier_test_young(walk->mm, addr)) @@ -660,8 +661,9 @@ static int damon_va_probe_hugetlb_entry(pte_t *pte, unsigned long hmask, if (!pte_present(entry)) goto out; - folio = pfn_folio(pte_pfn(entry)); - folio_get(folio); + folio = damon_get_monitor_folio(pte_pfn(entry)); + if (!folio) + goto out; damon_va_probe_folio(priv->ctx, priv->r, folio, &entry, NULL, walk->vma->vm_mm); folio_put(folio); diff --git a/mm/folio.c b/mm/folio.c index 47a437e0f7fd..35e242b48870 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -505,7 +505,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED)) + if (vma_test(vma, VMA_LOCKED_BIT)) mlock_new_folio(folio); else folio_add_lru(folio); @@ -621,7 +621,7 @@ static struct page *no_page_table(struct vm_area_struct *vma, * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { struct hstate *h = hstate_vma(vma); if (!hugetlbfs_pagecache_present(h, vma, address)) @@ -1213,7 +1213,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; - if ((gup_flags & FOLL_SPLIT_PMD) && is_vm_hugetlb_page(vma)) + if ((gup_flags & FOLL_SPLIT_PMD) && vma_is_hugetlb(vma)) return -EOPNOTSUPP; if (vma_is_secretmem(vma)) @@ -1836,6 +1836,10 @@ long populate_vma_page_range(struct vm_area_struct *vma, if (!vma_is_accessible(vma)) return -EFAULT; + /* Unreadable VMAs also cannot be faulted in. */ + if (!vma_test(vma, VMA_MAYREAD_BIT)) + return -EFAULT; + gup_flags = FOLL_TOUCH; /* * We want to touch writable mappings with a write fault in order diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d7901b8171b2..bd44d58e8684 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -110,14 +110,6 @@ static inline bool file_thp_enabled(const struct vm_area_struct *vma) return S_ISREG(inode->i_mode); } -/* If returns true, we are unable to access the VMA's folios. */ -static bool vma_is_special_huge(const struct vm_area_struct *vma) -{ - if (vma_is_dax(vma)) - return false; - return vma_test_any(vma, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); -} - static bool vma_file_bypass_thp_tuneables(const struct vm_area_struct *vma, enum tva_type type) { @@ -192,7 +184,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma) || vma_is_special_huge(vma)) + else if (vma_is_dax(vma) || !vma_is_mm_managed(vma)) supported_orders = THP_ORDERS_ALL_SPECIAL_DAX; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; @@ -212,11 +204,13 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, return in_pf ? orders : 0; /* - * khugepaged special VMA and hugetlb VMA. - * Must be checked after dax since some dax mappings may have - * VM_MIXEDMAP set. + * khugepaged moves data from VMAs once collapsed, after they have been + * faulted in, relying on refaulting for file-backed memory. + * + * Only mm-backed mappings can be reliably reconstructed from page + * faults, and hugetlb mappings cannot be collapsed. */ - if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) + if (!in_pf && !smaps && (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma))) return 0; /* @@ -259,12 +253,12 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* * THPeligible bit of smaps should show 1 for proper VMAs even - * though anon_vma is not initialized yet. + * though they don't have an anon rmap yet. * - * Allow page fault since anon_vma may be not initialized until - * the first page fault. + * Allow page fault since the VMA may not have an anon rmap until the + * first page fault. */ - if (!vma->anon_vma) + if (!vma_has_anon_rmap(vma)) return (smaps || in_pf) ? orders : 0; return orders; @@ -2179,7 +2173,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd); - VM_BUG_ON_VMA(!vma->anon_vma, vma); + VM_BUG_ON_VMA(!vma_has_anon_rmap(vma), vma); if (is_huge_zero_pmd(orig_pmd)) { vm_fault_t ret = do_huge_zero_wp_pmd(vmf); @@ -3053,7 +3047,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); - if (vma_is_special_huge(vma)) { + if (!vma_is_mm_managed(vma)) { spin_unlock(ptl); /* No zero page support yet */ } else { @@ -3209,7 +3203,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, */ if (arch_needs_pgtable_deposit()) zap_deposited_table(mm, pmd); - if (vma_is_special_huge(vma)) + if (!vma_is_mm_managed(vma)) return; if (unlikely(pmd_is_migration_entry(old_pmd))) { const softleaf_t old_entry = softleaf_from_pmd(old_pmd); @@ -3420,7 +3414,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, swp_entry = make_readable_device_private_entry( page_to_pfn(page + i)); /* - * Young and dirty bits are not progated via swp_entry + * Young and dirty bits are not propagated via swp_entry */ entry = swp_entry_to_pte(swp_entry); if (soft_dirty) @@ -4378,7 +4372,7 @@ out: * THP pages in the middle of migration, due to allocation issues on either * side. * - * anon_vma_lock is not required to be held, mmap_read_lock() or + * The anon rmap lock is not required to be held, mmap_read_lock() or * mmap_write_lock() should be held. @folio is expected to be locked by the * caller. device-private and non device-private folios are supported along * with folios that are in the swapcache. @folio should also be unmapped and @@ -4776,11 +4770,9 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) { if (vma_is_dax(vma)) return true; - if (vma_is_special_huge(vma)) - return true; - if (vma_test(vma, VMA_IO_BIT)) + if (!vma_is_mm_managed(vma)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; return false; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b017ae1dfd6f..da5d04edf7c1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3,6 +3,8 @@ * Generic hugetlb support. * (C) Nadia Yvette Chambers, April 2004 */ +#define pr_fmt(fmt) "HugeTLB: " fmt + #include <linux/list.h> #include <linux/init.h> #include <linux/mm.h> @@ -23,6 +25,7 @@ #include <linux/mmdebug.h> #include <linux/sched/signal.h> #include <linux/rmap.h> +#include <linux/rcupdate.h> #include <linux/string_choices.h> #include <linux/string_helpers.h> #include <linux/swap.h> @@ -457,7 +460,7 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma) * until the file is removed. Warn in the unlikely case of * allocation failure. */ - pr_warn_once("HugeTLB: unable to allocate vma specific lock\n"); + pr_warn_once("unable to allocate vma specific lock\n"); return; } @@ -986,7 +989,7 @@ void hugetlb_fix_reserve_counts(struct inode *inode) } if (!reserved) - pr_warn("hugetlb: Huge Page Reserved count may go negative.\n"); + pr_warn("Huge Page Reserved count may go negative.\n"); } /* @@ -1147,7 +1150,7 @@ static inline struct resv_map *inode_resv_map(struct inode *inode) static struct resv_map *vma_resv_map(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); if (vma->vm_flags & VM_MAYSHARE) { struct address_space *mapping = vma->vm_file->f_mapping; struct inode *inode = mapping->host; @@ -1162,7 +1165,7 @@ static struct resv_map *vma_resv_map(struct vm_area_struct *vma) static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map) { - VM_WARN_ON_ONCE_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); VM_WARN_ON_ONCE_VMA(vma_test(vma, VMA_MAYSHARE_BIT), vma); set_vma_private_data(vma, (unsigned long)map); @@ -1170,7 +1173,7 @@ static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map) static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags) { - VM_WARN_ON_ONCE_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); VM_WARN_ON_ONCE_VMA(vma_test(vma, VMA_MAYSHARE_BIT), vma); set_vma_private_data(vma, get_vma_private_data(vma) | flags); @@ -1178,7 +1181,7 @@ static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags) static int is_vma_resv_set(struct vm_area_struct *vma, unsigned long flag) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); return (get_vma_private_data(vma) & flag) != 0; } @@ -1192,7 +1195,7 @@ bool __vma_private_lock(struct vm_area_struct *vma) void hugetlb_dup_vma_private(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(!is_vm_hugetlb_page(vma), vma); + VM_WARN_ON_ONCE_VMA(!vma_is_hugetlb(vma), vma); /* * Clear vm_private_data * - For shared mappings this is a per-vma semaphore that may be @@ -2206,6 +2209,11 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, } spin_unlock_irq(&hugetlb_lock); + + if (folio) + lruvec_stat_mod_folio(folio, NR_HUGETLB, + folio_nr_pages(folio)); + return folio; } @@ -2213,24 +2221,30 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback) { - spin_lock_irq(&hugetlb_lock); - if (available_huge_pages(h)) { - struct folio *folio; + struct folio *folio = NULL; + spin_lock_irq(&hugetlb_lock); + if (available_huge_pages(h)) folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, preferred_nid, nmask); - if (folio) { - spin_unlock_irq(&hugetlb_lock); - return folio; - } - } spin_unlock_irq(&hugetlb_lock); - /* We cannot fallback to other nodes, as we could break the per-node pool. */ - if (!allow_alloc_fallback) - gfp_mask |= __GFP_THISNODE; + if (!folio) { + /* + * We cannot fallback to other nodes, as we could break the + * per-node pool. + */ + if (!allow_alloc_fallback) + gfp_mask |= __GFP_THISNODE; + + folio = alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, + nmask); + } + + if (folio) + lruvec_stat_mod_folio(folio, NR_HUGETLB, folio_nr_pages(folio)); - return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); + return folio; } static nodemask_t *policy_mbind_nodemask(gfp_t gfp) @@ -3396,7 +3410,7 @@ static unsigned long __init hugetlb_free_cross_zone_pages(struct hstate *h, int char buf[32]; string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); - pr_warn("HugeTLB: freed %lu cross-zone hugepages of size %s on node %d.\n", + pr_warn("freed %lu cross-zone hugepages of size %s on node %d.\n", freed, buf, nid); } @@ -3441,8 +3455,8 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) if (i == h->max_huge_pages_node[nid]) return; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_warn("allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", h->max_huge_pages_node[nid], buf, nid, i); h->max_huge_pages -= (h->max_huge_pages_node[nid] - i); h->max_huge_pages_node[nid] = i; @@ -3468,8 +3482,8 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, if (allocated < h->max_huge_pages) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_warn("allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", h->max_huge_pages, buf, allocated); h->max_huge_pages = allocated; } @@ -3584,7 +3598,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) jiffies_end = jiffies; - pr_info("HugeTLB: allocation took %dms with hugepage_allocation_threads=%ld\n", + pr_info("allocation took %dms with hugepage_allocation_threads=%ld\n", jiffies_to_msecs(jiffies_end - jiffies_start), hugepage_allocation_threads); @@ -3612,7 +3626,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) */ if (hstate_is_gigantic(h) && hugetlb_cma_total_size() && !hugetlb_early_cma(h)) { - pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); + pr_warn_once("hugetlb_cma is enabled, skip boot time allocation\n"); return; } @@ -3679,10 +3693,10 @@ static void __init report_hugepages(void) for_each_hstate(h) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_info("registered %s page size, pre-allocated %ld pages\n", buf, h->nr_huge_pages); - pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", + pr_info("%d KiB vmemmap can be freed for a %s page\n", hugetlb_vmemmap_optimizable_size(h) / SZ_1K, buf); } } @@ -4022,7 +4036,7 @@ long demote_pool_huge_page(struct hstate *src, nodemask_t *nodes_allowed, /* We should never get here if no demote order */ if (!src->demote_order) { - pr_warn("HugeTLB: NULL demote order passed to demote_pool_huge_page.\n"); + pr_warn("NULL demote order passed to demote_pool_huge_page.\n"); return -EINVAL; /* internal error */ } dst = size_to_hstate(PAGE_SIZE << src->demote_order); @@ -4167,7 +4181,7 @@ static int __init hugetlb_init(void) if (!hugepages_supported()) { if (hugetlb_max_hstate || default_hstate_max_huge_pages) - pr_warn("HugeTLB: huge pages not supported, ignoring associated command-line parameters\n"); + pr_warn("huge pages not supported, ignoring associated command-line parameters\n"); return 0; } @@ -4191,10 +4205,10 @@ static int __init hugetlb_init(void) char buf[32]; string_get_size(huge_page_size(&default_hstate), - 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: Ignoring hugepages=%lu associated with %s page size\n", + 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_warn("Ignoring hugepages=%lu associated with %s page size\n", default_hstate.max_huge_pages, buf); - pr_warn("HugeTLB: Using hugepages=%lu for number of default huge pages\n", + pr_warn("Using hugepages=%lu for number of default huge pages\n", default_hstate_max_huge_pages); } default_hstate.max_huge_pages = @@ -4219,7 +4233,7 @@ static int __init hugetlb_init(void) num_fault_mutexes = 1; #endif hugetlb_fault_mutex_table = - kmalloc_objs(struct mutex, num_fault_mutexes); + kvmalloc_objs(struct mutex, num_fault_mutexes); BUG_ON(!hugetlb_fault_mutex_table); for (i = 0; i < num_fault_mutexes; i++) @@ -4334,12 +4348,12 @@ static int __init hugepages_setup(char *s) char *p = s; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring hugepages=%s cmdline\n", s); + pr_warn("hugepages unsupported, ignoring hugepages=%s cmdline\n", s); return 0; } if (!parsed_valid_hugepagesz) { - pr_warn("HugeTLB: hugepages=%s does not follow a valid hugepagesz, ignoring\n", s); + pr_warn("hugepages=%s does not follow a valid hugepagesz, ignoring\n", s); parsed_valid_hugepagesz = true; return -EINVAL; } @@ -4356,7 +4370,7 @@ static int __init hugepages_setup(char *s) mhp = &parsed_hstate->max_huge_pages; if (mhp == last_mhp) { - pr_warn("HugeTLB: hugepages= specified twice without interleaving hugepagesz=, ignoring hugepages=%s\n", s); + pr_warn("hugepages= specified twice without interleaving hugepagesz=, ignoring hugepages=%s\n", s); return 1; } @@ -4367,7 +4381,7 @@ static int __init hugepages_setup(char *s) /* Parameter is node format */ if (p[count] == ':') { if (!hugetlb_node_alloc_supported()) { - pr_warn("HugeTLB: architecture can't support node specific alloc, ignoring!\n"); + pr_warn("architecture can't support node specific alloc, ignoring!\n"); return 1; } if (tmp >= MAX_NUMNODES || !node_online(tmp)) @@ -4400,7 +4414,7 @@ static int __init hugepages_setup(char *s) return 0; invalid: - pr_warn("HugeTLB: Invalid hugepages parameter %s\n", p); + pr_warn("Invalid hugepages parameter %s\n", p); hugepages_clear_pages_in_node(); return -EINVAL; } @@ -4419,7 +4433,7 @@ static int __init hugepagesz_setup(char *s) struct hstate *h; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring hugepagesz=%s cmdline\n", s); + pr_warn("hugepages unsupported, ignoring hugepagesz=%s cmdline\n", s); return 0; } @@ -4427,7 +4441,7 @@ static int __init hugepagesz_setup(char *s) size = (unsigned long)memparse(s, NULL); if (!arch_hugetlb_valid_size(size)) { - pr_err("HugeTLB: unsupported hugepagesz=%s\n", s); + pr_err("unsupported hugepagesz=%s\n", s); return -EINVAL; } @@ -4442,7 +4456,7 @@ static int __init hugepagesz_setup(char *s) */ if (!parsed_default_hugepagesz || h != &default_hstate || default_hstate.max_huge_pages) { - pr_warn("HugeTLB: hugepagesz=%s specified twice, ignoring\n", s); + pr_warn("hugepagesz=%s specified twice, ignoring\n", s); return -EINVAL; } @@ -4472,21 +4486,21 @@ static int __init default_hugepagesz_setup(char *s) int i; if (!hugepages_supported()) { - pr_warn("HugeTLB: hugepages unsupported, ignoring default_hugepagesz=%s cmdline\n", + pr_warn("hugepages unsupported, ignoring default_hugepagesz=%s cmdline\n", s); return 0; } parsed_valid_hugepagesz = false; if (parsed_default_hugepagesz) { - pr_err("HugeTLB: default_hugepagesz previously specified, ignoring %s\n", s); + pr_err("default_hugepagesz previously specified, ignoring %s\n", s); return -EINVAL; } size = (unsigned long)memparse(s, NULL); if (!arch_hugetlb_valid_size(size)) { - pr_err("HugeTLB: unsupported default_hugepagesz=%s\n", s); + pr_err("unsupported default_hugepagesz=%s\n", s); return -EINVAL; } @@ -4751,7 +4765,7 @@ static void hugetlb_vm_op_open(struct vm_area_struct *vma) vma->vm_private_data = NULL; hugetlb_vma_lock_alloc(vma); } else { - pr_warn("HugeTLB: vma_lock already exists in %s.\n", __func__); + pr_warn("vma_lock already exists in %s.\n", __func__); } } else { hugetlb_vma_lock_alloc(vma); @@ -4959,10 +4973,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, raw_write_seqcount_begin(&src->write_protect_seq); } else { /* - * For shared mappings the vma lock must be held before - * calling hugetlb_walk() in the src vma. Otherwise, the - * returned ptep could go away if part of a shared pmd and - * another thread calls huge_pmd_unshare. + * Shared mappings only need copying when we could have + * information stored in page tables (e.g., userfaultfd PTE + * bits or markers), which is incompatible with PMD sharing. + * Still take the hugetlb VMA lock for now to satisfy + * hugetlb_walk()'s locking requirements. */ hugetlb_vma_lock_read(src_vma); } @@ -4970,6 +4985,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; + + if (WARN_ON_ONCE(want_pmd_share(src_vma, addr) || + want_pmd_share(dst_vma, addr))) { + ret = -EINVAL; + break; + } + src_pte = hugetlb_walk(src_vma, addr, sz); if (!src_pte) { addr |= last_addr_mask; @@ -4981,14 +5003,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, break; } -#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING - /* If the pagetables are shared, there is nothing to do */ - if (ptdesc_pmd_is_shared(virt_to_ptdesc(dst_pte))) { - addr |= last_addr_mask; - continue; - } -#endif - dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -5265,7 +5279,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long last_addr_mask; i_mmap_assert_write_locked(vma->vm_file->f_mapping); - WARN_ON(!is_vm_hugetlb_page(vma)); + WARN_ON(!vma_is_hugetlb(vma)); BUG_ON(start & ~huge_page_mask(h)); BUG_ON(end & ~huge_page_mask(h)); @@ -5309,7 +5323,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, * drop the uffd-wp bit in this zap, then replace the * pte with a marker. */ - if (pte_swp_uffd_any(pte) && + if (userfaultfd_wp(vma) && pte_swp_uffd_any(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) set_huge_pte_at(mm, address, ptep, make_pte_marker(PTE_MARKER_UFFD_WP), @@ -5345,7 +5359,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, if (huge_pte_dirty(pte)) folio_mark_dirty(folio); /* Leave a uffd-wp pte marker if needed */ - if (huge_pte_uffd(pte) && + if (userfaultfd_wp(vma) && huge_pte_uffd(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) set_huge_pte_at(mm, address, ptep, make_pte_marker(PTE_MARKER_UFFD_WP), @@ -5385,7 +5399,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, int rc = vma_needs_reservation(h, vma, address); if (rc < 0) - /* Pressumably allocate_file_region_entries failed + /* Presumably allocate_file_region_entries failed * to allocate a file_region struct. Clear * hugetlb_restore_reserve so that global reserve * count will not be incremented by free_huge_folio. @@ -5415,10 +5429,12 @@ void __hugetlb_zap_begin(struct vm_area_struct *vma, if (!vma->vm_file) /* hugetlbfs_file_mmap error */ return; - adjust_range_if_pmd_sharing_possible(vma, start, end); hugetlb_vma_lock_write(vma); - if (vma->vm_file) + if (vma->vm_file) { i_mmap_lock_write(vma->vm_file->f_mapping); + if (hugetlbfs_pmd_sharing_active(file_inode(vma->vm_file))) + adjust_range_if_pmd_sharing_possible(vma, start, end); + } } void __hugetlb_zap_end(struct vm_area_struct *vma, @@ -5457,7 +5473,10 @@ void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, start, end); - adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); + i_mmap_assert_write_locked(vma->vm_file->f_mapping); + if (hugetlbfs_pmd_sharing_active(file_inode(vma->vm_file))) + adjust_range_if_pmd_sharing_possible(vma, &range.start, + &range.end); mmu_notifier_invalidate_range_start(&range); tlb_gather_mmu(&tlb, vma->vm_mm); @@ -5482,6 +5501,8 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping; pgoff_t pgoff; + VM_WARN_ON_ONCE(!folio_test_anon(folio)); + /* * vm_pgoff is in PAGE_SIZE units, hence the different calculation * from page cache lookup which is in HPAGE_SIZE units. @@ -5516,10 +5537,17 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, * could insert a zeroed page instead of the data existing * from the time of fork. This would look like data corruption */ - if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) - unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), - folio, 0); + if (is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) + continue; + + /* + * The address in other VMAs can differ. Even though our old + * folio is anon, we always operate on the underlying hugetlb + * file. + */ + address = vma_filebacked_address(iter_vma, pgoff, 1); + unmap_hugepage_range(iter_vma, address, + address + huge_page_size(h), folio, 0); } i_mmap_unlock_write(mapping); } @@ -5598,7 +5626,7 @@ retry_avoidcopy: * In order to determine where this is a COW on a MAP_PRIVATE mapping it * is enough to check whether the old_folio is anonymous. This means that * the reserve for this address was consumed. If reserves were used, a - * partial faulted mapping at the fime of fork() could consume its reserves + * partial faulted mapping at the time of fork() could consume its reserves * on COW instead of the full address range. */ if (is_vma_resv_set(vma, HPAGE_RESV_OWNER) && @@ -5666,7 +5694,7 @@ retry_avoidcopy: /* * When the original hugepage is shared one, it does not have - * anon_vma prepared. + * an anon rmap prepared. */ ret = __vmf_anon_prepare(vmf); if (unlikely(ret)) @@ -5972,8 +6000,11 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, /* * If this pte was previously wr-protected, keep it wr-protected even * if populated. + * + * TODO: fix all cases where uffd-wp markers remain in VMAs that are no + * longer registered with uffd. */ - if (unlikely(pte_is_uffd_wp_marker(vmf->orig_pte))) + if (unlikely(userfaultfd_wp(vma) && pte_is_uffd_wp_marker(vmf->orig_pte))) new_pte = huge_pte_mkuffd(new_pte); set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h)); @@ -5991,8 +6022,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, ret = hugetlb_wp(vmf); } - spin_unlock(vmf->ptl); - /* * Only set hugetlb_migratable in newly allocated pages. Existing pages * found in the pagecache may not have hugetlb_migratable if they have @@ -6003,6 +6032,12 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, if (folio_locked) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(vmf->ptl); out: hugetlb_vma_unlock_read(vma); @@ -6156,16 +6191,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * NUMA hinting), so let normal hugetlb fault handling proceed. */ if (pte_protnone(vmf.orig_pte) && vma_is_accessible(vma) && - userfaultfd_rwp(vma) && huge_pte_uffd(vmf.orig_pte)) { + huge_pte_uffd(vmf.orig_pte)) { spinlock_t *ptl; pte_t pte; /* Sync: drop hugetlb locks before blocking in handle_userfault() */ - if (!userfaultfd_rwp_async(vma)) + if (userfaultfd_rwp(vma) && !userfaultfd_rwp_async(vma)) return hugetlb_handle_userfault(&vmf, mapping, VM_UFFD_RWP); ptl = huge_pte_lock(h, mm, vmf.pte); pte = huge_ptep_get(mm, vmf.address, vmf.pte); + /* + * TODO: fix all cases where uffd PTE bits remain in VMAs that + * are no longer registered with uffd. + */ if (pte_protnone(pte) && huge_pte_uffd(pte)) { unsigned int shift = huge_page_shift(h); @@ -6213,15 +6252,19 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) && + if (huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { - if (!userfaultfd_wp_async(vma)) { + if (userfaultfd_wp(vma) && !userfaultfd_wp_async(vma)) { spin_unlock(vmf.ptl); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return handle_userfault(&vmf, VM_UFFD_WP); } + /* + * TODO: fix all cases where uffd PTE bits remain in VMAs that + * are no longer registered with uffd. + */ vmf.orig_pte = huge_pte_clear_uffd(vmf.orig_pte); set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, huge_page_size(hstate_vma(vma))); @@ -6516,11 +6559,16 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, /* No need to invalidate - it was non-present before */ update_mmu_cache(dst_vma, dst_addr, dst_pte); - spin_unlock(ptl); if (!is_continue) folio_set_hugetlb_migratable(folio); if (vm_shared || is_continue) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(ptl); ret = 0; out: return ret; @@ -7060,6 +7108,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, if (pud_none(*pud)) { pud_populate(mm, pud, (pmd_t *)((unsigned long)spte & PAGE_MASK)); + hugetlbfs_pmd_sharing_inc(file_inode(vma->vm_file)); mm_inc_nr_pmds(mm); } else { ptdesc_pmd_pts_dec(virt_to_ptdesc(spte)); @@ -7091,6 +7140,7 @@ static int __huge_pmd_unshare(struct mmu_gather *tlb, pud_clear(pud); tlb_unshare_pmd_ptdesc(tlb, virt_to_ptdesc(ptep), addr); + hugetlbfs_pmd_sharing_dec(file_inode(vma->vm_file)); mm_dec_nr_pmds(mm); return 1; @@ -7356,12 +7406,36 @@ void folio_putback_hugetlb(struct folio *folio) folio_put(folio); } +static void move_hugetlb_lruvec_stat(struct folio *old_folio, + struct folio *new_folio) +{ + struct mem_cgroup *memcg; + long nr_pages = folio_nr_pages(old_folio); + int old_nid = folio_nid(old_folio); + int new_nid = folio_nid(new_folio); + + if (old_nid == new_nid) + return; + + guard(rcu)(); + + memcg = folio_memcg(new_folio); + if (!memcg) + return; + + mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)), + NR_HUGETLB, -nr_pages); + mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)), + NR_HUGETLB, nr_pages); +} + void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, enum migrate_reason reason) { struct hstate *h = folio_hstate(old_folio); hugetlb_cgroup_migrate(old_folio, new_folio); + move_hugetlb_lruvec_stat(old_folio, new_folio); folio_set_owner_migrate_reason(new_folio, reason); /* @@ -7491,6 +7565,15 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) */ void fixup_hugetlb_reservations(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) - clear_vma_resv_huge_pages(vma); + if (!vma_is_hugetlb(vma)) + return; + + /* + * Rmap walks might still find this VMA and take the lock + * inside the reservation map for __vma_private_lock(). Don't clear + * the reservation map while that is possible. + */ + i_mmap_lock_write(vma->vm_file->f_mapping); + clear_vma_resv_huge_pages(vma); + i_mmap_unlock_write(vma->vm_file->f_mapping); } diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 326a54b4d991..c89e4c847575 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -462,6 +462,8 @@ static void __init hugetlb_register_all_nodes(void) { int nid; + hugetlb_sysfs_initialized = true; + for_each_online_node(nid) hugetlb_register_node(node_devices[nid]); } @@ -495,8 +497,5 @@ void __init hugetlb_sysfs_init(void) pr_err("HugeTLB: Unable to add hstate %s\n", h->name); } -#ifdef CONFIG_NUMA - hugetlb_sysfs_initialized = true; -#endif hugetlb_register_all_nodes(); } diff --git a/mm/internal.h b/mm/internal.h index 8c6c9db56aa8..566cbd2dce48 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -7,6 +7,7 @@ #ifndef __MM_INTERNAL_H #define __MM_INTERNAL_H +#include <linux/file.h> #include <linux/fs.h> #include <linux/khugepaged.h> #include <linux/mm.h> @@ -218,6 +219,24 @@ static inline void *folio_raw_mapping(const struct folio *folio) } /* + * If the VMA has a close hook then close it, and since closing it might leave + * it in an inconsistent state which makes the use of any hooks suspect, clear + * them down by installing dummy empty hooks. + */ +static inline void vma_close(struct vm_area_struct *vma) +{ + if (vma->vm_ops && vma->vm_ops->close) { + vma->vm_ops->close(vma); + + /* + * The mapping is in an inconsistent state, and no further hooks + * may be invoked upon it. + */ + vma->vm_ops = &vma_dummy_vm_ops; + } +} + +/* * This is a file-backed mapping, and is about to be memory mapped - invoke its * mmap hook and safely handle error conditions. On error, VMA hooks will be * mutated. @@ -229,8 +248,12 @@ static inline void *folio_raw_mapping(const struct folio *folio) */ static inline int mmap_file(struct file *file, struct vm_area_struct *vma) { - int err = vfs_mmap(file, vma); + const unsigned long orig_start = vma->vm_start; + const unsigned long orig_end = vma->vm_end; + const vma_flags_t orig_flags = vma->flags; + int err; + err = vfs_mmap(file, vma); /* * Either we tried to call the file hook for mmap() and an error arose * or a driver set vma->vm_ops = NULL intending there to be no VMA @@ -243,26 +266,16 @@ static inline int mmap_file(struct file *file, struct vm_area_struct *vma) */ if (unlikely(err || !vma->vm_ops)) vma->vm_ops = &vma_dummy_vm_ops; + if (unlikely(err)) + return err; - return err; -} - -/* - * If the VMA has a close hook then close it, and since closing it might leave - * it in an inconsistent state which makes the use of any hooks suspect, clear - * them down by installing dummy empty hooks. - */ -static inline void vma_close(struct vm_area_struct *vma) -{ - if (vma->vm_ops && vma->vm_ops->close) { - vma->vm_ops->close(vma); - - /* - * The mapping is in an inconsistent state, and no further hooks - * may be invoked upon it. - */ - vma->vm_ops = &vma_dummy_vm_ops; + err = mmap_hook_validate(orig_start, orig_end, &orig_flags, vma); + if (unlikely(err)) { + vma->vm_start = orig_start; + vma->vm_end = orig_end; } + + return err; } /* unmap_vmas is in mm/memory.c */ @@ -330,7 +343,7 @@ void unlink_anon_vmas(struct vm_area_struct *vma); static inline int anon_vma_prepare(struct vm_area_struct *vma) { - if (likely(vma->anon_vma)) + if (likely(vma_has_anon_rmap(vma))) return 0; return __anon_vma_prepare(vma); @@ -616,8 +629,8 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); -bool truncate_inode_partial_folio(struct folio *folio, loff_t start, - loff_t end); +bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, + loff_t lend, pgoff_t *pstart, pgoff_t *pend); long mapping_evict_folio(struct address_space *mapping, struct folio *folio); unsigned long mapping_try_invalidate(struct address_space *mapping, pgoff_t start, pgoff_t end, unsigned long *nr_failed); @@ -962,15 +975,7 @@ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, struct vm_area_struct *vma) { - /* - * The VM_SPECIAL check here serves two purposes. - * 1) VM_IO check prevents migration from double-counting during mlock. - * 2) Although mmap_region() and mlock_fixup() take care that VM_LOCKED - * is never left set on a VM_SPECIAL vma, there is an interval while - * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may - * still be set while VM_SPECIAL bits are added: so ignore it then. - */ - if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED)) + if (vma_test(vma, VMA_LOCKED_BIT)) mlock_folio(folio); } @@ -987,7 +992,12 @@ static inline void munlock_vma_folio(struct folio *folio, * always munlock the folio and page reclaim will correct it * if it's wrong. */ - if (unlikely(vma->vm_flags & VM_LOCKED)) + /* + * VMA_LOCKONFAULT_BIT alone marks an mlock walk in progress, see + * mlock_vma_pages_range(). An unmap racing with the walk must still + * munlock folios the walk has already counted. + */ + if (unlikely(vma_test_any_mask(vma, VMA_LOCKED_MASK))) munlock_folio(folio); } @@ -1107,11 +1117,12 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, static inline bool vma_supports_mlock(const struct vm_area_struct *vma) { - if (vma_test_any_mask(vma, VMA_SPECIAL_FLAGS)) + if (!vma_is_mm_backed(vma)) return false; + /* Core mm may discard droppable memory at any time. */ if (vma_test_single_mask(vma, VMA_DROPPABLE)) return false; - if (vma_is_dax(vma) || is_vm_hugetlb_page(vma)) + if (vma_is_dax(vma) || vma_is_hugetlb(vma)) return false; return vma != get_gate_vma(current->mm); } @@ -1506,6 +1517,12 @@ int remap_pfn_range_prepare(struct vm_area_desc *desc); int remap_pfn_range_complete(struct vm_area_struct *vma, struct mmap_action *action); int simple_ioremap_prepare(struct vm_area_desc *desc); +int map_kernel_pages_prepare(struct vm_area_desc *desc); +int map_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action); +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc); +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action); static inline int io_remap_pfn_range_prepare(struct vm_area_desc *desc) { diff --git a/mm/interval_tree.c b/mm/interval_tree.c index 7bbbf15cfbf0..9e24bb99fd80 100644 --- a/mm/interval_tree.c +++ b/mm/interval_tree.c @@ -64,6 +64,51 @@ void mapping_rmap_tree_remove(struct vm_area_struct *vma, __mapping_rmap_tree_remove(vma, &mapping->i_mmap); } +static void mapping_rmap_tree_update_inplace(struct vm_area_struct *vma) +{ + /* Propagate all the way up the tree. */ + __mapping_rmap_tree_augment.propagate(&vma->shared.rb, NULL); +} + +/** + * mapping_rmap_tree_pre_update() - Prepare the file rmap tree for a change to + * be made to @vma. + * @vma: The VMA about to be updated. + * @mapping: The file rmap to which @vma belongs. + * @pgoff_unchanged: Whether @vma's page offset will remain unchanged. + * + * The file rmap lock must be held across the entire update. + */ +void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ + /* If the pgoff has changed, then remove and reinsert afterwards. */ + if (!pgoff_unchanged) + mapping_rmap_tree_remove(vma, mapping); +} + +/** + * mapping_rmap_tree_post_update() - Update the file rmap tree to reflect a + * change that has been made to @vma. + * @vma: The VMA that has been updated. + * @mapping: The file rmap to which @vma belongs. + * @pgoff_unchanged: Whether @vma's page offset remained unchanged. + * + * mapping_rmap_tree_pre_update() must have been called prior to this. + * + * The file rmap lock must be held across the entire update. + */ +void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ + if (pgoff_unchanged) + mapping_rmap_tree_update_inplace(vma); + else + mapping_rmap_tree_insert(vma, mapping); +} + struct vm_area_struct * mapping_rmap_tree_iter_first(struct address_space *mapping, pgoff_t pgoff_start, pgoff_t pgoff_last) @@ -111,6 +156,62 @@ void anon_rmap_tree_remove(struct anon_vma_chain *avc, __anon_rmap_tree_remove(avc, &anon_vma->rb_root); } +static void anon_rmap_tree_update_inplace(struct anon_vma_chain *avc) +{ +#ifdef CONFIG_DEBUG_VM_RB + avc->cached_vma_last = avc_last_pgoff(avc); +#endif + /* Propagate all the way up the tree. */ + __anon_rmap_tree_augment.propagate(&avc->rb, NULL); +} + +/** + * anon_rmap_tree_pre_update_vma() - Prepare the anon rmap trees for a change + * to be made to @vma. + * @vma: The VMA about to be updated, which has an anon rmap assigned and is + * already inserted on its interval trees. + * @anon_pgoff_unchanged: Whether @vma's anonymous page offset will remain + * unchanged. + * + * The anon rmap lock must be held across the entire update. + */ +void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ + struct anon_vma_chain *avc; + + if (anon_pgoff_unchanged) + return; + + /* If the pgoff has changed, then remove and reinsert afterwards. */ + list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) + anon_rmap_tree_remove(avc, avc->anon_vma); +} + +/** + * anon_rmap_tree_post_update_vma() - Update the anon rmap trees to reflect a + * change that has been made to @vma. + * @vma: The VMA that has been updated. + * @anon_pgoff_unchanged: Whether @vma's anonymous page offset remained + * unchanged. + * + * anon_rmap_tree_pre_update_vma() must have been called prior to this. + * + * The anon rmap lock must be held across the entire update. + */ +void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ + struct anon_vma_chain *avc; + + list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) { + if (anon_pgoff_unchanged) + anon_rmap_tree_update_inplace(avc); + else + anon_rmap_tree_insert(avc, avc->anon_vma); + } +} + struct anon_vma_chain * anon_rmap_tree_iter_first(struct anon_vma *anon_vma, pgoff_t pgoff_start, pgoff_t pgoff_last) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 90925c646c4c..55d5c114eadf 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -156,7 +156,7 @@ atomic_t kfence_allocation_gate = ATOMIC_INIT(1); * allocations of the same source filling up the pool. * * Assuming a range of 15%-85% unique allocations in the pool at any point in - * time, the below parameters provide a probablity of 0.02-0.33 for false + * time, the below parameters provide a probability of 0.02-0.33 for false * positive hits respectively: * * P(alloc_traces) = (1 - e^(-HNUM * (alloc_traces / SIZE)) ^ HNUM diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 85ea095906fb..4b71d010b2af 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include <linux/cleanup.h> #include <asm/tlb.h> +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include <trace/events/huge_memory.h> @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepaged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS); static struct kmem_cache *mm_slot_cache __ro_after_init; -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit represents a single occupied (!none/zero) page. */ - DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; err = kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_none = max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; err = kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_swap = max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj, unsigned long max_ptes_shared; err = kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_shared = max_ptes_shared; @@ -367,30 +314,17 @@ static bool pte_none_or_zero(pte_t pte) static unsigned int collapse_max_ptes_none(struct collapse_control *cc, struct vm_area_struct *vma, unsigned int order) { - const unsigned int max_ptes_none = khugepaged_max_ptes_none; + unsigned int max_ptes_none; if (vma && userfaultfd_armed(vma)) return 0; - /* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for PMD collapse, respect the user defined maximum */ if (is_pmd_order(order)) - return max_ptes_none; - /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIMIT, - * scale the maximum number of PTEs to the order of the collapse. - */ - if (max_ptes_none == KHUGEPAGED_MAX_PTES_LIMIT) + return cc->policy.pmd.max_ptes_none; + + /* Below PMD order: all but one PTE of the window, or none */ + max_ptes_none = cc->policy.sub_pmd.max_ptes_none; + if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; - /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, - * emit a warning and return 0. - */ - if (max_ptes_none) - pr_warn_once("mTHP collapse does not support max_ptes_none" - " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); return 0; } @@ -406,20 +340,9 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc, static unsigned int collapse_max_ptes_shared(struct collapse_control *cc, unsigned int order) { - /* - * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared - * anonymous pages. - */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* - * for mTHP collapse do not allow collapsing anonymous memory pages that - * are shared between processes. - */ - if (!is_pmd_order(order)) - return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_shared; + if (is_pmd_order(order)) + return cc->policy.pmd.max_ptes_shared; + return cc->policy.sub_pmd.max_ptes_shared; } /** @@ -434,17 +357,9 @@ static unsigned int collapse_max_ptes_shared(struct collapse_control *cc, static unsigned int collapse_max_ptes_swap(struct collapse_control *cc, unsigned int order) { - /* - * For MADV_COLLAPSE, do not restrict the number PTEs entries or - * pagecache entries that are non-present. - */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for mTHP collapse do not allow any non-present PTEs or pagecache entries */ - if (!is_pmd_order(order)) - return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_swap; + if (is_pmd_order(order)) + return cc->policy.pmd.max_ptes_swap; + return cc->policy.sub_pmd.max_ptes_swap; } int hugepage_madvise(struct vm_area_struct *vma, @@ -476,7 +391,7 @@ int __init khugepaged_init(void) return -ENOMEM; khugepaged_pages_to_scan = HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap = HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared = HPAGE_PMD_NR / 2; @@ -579,17 +494,11 @@ static unsigned long collapse_possible_orders(struct vm_area_struct *vma, return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } -static bool collapse_possible(struct vm_area_struct *vma, - vm_flags_t vm_flags, enum tva_type tva_flags) -{ - return collapse_possible_orders(vma, vm_flags, tva_flags); -} - void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) { - if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() - && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED)) + if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() && + collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED)) __khugepaged_enter(vma->vm_mm); } @@ -623,11 +532,11 @@ void __khugepaged_exit(struct mm_struct *mm) } } -static void collapse_control_init_scan(struct collapse_control *cc) +static void collapse_scan_reset(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); } static void release_pte_folio(struct folio *folio) @@ -731,13 +640,13 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result = SCAN_PAGE_LAZYFREE; goto out; } - /* See collapse_scan_pmd(). */ + /* See collapse_scan_anon(). */ if (folio_maybe_mapped_shared(folio)) { /* * TODO: Support shared pages without leading to further @@ -820,12 +729,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - if (cc->is_khugepaged && + if (cc->policy.anon_require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (unlikely(cc->is_khugepaged && !referenced)) { + if (unlikely(cc->policy.anon_require_referenced && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -991,9 +900,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } -static struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; +static struct collapse_control khugepaged_collapse_control; static bool collapse_scan_abort(int nid, struct collapse_control *cc) { @@ -1029,6 +936,52 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT; } +/* khugepaged collapses on its own initiative, so it obeys its own settings */ +static void collapse_policy_khugepaged(struct collapse_policy *p) +{ + p->pmd.max_ptes_none = READ_ONCE(khugepaged_max_ptes_none); + p->pmd.max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap); + p->pmd.max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared); + + /* + * A sub-PMD window takes no swapped-out and no shared PTE: reading + * pages back or breaking CoW is not worth it for an mTHP. Empty PTEs + * it takes all or nothing, since anything in between would let one + * collapse feed the next. + */ + p->sub_pmd.max_ptes_none = p->pmd.max_ptes_none; + if (p->sub_pmd.max_ptes_none && + p->sub_pmd.max_ptes_none != COLLAPSE_MAX_PTES_LIMIT) + pr_warn_once("mTHP collapse does not support max_ptes_none values other than 0 or %u, defaulting to 0.\n", + COLLAPSE_MAX_PTES_LIMIT); + p->sub_pmd.max_ptes_swap = 0; + p->sub_pmd.max_ptes_shared = 0; + + p->anon_skip_lazyfree = true; + p->anon_require_referenced = true; + p->file_install_pmd = false; + p->file_writeback_dirty = false; + p->gfp = alloc_hugepage_khugepaged_gfpmask(); + p->tva_type = TVA_KHUGEPAGED; +} + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_madvise(struct collapse_policy *p) +{ + p->pmd.max_ptes_none = HPAGE_PMD_NR; + p->pmd.max_ptes_swap = HPAGE_PMD_NR; + p->pmd.max_ptes_shared = HPAGE_PMD_NR; + /* Never read: MADV_COLLAPSE collapses to PMD order only */ + p->sub_pmd = p->pmd; + + p->anon_skip_lazyfree = false; + p->anon_require_referenced = false; + p->file_install_pmd = true; + p->file_writeback_dirty = true; + p->gfp = GFP_TRANSHUGE; + p->tva_type = TVA_FORCED_COLLAPSE; +} + #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -1066,8 +1019,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l struct collapse_control *cc, unsigned int order) { struct vm_area_struct *vma; - enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED : - TVA_FORCED_COLLAPSE; + enum tva_type type = cc->policy.tva_type; if (unlikely(collapse_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; @@ -1094,7 +1046,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l * thp_vma_allowable_orders() may return true for qualified file * vmas. */ - if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap))) + if (expect_anon && (!vma_has_anon_rmap(vma) || !vma_is_anonymous(vma))) return SCAN_PAGE_ANON; return SCAN_SUCCEED; } @@ -1147,7 +1099,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if collapse_scan_pmd() believes it is worthwhile. + * Only done if collapse_scan_anon() believes it is worthwhile. * * For mTHP orders the function bails on the first swap entry, because * faulting pages back in during collapse could re-populate PTEs that @@ -1250,8 +1202,7 @@ out: static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, struct collapse_control *cc, unsigned int order) { - gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : - GFP_TRANSHUGE); + gfp_t gfp = cc->policy.gfp; int node = collapse_find_target_node(cc); struct folio *folio; @@ -1301,8 +1252,8 @@ static pgtable_t alloc_deposit_pte_table(struct mm_struct *mm) * while allocating a THP, as that could trigger direct reclaim/compaction. * Note that the VMA must be rechecked after grabbing the mmap_lock again. */ -static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr, - int referenced, int unmapped, struct collapse_control *cc, +static enum scan_result collapse_huge_page(struct mm_struct *mm, + unsigned long start_addr, struct collapse_control *cc, unsigned int order) { const unsigned long pmd_addr = start_addr & HPAGE_PMD_MASK; @@ -1349,14 +1300,14 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s goto out_nolock; } - if (unmapped) { + if (cc->scan_unmapped) { /* * __collapse_huge_page_swapin() will return with mmap_lock * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd, - referenced, order); + cc->scan_referenced, order); if (result != SCAN_SUCCEED) goto out_nolock; } @@ -1365,7 +1316,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s /* * Prevent all access to pagetables with the exception of * gup_fast later handled by the pmdp_collapse_flush() and the VM - * handled by the anon_vma lock + folio lock. + * handled by the anon rmap lock + folio lock. * * UFFDIO_MOVE is prevented to race as well thanks to the * mmap_lock. @@ -1432,8 +1383,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s } /* - * For PMD collapse all pages are isolated and locked so anon_vma - * rmap can't run anymore. For mTHP collapse the PMD entry has been + * For PMD collapse all pages are isolated and locked so the anon + * rmap walk can't run anymore. For mTHP collapse the PMD entry has been * removed and not all pages are isolated and locked, so we must hold * the lock to prevent neighboring folios from attempting to access * this PMD until its reinstalled. @@ -1509,27 +1460,26 @@ static unsigned int max_order_from_offset(unsigned int offset) } /* - * mthp_collapse() consumes the bitmap that is generated during - * collapse_scan_pmd() to determine what regions and mTHP orders fit best. + * collapse_anon() consumes the bitmap that is generated during + * collapse_scan_anon() to determine what regions and mTHP orders fit best. * - * Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero) - * page. We start at the PMD order and check if it is eligible for collapse; + * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse + * source. We start at the PMD order and check if it is eligible for collapse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. * - * For each of these, we determine how many PTE entries are occupied in the - * range of PTE entries we propose to collapse, then we compare this to a - * threshold number of PTE entries which would need to be occupied for a - * collapse to be permitted at that order (accounting for max_ptes_none). + * For each of these, we count the eligible PTEs in the range we propose to + * collapse, then we compare this to the number of eligible PTEs the range + * would need for a collapse to be permitted at that order (accounting for + * max_ptes_none). * * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. */ -static enum scan_result mthp_collapse(struct mm_struct *mm, - unsigned long address, int referenced, int unmapped, - struct collapse_control *cc, unsigned long enabled_orders) +static enum scan_result collapse_anon(struct mm_struct *mm, + unsigned long address, struct collapse_control *cc) { - unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none; + unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result = SCAN_FAIL; int collapsed = 0; bool alloc_failed = false; @@ -1540,27 +1490,26 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, while (offset < HPAGE_PMD_NR) { nr_ptes = 1UL << order; - if (!test_bit(order, &enabled_orders)) + if (!test_bit(order, &cc->scan_orders)) goto next_order; max_ptes_none = collapse_max_ptes_none(cc, NULL, order); - nr_occupied_ptes = bitmap_weight_from(cc->mthp_present_ptes, offset, + nr_eligible_ptes = bitmap_weight_from(cc->eligible_ptes, offset, offset + nr_ptes); /* - * Swap PTEs accepted during the scan are counted in @unmapped, - * not in the present-PTE bitmap. Account them for the PMD-order - * candidate. + * Swap PTEs accepted during the scan are counted in + * cc->scan_unmapped, not in cc->eligible_ptes. Account them for + * the PMD-order candidate. */ if (is_pmd_order(order)) - nr_occupied_ptes += unmapped; + nr_eligible_ptes += cc->scan_unmapped; - if (nr_occupied_ptes >= nr_ptes - max_ptes_none) { + if (nr_eligible_ptes >= nr_ptes - max_ptes_none) { enum scan_result ret; collapse_address = address + offset * PAGE_SIZE; - ret = collapse_huge_page(mm, collapse_address, referenced, - unmapped, cc, order); + ret = collapse_huge_page(mm, collapse_address, cc, order); switch (ret) { /* Cases where we continue to next collapse candidate */ @@ -1601,8 +1550,8 @@ next_order: * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (cc->scan_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1627,14 +1576,14 @@ done: return last_result; } -static enum scan_result collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, - bool *lock_dropped, struct collapse_control *cc) +static enum scan_result collapse_scan_anon(struct vm_area_struct *vma, + unsigned long start_addr, struct collapse_control *cc) { const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER); const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER); unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER); - enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE; + enum tva_type tva_flags = cc->policy.tva_type; + struct mm_struct *mm = vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1656,17 +1605,17 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out; } - collapse_control_init_scan(cc); + collapse_scan_reset(cc); enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags); /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap - * is then checked again in mthp_collapse() for each attempted order. + * is then checked again in collapse_anon() for each attempted order. */ if (enabled_orders != BIT(HPAGE_PMD_ORDER)) - max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { @@ -1734,7 +1683,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result = SCAN_PAGE_LAZYFREE; failed_pfn = folio_pfn(folio); @@ -1761,8 +1710,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, } } - /* Set bit for occupied pages */ - __set_bit(i, cc->mthp_present_ptes); + /* The scan accepted this PTE as a collapse source */ + __set_bit(i, cc->eligible_ptes); /* * Record which node the original page is from and save this * information to cc->node_load[]. @@ -1800,13 +1749,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } - if (cc->is_khugepaged && + if (cc->policy.anon_require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (cc->is_khugepaged && - (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + if (cc->policy.anon_require_referenced && + (!referenced || + (unmapped && referenced < HPAGE_PMD_NR / 2))) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1814,12 +1763,9 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { - /* collapse_huge_page() expects the lock to be dropped before calling */ - mmap_read_unlock(mm); - result = mthp_collapse(mm, start_addr, referenced, - unmapped, cc, enabled_orders); - /* mmap_lock was released above, set lock_dropped */ - *lock_dropped = true; + cc->scan_orders = enabled_orders; + cc->scan_referenced = referenced; + cc->scan_unmapped = unmapped; } out: trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, @@ -2137,6 +2083,9 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma) * Check vma->anon_vma to exclude MAP_PRIVATE mappings that * got written to. These VMAs are likely not worth removing * page tables from, as PMD-mapping is likely to be split later. + * + * Can't use vma_has_anon_rmap() here as the VMA may be stabilised + * by the file rmap lock. */ if (READ_ONCE(vma->anon_vma)) return false; @@ -2223,9 +2172,9 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) /* * Huge page lock is still held, so normally the page table must - * remain empty; and we have already skipped anon_vma and - * userfaultfd_wp() vmas. But since the mmap_lock is not held, - * it is still possible for a racing userfaultfd_ioctl() or + * remain empty; and we have already skipped vmas with an anon + * rmap and userfaultfd_wp() vmas. But since the mmap_lock is not + * held, it is still possible for a racing userfaultfd_ioctl() or * madvise() to have inserted ptes or markers. Now that we hold * ptlock, repeating the retractable checks protects us from * races against the prior checks. @@ -2658,11 +2607,11 @@ immap_locked: xas_unlock_irq(&xas); /* - * Remove pte page tables, so we can re-fault the page as huge. - * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). + * Remove pte page tables, so we can re-fault the page as huge. A + * caller that wants the PMD mapped now is told to go and do that. */ retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) + if (cc->policy.file_install_pmd) result = SCAN_PTE_MAPPED_HUGEPAGE; folio_unlock(new_folio); @@ -2727,7 +2676,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, present = 0; swap = 0; - collapse_control_init_scan(cc); + collapse_scan_reset(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) @@ -2809,53 +2758,78 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, else cc->progress += HPAGE_PMD_NR; - if (result == SCAN_SUCCEED) { - if (present < HPAGE_PMD_NR - max_ptes_none) { - result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - } else { - result = collapse_file(mm, addr, file, start, cc); - } + if (result == SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none) { + result = SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } - trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result); + trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, + result); return result; } -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) +static void collapse_control_init(struct collapse_control *cc) +{ + cc->progress = 0; + cc->scan_file = NULL; +} + +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc) { - struct mm_struct *mm = vma->vm_mm; - bool triggered_wb = false; enum scan_result result; - struct file *file; pgoff_t pgoff; - mmap_assert_locked(mm); - - if (vma_is_anonymous(vma)) { - result = collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); - goto end; + mmap_assert_locked(vma->vm_mm); + /* A scan that found work has to have been followed by collapse_pmd() */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file = NULL; } - file = get_file(vma->vm_file); - pgoff = linear_page_index(vma, addr); + if (vma_is_anonymous(vma)) + return collapse_scan_anon(vma, addr, cc); - mmap_read_unlock(mm); - *lock_dropped = true; -retry: - result = collapse_scan_file(mm, addr, file, pgoff, cc); + pgoff = linear_page_index(vma, addr); + result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); + /* + * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds + * the PMD folio, and collapse_pmd() still has to retract the PTE table. + */ + if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE) + return result; /* - * For MADV_COLLAPSE, when encountering dirty pages, try to writeback, - * then retry the collapse one time. + * A file collapse works on the page cache and never sees a VMA, so take + * what it needs from this one while it is still here. */ - if (!cc->is_khugepaged && result == SCAN_PAGE_DIRTY_OR_WRITEBACK && + cc->scan_file = get_file(vma->vm_file); + cc->scan_pgoff = pgoff; + return result; +} + +static enum scan_result collapse_pmd(struct mm_struct *mm, + unsigned long addr, enum scan_result result, + struct collapse_control *cc) +{ + struct file *file = cc->scan_file; + bool triggered_wb = false; + pgoff_t pgoff; + + if (!file) + return collapse_anon(mm, addr, cc); + + cc->scan_file = NULL; + pgoff = cc->scan_pgoff; + + /* The scan found the PMD folio in place: nothing to collapse */ + if (result == SCAN_PTE_MAPPED_HUGEPAGE) + goto put; +retry: + result = collapse_file(mm, addr, file, pgoff, cc); + + /* Dirty pages are worth a writeback and one more try, if asked for */ + if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { const loff_t lstart = (loff_t)pgoff << PAGE_SHIFT; const loff_t lend = lstart + HPAGE_PMD_SIZE - 1; @@ -2864,25 +2838,49 @@ retry: triggered_wb = true; goto retry; } +put: fput(file); + /* + * A PMD folio is in the page cache, whether the collapse just put it + * there or found it: retract the PTE table, and map the PMD if asked. + */ if (result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) result = SCAN_ANY_PROCESS; else result = try_collapse_pte_mapped_thp(mm, addr, - !cc->is_khugepaged); + cc->policy.file_install_pmd); if (result == SCAN_PMD_MAPPED) result = SCAN_SUCCEED; mmap_read_unlock(mm); } -end: - if (cc->is_khugepaged && result == SCAN_SUCCEED) - ++khugepaged_pages_collapsed; return result; } +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *lock_dropped, + struct collapse_control *cc) +{ + struct mm_struct *mm = vma->vm_mm; + enum scan_result result; + + result = collapse_scan_pmd(vma, addr, cc); + if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE) + return result; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped = true; + + return collapse_pmd(mm, addr, result, cc); +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2929,7 +2927,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + if (!collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED)) { cc->progress++; continue; } @@ -2956,6 +2955,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, *result = collapse_single_pmd(khugepaged_scan.address, vma, &lock_dropped, cc); + if (*result == SCAN_SUCCEED) + khugepaged_pages_collapsed++; /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; if (lock_dropped) @@ -3022,7 +3023,9 @@ static void khugepaged_do_scan(struct collapse_control *cc) lru_add_drain_all(); - cc->progress = 0; + collapse_control_init(cc); + collapse_policy_khugepaged(&cc->policy); + while (true) { cond_resched(); @@ -3141,13 +3144,8 @@ void set_recommended_min_free_kbytes(void) (unsigned long) nr_free_buffer_pages() / 20); recommended_min <<= (PAGE_SHIFT-10); - if (recommended_min > min_free_kbytes) { - if (user_min_free_kbytes >= 0) - pr_info_ratelimited("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n", - min_free_kbytes, recommended_min); - + if (user_min_free_kbytes < 0 && recommended_min > min_free_kbytes) min_free_kbytes = recommended_min; - } update_wmarks: setup_per_zone_wmarks(); @@ -3237,7 +3235,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); - if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) return -EINVAL; hstart = ALIGN(start, HPAGE_PMD_SIZE); @@ -3249,10 +3247,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, cc = kmalloc_obj(*cc); if (!cc) return -ENOMEM; - cc->is_khugepaged = false; - cc->progress = 0; + collapse_control_init(cc); + collapse_policy_madvise(&cc->policy); - mmgrab(mm); lru_add_drain_all(); for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { @@ -3308,7 +3305,6 @@ out_maybelock: } out_nolock: mmap_assert_locked(mm); - mmdrop(mm); kfree(cc); return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c index 90f427b95a21..3ad5f31d56b6 100644 --- a/mm/kmsan/core.c +++ b/mm/kmsan/core.c @@ -27,6 +27,7 @@ #include <linux/vmalloc.h> #include "../slab.h" +#include "../vmalloc.h" #include "kmsan.h" bool kmsan_enabled __read_mostly; @@ -240,11 +241,8 @@ struct page *kmsan_vmalloc_to_page_or_null(void *vaddr) { struct page *page; - if (!kmsan_internal_is_vmalloc_addr(vaddr) && - !kmsan_internal_is_module_addr(vaddr)) - return NULL; - page = vmalloc_to_page(vaddr); - if (pfn_valid(page_to_pfn(page))) + page = __vmalloc_to_page(vaddr); + if (page && pfn_valid(page_to_pfn(page))) return page; else return NULL; diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 5f1b8053f9fa..a706666db097 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -20,6 +20,8 @@ #include <linux/uaccess.h> #include <linux/usb.h> +#include <asm/tlbflush.h> + #include "../internal.h" #include "../vmalloc.h" #include "../slab.h" @@ -142,6 +144,36 @@ void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end) flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); } +#define KMSAN_IOREMAP_META_ORDER 1 + +static void kmsan_iounmap_pages(unsigned long start, unsigned long end) +{ + unsigned long shadow_start = vmalloc_shadow(start), + shadow_end = vmalloc_shadow(end); + unsigned long origin_start = vmalloc_origin(start), + origin_end = vmalloc_origin(end); + unsigned long v_shadow, v_origin; + struct page *shadow, *origin; + int nr; + + nr = (end - start) / PAGE_SIZE; + v_shadow = shadow_start; + v_origin = origin_start; + for (int i = 0; i < nr; + i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) { + shadow = kmsan_vmalloc_to_page_or_null((void *)v_shadow); + origin = kmsan_vmalloc_to_page_or_null((void *)v_origin); + if (shadow) + __free_pages(shadow, KMSAN_IOREMAP_META_ORDER); + if (origin) + __free_pages(origin, KMSAN_IOREMAP_META_ORDER); + } + __vunmap_range_noflush(shadow_start, shadow_end); + __vunmap_range_noflush(origin_start, origin_end); + flush_tlb_kernel_range(shadow_start, shadow_end); + flush_tlb_kernel_range(origin_start, origin_end); +} + /* * This function creates new shadow/origin pages for the physical pages mapped * into the virtual memory. If those physical pages already had shadow/origin, @@ -154,16 +186,17 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end, gfp_t gfp_mask = GFP_KERNEL | __GFP_ZERO; struct page *shadow, *origin; unsigned long off = 0; - int nr, err = 0, clean = 0, mapped; + unsigned long mapped_end = start; + int nr, err = 0, mapped; if (!kmsan_enabled || kmsan_in_runtime()) return 0; nr = (end - start) / PAGE_SIZE; kmsan_enter_runtime(); - for (int i = 0; i < nr; i++, off += PAGE_SIZE, clean = i) { - shadow = alloc_pages(gfp_mask, 1); - origin = alloc_pages(gfp_mask, 1); + for (int i = 0; i < nr; i++, off += PAGE_SIZE) { + shadow = alloc_pages(gfp_mask, KMSAN_IOREMAP_META_ORDER); + origin = alloc_pages(gfp_mask, KMSAN_IOREMAP_META_ORDER); if (!shadow || !origin) { err = -ENOMEM; goto ret; @@ -177,39 +210,27 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end, goto ret; } shadow = NULL; + mapped_end = start + off + PAGE_SIZE; mapped = __vmap_pages_range_noflush( vmalloc_origin(start + off), vmalloc_origin(start + off + PAGE_SIZE), prot, &origin, PAGE_SHIFT); if (mapped) { - __vunmap_range_noflush( - vmalloc_shadow(start + off), - vmalloc_shadow(start + off + PAGE_SIZE)); err = mapped; goto ret; } origin = NULL; } - /* Page mapping loop finished normally, nothing to clean up. */ - clean = 0; ret: - if (clean > 0) { - /* - * Something went wrong. Clean up shadow/origin pages allocated - * on the last loop iteration, then delete mappings created - * during the previous iterations. - */ + if (err) { if (shadow) - __free_pages(shadow, 1); + __free_pages(shadow, KMSAN_IOREMAP_META_ORDER); if (origin) - __free_pages(origin, 1); - __vunmap_range_noflush( - vmalloc_shadow(start), - vmalloc_shadow(start + clean * PAGE_SIZE)); - __vunmap_range_noflush( - vmalloc_origin(start), - vmalloc_origin(start + clean * PAGE_SIZE)); + __free_pages(origin, KMSAN_IOREMAP_META_ORDER); + + if (mapped_end > start) + kmsan_iounmap_pages(start, mapped_end); } flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); @@ -219,28 +240,11 @@ ret: void kmsan_iounmap_page_range(unsigned long start, unsigned long end) { - unsigned long v_shadow, v_origin; - struct page *shadow, *origin; - int nr; - if (!kmsan_enabled || kmsan_in_runtime()) return; - nr = (end - start) / PAGE_SIZE; kmsan_enter_runtime(); - v_shadow = (unsigned long)vmalloc_shadow(start); - v_origin = (unsigned long)vmalloc_origin(start); - for (int i = 0; i < nr; - i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) { - shadow = kmsan_vmalloc_to_page_or_null((void *)v_shadow); - origin = kmsan_vmalloc_to_page_or_null((void *)v_origin); - __vunmap_range_noflush(v_shadow, vmalloc_shadow(end)); - __vunmap_range_noflush(v_origin, vmalloc_origin(end)); - if (shadow) - __free_pages(shadow, 1); - if (origin) - __free_pages(origin, 1); - } + kmsan_iounmap_pages(start, end); flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); kmsan_leave_runtime(); @@ -747,9 +747,10 @@ static bool ksm_compatible(const struct file *file, vma_flags_t vma_flags) if (vma_flags_test_any(&vma_flags, VMA_SHARED_BIT, VMA_MAYSHARE_BIT, VMA_HUGETLB_BIT)) return false; - if (vma_flags_test_single_mask(&vma_flags, VMA_DROPPABLE)) + if (!vma_flags_is_mm_backed(&vma_flags)) return false; - if (vma_flags_test_any_mask(&vma_flags, VMA_SPECIAL_FLAGS)) + /* Core mm may discard droppable memory at any time. */ + if (vma_flags_test_single_mask(&vma_flags, VMA_DROPPABLE)) return false; if (file_is_dax(file)) return false; @@ -777,7 +778,7 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm, if (ksm_test_exit(mm)) return NULL; vma = vma_lookup(mm, addr); - if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) + if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma_has_anon_rmap(vma)) return NULL; return vma; } @@ -795,7 +796,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item) /* * It is not an accident that whenever we want to break COW - * to undo, we also need to drop a reference to the anon_vma. + * to undo, we also need to drop a reference to the anon rmap. */ put_anon_vma(rmap_item->anon_vma); /* @@ -1241,7 +1242,7 @@ static int unmerge_and_remove_all_rmap_items(void) goto mm_exiting; for_each_vma(vmi, vma) { - if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) + if (!(vma->vm_flags & VM_MERGEABLE) || !vma_has_anon_rmap(vma)) continue; err = break_ksm(vma, vma->vm_start, vma->vm_end, false); if (err) @@ -1415,7 +1416,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out; /* * Some THP functions use the sequence pmdp_huge_clear_flush(), set_pmd_at() - * without holding anon_vma lock for write. So when looking for a + * without holding the anon rmap lock for write. So when looking for a * genuine pmde (in which to find pte), test present and !THP together. */ pmde = pmdp_get_lockless(pmd); @@ -1619,7 +1620,7 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item, /* * We can consider the VMA only while still holding the mmap lock, - * so lock, so reference the anon_vma and calculate the linear + * so reference the anon rmap and calculate the linear * page index early, before stable_tree_append(). If anything goes * wrong that prevents the rmap_item from being added to the * stable_tree, break_cow() will clean it up. @@ -2691,7 +2692,7 @@ next_mm: continue; if (ksm_scan.address < vma->vm_start) ksm_scan.address = vma->vm_start; - if (!vma->anon_vma) + if (!vma_has_anon_rmap(vma)) ksm_scan.address = vma->vm_end; while (ksm_scan.address < vma->vm_end) { @@ -2882,7 +2883,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma) if (!(vma->vm_flags & VM_MERGEABLE)) return 0; - if (vma->anon_vma) { + if (vma_has_anon_rmap(vma)) { err = break_ksm(vma, vma->vm_start, vma->vm_end, true); if (err) return err; @@ -3034,7 +3035,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, if (!(*vm_flags & VM_MERGEABLE)) return 0; /* just ignore the advice */ - if (vma->anon_vma) { + if (vma_has_anon_rmap(vma)) { err = break_ksm(vma, start, end, true); if (err) return err; diff --git a/mm/list_lru.c b/mm/list_lru.c index 8a6dd0a489e1..d1d256822ccc 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -428,7 +428,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, xa_for_each(&lru->xa, index, mlru) { rcu_read_lock(); memcg = mem_cgroup_from_private_id(index); - if (!mem_cgroup_tryget(memcg)) { + if (!memcg || !mem_cgroup_tryget(memcg)) { rcu_read_unlock(); continue; } @@ -587,7 +587,7 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, */ do { /* - * Keep finding the farest parent that wasn't populated + * Keep finding the farthest parent that wasn't populated * until found memcg itself. */ pos = memcg; diff --git a/mm/madvise.c b/mm/madvise.c index bef4de4425f2..75b409e97dbf 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -430,10 +430,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } if (!pageout && pmd_young(orig_pmd)) { - pmdp_invalidate(vma, addr, pmd); - orig_pmd = pmd_mkold(orig_pmd); - - set_pmd_at(mm, addr, pmd, orig_pmd); + pmdp_test_and_clear_young(vma, addr, pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } @@ -454,6 +451,7 @@ huge_unlock: spin_unlock(ptl); if (pageout) reclaim_pages(&folio_list); + cond_resched(); return 0; } @@ -882,7 +880,7 @@ bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv_behavior) int behavior = madv_behavior->behavior; struct madvise_behavior_range *range = &madv_behavior->range; - if (!is_vm_hugetlb_page(vma)) { + if (!vma_is_hugetlb(vma)) { unsigned int forbidden = VM_PFNMAP; if (behavior != MADV_DONTNEED_LOCKED) @@ -1057,19 +1055,25 @@ static long madvise_remove(struct madvise_behavior *madv_behavior) return error; } -static bool is_valid_guard_vma(struct vm_area_struct *vma, bool allow_locked) +static bool is_valid_guard_vma(const struct vm_area_struct *vma, + bool allow_locked) { - vm_flags_t disallowed = VM_SPECIAL | VM_HUGETLB; - /* - * A user could lock after setting a guard range but that's fine, as + * A user could lock after setting a guard range but that's fine as * they'd not be able to fault in. The issue arises when we try to zap * existing locked VMAs. We don't want to do that. */ - if (!allow_locked) - disallowed |= VM_LOCKED; + if (!allow_locked && vma_test(vma, VMA_LOCKED_BIT)) + return false; + /* + * Guard regions require a VMA whose page tables are managed solely by + * core mm, so the VMA must be mm-backed (excluding hugetlb, in which + * guard markers cannot be installed). + */ + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma)) + return false; - return !(vma->vm_flags & disallowed); + return true; } static bool is_guard_pte_marker(pte_t ptent) @@ -1152,13 +1156,13 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior) /* * If anonymous and we are establishing page tables the VMA ought to - * have an anon_vma associated with it. + * have an anon rmap associated with it. * * We will hold an mmap read lock if this is necessary, this is checked * as part of the VMA lock logic. */ if (vma_is_anonymous(vma)) { - VM_WARN_ON_ONCE(!vma->anon_vma && + VM_WARN_ON_ONCE(!vma_has_anon_rmap(vma) && madv_behavior->lock_mode != MADVISE_MMAP_READ_LOCK); err = anon_vma_prepare(vma); @@ -1298,7 +1302,7 @@ static bool is_discard(int behavior) * We are restricted from madvise()'ing mseal()'d VMAs only in very particular * circumstances - discarding of data from read-only anonymous SEALED mappings. * - * This is because users cannot trivally discard data from these VMAs, and may + * This is because users cannot trivially discard data from these VMAs, and may * only do so via an appropriate madvise() call. */ static bool can_madvise_modify(struct madvise_behavior *madv_behavior) @@ -1396,7 +1400,7 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) new_flags |= VM_DONTCOPY; break; case MADV_DOFORK: - if (new_flags & VM_SPECIAL) + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma)) return -EINVAL; new_flags &= ~VM_DONTCOPY; break; @@ -1415,8 +1419,11 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) new_flags |= VM_DONTDUMP; break; case MADV_DODUMP: - if ((!is_vm_hugetlb_page(vma) && (new_flags & VM_SPECIAL)) || - (new_flags & VM_DROPPABLE)) + /* Only mm-backed memory can be meaningfully dumped. */ + if (!vma_is_mm_backed(vma)) + return -EINVAL; + /* Core mm may discard droppable memory at any time. */ + if (new_flags & VM_DROPPABLE) return -EINVAL; new_flags &= ~VM_DONTDUMP; break; @@ -1617,11 +1624,11 @@ static bool is_vma_lock_sufficient(struct vm_area_struct *vma, * anon_vma_prepare() explicitly requires an mmap lock for * serialisation, so we cannot use a VMA lock in this case. * - * Note we might race with anon_vma being set, however this makes this - * check overly paranoid which is safe. + * Note we might race with the anon rmap being assigned, however this + * makes this check overly paranoid which is safe. */ if (vma_is_anonymous(vma) && - prepares_anon_vma(madv_behavior->behavior) && !vma->anon_vma) + prepares_anon_vma(madv_behavior->behavior) && !vma_has_anon_rmap(vma)) return false; return true; diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index bf2c7d53b01b..b7f286888507 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -268,9 +268,10 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg) */ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) { - struct mem_cgroup *memcg, *swap_memcg; + struct mem_cgroup *memcg; struct obj_cgroup *objcg; unsigned int nr_entries; + unsigned short private_id; VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); @@ -293,25 +294,23 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci) /* * In case the memcg owning these pages has been offlined and doesn't * have an ID allocated to it anymore, charge the closest online - * ancestor for the swap instead and transfer the memory+swap charge. + * ancestor for the swap instead and cancel the memory+swap charge + * if the ID refers to the root memcg. */ nr_entries = folio_nr_pages(folio); - swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries); - mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); + private_id = mem_cgroup_private_id_get(memcg, nr_entries); + mod_memcg_state(memcg, MEMCG_SWAP, nr_entries); - __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries, - mem_cgroup_private_id(swap_memcg)); + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries, private_id); folio_unqueue_deferred_split(folio); folio->memcg_data = 0; - if (!obj_cgroup_is_root(objcg)) + if (!obj_cgroup_is_root(objcg)) { page_counter_uncharge(&memcg->memory, nr_entries); - if (memcg != swap_memcg) { - if (!mem_cgroup_is_root(swap_memcg)) - page_counter_charge(&swap_memcg->memsw, nr_entries); - page_counter_uncharge(&memcg->memsw, nr_entries); + if (mem_cgroup_private_id_is_root(private_id)) + page_counter_uncharge(&memcg->memsw, nr_entries); } /* diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 0952b2a783e5..820139657184 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -22,8 +22,12 @@ void drain_all_stock(struct mem_cgroup *root_memcg); int memory_stat_show(struct seq_file *m, void *v); -struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, - unsigned int n); +static inline bool mem_cgroup_private_id_is_root(unsigned short id) +{ + return id == mem_cgroup_private_id(root_mem_cgroup); +} + +unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int n); void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent, int idx); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4d00748c8a5b..a05d9a8baa1b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -741,7 +741,7 @@ static unsigned long *memcg_events_local_array(struct mem_cgroup *memcg) * * 2) Flush the stats synchronously on reader side only when there are more than * (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization - * will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but + * will let stats be out of sync by at most (MEMCG_CHARGE_BATCH * nr_cpus) but * only for 2 seconds due to (1). */ static void flush_memcg_stats_dwork(struct work_struct *w); @@ -1015,9 +1015,8 @@ static void __mod_memcg_lruvec_state(struct mem_cgroup_per_node *pn, put_cpu(); } -static void mod_memcg_lruvec_state(struct lruvec *lruvec, - enum node_stat_item idx, - int val) +void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) { struct pglist_data *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pn; @@ -2173,8 +2172,8 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) stock_pages -= nr_pages; WRITE_ONCE(stock->nr_pages[i], stock_pages); if (!stock_pages) { - css_put(&memcg->css); WRITE_ONCE(stock->cached[i], NULL); + css_put(&memcg->css); } ret = true; } @@ -2210,8 +2209,8 @@ static void drain_stock(struct memcg_stock_pcp *stock, int i) WRITE_ONCE(stock->nr_pages[i], 0); } - css_put(&old->css); WRITE_ONCE(stock->cached[i], NULL); + css_put(&old->css); } static void drain_stock_fully(struct memcg_stock_pcp *stock) @@ -4075,6 +4074,13 @@ static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) #define MEM_CGROUP_ID_MAX ((1UL << MEM_CGROUP_ID_SHIFT) - 1) static DEFINE_XARRAY_ALLOC1(mem_cgroup_private_ids); +static inline struct obj_cgroup *obj_cgroup_from_private_id(unsigned short id) +{ + lockdep_assert_once(rcu_read_lock_held()); + + return xa_load(&mem_cgroup_private_ids, id); +} + static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) { if (memcg->private_id > 0) { @@ -4083,19 +4089,44 @@ static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) } } -static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg, unsigned int n) +static void __mem_cgroup_private_id_put(struct obj_cgroup *objcg, + unsigned short id, unsigned int n) { - if (refcount_sub_and_test(n, &memcg->private_id_ref)) { - mem_cgroup_private_id_remove(memcg); + struct obj_cgroup *objcg_free; - /* Memcg ID pins CSS */ - css_put(&memcg->css); + if (refcount_sub_and_test(n, &objcg->private_id_ref)) { + objcg_free = xa_erase(&mem_cgroup_private_ids, id); + VM_WARN_ON(objcg_free != objcg); + + /* Memcg ID pins the objcg */ + obj_cgroup_put(objcg); } } -struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, unsigned int n) +static inline void mem_cgroup_private_id_put(unsigned short id, unsigned int n) { - while (!refcount_add_not_zero(n, &memcg->private_id_ref)) { + struct obj_cgroup *objcg; + + lockdep_assert_in_rcu_read_lock(); + + objcg = obj_cgroup_from_private_id(id); + __mem_cgroup_private_id_put(objcg, id, n); +} + +static void mem_cgroup_private_id_kill(struct mem_cgroup *memcg) +{ + __mem_cgroup_private_id_put(memcg->private_id_objcg, memcg->private_id, 1); +} + +unsigned short mem_cgroup_private_id_get(struct mem_cgroup *memcg, unsigned int n) +{ + struct obj_cgroup *objcg; + + lockdep_assert_once(rcu_read_lock_held()); + + objcg = memcg->private_id_objcg; + + while (!refcount_add_not_zero(n, &objcg->private_id_ref)) { /* * The root cgroup cannot be destroyed, so it's refcount must * always be >= 1. @@ -4105,8 +4136,10 @@ struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, un break; } memcg = parent_mem_cgroup(memcg); + objcg = memcg->private_id_objcg; } - return memcg; + + return mem_cgroup_private_id(memcg); } /** @@ -4114,11 +4147,25 @@ struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, un * @id: the memcg id to look up * * Caller must hold rcu_read_lock(). + * + * @return: the memcg, or NULL if the memcg referred to is already dead. */ struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id) { + struct obj_cgroup *objcg; + struct mem_cgroup *memcg; + WARN_ON_ONCE(!rcu_read_lock_held()); - return xa_load(&mem_cgroup_private_ids, id); + + objcg = obj_cgroup_from_private_id(id); + if (!objcg) + return NULL; + + memcg = obj_cgroup_memcg(objcg); + if (mem_cgroup_private_id(memcg) != id) + return NULL; + + return memcg; } struct mem_cgroup *mem_cgroup_get_from_id(u64 id) @@ -4365,9 +4412,10 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) FLUSH_TIME); lru_gen_online_memcg(memcg); - /* Online state pins memcg ID, memcg ID pins CSS */ - refcount_set(&memcg->private_id_ref, 1); - css_get(css); + /* CSS pins memcg ID, memcg ID pins obj cgroup */ + memcg->private_id_objcg = objcg; + refcount_set(&memcg->private_id_objcg->private_id_ref, 1); + obj_cgroup_get(memcg->private_id_objcg); /* * Ensure mem_cgroup_from_private_id() works once we're fully online. @@ -4379,7 +4427,7 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) * publish it here at the end of onlining. This matches the * regular ID destruction during offlining. */ - xa_store(&mem_cgroup_private_ids, memcg->private_id, memcg, GFP_KERNEL); + xa_store(&mem_cgroup_private_ids, memcg->private_id, memcg->private_id_objcg, GFP_KERNEL); return 0; free_objcg: @@ -4431,7 +4479,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); - mem_cgroup_private_id_put(memcg, 1); + mem_cgroup_private_id_kill(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) @@ -5817,8 +5865,6 @@ static void __init memcg_struct_check(void) memory_events_local); CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, vmpressure); - CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, - private_id_ref); #ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_write_hot, kmem_stat); @@ -5864,6 +5910,8 @@ static void __init memcg_struct_check(void) zswap_writeback); #endif CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, + private_id_objcg); + CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, private_id); CACHELINE_ASSERT_GROUP_MEMBER(struct mem_cgroup, memcg_read_mostly, kmemcg_id); @@ -5936,6 +5984,8 @@ int __init mem_cgroup_init(void) memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node, SLAB_PANIC | SLAB_HWCACHE_ALIGN); + shrinker_unit_cache_init(); + return 0; } @@ -5955,6 +6005,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) struct page_counter *counter; struct mem_cgroup *memcg; struct obj_cgroup *objcg; + unsigned short private_id; if (do_memsw_account()) return 0; @@ -5964,30 +6015,27 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) if (!objcg) return 0; - rcu_read_lock(); - memcg = obj_cgroup_memcg(objcg); - if (!folio_test_swapcache(folio)) { - memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - rcu_read_unlock(); - return 0; - } + scoped_guard(rcu) { + memcg = obj_cgroup_memcg(objcg); + if (!folio_test_swapcache(folio)) { + memcg_memory_event(memcg, MEMCG_SWAP_FAIL); + return 0; + } - memcg = mem_cgroup_private_id_get_online(memcg, nr_pages); - /* memcg is pined by memcg ID. */ - rcu_read_unlock(); + private_id = mem_cgroup_private_id_get(memcg, nr_pages); - if (!mem_cgroup_is_root(memcg) && - !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { - memcg_memory_event(memcg, MEMCG_SWAP_MAX); - memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - mem_cgroup_private_id_put(memcg, nr_pages); - return -ENOMEM; + if (!mem_cgroup_private_id_is_root(private_id) && + !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { + memcg_memory_event(memcg, MEMCG_SWAP_MAX); + memcg_memory_event(memcg, MEMCG_SWAP_FAIL); + mem_cgroup_private_id_put(private_id, nr_pages); + return -ENOMEM; + } + mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); } - mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); ci = swap_cluster_get_and_lock(folio); - __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages, - mem_cgroup_private_id(memcg)); + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages, private_id); swap_cluster_unlock(ci); return 0; @@ -6000,19 +6048,21 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) */ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) { + struct obj_cgroup *objcg; struct mem_cgroup *memcg; rcu_read_lock(); - memcg = mem_cgroup_from_private_id(id); + objcg = obj_cgroup_from_private_id(id); + memcg = obj_cgroup_memcg(objcg); if (memcg) { - if (!mem_cgroup_is_root(memcg)) { + if (!mem_cgroup_private_id_is_root(id)) { if (do_memsw_account()) page_counter_uncharge(&memcg->memsw, nr_pages); else page_counter_uncharge(&memcg->swap, nr_pages); } mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); - mem_cgroup_private_id_put(memcg, nr_pages); + __mem_cgroup_private_id_put(objcg, id, nr_pages); } rcu_read_unlock(); } diff --git a/mm/memfd.c b/mm/memfd.c index c708d92533f4..0f6fff004f5e 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -82,22 +82,31 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) struct hstate *h = hstate_file(memfd); int err = -ENOMEM; long nr_resv; + u32 hash; gfp_mask = htlb_alloc_mask(h); gfp_mask &= ~(__GFP_HIGHMEM | __GFP_MOVABLE); idx >>= huge_page_order(h); + /* + * Serialize hugepage allocation and instantiation to prevent + * races with concurrent allocations, as required by all other + * callers of hugetlb_add_to_page_cache(). + */ + hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); + mutex_lock(&hugetlb_fault_mutex_table[hash]); + nr_resv = hugetlb_reserve_pages(inode, idx, idx + 1, NULL, EMPTY_VMA_FLAGS); - if (nr_resv < 0) - return ERR_PTR(nr_resv); + if (nr_resv < 0) { + err = nr_resv; + goto out_unlock; + } folio = alloc_hugetlb_folio_reserve(h, numa_node_id(), NULL, gfp_mask); if (folio) { - u32 hash; - /* * Zero the folio to prevent information leaks to userspace. * Use folio_zero_user() which is optimized for huge/gigantic @@ -112,20 +121,9 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) */ __folio_mark_uptodate(folio); - /* - * Serialize hugepage allocation and instantiation to prevent - * races with concurrent allocations, as required by all other - * callers of hugetlb_add_to_page_cache(). - */ - hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - err = hugetlb_add_to_page_cache(folio, memfd->f_mapping, idx); - - mutex_unlock(&hugetlb_fault_mutex_table[hash]); - if (err) { folio_put(folio); goto err_unresv; @@ -133,11 +131,14 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) hugetlb_set_folio_subpool(folio, subpool_inode(inode)); folio_unlock(folio); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return folio; } err_unresv: if (nr_resv > 0) hugetlb_unreserve_pages(inode, idx, idx + 1, 0); +out_unlock: + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return ERR_PTR(err); } #endif diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 57275d4dd34c..b705f9f97f42 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2600,7 +2600,7 @@ try_again: /* * We're only intended to deal with the non-Compound page here. * The page cannot become compound pages again as folio has been - * splited and extra refcnt is held. + * split and extra refcnt is held. */ WARN_ON(folio_test_large(folio)); diff --git a/mm/memory.c b/mm/memory.c index 797d39572a09..0fb676dd2bd6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1291,8 +1291,8 @@ again: * copy_pmd_range()'s prior pmd_none_or_clear_bad(src_pmd), and the * error handling here, assume that exclusive mmap_lock on dst and src * protects anon from unexpected THP transitions; with shmem and file - * protected by mmap_lock-less collapse skipping areas with anon_vma - * (whereas vma_needs_copy() skips areas without anon_vma). A rework + * protected by mmap_lock-less collapse skipping areas with an anon rmap + * (whereas vma_needs_copy() skips areas without one). A rework * can remove such assumptions later, but this is good enough for now. */ dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); @@ -1533,10 +1533,10 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (dst_vma->vm_flags & VM_COPY_ON_FORK) return true; /* - * The presence of an anon_vma indicates an anonymous VMA has page - * tables which naturally cannot be reconstituted on page fault. + * The presence of an anon rmap indicates the VMA may map anonymous + * folios which naturally cannot be reconstituted on page fault. */ - if (src_vma->anon_vma) + if (vma_has_anon_rmap(src_vma)) return true; /* @@ -1564,7 +1564,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (!vma_needs_copy(dst_vma, src_vma)) return 0; - if (is_vm_hugetlb_page(src_vma)) + if (vma_is_hugetlb(src_vma)) return copy_hugetlb_page_range(dst_mm, src_mm, dst_vma, src_vma); /* @@ -2178,7 +2178,7 @@ static void __zap_vma_range(struct mmu_gather *tlb, struct vm_area_struct *vma, if (vma->vm_file && !reaping) uprobe_munmap(vma, start, end); - if (unlikely(is_vm_hugetlb_page(vma))) { + if (unlikely(vma_is_hugetlb(vma))) { zap_flags_t zap_flags = details ? details->zap_flags : 0; VM_WARN_ON_ONCE(reaping); @@ -2313,7 +2313,7 @@ void zap_vma_range_batched(struct mmu_gather *tlb, */ __zap_vma_range(tlb, vma, address, end, details); mmu_notifier_invalidate_range_end(&range); - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { /* * flush tlb and free resources before hugetlb_zap_end(), to * avoid concurrent page faults' allocation failure. @@ -2343,19 +2343,20 @@ void zap_vma_range(struct vm_area_struct *vma, unsigned long address, } /** - * zap_special_vma_range - zap all page table entries in a special vma range + * zap_special_vma_range - zap all page table entries in a VMA which is not + * mm-managed * @vma: the vma covering the range to zap * @address: starting address of the range to zap * @size: number of bytes to zap * * This function does nothing when the provided address range is not fully - * contained in @vma, or when the @vma is not VM_PFNMAP or VM_MIXEDMAP. + * contained in @vma, or when @vma is mm-managed. */ void zap_special_vma_range(struct vm_area_struct *vma, unsigned long address, unsigned long size) { if (!range_in_vma(vma, address, address + size) || - !(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))) + vma_is_mm_managed(vma)) return; zap_vma_range(vma, address, size); @@ -2599,27 +2600,34 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, if (!range_in_vma(vma, addr, end)) return -EFAULT; - if (!(vma->vm_flags & VM_MIXEDMAP)) { - VM_WARN_ON_ONCE(mmap_read_trylock(vma->vm_mm)); - VM_WARN_ON_ONCE(vma->vm_flags & VM_PFNMAP); - vm_flags_set(vma, VM_MIXEDMAP); + if (!vma_test(vma, VMA_MIXEDMAP_BIT)) { + mmap_assert_write_locked(vma->vm_mm); + VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT)); + vma_start_write(vma); + vma_set_flags(vma, VMA_MIXEDMAP_BIT); } /* Defer page refcount checking till we're about to map that page. */ return insert_pages(vma, addr, pages, num, vma->vm_page_prot); } EXPORT_SYMBOL(vm_insert_pages); +static void __map_kernel_pages_prepare(struct vm_area_desc *desc) +{ + if (vma_desc_test(desc, VMA_MIXEDMAP_BIT)) + return; + + mmap_assert_write_locked(desc->mm); + VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); + vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); +} + int map_kernel_pages_prepare(struct vm_area_desc *desc) { const struct mmap_action *action = &desc->action; const unsigned long addr = action->map_kernel.start; unsigned long nr_pages, end; - if (!vma_desc_test(desc, VMA_MIXEDMAP_BIT)) { - VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm)); - VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); - vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); - } + __map_kernel_pages_prepare(desc); nr_pages = action->map_kernel.nr_pages; end = addr + PAGE_SIZE * nr_pages; @@ -2628,7 +2636,6 @@ int map_kernel_pages_prepare(struct vm_area_desc *desc) return 0; } -EXPORT_SYMBOL(map_kernel_pages_prepare); int map_kernel_pages_complete(struct vm_area_struct *vma, struct mmap_action *action) @@ -2640,7 +2647,96 @@ int map_kernel_pages_complete(struct vm_area_struct *vma, action->map_kernel.pages, &nr_pages, vma->vm_page_prot); } -EXPORT_SYMBOL(map_kernel_pages_complete); + +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc) +{ + const struct mmap_action *action = &desc->action; + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + + /* At minimum need to be able to get pages. */ + if (WARN_ON_ONCE(!ops || !ops->get)) + return -EINVAL; + + __map_kernel_pages_prepare(desc); + return 0; +} + +static int apply_discontig_action(struct vm_area_struct *vma, + struct discontig_kernel_page_state *state) +{ + unsigned long nr_pages = state->__nr_pages; + unsigned long addr = state->addr; + unsigned long i; + + if (state->action == DISCONTIG_KERNEL_PAGE_MAP_PAGES) + return insert_pages(vma, addr, state->__page_arr, + &nr_pages, vma->vm_page_prot); + + VM_WARN_ON_ONCE(state->action != DISCONTIG_KERNEL_PAGE_MAP_CONTIG_PAGES); + /* Contiguous pages - have to iterate through each one. */ + for (i = 0; i < nr_pages; i++, addr += PAGE_SIZE) { + struct page *page = state->__page + i; + int err; + + err = insert_page(vma, addr, page, vma->vm_page_prot, + /*mkwrite=*/false); + if (err) + return err; + } + return 0; +} + +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action) +{ + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + struct discontig_kernel_page_state state = { + .start = vma->vm_start, + .end = vma->vm_end, + .addr = vma->vm_start, + .pgoff = vma->vm_pgoff, + .nr_pages_mapped = 0, + .nr_pages_remain = vma_pages(vma), + .vm_private_data = vma->vm_private_data, + .private = action->map_kernel_discontig.init_private, + }; + int err = 0; + + if (ops->init) + err = ops->init(vma->vm_private_data, &state.private); + if (err) + return err; + + do { + unsigned long end, pgoff_end; + unsigned long nr_pages; + + /* Default to abort. */ + state.action = DISCONTIG_KERNEL_PAGE_ABORT; + err = ops->get(&state); + if (err || state.action == DISCONTIG_KERNEL_PAGE_ABORT) + return err; + nr_pages = state.__nr_pages; + + if (!nr_pages || nr_pages > state.nr_pages_remain) + return -EINVAL; + end = state.addr + PAGE_SIZE * nr_pages; + pgoff_end = state.pgoff + nr_pages; + + err = apply_discontig_action(vma, &state); + if (err) + return err; + + state.addr = end; + state.pgoff = pgoff_end; + state.nr_pages_mapped += nr_pages; + state.nr_pages_remain -= nr_pages; + } while (state.addr < vma->vm_end); + + return 0; +} /** * vm_insert_page - insert single page into user vma @@ -2677,10 +2773,11 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, { if (addr < vma->vm_start || addr >= vma->vm_end) return -EFAULT; - if (!(vma->vm_flags & VM_MIXEDMAP)) { - BUG_ON(mmap_read_trylock(vma->vm_mm)); - BUG_ON(vma->vm_flags & VM_PFNMAP); - vm_flags_set(vma, VM_MIXEDMAP); + if (!vma_test(vma, VMA_MIXEDMAP_BIT)) { + mmap_assert_write_locked(vma->vm_mm); + VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT)); + vma_start_write(vma); + vma_set_flags(vma, VMA_MIXEDMAP_BIT); } return insert_page(vma, addr, page, vma->vm_page_prot, false); } @@ -3904,10 +4001,10 @@ static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) * * When preparing to insert an anonymous page into a VMA from a * fault handler, call this function rather than anon_vma_prepare(). - * If this vma does not already have an associated anon_vma and we are + * If this vma does not already have an anon rmap and we are * only protected by the per-VMA lock, the caller must retry with the * mmap_lock held. __anon_vma_prepare() will look at adjacent VMAs to - * determine if this VMA can share its anon_vma, and that's not safe to + * determine if this VMA can share its anon rmap, and that's not safe to * do with only the per-VMA lock held for this VMA. * * Return: 0 if fault handling can proceed. Any other value should be @@ -3918,7 +4015,7 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = 0; - if (likely(vma->anon_vma)) + if (likely(vma_has_anon_rmap(vma))) return 0; if (vmf->flags & FAULT_FLAG_VMA_LOCK) { if (!mmap_read_trylock(vma->vm_mm)) @@ -5069,7 +5166,7 @@ check_folio: * have changed, so sub pages might got charged to the wrong cgroup, * or even should be shmem. So we have to free it and fallback. * Nothing should have touched it, both anon and shmem checks if a - * large folio is fully appliable before use. + * large folio is fully applicable before use. * * This will be removed once we unify folio allocation in the swap cache * layer, where allocation of a folio stabilizes the swap entries. @@ -6840,7 +6937,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, lru_gen_enter_fault(vma); - if (unlikely(is_vm_hugetlb_page(vma))) + if (unlikely(vma_is_hugetlb(vma))) ret = hugetlb_fault(vma->vm_mm, vma, address, flags); else ret = __handle_mm_fault(vma, address, flags); @@ -7129,6 +7226,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; pgprot_t prot = __pgprot(0); void __iomem *maddr; + struct page *page; int offset = offset_in_page(addr); int ret = -EINVAL; bool writable; @@ -7143,6 +7241,19 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, retry: if (follow_pfnmap_start(&args)) return -EINVAL; + + /* + * Reject anything that looks like ordinary RAM, including anonymous + * folios in CoW mappings or allocated kernel memory. Note that we + * cannot distinguish memblock allocations from non-RAM reliably, so + * we cannot easily reject them here. + */ + page = pfn_to_online_page(args.pfn); + if (page && !PageReserved(page)) { + follow_pfnmap_end(&args); + return -EINVAL; + } + prot = args.pgprot; phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; writable = args.writable; @@ -7671,12 +7782,12 @@ void ptlock_free(struct ptdesc *ptdesc) void vma_pgtable_walk_begin(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_lock_read(vma); } void vma_pgtable_walk_end(struct vm_area_struct *vma) { - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_unlock_read(vma); } diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index dd9c8a519bfe..c9cc42e54f05 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -345,6 +345,8 @@ struct page *pfn_to_online_page(unsigned long pfn) struct dev_pagemap *pgmap; struct mem_section *ms; + BUILD_BUG_ON(IS_ENABLED(CONFIG_HAVE_ARCH_PFN_VALID)); + if (nr >= NR_MEM_SECTIONS) return NULL; @@ -352,13 +354,6 @@ struct page *pfn_to_online_page(unsigned long pfn) if (!online_section(ms)) return NULL; - /* - * Save some code text when online_section() + - * pfn_section_valid() are sufficient. - */ - if (IS_ENABLED(CONFIG_HAVE_ARCH_PFN_VALID) && !pfn_valid(pfn)) - return NULL; - if (!pfn_section_valid(ms, pfn)) return NULL; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index e9860fb9f73f..0884c629670d 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2018,7 +2018,7 @@ bool vma_migratable(struct vm_area_struct *vma) if (vma_is_dax(vma)) return false; - if (is_vm_hugetlb_page(vma) && + if (vma_is_hugetlb(vma) && !hugepage_migration_supported(hstate_vma(vma))) return false; @@ -2302,7 +2302,7 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, break; case MPOL_BIND: /* Restrict to nodemask (but not on lower zones) */ - if (apply_policy_zone(pol, gfp_zone(gfp)) && + if (apply_policy_zone(pol, gfp_zone(current_gfp_context(gfp))) && cpuset_nodemask_valid_mems_allowed(&pol->nodes)) nodemask = &pol->nodes; if (pol->home_node != NUMA_NO_NODE) diff --git a/mm/migrate.c b/mm/migrate.c index 7e3a81f06974..7bdcdb57652f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1174,7 +1174,7 @@ static void migrate_folio_undo_src(struct folio *src, int was_mapped, { if (was_mapped) remove_migration_ptes(src, src, 0); - /* Drop an anon_vma reference if we took one */ + /* Drop an anon rmap reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); if (locked) @@ -1281,15 +1281,15 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, /* * By try_to_migrate(), src->mapcount goes down to 0 here. In this case, - * we cannot notice that anon_vma is freed while we migrate a page. - * This get_anon_vma() delays freeing anon_vma pointer until the end + * we cannot notice that the anon rmap is freed while we migrate a page. + * This get_anon_vma() delays freeing the anon rmap until the end * of migration. File cache pages are no problem because of page_lock() * File Caches may use write_page() or lock_page() in migration, then, * just care Anon page here. * * Only folio_get_anon_vma() understands the subtleties of - * getting a hold on an anon_vma from outside one of its mms. - * But if we cannot get anon_vma, then we won't need it anyway, + * getting a hold on an anon rmap from outside one of its mms. + * But if we cannot get the anon rmap, then we won't need it anyway, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ @@ -1432,7 +1432,7 @@ out_unlock_both: * and will be freed. */ list_del(&src->lru); - /* Drop an anon_vma reference if we took one */ + /* Drop an anon rmap reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 0c437004329d..9e19fadfa65c 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -739,19 +739,21 @@ static void migrate_vma_unmap(struct migrate_vma *migrate) */ int migrate_vma_setup(struct migrate_vma *args) { + const struct vm_area_struct *vma = args->vma; long nr_pages = (args->end - args->start) >> PAGE_SHIFT; args->start &= PAGE_MASK; args->end &= PAGE_MASK; - if (!args->vma || is_vm_hugetlb_page(args->vma) || - (args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma)) + if (!vma) + return -EINVAL; + /* The migrate_vma page walk has no hugetlb support. */ + if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma) || vma_is_dax(vma)) return -EINVAL; if (nr_pages <= 0) return -EINVAL; - if (args->start < args->vma->vm_start || - args->start >= args->vma->vm_end) + if (args->start < vma->vm_start || args->start >= vma->vm_end) return -EINVAL; - if (args->end <= args->vma->vm_start || args->end > args->vma->vm_end) + if (args->end <= vma->vm_start || args->end > vma->vm_end) return -EINVAL; if (!args->src || !args->dst) return -EINVAL; diff --git a/mm/mlock.c b/mm/mlock.c index 39215a3eab1f..67ae9b18b923 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -316,22 +316,10 @@ static inline unsigned int folio_mlock_step(struct folio *folio, return folio_pte_batch(folio, pte, ptent, count); } -static inline bool allow_mlock_munlock(struct folio *folio, +static inline bool allow_mlock(struct folio *folio, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned int step) { - /* - * For unlock, allow munlock large folio which is partially - * mapped to VMA. As it's possible that large folio is - * mlocked and VMA is split later. - * - * During memory pressure, such kind of large folio can - * be split. And the pages are not in VM_LOCKed VMA - * can be reclaimed. - */ - if (!vma_test(vma, VMA_LOCKED_BIT)) - return true; - /* folio_within_range() cannot take KSM, but any small folio is OK */ if (!folio_test_large(folio)) return true; @@ -352,6 +340,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, { struct vm_area_struct *vma = walk->vma; + const bool lock = walk->private; spinlock_t *ptl; pte_t *start_pte, *pte; pte_t ptent; @@ -368,7 +357,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = pmd_folio(*pmd); if (folio_is_zone_device(folio)) goto out; - if (vma_test(vma, VMA_LOCKED_BIT)) + if (lock) mlock_folio(folio); else munlock_folio(folio); @@ -390,10 +379,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, continue; step = folio_mlock_step(folio, pte, addr, end); - if (!allow_mlock_munlock(folio, vma, start, end, step)) + if (lock && !allow_mlock(folio, vma, start, end, step)) goto next_entry; - if (vma_test(vma, VMA_LOCKED_BIT)) + if (lock) mlock_folio(folio); else munlock_folio(folio); @@ -428,31 +417,29 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, .pmd_entry = mlock_pte_range, .walk_lock = PGWALK_WRLOCK_VERIFY, }; + const bool lock = vma_flags_test(new_vma_flags, VMA_LOCKED_BIT); + vma_flags_t walk_flags = *new_vma_flags; /* - * There is a slight chance that concurrent page migration, - * or page reclaim finding a page of this now-VMA_LOCKED_BIT vma, - * will call mlock_vma_folio() and raise page's mlock_count: - * double counting, leaving the page unevictable indefinitely. - * Communicate this danger to mlock_vma_folio() with VMA_IO_BIT, - * which is a VMA_SPECIAL_FLAGS flag not allowed on VMA_LOCKED_BIT vmas. - * mmap_lock is held in write mode here, so this weird - * combination should not be visible to other mmap_lock users; - * but WRITE_ONCE so rmap walkers must see VMA_IO_BIT if VMA_LOCKED_BIT. + * LOCKONFAULT without LOCKED never otherwise occurs: it marks a walk in + * progress so that rmap-side callers, which test VMA_LOCKED_BIT, do not + * count folios, while try_to_unmap_one(), which tests VMA_LOCKED_MASK, + * still refuses to unmap them. */ - if (vma_flags_test(new_vma_flags, VMA_LOCKED_BIT)) - vma_flags_set(new_vma_flags, VMA_IO_BIT); + if (lock) { + vma_flags_clear(&walk_flags, VMA_LOCKED_BIT); + vma_flags_set(&walk_flags, VMA_LOCKONFAULT_BIT); + } + vma_start_write(vma); - vma_flags_reset_once(vma, new_vma_flags); + vma_flags_reset_once(vma, &walk_flags); lru_add_drain(); - walk_page_range_vma(vma, start, end, &mlock_walk_ops, NULL); + walk_page_range_vma(vma, start, end, &mlock_walk_ops, (void *)lock); lru_add_drain(); - if (vma_flags_test(new_vma_flags, VMA_IO_BIT)) { - vma_flags_clear(new_vma_flags, VMA_IO_BIT); + if (lock) vma_flags_reset_once(vma, new_vma_flags); - } } /* @@ -470,7 +457,7 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, { const vma_flags_t old_vma_flags = vma->flags; struct mm_struct *mm = vma->vm_mm; - int nr_pages; + long nr_pages; int ret = 0; if (vma_flags_same_pair(&old_vma_flags, new_vma_flags) || diff --git a/mm/mm_init.c b/mm/mm_init.c index 41a27860d2f6..2a0e782a151d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -593,6 +593,15 @@ out: node_states[N_MEMORY] = saved_node_state; } +static inline void vmemmap_optimization_verify_zone(struct page *page) +{ + const unsigned int order = pfn_to_section_compound_order(page_to_pfn(page)); + + VM_WARN_ON_ONCE(vmemmap_optimizable_order(order) && + page_zone_id(page + VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES) != + page_zone_id(page)); +} + void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { @@ -609,6 +618,11 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif + /* + * Shared tail struct pages are initialized during vmemmap population, + * before retained struct pages reach here. + */ + vmemmap_optimization_verify_zone(page); } #ifdef CONFIG_NUMA @@ -1076,7 +1090,7 @@ static void zone_device_page_init_from_template(struct page *page, * initialize is a lot smaller that the total amount of struct pages being * mapped. This is a paired / mild layering violation with explicit knowledge * of how the sparse_vmemmap internals handle compound pages in the lack - * of an altmap. See vmemmap_populate_compound_pages(). + * of an altmap. */ static inline unsigned long compound_nr_pages(unsigned long pfn, struct dev_pagemap *pgmap) @@ -1158,6 +1172,8 @@ void __ref memmap_init_zone_device(struct zone *zone, if (!nr_pages) return; + VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound)); + /* * Seed the reusable head-page template from the first real struct * page. The normal page-init and refcount helpers must operate on diff --git a/mm/mmap.c b/mm/mmap.c index 4bf26b0f1e6e..148ba01c0373 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -547,7 +547,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, } case MAP_PRIVATE: /* - * Set pgoff according to addr for anon_vma. + * Set pgoff according to addr for the anon rmap. */ pgoff = addr >> PAGE_SHIFT; break; @@ -1774,8 +1774,8 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) if (vma_test(tmp, VMA_WIPEONFORK_BIT)) { /* * VMA_WIPEONFORK_BIT gets a clean slate in the child. - * Don't prepare anon_vma until fault since we don't - * copy page for current vma. + * Don't prepare the anon rmap until fault since we + * don't copy pages for the current vma. */ tmp->anon_vma = NULL; } else if (anon_vma_fork(tmp, mpnt)) @@ -1786,7 +1786,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) /* * Copy/update hugetlb private vma information. */ - if (is_vm_hugetlb_page(tmp)) + if (vma_is_hugetlb(tmp)) hugetlb_dup_vma_private(tmp); /* diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index 2f94ee0fdee2..83b7d3ded0dc 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -194,21 +194,38 @@ void __vma_exclude_readers_for_detach(struct vm_area_struct *vma) WARN_ON_ONCE(!ves.detached); } +/* Do the VMA, mm seqcounts indicate the VMA is write locked? */ +static bool vma_seqcount_is_write_locked(unsigned int vma_seq, unsigned int mm_seq) +{ + /* VMA and mm sequence numbers must match for a write lock. */ + if (vma_seq != mm_seq) + return false; + + /* + * The VMA seqcount is always odd if locked. For an attached VMA an even + * value can only be the zero stored by __mm_reset_vma_seqcounts(), + * never a write lock. + */ + return vma_seq & 1; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. - * False locked result is possible if mm_lock_seq overflows or if vma gets - * reused and attached to a different mm before we lock it. + * False locked result is possible if vma gets reused and attached to a + * different mm before we lock it. * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got * detached. * * IMPORTANT: RCU lock must be held upon entering the function, but upon error * IT IS RELEASED. The caller must handle this correctly. */ -static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, - struct vm_area_struct *vma) +static struct vm_area_struct *vma_start_read(struct mm_struct *mm, + struct vm_area_struct *vma) { + unsigned int vma_seq; + unsigned int mm_seq; struct mm_struct *other_mm; int oldcnt; @@ -220,7 +237,9 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) { + vma_seq = READ_ONCE(vma->vm_lock_seq); + mm_seq = READ_ONCE(mm->mm_lock_seq.sequence); + if (vma_seqcount_is_write_locked(vma_seq, mm_seq)) { vma = NULL; goto err; } @@ -246,17 +265,26 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, goto err_unstable; /* - * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. - * False unlocked result is impossible because we modify and check + * False unlocked result is impossible because a write lock sets * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * - * We must use ACQUIRE semantics for the mm_lock_seq so that if we are - * racing with vma_end_write_all(), we only start reading from the VMA - * after it has been unlocked. - * This pairs with RELEASE semantics in vma_end_write_all(). + * ACQUIRE semantics are required for both reads. + * + * __mm_reset_vma_seqcounts() might reset vma->vm_lock_seq to zero + * after an overflow. A reader observing that zero must also observe + * the VMA modifications made under the write lock which just ended, so + * the ACQUIRE pairs with the smp_wmb() in do_raw_write_seqcount_end() + * that precedes the reset stores. + * + * The ACQUIRE semantics for the mm_lock_seq read is required so that, + * if we are racing with vma_end_write_all(), we only start reading from + * the VMA after it has been unlocked. This pairs with the RELEASE + * semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { + vma_seq = smp_load_acquire(&vma->vm_lock_seq); + mm_seq = raw_read_seqcount(&mm->mm_lock_seq); + if (unlikely(vma_seqcount_is_write_locked(vma_seq, mm_seq))) { vma_refcount_put(vma); vma = NULL; goto err; @@ -466,6 +494,43 @@ fallback: return vma; } +/* + * Called with mmap write lock held after mm->mm_lock_seq just overflowed (and + * therefore zero), with all VMA write locks now released. + * + * The stores below follow the smp_wmb() in do_raw_write_seqcount_end(), so + * a reader that observes a reset value also observes this cycle's VMA + * modifications (ordered by the ACQUIRE on vma->vm_lock_seq in + * vma_start_read()). + */ +void __mm_reset_vma_seqcounts(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); + + mmap_assert_write_locked(mm); + VM_WARN_ON_ONCE(mm->mm_lock_seq.sequence); + + /* + * Failing to reset means stale vma->vm_lock_seq values could result in + * those VMAs becoming spuriously write locked when mm->mm_lock_seq + * happens to match their values. + * + * This is problematic, as it causes the VMA write lock to not exclude + * readers, which can result in concurrent readers while a VMA write + * lock is held. + * + * Avoid this by resetting to an even value (indicates no write lock + * held). + * + * Readers will either observe zero (not write locked) or the previous + * value, so this is safe. + */ + for_each_vma(vmi, vma) + WRITE_ONCE(vma->vm_lock_seq, 0); +} +EXPORT_SYMBOL(__mm_reset_vma_seqcounts); + #ifdef CONFIG_LOCK_MM_AND_FIND_VMA #include <linux/extable.h> diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 2a72a9686773..9f353f0e2ef4 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -480,7 +480,7 @@ void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) { tlb_gather_mmu(tlb, vma->vm_mm); tlb_update_vma_flags(tlb, vma); - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) /* All entries have the same size. */ tlb_change_page_size(tlb, huge_page_size(hstate_vma(vma))); } diff --git a/mm/mprotect.c b/mm/mprotect.c index 2888ee638d87..754333698c1e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -717,7 +717,7 @@ long change_protection(struct mmu_gather *tlb, (cp_flags & MM_CP_UFFD_RWP)) newprot = PAGE_NONE; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) pages = hugetlb_change_protection(vma, start, end, newprot, cp_flags); else @@ -783,8 +783,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, * uncommon case, so doesn't need to be very optimized. */ if (arch_has_pfn_modify_check() && - vma_flags_test_any(&old_vma_flags, VMA_PFNMAP_BIT, - VMA_MIXEDMAP_BIT) && + !vma_flags_is_mm_managed(&old_vma_flags) && !vma_flags_test_any_mask(&new_vma_flags, VMA_ACCESS_FLAGS)) { pgprot_t new_pgprot = vm_get_page_prot(newflags); @@ -797,8 +796,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, /* * If we make a private mapping writable we increase our commit; * but (without finer accounting) cannot reduce our commit if we - * make it unwritable again except in the anonymous case where no - * anon_vma has yet to be assigned. + * make it unwritable again except in the anonymous case where the + * VMA's anon rmap has yet to be assigned. * * hugetlb mapping were accounted for even if read-only so there is * no need to account for them here. @@ -817,7 +816,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, vma_flags_set(&new_vma_flags, VMA_ACCOUNT_BIT); } } else if (vma_flags_test(&old_vma_flags, VMA_ACCOUNT_BIT) && - vma_is_anonymous(vma) && !vma->anon_vma) { + vma_is_anonymous(vma) && !vma_has_anon_rmap(vma)) { vma_flags_clear(&new_vma_flags, VMA_ACCOUNT_BIT); } diff --git a/mm/mremap.c b/mm/mremap.c index 444f37d7cd28..0d06d7f37f19 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -144,13 +144,13 @@ static void take_rmap_locks(struct vm_area_struct *vma) { if (vma->vm_file) i_mmap_lock_write(vma->vm_file->f_mapping); - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) anon_vma_lock_write(vma->anon_vma); } static void drop_rmap_locks(struct vm_area_struct *vma) { - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) anon_vma_unlock_write(vma->anon_vma); if (vma->vm_file) i_mmap_unlock_write(vma->vm_file->f_mapping); @@ -214,7 +214,7 @@ static int move_ptes(struct pagetable_move_control *pmc, int err = 0; /* - * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma + * When need_rmap_locks is true, we take the i_mmap_rwsem and anon rmap * locks to ensure that rmap will always observe either the old or the * new ptes. This is the easiest way to avoid races with * truncate_pagecache(), page migration, etc... @@ -812,7 +812,7 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) if (!pmc->len_in) return 0; - if (is_vm_hugetlb_page(pmc->old)) + if (vma_is_hugetlb(pmc->old)) return move_hugetlb_page_tables(pmc->old, pmc->new, pmc->old_addr, pmc->new_addr, pmc->len_in); @@ -887,12 +887,6 @@ again: return pmc_progress(pmc); } -/* Set vrm->delta to the difference in VMA size specified by user. */ -static void vrm_set_delta(struct vma_remap_struct *vrm) -{ - vrm->delta = abs_diff(vrm->old_len, vrm->new_len); -} - /* Determine what kind of remap this is - shrink, expand or no resize at all. */ static enum mremap_type vrm_remap_type(struct vma_remap_struct *vrm) { @@ -905,6 +899,13 @@ static enum mremap_type vrm_remap_type(struct vma_remap_struct *vrm) return MREMAP_EXPAND; } +/* Populate calculated fields. */ +static void vrm_populate_calculated(struct vma_remap_struct *vrm) +{ + vrm->delta = abs_diff(vrm->old_len, vrm->new_len); + vrm->remap_type = vrm_remap_type(vrm); +} + /* * When moving a VMA to vrm->new_adr, does this result in the new and old VMAs * overlapping? @@ -1338,6 +1339,20 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, pmc_revert.need_rmap_locks = true; move_page_tables(&pmc_revert); + /* + * When copy_vma_and_data() returns, the new VMA will be + * unmapped rather than the old one. + * + * When an accountable move is being performed, this unmapping + * will not uncharge any expansion, so do that now. + * + * A move is accountable if the VMA is accountable and + * MREMAP_DONTUNMAP is not specified, so only do so if the + * latter is true. + */ + if (!(vrm->flags & MREMAP_DONTUNMAP)) + vrm_uncharge(vrm); + vrm->vma = new_vma; vrm->old_len = vrm->new_len; vrm->addr = vrm->new_addr; @@ -1345,7 +1360,7 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, mremap_userfaultfd_prep(new_vma, vrm->uf); } - fixup_hugetlb_reservations(vma); + fixup_hugetlb_reservations(vrm->vma); *new_vma_ptr = new_vma; return err; @@ -1373,8 +1388,8 @@ static void dontunmap_complete(struct vma_remap_struct *vrm, vma_clear_flags_mask(vma, VMA_LOCKED_MASK); /* - * anon_vma links of the old vma is no longer needed after its page - * table has been moved. + * The anon rmap links of the old vma are no longer needed after its + * page table has been moved. */ if (start == old_start && end == old_end) { const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT; @@ -1523,6 +1538,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm) /* Set up for the move now shrink has been executed. */ vrm->old_len = vrm->new_len; + vrm_populate_calculated(vrm); } /* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ @@ -1769,7 +1785,7 @@ static bool vma_multi_allowed(struct vm_area_struct *vma) /* Known good. */ if (vma_is_shmem(vma)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; if (file->f_op->get_unmapped_area == thp_get_unmapped_area) return true; @@ -1792,11 +1808,10 @@ static int check_prep_vma(struct vma_remap_struct *vrm) return -EPERM; /* Align to hugetlb page size, if required. */ - if (is_vm_hugetlb_page(vma) && !align_hugetlb(vrm)) + if (vma_is_hugetlb(vma) && !align_hugetlb(vrm)) return -EINVAL; - vrm_set_delta(vrm); - vrm->remap_type = vrm_remap_type(vrm); + vrm_populate_calculated(vrm); /* For convenience, we set new_addr even if VMA won't move. */ if (!vrm_implies_new_addr(vrm)) vrm->new_addr = addr; @@ -1822,8 +1837,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) return -EINVAL; } - if ((vrm->flags & MREMAP_DONTUNMAP) && - vma_test_any(vma, VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT)) + if ((vrm->flags & MREMAP_DONTUNMAP) && vma_is_fixed_mapping(vma)) return -EINVAL; /* @@ -1861,7 +1875,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm) if (pgoff + (new_len >> PAGE_SHIFT) < pgoff) return -EINVAL; - if (vma_test_any(vma, VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT)) + if (vma_is_fixed_mapping(vma)) return -EFAULT; if (!mlock_future_ok(mm, vma_test(vma, VMA_LOCKED_BIT), vrm->delta)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3322f9e75616..a8cac5e01066 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -971,10 +971,12 @@ static inline void __free_one_page(struct page *page, * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order. */ - if (page_is_guard(buddy)) + if (page_is_guard(buddy)) { clear_page_guard(zone, buddy, order); - else + account_freepages(zone, 1 << order, migratetype); + } else { __del_page_from_free_list(buddy, zone, order, buddy_mt); + } if (unlikely(buddy_mt != migratetype)) { /* @@ -4117,6 +4119,12 @@ out: return page; } +static inline bool order_defrag_promoted(unsigned int order, + const struct alloc_context *ac) +{ + return ac->reclaim_compact_order != order; +} + /* * Maximum number of compaction retries with a progress before OOM * killer is consider as the only way to move forward. @@ -4139,22 +4147,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, .order = order, .page = NULL, }; - int compact_order = order; - - /* - * If fallbacks are not permitted (defrag_mode), we either - * need to reclaim space in a block of matching type, or clear - * out an entire block to allow __rmqueue_claim() to convert. - * - * Reclaim by itself is primarily freeing space in movable - * blocks, since that's where the LRU pages live. So this - * works for movable requests, but not for others. - * - * For those, promote the order to help make blocks, instead - * of spinning in reclaim alone unproductively. - */ - if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) - compact_order = max(order, pageblock_order); + int compact_order = ac->reclaim_compact_order; if (!compact_order) return NULL; @@ -4247,7 +4240,7 @@ should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order, int retries = *compaction_retries; enum compact_priority priority = *compact_priority; - if (!order) + if (!ac->reclaim_compact_order) return false; if (fatal_signal_pending(current)) @@ -4256,10 +4249,12 @@ should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order, /* * Compaction was skipped due to a lack of free order-0 * migration targets. Continue if reclaim can help. + * Promoted requests fall back instead. */ if (compact_result == COMPACT_SKIPPED) { - ret = compaction_zonelist_suitable(ac, order, alloc_flags, - gfp_mask); + if (!order_defrag_promoted(order, ac)) + ret = compaction_zonelist_suitable(ac, order, + alloc_flags, gfp_mask); goto out; } @@ -4458,11 +4453,7 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; bool drained = false; - int reclaim_order = order; - - /* Match the slowpath compaction promotion in __alloc_pages_direct_compact */ - if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) - reclaim_order = max(order, pageblock_order); + int reclaim_order = ac->reclaim_compact_order; psi_memstall_enter(&pflags); *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac); @@ -4638,9 +4629,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, /* * Costly allocations might have made a progress but this doesn't mean * their order will become available due to high fragmentation so - * always increment the no progress counter for them + * always increment the no progress counter for them. The same goes for + * promoted requests: their reclaim reports progress when it backs off + * for compaction without freeing anything. */ - if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) + if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER && + !order_defrag_promoted(order, ac)) *no_progress_loops = 0; else (*no_progress_loops)++; @@ -4924,6 +4918,22 @@ retry: /* If allocation has taken excessively long, warn about it */ check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time); + /* + * If fallbacks are not permitted (defrag_mode), we either + * need to reclaim space in a block of matching type, or clear + * out an entire block to allow __rmqueue_claim() to convert. + * + * Reclaim by itself is primarily freeing space in movable + * blocks, since that's where the LRU pages live. So this + * works for movable requests, but not for others. + * + * For those, promote the order to help make blocks, instead + * of spinning in reclaim alone unproductively. + */ + ac->reclaim_compact_order = order; + if ((alloc_flags & ALLOC_NOFRAGMENT) && ac->migratetype != MIGRATE_MOVABLE) + ac->reclaim_compact_order = max(order, pageblock_order); + /* Try direct reclaim and then allocating */ if (!compact_first) { page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, @@ -5005,6 +5015,11 @@ retry: /* Reclaim/compaction failed to prevent the fallback */ if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT)) { alloc_flags &= ~ALLOC_NOFRAGMENT; + /* Retries went into making blocks, start the fallback afresh */ + if (order_defrag_promoted(order, ac)) { + no_progress_loops = 0; + compaction_retries = 0; + } goto retry; } @@ -6709,12 +6724,8 @@ void calculate_min_free_kbytes(void) lowmem_kbytes = nr_free_buffer_pages() * (PAGE_SIZE >> 10); new_min_free_kbytes = int_sqrt(lowmem_kbytes * 16); - if (new_min_free_kbytes > user_min_free_kbytes) + if (user_min_free_kbytes < 0) min_free_kbytes = clamp(new_min_free_kbytes, 128, 262144); - else - pr_warn_ratelimited("min_free_kbytes is not updated to %d because user defined value %d is preferred\n", - new_min_free_kbytes, user_min_free_kbytes); - } int __meminit init_per_zone_wmark_min(void) diff --git a/mm/page_alloc.h b/mm/page_alloc.h index d4182f7cb7dd..e238ee91ed24 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -93,6 +93,8 @@ struct alloc_context { bool spread_dirty_pages; /* Only flags that are global to the whole allocation go here. */ unsigned int alloc_flags; + /* Set by __alloc_pages_slowpath() before reclaim and compaction */ + unsigned int reclaim_compact_order; }; /* diff --git a/mm/page_counter.c b/mm/page_counter.c index 9167ffd1380c..5293460831f8 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -472,8 +472,8 @@ void page_counter_calculate_protection(struct page_counter *root, return; if (parent == root) { - counter->emin = READ_ONCE(counter->min); - counter->elow = READ_ONCE(counter->low); + WRITE_ONCE(counter->emin, READ_ONCE(counter->min)); + WRITE_ONCE(counter->elow, READ_ONCE(counter->low)); return; } diff --git a/mm/page_io.c b/mm/page_io.c index 0808808f0309..c6824fcd483e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -135,7 +135,7 @@ static bool is_folio_zero_filled(struct folio *folio) for (i = 0; i < folio_nr_pages(folio); i++) { data = kmap_local_folio(folio, i * PAGE_SIZE); /* - * Check last word first, incase the page is zero-filled at + * Check last word first, in case the page is zero-filled at * the start and has non-zero data at the end, which is common * in real-world workloads. */ diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 28e306fdb3a5..e6b6170c2f4f 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -109,7 +109,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) unsigned long pfn; pte_t ptent; - if (is_vm_hugetlb_page(pvmw->vma)) + if (vma_is_hugetlb(pvmw->vma)) ptent = huge_ptep_get(pvmw->vma->vm_mm, pvmw->address, pvmw->pte); else @@ -206,7 +206,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pmd && !pvmw->pte) return not_found(pvmw); - if (unlikely(is_vm_hugetlb_page(vma))) { + if (unlikely(vma_is_hugetlb(vma))) { struct hstate *hstate = hstate_vma(vma); unsigned long size = huge_page_size(hstate); /* The only possible mapping was handled on last iteration */ @@ -265,28 +265,28 @@ restart: softleaf_t entry; if (!(pvmw->flags & PVMW_MIGRATION)) - return not_found(pvmw); + goto next_pmd; entry = softleaf_from_pmd(pmde); if (!check_pmd(softleaf_to_pfn(entry), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } else if (pmd_is_device_private_entry(pmde)) { softleaf_t entry; if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); + goto next_pmd; entry = softleaf_from_pmd(pmde); if (!check_pmd(softleaf_to_pfn(entry), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } else if (!pmd_present(pmde)) { - return not_found(pvmw); + goto next_pmd; } if (likely(pmd_trans_huge(pmde))) { if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); + goto next_pmd; if (!check_pmd(pmd_pfn(pmde), pvmw)) - return not_found(pvmw); + goto next_pmd; return true; } /* THP/device-private pmd was split under us: handle on pte level */ @@ -299,8 +299,7 @@ restart: (pvmw->nr_pages >= HPAGE_PMD_NR)) sync_with_folio_pmd_zap(mm, pvmw->pmd); - step_forward(pvmw, PMD_SIZE); - continue; + goto next_pmd; } if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) @@ -343,6 +342,17 @@ next_pte: pvmw->ptl = ptl; } goto this_pte; +next_pmd: + /* + * A large folio can be PTE-mapped across PMDs, so a + * nonmatching PMD does not end the walk. + */ + if (pvmw->ptl) { + spin_unlock(pvmw->ptl); + pvmw->ptl = NULL; + } + pvmw->pmd = NULL; + step_forward(pvmw, PMD_SIZE); } while (pvmw->address < end); return false; diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7411702a37f5..e6493bbe6919 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -408,7 +408,7 @@ static int __walk_page_range(unsigned long start, unsigned long end, int err = 0; struct vm_area_struct *vma = walk->vma; const struct mm_walk_ops *ops = walk->ops; - bool is_hugetlb = is_vm_hugetlb_page(vma); + bool is_hugetlb = vma_is_hugetlb(vma); /* We do not support hugetlb PTE installation. */ if (ops->install_pte && is_hugetlb) diff --git a/mm/percpu.c b/mm/percpu.c index 3eff382e565a..d7883507dc15 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, - current->alloc_tag, size); + current->alloc_tag, pcpu_obj_full_size(size)); } } static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) - alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + pcpu_obj_full_size(size)); } #else static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 26643d76bfb0..6e83ce3801b8 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -349,7 +349,7 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * pte_offset_map_lock(mm, pmd, addr, ptlp) is usually called with the pmd * pointer for addr, reached by walking down the mm's pgd, p4d, pud for addr: * either while holding mmap_lock or vma lock for read or for write; or in - * truncate or rmap context, while holding file's i_mmap_lock or anon_vma lock + * truncate or rmap context, while holding file's i_mmap_lock or anon rmap lock * for read (or for write). In a few cases, it may be used with pmd pointing to * a pmd_t already copied to or constructed on the stack. * diff --git a/mm/rmap.c b/mm/rmap.c index 5332c52909be..1131b76bbbc2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -208,7 +208,7 @@ int __anon_vma_prepare(struct vm_area_struct *vma) anon_vma_lock_write(anon_vma); /* page_table_lock to protect against threads */ spin_lock(&mm->page_table_lock); - if (likely(!vma->anon_vma)) { + if (likely(!vma_has_anon_rmap(vma))) { /* * Make anon_vma fields visible before anon_vma is published. * Paired with an address dependency in reusable_anon_vma(). @@ -246,21 +246,21 @@ static void check_anon_vma_clone(struct vm_area_struct *dst, VM_WARN_ON_ONCE(operation != VMA_OP_FORK && dst->vm_mm != src->vm_mm); /* If we have anything to do src->anon_vma must be provided. */ - VM_WARN_ON_ONCE(!src->anon_vma && !list_empty(&src->anon_vma_chain)); - VM_WARN_ON_ONCE(!src->anon_vma && dst->anon_vma); + VM_WARN_ON_ONCE(!vma_has_anon_rmap(src) && !list_empty(&src->anon_vma_chain)); + VM_WARN_ON_ONCE(!vma_has_anon_rmap(src) && vma_has_anon_rmap(dst)); /* We are establishing a new anon_vma_chain. */ VM_WARN_ON_ONCE(!list_empty(&dst->anon_vma_chain)); /* * On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma * must be the same across dst and src. */ - VM_WARN_ON_ONCE(dst->anon_vma && dst->anon_vma != src->anon_vma); + VM_WARN_ON_ONCE(vma_has_anon_rmap(dst) && dst->anon_vma != src->anon_vma); /* * Essentially equivalent to above - if not a no-op, we should expect * dst->anon_vma to be set for everything except a fork. */ - VM_WARN_ON_ONCE(operation != VMA_OP_FORK && src->anon_vma && - !dst->anon_vma); + VM_WARN_ON_ONCE(operation != VMA_OP_FORK && vma_has_anon_rmap(src) && + !vma_has_anon_rmap(dst)); /* For the anon_vma to be compatible, it can only be singular. */ VM_WARN_ON_ONCE(operation == VMA_OP_MERGE_UNFAULTED && !list_is_singular(&src->anon_vma_chain)); @@ -273,7 +273,7 @@ static void maybe_reuse_anon_vma(struct vm_area_struct *dst, struct anon_vma *anon_vma) { /* If already populated, nothing to do.*/ - if (dst->anon_vma) + if (vma_has_anon_rmap(dst)) return; /* @@ -327,7 +327,7 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, check_anon_vma_clone(dst, src, operation); - if (!active_anon_vma) + if (!vma_has_anon_rmap(src)) return 0; /* @@ -384,7 +384,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) int rc; /* Don't bother if the parent process has no anon_vma here. */ - if (!pvma->anon_vma) + if (!vma_has_anon_rmap(pvma)) return 0; /* Drop inherited anon_vma, we'll reuse existing or allocate new. */ @@ -405,7 +405,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) */ rc = anon_vma_clone(vma, pvma, VMA_OP_FORK); /* An error arose or an existing anon_vma was reused, all done then. */ - if (rc || vma->anon_vma) { + if (rc || vma_has_anon_rmap(vma)) { put_anon_vma(anon_vma); anon_vma_chain_free(avc); return rc; @@ -864,7 +864,7 @@ unsigned long page_address_in_vma(const struct folio *folio, * Note: swapoff's unuse_vma() is more efficient with this * check, and needs it to match anon_vma when KSM is active. */ - if (!vma->anon_vma || !anon_vma || + if (!vma_has_anon_rmap(vma) || !anon_vma || vma->anon_vma->root != anon_vma->root) return -EFAULT; /* KSM folios don't reach here because of the !anon_vma check */ @@ -2239,9 +2239,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * If the folio is in an mlock()d vma, we must not swap it out. + * VMA_LOCKONFAULT_BIT alone marks an mlock walk in progress, see + * mlock_vma_pages_range(). */ if (!(flags & TTU_IGNORE_MLOCK) && - (vma->vm_flags & VM_LOCKED)) { + vma_test_any_mask(vma, VMA_LOCKED_MASK)) { ptes++; /* @@ -2273,17 +2275,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { - if (folio_test_lazyfree(folio)) { - if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) - goto walk_done; - /* - * unmap_huge_pmd_locked has either already marked - * the folio as swap-backed or decided to retain it - * due to GUP or speculative references. - */ - goto walk_abort; - } - if (flags & TTU_SPLIT_HUGE_PMD) { /* * We temporarily have to drop the PTL and @@ -2395,7 +2386,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } finish_unmap: folio_remove_rmap_ptes(folio, page, nr_pages, vma); - if (vma->vm_flags & VM_LOCKED) + if (vma_test_any_mask(vma, VMA_LOCKED_MASK)) mlock_drain_local(); folio_put_refs(folio, nr_pages); @@ -2770,7 +2761,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_remove_rmap(folio); else folio_remove_rmap_pte(folio, subpage, vma); - if (vma->vm_flags & VM_LOCKED) + if (vma_test_any_mask(vma, VMA_LOCKED_MASK)) mlock_drain_local(); folio_put(folio); } diff --git a/mm/shmem.c b/mm/shmem.c index eab65f3c268c..af21a89abb0f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1386,11 +1386,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, if (folio) { same_folio = lend < folio_next_pos(folio); folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) { - start = folio_next_index(folio); - if (same_folio) - end = folio->index; - } + truncate_inode_partial_folio(folio, lstart, lend, &start, + same_folio ? &end : NULL); folio_unlock(folio); folio_put(folio); folio = NULL; @@ -1400,8 +1397,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, folio = shmem_get_partial_folio(inode, lend >> PAGE_SHIFT); if (folio) { folio_mark_dirty(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) - end = folio->index; + truncate_inode_partial_folio(folio, lstart, lend, NULL, &end); folio_unlock(folio); folio_put(folio); } @@ -1469,7 +1465,8 @@ whole_folios: if (!folio_test_large(folio)) { truncate_inode_folio(mapping, folio); - } else if (truncate_inode_partial_folio(folio, lstart, lend)) { + } else if (truncate_inode_partial_folio(folio, + lstart, lend, NULL, NULL)) { /* * If we split a page, reset the loop so * that we pick up the new sub pages. diff --git a/mm/shrinker.c b/mm/shrinker.c index 7ec2a9704f6f..4fb424e721bc 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -12,6 +12,12 @@ DEFINE_MUTEX(shrinker_mutex); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +static struct kmem_cache *shrinker_unit_cachep; + +void __init shrinker_unit_cache_init(void) +{ + shrinker_unit_cachep = KMEM_CACHE(shrinker_info_unit, SLAB_PANIC); +} static inline int shrinker_unit_size(int nr_items) { @@ -33,7 +39,7 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start) if (!unit[i]) break; - kfree(unit[i]); + kmem_cache_free(shrinker_unit_cachep, unit[i]); unit[i] = NULL; } } @@ -47,7 +53,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new, int i; for (i = start; i < nr; i++) { - unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid); + unit = kmem_cache_alloc_node(shrinker_unit_cachep, + GFP_KERNEL | __GFP_ZERO, nid); if (!unit) { shrinker_unit_free(new, start); return -ENOMEM; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 34be00c67009..5d7e982f94d0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,12 +32,6 @@ #include <asm/dma.h> #include <asm/tlbflush.h> -/* - * Flags for vmemmap_populate_range and friends. - */ -/* Vmemmap population for ZONE_DEVICE compound pages */ -#define VMEMMAP_POPULATE_DAX 0x0001 - #include "internal.h" #include "mm_init.h" #include "sparse.h" @@ -241,61 +235,66 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon #endif static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap, unsigned long flags) + struct vmem_altmap *altmap) { struct zone *zone; struct page *page; const unsigned int order = pfn_to_section_compound_order(pfn); - /* - * Device DAX still relies on vmemmap_populate_compound_pages() for - * head/first-tail allocation and tail-page reuse. - */ - if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) + if (!vmemmap_optimizable_pfn(pfn)) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - zone = pfn_to_zone(pfn, node); + /* + * Before slab is available, vmemmap optimization is used for early + * system RAM, whose zone can be determined from the PFN. + * + * Once slab is available, only ZONE_DEVICE memory reaches this + * optimized population path. Its zone span has not been initialized + * while its vmemmap is being populated, so pfn_to_zone() cannot be + * used. Obtain ZONE_DEVICE directly from the node instead. + */ + zone = slab_is_available() ? device_zone(node) : pfn_to_zone(pfn, node); page = vmemmap_shared_tail_page(order, zone); if (!page) return NULL; + /* + * During early vmemmap population, the shared tail vmemmap backing + * page is allocated from memblock before its struct page can safely + * participate in page refcounting. Therefore, no reference can be + * held for each shared PTE mapping, and the mappings must be unshared + * before the vmemmap is depopulated. + * + * Once slab is available, the shared backing page is allocated from + * the buddy allocator and can be refcounted. Hold one reference for + * each shared PTE mapping. The architecture vmemmap teardown drops + * the reference through __free_pages() when removing the mapping, + * preventing the backing page from being freed while it is shared. + * + * The backing page may be shared by enough PTE mappings to exhaust + * the positive range of its reference count. Stop populating the + * vmemmap if another reference cannot be acquired. + */ + if (slab_is_available() && !try_get_page(page)) + return NULL; + return page_address(page); } static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn, unsigned long flags) + struct vmem_altmap *altmap) { pte_t *pte = pte_offset_kernel(pmd, addr); unsigned long pfn = page_to_pfn((struct page *)addr); if (pte_none(ptep_get(pte))) { pte_t entry; + void *p = vmemmap_alloc_pte(pfn, node, altmap); - if (ptpfn == (unsigned long)-1) { - void *p = vmemmap_alloc_pte(pfn, node, altmap, flags); - - if (!p) - return NULL; - ptpfn = PHYS_PFN(__pa(p)); - } else { - /* - * When a PTE/PMD entry is freed from the init_mm - * there's a free_pages() call to this page allocated - * above. Thus this try_get_page() is paired with the - * put_page_testzero() on the freeing path. - * This can only called by certain ZONE_DEVICE path, - * and through vmemmap_populate_compound_pages() when - * slab is available. - * - * Use try_get_page() to prevent the shared page refcount - * from overflowing. - */ - if ((flags & VMEMMAP_POPULATE_DAX) && - !try_get_page(pfn_to_page(ptpfn))) - return NULL; - } - entry = pfn_pte(ptpfn, PAGE_KERNEL); + if (!p) + return NULL; + + entry = pfn_pte(PHYS_PFN(__pa(p)), PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) return NULL; @@ -353,10 +352,8 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) { pgd_t *pgd; p4d_t *p4d; @@ -364,51 +361,28 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd_t *pmd; pte_t *pte; - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return NULL; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return NULL; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return NULL; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn, flags); - if (!pte) - return NULL; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - -static int __meminit vmemmap_populate_range(unsigned long start, - unsigned long end, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) -{ - unsigned long addr = start; - pte_t *pte; - - for (; addr < end; addr += PAGE_SIZE) { - pte = vmemmap_populate_address(addr, node, altmap, - ptpfn, flags); + for (unsigned long addr = start; addr < end; addr += PAGE_SIZE) { + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); if (!pte) return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); } return 0; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) -{ - return vmemmap_populate_range(start, end, node, altmap, -1, 0); -} - /* * Write protect the mirrored tail page structs for HVO. This will be * called from the hugetlb code when gathering and initializing the @@ -506,72 +480,6 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return 0; } -#ifndef vmemmap_populate_compound_pages -/* - * For compound pages bigger than section size (e.g. x86 1G compound - * pages with 2M subsection size) fill the rest of sections as tail - * pages. - * - * Note that memremap_pages() resets @nr_range value and will increment - * it after each range successful onlining. Thus the value or @nr_range - * at section memmap populate corresponds to the in-progress range - * being onlined here. - */ -static bool __meminit reuse_compound_section(unsigned long start_pfn, - struct dev_pagemap *pgmap) -{ - unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); - unsigned long offset = start_pfn - - PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); - - return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; -} - -static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, - unsigned long start, - unsigned long end, int node, - struct dev_pagemap *pgmap) -{ - const unsigned long flags = VMEMMAP_POPULATE_DAX; - const unsigned int order = pfn_to_section_compound_order(start_pfn); - unsigned long size, addr; - pte_t *pte; - struct page *page; - int rc; - - page = vmemmap_shared_tail_page(order, device_zone(node)); - if (!page) - return -ENOMEM; - - if (reuse_compound_section(start_pfn, pgmap)) - return vmemmap_populate_range(start, end, node, NULL, - page_to_pfn(page), flags); - - size = min(end - start, (1UL << order) * sizeof(struct page)); - for (addr = start; addr < end; addr += size) { - unsigned long next, last = addr + size; - - /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, -1, flags); - if (!pte) - return -ENOMEM; - - /* - * Reuse the shared page for the rest of tail pages - * See layout diagram in Documentation/mm/vmemmap_dedup.rst - */ - next = addr + PAGE_SIZE; - rc = vmemmap_populate_range(next, last, node, NULL, - page_to_pfn(page), flags); - if (rc) - return -ENOMEM; - } - - return 0; -} - -#endif - struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -584,11 +492,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) - r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); - else - r = vmemmap_populate(start, end, nid, altmap); - + r = vmemmap_populate(start, end, nid, altmap); if (r < 0) return NULL; @@ -903,12 +807,12 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (!section_vmemmap_optimizable(ms)) page_init_poison(memmap, sizeof(struct page) * nr_pages); - __section_mark_present(ms, section_nr); + __highest_used_section_nr = max(section_nr, __highest_used_section_nr); /* Align memmap to section boundary in the subsection case */ if (section_nr_to_pfn(section_nr) != start_pfn) memmap = pfn_to_page(section_nr_to_pfn(section_nr)); - sparse_init_one_section(ms, section_nr, memmap, ms->usage, 0); + sparse_init_one_section(ms, section_nr, memmap, ms->usage); return 0; } diff --git a/mm/sparse.c b/mm/sparse.c index 36692b1dd733..6acb628c8a73 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -103,11 +103,22 @@ int __meminit sparse_index_init(unsigned long section_nr, int nid) return 0; } + +static void __init sparse_extreme_init(void) +{ + const unsigned long size = sizeof(struct mem_section *) * NR_SECTION_ROOTS; + + mem_section = memblock_alloc_or_panic(size, INTERNODE_CACHE_BYTES); +} #else /* !SPARSEMEM_EXTREME */ int __meminit sparse_index_init(unsigned long section_nr, int nid) { return 0; } + +static void __init sparse_extreme_init(void) +{ +} #endif /* @@ -153,61 +164,44 @@ static void __init mminit_validate_memmodel_limits(unsigned long *start_pfn, } /* - * There are a number of times that we loop over NR_MEM_SECTIONS, - * looking for section_present() on each. But, when we have very - * large physical address spaces, NR_MEM_SECTIONS can also be - * very large which makes the loops quite long. - * - * Keeping track of this gives us an easy way to break out of - * those loops early. + * Looping over all possible memory sections is expensive, especially if + * NR_MEM_SECTIONS is large but only a fraction is actually used. Keep track of + * the highest section number we ever used. */ -unsigned long __highest_present_section_nr; - -static inline unsigned long first_present_section_nr(void) -{ - return next_present_section_nr(-1); -} +unsigned long __highest_used_section_nr; -/* Record a memory area against a node. */ -static void __init memory_present(int nid, unsigned long start, unsigned long end) +static inline unsigned long first_early_section_nr(void) { - unsigned long pfn; - - start &= PAGE_SECTION_MASK; - mminit_validate_memmodel_limits(&start, &end); - for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) { - unsigned long section_nr = pfn_to_section_nr(pfn); - struct mem_section *ms; - - sparse_index_init(section_nr, nid); - set_section_nid(section_nr, nid); - - ms = __nr_to_section(section_nr); - if (!ms->section_mem_map) { - ms->section_mem_map = sparse_encode_early_nid(nid) | - SECTION_IS_ONLINE; - __section_mark_present(ms, section_nr); - } - } + return next_early_section_nr(-1); } void __init sparse_sections_init(void) { - unsigned long start, end; + const unsigned long flags = SECTION_IS_EARLY | SECTION_IS_ONLINE; + unsigned long pfn, start_pfn, end_pfn, section_nr; int i, nid; -#ifdef CONFIG_SPARSEMEM_EXTREME - unsigned long size, align; + sparse_extreme_init(); - size = sizeof(struct mem_section *) * NR_SECTION_ROOTS; - align = 1 << (INTERNODE_CACHE_SHIFT); - mem_section = memblock_alloc_or_panic(size, align); -#endif + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + start_pfn &= PAGE_SECTION_MASK; + mminit_validate_memmodel_limits(&start_pfn, &end_pfn); - for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) - memory_present(nid, start, end); -} + for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) { + struct mem_section *ms; + + section_nr = pfn_to_section_nr(pfn); + sparse_index_init(section_nr, nid); + ms = __nr_to_section(section_nr); + if (ms->section_mem_map) + continue; + set_section_nid(section_nr, nid); + ms->section_mem_map = sparse_encode_early_nid(nid) | flags; + } + } + __highest_used_section_nr = section_nr; +} #ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, @@ -223,72 +217,72 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -/* - * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_end) - * And number of present sections in this node is map_count. - */ -static void __init sparse_init_nid(int nid, unsigned long pnum_begin, - unsigned long pnum_end, - unsigned long map_count) +static void __init sparse_metadata_init_nid(int nid, + unsigned long start_section_nr, unsigned long end_section_nr, + unsigned long nr_sections) { - unsigned long pnum; struct mem_section_usage *usage; + unsigned long section_nr; - usage = memblock_alloc_node(map_count * mem_section_usage_size(), + usage = memblock_alloc_node(nr_sections * mem_section_usage_size(), SMP_CACHE_BYTES, nid); if (!usage) panic("Failed to allocate usemap for node %d\n", nid); - for_each_present_section_nr(pnum_begin, pnum) { - unsigned long pfn = section_nr_to_pfn(pnum); - struct page *map; + for_each_early_section_nr(start_section_nr, section_nr) { + const unsigned long pfn = section_nr_to_pfn(section_nr); + struct page *mem_map; - if (pnum >= pnum_end) + if (section_nr >= end_section_nr) break; - map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); + mem_map = __populate_section_memmap(pfn, PAGES_PER_SECTION, nid, + NULL, NULL); + if (!mem_map) + panic("Failed to allocate memmap for section %lu\n", + section_nr); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, - SECTION_IS_EARLY); + sparse_init_one_section(__nr_to_section(section_nr), section_nr, + mem_map, usage); usage = (void *)usage + mem_section_usage_size(); } } +static void __init sparse_metadata_init(void) +{ + unsigned long start_section_nr = first_early_section_nr(); + int nid_begin = sparse_early_nid(__nr_to_section(start_section_nr)); + unsigned long section_nr, nr_sections = 1; + + for_each_early_section_nr(start_section_nr + 1, section_nr) { + const int nid = sparse_early_nid(__nr_to_section(section_nr)); + + if (nid == nid_begin) { + nr_sections++; + continue; + } + sparse_metadata_init_nid(nid_begin, start_section_nr, + section_nr, nr_sections); + nid_begin = nid; + start_section_nr = section_nr; + nr_sections = 1; + } + sparse_metadata_init_nid(nid_begin, start_section_nr, section_nr, + nr_sections); +} + /* * Allocate the accumulated non-linear sections, allocate a mem_map * for each and record the physical to section mapping. */ void __init sparse_init(void) { - unsigned long pnum_end, pnum_begin, map_count = 1; - int nid_begin; - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); } - pnum_begin = first_present_section_nr(); - nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); - - for_each_present_section_nr(pnum_begin + 1, pnum_end) { - int nid = sparse_early_nid(__nr_to_section(pnum_end)); - - if (nid == nid_begin) { - map_count++; - continue; - } - /* Init node with sections in range [pnum_begin, pnum_end) */ - sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count); - nid_begin = nid; - pnum_begin = pnum_end; - map_count = 1; - } - /* cover the last node */ - sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count); + sparse_metadata_init(); sparse_init_subsection_map(); vmemmap_populate_print_last(); } diff --git a/mm/sparse.h b/mm/sparse.h index 530692cdd516..fdbc4cacf91d 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -20,36 +20,27 @@ void sparse_sections_init(void); int sparse_index_init(unsigned long section_nr, int nid); static inline void sparse_init_one_section(struct mem_section *ms, - unsigned long pnum, struct page *mem_map, - struct mem_section_usage *usage, unsigned long flags) + unsigned long section_nr, struct page *mem_map, + struct mem_section_usage *usage) { unsigned long coded_mem_map; BUILD_BUG_ON(SECTION_MAP_LAST_BIT > PFN_SECTION_SHIFT); + VM_WARN_ON_ONCE(section_nr > __highest_used_section_nr); /* * We encode the start PFN of the section into the mem_map such that * page_to_pfn() on !CONFIG_SPARSEMEM_VMEMMAP can simply subtract it * from the page pointer to obtain the PFN. */ - coded_mem_map = (unsigned long)(mem_map - section_nr_to_pfn(pnum)); + coded_mem_map = (unsigned long)(mem_map - section_nr_to_pfn(section_nr)); VM_WARN_ON_ONCE(coded_mem_map & ~SECTION_MAP_MASK); ms->section_mem_map &= ~SECTION_MAP_MASK; - ms->section_mem_map |= coded_mem_map; - ms->section_mem_map |= flags | SECTION_HAS_MEM_MAP; + ms->section_mem_map |= coded_mem_map | SECTION_HAS_MEM_MAP; ms->usage = usage; } -static inline void __section_mark_present(struct mem_section *ms, - unsigned long section_nr) -{ - if (section_nr > __highest_present_section_nr) - __highest_present_section_nr = section_nr; - - ms->section_mem_map |= SECTION_MARKED_PRESENT; -} - static inline size_t mem_section_usage_size(void) { return struct_size_t(struct mem_section_usage, pageblock_flags, diff --git a/mm/swap_state.c b/mm/swap_state.c index 2475ba29126d..d9e36c505398 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -92,17 +92,27 @@ void show_swap_cache_info(void) */ struct folio *swap_cache_get_folio(swp_entry_t entry) { + struct swap_cluster_info *ci = __swap_entry_to_cluster(entry); + unsigned int ci_off = swp_cluster_offset(entry); unsigned long swp_tb; struct folio *folio; for (;;) { - swp_tb = swap_table_get(__swap_entry_to_cluster(entry), - swp_cluster_offset(entry)); + swp_tb = swap_table_get(ci, ci_off); if (!swp_tb_is_folio(swp_tb)) return NULL; folio = swp_tb_to_folio(swp_tb); - if (likely(folio_try_get(folio))) + if (!folio_try_get(folio)) + continue; + /* + * The folio may have been freed and its memory reused before + * we got the reference. Make sure it is still in the slot. + */ + swp_tb = swap_table_get(ci, ci_off); + if (likely(swp_tb_is_folio(swp_tb) && + swp_tb_to_folio(swp_tb) == folio)) return folio; + folio_put(folio); } return NULL; @@ -906,8 +916,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, continue; folio_put(folio); } - blk_finish_plug(&plug); swap_read_submit(&ctx); + blk_finish_plug(&plug); skip: return swap_cache_read_folio_sync(entry, gfp_mask, mpol, ilx); } @@ -1019,8 +1029,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, } if (pte) pte_unmap(pte); - blk_finish_plug(&plug); swap_read_submit(&ctx); + blk_finish_plug(&plug); skip: /* The folio was likely read above, so no need for plugging here */ return swap_cache_read_folio_sync(targ_entry, gfp_mask, mpol, targ_ilx); diff --git a/mm/swapfile.c b/mm/swapfile.c index 16ceaa28db3a..254ce86fa923 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2313,6 +2313,67 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset) } /** + * repin_hibernation_swap_type - Atomically replace the hibernation pin + * @old_type: Swap type currently pinned (or < 0 if none). + * @device: Block device of the new resume image. + * @offset: Offset identifying the new swap area. + * + * Look up the swap device for @device/@offset and atomically transfer + * the SWP_HIBERNATION pin from @old_type (if valid) to the new device, + * all under a single swap_lock critical section. This closes the + * swapoff() window that exists when callers unpin and re-pin in two + * separate operations. + * + * If the new device cannot be located, the existing pin on @old_type + * is preserved and an error is returned. If @old_type already refers + * to the same swap_info_struct as the new lookup, no flag changes are + * made and @old_type is returned. + * + * Return: + * >= 0 on success (new swap type). + * -EINVAL if @device is invalid. + * -ENODEV if the swap device is not found. + * -EBUSY if the new device is already pinned by another context. + */ +int repin_hibernation_swap_type(int old_type, dev_t device, sector_t offset) +{ + struct swap_info_struct *old_si, *new_si; + int new_type; + + spin_lock(&swap_lock); + + new_type = __find_hibernation_swap_type(device, offset); + if (new_type < 0) { + spin_unlock(&swap_lock); + return new_type; + } + + new_si = swap_type_to_info(new_type); + if (WARN_ON_ONCE(!new_si)) { + spin_unlock(&swap_lock); + return -ENODEV; + } + + old_si = swap_type_to_info(old_type); + if (new_si == old_si) { + spin_unlock(&swap_lock); + return new_type; + } + + if (WARN_ON_ONCE(new_si->flags & SWP_HIBERNATION)) { + spin_unlock(&swap_lock); + return -EBUSY; + } + + if (old_si) + old_si->flags &= ~SWP_HIBERNATION; + new_si->flags |= SWP_HIBERNATION; + + spin_unlock(&swap_lock); + return new_type; +} + +/** * unpin_hibernation_swap_type - Unpin the swap device for hibernation * @type: Swap type previously returned by pin_hibernation_swap_type() * @@ -2707,7 +2768,7 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type) if (check_stable_address_space(mm)) goto unlock; for_each_vma(vmi, vma) { - if (vma->anon_vma && !is_vm_hugetlb_page(vma)) { + if (vma_has_anon_rmap(vma) && !vma_is_hugetlb(vma)) { ret = unuse_vma(vma, type); if (ret) break; diff --git a/mm/truncate.c b/mm/truncate.c index b58ba940be47..733d2713a875 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -206,35 +206,47 @@ static int folio_split_or_unmap(struct folio *folio, struct page *split_at, /* * Handle partial folios. The folio may be entirely within the * range if a split has raced with us. If not, we zero the part of the - * folio that's within the [start, end] range, and then split the folio if + * folio that's within the [lstart, lend] range, and then split the folio if * it's large. split_page_range() will discard pages which now lie beyond * i_size, and we rely on the caller to discard pages which lie within a * newly created hole. * - * Returns false if splitting failed so the caller can avoid - * discarding the entire folio which is stubbornly unsplit. + * When @pstart and/or @pend are non-NULL they receive the indexes of the + * folio range fully covered by [lstart, lend] after any split (or none), + * aligned inwards to min_order, i.e. the range of folios wholly within + * [lstart, lend] and so safe to discard. + * + * Return %true if at least one split succeeded, %false otherwise. */ -bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) +bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, + loff_t lend, pgoff_t *pstart, pgoff_t *pend) { loff_t pos = folio_pos(folio); size_t size = folio_size(folio); unsigned int offset, length; struct page *split_at, *split_at2; + unsigned long min_nrbytes; unsigned int min_order; - if (pos < start) - offset = start - pos; + if (pos < lstart) + offset = lstart - pos; else offset = 0; - if (pos + size <= (u64)end) + if (pos + size <= (u64)lend) length = size - offset; else - length = end + 1 - pos - offset; + length = lend + 1 - pos - offset; + + if (pstart) + *pstart = offset ? folio_next_index(folio) : folio->index; + if (pend) + *pend = (pos + size > (u64)lend) ? folio->index : + folio_next_index(folio); folio_wait_writeback(folio); if (length == size) { truncate_inode_folio(folio->mapping, folio); - return true; + return false; } /* @@ -247,10 +259,12 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) if (folio_needs_release(folio)) folio_invalidate(folio, offset, length); - if (!folio_test_large(folio)) - return true; min_order = mapping_min_folio_order(folio->mapping); + min_nrbytes = mapping_min_folio_nrbytes(folio->mapping); + if (folio_order(folio) == min_order) + return false; + split_at = folio_page(folio, PAGE_ALIGN_DOWN(offset) / PAGE_SIZE); if (!folio_split_or_unmap(folio, split_at, min_order)) { /* @@ -259,38 +273,62 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) * for shmem truncate */ struct folio *folio2; + pgoff_t end, aligned_end = round_down(pos + offset + length, + min_nrbytes) >> PAGE_SHIFT; - if (offset + length == size) - goto no_split; + if (pstart) + *pstart = round_up(pos + offset, + min_nrbytes) >> PAGE_SHIFT; - split_at2 = folio_page(folio, - PAGE_ALIGN_DOWN(offset + length) / PAGE_SIZE); - folio2 = page_folio(split_at2); - - if (!folio_try_get(folio2)) - goto no_split; - - if (!folio_test_large(folio2)) + end = aligned_end; + if (offset + length == size) goto out; - if (!folio_trylock(folio2)) + /* + * After the first split at the start edge, the folio at the + * end edge may be freed and reused concurrently. + * __filemap_get_folio() looks up the straddler at aligned_end + * and returns it locked and ref'd with the mapping + * validated. + */ + folio2 = __filemap_get_folio(folio->mapping, aligned_end, + FGP_LOCK | FGP_NOWAIT, 0); + if (IS_ERR(folio2)) { + /* + * No sub-folio straddles the boundary when + * aligned_end is empty, so discarding up to it is + * safe. Otherwise the straddler is locked by + * someone else and we cannot obtain a reliable end + * position, so we fall back to folio->index, which + * is safe but leaves the sub-folios split off at + * the offset edge in the page cache. + */ + if (PTR_ERR(folio2) != -ENOENT) + end = folio->index; goto out; + } + + /* Already at the minimum order, nothing to split */ + if (folio_order(folio2) == min_order) + goto out_put; + + split_at2 = folio_page(folio2, (aligned_end - folio2->index)); - /* make sure folio2 is large and does not change its mapping */ - if (folio_test_large(folio2) && - folio2->mapping == folio->mapping) - folio_split_or_unmap(folio2, split_at2, min_order); + /* Split failed, keep the straddler intact */ + if (folio_split_or_unmap(folio2, split_at2, min_order)) + end = folio2->index; +out_put: folio_unlock(folio2); -out: folio_put(folio2); -no_split: +out: + if (pend) + *pend = end; return true; } - if (folio_test_dirty(folio)) - return false; - truncate_inode_folio(folio->mapping, folio); - return true; + if (!folio_test_dirty(folio)) + truncate_inode_folio(folio->mapping, folio); + return false; } /* @@ -345,9 +383,11 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio) * @lstart: offset from which to truncate * @lend: offset to which to truncate (inclusive) * - * Truncate the page cache, removing the pages that are between - * specified offsets (and zeroing out partial pages - * if lstart or lend + 1 is not page aligned). + * Truncate the page cache, removing the folios that are between specified + * offsets (and zeroing out partial folios if lstart or lend + 1 is not + * folio aligned). For mappings with a non-zero minimum folio order, the + * boundaries are aligned inwards to 1 << min_order so the edge sub-folio + * straddling the range is kept. * * Truncate takes two passes - the first pass is nonblocking. It will not * block on page locks and it will not block on writeback. The second pass @@ -366,6 +406,7 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio) void truncate_inode_pages_range(struct address_space *mapping, loff_t lstart, uoff_t lend) { + pgoff_t min_nrpages = mapping_min_folio_nrpages(mapping); pgoff_t start; /* inclusive */ pgoff_t end; /* exclusive */ struct folio_batch fbatch; @@ -379,9 +420,8 @@ void truncate_inode_pages_range(struct address_space *mapping, return; /* - * 'start' and 'end' always covers the range of pages to be fully - * truncated. Partial pages are covered with 'partial_start' at the - * start of the range and 'partial_end' at the end of the range. + * 'start' and 'end' always covers the range of folios to be fully + * truncated, with both boundaries aligned inwards to 1 << min_order. * Note that 'end' is exclusive while 'lend' is inclusive. */ start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT; @@ -395,6 +435,10 @@ void truncate_inode_pages_range(struct address_space *mapping, else end = (lend + 1) >> PAGE_SHIFT; + start = round_up(start, min_nrpages); + if (end != (pgoff_t)-1) + end = round_down(end, min_nrpages); + folio_batch_init(&fbatch); index = start; while (index < end && find_lock_entries(mapping, &index, end - 1, @@ -413,11 +457,8 @@ void truncate_inode_pages_range(struct address_space *mapping, folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0); if (!IS_ERR(folio)) { same_folio = lend < folio_next_pos(folio); - if (!truncate_inode_partial_folio(folio, lstart, lend)) { - start = folio_next_index(folio); - if (same_folio) - end = folio->index; - } + truncate_inode_partial_folio(folio, lstart, lend, &start, + same_folio ? &end : NULL); folio_unlock(folio); folio_put(folio); folio = NULL; @@ -427,8 +468,8 @@ void truncate_inode_pages_range(struct address_space *mapping, folio = __filemap_get_folio(mapping, lend >> PAGE_SHIFT, FGP_LOCK, 0); if (!IS_ERR(folio)) { - if (!truncate_inode_partial_folio(folio, lstart, lend)) - end = folio->index; + truncate_inode_partial_folio(folio, lstart, lend, + NULL, &end); folio_unlock(folio); folio_put(folio); } @@ -878,8 +919,7 @@ void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to) unsigned int offset, end; offset = from - folio_pos(folio); - end = min_t(unsigned int, to - folio_pos(folio), - folio_size(folio)); + end = min_t(loff_t, to - folio_pos(folio), folio_size(folio)); folio_zero_segment(folio, offset, end); } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 938f20e9a78f..928ad95eeb2b 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -115,8 +115,7 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct mm_struct *mm, vma = vma_lookup(mm, addr); if (!vma) vma = ERR_PTR(-ENOENT); - else if (!(vma->vm_flags & VM_SHARED) && - unlikely(anon_vma_prepare(vma))) + else if (vma_is_cow_mapping(vma) && unlikely(anon_vma_prepare(vma))) vma = ERR_PTR(-ENOMEM); return vma; @@ -130,7 +129,7 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct mm_struct *mm, * Should be called without holding mmap_lock. * * Return: A locked vma containing @address, -ENOENT if no vma is found, - * -ENOMEM if anon_vma couldn't be allocated, or -EAGAIN if vma refcount + * -ENOMEM if the anon rmap couldn't be allocated, or -EAGAIN if vma refcount * overflow happened due to high number of readers and the caller should * retry later. */ @@ -142,10 +141,10 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, vma = lock_vma_under_rcu(mm, address); if (vma) { /* - * We know we're going to need to use anon_vma, so check - * that early. + * We know we're going to need an anon rmap, so check that + * early. */ - if (!(vma->vm_flags & VM_SHARED) && unlikely(!vma->anon_vma)) + if (vma_is_cow_mapping(vma) && unlikely(!vma_has_anon_rmap(vma))) vma_end_read(vma); else return vma; @@ -237,7 +236,7 @@ static int mfill_get_vma(struct mfill_state *state) if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP)) goto out_unlock; - if (is_vm_hugetlb_page(dst_vma)) + if (vma_is_hugetlb(dst_vma)) return 0; ops = vma_uffd_ops(dst_vma); @@ -804,7 +803,7 @@ retry: } err = -ENOENT; - if (!is_vm_hugetlb_page(dst_vma)) + if (!vma_is_hugetlb(dst_vma)) goto out_unlock_vma; err = -EINVAL; @@ -967,7 +966,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, /* * If this is a HUGETLB vma, pass off to appropriate routine */ - if (is_vm_hugetlb_page(state.vma)) + if (vma_is_hugetlb(state.vma)) return mfill_atomic_hugetlb(ctx, state.vma, dst_start, src_start, len, flags); @@ -1114,7 +1113,7 @@ static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, break; } - if (is_vm_hugetlb_page(dst_vma)) { + if (vma_is_hugetlb(dst_vma)) { err = -EINVAL; page_mask = vma_kernel_pagesize(dst_vma) - 1; if ((start & page_mask) || (len & page_mask)) @@ -1172,7 +1171,7 @@ int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start, if (!userfaultfd_rwp(dst_vma)) return -ENOENT; - if (is_vm_hugetlb_page(dst_vma)) { + if (vma_is_hugetlb(dst_vma)) { unsigned long page_mask; page_mask = vma_kernel_pagesize(dst_vma) - 1; @@ -1636,8 +1635,12 @@ retry: pte_unmap(dst_pte); src_pte = dst_pte = NULL; ret = split_folio(src_folio); - if (ret) + if (ret) { + /* do not retry, extra refs may be held forever */ + if (ret == -EAGAIN) + ret = -EBUSY; goto out; + } /* have to reacquire the folio after it got split */ folio_unlock(src_folio); folio_put(src_folio); @@ -1680,7 +1683,7 @@ retry: /* * Verify the existence of the swapcache. If present, the folio's * index and mapping must be updated even when the PTE is a swap - * entry. The anon_vma lock is not taken during this process since + * entry. The anon rmap lock is not taken during this process since * the folio has already been unmapped, and the swap entry is * exclusive, preventing rmap walks. * @@ -1754,10 +1757,16 @@ static inline bool move_splits_huge_pmd(unsigned long dst_addr, } #endif -static inline bool vma_move_compatible(struct vm_area_struct *vma) +static inline bool vma_move_compatible(const struct vm_area_struct *vma) { - return !(vma->vm_flags & (VM_PFNMAP | VM_IO | VM_HUGETLB | - VM_MIXEDMAP | VM_SHADOW_STACK)); + /* uffd is generally incompatible with mappings not managed by core mm. */ + if (!vma_is_mm_managed(vma)) + return false; + /* The shadow stack should not be written to by userspace. */ + if (vma_test_single_mask(vma, VMA_SHADOW_STACK)) + return false; + /* hugetlb mappings cannot be safely moved. */ + return !vma_is_hugetlb(vma); } static int validate_move_areas(struct userfaultfd_ctx *ctx, @@ -1909,7 +1918,7 @@ static void uffd_move_unlock(struct vm_area_struct *dst_vma, * * move_pages() remaps arbitrary anonymous pages atomically in zero * copy. It only works on non shared anonymous pages because those can - * be relocated without generating non linear anon_vmas in the rmap + * be relocated without generating non linear anon rmaps in the rmap * code. * * It provides a zero copy mechanism to handle userspace page faults. @@ -2146,10 +2155,14 @@ static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, { const struct vm_uffd_ops *ops = vma_uffd_ops(vma); - if (vma->vm_flags & (VM_DROPPABLE | VM_SHADOW_STACK)) + /* Only mm-backed memory can have its faults handled by userspace. */ + if (!vma_is_mm_backed(vma)) return false; - - if (!is_vm_hugetlb_page(vma) && (vma->vm_flags & VM_SPECIAL)) + /* Core mm may discard droppable memory at any time. */ + if (vma_test_single_mask(vma, VMA_DROPPABLE)) + return false; + /* The shadow stack should not be written to by userspace. */ + if (vma_test_single_mask(vma, VMA_SHADOW_STACK)) return false; vm_flags &= __VM_UFFD_FLAGS; @@ -2319,7 +2332,7 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx, */ userfaultfd_set_ctx(vma, ctx, vm_flags); - if (is_vm_hugetlb_page(vma) && uffd_disable_huge_pmd_share(vma)) + if (vma_is_hugetlb(vma) && uffd_disable_huge_pmd_share(vma)) hugetlb_unshare_all_pmds(vma); skip: @@ -2895,7 +2908,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * (sleepable) vma lock can modify the current task state, that * must be before explicitly calling set_current_state(). */ - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_vma_lock_read(vma); spin_lock_irq(&ctx->fault_pending_wqh.lock); @@ -2912,7 +2925,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) set_current_state(blocking_state); spin_unlock_irq(&ctx->fault_pending_wqh.lock); - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason); hugetlb_vma_unlock_read(vma); } else { @@ -3744,7 +3757,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * If the first vma contains huge pages, make sure start address * is aligned to huge page size. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { unsigned long vma_hpagesize = vma_kernel_pagesize(vma); if (start & (vma_hpagesize - 1)) @@ -3795,7 +3808,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * If this vma contains ending address, and huge pages * check alignment. */ - if (is_vm_hugetlb_page(cur) && end <= cur->vm_end && + if (vma_is_hugetlb(cur) && end <= cur->vm_end && end > cur->vm_start) { unsigned long vma_hpagesize = vma_kernel_pagesize(cur); @@ -3831,7 +3844,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, /* * Note vmas containing huge pages */ - if (is_vm_hugetlb_page(cur)) + if (vma_is_hugetlb(cur)) basic_ioctls = true; found = true; @@ -3917,7 +3930,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * If the first vma contains huge pages, make sure start address * is aligned to huge page size. */ - if (is_vm_hugetlb_page(vma)) { + if (vma_is_hugetlb(vma)) { unsigned long vma_hpagesize = vma_kernel_pagesize(vma); if (start & (vma_hpagesize - 1)) diff --git a/mm/util.c b/mm/util.c index 75c0fc2061aa..8acfe2148ba5 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1284,23 +1284,35 @@ EXPORT_SYMBOL(compat_set_desc_from_vma); int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma) { + struct vm_area_desc orig_desc; int err; + /* Derive state prior to mmap_prepare hook. */ + compat_set_desc_from_vma(&orig_desc, desc->file, vma); /* Perform any preparatory tasks for mmap action. */ err = mmap_action_prepare(desc); if (err) - return err; + goto err_put; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); + if (err) + goto err_put; /* Update the VMA from the descriptor. */ compat_set_vma_from_desc(vma, desc); /* Complete any specified mmap actions. */ return mmap_action_complete(vma, &desc->action, /*is_compat=*/true); + +err_put: + if (desc->vm_file != vma->vm_file) + fput(desc->vm_file); + return err; } EXPORT_SYMBOL(__compat_vma_mmap); /** * compat_vma_mmap() - Apply the file's .mmap_prepare() hook to an * existing VMA and execute any requested actions. - * @file: The file which possesss an f_op->mmap_prepare() hook. + * @file: The file which possesses an f_op->mmap_prepare() hook. * @vma: The VMA to apply the .mmap_prepare() hook to. * * Ordinarily, .mmap_prepare() is invoked directly upon mmap(). However, certain @@ -1449,6 +1461,15 @@ static int call_vma_mapped(struct vm_area_struct *vma) return 0; } +/* An mmap action failed under the compatibility layer - unmap and close. */ +static void compat_mmap_action_abort(struct vm_area_struct *vma) +{ +#ifdef CONFIG_MMU + zap_vma_range(vma, vma->vm_start, vma_pages(vma) << PAGE_SHIFT); +#endif + vma_close(vma); +} + static int mmap_action_finish(struct vm_area_struct *vma, struct mmap_action *action, int err, bool is_compat) @@ -1460,12 +1481,13 @@ static int mmap_action_finish(struct vm_area_struct *vma, /* do_munmap() might take rmap lock, so release if held. */ maybe_rmap_unlock_action(vma, action); - /* - * If this is invoked from the compatibility layer, post-mmap() hook - * logic will handle cleanup for us. - */ - if (!err || is_compat) + if (!err) + return 0; + + if (is_compat) { + compat_mmap_action_abort(vma); return err; + } /* * If an error occurs, unmap the VMA altogether and return an error. We @@ -1515,8 +1537,10 @@ int mmap_action_prepare(struct vm_area_desc *desc) return io_remap_pfn_range_prepare(desc); case MMAP_SIMPLE_IO_REMAP: return simple_ioremap_prepare(desc); - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: return map_kernel_pages_prepare(desc); + case MMAP_DISCONTIG_KERNEL_PAGES: + return map_discontig_kernel_pages_prepare(desc); } WARN_ON_ONCE(1); @@ -1546,9 +1570,12 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_REMAP_PFN: err = remap_pfn_range_complete(vma, action); break; - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: err = map_kernel_pages_complete(vma, action); break; + case MMAP_DISCONTIG_KERNEL_PAGES: + err = map_discontig_kernel_pages_complete(vma, action); + break; case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: /* Should have been delegated. */ @@ -1569,7 +1596,8 @@ int mmap_action_prepare(struct vm_area_desc *desc) case MMAP_REMAP_PFN: case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle these. */ break; } @@ -1590,7 +1618,8 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_REMAP_PFN: case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: - case MMAP_MAP_KERNEL_PAGES: + case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle this. */ err = -EINVAL; @@ -24,7 +24,8 @@ struct mmap_state { vm_flags_t vm_flags; vma_flags_t vma_flags; }; - struct file *file; + struct file *file; /* mmap()-specified file. */ + struct file *vm_file; /* May be updated by mmap_prepare. */ pgprot_t page_prot; /* User-defined fields, perhaps updated by .mmap_prepare(). */ @@ -43,8 +44,6 @@ struct mmap_state { /* Determine if we can check KSM flags early in mmap() logic. */ bool check_ksm_early :1; - /* If .mmap_prepare changed the file, we don't need to pin. */ - bool file_doesnt_need_get :1; }; #define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, anon_pgoff_, vma_flags_, file_) \ @@ -58,6 +57,7 @@ struct mmap_state { .pglen = PHYS_PFN(len_), \ .vma_flags = vma_flags_, \ .file = file_, \ + .vm_file = file_, \ .page_prot = vma_flags_to_page_prot(vma_flags_), \ } @@ -70,7 +70,7 @@ struct mmap_state { .vma_flags = (map_)->vma_flags, \ .pgoff = (map_)->pgoff, \ .anon_pgoff = (map_)->anon_pgoff, \ - .file = (map_)->file, \ + .file = (map_)->vm_file, \ .prev = (map_)->prev, \ .middle = vma_, \ .next = (vma_) ? NULL : (map_)->next, \ @@ -97,10 +97,11 @@ static bool vma_is_fork_child(struct vm_area_struct *vma) { /* * The list_is_singular() test is to avoid merging VMA cloned from - * parents. This can improve scalability caused by the anon_vma root + * parents. This can improve scalability caused by the anon rmap root * lock. */ - return vma && vma->anon_vma && !list_is_singular(&vma->anon_vma_chain); + return vma && vma_has_anon_rmap(vma) && + !list_is_singular(&vma->anon_vma_chain); } static inline bool is_mergeable_vma(struct vma_merge_struct *vmg, bool merge_next) @@ -134,13 +135,13 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) /* * We _can_ have !src, vmg->anon_vma via copy_vma(). In this instance we - * will remove the existing VMA's anon_vma's so there's no scalability + * will remove the existing VMA's anon rmap so there's no scalability * concerns. */ VM_WARN_ON(src && src_anon != src->anon_vma); /* Case 1 - we will dup_anon_vma() from src into tgt. */ - if (!tgt_anon && src_anon) { + if (!vma_has_anon_rmap(tgt) && src_anon) { struct vm_area_struct *copied_from = vmg->copied_from; if (vma_is_fork_child(src)) @@ -150,10 +151,10 @@ static bool is_mergeable_anon_vma(struct vma_merge_struct *vmg, bool merge_next) return true; } - /* Case 2 - we will simply use tgt's anon_vma. */ - if (tgt_anon && !src_anon) + /* Case 2 - we will simply use tgt's anon rmap. */ + if (vma_has_anon_rmap(tgt) && !src_anon) return !vma_is_fork_child(tgt); - /* Case 3 - the anon_vma's are already shared. */ + /* Case 3 - src and tgt already share an anon rmap. */ return src_anon == tgt_anon; } @@ -190,18 +191,25 @@ static void init_multi_vma_prep(struct vma_prepare *vp, adjust = NULL; vp->adj_next = adjust; - if (!vp->anon_vma && adjust) + if (!vma_has_anon_rmap(vma) && adjust) vp->anon_vma = adjust->anon_vma; - VM_WARN_ON(vp->anon_vma && adjust && adjust->anon_vma && + VM_WARN_ON(vma_has_anon_rmap(vma) && adjust && vma_has_anon_rmap(adjust) && vp->anon_vma != adjust->anon_vma); vp->file = vma->vm_file; if (vp->file) vp->mapping = vma->vm_file->f_mapping; - if (vmg && vmg->skip_vma_uprobe) + if (!vmg) + return; + + if (vmg->skip_vma_uprobe) vp->skip_vma_uprobe = true; + if (vma_start_pgoff(vma) == vmg_start_pgoff(vmg)) + vp->file_pgoff_unchanged = true; + if (vma_start_anon_pgoff(vma) == vmg_start_anon_pgoff(vmg)) + vp->anon_pgoff_unchanged = true; } /* @@ -227,8 +235,8 @@ static bool needs_adjacent_anon_pgoff(const struct vma_merge_struct *vmg) * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff) * in front of (at a lower virtual address and file offset than) the vma. * - * We cannot merge two vmas if they have differently assigned (non-NULL) - * anon_vmas, nor if same anon_vma is assigned but offsets incompatible. + * We cannot merge two vmas if they have differently assigned anon rmaps, + * nor if the same anon rmap is assigned but offsets incompatible. * * We don't check here for the merged mmap wrapping around the end of pagecache * indices (16TB on ia32) because do_mmap() does not permit mmap's which @@ -254,8 +262,8 @@ static bool can_vma_merge_before(struct vma_merge_struct *vmg) * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff) * beyond (at a higher virtual address and file offset than) the vma. * - * We cannot merge two vmas if they have differently assigned (non-NULL) - * anon_vmas, nor if same anon_vma is assigned but offsets incompatible. + * We cannot merge two vmas if they have differently assigned anon rmaps, + * nor if the same anon rmap is assigned but offsets incompatible. * * We assume that vma is not removed as part of the merge. */ @@ -299,38 +307,6 @@ static void __remove_shared_vm_struct(struct vm_area_struct *vma, } /* - * vma has some anon_vma assigned, and is already inserted on that - * anon_vma's interval trees. - * - * Before updating the vma's vm_start / vm_end / vm_pgoff fields, the - * vma must be removed from the anon_vma's interval trees using - * anon_rmap_tree_pre_update_vma(). - * - * After the update, the vma will be reinserted using - * anon_rmap_tree_post_update_vma(). - * - * The entire update must be protected by exclusive mmap_lock and by - * the root anon_vma's mutex. - */ -static void -anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma) -{ - struct anon_vma_chain *avc; - - list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) - anon_rmap_tree_remove(avc, avc->anon_vma); -} - -static void -anon_rmap_tree_post_update_vma(struct vm_area_struct *vma) -{ - struct anon_vma_chain *avc; - - list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) - anon_rmap_tree_insert(avc, avc->anon_vma); -} - -/* * vma_prepare() - Helper function for handling locking VMAs prior to altering * @vp: The initialized vma_prepare struct */ @@ -358,16 +334,19 @@ static void vma_prepare(struct vma_prepare *vp) if (vp->anon_vma) { anon_vma_lock_write(vp->anon_vma); - anon_rmap_tree_pre_update_vma(vp->vma); + anon_rmap_tree_pre_update_vma(vp->vma, vp->anon_pgoff_unchanged); + /* The adjacent VMA's start is moved, so its page offset changes. */ if (vp->adj_next) - anon_rmap_tree_pre_update_vma(vp->adj_next); + anon_rmap_tree_pre_update_vma(vp->adj_next, false); } if (vp->file) { flush_dcache_mmap_lock(vp->mapping); - mapping_rmap_tree_remove(vp->vma, vp->mapping); + mapping_rmap_tree_pre_update(vp->vma, vp->mapping, + vp->file_pgoff_unchanged); if (vp->adj_next) - mapping_rmap_tree_remove(vp->adj_next, vp->mapping); + mapping_rmap_tree_pre_update(vp->adj_next, vp->mapping, + false); } } @@ -385,8 +364,10 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, { if (vp->file) { if (vp->adj_next) - mapping_rmap_tree_insert(vp->adj_next, vp->mapping); - mapping_rmap_tree_insert(vp->vma, vp->mapping); + mapping_rmap_tree_post_update(vp->adj_next, vp->mapping, + false); + mapping_rmap_tree_post_update(vp->vma, vp->mapping, + vp->file_pgoff_unchanged); flush_dcache_mmap_unlock(vp->mapping); } @@ -405,9 +386,9 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, } if (vp->anon_vma) { - anon_rmap_tree_post_update_vma(vp->vma); + anon_rmap_tree_post_update_vma(vp->vma, vp->anon_pgoff_unchanged); if (vp->adj_next) - anon_rmap_tree_post_update_vma(vp->adj_next); + anon_rmap_tree_post_update_vma(vp->adj_next, false); anon_vma_unlock_write(vp->anon_vma); } @@ -430,7 +411,7 @@ again: vp->remove->vm_end); fput(vp->file); } - if (vp->remove->anon_vma) + if (vma_has_anon_rmap(vp->remove)) unlink_anon_vmas(vp->remove); mm->map_count--; mpol_put(vma_policy(vp->remove)); @@ -478,7 +459,7 @@ static bool can_vma_merge_left(struct vma_merge_struct *vmg) * account the end position of the proposed range. * * In addition, if we can merge with the left VMA, ensure that left and right - * anon_vma's are also compatible. + * anon rmaps are also compatible. */ static bool can_vma_merge_right(struct vma_merge_struct *vmg, bool can_merge_left) @@ -494,13 +475,13 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg, /* * If we can merge with prev (left) and next (right), indicating that - * each VMA's anon_vma is compatible with the proposed anon_vma, this + * each VMA's anon rmap is compatible with the proposed anon rmap, this * does not mean prev and next are compatible with EACH OTHER. * * We therefore check this in addition to mergeability to either side. */ prev = vmg->prev; - return !prev->anon_vma || !next->anon_vma || + return !vma_has_anon_rmap(prev) || !vma_has_anon_rmap(next) || prev->anon_vma == next->anon_vma; } @@ -592,6 +573,8 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, init_vma_prep(&vp, vma); vp.insert = new; + vp.file_pgoff_unchanged = !new_below; + vp.anon_pgoff_unchanged = !new_below; vma_prepare(&vp); /* @@ -599,7 +582,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, * boundary. */ vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL); - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) hugetlb_split(vma, addr); if (new_below) { @@ -644,8 +627,8 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, } /* - * dup_anon_vma() - Helper function to duplicate anon_vma on VMA merge in the - * instance that the destination VMA has no anon_vma but the source does. + * dup_anon_vma() - Helper function to duplicate the anon rmap on VMA merge in + * the instance that the destination VMA has no anon rmap but the source does. * * @dst: The destination VMA * @src: The source VMA @@ -658,19 +641,19 @@ static int dup_anon_vma(struct vm_area_struct *dst, { /* * There are three cases to consider for correctly propagating - * anon_vma's on merge. + * anon rmaps on merge. * - * The first is trivial - neither VMA has anon_vma, we need not do + * The first is trivial - neither VMA has an anon rmap, we need not do * anything. * - * The second where both have anon_vma is also a no-op, as they must + * The second where both have an anon rmap is also a no-op, as they must * then be the same, so there is simply nothing to copy. * - * Here we cover the third - if the destination VMA has no anon_vma, + * Here we cover the third - if the destination VMA has no anon rmap, * that is it is unfaulted, we need to ensure that the newly merged - * range is referenced by the anon_vma's of the source. + * range is referenced by the anon rmap of the source. */ - if (src->anon_vma && !dst->anon_vma) { + if (vma_has_anon_rmap(src) && !vma_has_anon_rmap(dst)) { int ret; vma_assert_write_locked(dst); @@ -720,7 +703,7 @@ void validate_mm(struct mm_struct *mm) } #ifdef CONFIG_DEBUG_VM_RB - if (anon_vma) { + if (vma_has_anon_rmap(vma)) { anon_vma_lock_read(anon_vma); list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) anon_rmap_tree_verify(avc); @@ -924,13 +907,14 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->state = VMA_MERGE_NOMERGE; + if (!vma_flags_can_merge(&vmg->vma_flags)) + return NULL; /* - * If a special mapping or if the range being modified is neither at the - * furthermost left or right side of the VMA, then we have no chance of - * merging and should abort. + * If the range being modified is neither at the furthermost left or + * right side of the VMA, then we have no chance of merging and should + * abort. */ - if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) || - (!left_side && !right_side)) + if (!left_side && !right_side) return NULL; if (left_side) @@ -1015,11 +999,11 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->anon_pgoff = vma_start_anon_pgoff(prev); /* - * We already ensured anon_vma compatibility above, so now it's - * simply a case of, if prev has no anon_vma object, which of - * next or middle contains the anon_vma we must duplicate. + * We already ensured anon rmap compatibility above, so now it's + * simply a case of, if prev has no anon rmap, which of next or + * middle contains the anon rmap we must duplicate. */ - err = dup_anon_vma(prev, next->anon_vma ? next : middle, + err = dup_anon_vma(prev, vma_has_anon_rmap(next) ? next : middle, &anon_dup); } else if (merge_left) { /* @@ -1083,7 +1067,7 @@ abort: unlink_anon_vmas(anon_dup); /* - * This means we have failed to clone anon_vma's correctly, but no + * This means we have failed to clone the anon rmap correctly, but no * actual changes to VMAs have occurred, so no harm no foul - if the * user doesn't want this reported and instead just wants to give up on * the merge, allow it. @@ -1152,9 +1136,11 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg) vmg->state = VMA_MERGE_NOMERGE; - /* Special VMAs are unmergeable, also if no prev/next. */ - if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) || - (!prev && !next)) + if (!vma_flags_can_merge(&vmg->vma_flags)) + return NULL; + + /* VMAs with no prev/next are unmergeable. */ + if (!prev && !next) return NULL; can_merge_left = can_vma_merge_left(vmg); @@ -1225,6 +1211,11 @@ static struct vm_area_struct *vma_merge_copied_range(struct vma_merge_struct *vm vmg->copied_from = vmg->middle; vmg->middle = NULL; + + /* If not registered for remap, uffd context will be cleared. */ + if (vma_has_uffd_without_event_remap(vmg->copied_from)) + return NULL; + return vma_merge_new_range(vmg); } @@ -1278,7 +1269,7 @@ int vma_expand(struct vma_merge_struct *vmg) /* * If we are removing the next VMA or copying from a VMA - * (e.g. mremap()'ing), we must propagate anon_vma state. + * (e.g. mremap()'ing), we must propagate anon rmap state. * * Note that, by convention, callers ignore OOM for this case, so * we don't need to account for vmg->give_up_on_mm here. @@ -1342,6 +1333,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, vma_start_write(vma); init_vma_prep(&vp, vma); + vp.file_pgoff_unchanged = true; + vp.anon_pgoff_unchanged = true; vma_prepare(&vp); vma_adjust_trans_huge(vma, vma->vm_start, end, NULL); @@ -1957,7 +1950,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, * If a vma has not yet been faulted, update its anonymous pgoff to * match the new location to increase its chance of merging. */ - if (!vma->anon_vma) { + if (!vma_has_anon_rmap(vma)) { anon_pgoff = addr >> PAGE_SHIFT; if (vma_is_anonymous(vma)) { @@ -2055,16 +2048,16 @@ out: /* * Rough compatibility check to quickly see if it's even worth looking - * at sharing an anon_vma. + * at sharing an anon rmap. * * They need to have the same vm_file, and the flags can only differ * in things that mprotect may change. * - * NOTE! The fact that we share an anon_vma doesn't _have_ to mean that + * NOTE! The fact that we share an anon rmap doesn't _have_ to mean that * we can merge the two vma's. For example, we refuse to merge a vma if * there is a vm_ops->close() function, because that indicates that the * driver is doing some kind of reference counting. But that doesn't - * really matter for the anon_vma sharing case. + * really matter for the anon rmap sharing case. */ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *b) { @@ -2097,13 +2090,13 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * } /* - * Do some basic sanity checking to see if we can re-use the anon_vma + * Do some basic sanity checking to see if we can re-use the anon rmap * from 'old'. The 'a'/'b' vma's are in VM order - one of them will be * the same as 'old', the other will be the new one that is trying - * to share the anon_vma. + * to share the anon rmap. * * NOTE! This runs with mmap_lock held for reading, so it is possible that - * the anon_vma of 'old' is concurrently in the process of being set up + * the anon rmap of 'old' is concurrently in the process of being set up * by another page fault trying to merge _that_. But that's ok: if it * is being set up, that automatically means that it will be a singleton * acceptable for merging, so we can do all of this optimistically. But @@ -2117,8 +2110,8 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * * accessing an uninitialised anon_vma's fields may result in a UAF. * * IOW: that the "list_is_singular()" test on the anon_vma_chain only - * matters for the 'stable anon_vma' case (ie the thing we want to avoid - * is to return an anon_vma that is "complex" due to having gone through + * matters for the 'stable anon rmap' case (ie the thing we want to avoid + * is to return an anon rmap that is "complex" due to having gone through * a fork). * * We also make sure that the two vma's are compatible (adjacent, @@ -2141,10 +2134,10 @@ static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old, /* * find_mergeable_anon_vma is used by anon_vma_prepare, to check - * neighbouring vmas for a suitable anon_vma, before it goes off - * to allocate a new anon_vma. It checks because a repetitive + * neighbouring vmas for a suitable anon rmap, before it goes off + * to allocate a new anon rmap. It checks because a repetitive * sequence of mprotects and faults may otherwise lead to distinct - * anon_vmas being allocated, preventing vma merge in subsequent + * anon rmaps being allocated, preventing vma merge in subsequent * mprotect. */ struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma) @@ -2170,13 +2163,13 @@ struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma) /* * We might reach here with anon_vma == NULL if we can't find - * any reusable anon_vma. + * any reusable anon rmap. * There's no absolute need to look only at touching neighbours: - * we could search further afield for "compatible" anon_vmas. + * we could search further afield for "compatible" anon rmaps. * But it would probably just be a waste of time searching, - * or lead to too many vmas hanging off the same anon_vma. + * or lead to too many vmas hanging off the same anon rmap. * We're trying to allow mprotect remerging later on, - * not trying to minimize memory used for anon_vmas. + * not trying to minimize memory used for anon rmaps. */ return anon_vma; } @@ -2248,7 +2241,7 @@ bool vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot) * Do we need to track softdirty? hugetlb does not support softdirty * tracking yet. */ - if (vma_soft_dirty_enabled(vma) && !is_vm_hugetlb_page(vma)) + if (vma_soft_dirty_enabled(vma) && !vma_is_hugetlb(vma)) return true; /* Do we need write faults for uffd-wp tracking? */ @@ -2272,7 +2265,7 @@ static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma) /* * We can safely modify head.next after taking the * anon_vma->root->rwsem. If some other vma in this mm shares - * the same anon_vma we won't take it again. + * the same anon rmap we won't take it again. * * No need of atomic instructions here, head.next * can't change from under us thanks to the @@ -2314,14 +2307,14 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * mmap_lock in write mode is required in order to block all operations * that could modify pagetables and free pages without need of * altering the vma layout. It's also needed in write mode to avoid new - * anon_vmas to be associated with existing vmas. + * anon rmaps being associated with existing vmas. * * A single task can't take more than one mm_take_all_locks() in a row * or it would deadlock. * * The LSB in anon_vma->rb_root.rb_node and the AS_MM_ALL_LOCKS bitflag in * mapping->flags avoid to take the same lock twice, if more than one - * vma in this mm is backed by the same anon_vma or address_space. + * vma in this mm is backed by the same anon rmap or address_space. * * We take locks in following order, accordingly to comment at beginning * of mm/rmap.c: @@ -2367,7 +2360,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (signal_pending(current)) goto out_unlock; if (vma->vm_file && vma->vm_file->f_mapping && - is_vm_hugetlb_page(vma)) + vma_is_hugetlb(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); } @@ -2376,7 +2369,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (signal_pending(current)) goto out_unlock; if (vma->vm_file && vma->vm_file->f_mapping && - !is_vm_hugetlb_page(vma)) + !vma_is_hugetlb(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); } @@ -2384,7 +2377,7 @@ int mm_take_all_locks(struct mm_struct *mm) for_each_vma(vmi, vma) { if (signal_pending(current)) goto out_unlock; - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); } @@ -2446,7 +2439,7 @@ void mm_drop_all_locks(struct mm_struct *mm) BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); for_each_vma(vmi, vma) { - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_unlock_anon_vma(avc->anon_vma); if (vma->vm_file && vma->vm_file->f_mapping) @@ -2462,7 +2455,7 @@ void mm_drop_all_locks(struct mm_struct *mm) */ static bool accountable_mapping(struct mmap_state *map) { - const struct file *file = map->file; + const struct file *file = map->vm_file; /* * hugetlb has its own accounting separate from the core VM @@ -2511,7 +2504,7 @@ static void vms_abort_munmap_vmas(struct vma_munmap_struct *vms, static void update_ksm_flags(struct mmap_state *map) { - map->vma_flags = ksm_vma_flags(map->mm, map->file, map->vma_flags); + map->vma_flags = ksm_vma_flags(map->mm, map->vm_file, map->vma_flags); } static void set_desc_from_map(struct vm_area_desc *desc, @@ -2521,7 +2514,7 @@ static void set_desc_from_map(struct vm_area_desc *desc, desc->end = map->end; desc->pgoff = map->pgoff; - desc->vm_file = map->file; + desc->vm_file = map->vm_file; desc->vma_flags = map->vma_flags; desc->page_prot = map->page_prot; } @@ -2577,14 +2570,17 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, /* Private writable mapping: check memory availability. */ if (accountable_mapping(map)) { - map->charged = map->pglen; - map->charged -= vms->nr_accounted; - if (map->charged) { - error = security_vm_enough_memory_mm(map->mm, map->charged); + const unsigned long uncharged = map->pglen - vms->nr_accounted; + + /* Charge only what's not already charged by mappings we replace. */ + if (uncharged) { + error = security_vm_enough_memory_mm(map->mm, uncharged); if (error) return error; } + /* Our new mapping owns the entire charge now. */ + map->charged = map->pglen; vms->nr_accounted = 0; vma_flags_set(&map->vma_flags, VMA_ACCOUNT_BIT); } @@ -2601,6 +2597,10 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, return 0; } +static bool map_same_file(struct mmap_state *map) +{ + return map->vm_file == map->file; +} static int __mmap_new_file_vma(struct mmap_state *map, struct vm_area_struct *vma) @@ -2608,37 +2608,50 @@ static int __mmap_new_file_vma(struct mmap_state *map, struct vma_iterator *vmi = map->vmi; int error; - vma->vm_file = map->file; - if (!map->file_doesnt_need_get) - get_file(map->file); + vma->vm_file = map->vm_file; + if (map_same_file(map)) + get_file(map->vm_file); - if (!map->file->f_op->mmap) + if (!map->vm_file->f_op->mmap) return 0; + /* + * Driver-specified flags may make the lock flags invalid, so clear + * VMA_LOCKED_MASK and reinstate it afterwards if appropriate. + */ + vma_clear_flags_mask(vma, VMA_LOCKED_MASK); error = mmap_file(vma->vm_file, vma); + map->vm_file = vma->vm_file; + if (error) { UNMAP_STATE(unmap, vmi, vma, vma->vm_start, vma->vm_end, map->prev, map->next); - fput(vma->vm_file); - vma->vm_file = NULL; + if (vma->vm_ops->close) { + zap_vma(vma); + /* Safe, because VMA zapped. */ + vma_close(vma); + } + if (map_same_file(map)) + fput(map->vm_file); + + vma->vm_file = NULL; vma_iter_set(vmi, vma->vm_end); /* Undo any partial mapping done by a device driver. */ unmap_region(&unmap); return error; } - /* Drivers cannot alter the address of the VMA. */ - WARN_ON_ONCE(map->addr != vma->vm_start); /* - * Drivers should not permit writability when previously it was - * disallowed. + * If the VMA is still eligible for mlock(), reinstate any original + * VMA_LOCKED_BIT and/or VMA_LOCKONFAULT_BIT flags. */ - VM_WARN_ON_ONCE(!vma_flags_same_pair(&map->vma_flags, &vma->flags) && - !vma_flags_test(&map->vma_flags, VMA_MAYWRITE_BIT) && - vma_test(vma, VMA_MAYWRITE_BIT)); + if (vma_supports_mlock(vma)) { + const vma_flags_t mask = vma_flags_and_mask(&map->vma_flags, VMA_LOCKED_MASK); + + vma_set_flags_mask(vma, mask); + } - map->file = vma->vm_file; map->vma_flags = vma->flags; return 0; @@ -2646,7 +2659,7 @@ static int __mmap_new_file_vma(struct mmap_state *map, static void map_set_anon(struct mmap_state *map) { - map->file = NULL; + map->vm_file = NULL; map->vm_ops = NULL; map->pgoff = map->addr >> PAGE_SHIFT; } @@ -2658,7 +2671,7 @@ static bool map_is_private(const struct mmap_state *map) static bool map_is_anon(const struct mmap_state *map) { - return map_is_private(map) && !map->file; + return map_is_private(map) && !map->vm_file; } /* @@ -2703,7 +2716,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, } /* Invoke callbacks. */ - if (map->file) + if (map->vm_file) error = __mmap_new_file_vma(map, vma); else if (!is_anon) error = shmem_zero_setup(vma); @@ -2716,11 +2729,6 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, vma->flags = map->vma_flags; } -#ifdef CONFIG_SPARC64 - /* TODO: Fix SPARC ADI! */ - WARN_ON_ONCE(!arch_validate_flags(map->vm_flags)); -#endif - /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); vma_iter_store_new(vmi, vma); @@ -2783,6 +2791,103 @@ static void __mmap_complete(struct mmap_state *map, struct vm_area_struct *vma) vma_set_page_prot(vma); } +/* Check to ensure that the VMA flags of a newly mapped VMA are sane. */ +static int mmap_validate_vma_flags(const vma_flags_t *flags) +{ +#ifdef CONFIG_SPARC64 + const vm_flags_t legacy_flags = vma_flags_to_legacy(*flags); + + /* TODO: Fix SPARC ADI! */ + if (WARN_ON_ONCE(!arch_validate_flags(legacy_flags))) + return -EINVAL; +#endif + + if (vma_flags_is_mm_managed(flags)) { + /* mm-managed mappings may not set VMA_IO_BIT. */ + if (WARN_ON_ONCE(vma_flags_test(flags, VMA_IO_BIT))) + return -EINVAL; + } + + return 0; +} + +/* Check to ensure a driver hasn't done something crazy. */ +static int mmap_validate(unsigned long orig_start, unsigned long orig_end, + unsigned long curr_start, unsigned long curr_end, + const vma_flags_t *orig_flags, + const vma_flags_t *curr_flags) +{ + bool was_maywrite, is_maywrite; + + /* Drivers cannot alter the range of the VMA. */ + if (WARN_ON_ONCE(orig_start != curr_start || orig_end != curr_end)) + return -EINVAL; + + was_maywrite = vma_flags_test(orig_flags, VMA_MAYWRITE_BIT); + is_maywrite = vma_flags_test(curr_flags, VMA_MAYWRITE_BIT); + + /* A driver may not make a previously unwritable mapping writable. */ + if (WARN_ON_ONCE(!was_maywrite && is_maywrite)) + return -EINVAL; + + /* mm-managed mappings may not clear VMA_MAYWRITE_BIT. */ + if (vma_flags_is_mm_managed(curr_flags) && + WARN_ON_ONCE(was_maywrite && !is_maywrite)) + return -EINVAL; + + return mmap_validate_vma_flags(curr_flags); +} + +/** + * mmap_prepare_validate() - Ensure the driver hasn't violated invariants in its + * f_op->mmap_prepare hook. + * @orig_desc: The VMA descriptor prior to the mmap_prepare hook being called. + * @desc: The VMA descriptor after the mmap_prepare hook has been called. + * + * Returns: 0 on success, otherwise an error. + */ +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc) +{ + /* + * It is not valid to execute mmap actions for VMAs which can be merged, + * as any such merge would leave portions of the mapping incorrectly + * unmapped. + * + * This is checked after mmap_action_prepare(), as it may update the VMA + * flags and therefore whether the VMA can be merged. + */ + if (vma_flags_can_merge(&desc->vma_flags) && + WARN_ON_ONCE(desc->action.type != MMAP_NOTHING)) + return -EINVAL; + + return mmap_validate(orig_desc->start, orig_desc->end, + desc->start, desc->end, + &orig_desc->vma_flags, &desc->vma_flags); +} + +/** + * mmap_hook_validate() - Ensure the driver hasn't violated invariants in + * its f_op->mmap hook. + * @orig_start: The start of the mapping prior to the mmap hook. + * @orig_end: The end of the mapping prior to the mmap hook. + * @orig_flags: The VMA flags set for the VMA prior to the mmap hook. + * @vma: The VMA after the hook has been applied. + * + * Returns: 0 on success, otherwise an error. + */ +int mmap_hook_validate(unsigned long orig_start, unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma) +{ + const unsigned long start = vma->vm_start; + const unsigned long end = vma->vm_end; + const vma_flags_t *flags = &vma->flags; + + return mmap_validate(orig_start, orig_end, start, end, orig_flags, + flags); +} + static int call_action_prepare(struct mmap_state *map, struct vm_area_desc *desc) { @@ -2809,13 +2914,18 @@ static int call_action_prepare(struct mmap_state *map, static int call_mmap_prepare(struct mmap_state *map, struct vm_area_desc *desc) { + const struct vm_area_desc orig_desc = *desc; int err; /* Invoke the hook. */ - err = vfs_mmap_prepare(map->file, desc); + err = vfs_mmap_prepare(map->vm_file, desc); if (err) return err; + /* Update first so file refcount tracked correctly. */ + if (desc->vm_file != map->vm_file) + map->vm_file = desc->vm_file; + /* It's invalid for mmap_prepare hooks to clear vm_ops. */ if (!desc->vm_ops) return -EINVAL; @@ -2824,12 +2934,13 @@ static int call_mmap_prepare(struct mmap_state *map, if (err) return err; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); + if (err) + return err; + /* Update fields permitted to be changed. */ map->pgoff = desc->pgoff; - if (desc->vm_file != map->file) { - map->file_doesnt_need_get = true; - map->file = desc->vm_file; - } map->vma_flags = desc->vma_flags; map->page_prot = desc->page_prot; /* User-defined fields. */ @@ -2841,7 +2952,7 @@ static int call_mmap_prepare(struct mmap_state *map, * anonymous mappings. Rather than allowing these mappings to be odd * outliers, simply make them truly anonymous. */ - if (map_is_private(map) && file_is_dev_zero(map->file)) + if (map_is_private(map) && file_is_dev_zero(map->vm_file)) map_set_anon(map); return 0; @@ -2860,7 +2971,7 @@ static void set_vma_user_defined_fields(struct vm_area_struct *vma, */ static bool can_set_ksm_flags_early(struct mmap_state *map) { - struct file *file = map->file; + struct file *file = map->vm_file; /* Anonymous mappings have no driver which can change them. */ if (!file) @@ -2883,6 +2994,20 @@ static bool can_set_ksm_flags_early(struct mmap_state *map) return false; } +static void map_put_vm_file(struct mmap_state *map) +{ + /* + * An error occurred or the VMA was merged. + * + * If the file was changed by the driver (which is required to increment + * the replacement file's reference count), drop its reference count. + * + * On error, the caller always drops the original file regardless. + */ + if (map->vm_file && !map_same_file(map)) + fput(map->vm_file); +} + static unsigned long __mmap_region(struct file *file, unsigned long addr, unsigned long len, vma_flags_t vma_flags, unsigned long pgoff, struct list_head *uf) @@ -2937,7 +3062,10 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, __mmap_complete(&map, vma); - if (have_mmap_prepare && allocated_new) { + if (!allocated_new) { + /* Merged, so need to drop refcount. */ + map_put_vm_file(&map); + } else if (have_mmap_prepare) { error = mmap_action_complete(vma, &desc.action, /*is_compat=*/false); if (error) @@ -2951,13 +3079,7 @@ unacct_error: if (map.charged) vm_unacct_memory(map.charged); abort_munmap: - /* - * This indicates that .mmap_prepare has set a new file, differing from - * desc->vm_file. But since we're aborting the operation, only the - * original file will be cleaned up. Ensure we clean up both. - */ - if (map.file_doesnt_need_get) - fput(map.file); + map_put_vm_file(&map); vms_abort_munmap_vmas(&map.vms, &map.mas_detach); return error; } @@ -3299,7 +3421,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (next && vma_is_accessible(next)) { if (!vma_test(next, VMA_GROWSUP_BIT)) return -ENOMEM; - /* Check that both stack segments have the same anon_vma? */ + /* Check that both stack segments have the same anon rmap? */ } if (next) @@ -3309,7 +3431,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (vma_iter_prealloc(&vmi, vma)) return -ENOMEM; - /* We must make sure the anon_vma is allocated. */ + /* We must make sure the anon rmap is allocated. */ if (unlikely(anon_vma_prepare(vma))) { vma_iter_free(&vmi); return -ENOMEM; @@ -3333,11 +3455,11 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) if (vma_test(vma, VMA_LOCKED_BIT)) mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); - anon_rmap_tree_pre_update_vma(vma); + anon_rmap_tree_pre_update_vma(vma, true); vma->vm_end = address; /* Overwrite old entry in mtree. */ vma_iter_store_overwrite(&vmi, vma); - anon_rmap_tree_post_update_vma(vma); + anon_rmap_tree_post_update_vma(vma, true); perf_event_mmap(vma); } @@ -3372,7 +3494,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) /* Enforce stack_guard_gap */ prev = vma_prev(&vmi); - /* Check that both stack segments have the same anon_vma? */ + /* Check that both stack segments have the same anon rmap? */ if (prev) { if (!vma_test(prev, VMA_GROWSDOWN_BIT) && vma_is_accessible(prev) && @@ -3387,7 +3509,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) if (vma_iter_prealloc(&vmi, vma)) return -ENOMEM; - /* We must make sure the anon_vma is allocated. */ + /* We must make sure the anon rmap is allocated. */ if (unlikely(anon_vma_prepare(vma))) { vma_iter_free(&vmi); return -ENOMEM; @@ -3410,12 +3532,12 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) if (vma_test(vma, VMA_LOCKED_BIT)) mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); - anon_rmap_tree_pre_update_vma(vma); + anon_rmap_tree_pre_update_vma(vma, false); vma->vm_start = address; vma_sub_pgoff(vma, grow); /* Overwrite old entry in mtree. */ vma_iter_store_overwrite(&vmi, vma); - anon_rmap_tree_post_update_vma(vma); + anon_rmap_tree_post_update_vma(vma, false); perf_event_mmap(vma); } @@ -3452,17 +3574,22 @@ int __vm_munmap(unsigned long start, size_t len, bool unlock) int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) { unsigned long charged = vma_pages(vma); + int err; if (find_vma_intersection(mm, vma->vm_start, vma->vm_end)) return -ENOMEM; + err = mmap_validate_vma_flags(&vma->flags); + if (err) + return err; + if (vma_test(vma, VMA_ACCOUNT_BIT) && security_vm_enough_memory_mm(mm, charged)) return -ENOMEM; /* * The vm_pgoff of a purely anonymous vma should be irrelevant - * until its first write fault, when page's anon_vma and index + * until its first write fault, when page's anon rmap and index * are set. But now set the vm_pgoff it will almost certainly * end up with (unless mremap moves it elsewhere before that * first wfault), so /proc/pid/maps tells a consistent story. @@ -3473,7 +3600,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) * Similarly in do_mmap and in do_brk_flags. */ if (vma_is_anonymous(vma)) { - WARN_ON_ONCE(vma->anon_vma); + WARN_ON_ONCE(vma_has_anon_rmap(vma)); vma_set_pgoff(vma, vma->vm_start >> PAGE_SHIFT); } vma_set_anon_pgoff(vma, vma->vm_start >> PAGE_SHIFT); @@ -28,6 +28,8 @@ struct vma_prepare { struct vm_area_struct *remove2; bool skip_vma_uprobe :1; + bool file_pgoff_unchanged :1; + bool anon_pgoff_unchanged :1; }; struct unlink_vma_file_batch { @@ -255,6 +257,33 @@ static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg) return vmg_start_pgoff(vmg) + vmg_pages(vmg); } +/** + * vma_has_anon_rmap() - does @vma possess an anonymous reverse mapping? + * @vma: The VMA to be checked. + * + * If the VMA is attached, a VMA or mmap lock must be held. + * + * This state is only possible for CoW mappings, see the comment for + * vma_flags_is_cow_mapping() for details. + * + * Importantly, a VMA which possesses an anonymous rmap may map anonymous + * folios. + * + * This function will not result in a false positive. + * + * However, if only a read lock is held, it may give a false negative, in which + * case it should be re-checked with mm->page_table_lock held. + * + * Returns: true if @vma has an anonymous reverse mapping, otherwise false. + */ +static inline bool vma_has_anon_rmap(const struct vm_area_struct *vma) +{ + if (vma_is_attached(vma)) + vma_assert_stabilised(vma); + /* KCSAN gets confused about the optimistic check. Silence it. */ + return data_race(vma->anon_vma); +} + static inline void assert_sane_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) { /* nommu doesn't set a virtual pgoff for anon VMAs. */ @@ -268,7 +297,7 @@ static inline void assert_sane_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) if (!vma_is_anonymous(vma)) return; /* If faulted in, could have been remapped. */ - if (vma->anon_vma) + if (vma_has_anon_rmap(vma)) return; /* OK this is really an anon VMA - expect virtual page offset. */ VM_WARN_ON_ONCE(pgoff != vma->vm_start >> PAGE_SHIFT); @@ -394,8 +423,10 @@ static inline void compat_set_vma_from_desc(struct vm_area_struct *vma, /* Mutable fields. Populated with initial state. */ vma_set_pgoff(vma, desc->pgoff); - if (desc->vm_file != vma->vm_file) - vma_set_file(vma, desc->vm_file); + if (desc->vm_file != vma->vm_file) { + fput(vma->vm_file); + vma->vm_file = desc->vm_file; + } vma->flags = desc->vma_flags; vma->vm_page_prot = desc->page_prot; @@ -785,14 +816,19 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm); struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig); void vm_area_free(struct vm_area_struct *vma); -/* vma_exec.c */ #ifdef CONFIG_MMU +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc); + +int mmap_hook_validate(unsigned long orig_start, unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma); + +/* vma_exec.c */ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap, unsigned long *top_mem_p); int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift); -#endif -#ifdef CONFIG_MMU /* * Denies creating a writable executable mapping or gaining executable permissions. * @@ -841,6 +877,20 @@ static inline bool map_deny_write_exec(const vma_flags_t *old, return false; } +#else +static inline int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc) +{ + return 0; +} + +static inline int mmap_hook_validate(unsigned long orig_start, + unsigned long orig_end, + const vma_flags_t *orig_flags, + const struct vm_area_struct *vma) +{ + return 0; +} #endif struct vm_area_struct *__install_special_mapping(struct mm_struct *mm, diff --git a/mm/vma_internal.h b/mm/vma_internal.h index 4d300e7bbaf4..4f73f0a4db79 100644 --- a/mm/vma_internal.h +++ b/mm/vma_internal.h @@ -18,7 +18,6 @@ #include <linux/fs.h> #include <linux/huge_mm.h> #include <linux/hugetlb.h> -#include <linux/hugetlb_inline.h> #include <linux/kernel.h> #include <linux/ksm.h> #include <linux/khugepaged.h> diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a009cc73f1b7..f7aa4686fe5b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -684,7 +684,8 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); + if (WARN_ON_ONCE(page_shift < PAGE_SHIFT)) + return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift == PAGE_SHIFT) @@ -817,9 +818,10 @@ EXPORT_SYMBOL_GPL(is_vmalloc_or_module_addr); /* * Walk a vmap address to the struct page it maps. Huge vmap mappings will * return the tail page that corresponds to the base page address, which - * matches small vmap mappings. + * matches small vmap mappings. Unlike vmalloc_to_page(), this also accepts + * addresses outside the vmalloc and module ranges, such as KMSAN metadata. */ -struct page *vmalloc_to_page(const void *vmalloc_addr) +__always_inline struct page *__vmalloc_to_page(const void *vmalloc_addr) { unsigned long addr = (unsigned long) vmalloc_addr; struct page *page = NULL; @@ -829,12 +831,6 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) pmd_t *pmd; pte_t *ptep, pte; - /* - * XXX we might need to change this if we add VIRTUAL_BUG_ON for - * architectures that do not vmalloc module space - */ - VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr)); - if (pgd_none(*pgd)) return NULL; if (WARN_ON_ONCE(pgd_leaf(*pgd))) @@ -873,6 +869,16 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) return page; } + +struct page *vmalloc_to_page(const void *vmalloc_addr) +{ + /* + * XXX we might need to change this if we add VIRTUAL_BUG_ON for + * architectures that do not vmalloc module space + */ + VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr)); + return __vmalloc_to_page(vmalloc_addr); +} EXPORT_SYMBOL(vmalloc_to_page); /* @@ -2503,67 +2509,94 @@ static void free_unmap_vmap_area(struct vmap_area *va) free_vmap_area_noflush(va); } -struct vmap_area *find_vmap_area(unsigned long addr) +static inline int next_vmap_node_id(int i) +{ + return (i + nr_vmap_nodes - 1) % nr_vmap_nodes; +} + +enum vmap_lock_mode { + VMAP_LOCK, + VMAP_TRYLOCK, +}; + +/* + * Search for a vmap_area at @addr across all vmap nodes. An + * addr_to_node_id(addr) converts an address to a node index where + * a VA is located. If VA spans several zones and passed addr is not + * the same as va->va_start, what is not common, we may need to scan + * extra nodes. See an example: + * + * <----va----> + * -|-----|-----|-----|-----|- + * 1 2 0 1 + * + * VA resides in node 1 whereas it spans 1, 2 an 0. If passed addr + * is within 2 or 0 nodes we should do extra work. + * + * Returns the VA with @locked_vn->busy.lock held; the caller must + * release it. If @mode is VMAP_TRYLOCK, nodes that cannot be locked + * are skipped. + */ +static struct vmap_area * +find_vmap_area_lock(unsigned long addr, struct vmap_node **locked_vn, + enum vmap_lock_mode mode) { struct vmap_node *vn; struct vmap_area *va; int i, j; + *locked_vn = NULL; + if (unlikely(!vmap_initialized)) return NULL; - /* - * An addr_to_node_id(addr) converts an address to a node index - * where a VA is located. If VA spans several zones and passed - * addr is not the same as va->va_start, what is not common, we - * may need to scan extra nodes. See an example: - * - * <----va----> - * -|-----|-----|-----|-----|- - * 1 2 0 1 - * - * VA resides in node 1 whereas it spans 1, 2 an 0. If passed - * addr is within 2 or 0 nodes we should do extra work. - */ i = j = addr_to_node_id(addr); do { vn = &vmap_nodes[i]; - spin_lock(&vn->busy.lock); - va = __find_vmap_area(addr, &vn->busy.root); - spin_unlock(&vn->busy.lock); + if (mode == VMAP_LOCK) { + spin_lock(&vn->busy.lock); + } else { + if (!spin_trylock(&vn->busy.lock)) + continue; + } - if (va) + va = __find_vmap_area(addr, &vn->busy.root); + if (va) { + *locked_vn = vn; return va; - } while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j); + } + + spin_unlock(&vn->busy.lock); + } while ((i = next_vmap_node_id(i)) != j); return NULL; } -static struct vmap_area *find_unlink_vmap_area(unsigned long addr) +struct vmap_area *find_vmap_area(unsigned long addr) { struct vmap_node *vn; struct vmap_area *va; - int i, j; - /* - * Check the comment in the find_vmap_area() about the loop. - */ - i = j = addr_to_node_id(addr); - do { - vn = &vmap_nodes[i]; - - spin_lock(&vn->busy.lock); - va = __find_vmap_area(addr, &vn->busy.root); - if (va) - unlink_va(va, &vn->busy.root); + va = find_vmap_area_lock(addr, &vn, VMAP_LOCK); + if (va) spin_unlock(&vn->busy.lock); - if (va) - return va; - } while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j); + return va; +} - return NULL; +static struct vmap_area *find_unlink_vmap_area(unsigned long addr) +{ + struct vmap_node *vn; + struct vmap_area *va; + + va = find_vmap_area_lock(addr, &vn, VMAP_LOCK); + if (va) { + unlink_va(va, &vn->busy.root); + spin_unlock(&vn->busy.lock); + } + + return va; } /*** Per cpu kva allocator ***/ @@ -3685,7 +3718,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, /* * Initially, attempt to have the page allocator give us large order * pages. Do not attempt allocating smaller than order chunks since - * __vmap_pages_range() expects physically contigous pages of exactly + * __vmap_pages_range() expects physically contiguous pages of exactly * order long chunks. */ while (large_order > order && nr_remaining) { @@ -5122,9 +5155,14 @@ retry: ret = va_clip(&free_vmap_area_root, &free_vmap_area_list, va, start, size); - if (WARN_ON_ONCE(unlikely(ret))) - /* It is a BUG(), but trigger recovery instead. */ + if (unlikely(ret)) { + /* + * -ENOMEM from the GFP_NOWAIT fallback is expected. + * Anything else is a BUG(), but trigger recovery instead. + */ + WARN_ON_ONCE(ret != -ENOMEM); goto recovery; + } /* Allocated area. */ va = vas[area]; @@ -5263,15 +5301,12 @@ bool vmalloc_dump_obj(void *object) unsigned long addr; unsigned long nr_pages; - addr = PAGE_ALIGN((unsigned long) object); - vn = addr_to_node(addr); + addr = PAGE_ALIGN_DOWN((unsigned long) object); - if (!spin_trylock(&vn->busy.lock)) - return false; - - va = __find_vmap_area(addr, &vn->busy.root); + va = find_vmap_area_lock(addr, &vn, VMAP_TRYLOCK); if (!va || !va->vm) { - spin_unlock(&vn->busy.lock); + if (va) + spin_unlock(&vn->busy.lock); return false; } diff --git a/mm/vmalloc.h b/mm/vmalloc.h index 211869f36509..d8372a4f65c4 100644 --- a/mm/vmalloc.h +++ b/mm/vmalloc.h @@ -8,6 +8,7 @@ #include <linux/vmalloc.h> #ifdef CONFIG_MMU +struct page *__vmalloc_to_page(const void *vmalloc_addr); void __init vmalloc_init(void); int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, diff --git a/mm/vmscan.c b/mm/vmscan.c index 37467004336b..9959ca158ce8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3515,13 +3515,13 @@ static int should_skip_vma(unsigned long start, unsigned long end, struct mm_wal if (!vma_is_accessible(vma)) return true; - if (is_vm_hugetlb_page(vma)) + if (vma_is_hugetlb(vma)) return true; if (!vma_has_recency(vma)) return true; - if (vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) + if (vma_test(vma, VMA_LOCKED_BIT) || !vma_is_mm_backed(vma)) return true; if (vma == get_gate_vma(vma->vm_mm)) @@ -4461,8 +4461,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) if (spin_is_contended(pvmw->ptl)) return true; - /* exclude special VMAs containing anon pages from COW */ - if (vma->vm_flags & VM_SPECIAL) + /* exclude non-mm-backed VMAs containing anon pages from COW */ + if (!vma_is_mm_backed(vma)) return true; /* avoid taking the LRU lock under the PTL when possible */ @@ -5081,6 +5081,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &list, &isolated, &type, &type_scanned); nr_isolated = isolated; + if (nr_isolated) + __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, + nr_isolated); /* Scanning may have emptied the oldest gen, flush it */ if (scanned) @@ -5149,6 +5152,8 @@ retry: goto retry; } + mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, -nr_isolated); + if (nr_isolated > total_reclaimed) mod_lruvec_state(lruvec, PGROTATE_ANON + type, nr_isolated - total_reclaimed); @@ -5449,7 +5454,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * sc->nr_reclaimed = 0; if (mem_cgroup_disabled()) - shrink_one(&pgdat->__lruvec, sc); + shrink_one(mem_cgroup_lruvec(NULL, pgdat), sc); else shrink_many(pgdat, sc); @@ -7993,7 +7998,8 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, sc->gfp_mask); cond_resched(); - psi_memstall_enter(&pflags); + if (!sc->proactive) + psi_memstall_enter(&pflags); delayacct_freepages_start(); fs_reclaim_acquire(sc->gfp_mask); /* @@ -8010,7 +8016,8 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(sc->gfp_mask); delayacct_freepages_end(); - psi_memstall_leave(&pflags); + if (!sc->proactive) + psi_memstall_leave(&pflags); trace_mm_vmscan_node_reclaim_end(sc->nr_reclaimed, NULL); @@ -8178,7 +8185,7 @@ int user_proactive_reclaim(char *buf, } else { struct scan_control sc = { .gfp_mask = current_gfp_context(gfp_mask), - .reclaim_idx = gfp_zone(gfp_mask), + .reclaim_idx = MAX_NR_ZONES - 1, .proactive_swappiness = swappiness == -1 ? NULL : &swappiness, .priority = DEF_PRIORITY, .may_writepage = 1, diff --git a/mm/workingset.c b/mm/workingset.c index 8412f4840ae3..1504f91cdca5 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -470,7 +470,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, * configurations instead. */ eviction_memcg = mem_cgroup_from_private_id(memcgid); - if (!mem_cgroup_tryget(eviction_memcg)) + if (eviction_memcg && !mem_cgroup_tryget(eviction_memcg)) eviction_memcg = NULL; rcu_read_unlock(); diff --git a/samples/damon/mtier.c b/samples/damon/mtier.c index 27dc88bdf7a0..a2e311082cd4 100644 --- a/samples/damon/mtier.c +++ b/samples/damon/mtier.c @@ -163,7 +163,7 @@ static struct damon_ctx *damon_sample_mtier_build_ctx(bool promote) damon_set_schemes(ctx, &scheme, 1); quota_goal = damos_new_quota_goal( promote ? DAMOS_QUOTA_NODE_MEM_USED_BP : - DAMOS_QUOTA_NODE_MEM_FREE_BP, + DAMOS_QUOTA_NODE_MEM_FREE_BP, false, promote ? node0_mem_used_bp : node0_mem_free_bp); if (!quota_goal) goto free_out; diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 193a88d763ab..7398105ff3f6 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -71,12 +71,8 @@ class x86_page_ops(): self.NR_SECTION_ROOTS = DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.SECTIONS_PER_ROOT) - try: - self.SECTION_HAS_MEM_MAP = 1 << int(gdb.parse_and_eval('SECTION_HAS_MEM_MAP_BIT')) - self.SECTION_IS_EARLY = 1 << int(gdb.parse_and_eval('SECTION_IS_EARLY_BIT')) - except: - self.SECTION_HAS_MEM_MAP = 1 << 0 - self.SECTION_IS_EARLY = 1 << 3 + self.SECTION_HAS_MEM_MAP = 1 << int(gdb.parse_and_eval('SECTION_HAS_MEM_MAP_BIT')) + self.SECTION_IS_EARLY = 1 << int(gdb.parse_and_eval('SECTION_IS_EARLY_BIT')) self.SUBSECTION_SHIFT = 21 self.PAGES_PER_SUBSECTION = 1 << (self.SUBSECTION_SHIFT - self.PAGE_SHIFT) diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index 4256838d496c..310e19efff1d 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -334,6 +334,9 @@ static int sel_open_policy(struct inode *inode, struct file *filp) struct policy_load_memory *plm = NULL; int rc; + if (filp->f_mode & FMODE_WRITE) + return -EACCES; + rc = avc_has_perm(current_sid(), SECINITSID_SECURITY, SECCLASS_SECURITY, SECURITY__READ_POLICY, NULL); if (rc) @@ -418,14 +421,6 @@ static const struct vm_operations_struct sel_mmap_policy_ops = { static int sel_mmap_policy(struct file *filp, struct vm_area_struct *vma) { - if (vma->vm_flags & VM_SHARED) { - /* do not allow mprotect to make mapping writable */ - vm_flags_clear(vma, VM_MAYWRITE); - - if (vma->vm_flags & VM_WRITE) - return -EACCES; - } - vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &sel_mmap_policy_ops; diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c index 35152fae74ca..5cbf88da7626 100644 --- a/sound/core/pcm_native.c +++ b/sound/core/pcm_native.c @@ -3831,39 +3831,27 @@ static __poll_t snd_pcm_poll(struct file *file, poll_table *wait) /* * mmap status record */ -static vm_fault_t snd_pcm_mmap_status_fault(struct vm_fault *vmf) +static int snd_pcm_mmap_status(struct snd_pcm_substream *substream, struct file *file, + struct vm_area_struct *vma) { - struct snd_pcm_substream *substream = vmf->vma->vm_private_data; + const unsigned long size = vma->vm_end - vma->vm_start; struct snd_pcm_runtime *runtime; - - if (substream == NULL) - return VM_FAULT_SIGBUS; - runtime = substream->runtime; - vmf->page = virt_to_page(runtime->status); - get_page(vmf->page); - return 0; -} + struct page *page; -static const struct vm_operations_struct snd_pcm_vm_ops_status = -{ - .fault = snd_pcm_mmap_status_fault, -}; + BUILD_BUG_ON(sizeof(struct snd_pcm_mmap_status) > PAGE_SIZE); -static int snd_pcm_mmap_status(struct snd_pcm_substream *substream, struct file *file, - struct vm_area_struct *area) -{ - long size; - if (!(area->vm_flags & VM_READ)) + if (!(vma->vm_flags & VM_READ)) return -EINVAL; - size = area->vm_end - area->vm_start; - if (size != PAGE_ALIGN(sizeof(struct snd_pcm_mmap_status))) + if (size != PAGE_SIZE) return -EINVAL; - area->vm_ops = &snd_pcm_vm_ops_status; - area->vm_private_data = substream; - vm_flags_mod(area, VM_DONTEXPAND | VM_DONTDUMP, + + vm_flags_mod(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_WRITE | VM_MAYWRITE); + vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); - return 0; + runtime = substream->runtime; + page = virt_to_page(runtime->status); + return vm_insert_page(vma, vma->vm_start, page); } /* diff --git a/sound/usb/usx2y/us122l.c b/sound/usb/usx2y/us122l.c index a5db0d044ef9..2986a90ea335 100644 --- a/sound/usb/usx2y/us122l.c +++ b/sound/usb/usx2y/us122l.c @@ -193,7 +193,7 @@ static int usb_stream_hwdep_mmap(struct snd_hwdep *hw, } area->vm_ops = &usb_stream_hwdep_vm_ops; - vm_flags_set(area, VM_DONTDUMP); + vm_flags_set(area, VM_MIXEDMAP | VM_DONTDUMP); if (!read) vm_flags_set(area, VM_DONTEXPAND); area->vm_private_data = us122l; diff --git a/tools/cgroup/memcg_shrinker.py b/tools/cgroup/memcg_shrinker.py index e81c3017ada9..714772449c6b 100644 --- a/tools/cgroup/memcg_shrinker.py +++ b/tools/cgroup/memcg_shrinker.py @@ -31,7 +31,7 @@ def scan_shrinkers(shrinker_debugfs): items = line.split(' ') ino = int(items[0]) # (count, shrinker, memcg ino) - shrinkers.append((int(items[1]), shrinker, ino)) + shrinkers.append((sum(map(int, items[1:])), shrinker, ino)) return shrinkers diff --git a/tools/testing/selftests/damon/sysfs.sh b/tools/testing/selftests/damon/sysfs.sh index b66593c9ac47..1cc7ab2d7e22 100755 --- a/tools/testing/selftests/damon/sysfs.sh +++ b/tools/testing/selftests/damon/sysfs.sh @@ -212,6 +212,9 @@ test_goal() ensure_write_succ "$fpath" "node_eligible_mem_bp" "valid input" ensure_write_succ "$fpath" "hugepage_mem_bp" "valid input" ensure_write_fail "$fpath" "foo" "invalid input" + ensure_file "$goal_dir/complement" "exist" "600" + ensure_write_succ "$goal_dir/complement" "Y" "valid input" + ensure_write_succ "$goal_dir/complement" "N" "valid input" ensure_file "$goal_dir/nid" "exist" "600" ensure_file "$goal_dir/path" "exist" "600" } diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 311811dc55a0..45fb8ec38493 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -36,10 +36,19 @@ tap_prefix() fi } +is_nommu() +{ + grep -q '^MmapCopy:' /proc/meminfo 2>/dev/null +} + tap_timeout() { + # nommu doesn't support timeout command (missing fork(2)) + if is_nommu; then + echo "timeout isn't supported for NOMMU" + $1 # Make sure tests will time out if utility is available. - if [ -x /usr/bin/timeout ] ; then + elif [ -x /usr/bin/timeout ]; then /usr/bin/timeout --foreground "$kselftest_timeout" \ /usr/bin/timeout "$kselftest_timeout" $1 else diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index beacc0f87304..0ea72e997934 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -266,7 +266,7 @@ $(OUTPUT)/migration: LDLIBS += -lnuma $(OUTPUT)/rmap: LDLIBS += -lnuma local_config.mk local_config.h_gen: check_config.sh - $(call msg,CHK,config,$@) + @{ $(patsubst @%,%,$(call msg,CHK,config,$@)) :; } >&2 $(Q)CC="$(CC)" CFLAGS="$(CFLAGS)" ./check_config.sh EXTRA_CLEAN += local_config.mk local_config.h_gen diff --git a/tools/testing/selftests/mm/charge_reserved_hugetlb.sh b/tools/testing/selftests/mm/charge_reserved_hugetlb.sh index a1cfd3a349db..65065b1377b2 100755 --- a/tools/testing/selftests/mm/charge_reserved_hugetlb.sh +++ b/tools/testing/selftests/mm/charge_reserved_hugetlb.sh @@ -39,6 +39,14 @@ if [[ $cgroup2 ]]; then mount -t cgroup2 none $cgroup_path do_umount=1 fi + if ! grep -qw hugetlb $cgroup_path/cgroup.controllers; then + echo "hugetlb controller not available in cgroup v2. Skipping..." + if [[ $do_umount ]]; then + umount $cgroup_path + rmdir $cgroup_path + fi + exit $ksft_skip + fi echo "+hugetlb" >$cgroup_path/cgroup.subtree_control else cgroup_path=$(mount -t cgroup | grep ",hugetlb" | awk '{print $3}') diff --git a/tools/testing/selftests/mm/folio_split_race_test.c b/tools/testing/selftests/mm/folio_split_race_test.c index e4660bf89b62..1a9840b73e61 100644 --- a/tools/testing/selftests/mm/folio_split_race_test.c +++ b/tools/testing/selftests/mm/folio_split_race_test.c @@ -181,7 +181,7 @@ static uint64_t run_iteration(void) for (i = 0; i < TOTAL_PAGES; i++) fill_page(mmap_base, i); - if (!check_huge_shmem(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize)) + if (!check_huge_file(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize)) ksft_exit_fail_msg("No shmem THP is allocated\n"); if (pthread_barrier_init(&ctl.barrier, NULL, NUM_READER_THREADS + 1) != 0) diff --git a/tools/testing/selftests/mm/hugetlb-read-hwpoison.c b/tools/testing/selftests/mm/hugetlb-read-hwpoison.c index 70b24e3660c4..31748a24448f 100644 --- a/tools/testing/selftests/mm/hugetlb-read-hwpoison.c +++ b/tools/testing/selftests/mm/hugetlb-read-hwpoison.c @@ -13,6 +13,7 @@ #include <signal.h> #include "kselftest.h" +#include "vm_util.h" #define MAX_WRITE_READ_CHUNK_SIZE (getpagesize() * 16) #define MAX(a, b) (((a) > (b)) ? (a) : (b)) @@ -281,6 +282,13 @@ int main(void) size_t i; ksft_print_header(); + + if (!hugetlb_available()) + ksft_exit_skip("HugeTLB is not available\n"); + + if (!hugetlb_setup_default(ARRAY_SIZE(wr_chunk_sizes) * 2)) + ksft_exit_skip("Not enough free huge pages\n"); + ksft_set_plan(ARRAY_SIZE(wr_chunk_sizes) * 3); signal(SIGBUS, sigbus_handler); diff --git a/tools/testing/selftests/mm/hugetlb_reparenting_test.sh b/tools/testing/selftests/mm/hugetlb_reparenting_test.sh index 95f517c3bd16..811912ae7d97 100755 --- a/tools/testing/selftests/mm/hugetlb_reparenting_test.sh +++ b/tools/testing/selftests/mm/hugetlb_reparenting_test.sh @@ -29,6 +29,16 @@ if [[ $cgroup2 ]]; then mount -t cgroup2 none $CGROUP_ROOT do_umount=1 fi + for ctrl in hugetlb memory; do + if ! grep -qw $ctrl $CGROUP_ROOT/cgroup.controllers; then + echo "$ctrl controller not available in cgroup v2. Skipping..." + if [[ $do_umount ]]; then + umount $CGROUP_ROOT + rmdir $CGROUP_ROOT + fi + exit $ksft_skip + fi + done echo "+hugetlb +memory" >$CGROUP_ROOT/cgroup.subtree_control else CGROUP_ROOT=$(mount -t cgroup | grep ",hugetlb" | awk '{print $3}') diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index a2ac3b3ca5de..1bc7acc66a46 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -57,7 +57,7 @@ struct mem_ops { void *(*setup_area)(int nr_hpages); void (*cleanup_area)(void *p, unsigned long size); void (*fault)(void *p, unsigned long start, unsigned long end); - bool (*check_huge)(void *addr, size_t len, int nr_hpages, unsigned long hpage_size); + bool (*check_huge)(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); const char *name; }; @@ -343,12 +343,6 @@ static void anon_fault(void *p, unsigned long start, unsigned long end) fill_memory(p, start, end); } -static bool anon_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - return check_huge_anon(addr, len, nr_hpages, hpage_size); -} - static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup) { const int open_opt = setup == FILE_SETUP_READ_ONLY_FS ? O_RDONLY : O_RDWR; @@ -444,20 +438,6 @@ static void file_fault_write(void *p, unsigned long start, unsigned long end) ksft_exit_fail_perror("madvise(MADV_POPULATE_WRITE)"); } -static bool file_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - switch (finfo.type) { - case VMA_FILE: - return check_huge_file(addr, len, nr_hpages, hpage_size); - case VMA_SHMEM: - return check_huge_shmem(addr, len, nr_hpages, hpage_size); - default: - ksft_exit_fail_msg("Unknown VMA type\n"); - return false; - } -} - static void *shmem_setup_area(int nr_hpages) { void *p; @@ -481,17 +461,11 @@ static void shmem_cleanup_area(void *p, unsigned long size) close(finfo.fd); } -static bool shmem_check_huge(void *addr, size_t len, int nr_hpages, - unsigned long hpage_size) -{ - return check_huge_shmem(addr, len, nr_hpages, hpage_size); -} - static struct mem_ops __anon_ops = { .setup_area = &anon_setup_area, .cleanup_area = &anon_cleanup_area, .fault = &anon_fault, - .check_huge = &anon_check_huge, + .check_huge = &check_huge_anon, .name = "anon", }; @@ -499,7 +473,7 @@ static struct mem_ops __read_only_file_ops = { .setup_area = &file_setup_read_only_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_read, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -507,7 +481,7 @@ static struct mem_ops __read_write_file_read_ops = { .setup_area = &file_setup_read_write_fs_read_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_read_and_flush, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -515,7 +489,7 @@ static struct mem_ops __read_write_file_write_ops = { .setup_area = &file_setup_read_write_fs_write_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_write, - .check_huge = &file_check_huge, + .check_huge = &check_huge_file, .name = "file", }; @@ -523,7 +497,7 @@ static struct mem_ops __shmem_ops = { .setup_area = &shmem_setup_area, .cleanup_area = &shmem_cleanup_area, .fault = &anon_fault, - .check_huge = &shmem_check_huge, + .check_huge = &check_huge_file, .name = "shmem", }; @@ -618,6 +592,16 @@ static bool wait_for_scan(const char *msg, char *p, size_t len, usleep(TICK); } + /* + * The file and shmem tests rely on refaults to install PMD mappings + * after collapse. MADV_NOHUGEPAGE would prevent those mappings. + * + * Apply MADV_NOHUGEPAGE only to anonymous VMAs to prevent khugepaged + * from unexpectedly collapsing pages during the test. + */ + if (is_anon(ops)) + madvise(p, len, MADV_NOHUGEPAGE); + return timeout == -1; } @@ -1053,6 +1037,8 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has small page..."); if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) @@ -1060,6 +1046,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) else fail("Fail"); validate_memory(p, 0, page_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } @@ -1100,6 +1087,8 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has huge page..."); if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) @@ -1107,6 +1096,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o else fail("Fail"); validate_memory(p, 0, hpage_pmd_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } @@ -1158,6 +1148,8 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops wait(&wstatus); exit_status = WEXITSTATUS(wstatus); + if (exit_status == KSFT_FAIL) + goto out; ksft_print_msg("Check if parent still has huge page..."); if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) @@ -1165,6 +1157,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops else fail("Fail"); validate_memory(p, 0, hpage_pmd_size); +out: ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } diff --git a/tools/testing/selftests/mm/mlock2-tests.c b/tools/testing/selftests/mm/mlock2-tests.c index e16e288cc7c1..144b550813a6 100644 --- a/tools/testing/selftests/mm/mlock2-tests.c +++ b/tools/testing/selftests/mm/mlock2-tests.c @@ -502,7 +502,7 @@ int main(int argc, char **argv) ret = mlock2_(map, size, MLOCK_ONFAULT); if (ret && errno == ENOSYS) - ksft_finished(); + ksft_exit_skip("mlock2() syscall is not supported\n"); munmap(map, size); diff --git a/tools/testing/selftests/mm/mlock2.h b/tools/testing/selftests/mm/mlock2.h index 81e77fa41901..4417eaa5cfb7 100644 --- a/tools/testing/selftests/mm/mlock2.h +++ b/tools/testing/selftests/mm/mlock2.h @@ -6,13 +6,7 @@ static int mlock2_(void *start, size_t len, int flags) { - int ret = syscall(__NR_mlock2, start, len, flags); - - if (ret) { - errno = ret; - return -1; - } - return 0; + return syscall(__NR_mlock2, start, len, flags); } static FILE *seek_to_smaps_entry(unsigned long addr) diff --git a/tools/testing/selftests/mm/mrelease_test.c b/tools/testing/selftests/mm/mrelease_test.c index 64e8d00ae944..a5124c9e9f43 100644 --- a/tools/testing/selftests/mm/mrelease_test.c +++ b/tools/testing/selftests/mm/mrelease_test.c @@ -157,7 +157,7 @@ retry: * Retry until we succeed or reach MAX_SIZE_MB. */ if (errno == ESRCH) { - retry = (size <= MAX_SIZE_MB); + retry = (size < MAX_SIZE_MB); } else { waitpid(pid, NULL, 0); ksft_exit_fail_msg("process_mrelease: %s\n", strerror(errno)); diff --git a/tools/testing/selftests/mm/mremap_dontunmap.c b/tools/testing/selftests/mm/mremap_dontunmap.c index 96ba537facf7..5ca33d9d42af 100644 --- a/tools/testing/selftests/mm/mremap_dontunmap.c +++ b/tools/testing/selftests/mm/mremap_dontunmap.c @@ -7,6 +7,7 @@ */ #define _GNU_SOURCE #include <sys/mman.h> +#include <sys/syscall.h> #include <linux/mman.h> #include <errno.h> #include <stdio.h> @@ -15,6 +16,7 @@ #include <unistd.h> #include "kselftest.h" +#include "vm_util.h" unsigned long page_size; char *page_buffer; @@ -335,6 +337,62 @@ static void mremap_dontunmap_partial_mapping_overwrite(void) ksft_test_result_pass("%s\n", __func__); } +/* VmLck from /proc/self/status, which is mm->locked_vm in kB. */ +static unsigned long locked_vm_kb(void) +{ + unsigned long kb; + char buf[256]; + FILE *fp; + + fp = fopen("/proc/self/status", "r"); + BUG_ON(!fp, "unable to open /proc/self/status"); + BUG_ON(!check_for_pattern(fp, "VmLck:", buf, sizeof(buf)) || + sscanf(buf, "VmLck: %lu kB", &kb) != 1, "unable to read VmLck"); + fclose(fp); + + return kb; +} + +/* + * MREMAP_DONTUNMAP clears the mlock flags of the source VMA and sets them on + * the destination, and mm->locked_vm has to follow. mlock2() a source of + * num_pages and move its first move_pages to gap pages past its end: + * + * |------ source ------|... gap ...|-- dest --| + * + * then unmap everything and check that VmLck is back where it started. With + * no gap the destination is adjacent to the source and could merge with it, + * otherwise the gap is left PROT_NONE so that it cannot. + */ +static void mremap_dontunmap_mlock(const char *desc, unsigned long num_pages, + unsigned long move_pages, unsigned long gap, + int mlock_flags) +{ + unsigned long size = (num_pages + gap + move_pages) * page_size; + unsigned long locked = locked_vm_kb(); + void *source, *dest; + + source = mmap(NULL, size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + BUG_ON(source == MAP_FAILED, "mmap"); + BUG_ON(mprotect(source, num_pages * page_size, + PROT_READ | PROT_WRITE) == -1, "mprotect"); + dest = source + (num_pages + gap) * page_size; + + if (syscall(__NR_mlock2, source, num_pages * page_size, mlock_flags)) { + ksft_test_result_skip("%s: %s: mlock2: %s\n", __func__, desc, + strerror(errno)); + BUG_ON(munmap(source, size) == -1, "unable to unmap mappings"); + return; + } + + BUG_ON(mremap(source, move_pages * page_size, move_pages * page_size, + MREMAP_DONTUNMAP | MREMAP_MAYMOVE | MREMAP_FIXED, + dest) != dest, "mremap"); + + BUG_ON(munmap(source, size) == -1, "unable to unmap mappings"); + ksft_test_result(locked_vm_kb() == locked, "%s: %s\n", __func__, desc); +} + int main(void) { ksft_print_header(); @@ -348,7 +406,7 @@ int main(void) ksft_finished(); } - ksft_set_plan(5); + ksft_set_plan(8); // Keep a page sized buffer around for when we need it. page_buffer = @@ -361,6 +419,9 @@ int main(void) mremap_dontunmap_simple_fixed(); mremap_dontunmap_partial_mapping(); mremap_dontunmap_partial_mapping_overwrite(); + mremap_dontunmap_mlock("onfault self-merge", 1, 1, 0, MLOCK_ONFAULT); + mremap_dontunmap_mlock("partial", 3, 2, 1, 0); + mremap_dontunmap_mlock("onfault partial", 3, 2, 1, MLOCK_ONFAULT); BUG_ON(munmap(page_buffer, page_size) == -1, "unable to unmap page buffer"); diff --git a/tools/testing/selftests/mm/mremap_test.c b/tools/testing/selftests/mm/mremap_test.c index 97abf4713cc5..31b484517bda 100644 --- a/tools/testing/selftests/mm/mremap_test.c +++ b/tools/testing/selftests/mm/mremap_test.c @@ -1006,8 +1006,10 @@ static long long remap_region(struct config c, unsigned int threshold_mb, if (!((unsigned long long) addr & c.dest_alignment)) addr = (void *) ((unsigned long long) addr | c.dest_alignment); - /* Don't destroy existing mappings unless expected to overlap */ - while (!is_remap_region_valid(addr, c.region_size) && !c.overlapping) { + /* Don't destroy existing mappings unless expected to overlap. */ + while (!is_remap_region_valid(addr - c.dest_preamble_size, + c.dest_preamble_size + c.region_size) && + !c.overlapping) { /* Check for unsigned overflow */ tmp_addr = addr + c.dest_alignment; if (tmp_addr < addr) { diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index c5d96a4b1db3..36a6719ff09a 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -108,6 +108,15 @@ static char *allocate_zero_filled_hugepage(size_t len) return result; } +static void disable_khugepaged(void *addr, size_t len) +{ + /* Disables khugepaged from collapsing pages in range into THPs */ + if (!madvise(addr, len, MADV_NOHUGEPAGE)) + return; + + ksft_exit_fail_perror("MADV_NOHUGEPAGE failed"); +} + static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hpages, size_t len) { unsigned long rss_anon_before, rss_anon_after; @@ -120,6 +129,8 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp if (!rss_anon_before) ksft_exit_fail_msg("No RssAnon is allocated before split\n"); + disable_khugepaged(one_page, len); + /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, (uint64_t)one_page + len, 0); @@ -167,6 +178,8 @@ static void split_pmd_thp_to_order(int order) if (!check_huge_anon(one_page, 4 * pmd_pagesize, 4, pmd_pagesize)) ksft_exit_fail_msg("No THP is allocated\n"); + disable_khugepaged(one_page, len); + /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, (uint64_t)one_page + len, order); @@ -215,6 +228,8 @@ static void split_pte_mapped_thp(void) goto out; } + disable_khugepaged(thp_area, thp_area_size); + /* * To challenge spitting code, we will mremap a single page of each * THP (page[i] of thp[i]) in the thp_area into page_area. This will @@ -482,6 +497,7 @@ static int create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, ksft_test_result_skip("Pagecache folio split skipped\n"); return -2; } + disable_khugepaged(*addr, fd_size); return 0; err_out_close: close(*fd); diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index 1fb967ef4985..a76851aced4d 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -196,8 +196,8 @@ static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *p, int { size_t len = expect_nr_hpages * read_pmd_pagesize(); - if (!check_huge_shmem(gopts->area_dst_alias, len, expect_nr_hpages, - read_pmd_pagesize())) + if (!check_huge_file(gopts->area_dst_alias, len, expect_nr_hpages, + read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); } diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c index ef9b3956bdcf..2a97ba1d6feb 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -1937,6 +1937,11 @@ uffd_move_test_common(uffd_global_test_opts_t *gopts, args.gopts = gopts; + if (chunk_size > gopts->nr_pages * gopts->page_size) { + uffd_test_skip("chunk size does not fit in the test area"); + return; + } + /* Prevent source pages from being mapped more than once */ if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DONTFORK)) err("madvise(MADV_DONTFORK) failure"); diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index a0ab78ceedb1..515031b8283d 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -351,24 +351,6 @@ err_out: return entry; } -static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages, - uint64_t hpage_size) -{ - char buffer[MAX_LINE_LENGTH]; - uint64_t thp = -1; - char *entry; - - entry = __get_smap_entry(addr, pattern, buffer, sizeof(buffer)); - if (!entry) - goto err_out; - - if (sscanf(entry, "%9" SCNu64 " kB", &thp) != 1) - ksft_exit_fail_msg("Reading smap error\n"); - -err_out: - return thp == (nr_hpages * (hpage_size >> 10)); -} - static bool check_large_folios(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { @@ -410,43 +392,98 @@ out: return ret; } -bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) +enum check_type { + CHECK_TYPE_ANON, + CHECK_TYPE_FILE, +}; + +static bool __check_type(void *addr, size_t len, uint64_t page_size, + enum check_type type) { - uint64_t pmd_pagesize = read_pmd_pagesize(); + bool ret = false; + int pagemap_fd; + char *start = addr; + char *end = start + len; + uint64_t categories; - if (!pmd_pagesize) - ksft_exit_fail_msg("reading PMD pagesize failed\n"); + pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + for (; start < end; start += page_size) { + categories = pagemap_scan_get_categories(pagemap_fd, start); + if ((categories & PAGE_IS_PRESENT) != PAGE_IS_PRESENT) + continue; + + if ((type == CHECK_TYPE_FILE) != !!(categories & PAGE_IS_FILE)) + goto out; + } - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size); + ret = true; - return check_large_folios(addr, len, nr_hpages, hpage_size); +out: + close(pagemap_fd); + return ret; } -bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) +static bool __check_huge(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size) { - uint64_t pmd_pagesize = read_pmd_pagesize(); - + bool ret = false; + int pagemap_fd; + int nr_pmd_mappings = 0; + uint64_t pmd_pagesize; + uint64_t categories; + char *start = addr; + char *end = start + len; + + pmd_pagesize = read_pmd_pagesize(); if (!pmd_pagesize) ksft_exit_fail_msg("reading PMD pagesize failed\n"); - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size); + pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + if (hpage_size != pmd_pagesize) { + ret = check_large_folios(addr, len, nr_hpages, hpage_size); + goto out; + } + + for (; start < end; start += hpage_size) { + categories = pagemap_scan_get_categories(pagemap_fd, start); + if (categories & PAGE_IS_HUGE) + nr_pmd_mappings++; + } + + if (nr_pmd_mappings != nr_hpages) + goto out; + + ret = true; - return check_large_folios(addr, len, nr_hpages, hpage_size); +out: + close(pagemap_fd); + return ret; } -bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) +bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - uint64_t pmd_pagesize = read_pmd_pagesize(); + const uint64_t scan_mapping_size = (nr_hpages > 0) ? hpage_size : psize(); - if (!pmd_pagesize) - ksft_exit_fail_msg("reading PMD pagesize failed\n"); + if (!__check_huge(addr, len, nr_hpages, hpage_size)) + return false; - if (hpage_size == pmd_pagesize) - return __check_pmd_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size); + return __check_type(addr, len, scan_mapping_size, CHECK_TYPE_ANON); +} + +bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) +{ + const uint64_t scan_mapping_size = (nr_hpages > 0) ? hpage_size : psize(); + + if (!__check_huge(addr, len, nr_hpages, hpage_size)) + return false; - return check_large_folios(addr, len, nr_hpages, hpage_size); + return __check_type(addr, len, scan_mapping_size, CHECK_TYPE_FILE); } int64_t allocate_transhuge(void *ptr, int pagemap_fd) diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index 072a6c756c51..b5d59729d432 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -96,7 +96,6 @@ uint64_t read_pmd_pagesize(void); unsigned long rss_anon(void); bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); -bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); int64_t allocate_transhuge(void *ptr, int pagemap_fd); int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags); int gather_folio_orders(char *vaddr_start, size_t len, diff --git a/tools/testing/selftests/nommu/Makefile b/tools/testing/selftests/nommu/Makefile new file mode 100644 index 000000000000..87bc3bbfa2fd --- /dev/null +++ b/tools/testing/selftests/nommu/Makefile @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +# Makefile for nommu selftests + +TEST_GEN_PROGS += nommu_mmap_test +TEST_GEN_PROGS += nommu_mremap_test + +include ../lib.mk diff --git a/tools/testing/selftests/nommu/nommu.h b/tools/testing/selftests/nommu/nommu.h new file mode 100644 index 000000000000..2c9a2f4e013a --- /dev/null +++ b/tools/testing/selftests/nommu/nommu.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef NOMMU_H +#define NOMMU_H + +#include <stdio.h> +#include <string.h> + +/* Returns 1 for NOMMU, 0 for MMU, and -1 if /proc/meminfo is unavailable. */ +static inline int ksft_is_nommu(void) +{ + FILE *fp; + char line[256]; + int nommu = 0; + + fp = fopen("/proc/meminfo", "r"); + if (!fp) + return -1; + + while (fgets(line, sizeof(line), fp)) { + if (strncmp(line, "MmapCopy:", sizeof("MmapCopy:") - 1) == 0) { + nommu = 1; + break; + } + } + + if (ferror(fp)) + nommu = -1; + fclose(fp); + + return nommu; +} + +#endif diff --git a/tools/testing/selftests/nommu/nommu_mmap_test.c b/tools/testing/selftests/nommu/nommu_mmap_test.c new file mode 100644 index 000000000000..2b3639cf72ef --- /dev/null +++ b/tools/testing/selftests/nommu/nommu_mmap_test.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <sys/mman.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <string.h> +#include <limits.h> +#include "kselftest.h" +#include "nommu.h" + +#include <sys/vfs.h> +#ifndef RAMFS_MAGIC +#define RAMFS_MAGIC 0x858458f6 +#endif + +static size_t ps; +static int nommu; + +struct test_case_t { + const char *name; + const char *pathname; + int open_flags; + int mmap_prot; + int mmap_flags; + int exp_err; + int (*resolve_exp_err)(const char *path); +}; + +static int get_shm_expected_error(const char *path) +{ + struct statfs fs; + + if (!nommu) + return 0; + + if (statfs(path, &fs) == 0) { + if (fs.f_type == RAMFS_MAGIC) + return 0; /* ramfs succeed with contiguous memory */ + } + /* hostfs, etc returns ENODEV due to lack of contiguous allocation */ + return ENODEV; +} + +static struct test_case_t test_cases[] = { + { + .name = "anonymous private allocation", + .pathname = NULL, + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_ANONYMOUS | MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous private file mapping (rw-)", + .pathname = "/tmp/ksft.nommu-reg-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous private file mapping (r--)", + .pathname = "/tmp/ksft.nommu-reg-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = NULL, + }, + { + .name = "non-anonymous shared file mapping (rw-)", + .pathname = "/tmp/ksft.nommu-shm-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shm_expected_error, + }, + { + .name = "non-anonymous shared file mapping (r--)", + .pathname = "/tmp/ksft.nommu-shm-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shm_expected_error, + }, +}; + +static int run_mapping_matrix_test(struct test_case_t *tcase) +{ + int fd; + void *ptr = MAP_FAILED; + char path_buf[PATH_MAX]; + const char *path = tcase->pathname; + int rc = KSFT_PASS; + int expected_error; + + ksft_print_msg("[RUN] %s\n", tcase->name); + + if (tcase->pathname == NULL) { + fd = -1; + } else if (strstr(tcase->pathname, "XXXXXX")) { + strncpy(path_buf, tcase->pathname, sizeof(path_buf) - 1); + path_buf[sizeof(path_buf) - 1] = '\0'; + fd = mkstemp(path_buf); + if (fd < 0) { + ksft_print_msg("Failed to setup temp node: %s\n", + tcase->pathname); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + if (ftruncate(fd, ps) != 0) { + ksft_print_msg("ftruncate failed for: %s\n", tcase->pathname); + rc = KSFT_FAIL; + goto cleanup; + } + path = path_buf; + } else { + fd = open(tcase->pathname, tcase->open_flags, 0600); + if (fd < 0) { + ksft_print_msg("Device node not accessible: %s\n", + tcase->pathname); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + } + + expected_error = tcase->exp_err; + if (tcase->resolve_exp_err && fd >= 0) + expected_error = tcase->resolve_exp_err(path); + + ptr = mmap(NULL, ps, tcase->mmap_prot, tcase->mmap_flags, fd, 0); + + if (expected_error != 0) { + if (ptr != MAP_FAILED) { + ksft_print_msg("mmap unexpectedly succeeded (exp error %d)\n", + expected_error); + rc = KSFT_FAIL; + goto cleanup; + } + if (errno != expected_error) { + ksft_print_msg("mmap failed with %d (%s), but expected %d\n", + errno, strerror(errno), expected_error); + rc = KSFT_FAIL; + goto cleanup; + } + ksft_print_msg("Correctly rejected with expected error %s(%d)\n", + strerror(expected_error), expected_error); + rc = KSFT_PASS; + goto cleanup; + } + + if (ptr == MAP_FAILED) { + ksft_print_msg("mmap failed unexpectedly: %s\n", strerror(errno)); + rc = KSFT_FAIL; + goto cleanup; + } + +cleanup: + if (ptr != MAP_FAILED && munmap(ptr, ps)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + rc = KSFT_FAIL; + } + if (fd >= 0) { + close(fd); + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX") && + unlink(path_buf)) { + ksft_print_msg("unlink failed for %s: %s\n", + path_buf, strerror(errno)); + rc = KSFT_FAIL; + } + } + ksft_test_result_report(rc, "%s\n", tcase->name); + return rc; +} + +static int test_map_fixed(void) +{ + void *fixed_addr; + void *ptr; + + ksft_print_msg("[RUN] %s\n", __func__); + + fixed_addr = mmap(NULL, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (fixed_addr == MAP_FAILED) { + ksft_print_msg("Unable to reserve test address: %s\n", + strerror(errno)); + ksft_test_result_skip("MAP_FIXED behavior\n"); + return KSFT_SKIP; + } + + if (munmap(fixed_addr, ps)) { + ksft_print_msg("Unable to release test address: %s\n", + strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + + ptr = mmap(fixed_addr, ps, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + + if (nommu) { + if (ptr == MAP_FAILED && (errno == ENODEV || errno == EINVAL)) { + ksft_print_msg("MAP_FIXED correctly rejected under nommu\n"); + ksft_test_result_pass("MAP_FIXED behavior\n"); + return KSFT_PASS; + } + if (ptr != MAP_FAILED) { + ksft_print_msg("MAP_FIXED unexpectedly allowed under nommu\n"); + ksft_test_result_fail("MAP_FIXED behavior\n"); + if (munmap(ptr, ps)) + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + return KSFT_FAIL; + } + ksft_print_msg("MAP_FIXED failed under NOMMU: %s\n", + strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + if (ptr != MAP_FAILED) { + ksft_print_msg("MAP_FIXED successfully allocated under MMU\n"); + if (munmap(ptr, ps)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; + } + ksft_test_result_pass("MAP_FIXED behavior\n"); + return KSFT_PASS; + } + ksft_print_msg("MAP_FIXED failed allocation under MMU\n"); + ksft_test_result_fail("MAP_FIXED behavior\n"); + return KSFT_FAIL; +} + +int main(int argc, char **argv) +{ + int i; + + nommu = ksft_is_nommu(); + if (nommu < 0) + ksft_exit_skip("Cannot read /proc/meminfo; mount procfs at /proc\n"); + + ps = sysconf(_SC_PAGESIZE); + ksft_print_header(); + ksft_set_plan(ARRAY_SIZE(test_cases) + 1); + + if (nommu) + ksft_print_msg("Running strict MMAP test criteria under nommu architecture\n"); + else + ksft_print_msg("Running MMAP test criteria under MMU architecture\n"); + + test_map_fixed(); + for (i = 0; i < (int)ARRAY_SIZE(test_cases); i++) + run_mapping_matrix_test(&test_cases[i]); + + ksft_finished(); +} diff --git a/tools/testing/selftests/nommu/nommu_mremap_test.c b/tools/testing/selftests/nommu/nommu_mremap_test.c new file mode 100644 index 000000000000..6ee89cb25f61 --- /dev/null +++ b/tools/testing/selftests/nommu/nommu_mremap_test.c @@ -0,0 +1,356 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <sys/mman.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <string.h> +#include <limits.h> +#include "kselftest.h" +#include "nommu.h" + +#include <sys/vfs.h> +#ifndef RAMFS_MAGIC +#define RAMFS_MAGIC 0x858458f6 +#endif + +static size_t ps; +static int nommu; + +static long get_fs_type(const char *path) +{ + struct statfs fs; + + if (statfs(path, &fs) == 0) + return fs.f_type; + + return 0; +} + +static void munmap_shrink_test(void) +{ + void *addr; + int ret; + + /* munmap shrink test */ + for (int i = 0; i < 4; i++) { + addr = mmap(NULL, ps * 4, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap failed: %s(%d)\n", strerror(errno), errno); + ksft_test_result_fail("munmap shrink\n"); + return; + } + ret = munmap((char *)addr + ps * i, ps); + if (ret != 0) { + ksft_print_msg("memory %p isn't unmapped at %p\n", + addr, (char *)addr + ps * i); + goto error; + } + + if (i == 0) { + if (munmap(addr + ps, ps * 3)) + goto error; + } else if (i == 1) { + if (munmap(addr, ps) || munmap(addr + (ps * 2), ps * 2)) + goto error; + } else if (i == 2) { + if (munmap(addr, ps * 2) || munmap(addr + (ps * 3), ps)) + goto error; + } else if (i == 3) { + if (munmap(addr, ps * 3)) + goto error; + } + } + + ksft_test_result_pass("munmap shrink\n"); + return; +error: + for (int j = 0; j < 4; j++) { + if (munmap((char *)addr + j * ps, ps)) + ksft_print_msg("cleanup munmap failed: %s\n", strerror(errno)); + } + ksft_print_msg("clean up failures\n"); + ksft_test_result_fail("munmap shrink\n"); +} + +static size_t page_align(size_t len) +{ + return (len + ps - 1) / ps * ps; +} + +static void mremap_shrink_test(void) +{ + void *addr, *addr2; + size_t current_len; + size_t old_len, new_len; + struct param { + size_t old; + size_t new; + } params[] = { + /* should not happen any shrink */ + { .old = ps * 4 - 1, .new = ps * 4 - 2 }, + /* should not happen any shrink */ + { .old = ps * 4 - 1, .new = ps * 4 }, + { .old = ps * 4, .new = ps * 2 }, + /* should not happen any shrink */ + { .old = ps * 2, .new = ps * 2 - 2 }, + { .old = ps * 2 - 2, .new = ps * 1 }, + }; + + /* mremap shrink test */ + current_len = page_align(ps * 4 - 1); + addr = mmap(NULL, ps * 4 - 1, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap failed: %s(%d)\n", strerror(errno), errno); + ksft_test_result_fail("mremap shrink\n"); + return; + } + + for (int i = 0; i < ARRAY_SIZE(params); i++) { + old_len = params[i].old; + new_len = params[i].new; + current_len = page_align(new_len); + addr2 = mremap(addr, old_len, new_len, MREMAP_MAYMOVE); + if (addr2 == MAP_FAILED) { + ksft_print_msg("memory %p isn't remapped at %p\n", addr, addr2); + ksft_test_result_fail("mremap shrink\n"); + munmap(addr, page_align(old_len)); + return; + } + + addr = addr2; + } + + if (munmap(addr, current_len)) { + ksft_print_msg("cleanup failed: %s\n", strerror(errno)); + ksft_test_result_fail("mremap shrink\n"); + return; + } + ksft_test_result_pass("mremap shrink\n"); +} + +static int get_shared_writable_file_expected_error(const char *path) +{ + if (nommu && get_fs_type(path) == RAMFS_MAGIC) + return EPERM; /* ramfs failed */ + + return 0; +} + +struct mremap_case_t { + const char *name; + const char *pathname; + int open_flags; + int mmap_prot; + int mmap_flags; + int exp_err; + int nommu_exp_err; + int (*resolve_exp_err)(const char *path); + unsigned int old_pages; + unsigned int new_pages; +}; + +static struct mremap_case_t mremap_cases[] = { + { + .name = "anonymous shrink (r--)", + .pathname = NULL, + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_ANONYMOUS | MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = 0, + }, + { + .name = "shared file shrink (r--)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_SHARED, + .exp_err = 0, + .resolve_exp_err = get_shared_writable_file_expected_error, + }, + { + .name = "private file unchanged length (r-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = EPERM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 4, + }, + { + .name = "private file unchanged length (rw-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 4, + }, + { + .name = "private file growth (r-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = EPERM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 8, + }, + { + .name = "private file growth (rw-)", + .pathname = "/tmp/ksft.nommu-remap-XXXXXX", + .open_flags = O_CREAT | O_RDWR | O_EXCL, + .mmap_prot = PROT_READ | PROT_WRITE, + .mmap_flags = MAP_PRIVATE, + .exp_err = 0, + .nommu_exp_err = ENOMEM, + .resolve_exp_err = 0, + .old_pages = 4, + .new_pages = 8, + }, +}; + +static int run_mremap_test(struct mremap_case_t *tcase) +{ + int fd = -1; + void *addr, *addr2; + char pb[PATH_MAX]; + const char *path = tcase->pathname; + int rc = KSFT_PASS; + int expected_error; + unsigned int old_pages = tcase->old_pages ?: 4; + unsigned int new_pages = tcase->new_pages ?: 2; + unsigned int file_pages = old_pages > new_pages ? + old_pages : new_pages; + + ksft_print_msg("[RUN] Testing mremap: %s\n", tcase->name); + + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX")) { + strncpy(pb, tcase->pathname, sizeof(pb) - 1); + pb[sizeof(pb) - 1] = '\0'; + fd = mkstemp(pb); + if (fd < 0) { + ksft_print_msg("Failed to setup file backing\n"); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + if (ftruncate(fd, ps * file_pages) != 0) { + ksft_print_msg("Failed to setup file backing\n"); + rc = KSFT_FAIL; + goto out; + } + + if (nommu && (tcase->mmap_flags & MAP_SHARED) && + get_fs_type(pb) != RAMFS_MAGIC) { + ksft_print_msg("Skip the test under non-ramfs filesystem (%s)\n", + pb); + rc = KSFT_SKIP; + goto out; + } + path = pb; + } else if (tcase->pathname) { + fd = open(tcase->pathname, tcase->open_flags, 0600); + if (fd < 0) { + ksft_print_msg("Backing node not accessible\n"); + ksft_test_result_skip("%s\n", tcase->name); + return KSFT_SKIP; + } + + if (nommu && (tcase->mmap_flags & MAP_SHARED) && + get_fs_type(tcase->pathname) != RAMFS_MAGIC) { + ksft_print_msg("Skip the test under non-ramfs filesystem (%s)\n", + tcase->pathname); + rc = KSFT_SKIP; + goto out; + } + } + + addr = mmap(NULL, ps * old_pages, tcase->mmap_prot, + tcase->mmap_flags, fd, 0); + if (addr == MAP_FAILED) { + ksft_print_msg("mmap mapping failed %s(%d)\n", strerror(errno), errno); + rc = KSFT_FAIL; + goto out; + } + + expected_error = nommu ? tcase->nommu_exp_err : tcase->exp_err; + if (tcase->resolve_exp_err && fd >= 0) + expected_error = tcase->resolve_exp_err(path); + + addr2 = mremap(addr, ps * old_pages, ps * new_pages, + MREMAP_MAYMOVE); + + if (expected_error != 0) { + if (addr2 != MAP_FAILED) { + ksft_print_msg("Expected error %d, but mremap unexpectedly succeeded\n", + expected_error); + rc = KSFT_FAIL; + } else if (errno != expected_error) { + ksft_print_msg("Expected error %d, got %s(%d)\n", + expected_error, strerror(errno), errno); + rc = KSFT_FAIL; + } else { + ksft_print_msg("%s: Handled expected error path (errno=%d)\n", + tcase->name, expected_error); + } + } else if (addr2 == MAP_FAILED) { + ksft_print_msg("mremap shrink failed unexpectedly: %s\n", + strerror(errno)); + rc = KSFT_FAIL; + } else { + ksft_print_msg("%s step successful\n", tcase->name); + } + + /* clean up */ + if (munmap(addr2 == MAP_FAILED ? addr : addr2, + addr2 == MAP_FAILED ? ps * old_pages : ps * new_pages)) { + ksft_print_msg("munmap failed: %s\n", strerror(errno)); + rc = KSFT_FAIL; + } + +out: + if (fd >= 0) { + close(fd); + if (tcase->pathname && strstr(tcase->pathname, "XXXXXX") && + unlink(pb)) { + ksft_print_msg("unlink failed for %s: %s\n", pb, strerror(errno)); + rc = KSFT_FAIL; + } + } + + ksft_test_result_report(rc, "%s\n", tcase->name); + return rc; +} + +int main(int argc, char **argv) +{ + int i; + + nommu = ksft_is_nommu(); + if (nommu < 0) + ksft_exit_skip("Cannot read /proc/meminfo; mount procfs at /proc\n"); + + ps = sysconf(_SC_PAGESIZE); + ksft_print_header(); + ksft_set_plan(ARRAY_SIZE(mremap_cases) + 2); + + munmap_shrink_test(); + mremap_shrink_test(); + + for (i = 0; i < (int)ARRAY_SIZE(mremap_cases); i++) + run_mremap_test(&mremap_cases[i]); + + ksft_finished(); +} diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c index e0a0693df08f..a37d2f86d325 100644 --- a/tools/testing/shared/linux.c +++ b/tools/testing/shared/linux.c @@ -83,7 +83,7 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, } else { pthread_mutex_unlock(&cachep->lock); if (cachep->align) { - if (posix_memalign(&p, cachep->align, cachep->size) < 0) + if (posix_memalign(&p, cachep->align, cachep->size)) return NULL; } else { p = malloc(cachep->size); @@ -191,7 +191,7 @@ bool kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, if (cachep->align) { if (posix_memalign(&p[i], cachep->align, - cachep->size) < 0) + cachep->size)) break; } else { p[i] = malloc(cachep->size); diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 16c09dac59d9..f449129ea97a 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -352,14 +352,6 @@ enum { #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC) #define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT) -/* - * Special vmas that are non-mergable, non-mlock()able. - */ -#define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP) - -#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ - VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) - #define VMA_REMAP_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, \ VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT) @@ -454,17 +446,20 @@ static __always_inline bool vma_flags_empty(const vma_flags_t *flags) /* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type { - MMAP_NOTHING, /* Mapping is complete, no further action. */ - MMAP_REMAP_PFN, /* Remap PFN range. */ - MMAP_IO_REMAP_PFN, /* I/O remap PFN range. */ - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_MAP_KERNEL_PAGES, /* Map kernel page range from an array. */ + MMAP_NOTHING, + MMAP_REMAP_PFN, + MMAP_IO_REMAP_PFN, + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ }; /* * Describes an action an mmap_prepare hook can instruct to be taken to complete * the mapping of a VMA. Specified in vm_area_desc. */ +struct discontig_kernel_page_ops; + struct mmap_action { union { struct { @@ -483,6 +478,10 @@ struct mmap_action { unsigned long nr_pages; pgoff_t pgoff; } map_kernel; + struct { + void *init_private; + const struct discontig_kernel_page_ops *ops; + } map_kernel_discontig; }; enum mmap_action_type type; @@ -1179,7 +1178,7 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_is_attached(struct vm_area_struct *vma) +static inline bool vma_is_attached(const struct vm_area_struct *vma) { return refcount_read(&vma->vm_refcnt); } @@ -1226,7 +1225,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; } /* @@ -1359,15 +1358,25 @@ static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc) return file->f_op->mmap_prepare(desc); } +int mmap_prepare_validate(const struct vm_area_desc *orig_desc, + const struct vm_area_desc *desc); + static inline int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma) { + struct vm_area_desc orig_desc; int err; + /* Derive state prior to mmap_prepare hook. */ + compat_set_desc_from_vma(&orig_desc, desc->file, vma); /* Perform any preparatory tasks for mmap action. */ err = mmap_action_prepare(desc); if (err) return err; + /* Check the caller did nothing crazy. */ + err = mmap_prepare_validate(&orig_desc, desc); + if (err) + return err; /* Update the VMA from the descriptor. */ compat_set_vma_from_desc(vma, desc); /* Complete any specified mmap actions. */ @@ -1647,3 +1656,74 @@ static inline bool file_is_dev_zero(const struct file *file) { return file && file->f_op == &zero_fops; } + +static inline bool vma_flags_is_mm_managed(const vma_flags_t *flags) +{ + return !vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); +} + +static inline bool vma_is_mm_managed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_managed(&vma->flags); +} + +static inline bool vma_flags_is_fixed_mapping(const vma_flags_t *flags) +{ + /* + * VMA_PFNMAP_BIT should imply VMA_DONTEXPAND_BIT, but some callers set + * only the former. + */ + return vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT); +} + +static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_fixed_mapping(&vma->flags); +} + +static inline bool vma_flags_is_hugetlb(const vma_flags_t *flags) +{ + return IS_ENABLED(CONFIG_HUGETLB_PAGE) && + vma_flags_test(flags, VMA_HUGETLB_BIT); +} + +static inline bool vma_flags_is_mm_backed(const vma_flags_t *flags) +{ + /* hugetlb is a fixed mapping, but core mm owns it entirely. */ + if (vma_flags_is_hugetlb(flags)) + return true; + /* + * Mappings not managed by core mm are populated by their owner. + * + * Fixed mappings may be mm-managed, but their contents are established + * by a custom ->fault handler, so core mm cannot assume ordinary fault + * semantics over them. + */ + return vma_flags_is_mm_managed(flags) && !vma_flags_is_fixed_mapping(flags); +} + +static inline bool vma_is_mm_backed(const struct vm_area_struct *vma) +{ + return vma_flags_is_mm_backed(&vma->flags); +} + +static inline bool vma_flags_can_merge(const vma_flags_t *flags) +{ + /* + * VMA merging assumes that a VMA's flags and fields completely describe + * its state, which is only the case for mm-backed mappings. + * + * Mappings which are not mm-managed may have established state upon + * mapping not embodied in any attribute of the VMA. + * + * Additionally, private (CoW) PFN maps encode the source PFN of the + * range in vma->vm_pgoff, which may otherwise cause spurious merges. + */ + if (!vma_flags_is_mm_backed(flags)) + return false; + /* + * hugetlb mappings contain per-VMA state, so their flags and fields do + * not completely describe them. + */ + return !vma_flags_is_hugetlb(flags); +} diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index d6136e19a8af..f786a585caf0 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -193,7 +193,7 @@ static inline bool mapping_can_writeback(struct address_space *mapping) return true; } -static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma) +static inline bool vma_is_hugetlb(struct vm_area_struct *vma) { return false; } @@ -267,6 +267,18 @@ static inline void mapping_rmap_tree_remove(struct vm_area_struct *vma, { } +static inline void mapping_rmap_tree_pre_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ +} + +static inline void mapping_rmap_tree_post_update(struct vm_area_struct *vma, + struct address_space *mapping, + bool pgoff_unchanged) +{ +} + static inline void flush_dcache_mmap_unlock(struct address_space *mapping) { } @@ -281,6 +293,16 @@ static inline void anon_rmap_tree_remove(struct anon_vma_chain *avc, { } +static inline void anon_rmap_tree_pre_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ +} + +static inline void anon_rmap_tree_post_update_vma(struct vm_area_struct *vma, + bool anon_pgoff_unchanged) +{ +} + static inline void uprobe_mmap(struct vm_area_struct *vma) { } @@ -302,6 +324,10 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) { } +static inline void vma_assert_stabilised(const struct vm_area_struct *vma) +{ +} + static inline void ksm_add_vma(struct vm_area_struct *vma) { } @@ -329,6 +355,10 @@ static inline void vma_close(struct vm_area_struct *vma) { } +static inline void zap_vma(struct vm_area_struct *vma) +{ +} + static inline int mmap_file(struct file *file, struct vm_area_struct *vma) { return 0; @@ -356,6 +386,11 @@ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, return true; } +static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma) +{ + return false; +} + static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, struct anon_vma_name *anon_name2) { diff --git a/tools/testing/vma/shared.c b/tools/testing/vma/shared.c index 8c4826499f40..c1e0c2280101 100644 --- a/tools/testing/vma/shared.c +++ b/tools/testing/vma/shared.c @@ -65,7 +65,7 @@ struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, * begun. Linking to the tree will have caused this to be incremented, * which means we will get a false positive otherwise. */ - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; return vma; } @@ -97,13 +97,13 @@ int cleanup_mm(struct mm_struct *mm, struct vma_iterator *vmi) bool vma_write_started(struct vm_area_struct *vma) { - int seq = vma->vm_lock_seq; + unsigned int seq = vma->vm_lock_seq; /* We reset after each check. */ - vma->vm_lock_seq = UINT_MAX; + vma->vm_lock_seq = 0; /* The vma_start_write() stub simply increments this value. */ - return seq > -1; + return seq > 0; } void __vma_set_dummy_anon_vma(struct vm_area_struct *vma, diff --git a/tools/testing/vma/tests/merge.c b/tools/testing/vma/tests/merge.c index acaab282939c..b26f1a66a170 100644 --- a/tools/testing/vma/tests/merge.c +++ b/tools/testing/vma/tests/merge.c @@ -496,17 +496,11 @@ static bool test_vma_merge_special_flags(void) .mm = &mm, .vmi = &vmi, }; - vma_flag_t special_flags[] = { VMA_IO_BIT, VMA_DONTEXPAND_BIT, + vma_flag_t special_flags[] = { VMA_DONTEXPAND_BIT, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT }; - vma_flags_t all_special_flags = EMPTY_VMA_FLAGS; int i; struct vm_area_struct *vma_left, *vma; - /* Make sure there aren't new VM_SPECIAL flags. */ - for (i = 0; i < ARRAY_SIZE(special_flags); i++) - vma_flags_set(&all_special_flags, special_flags[i]); - ASSERT_FLAGS_SAME_MASK(&all_special_flags, VMA_SPECIAL_FLAGS); - /* * 01234 * AAA @@ -520,7 +514,7 @@ static bool test_vma_merge_special_flags(void) * 01234 * AAA* * - * This should merge if not for the VM_SPECIAL flag. + * This should merge if not for the 'special' flag. */ vmg_set_range(&vmg, 0x3000, 0x4000, 3, vma_flags); for (i = 0; i < ARRAY_SIZE(special_flags); i++) { |