linux-toradex.git/arch/x86/include/asm/tlbflush.h, branch v6.11

Merge tag 'x86_shstk_for_6.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2023-08-31T19:20:12+00:00

Pull x86 shadow stack support from Dave Hansen:
 "This is the long awaited x86 shadow stack support, part of Intel's
  Control-flow Enforcement Technology (CET).

  CET consists of two related security features: shadow stacks and
  indirect branch tracking. This series implements just the shadow stack
  part of this feature, and just for userspace.

  The main use case for shadow stack is providing protection against
  return oriented programming attacks. It works by maintaining a
  secondary (shadow) stack using a special memory type that has
  protections against modification. When executing a CALL instruction,
  the processor pushes the return address to both the normal stack and
  to the special permission shadow stack. Upon RET, the processor pops
  the shadow stack copy and compares it to the normal stack copy.

  For more information, refer to the links below for the earlier
  versions of this patch set"

Link: https://lore.kernel.org/lkml/20220130211838.8382-1-rick.p.edgecombe@intel.com/
Link: https://lore.kernel.org/lkml/20230613001108.3040476-1-rick.p.edgecombe@intel.com/

* tag 'x86_shstk_for_6.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (47 commits)
  x86/shstk: Change order of __user in type
  x86/ibt: Convert IBT selftest to asm
  x86/shstk: Don't retry vm_munmap() on -EINTR
  x86/kbuild: Fix Documentation/ reference
  x86/shstk: Move arch detail comment out of core mm
  x86/shstk: Add ARCH_SHSTK_STATUS
  x86/shstk: Add ARCH_SHSTK_UNLOCK
  x86: Add PTRACE interface for shadow stack
  selftests/x86: Add shadow stack test
  x86/cpufeatures: Enable CET CR4 bit for shadow stack
  x86/shstk: Wire in shadow stack interface
  x86: Expose thread features in /proc/$PID/status
  x86/shstk: Support WRSS for userspace
  x86/shstk: Introduce map_shadow_stack syscall
  x86/shstk: Check that signal frame is shadow stack mem
  x86/shstk: Check that SSP is aligned on sigreturn
  x86/shstk: Handle signals for shadow stack
  x86/shstk: Introduce routines modifying shstk
  x86/shstk: Handle thread shadow stack
  x86/shstk: Add user-mode shadow stack support
  ...

mmu_notifiers: rename invalidate_range notifier

2023-08-18T17:12:41+00:00

There are two main use cases for mmu notifiers.  One is by KVM which uses
mmu_notifier_invalidate_range_start()/end() to manage a software TLB.

The other is to manage hardware TLBs which need to use the
invalidate_range() callback because HW can establish new TLB entries at
any time.  Hence using start/end() can lead to memory corruption as these
callbacks happen too soon/late during page unmap.

mmu notifier users should therefore either use the start()/end() callbacks
or the invalidate_range() callbacks.  To make this usage clearer rename
the invalidate_range() callback to arch_invalidate_secondary_tlbs() and
update documention.

Link: https://lkml.kernel.org/r/6f77248cd25545c8020a54b4e567e8b72be4dca1.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Suggested-by: Jason Gunthorpe 
Acked-by: Catalin Marinas 
Reviewed-by: Jason Gunthorpe 
Cc: Andrew Donnellan 
Cc: Chaitanya Kumar Borah 
Cc: Frederic Barrat 
Cc: Jason Gunthorpe 
Cc: John Hubbard 
Cc: Kevin Tian 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Nicolin Chen 
Cc: Robin Murphy 
Cc: Sean Christopherson 
Cc: SeongJae Park 
Cc: Tvrtko Ursulin 
Cc: Will Deacon 
Cc: Zhi Wang 
Signed-off-by: Andrew Morton

mmu_notifiers: call invalidate_range() when invalidating TLBs

2023-08-18T17:12:41+00:00

The invalidate_range() is going to become an architecture specific mmu
notifier used to keep the TLB of secondary MMUs such as an IOMMU in sync
with the CPU page tables.  Currently it is called from separate code paths
to the main CPU TLB invalidations.  This can lead to a secondary TLB not
getting invalidated when required and makes it hard to reason about when
exactly the secondary TLB is invalidated.

To fix this move the notifier call to the architecture specific TLB
maintenance functions for architectures that have secondary MMUs requiring
explicit software invalidations.

This fixes a SMMU bug on ARM64.  On ARM64 PTE permission upgrades require
a TLB invalidation.  This invalidation is done by the architecture
specific ptep_set_access_flags() which calls flush_tlb_page() if required.
However this doesn't call the notifier resulting in infinite faults being
generated by devices using the SMMU if it has previously cached a
read-only PTE in it's TLB.

Moving the invalidations into the TLB invalidation functions ensures all
invalidations happen at the same time as the CPU invalidation.  The
architecture specific flush_tlb_all() routines do not call the notifier as
none of the IOMMUs require this.

Link: https://lkml.kernel.org/r/0287ae32d91393a582897d6c4db6f7456b1001f2.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple 
Suggested-by: Jason Gunthorpe 
Tested-by: SeongJae Park 
Acked-by: Catalin Marinas 
Reviewed-by: Jason Gunthorpe 
Tested-by: Luis Chamberlain 
Cc: Andrew Donnellan 
Cc: Chaitanya Kumar Borah 
Cc: Frederic Barrat 
Cc: John Hubbard 
Cc: Kevin Tian 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Nicolin Chen 
Cc: Robin Murphy 
Cc: Sean Christopherson 
Cc: Tvrtko Ursulin 
Cc: Will Deacon 
Cc: Zhi Wang 
Signed-off-by: Andrew Morton

mm/tlbbatch: introduce arch_flush_tlb_batched_pending()

2023-08-18T17:12:37+00:00

Currently we'll flush the mm in flush_tlb_batched_pending() to avoid race
between reclaim unmaps pages by batched TLB flush and mprotect/munmap/etc.
Other architectures like arm64 may only need a synchronization
barrier(dsb) here rather than a full mm flush.  So add
arch_flush_tlb_batched_pending() to allow an arch-specific implementation
here.  This intends no functional changes on x86 since still a full mm
flush for x86.

Link: https://lkml.kernel.org/r/20230717131004.12662-4-yangyicong@huawei.com
Signed-off-by: Yicong Yang 
Reviewed-by: Catalin Marinas 
Cc: Anshuman Khandual 
Cc: Anshuman Khandual 
Cc: Arnd Bergmann 
Cc: Barry Song 
Cc: Barry Song 
Cc: Darren Hart 
Cc: Jonathan Cameron 
Cc: Jonathan Corbet 
Cc: Kefeng Wang 
Cc: lipeifeng 
Cc: Mark Rutland 
Cc: Mel Gorman 
Cc: Nadav Amit 
Cc: Peter Zijlstra 
Cc: Punit Agrawal 
Cc: Ryan Roberts 
Cc: Steven Miao 
Cc: Will Deacon 
Cc: Xin Hao 
Cc: Zeng Tao 
Signed-off-by: Andrew Morton

mm/tlbbatch: rename and extend some functions

2023-08-18T17:12:36+00:00

This patch does some preparation works to extend batched TLB flush to
arm64. Including:
- Extend set_tlb_ubc_flush_pending() and arch_tlbbatch_add_mm()
  to accept an additional argument for address, architectures
  like arm64 may need this for tlbi.
- Rename arch_tlbbatch_add_mm() to arch_tlbbatch_add_pending()
  to match its current function since we don't need to handle
  mm on architectures like arm64 and add_mm is not proper,
  add_pending will make sense to both as on x86 we're pending the
  TLB flush operations while on arm64 we're pending the synchronize
  operations.

This intends no functional changes on x86.

Link: https://lkml.kernel.org/r/20230717131004.12662-3-yangyicong@huawei.com
Tested-by: Yicong Yang 
Tested-by: Xin Hao 
Tested-by: Punit Agrawal 
Signed-off-by: Barry Song 
Signed-off-by: Yicong Yang 
Reviewed-by: Kefeng Wang 
Reviewed-by: Xin Hao 
Reviewed-by: Anshuman Khandual 
Reviewed-by: Catalin Marinas 
Cc: Jonathan Corbet 
Cc: Nadav Amit 
Cc: Mel Gorman 
Cc: Anshuman Khandual 
Cc: Arnd Bergmann 
Cc: Barry Song 
Cc: Darren Hart 
Cc: Jonathan Cameron 
Cc: lipeifeng 
Cc: Mark Rutland 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Steven Miao 
Cc: Will Deacon 
Cc: Zeng Tao 
Signed-off-by: Andrew Morton

mm/tlbbatch: introduce arch_tlbbatch_should_defer()

2023-08-18T17:12:36+00:00

Patch series "arm64: support batched/deferred tlb shootdown during page
reclamation/migration", v11.

Though ARM64 has the hardware to do tlb shootdown, the hardware
broadcasting is not free.  A simplest micro benchmark shows even on
snapdragon 888 with only 8 cores, the overhead for ptep_clear_flush is
huge even for paging out one page mapped by only one process: 5.36% a.out
[kernel.kallsyms] [k] ptep_clear_flush

While pages are mapped by multiple processes or HW has more CPUs, the cost
should become even higher due to the bad scalability of tlb shootdown. 
The same benchmark can result in 16.99% CPU consumption on ARM64 server
with around 100 cores according to the test on patch 4/4.

This patchset leverages the existing BATCHED_UNMAP_TLB_FLUSH by
1. only send tlbi instructions in the first stage -
	arch_tlbbatch_add_mm()
2. wait for the completion of tlbi by dsb while doing tlbbatch
	sync in arch_tlbbatch_flush()

Testing on snapdragon shows the overhead of ptep_clear_flush is removed by
the patchset.  The micro benchmark becomes 5% faster even for one page
mapped by single process on snapdragon 888.

Since BATCHED_UNMAP_TLB_FLUSH is implemented only on x86, the patchset
does some renaming/extension for the current implementation first (Patch
1-3), then add the support on arm64 (Patch 4).
		

This patch (of 4):

The entire scheme of deferred TLB flush in reclaim path rests on the fact
that the cost to refill TLB entries is less than flushing out individual
entries by sending IPI to remote CPUs.  But architecture can have
different ways to evaluate that.  Hence apart from checking
TTU_BATCH_FLUSH in the TTU flags, rest of the decision should be
architecture specific.

[yangyicong@hisilicon.com: rebase and fix incorrect return value type]
Link: https://lkml.kernel.org/r/20230717131004.12662-1-yangyicong@huawei.com
Link: https://lkml.kernel.org/r/20230717131004.12662-2-yangyicong@huawei.com
Signed-off-by: Anshuman Khandual 
[https://lore.kernel.org/linuxppc-dev/20171101101735.2318-2-khandual@linux.vnet.ibm.com/]
Signed-off-by: Yicong Yang 
Reviewed-by: Kefeng Wang 
Reviewed-by: Anshuman Khandual 
Reviewed-by: Barry Song 
Reviewed-by: Xin Hao 
Tested-by: Punit Agrawal 
Reviewed-by: Catalin Marinas 
Cc: Arnd Bergmann 
Cc: Darren Hart 
Cc: Jonathan Cameron 
Cc: Jonathan Corbet 
Cc: lipeifeng 
Cc: Mark Rutland 
Cc: Peter Zijlstra 
Cc: Ryan Roberts 
Cc: Steven Miao 
Cc: Will Deacon 
Cc: Zeng Tao 
Cc: Barry Song 
Cc: Mel Gorman 
Cc: Nadav Amit 
Signed-off-by: Andrew Morton

x86/mm: Introduce _PAGE_SAVED_DIRTY

2023-07-11T21:12:19+00:00

Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.

The reason it's lightly used is that Dirty=1 is normally set by HW
_before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.

So that leaves Write=0,Dirty=1 PTEs created in software. To avoid
inadvertently created shadow stack memory, in places where Linux normally
creates Write=0,Dirty=1, it can use the software-defined _PAGE_SAVED_DIRTY
in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs
to create Write=0,Dirty=1, it instead creates Write=0,SavedDirty=1 except
for shadow stack, which is Write=0,Dirty=1.

There are six bits left available to software in the 64-bit PTE after
consuming a bit for _PAGE_SAVED_DIRTY. For 32 bit, the same bit as
_PAGE_BIT_UFFD_WP is used, since user fault fd is not supported on 32
bit. This leaves one unused software bit on 32 bit (_PAGE_BIT_SOFT_DIRTY,
as this is also not supported on 32 bit).

Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to
actually begin creating _PAGE_SAVED_DIRTY PTEs will follow once other
pieces are in place.

Since this SavedDirty shifting is done for all x86 CPUs, this leaves
the possibility for the hardware oddity to still create Write=0,Dirty=1
PTEs in rare cases. Since these CPUs also don't support shadow stack, this
will be harmless as it was before the introduction of SavedDirty.

Implement the shifting logic to be branchless. Embed the logic of whether
to do the shifting (including checking the Write bits) so that it can be
called by future callers that would otherwise need additional branching
logic. This efficiency allows the logic of when to do the shifting to be
centralized, making the code easier to reason about.

Co-developed-by: Yu-cheng Yu 
Signed-off-by: Yu-cheng Yu 
Signed-off-by: Rick Edgecombe 
Signed-off-by: Dave Hansen 
Tested-by: Pengfei Xu 
Tested-by: John Allen 
Tested-by: Kees Cook 
Link: https://lore.kernel.org/all/20230613001108.3040476-11-rick.p.edgecombe%40intel.com

x86/mm: Remove unused current_untag_mask()

2023-06-16T08:50:16+00:00

e0bddc19ba95 ("x86/mm: Reduce untagged_addr() overhead for systems without LAM")

removed its only usage site so drop it.

Move the tlbstate_untag_mask up in the header and drop the ugly
ifdeffery as the unused declaration should be properly discarded.

Signed-off-by: Borislav Petkov (AMD) 
Acked-by: Kirill A. Shutemov 
Link: https://lore.kernel.org/r/20230614174148.5439-1-bp@alien8.de

x86/uaccess: Provide untagged_addr() and remove tags before address check

2023-03-16T20:08:39+00:00

untagged_addr() is a helper used by the core-mm to strip tag bits and
get the address to the canonical shape based on rules of the current
thread. It only handles userspace addresses.

The untagging mask is stored in per-CPU variable and set on context
switching to the task.

The tags must not be included into check whether it's okay to access the
userspace address. Strip tags in access_ok().

Signed-off-by: Kirill A. Shutemov 
Signed-off-by: Dave Hansen 
Acked-by: Peter Zijlstra (Intel) 
Tested-by: Alexander Potapenko 
Link: https://lore.kernel.org/all/20230312112612.31869-7-kirill.shutemov%40linux.intel.com

x86/mm: Handle LAM on context switch

2023-03-16T20:08:39+00:00

Linear Address Masking mode for userspace pointers encoded in CR3 bits.
The mode is selected per-process and stored in mm_context_t.

switch_mm_irqs_off() now respects selected LAM mode and constructs CR3
accordingly.

The active LAM mode gets recorded in the tlb_state.

Signed-off-by: Kirill A. Shutemov 
Signed-off-by: Dave Hansen 
Acked-by: Peter Zijlstra (Intel) 
Tested-by: Alexander Potapenko 
Link: https://lore.kernel.org/all/20230312112612.31869-5-kirill.shutemov%40linux.intel.com