linux-toradex.git/include/asm-generic/tlb.h, branch v5.12

tlb: mmu_gather: Introduce tlb_gather_mmu_fullmm()

2021-01-29T19:02:29+00:00

Passing the range '0, -1' to tlb_gather_mmu() sets the 'fullmm' flag,
which indicates that the mm_struct being operated on is going away. In
this case, some architectures (such as arm64) can elide TLB invalidation
by ensuring that the TLB tag (ASID) associated with this mm is not
immediately reclaimed. Although this behaviour is documented in
asm-generic/tlb.h, it's subtle and easily missed.

Introduce tlb_gather_mmu_fullmm() to make it clearer that this is for the
entire mm and WARN() if tlb_gather_mmu() is called with the 'fullmm'
address range.

Signed-off-by: Will Deacon 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Yu Zhao 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Linus Torvalds 
Link: https://lkml.kernel.org/r/20210127235347.1402-4-will@kernel.org

mm: remove unneeded includes of

2020-08-07T18:33:26+00:00

Patch series "mm: cleanup usage of "

Most architectures have very similar versions of pXd_alloc_one() and
pXd_free_one() for intermediate levels of page table.  These patches add
generic versions of these functions in  and enable
use of the generic functions where appropriate.

In addition, functions declared and defined in  headers are
used mostly by core mm and early mm initialization in arch and there is no
actual reason to have the  included all over the place.
The first patch in this series removes unneeded includes of


In the end it didn't work out as neatly as I hoped and moving
pXd_alloc_track() definitions to  would require
unnecessary changes to arches that have custom page table allocations, so
I've decided to move lib/ioremap.c to mm/ and make pgalloc-track.h local
to mm/.

This patch (of 8):

In most cases  header is required only for allocations of
page table memory.  Most of the .c files that include that header do not
use symbols declared in  and do not require that header.

As for the other header files that used to include , it is
possible to move that include into the .c file that actually uses symbols
from  and drop the include from the header file.

The process was somewhat automated using

	sed -i -E '/[<"]asm\/pgalloc\.h/d' \
                $(grep -L -w -f /tmp/xx \
                        $(git grep -E -l '[<"]asm/pgalloc\.h'))

where /tmp/xx contains all the symbols defined in
arch/*/include/asm/pgalloc.h.

[rppt@linux.ibm.com: fix powerpc warning]

Signed-off-by: Mike Rapoport 
Signed-off-by: Andrew Morton 
Reviewed-by: Pekka Enberg 
Acked-by: Geert Uytterhoeven 	[m68k]
Cc: Abdul Haleem 
Cc: Andy Lutomirski 
Cc: Arnd Bergmann 
Cc: Christophe Leroy 
Cc: Joerg Roedel 
Cc: Max Filippov 
Cc: Peter Zijlstra 
Cc: Satheesh Rajendran 
Cc: Stafford Horne 
Cc: Stephen Rothwell 
Cc: Steven Rostedt 
Cc: Joerg Roedel 
Cc: Matthew Wilcox 
Link: http://lkml.kernel.org/r/20200627143453.31835-1-rppt@kernel.org
Link: http://lkml.kernel.org/r/20200627143453.31835-2-rppt@kernel.org
Signed-off-by: Linus Torvalds

tlb: mmu_gather: add tlb_flush_*_range APIs

2020-07-07T10:23:46+00:00

tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and
tlb->end, then set corresponding cleared_*.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Zhenyu Ye 
Acked-by: Catalin Marinas 
Link: https://lore.kernel.org/r/20200625080314.230-5-yezhenyu2@huawei.com
Signed-off-by: Catalin Marinas

mm/vma: replace all remaining open encodings with is_vm_hugetlb_page()

2020-04-07T17:43:37+00:00

This replaces all remaining open encodings with is_vm_hugetlb_page().

Signed-off-by: Anshuman Khandual 
Signed-off-by: Andrew Morton 
Acked-by: Vlastimil Babka 
Cc: Paul Mackerras 
Cc: Benjamin Herrenschmidt 
Cc: Michael Ellerman 
Cc: Alexander Viro 
Cc: Will Deacon 
Cc: "Aneesh Kumar K.V" 
Cc: Nick Piggin 
Cc: Peter Zijlstra 
Cc: Arnd Bergmann 
Cc: Ingo Molnar 
Cc: Arnaldo Carvalho de Melo 
Cc: Andy Lutomirski 
Cc: Dave Hansen 
Cc: Geert Uytterhoeven 
Cc: Guo Ren 
Cc: Mel Gorman 
Cc: Paul Burton 
Cc: Paul Mackerras 
Cc: Ralf Baechle 
Cc: Rich Felker 
Cc: Steven Rostedt 
Cc: Thomas Gleixner 
Cc: Yoshinori Sato 
Link: http://lkml.kernel.org/r/1582520593-30704-4-git-send-email-anshuman.khandual@arm.com
Signed-off-by: Linus Torvalds

asm-generic/tlb: provide MMU_GATHER_TABLE_FREE

2020-02-04T03:05:26+00:00

As described in the comment, the correct order for freeing pages is:

 1) unhook page
 2) TLB invalidate page
 3) free page

This order equally applies to page directories.

Currently there are two correct options:

 - use tlb_remove_page(), when all page directores are full pages and
   there are no futher contraints placed by things like software
   walkers (HAVE_FAST_GUP).

 - use MMU_GATHER_RCU_TABLE_FREE and tlb_remove_table() when the
   architecture does not do IPI based TLB invalidate and has
   HAVE_FAST_GUP (or software TLB fill).

This however leaves architectures that don't have page based directories
but don't need RCU in a bind.  For those, provide MMU_GATHER_TABLE_FREE,
which provides the independent batching for directories without the
additional RCU freeing.

Link: http://lkml.kernel.org/r/20200116064531.483522-10-aneesh.kumar@linux.ibm.com
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER

2020-02-04T03:05:26+00:00

Towards a more consistent naming scheme.

Link: http://lkml.kernel.org/r/20200116064531.483522-9-aneesh.kumar@linux.ibm.com
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-generic/tlb: rename HAVE_MMU_GATHER_PAGE_SIZE

2020-02-04T03:05:26+00:00

Towards a more consistent naming scheme.

Link: http://lkml.kernel.org/r/20200116064531.483522-8-aneesh.kumar@linux.ibm.com
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-generic/tlb: rename HAVE_RCU_TABLE_FREE

2020-02-04T03:05:26+00:00

Towards a more consistent naming scheme.

[akpm@linux-foundation.org: fix sparc64 Kconfig]
Link: http://lkml.kernel.org/r/20200116064531.483522-7-aneesh.kumar@linux.ibm.com
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-gemeric/tlb: remove stray function declarations

2020-02-04T03:05:26+00:00

We removed the actual functions a while ago.

Link: http://lkml.kernel.org/r/20200116064531.483522-5-aneesh.kumar@linux.ibm.com
Fixes: 1808d65b55e4 ("asm-generic/tlb: Remove arch_tlb*_mmu()")
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

asm-generic/tlb: avoid potential double flush

2020-02-04T03:05:26+00:00

Aneesh reported that:

	tlb_flush_mmu()
	  tlb_flush_mmu_tlbonly()
	    tlb_flush()			<-- #1
	  tlb_flush_mmu_free()
	    tlb_table_flush()
	      tlb_table_invalidate()
		tlb_flush_mmu_tlbonly()
		  tlb_flush()		<-- #2

does two TLBIs when tlb->fullmm, because __tlb_reset_range() will not
clear tlb->end in that case.

Observe that any caller to __tlb_adjust_range() also sets at least one of
the tlb->freed_tables || tlb->cleared_p* bits, and those are
unconditionally cleared by __tlb_reset_range().

Change the condition for actually issuing TLBI to having one of those bits
set, as opposed to having tlb->end != 0.

Link: http://lkml.kernel.org/r/20200116064531.483522-4-aneesh.kumar@linux.ibm.com
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Aneesh Kumar K.V 
Reported-by: "Aneesh Kumar K.V" 
Cc: Michael Ellerman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds