linux-toradex.git/arch/mips/include/asm/pgtable.h, branch v4.4.106

MIPS: Sync icache & dcache in set_pte_at

2016-06-08T01:14:31+00:00

commit 37d22a0d798b5c938b277d32cfd86dc231381342 upstream.

It's possible for pages to become visible prior to update_mmu_cache
running if a thread within the same address space preempts the current
thread or runs simultaneously on another CPU. That is, the following
scenario is possible:

    CPU0                            CPU1

    write to page
    flush_dcache_page
    flush_icache_page
    set_pte_at
                                    map page
    update_mmu_cache

If CPU1 maps the page in between CPU0's set_pte_at, which marks it valid
& visible, and update_mmu_cache where the dcache flush occurs then CPU1s
icache will fill from stale data (unless it fills from the dcache, in
which case all is good, but most MIPS CPUs don't have this property).
Commit 4d46a67a3eb8 ("MIPS: Fix race condition in lazy cache flushing.")
attempted to fix that by performing the dcache flush in
flush_icache_page such that it occurs before the set_pte_at call makes
the page visible. However it has the problem that not all code that
writes to pages exposed to userland call flush_icache_page. There are
many callers of set_pte_at under mm/ and only 2 of them do call
flush_icache_page. Thus the race window between a page becoming visible
& being coherent between the icache & dcache remains open in some cases.

To illustrate some of the cases, a WARN was added to __update_cache with
this patch applied that triggered in cases where a page about to be
flushed from the dcache was not the last page provided to
flush_icache_page. That is, backtraces were obtained for cases in which
the race window is left open without this patch. The 2 standout examples
follow.

When forking a process:

[   15.271842] [<80417630>] __update_cache+0xcc/0x188
[   15.277274] [<80530394>] copy_page_range+0x56c/0x6ac
[   15.282861] [<8042936c>] copy_process.part.54+0xd40/0x17ac
[   15.289028] [<80429f80>] do_fork+0xe4/0x420
[   15.293747] [<80413808>] handle_sys+0x128/0x14c

When exec'ing an ELF binary:

[   14.445964] [<80417630>] __update_cache+0xcc/0x188
[   14.451369] [<80538d88>] move_page_tables+0x414/0x498
[   14.457075] [<8055d848>] setup_arg_pages+0x220/0x318
[   14.462685] [<805b0f38>] load_elf_binary+0x530/0x12a0
[   14.468374] [<8055ec3c>] search_binary_handler+0xbc/0x214
[   14.474444] [<8055f6c0>] do_execveat_common+0x43c/0x67c
[   14.480324] [<8055f938>] do_execve+0x38/0x44
[   14.485137] [<80413808>] handle_sys+0x128/0x14c

These code paths write into a page, call flush_dcache_page then call
set_pte_at without flush_icache_page inbetween. The end result is that
the icache can become corrupted & userland processes may execute
unexpected or invalid code, typically resulting in a reserved
instruction exception, a trap or a segfault.

Fix this race condition fully by performing any cache maintenance
required to keep the icache & dcache in sync in set_pte_at, before the
page is made valid. This has the added bonus of ensuring the cache
maintenance always happens in one location, rather than being duplicated
in flush_icache_page & update_mmu_cache. It also matches the way other
architectures solve the same problem (see arm, ia64 & powerpc).

Signed-off-by: Paul Burton 
Reported-by: Ionela Voinescu 
Cc: Lars Persson 
Fixes: 4d46a67a3eb8 ("MIPS: Fix race condition in lazy cache flushing.")
Cc: Steven J. Hill 
Cc: David Daney 
Cc: Huacai Chen 
Cc: Aneesh Kumar K.V 
Cc: Andrew Morton 
Cc: Jerome Marchand 
Cc: Kirill A. Shutemov 
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12722/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

MIPS: Fix some missing CONFIG_CPU_MIPSR6 #ifdefs

2016-03-03T23:07:17+00:00

commit 4f33f6c522948fffc345261896042b58dea23754 upstream.

Commit be0c37c985eddc4 (MIPS: Rearrange PTE bits into fixed positions.)
defines fixed PTE bits for MIPS R2. Then, commit d7b631419b3d230a4d383
(MIPS: pgtable-bits: Fix XPA damage to R6 definitions.) adds the MIPS
R6 definitions in the same way as MIPS R2. But some R6 #ifdefs in the
later commit are missing, so in this patch I fix that.

Signed-off-by: Huacai Chen 
Cc: Aurelien Jarno 
Cc: Steven J. Hill 
Cc: Fuxin Zhang 
Cc: Zhangjin Wu 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/12164/
Signed-off-by: Ralf Baechle 
Signed-off-by: Greg Kroah-Hartman

MIPS: Fix definition of pgprot_writecombine()

2015-09-03T10:08:00+00:00

If pgprot_writecombine is not #defined, asm-generic/pgtable.h will try
to provide a default implementation by #defining it to pgprot_noncached.
However our implementation is an inline function rather than a #define,
so it was never actually used because of the #define in generic code.

Add "#define pgprot_writecombine pgprot_writecombine" to prevent generic
code from re-defining it.

Signed-off-by: Alex Smith 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10767/
Signed-off-by: Ralf Baechle

MIPS: Make set_pte() SMP safe.

2015-08-05T09:11:10+00:00

On MIPS the GLOBAL bit of the PTE must have the same value in any
aligned pair of PTEs.  These pairs of PTEs are referred to as
"buddies".  In a SMP system is is possible for two CPUs to be calling
set_pte() on adjacent PTEs at the same time.  There is a race between
setting the PTE and a different CPU setting the GLOBAL bit in its
buddy PTE.

This race can be observed when multiple CPUs are executing
vmap()/vfree() at the same time.

Make setting the buddy PTE's GLOBAL bit an atomic operation to close
the race condition.

The case of CONFIG_64BIT_PHYS_ADDR && CONFIG_CPU_MIPS32 is *not*
handled.

Signed-off-by: David Daney 
Cc: 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/10835/
Signed-off-by: Ralf Baechle

mm: clarify that the function operates on hugepage pte

2015-06-25T00:49:44+00:00

We have confusing functions to clear pmd, pmd_clear_* and pmd_clear.  Add
_huge_ to pmdp_clear functions so that we are clear that they operate on
hugepage pte.

We don't bother about other functions like pmdp_set_wrprotect,
pmdp_clear_flush_young, because they operate on PTE bits and hence
indicate they are operating on hugepage ptes

Signed-off-by: Aneesh Kumar K.V 
Acked-by: Kirill A. Shutemov 
Cc: Benjamin Herrenschmidt 
Cc: Paul Mackerras 
Cc: Michael Ellerman 
Cc: Andrea Arcangeli 
Cc: Martin Schwidefsky 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

Revert "MIPS: Remove race window in page fault handling"

2015-03-25T12:48:00+00:00

Revert commit 2a4a8b1e5d9d ("MIPS: Remove race window in page fault
handling") because it increased the number of flushed dcache pages and
became a performance problem for some workloads.

Signed-off-by: Lars Persson 
Cc: linux-mips@linux-mips.org
Cc: paul.burton@imgtec.com
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/9345/
Signed-off-by: Ralf Baechle

MIPS: Add support for XPA.

2015-03-19T16:39:49+00:00

Add support for extended physical addressing (XPA) so that
32-bit platforms can access equal to or greater than 40 bits
of physical addresses.

NOTE:
      1) XPA and EVA are not the same and cannot be used
         simultaneously.
      2) If you configure your kernel for XPA, the PTEs
         and all address sizes become 64-bit.
      3) Your platform MUST have working HIGHMEM support.

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9355/
Signed-off-by: Ralf Baechle

MIPS: Rearrange PTE bits into fixed positions.

2015-03-18T15:19:35+00:00

This patch rearranges the PTE bits into fixed positions for R2
and later cores. In the past, the TLB handling code did runtime
checking of RI/XI and adjusted the shifts and rotates in order
to fit the largest PFN value into the PTE. The checking now
occurs when building the TLB handler, thus eliminating those
checks. These new arrangements also define the largest possible
PFN value that can fit in the PTE. HUGE page support is only
available for 64-bit cores. Layouts of the PTE bits are now:

   64-bit, R1 or earlier:     CCC D V G [S H] M A W R P
   32-bit, R1 or earler:      CCC D V G M A W R P
   64-bit, R2 or later:       CCC D V G RI/R XI [S H] M A W P
   32-bit, R2 or later:       CCC D V G RI/R XI M A W P

[ralf@linux-mips.org: Fix another build error *rant* *rant*]

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9353/
Signed-off-by: Ralf Baechle

Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus

2015-02-22T03:41:38+00:00

Pull MIPS updates from Ralf Baechle:
 "This is the main pull request for MIPS:

   - a number of fixes that didn't make the 3.19 release.

   - a number of cleanups.

   - preliminary support for Cavium's Octeon 3 SOCs which feature up to
     48 MIPS64 R3 cores with FPU and hardware virtualization.

   - support for MIPS R6 processors.

     Revision 6 of the MIPS architecture is a major revision of the MIPS
     architecture which does away with many of original sins of the
     architecture such as branch delay slots.  This and other changes in
     R6 require major changes throughout the entire MIPS core
     architecture code and make up for the lion share of this pull
     request.

   - finally some preparatory work for eXtendend Physical Address
     support, which allows support of up to 40 bit of physical address
     space on 32 bit processors"

     [ Ahh, MIPS can't leave the PAE brain damage alone.  It's like
       every CPU architect has to make that mistake, but pee in the snow
       by changing the TLA.  But whether it's called PAE, LPAE or XPA,
       it's horrid crud   - Linus ]

* 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus: (114 commits)
  MIPS: sead3: Corrected get_c0_perfcount_int
  MIPS: mm: Remove dead macro definitions
  MIPS: OCTEON: irq: add CIB and other fixes
  MIPS: OCTEON: Don't do acknowledge operations for level triggered irqs.
  MIPS: OCTEON: More OCTEONIII support
  MIPS: OCTEON: Remove setting of processor specific CVMCTL icache bits.
  MIPS: OCTEON: Core-15169 Workaround and general CVMSEG cleanup.
  MIPS: OCTEON: Update octeon-model.h code for new SoCs.
  MIPS: OCTEON: Implement DCache errata workaround for all CN6XXX
  MIPS: OCTEON: Add little-endian support to asm/octeon/octeon.h
  MIPS: OCTEON: Implement the core-16057 workaround
  MIPS: OCTEON: Delete unused COP2 saving code
  MIPS: OCTEON: Use correct instruction to read 64-bit COP0 register
  MIPS: OCTEON: Save and restore CP2 SHA3 state
  MIPS: OCTEON: Fix FP context save.
  MIPS: OCTEON: Save/Restore wider multiply registers in OCTEON III CPUs
  MIPS: boot: Provide more uImage options
  MIPS: Remove unneeded #ifdef __KERNEL__ from asm/processor.h
  MIPS: ip22-gio: Remove legacy suspend/resume support
  mips: pci: Add ifdef around pci_proc_domain
  ...

MIPS: Usage and cosmetic cleanups of page table bits.

2015-02-19T21:15:59+00:00

* Clean up white spaces and tabs.
   * Get rid of remaining hardcoded values for calculating
     shifts and masks.
   * Get rid of redundant macro values.
   * Do not use page table bits directly in #ifdef's.

Signed-off-by: Steven J. Hill 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9287/
Signed-off-by: Ralf Baechle