linux-toradex.git/arch/arm64/kernel/jump_label.c, branch v6.7

jump_label: make initial NOP patching the special case

2022-06-24T07:48:55+00:00

Instead of defaulting to patching NOP opcodes at init time, and leaving
it to the architectures to override this if this is not needed, switch
to a model where doing nothing is the default. This is the common case
by far, as only MIPS requires NOP patching at init time. On all other
architectures, the correct encodings are emitted by the compiler and so
no initial patching is needed.

Signed-off-by: Ard Biesheuvel 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Mark Rutland 
Acked-by: Peter Zijlstra (Intel) 
Link: https://lore.kernel.org/r/20220615154142.1574619-4-ardb@kernel.org

arm64: insn: decouple patching from insn code

2021-06-11T10:19:27+00:00

Currently,  includes . We intend that
 will be usable from userspace, so it doesn't make sense to
include headers for kernel-only features such as the patching routines,
and we'd intended to restrict  to instruction encoding
details.

Let's decouple the patching code from , and explicitly
include  where it is needed. Since 
isn't included from assembly, we can drop the __ASSEMBLY__ guards.

At the same time, sort the kprobes includes so that it's easier to see
what is and isn't incldued.

Signed-off-by: Mark Rutland 
Cc: Catalin Marinas 
Cc: Will Deacon 
Link: https://lore.kernel.org/r/20210609102301.17332-2-mark.rutland@arm.com
Signed-off-by: Will Deacon

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 234

2019-06-19T15:09:07+00:00

Based on 1 normalized pattern(s):

  this program is free software you can redistribute it and or modify
  it under the terms of the gnu general public license version 2 as
  published by the free software foundation this program is
  distributed in the hope that it will be useful but without any
  warranty without even the implied warranty of merchantability or
  fitness for a particular purpose see the gnu general public license
  for more details you should have received a copy of the gnu general
  public license along with this program if not see http www gnu org
  licenses

extracted by the scancode license scanner the SPDX license identifier

  GPL-2.0-only

has been chosen to replace the boilerplate/reference in 503 file(s).

Signed-off-by: Thomas Gleixner 
Reviewed-by: Alexios Zavras 
Reviewed-by: Allison Randal 
Reviewed-by: Enrico Weigelt 
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190602204653.811534538@linutronix.de
Signed-off-by: Greg Kroah-Hartman

jump_label: move 'asm goto' support test to Kconfig

2019-01-06T00:46:51+00:00

Currently, CONFIG_JUMP_LABEL just means "I _want_ to use jump label".

The jump label is controlled by HAVE_JUMP_LABEL, which is defined
like this:

  #if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_JUMP_LABEL)
  # define HAVE_JUMP_LABEL
  #endif

We can improve this by testing 'asm goto' support in Kconfig, then
make JUMP_LABEL depend on CC_HAS_ASM_GOTO.

Ugly #ifdef HAVE_JUMP_LABEL will go away, and CONFIG_JUMP_LABEL will
match to the real kernel capability.

Signed-off-by: Masahiro Yamada 
Acked-by: Michael Ellerman  (powerpc)
Tested-by: Sedat Dilek

arm64/kernel: jump_label: Switch to relative references

2018-09-27T15:56:47+00:00

On a randomly chosen distro kernel build for arm64, vmlinux.o shows the
following sections, containing jump label entries, and the associated
RELA relocation records, respectively:

  ...
  [38088] __jump_table      PROGBITS         0000000000000000  00e19f30
       000000000002ea10  0000000000000000  WA       0     0     8
  [38089] .rela__jump_table RELA             0000000000000000  01fd8bb0
       000000000008be30  0000000000000018   I      38178   38088     8
  ...

In other words, we have 190 KB worth of 'struct jump_entry' instances,
and 573 KB worth of RELA entries to relocate each entry's code, target
and key members. This means the RELA section occupies 10% of the .init
segment, and the two sections combined represent 5% of vmlinux's entire
memory footprint.

So let's switch from 64-bit absolute references to 32-bit relative
references for the code and target field, and a 64-bit relative
reference for the 'key' field (which may reside in another module or the
core kernel, which may be more than 4 GB way on arm64 when running with
KASLR enable): this reduces the size of the __jump_table by 33%, and
gets rid of the RELA section entirely.

Signed-off-by: Ard Biesheuvel 
Signed-off-by: Thomas Gleixner 
Acked-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: Arnd Bergmann 
Cc: Heiko Carstens 
Cc: Kees Cook 
Cc: Catalin Marinas 
Cc: Steven Rostedt 
Cc: Martin Schwidefsky 
Cc: Jessica Yu 
Link: https://lkml.kernel.org/r/20180919065144.25010-4-ard.biesheuvel@linaro.org

arm64: Avoid calling stop_machine() when patching jump labels

2018-08-17T09:26:44+00:00

Patching a jump label involves patching a single instruction at a time,
swizzling between a branch and a NOP. The architecture treats these
instructions specially, so a concurrently executing CPU is guaranteed to
see either the NOP or the branch, rather than an amalgamation of the two
instruction encodings.

However, in order to guarantee that the new instruction is visible, it
is necessary to send an IPI to the concurrently executing CPU so that it
discards any previously fetched instructions from its pipeline. This
operation therefore cannot be completed from a context with IRQs
disabled, but this is exactly what happens on the jump label path where
the hotplug lock is held and irqs are subsequently disabled by
stop_machine_cpuslocked(). This results in a deadlock during boot on
Hikey-960.

Due to the architectural guarantees around patching NOPs and branches,
we don't actually need to stop_machine() at all on the jump label path,
so we can avoid the deadlock by using the "nosync" variant of our
instruction patching routine.

Fixes: 693350a79980 ("arm64: insn: Don't fallback on nosync path for general insn patching")
Reported-by: Tuomas Tynkkynen 
Reported-by: John Stultz 
Tested-by: Valentin Schneider 
Tested-by: Tuomas Tynkkynen 
Tested-by: John Stultz 
Signed-off-by: Will Deacon

jump_label: Rename JUMP_LABEL_{EN,DIS}ABLE to JUMP_LABEL_{JMP,NOP}

2015-08-03T09:34:12+00:00

Since we've already stepped away from ENABLE is a JMP and DISABLE is a
NOP with the branch_default bits, and are going to make it even worse,
rename it to make it all clearer.

This way we don't mix multiple levels of logic attributes, but have a
plain 'physical' name for what the current instruction patching status
of a jump label is.

This is a first step in removing the naming confusion that has led to
a stream of avoidable bugs such as:

  a833581e372a ("x86, perf: Fix static_key bug in load_mm_cr4()")

Signed-off-by: Peter Zijlstra (Intel) 
Cc: Andrew Morton 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: linux-kernel@vger.kernel.org
[ Beefed up the changelog. ]
Signed-off-by: Ingo Molnar

arm64: jump labels: NOP out NOP -> NOP replacement

2014-11-26T17:19:47+00:00

In the arm64 arch_static_branch implementation we place an A64 NOP into
the instruction stream and log relevant details to a jump_entry in a
__jump_table section. Later this may be replaced with an immediate
branch without link to the code for the unlikely case.

At init time, the core calls arch_jump_label_transform_static to
initialise the NOPs. On x86 this involves inserting the optimal NOP for
a given microarchitecture, but on arm64 we only use the architectural
NOP, and hence replace each NOP with the exact same NOP. This is
somewhat pointless.

Additionally, at module load time we don't call jump_label_apply_nops to
patch the optimal NOPs in, unlike other architectures, but get away with
this because we only use the architectural NOP anyway. A later notifier
will patch NOPs with branches as required.

Similarly to x86 commit 11570da1c5b1dee1 (x86/jump-label: Do not bother
updating NOPs if they are correct), we can avoid patching NOPs with
identical NOPs. Given that we only use a single NOP encoding, this means
we can NOP-out the body of arch_jump_label_transform_static entirely. As
the default __weak arch_jump_label_transform_static implementation
performs a patch, we must use an empty function to achieve this.

Cc: Catalin Marinas 
Cc: Jiang Liu 
Cc: Laura Abbott 
Acked-by: Will Deacon 
Signed-off-by: Mark Rutland 
Signed-off-by: Will Deacon

arm64, jump label: optimize jump label implementation

2014-01-08T15:23:53+00:00

Optimize jump label implementation for ARM64 by dynamically patching
kernel text.

Reviewed-by: Will Deacon 
Signed-off-by: Jiang Liu 
Signed-off-by: Catalin Marinas