linux-toradex.git/arch/mips/include/asm/jump_label.h, branch v5.18

MIPS: jump_label: Use compact branches for >= r6

2019-04-09T23:21:30+00:00

MIPSr6 introduced compact branches which have no delay slots. Make use
of them for jump labels in order to avoid the need for a nop to fill the
branch or jump delay slot, saving 4 bytes of code for each static branch.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org

MIPS: jump_label: Remove redundant nops

2019-04-09T23:21:28+00:00

Both arch_static_branch() & arch_static_branch_jump() emit a control
transfer instruction (ie. branch or jump) without disabling assembler
re-ordering. As such the assembler will automatically fill their delay
slots.

Both functions follow their branch or jump with an explicit nop that at
first appears to be there to fill the delay slot, but given that the
assembler will do that the explicit nops serve no purpose & we end up
with our branch or jump followed by 2 nops. Remove the redundant nops.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org

MIPS: Fix kernel crash for R6 in jump label branch function

2019-03-11T18:11:11+00:00

Insert Branch instruction instead of NOP to make sure assembler don't
patch code in forbidden slot. In jump label function, it might
be possible to patch Control Transfer Instructions(CTIs) into
forbidden slot, which will generate Reserved Instruction exception
in MIPS release 6.

Signed-off-by: Archer Yan 
Reviewed-by: Paul Burton 
[paul.burton@mips.com:
  - Add MIPS prefix to subject.
  - Mark for stable from v4.0, which introduced r6 support, onwards.]
Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: stable@vger.kernel.org # v4.0+

locking/static_keys: Add a new static_key interface

2015-08-03T09:34:15+00:00

There are various problems and short-comings with the current
static_key interface:

 - static_key_{true,false}() read like a branch depending on the key
   value, instead of the actual likely/unlikely branch depending on
   init value.

 - static_key_{true,false}() are, as stated above, tied to the
   static_key init values STATIC_KEY_INIT_{TRUE,FALSE}.

 - we're limited to the 2 (out of 4) possible options that compile to
   a default NOP because that's what our arch_static_branch() assembly
   emits.

So provide a new static_key interface:

  DEFINE_STATIC_KEY_TRUE(name);
  DEFINE_STATIC_KEY_FALSE(name);

Which define a key of different types with an initial true/false
value.

Then allow:

   static_branch_likely()
   static_branch_unlikely()

to take a key of either type and emit the right instruction for the
case.

This means adding a second arch_static_branch_jump() assembly helper
which emits a JMP per default.

In order to determine the right instruction for the right state,
encode the branch type in the LSB of jump_entry::key.

This is the final step in removing the naming confusion that has led to
a stream of avoidable bugs such as:

  a833581e372a ("x86, perf: Fix static_key bug in load_mm_cr4()")

... but it also allows new static key combinations that will give us
performance enhancements in the subsequent patches.

Tested-by: Rabin Vincent  # arm
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Michael Ellerman  # ppc
Acked-by: Heiko Carstens  # s390
Cc: Andrew Morton 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar

jump_label: Allow asm/jump_label.h to be included in assembly

2015-04-09T07:40:23+00:00

Wrap asm/jump_label.h for all archs with #ifndef __ASSEMBLY__.
Since these are kernel only headers, we don't need #ifdef
__KERNEL__ so can simplify things a bit.

If an architecture wants to use jump labels in assembly, it
will still need to define a macro to create the __jump_table
entries (see ARCH_STATIC_BRANCH in the powerpc asm/jump_label.h
for an example).

Signed-off-by: Anton Blanchard 
Acked-by: Peter Zijlstra (Intel) 
Cc: Andrew Morton 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: benh@kernel.crashing.org
Cc: catalin.marinas@arm.com
Cc: davem@davemloft.net
Cc: heiko.carstens@de.ibm.com
Cc: jbaron@akamai.com
Cc: linux@arm.linux.org.uk
Cc: linuxppc-dev@lists.ozlabs.org
Cc: liuj97@gmail.com
Cc: mgorman@suse.de
Cc: mmarek@suse.cz
Cc: mpe@ellerman.id.au
Cc: paulus@samba.org
Cc: ralf@linux-mips.org
Cc: rostedt@goodmis.org
Cc: schwidefsky@de.ibm.com
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1428551492-21977-1-git-send-email-anton@samba.org
Signed-off-by: Ingo Molnar

MIPS: jump_label.c: Handle the microMIPS J instruction encoding

2014-11-19T17:22:09+00:00

Implement the microMIPS encoding of the J instruction for the purpose of
the static keys feature, fixing a crash early on in bootstrap as the
kernel is unhappy seeing the ISA bit set in jump table entries.  Make
sure the ISA bit correctly reflects the instruction encoding chosen for
the kernel, 0 for the standard MIPS and 1 for the microMIPS encoding.

Also make sure the instruction to patch is a 32-bit NOP in the microMIPS
mode as by default the 16-bit short encoding is assumed

Signed-off-by: Maciej W. Rozycki 
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8516/
Signed-off-by: Ralf Baechle

compiler/gcc4: Add quirk for 'asm goto' miscompilation bug

2013-10-11T05:39:14+00:00

Fengguang Wu, Oleg Nesterov and Peter Zijlstra tracked down
a kernel crash to a GCC bug: GCC miscompiles certain 'asm goto'
constructs, as outlined here:

  http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670

Implement a workaround suggested by Jakub Jelinek.

Reported-and-tested-by: Fengguang Wu 
Reported-by: Oleg Nesterov 
Reported-by: Peter Zijlstra 
Suggested-by: Jakub Jelinek 
Reviewed-by: Richard Henderson 
Cc: Linus Torvalds 
Cc: Andrew Morton 
Cc: 
Signed-off-by: Ingo Molnar

static keys: Introduce 'struct static_key', static_key_true()/false() and static_key_slow_[inc|dec]()

2012-02-24T09:05:59+00:00

So here's a boot tested patch on top of Jason's series that does
all the cleanups I talked about and turns jump labels into a
more intuitive to use facility. It should also address the
various misconceptions and confusions that surround jump labels.

Typical usage scenarios:

        #include 

        struct static_key key = STATIC_KEY_INIT_TRUE;

        if (static_key_false(&key))
                do unlikely code
        else
                do likely code

Or:

        if (static_key_true(&key))
                do likely code
        else
                do unlikely code

The static key is modified via:

        static_key_slow_inc(&key);
        ...
        static_key_slow_dec(&key);

The 'slow' prefix makes it abundantly clear that this is an
expensive operation.

I've updated all in-kernel code to use this everywhere. Note
that I (intentionally) have not pushed through the rename
blindly through to the lowest levels: the actual jump-label
patching arch facility should be named like that, so we want to
decouple jump labels from the static-key facility a bit.

On non-jump-label enabled architectures static keys default to
likely()/unlikely() branches.

Signed-off-by: Ingo Molnar 
Acked-by: Jason Baron 
Acked-by: Steven Rostedt 
Cc: a.p.zijlstra@chello.nl
Cc: mathieu.desnoyers@efficios.com
Cc: davem@davemloft.net
Cc: ddaney.cavm@gmail.com
Cc: Linus Torvalds 
Link: http://lkml.kernel.org/r/20120222085809.GA26397@elte.hu
Signed-off-by: Ingo Molnar

jump label: Introduce static_branch() interface

2011-04-04T16:48:08+00:00

Introduce:

static __always_inline bool static_branch(struct jump_label_key *key);

instead of the old JUMP_LABEL(key, label) macro.

In this way, jump labels become really easy to use:

Define:

        struct jump_label_key jump_key;

Can be used as:

        if (static_branch(&jump_key))
                do unlikely code

enable/disale via:

        jump_label_inc(&jump_key);
        jump_label_dec(&jump_key);

that's it!

For the jump labels disabled case, the static_branch() becomes an
atomic_read(), and jump_label_inc()/dec() are simply atomic_inc(),
atomic_dec() operations. We show testing results for this change below.

Thanks to H. Peter Anvin for suggesting the 'static_branch()' construct.

Since we now require a 'struct jump_label_key *key', we can store a pointer into
the jump table addresses. In this way, we can enable/disable jump labels, in
basically constant time. This change allows us to completely remove the previous
hashtable scheme. Thanks to Peter Zijlstra for this re-write.

Testing:

I ran a series of 'tbench 20' runs 5 times (with reboots) for 3
configurations, where tracepoints were disabled.

jump label configured in
avg: 815.6

jump label *not* configured in (using atomic reads)
avg: 800.1

jump label *not* configured in (regular reads)
avg: 803.4

Signed-off-by: Peter Zijlstra 
LKML-Reference: <20110316212947.GA8792@redhat.com>
Signed-off-by: Jason Baron 
Suggested-by: H. Peter Anvin 
Tested-by: David Daney 
Acked-by: Ralf Baechle 
Acked-by: David S. Miller 
Acked-by: Mathieu Desnoyers 
Signed-off-by: Steven Rostedt

MIPS: jump label: Add MIPS support.

2011-01-18T18:30:24+00:00

In order not to be left behind, we add jump label support for MIPS.

Tested on 64-bit big endian (Octeon), and 32-bit little endian
(malta/qemu).

Signed-off-by: David Daney 
To: linux-mips@linux-mips.org
Cc: Steven Rostedt 
Cc: Jason Baron 
Patchwork: https://patchwork.linux-mips.org/patch/1923/
Signed-off-by: Ralf Baechle