linux-toradex.git/arch/mips/include/asm/barrier.h, branch v5.18

MIPS: barrier: Make __smp_mb__before_atomic() a no-op for Loongson3

2019-10-07T16:43:08+00:00

Loongson3 systems with CONFIG_CPU_LOONGSON3_WORKAROUNDS enabled already
emit a full completion barrier as part of the inline assembly containing
LL/SC loops for atomic operations. As such the barrier emitted by
__smp_mb__before_atomic() is redundant, and we can remove it.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Remove loongson_llsc_mb()

2019-10-07T16:43:06+00:00

The loongson_llsc_mb() macro is no longer used - instead barriers are
emitted as part of inline asm using the __SYNC() macro. Remove the
now-defunct loongson_llsc_mb() macro.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: futex: Emit Loongson3 sync workarounds within asm

2019-10-07T16:43:03+00:00

Generate the sync instructions required to workaround Loongson3 LL/SC
errata within inline asm blocks, which feels a little safer than doing
it from C where strictly speaking the compiler would be well within its
rights to insert a memory access between the separate asm statements we
previously had, containing sync & ll instructions respectively.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Clean up sync_ginv()

2019-10-07T16:42:24+00:00

Use the new __SYNC() infrastructure to implement sync_ginv(), for
consistency with much of the rest of the asm/barrier.h.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Clean up __sync() definition

2019-10-07T16:42:22+00:00

Implement __sync() using the new __SYNC() infrastructure, which will
take care of not emitting an instruction for old R3k CPUs that don't
support it. The only behavioral difference is that __sync() will now
provide a compiler barrier on these old CPUs, but that seems like
reasonable behavior anyway.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Remove fast_mb() Octeon #ifdef'ery

2019-10-07T16:42:21+00:00

The definition of fast_mb() is the same in both the Octeon & non-Octeon
cases, so remove the duplication & define it only once.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Clean up __smp_mb() definition

2019-10-07T16:42:20+00:00

We #ifdef on Cavium Octeon CPUs, but emit the same sync instruction in
both cases. Remove the #ifdef & simply expand to the __sync() macro.

Whilst here indent the strong ordering case definitions to match the
indentation of the weak ordering ones, helping readability.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Clean up rmb() & wmb() definitions

2019-10-07T16:42:18+00:00

Simplify our definitions of rmb() & wmb() using the new __SYNC()
infrastructure.

The fast_rmb() & fast_wmb() macros are removed, since they only provided
a level of indirection that made the code less readable & weren't
directly used anywhere in the kernel tree.

The Octeon #ifdef'ery is removed, since the "syncw" instruction
previously used is merely an alias for "sync 4" which __SYNC() will emit
for the wmb sync type when the kernel is configured for an Octeon CPU.
Similarly __SYNC() will emit nothing for the rmb sync type in Octeon
configurations.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

MIPS: barrier: Add __SYNC() infrastructure

2019-10-07T16:42:17+00:00

Introduce an asm/sync.h header which provides infrastructure that can be
used to generate sync instructions of various types, and for various
reasons. For example if we need a sync instruction that provides a full
completion barrier but only on systems which have weak memory ordering,
we can generate the appropriate assembly code using:

  __SYNC(full, weak_ordering)

When the kernel is configured to run on systems with weak memory
ordering (ie. CONFIG_WEAK_ORDERING is selected) we'll emit a sync
instruction. When the kernel is configured to run on systems with strong
memory ordering (ie. CONFIG_WEAK_ORDERING is not selected) we'll emit
nothing. The caller doesn't need to know which happened - it simply says
what it needs & when, with no concern for checking the kernel
configuration.

There are some scenarios in which we may want to emit code only when we
*didn't* emit a sync instruction. For example, some Loongson3 CPUs
suffer from a bug that requires us to emit a sync instruction prior to
each ll instruction (enabled by CONFIG_CPU_LOONGSON3_WORKAROUNDS). In
cases where this bug workaround is enabled, it's wasteful to then have
more generic code emit another sync instruction to provide barriers we
need in general. A __SYNC_ELSE() macro allows for this, providing an
extra argument that contains code to be assembled only in cases where
the sync instruction was not emitted. For example if we have a scenario
in which we generally want to emit a release barrier but for affected
Loongson3 configurations upgrade that to a full completion barrier, we
can do that like so:

  __SYNC_ELSE(full, loongson3_war, __SYNC(rl, always))

The assembly generated by these macros can be used either as inline
assembly or in assembly source files.

Differing types of sync as provided by MIPSr6 are defined, but currently
they all generate a full completion barrier except in kernels configured
for Cavium Octeon systems. There the wmb sync-type is used, and rmb
syncs are omitted, as has been the case since commit 6b07d38aaa52
("MIPS: Octeon: Use optimized memory barrier primitives."). Using
__SYNC() with the wmb or rmb types will abstract away the Octeon
specific behavior and allow us to later clean up asm/barrier.h code that
currently includes a plethora of #ifdef's.

Signed-off-by: Paul Burton 
Cc: linux-mips@vger.kernel.org
Cc: Huacai Chen 
Cc: Jiaxun Yang 
Cc: linux-kernel@vger.kernel.org

mips/atomic: Fix smp_mb__{before,after}_atomic()

2019-08-31T10:06:02+00:00

Recent probing at the Linux Kernel Memory Model uncovered a
'surprise'. Strongly ordered architectures where the atomic RmW
primitive implies full memory ordering and
smp_mb__{before,after}_atomic() are a simple barrier() (such as MIPS
without WEAK_REORDERING_BEYOND_LLSC) fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it
(surprisingly) does not provide a compiler barrier. This then allows
the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended. Therefore strengthen the atomic
RmW ops to include a compiler barrier.

Reported-by: Andrea Parri 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Paul Burton