linux-toradex.git/include/linux/seqlock.h, branch v6.12-rc4

seqcount: replace smp_rmb() in read_seqcount() with load acquire

2024-09-22T20:35:36+00:00

Many architectures support load acquire which can replace a memory
barrier and save some cycles.

A typical sequence

do {
seq = read_seqcount_begin(&s);

} while (read_seqcount_retry(&s, seq);

requires 13 cycles on an N1 Neoverse arm64 core (Ampere Altra, to be
specific) for an empty loop. Two read memory barriers are needed. One
for each of the seqcount_* functions.

We can replace the first read barrier with a load acquire of the
seqcount which saves us one barrier.

On the Altra doing so reduces the cycle count from 13 to 8.

According to ARM, this is a general improvement for the ARM64
architecture and not specific to a certain processor.

See

https://developer.arm.com/documentation/102336/0100/Load-Acquire-and-Store-Release-instructions

"Weaker ordering requirements that are imposed by Load-Acquire and
Store-Release instructions allow for micro-architectural
optimizations, which could reduce some of the performance impacts that
are otherwise imposed by an explicit memory barrier.

If the ordering requirement is satisfied using either a Load-Acquire
or Store-Release, then it would be preferable to use these
instructions instead of a DMB"

[ NOTE! This is my original minimal patch that unconditionally switches
over to using smp_load_acquire(), instead of the much more involved
and subtle patch that Christoph Lameter wrote that made it
conditional.

But Christoph gets authorship credit because I had initially thought
that we needed the more complex model, and Christoph ran with it it
and did the work. Only after looking at code generation for all the
relevant architectures, did I come to the conclusion that nobody
actually really needs the old "smp_rmb()" model.

Even architectures without load-acquire support generally do as well
or better with smp_load_acquire().

So credit to Christoph, but if this then causes issues on other
architectures, put the blame solidly on me.

Also note as part of the ruthless simplification, this gets rid of the
overly subtle optimization where some code uses a non-barrier version
of the sequence count (see the __read_seqcount_begin() users in
fs/namei.c). They then play games with their own barriers and/or with
nested sequence counts.

Those optimizations are literally meaningless on x86, and questionable
elsewhere. If somebody can show that they matter, we need to re-do
them more cleanly than "use an internal helper". - Linus ]

Signed-off-by: Christoph Lameter (Ampere)
Link: https://lore.kernel.org/all/20240912-seq_optimize-v3-1-8ee25e04dffa@gentwo.org/
Signed-off-by: Linus Torvalds

locking/seqlock: Split out seqlock_types.h

2023-12-21T00:26:31+00:00

Trimming down sched.h dependencies: we don't want to include more than
the base types.

Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Will Deacon 
Cc: Waiman Long 
Cc: Boqun Feng 
Signed-off-by: Kent Overstreet

locking/seqlock: Fix grammar in comment

2023-10-17T11:28:12+00:00

The "neither writes before and after ..." for the description
of do_write_seqcount_end() should be "neither writes before nor after".

Signed-off-by: Cuda-Chen 
Signed-off-by: Ingo Molnar 
Link: https://lore.kernel.org/r/20231017053703.11312-1-clh960524@gmail.com

locking/seqlock: Propagate 'const' pointers within read-only methods, remove forced type casts

2023-10-14T09:06:15+00:00

Currently __seqprop_ptr() is an inline function that must chose to either
use 'const' or non-const seqcount related pointers - but this results in
the undesirable loss of 'const' propagation, via a forced type cast.

The easiest solution would be to turn the pointer wrappers into macros that
pass through whatever type is passed to them - but the clever maze of
seqlock API instantiation macros relies on the GCC CPP '##' macro
extension, which isn't recursive, so inline functions must be used here.

So create two wrapper variants instead: 'ptr' and 'const_ptr', and pick the
right one for the codepaths that are const: read_seqcount_begin() and
read_seqcount_retry().

This cleans up type handling and allows the removal of all type forcing.

No change in functionality.

Signed-off-by: Ingo Molnar 
Reviewed-by: Oleg Nesterov 
Cc: Linus Torvalds 
Cc: Peter Zijlstra 
Cc: Waiman Long 
Cc: Will Deacon 
Cc: Thomas Gleixner 
Cc: Paul E. McKenney

locking/seqlock: Change __seqprop() to return the function pointer

2023-10-12T18:18:21+00:00

This simplifies the macro and makes it easy to add the new seqprop's
with 2 or more args.

Plus this way we do not lose the type info, the (void*) type cast is
no longer needed.

And the latter reveals the problem: a lot of seqcount_t helpers pass
the "const seqcount_t *s" argument to __seqprop_ptr(seqcount_t *s)
but (before this patch) "(void *)(s)" masked the problem.

So this patch changes __seqprop_ptr() and __seqprop_##lockname##_ptr()
to accept the "const LOCKNAME *s" argument. This is not nice either,
they need to drop the constness on return because these helpers are used
by both the readers and writers, but at least it is clear what's going on.

Signed-off-by: Oleg Nesterov 
Signed-off-by: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Waiman Long 
Cc: Will Deacon 
Cc: Thomas Gleixner 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Link: https://lore.kernel.org/r/20231012143227.GA16143@redhat.com

locking/seqlock: Simplify SEQCOUNT_LOCKNAME()

2023-10-12T18:18:20+00:00

1. Kill the "lockmember" argument. It is always s->lock plus
   __seqprop_##lockname##_sequence() already uses s->lock and
   ignores "lockmember".

2. Kill the "lock_acquire" argument. __seqprop_##lockname##_sequence()
   can use the same "lockbase" prefix for _lock and _unlock.

Apart from line numbers, gcc -E outputs the same code.

Signed-off-by: Oleg Nesterov 
Signed-off-by: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Waiman Long 
Cc: Will Deacon 
Cc: Thomas Gleixner 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Link: https://lore.kernel.org/r/20231012143158.GA16133@redhat.com

Merge tag 'v6.6-rc5' into locking/core, to pick up fixes

2023-10-09T16:09:23+00:00

Signed-off-by: Ingo Molnar

locking/seqlock: Fix typo in comment

2023-10-05T08:39:44+00:00

s/the the
 /the

[ mingo: Cleaned up the changelog. ]

Signed-off-by: Zizhen Pang 
Signed-off-by: Ingo Molnar 
Link: https://lore.kernel.org/r/70293ecd5bb7a1cd370fd4d95c35f936@208suo.com

locking/seqlock: Do the lockdep annotation before locking in do_write_seqcount_begin_nested()

2023-09-21T06:37:44+00:00

It was brought up by Tetsuo that the following sequence:

   write_seqlock_irqsave()
   printk_deferred_enter()

could lead to a deadlock if the lockdep annotation within
write_seqlock_irqsave() triggers.

The problem is that the sequence counter is incremented before the lockdep
annotation is performed. The lockdep splat would then attempt to invoke
printk() but the reader side, of the same seqcount, could have a
tty_port::lock acquired waiting for the sequence number to become even again.

The other lockdep annotations come before the actual locking because "we
want to see the locking error before it happens". There is no reason why
seqcount should be different here.

Do the lockdep annotation first then perform the locking operation (the
sequence increment).

Fixes: 1ca7d67cf5d5a ("seqcount: Add lockdep functionality to seqcount/seqlock structures")
Reported-by: Tetsuo Handa 
Signed-off-by: Sebastian Andrzej Siewior 
Signed-off-by: Ingo Molnar 
Link: https://lore.kernel.org/r/20230920104627._DTHgPyA@linutronix.de

Closes: https://lore.kernel.org/20230621130641.-5iueY1I@linutronix.de

seqlock/latch: Provide raw_read_seqcount_latch_retry()

2023-06-05T19:11:03+00:00

The read side of seqcount_latch consists of:

  do {
    seq = raw_read_seqcount_latch(&latch->seq);
    ...
  } while (read_seqcount_latch_retry(&latch->seq, seq));

which is asymmetric in the raw_ department, and sure enough,
read_seqcount_latch_retry() includes (explicit) instrumentation where
raw_read_seqcount_latch() does not.

This inconsistency becomes a problem when trying to use it from
noinstr code. As such, fix it by renaming and re-implementing
raw_read_seqcount_latch_retry() without the instrumentation.

Specifically the instrumentation in question is kcsan_atomic_next(0)
in do___read_seqcount_retry(). Loosing this annotation is not a
problem because raw_read_seqcount_latch() does not pass through
kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX).

Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Thomas Gleixner 
Reviewed-by: Petr Mladek 
Tested-by: Michael Kelley   # Hyper-V
Link: https://lore.kernel.org/r/20230519102715.233598176@infradead.org