linux-toradex.git/include/linux/srcutiny.h, branch v6.16-rc6

Merge branches 'docs.2025.02.04a', 'lazypreempt.2025.03.04a', 'misc.2025.03.04a', 'srcu.2025.02.05a' and 'torture.2025.02.05a'

2025-03-05T02:47:32+00:00

srcu: Add SRCU-fast readers

2025-02-05T15:12:05+00:00

This commit adds srcu_read_{,un}lock_fast(), which is similar
to srcu_read_{,un}lock_lite(), but avoids the array-indexing and
pointer-following overhead.  On a microbenchmark featuring tight
loops around empty readers, this results in about a 20% speedup
compared to RCU Tasks Trace on my x86 laptop.

Please note that SRCU-fast has drawbacks compared to RCU Tasks
Trace, including:

o	Lack of CPU stall warnings.
o	SRCU-fast readers permitted only where rcu_is_watching().
o	A pointer-sized return value from srcu_read_lock_fast() must
	be passed to the corresponding srcu_read_unlock_fast().
o	In the absence of readers, a synchronize_srcu() having _fast()
	readers will incur the latency of at least two normal RCU grace
	periods.
o	RCU Tasks Trace priority boosting could be easily added.
	Boosting SRCU readers is more difficult.

SRCU-fast also has a drawback compared to SRCU-lite, namely that the
return value from srcu_read_lock_fast()-fast is a 64-bit pointer and
that from srcu_read_lock_lite() is only a 32-bit int.

[ paulmck: Apply feedback from Akira Yokosawa. ]

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Signed-off-by: Boqun Feng

srcu: Move SRCU Tree/Tiny definitions from srcu.h

2025-02-05T15:12:05+00:00

There are a couple of definitions under "#ifdef CONFIG_TINY_SRCU"
in include/linux/srcu.h.  There is no point in them being there,
so this commit moves them to include/linux/srcutiny.h and
include/linux/srcutree.c, thus eliminating that #ifdef.

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Signed-off-by: Boqun Feng

srcu: Rename srcu_check_read_flavor_lite() to srcu_check_read_flavor_force()

2025-02-05T15:12:05+00:00

This commit renames the srcu_check_read_flavor_lite() function to
srcu_check_read_flavor_force() and adds a read_flavor argument in order to
support an srcu_read_lock_fast() variant that is to avoid array indexing
in both the lock and unlock primitives.

Signed-off-by: Paul E. McKenney 
Cc: Alexei Starovoitov 
Cc: Andrii Nakryiko 
Cc: Peter Zijlstra 
Cc: Kent Overstreet 
Cc: 
Signed-off-by: Boqun Feng

rcu: rename PREEMPT_AUTO to PREEMPT_LAZY

2025-02-05T15:01:55+00:00

Replace mentions of PREEMPT_AUTO with PREEMPT_LAZY.

Also, since PREMPT_LAZY implies PREEMPTION, we can reduce the
TASKS_RCU selection criteria from this:

  NEED_TASKS_RCU && (PREEMPTION || PREEMPT_AUTO)

to this:

  NEED_TASKS_RCU && PREEMPTION

CC: Paul E. McKenney 
Reviewed-by: Frederic Weisbecker 
Reviewed-by: Sebastian Andrzej Siewior 
Signed-off-by: Ankur Arora 
Signed-off-by: Paul E. McKenney 
Signed-off-by: Boqun Feng

srcu: Unconditionally record srcu_read_lock_lite() in ->srcu_reader_flavor

2024-11-15T21:13:37+00:00

Currently, srcu_read_lock_lite() uses the SRCU_READ_FLAVOR_LITE bit in
->srcu_reader_flavor to communicate to the grace-period processing in
srcu_readers_active_idx_check() that the smp_mb() must be replaced by a
synchronize_rcu().  Unfortunately, ->srcu_reader_flavor is not updated
unless the kernel is built with CONFIG_PROVE_RCU=y.  Therefore in all
kernels built with CONFIG_PROVE_RCU=n, srcu_readers_active_idx_check()
incorrectly uses smp_mb() instead of synchronize_rcu() for srcu_struct
structures whose readers use srcu_read_lock_lite().

This commit therefore causes Tree SRCU srcu_read_lock_lite()
to unconditionally update ->srcu_reader_flavor so that
srcu_readers_active_idx_check() can make the correct choice.

Reported-by: Neeraj Upadhyay 
Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/
Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()")
Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Reviewed-by: Neeraj Upadhyay 
Signed-off-by: Frederic Weisbecker

srcu: Make Tiny SRCU explicitly disable preemption

2024-04-15T09:29:48+00:00

Because Tiny SRCU is used only in kernels built with either
CONFIG_PREEMPT_NONE=y or CONFIG_PREEMPT_VOLUNTARY=y, there has not
been any need for TINY SRCU to explicitly disable preemption.  However,
the prospect of lazy preemption changes that, and the lazy-preemption
patches do result in rcutorture runs finding both too-short grace periods
and grace-period hangs for Tiny SRCU.

This commit therefore adds the needed preempt_disable() and
preempt_enable() calls to Tiny SRCU.

Signed-off-by: Paul E. McKenney 
Cc: Ankur Arora 
Cc: Thomas Gleixner 
Signed-off-by: Uladzislau Rezki (Sony)

srcu,notifier: Remove #ifdefs in favor of SRCU Tiny srcu_usage

2023-08-16T21:27:01+00:00

This commit removes two #ifdef directives from include/linux/notifier.h
by causing SRCU Tiny to provide a dummy srcu_usage structure and a dummy
__SRCU_USAGE_INIT() macro.

Suggested-by: Linus Torvalds 
Cc: Matthias Brugger 
Cc: "Rafael J. Wysocki" 
Cc: "Michał Mirosław" 
Cc: Dmitry Osipenko 
Cc: Sachin Sant 
Cc: Joel Fernandes (Google) 
Signed-off-by: Paul E. McKenney

srcu: Begin offloading srcu_struct fields to srcu_update

2023-04-04T15:35:34+00:00

The current srcu_struct structure is on the order of 200 bytes in size
(depending on architecture and .config), which is much better than the
old-style 26K bytes, but still all too inconvenient when one is trying
to achieve good cache locality on a fastpath involving SRCU readers.

However, only a few fields in srcu_struct are used by SRCU readers.
The remaining fields could be offloaded to a new srcu_update
structure, thus shrinking the srcu_struct structure down to a few
tens of bytes.  This commit begins this noble quest, a quest that is
complicated by open-coded initialization of the srcu_struct within the
srcu_notifier_head structure.  This complication is addressed by updating
the srcu_notifier_head structure's open coding, given that there does
not appear to be a straightforward way of abstracting that initialization.

This commit moves only the ->node pointer to srcu_update.  Later commits
will move additional fields.

[ paulmck: Fold in qiang1.zhang@intel.com's memory-leak fix. ]

Link: https://lore.kernel.org/all/20230320055751.4120251-1-qiang1.zhang@intel.com/
Suggested-by: Christoph Hellwig 
Cc: "Rafael J. Wysocki" 
Cc: "Michał Mirosław" 
Cc: Dmitry Osipenko 
Tested-by: Sachin Sant 
Tested-by: "Zhang, Qiang1" 
Acked-by: Rafael J. Wysocki 
Tested-by: Joel Fernandes (Google) 
Signed-off-by: Paul E. McKenney

srcu: Make Tiny SRCU use full-sized grace-period counters

2022-08-31T12:10:15+00:00

This commit makes Tiny SRCU use full-sized grace-period counters to
further avoid counter-wrap issues when using polled grace-period APIs.

Signed-off-by: Paul E. McKenney