linux-toradex.git/include/linux/rcu_segcblist.h, branch v7.0-rc7

rcu/nocb: Simplify (de-)offloading state machine

2024-09-08T18:33:55+00:00

Now that the (de-)offloading process can only apply to offline CPUs,
there is no more concurrency between rcu_core and nocb kthreads. Also
the mutation now happens on empty queues.

Therefore the state machine can be reduced to a single bit called
SEGCBLIST_OFFLOADED. Simplify the transition as follows:

* Upon offloading: queue the rdp to be added to the rcuog list and
  wait for the rcuog kthread to set the SEGCBLIST_OFFLOADED bit. Unpark
  rcuo kthread.

* Upon de-offloading: Park rcuo kthread. Queue the rdp to be removed
  from the rcuog list and wait for the rcuog kthread to clear the
  SEGCBLIST_OFFLOADED bit.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney 
Reviewed-by: Paul E. McKenney 
Signed-off-by: Neeraj Upadhyay

rcu/nocb: Remove SEGCBLIST_KTHREAD_CB

2024-07-29T02:04:32+00:00

This state excerpt from the (de-)offloading state machine was used to
implement an ad-hoc kthread parking of rcuo kthreads. This code has
been removed and therefore the related state can be erased as well.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney 
Reviewed-by: Paul E. McKenney 
Signed-off-by: Neeraj Upadhyay

rcu/nocb: Remove SEGCBLIST_RCU_CORE

2024-07-29T02:04:32+00:00

RCU core can't be running anymore while in the middle of (de-)offloading
since this sort of transition now only applies to offline CPUs.

The SEGCBLIST_RCU_CORE state can therefore be removed.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney 
Reviewed-by: Paul E. McKenney 
Signed-off-by: Neeraj Upadhyay

rcu/nocb: Use kthread parking instead of ad-hoc implementation

2024-06-04T00:26:26+00:00

Upon NOCB deoffloading, the rcuo kthread must be forced to sleep
until the corresponding rdp is ever offloaded again. The deoffloader
clears the SEGCBLIST_OFFLOADED flag, wakes up the rcuo kthread which
then notices that change and clears in turn its SEGCBLIST_KTHREAD_CB
flag before going to sleep, until it ever sees the SEGCBLIST_OFFLOADED
flag again, should a re-offloading happen.

Upon NOCB offloading, the rcuo kthread must be forced to wake up and
handle callbacks until the corresponding rdp is ever deoffloaded again.
The offloader sets the SEGCBLIST_OFFLOADED flag, wakes up the rcuo
kthread which then notices that change and sets in turn its
SEGCBLIST_KTHREAD_CB flag before going to check callbacks, until it
ever sees the SEGCBLIST_OFFLOADED flag cleared again, should a
de-offloading happen again.

This is all a crude ad-hoc and error-prone kthread (un-)parking
re-implementation.

Consolidate the behaviour with the appropriate API instead.

[ paulmck: Apply Qiang Zhang feedback provided in Link: below. ]
Link: https://lore.kernel.org/all/20240509074046.15629-1-qiang.zhang1211@gmail.com/

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Fix segcblist state machine stale comments about timers

2024-06-04T00:26:25+00:00

The (de-)offloading process used to take care about the NOCB timer when
it depended on the per-rdp NOCB locking. However this isn't the case
anymore for a long while. It can now safely be armed and run during the
(de-)offloading process, which doesn't care about it anymore.

Update the comments accordingly.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Fix segcblist state machine comments about bypass

2024-06-04T00:26:25+00:00

The parts explaining the bypass lifecycle in (de-)offloading are out
of date and/or wrong. Bypass is simply enabled whenever SEGCBLIST_RCU_CORE
flag is off.

Fix the comments accordingly.

Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Invoke rcu_core() at the start of deoffloading

2021-12-08T00:24:44+00:00

On PREEMPT_RT, if rcu_core() is preempted by the de-offloading process,
some work, such as callbacks acceleration and invocation, may be left
unattended due to the volatile checks on the offloaded state.

In the worst case this work is postponed until the next rcu_pending()
check that can take a jiffy to reach, which can be a problem in case
of callbacks flooding.

Solve that with invoking rcu_core() early in the de-offloading process.
This way any work dismissed by an ongoing rcu_core() call fooled by
a preempting deoffloading process will be caught up by a nearby future
recall to rcu_core(), this time fully aware of the de-offloading state.

Tested-by: Valentin Schneider 
Tested-by: Sebastian Andrzej Siewior 
Signed-off-by: Frederic Weisbecker 
Cc: Valentin Schneider 
Cc: Peter Zijlstra 
Cc: Sebastian Andrzej Siewior 
Cc: Josh Triplett 
Cc: Joel Fernandes 
Cc: Boqun Feng 
Cc: Neeraj Upadhyay 
Cc: Uladzislau Rezki 
Cc: Thomas Gleixner 
Signed-off-by: Paul E. McKenney

rcu/nocb: Prepare state machine for a new step

2021-12-08T00:24:44+00:00

Currently SEGCBLIST_SOFTIRQ_ONLY is a bit of an exception among the
segcblist flags because it is an exclusive state that doesn't mix up
with the other flags. Remove it in favour of:

_ A flag specifying that rcu_core() needs to perform callbacks execution
  and acceleration

and

_ A flag specifying we want the nocb lock to be held in any needed
  circumstances

This clarifies the code and is more flexible: It allows to have a state
where rcu_core() runs with locking while offloading hasn't started yet.
This is a necessary step to prepare for triggering rcu_core() at the
very beginning of the de-offloading process so that rcu_core() won't
dismiss work while being preempted by the de-offloading process, at
least not without a pending subsequent rcu_core() that will quickly
catch up.

Reviewed-by: Valentin Schneider 
Tested-by: Valentin Schneider 
Tested-by: Sebastian Andrzej Siewior 
Signed-off-by: Frederic Weisbecker 
Cc: Peter Zijlstra 
Cc: Sebastian Andrzej Siewior 
Cc: Josh Triplett 
Cc: Joel Fernandes 
Cc: Boqun Feng 
Cc: Neeraj Upadhyay 
Cc: Uladzislau Rezki 
Cc: Thomas Gleixner 
Signed-off-by: Paul E. McKenney

rcu/nocb: Disable bypass when CPU isn't completely offloaded

2021-03-15T20:54:54+00:00

Currently, the bypass is flushed at the very last moment in the
deoffloading procedure.  However, this approach leads to a larger state
space than would be preferred.  This commit therefore disables the
bypass at soon as the deoffloading procedure begins, then flushes it.
This guarantees that the bypass remains empty and thus out of the way
of the deoffloading procedure.

Symmetrically, this commit waits to enable the bypass until the offloading
procedure has completed.

Reported-by: Paul E. McKenney 
Cc: Josh Triplett 
Cc: Lai Jiangshan 
Cc: Joel Fernandes 
Cc: Neeraj Upadhyay 
Cc: Boqun Feng 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney

rcu/nocb: Provide basic callback offloading state machine bits

2021-01-07T00:24:19+00:00

Offloading and de-offloading RCU callback processes must be done
carefully.  There must never be a time at which callback processing is
disabled because the task driving the offloading or de-offloading might be
preempted or otherwise stalled at that point in time, which would result
in OOM due to calbacks piling up indefinitely.  This implies that there
will be times during which a given CPU's callbacks might be concurrently
invoked by both that CPU's RCU_SOFTIRQ handler (or, equivalently, that
CPU's rcuc kthread) and by that CPU's rcuo kthread.

This situation could fatally confuse both rcu_barrier() and the
CPU-hotplug offlining process, so these must be excluded during any
concurrent-callback-invocation period.  In addition, during times of
concurrent callback invocation, changes to ->cblist must be protected
both as needed for RCU_SOFTIRQ and as needed for the rcuo kthread.

This commit therefore defines and documents the states for a state
machine that coordinates offloading and deoffloading.

Cc: Josh Triplett 
Cc: Steven Rostedt 
Cc: Mathieu Desnoyers 
Cc: Lai Jiangshan 
Cc: Joel Fernandes 
Cc: Neeraj Upadhyay 
Cc: Thomas Gleixner 
Inspired-by: Paul E. McKenney 
Tested-by: Boqun Feng 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Paul E. McKenney