linux-toradex.git/kernel/rcu/tree_exp.h, branch v7.0-rc5

rcu: Make expedited RCU CPU stall warnings detect stall-end races

2026-01-07T13:58:26+00:00

If an expedited RCU CPU stall ends just at the stall-warning timeout,
the current code will print an expedited stall-warning message, but one
that doesn't identify any CPUs or tasks causing the stall.  This is most
likely to happen for short-timeout stalls, for example, the 20-millisecond
timeouts that are sometimes used for small embedded devices.  Needless to
say, these semi-empty stall-warning messages can be rather confusing.

One option would be to suppress the stall-warning message entirely in
this case, but the near-miss information can be quite valuable.

Detect this race condition and emits a "INFO: Expedited stall ended
before state dump start" message to clarify matters.

[boqun: Apply feedback from Borislav]

Reported-by: Borislav Petkov 
Signed-off-by: Paul E. McKenney 
Acked-by: Borislav Petkov (AMD) 
Signed-off-by: Boqun Feng

sched: Provide and use set_need_resched_current()

2025-11-20T21:26:09+00:00

set_tsk_need_resched(current) requires set_preempt_need_resched(current) to
work correctly outside of the scheduler.

Provide set_need_resched_current() which wraps this correctly and replace
all the open coded instances.

Signed-off-by: Peter Zijlstra 
Signed-off-by: Thomas Gleixner 
Link: https://patch.msgid.link/20251116174750.665769842@linutronix.de

rcu/exp: Remove needless CPU up quiescent state report

2025-07-08T17:51:13+00:00

A CPU coming online checks for an ongoing grace period and reports
a quiescent state accordingly if needed. This special treatment that
shortcuts the expedited IPI finds its origin as an optimization purpose
on the following commit:

	338b0f760e84 (rcu: Better hotplug handling for synchronize_sched_expedited()

The point is to avoid an IPI while waiting for a CPU to become online
or failing to become offline.

However this is pointless and even error prone for several reasons:

* If the CPU has been seen offline in the first round scanning offline
  and idle CPUs, no IPI is even tried and the quiescent state is
  reported on behalf of the CPU.

* This means that if the IPI fails, the CPU just became offline. So
  it's unlikely to become online right away, unless the cpu hotplug
  operation failed and rolled back, which is a rare event that can
  wait a jiffy for a new IPI to be issued.

* But then the "optimization" applying on failing CPU hotplug down only
  applies to !PREEMPT_RCU.

* This force reports a quiescent state even if ->cpu_no_qs.b.exp is not
  set. As a result it can race with remote QS reports on the same rdp.
  Fortunately it happens to be OK but an accident is waiting to happen.

For all those reasons, remove this optimization that doesn't look worthy
to keep around.

Reported-by: Paul E. McKenney 
Signed-off-by: Frederic Weisbecker 
Reviewed-by: Paul E. McKenney 
Signed-off-by: Joel Fernandes 
Signed-off-by: Neeraj Upadhyay (AMD)

rcu/exp: Protect against early QS report

2025-06-25T03:06:18+00:00

When a grace period is started, the ->expmask of each node is set up
from sync_exp_reset_tree(). Then later on each leaf node also initialize
its ->exp_tasks pointer.

This means that the initialization of the quiescent state of a node and
the initialization of its blocking tasks happen with an unlocked node
gap in-between.

It happens to be fine because nothing is expected to report an exp
quiescent state within this gap, since no IPI have been issued yet and
every rdp's ->cpu_no_qs.b.exp should be false.

However if it were to happen by accident, the quiescent state could be
reported and propagated while ignoring tasks that blocked _before_ the
start of the grace period.

Prevent such trouble to happen in the future and initialize both the
quiescent states mask to report and the blocked tasks head from the same
node locked block.

If a task blocks within an RCU read side critical section before
sync_exp_reset_tree() is called and is then unblocked between
sync_exp_reset_tree() and __sync_rcu_exp_select_node_cpus(), the QS
won't be reported because no RCU exp IPI had been issued to request it
through the setting of srdp->cpu_no_qs.b.exp.

Reviewed-by: Paul E. McKenney 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Joel Fernandes 
Signed-off-by: Neeraj Upadhyay (AMD)

rcu: Remove swake_up_one_online() bandaid

2025-05-16T13:00:54+00:00

It's now ok to perform a wake-up from an offline CPU because the
resulting armed scheduler bandwidth hrtimers are now correctly targeted
by hrtimer infrastructure.

Remove the obsolete hackerry.

Link: https://lore.kernel.org/all/20241231170712.149394-3-frederic@kernel.org/
Reviewed-by: Usama Arif 
Reviewed-by: Joel Fernandes 
Signed-off-by: Frederic Weisbecker 
Signed-off-by: Joel Fernandes

rcu: Split rcu_report_exp_cpu_mult() mask parameter and use for tracing

2025-02-05T05:52:38+00:00

This commit renames the rcu_report_exp_cpu_mult() function from "mask"
to "mask_in" and introduced a "mask" local variable to better support
upcoming event-tracing additions.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Boqun Feng

rcu: Add lockdep_assert_irqs_disabled() to rcu_exp_need_qs()

2024-12-14T16:10:38+00:00

Callers to rcu_exp_need_qs() are supposed to disable interrupts, so this
commit enlists lockdep's aid in checking this.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Add KCSAN exclusive-writer assertions for rdp->cpu_no_qs.b.exp

2024-12-14T16:10:29+00:00

The value of rdp->cpu_no_qs.b.exp may be changed only by the corresponding
CPU, and that CPU is not even allowed to race with itself, for example,
via interrupt handlers.  This commit therefore adds KCSAN exclusive-writer
assertions to check this constraint.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Make preemptible rcu_exp_handler() check idempotency

2024-12-14T16:10:20+00:00

Although the non-preemptible implementation of rcu_exp_handler()
contains checks to enforce idempotency, the preemptible version does not.
The reason for this omission is that in preemptible kernels, there is
no reporting of quiescent states from CPU hotplug notifiers, and thus
no need for idempotency.

In theory, anyway.

In practice, accidents happen.  This commit therefore adds checks under
WARN_ON_ONCE() to catch any such accidents.

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)

rcu: Replace open-coded rcu_exp_need_qs() from rcu_exp_handler() with call

2024-12-14T16:10:14+00:00

Currently, the preemptible implementation of rcu_exp_handler()
almost open-codes rcu_exp_need_qs().  A call to that function would be
shorter and would improve expediting in cases where rcu_exp_handler()
interrupted a preemption-disabled or bh-disabled region of code.
This commit therefore moves rcu_exp_need_qs() out of the non-preemptible
leg of the enclosing #ifdef and replaces the open coding in preemptible
rcu_exp_handler() with a call to rcu_exp_need_qs().

Signed-off-by: Paul E. McKenney 
Cc: Frederic Weisbecker 
Signed-off-by: Uladzislau Rezki (Sony)