| Age | Commit message (Collapse) | Author |
|
nmi_panic() used to accept variable arguments until commit ebc41f20d77f
("panic: change nmi_panic from macro to function") flattened it to a final
message string. vpanic() did not exist back then, so the function had to
format through panic("%s", msg).
Bring the variable arguments back and format with vpanic() directly.
The next patch makes nmi_panic() try the panic_force_cpu= redirect
before claiming panic_cpu. panic_try_force_cpu() needs to print the
message but it is used also by panic() which accepts variable argument
list. The string could be formatted only when a CPU gets assigned to
process the redirection. Otherwise, there might be a race when writing
to the `panic_force_buf`.
No current caller passes a string with format specifiers. The closest one
is hpwdt_pretimeout(), which builds panic_msg with hex_byte_pack() and has
only two variants, both plain strings. But the new __printf() annotation
on nmi_panic() would warn with -Wformat-security there because the buffer
is passed directly as the format argument, so switch it to nmi_panic(regs,
"%s", panic_msg).
[pmladek@suse.com: changelog fix]
Link: https://lore.kernel.org/arJxFybmtD7OBIcL@pathway.suse.cz
Link: https://lore.kernel.org/20260916182957.7788-5-brads@mainlining.org
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Cc: Guenetr Roeck <linux@roeck-us.net>
Cc: Sashiko <sashiko-bot@kernel.org>
Cc: Wang Jinchao <wangjinchao600@gmail.com>
Cc: Wim Van Sebroeck <wim@linux-watchdog.org>
Cc: <stable@vger.kernel.org>
|
|
vsnprintf() consumes the caller's va_list. When the redirect fails,
vpanic() reuses it for the panic message, which is undefined behavior.
Use va_copy().
Link: https://lore.kernel.org/20260916182957.7788-4-brads@mainlining.org
Fixes: 2e171ab29f91 ("panic: add panic_force_cpu= parameter to redirect panic to a specific CPU")
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Cc: Guenetr Roeck <linux@roeck-us.net>
Cc: Sashiko <sashiko-bot@kernel.org>
Cc: Wang Jinchao <wangjinchao600@gmail.com>
Cc: Wim Van Sebroeck <wim@linux-watchdog.org>
Cc: <stable@vger.kernel.org>
|
|
panic() is __noreturn, so the else after panic_try_start() is dead. Drop
it so the force_cpu path can be added cleanly on top.
Link: https://lore.kernel.org/20260916182957.7788-3-brads@mainlining.org
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Cc: Guenetr Roeck <linux@roeck-us.net>
Cc: Sashiko <sashiko-bot@kernel.org>
Cc: Wang Jinchao <wangjinchao600@gmail.com>
Cc: Wim Van Sebroeck <wim@linux-watchdog.org>
Cc: <stable@vger.kernel.org>
|
|
Patch series "panic: fix panic_force_cpu= redirect races and NMI bypass",
v7.
The panic_force_cpu= parameter redirects a panic to a specific CPU so the
crash kernel runs there. The redirect code in panic_try_force_cpu() had
two races and an NMI bypass, all found by Sashiko. This series closes
them and kills one more hack that the rework turned up.
Panic from NMI still goes through the redirect path first, so the crash
kernel ends up on the CPU the admin asked for. The redirect buffer is a
static 1KB now, no initcall, no fallback message.
This patch (of 6):
The cmpxchg() in panic_try_force_cpu() makes sure that only one CPU tries
to redirect panic() to the requested CPU. It is similar to the cmpxchg()
in panic_try_start() which makes sure that only one CPU does the panic().
In both situations, only the winner of cmpxchg() should proceed further.
Other CPUs should go offline.
There is a bug because the cmpxchg loser returns false and falls through
into vpanic(). Two non-target CPUs A and B panic, the requested CPU is C:
cpu A cpu B
---------- ----------
panic() panic()
vpanic() vpanic()
panic_try_force_cpu() panic_try_force_cpu()
cmpxchg wins cmpxchg fails
redirect = A old_cpu = A
IPI -> C return false <- BUG
return true panic_try_start() wins
panic_smp_self_stop() __crash_kexec() on B
(A stops) (target C bypassed)
The loser must stop, not fall through. It cannot just return true,
though. A CPU that already won the redirect cmpxchg can reenter
panic_try_force_cpu() on the same CPU, for example a nested NMI during the
message formatting, before the IPI is sent:
cpu A (1st) cpu A (nested)
---------- ----------
panic()
vpanic()
panic_try_force_cpu()
cmpxchg wins (redirect = A)
vsnprintf(msg) ...
<-- NMI, nested panic -->
panic()
vpanic()
panic_try_force_cpu()
cmpxchg fails
old_cpu == A (this CPU)
return true <- would halt
panic_smp_self_stop()
(IPI never sent, panic abandoned)
Check old_cpu against this_cpu so a second call from the same CPU returns
false and falls through to panic_try_start() instead.
Also fix the panic_in_progress() check. We must not redirect when
panic_cpu is already assigned. Return true to stop when the panic is on
another CPU, false to proceed when it is this one.
Update the panic_try_force_cpu() doc comment for the new return value
semantics.
Link: https://lore.kernel.org/20260916182957.7788-1-brads@mainlining.org
Link: https://lore.kernel.org/20260916182957.7788-2-brads@mainlining.org
Fixes: 2e171ab29f91 ("panic: add panic_force_cpu= parameter to redirect panic to a specific CPU")
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260705164123.18746-1-include@grrlz.net
Closes: https://sashiko.dev/#/patchset/20260707172252.4842-1-include@grrlz.net
Reviewed-by: Petr Mladek <pmladek@suse.com>
Cc: Guenetr Roeck <linux@roeck-us.net>
Cc: Wang Jinchao <wangjinchao600@gmail.com>
Cc: Wim Van Sebroeck <wim@linux-watchdog.org>
Cc: <stable@vger.kernel.org>
|
|
In iomem_map_sanity_check() a piece of code resembles the content of the
resource_overlaps(). Replace open coded piece with the call to the
existing helper.
Link: https://lore.kernel.org/20260915085353.3518153-1-andriy.shevchenko@linux.intel.com
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>
|
|
CPU_DONT_CARE was added by the original listener cpumask code in 2006
(f9fd8914c1ac) and has never had a single user, not even in the
commit that introduced it. It survived every refactor of the listener
path since, including the recent handler folds.
Kill it, twenty years is enough.
Link: https://lore.kernel.org/20260915180623.22058-1-brads@mainlining.org
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
|
|
Pull to receive 2935062be771 ("sched_ext: Wait for SCX_OPSS_DISPATCHING
before moving tasks off DSQs") and resolve its conflicts with the proxy
execution and reject DSQ work here so that for-next merges clean.
Conflicts:
- unlink_dsq_and_switch_rq_lock(): efa7bc7d7195 ("sched_ext: Avoid relocking
DSQ during remote consumption") clears p->scx.dsq in the success branch
and 2935062be771 adds the wait there. Kept both, the wait after the clear,
so that the proxy reject in consume_remote_task() is covered too.
- scx_reenq_reject(): 18f5c699d1d0 ("sched_ext: Rename
scx_reenq_wait_dispatching() to scx_wait_dispatching()") renames the call
in sub.c while cea8b60e4aee ("sched_ext: Move reject DSQ draining into
core") moved the function into ext.c. Dropped the sub.c side and applied
the rename at the new home.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A task can be found on a non-local DSQ while the CPU which dispatched it
is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING.
The DSQ consume and move paths don't wait for the dispatch to finish and
can operate on the task concurrently with the dispatcher.
Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(),
unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they
operate on the task. In scx_move_local_task_to_local_dsq(), wait before
scx_resolve_local_dsq() as it may update p->scx.flags.
Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Fixes: cb86607ada73 ("sched_ext: Don't run ops.dequeue() with a DSQ lock held")
Cc: stable@vger.kernel.org # v7.1+
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue
paths. Any path which takes a task off a DSQ can find it while the
dispatch is still in flight, and the next patch calls the helper from the
DSQ consume and move paths. Rename it to scx_wait_dispatching() and
update its comment accordingly.
No functional change.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Validation logic allowed a BTF_KIND_LOC_PARAM with size <= 4
and vlen == 2, an inconsistent representation.
Fixes: a3df11403e8c8 ("bpf: Verify BTF_KIND_LOC_PARAM vlen, flags")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261012140640.2932195-2-alan.maguire@oracle.com
|
|
update_protect_slice() is only called from wakeup_preempt_fair() when
RUN_TO_PARITY is enabled. Remove RUN_TO_PARITY check inside
update_protect_slice() and use cfs_rq_min_slice() directly.
Signed-off-by: Elif Topuz <elif.topuz@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20261012105001.131311-1-elif.topuz@arm.com
|
|
add_wait_queue_priority_exclusive() sets WQ_FLAG_EXCLUSIVE and
WQ_FLAG_PRIORITY before checking whether another priority waiter is
already present. Thus, when registration fails with -EBUSY, the entry
is left unqueued but with modified flags.
There is no need to modify the entry before the failure check. Move the
flag update after the check so that a rejected registration does not
unnecessarily alter the entry.
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Sean Christopherson <seanjc@google.com>
Link: https://patch.msgid.link/20261011015019.4056529-1-sh_def@163.com
|
|
RB_DECLARE_CALLBACKS_MULTI() asks its user for a function that copies
the augmented fields and one that recomputes them from the children,
with the early-exit protocol of ->propagate() hand-coded in the latter.
sched/eevdf, its only user, needs five helpers for three fields.
Describe each augmented field instead: RB_AUG(val, aug, fold) names the
per-node value, the member holding the subtree aggregate and how two
aggregates combine (min, max, a sum, ...). RB_AUG_FUNC() takes a
function for the per-node value. The template then generates, per
field, a recompute that works on a local and stores once, and a copy,
and combines them into the callbacks; the early exit is the AND of the
per-field results.
RB_DECLARE_CALLBACKS() is the only template left.
RB_DECLARE_CALLBACKS_MAX() becomes RB_AUG_FUNC(RBVALUE, RBAUGMENTED,
max) in a wrapper, with its RBCOMPUTE argument renamed to RBVALUE,
since it returns the per-node scalar and is a different thing from the
RBCOMPUTE of the generic template it used to build on.
RB_DECLARE_CALLBACKS_MULTI() goes away.
sched/eevdf shrinks to a wrapping-safe min() for min_vruntime and
three RB_AUG() lines.
No functional change intended.
[yiwei: take a fold(a, b) instead of a "replace?" compare so that sums
and counts can be expressed too, which also lets min()/max() replace
RB_MIN()/RB_MAX(); wrapped the lines over 100 columns]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Yiwei Lin <s921975628@gmail.com>
Assisted-by: LLM
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20260928213736.GA2947991@noisy.programming.kicks-ass.net
Link: https://patch.msgid.link/20261005093236.62702-3-s921975628@gmail.com
|
|
Pickup dependencies; notably the RB-tree fix.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
|
|
misc.2026.10.06a: Miscellaneous fixes.
torture.2026.10.12a: Make call_{s,}rcu() invocable from any environment,
add atomic SRCU, additional SRCU updates, and torture-test updates.
|
|
Add testtype 7 to verify that lockdep detects an IRQ-context
mismatch when an atomic SRCU read-side critical section is held
with IRQs enabled on one CPU and synchronize_srcu_atomic() is
called from an IPI handler on another CPU.
This covers the cross-CPU case that cannot be detected by
checking the current task's held locks.
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add three testtypes covering atomic SRCU lockdep behavior:
testtype 4: atomic SRCU same-type deadlock
testtype 5: atomic SRCU + raw spinlock dependency cycle
testtype 6: synchronize_srcu_atomic() inside rcu_read_lock()
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The rcu_torture_reader() drives RCU readers from interrupt context via
smp_call_function_single(cpu, rcu_torture_irq, NULL, 0) with wait=0, to
runs rcu_torture_irq() on a remote CPU. this is async, nothing waits for
the remote handler to run.
On shutdown, torture_stop_kthread() only waits for each reader kthread to
return, and the reader's timer_delete_sync() only drains its timer. Neither
waits for a rcu_torture_irq() which still pending or executing on a remote
CPU, so it can run after all readers have exited and rcu_torture_cleanup()
has already advanced.
1. rcu_torture_irq() may issue cur_ops->call(rhp, rcu_torture_timer_cb)
after cur_ops->cb_barrier() has been waiting for all outstanding
callbacks complete. once the module is unloaded, fires into freed
module text, a use-after-free happen.
2. rcu_torture_irq() may still be inside rcu_torture_one_read(), holding
a read-side critical section, when cur_ops->cleanup() tears the flavor
down (e.g. cleanup_srcu_struct()), triggering an active-reader warning
or use-after-free of the torn-down structure.
This commit therefore issue a kick_all_cpus_sync() after all readers
kthread have returned and before cur_ops->cb_barrier(), synchronous IPI
round trip to every CPU guarantees that every rcu_torture_irq() which
previously issued by any reader has completed.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When fwd_progress_div=1, the forward-progress test computes:
sd4 = (sd + div - 1) / div = sd
dur = sd4 + torture_random(&trs) % (sd - sd4) = sd4 + % 0
The modulo operation with a zero divisor triggers an integer
division by zero (undefined behavior at the C level, #DE trap on x86),
causing a kernel Oops and panic. On x86_64, this manifests as:
rcu_torture_fwd_prog_nr: Starting forward-progress test 0
Oops: divide error: 0000 [#1] SMP PTI
RIP: 0010:rcu_torture_fwd_prog+0x90b/0x1160
R12: 0000000000000000
The existing guard only handles non-positive values. However,
fwd_progress_div=1 also makes the random range empty because
sd4 == sd.
Change the guard to reject values below 2. The forward-progress test
only reaches this calculation when stall_dur() is positive, so
sd = stall_dur() + 1 >= 2. For fwd_progress_div >= 2, sd4 < sd,
ensuring that sd - sd4 is at least 1.
Keep the existing fallback to the default value of 4 for invalid
values.
Verified with QEMU/KVM: a 138-second run with fwd_progress_div=1
completed 81 forward-progress test cycles without a crash.
Fixes: 1b27291b1ea4f ("rcutorture: Add forward-progress tests for RCU grace periods")
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
synchronize_srcu_atomic() may be called with interrupts disabled, but
its callchain uses raw_spin_lock_irq_rcu_node() /
raw_spin_unlock_irq_rcu_node(), which unconditionally enables
interrupts on unlock. This can incorrectly change the caller's IRQ
state.
Use the irqsave/irqrestore variants for the locks in this callchain
so that the caller's IRQ state is preserved.
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add the RCU_LOCKDEP_WARN and early-boot check that the Tree version
gained, srcu_lock_sync() was already present.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add the lockdep annotation, same-type SRCU nesting warning, and
early-boot check used by __synchronize_srcu().
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Atomic SRCU does not use the srcu_node combining tree. Exclude it
from the SRCU_SIZING_IS_TORTURE() transition in
srcu_torture_stats_print(), and warn if such a transition is requested.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Atomic SRCU must remain in the small size state. Warn if this
invariant is violated and avoid transitioning to big in that case.
[ paulmck: Folded "if" onto one line for 100-character limit. ]
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
|
|
The is_atomic parameter of init_srcu_struct_fields() exists so that
atomic SRCU never transitions to big, but neither
init_srcu_struct_atomic() nor its lockdep counterpart
__init_srcu_struct_atomic() sets it.
On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
this needlessly allocates a full srcu_node combining tree for any
dynamically initialized atomic srcu_struct, despite atomic SRCU
having neither callbacks nor srcu_barrier() operations.
Pass true from both atomic entry points, adding an is_atomic parameter
to __init_srcu_struct_common().
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The atomic-SRCU path in srcu_barrier() sleeps for 100 milliseconds just
in case there are callbacks to wait for. But call_srcu() refuses
atomic SRCU with a WARN_ON_ONCE() before reaching the deferred-enqueue
path, so there can be no callbacks, deferred or otherwise.
Drop the sleep.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_srcu() is forbidden on atomic srcu_struct, so srcu_gp_end() never
has callbacks to invoke for them. Yet it schedules callback invocation,
which for atomic SRCU's SRCU_SIZE_SMALL state arms the boot CPU's
->delay_work timer every grace period, only for srcu_invoke_callbacks()
to find nothing to do.
Skip this for atomic SRCU.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
srcu_read_unlock_atomic(), whose read-side critical sections disable
preemption. In the common case where there are no readers at all, the
grace period therefore need not do the index flip. Add a fastpath
that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
lock counts match the unlock counts on both ranks, ends the grace
period immediately, skipping the srcu_advance_state() scans, mirroring
the similar Tiny SRCU fastpath.
Correctness requires the counter-sum proof to follow the grace-period
anchor written by srcu_gp_start(); placing it before the anchor could
let this grace period miss a pre-existing reader and return without
waiting for it. The smp_mb() between the unlock and lock sums pairs
with the smp_mb() in __srcu_read_lock(). The grace period is ended
manually under ->lock and ->srcu_atomic_gp_flag.
In theory, this is slower than David Woodhouse's earlier patch, but
David's measurements showed that the performance was close enough
that it makes sense to keep the get_state_synchronize_srcu() and
poll_state_synchronize_srcu semantics.
Link: https://lore.kernel.org/all/20260907075829.2073224-4-kunwu.chan@linux.dev/
Link: https://lore.kernel.org/all/0f4dea21bac43685d3286df329401177e4452b36.camel@infradead.org/
Link: https://lore.kernel.org/all/0d4af6318ac67486858be1df8d436147b444a2d2.camel@infradead.org/
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: David Woodhouse <dwmw2@infradead.org>
|
|
The fastpath of synchronize_srcu_atomic() should always be taken
because readers always disable preemption. Therefore, the fact that
synchronize_srcu_atomic() is running at all should mean that all readers
have ended.
But there are always bugs. And responding to a usage bug with a
too-short SRCU grace period, and thus possibly corrupting memory
is at best a sadistic response so such a bug. For this reason,
synchronize_srcu_atomic() explicitly waits for readers. Except that
it does so silently, possibly failing to flag this bug. This commit
therefore adds a splat if synchronize_srcu_atomic() fails to take the
early exit.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The irq_work in srcu is used to schedule a delayed work. The work item
is not scheduled directly because it is not always possible wake a
thread directly.
On PREEMPT_RT the default irq_work is initialized with IRQ_WORK_LAZY and
is delayed to the irq_work thread. A system booted with the command line
"trace_event=…" will freeze during boot because the irq_work thread is
not yet deployed (and tracing uses synchronize_srcu() in
tp_rcu_cond_sync()).
The irq_work performs just a wakeup a thread, there is nothing wrong
with doing this from hardirq context on PREEMPT_RT.
Use IRQ_WORK_INIT_HARD for srcu's irq_work.
Fixes: 7c405fb3279b3 ("rcu: Use an intermediate irq_work to start process_srcu()")
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Because synchronize_srcu_atomic() cannot sleep, if a pair of them run
concurrently pinned to the same CPU, it is possible that one will spin
uselessly waiting for the other while at the same time preventing that
other from running. This commit therefore disables preemption to prevent
this failure mode.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds support for the value 0x10 for reader_flavor, which
specifies SRCU_READ_FLAVOR_ATOMIC, that is, srcu_read_lock_atomic(),
srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
[ paulmck: Apply Kunwu Chan feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the Tiny SRCU counterpart to Tree SRCU's
synchronize_srcu_atomic(). One might hope that this could be as trivial
as Tiny RCU's synchronize_rcu(), and there was a time when it would
have been. But lazy preemption really can preempt an SRCU read-side
critical section, which means that synchronize_srcu_atomic() really must
be prepared to spin waiting for it.
This spinning currently consists of cond_resched_tasks_rcu_qs()
and cpu_relax(). It would be better to have some way of telling the
scheduler that there is nothing useful for us to do. We cannot use
the traditional wait_event() approach because synchronize_srcu_atomic()
is not permitted to block.
[ paulmck: Apply Kunwu Chan and Joel Fernandes feedback. ]
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Some dedicated srcu_struct users have read-side critical sections
which are short, never sleep, and never block on anything which may
itself depend on memory allocation — because they were, until
recently, spinlock or rwlock critical sections. For such a domain the
update side can safely wait for readers by spinning, from contexts
where sleeping is undesirable or the grace-period machinery's
latency (workqueue scheduling, jiffy-paced retries) dominates the
actual reader drain time.
But that is only safe if every reader keeps the promise. Make the
promise explicit and machine-checkable:
- srcu_read_lock_atomic() / srcu_read_unlock_atomic() enter the
usual (smp_mb-based) read-side critical section with preemption
disabled and (except in hardirq, where it is redundant)
non_block_start() armed, recording SRCU_READ_FLAVOR_ATOMIC in the
per-CPU reader flavor. Disabling preemption enforces the
no-sleeping promise on every configuration and bounds the section,
so it is always running on some CPU; non_block_start() extends the
enforcement to even potentially-sleeping calls on paths which
happen not to block. The existing reader-flavor consistency checks
complain about any mixing with other flavors.
- synchronize_srcu_atomic() waits for all pre-existing readers by
repeating the try_synchronize_srcu() both-epoch counter proof with
cpu_relax() until it succeeds: no sleeping, no index flip, no
grace-period sequence update, and therefore no interaction with
concurrent call_srcu(), synchronize_srcu() or srcu_barrier(). It
always provides the full grace-period guarantee: if the domain
turns out to have had readers of any other flavor — a caller bug,
since such a reader may be asleep and spinning on it would be
unbounded — it complains and falls back to a real (sleeping) grace
period internally, that being the only correct wait for a
possibly-sleeping reader. The flavor mask is rechecked on every
iteration so a first non-atomic reader appearing mid-spin takes
the same path.
The immediate motivation is the proposed conversion of KVM's
gfn_to_pfn_cache to SRCU¹, whose mmu_notifier invalidation path drains
readers before the primary MMU zaps a page. With the readers declared
atomic, that drain becomes spin-only: no sleeping at all in the
notifier, bounded by the longest reader section, satisfying even the
strictest reading of the OOM-reaper non-blocking requirement without
needing to touch the non_block_start() annotation².
This commit also adds atomic-SRCU-specific initializers:
init_srcu_struct_atomic(), DEFINE_SRCU_ATOMIC(), and
DEFINE_STATIC_SRCU_ATOMIC().
This commit implements only Tree SRCU. Tiny SRCU will follow.
¹ https://lore.kernel.org/all/20260811132237.102400-1-dwmw2@infradead.org/
² https://lore.kernel.org/all/20260812134934.GC662699@ziepe.ca/
[ paulmck: Apply Kunwu Chan feedback. ]
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The restricted select_cpu kfunc path handles single-CPU and
migration-disabled tasks without going through scx_select_cpu_dfl().
It currently uses prev_cpu as the candidate, even though prev_cpu is only a
hint and may no longer be usable after an affinity change. With a
caller-provided @allowed mask, this can also return an idle CPU outside
p->cpus_ptr.
Derive the candidate from task state instead: use the sole CPU in
p->cpus_ptr for single-CPU tasks and task_cpu(p) for migration-disabled
tasks. Validate it against p->cpus_ptr and @allowed before claiming its
idle state.
Since the derived candidate may be on a different NUMA node from
prev_cpu, also preserve SCX_PICK_IDLE_IN_NODE by rejecting cross-node
candidates.
Perform all checks before scx_idle_test_and_clear_cpu(), as a successful
call consumes the CPU's idle bit.
Fixes: 683d2d0faba1 ("sched_ext: idle: Introduce scx_bpf_select_cpu_and()")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 1765a153d985 ("cgroup/pids: Restore pids.events
notifications in local mode") for the pids.events notification
selftest.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_bypass() claims bypass dispatch when the scheduler's depth goes from
zero to one and releases it when the depth returns to zero, but propagates
only the depth to the descendants, not the claim. A descendant that starts
bypassing with its ancestor never claims, and one whose own bypass ends
under the ancestor's keeps its claim, as its depth does not return to zero.
A sub-scheduler enable whose bypass ends under a PM suspend thus leaks its
claim: the parent keeps hosting bypass dispatch and the sub-scheduler's next
bypass trips the WARN_ON_ONCE() on the claim. A sub-scheduler disable that
starts under a PM suspend has no claim, so once the resume lifts the root's
bypass its tasks sit on the parent's bypass DSQs with nobody consuming them.
Claim for every descendant whose depth goes from zero to one in the
propagation and release every scheduler in the subtree whose depth returned
to zero once the tasks have moved out of the bypass DSQs. A parent that is
bypassing itself when a descendant claims consumes only its own bypass DSQs,
empty until it leaves bypass and becomes the host.
Fixes: aa2a0a19686c ("sched_ext: Separate bypass dispatch enabling from bypass depth tracking")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_bypass() propagates a depth change to the descendants only when the
scheduler's own depth crosses zero, so an own bypass that starts under an
ancestor's never reaches them. When the ancestor's bypass ends first, the
descendants leave bypass while the scheduler is still bypassing: a
sub-scheduler disabled under a PM suspend has its children re-queue their
tasks through their own BPF schedulers at resume, mid-teardown. If the own
bypass then ends, every descendant is decremented once more than incremented
and dec_bypass_depth() warns, though no current caller gets there.
Propagate on every call so that a descendant's depth is the sum of its own
bypasses and its ancestors'. The early return follows the walk, as a
descendant's depth is at least the scheduler's and nothing changes bypass
state unless the scheduler's own does.
Fixes: c7f0e467a27a ("sched_ext: Prepare bypass mode for hierarchical operation")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
kernel-doc
btf_find_struct_member() returns the containing structure or union type
(which can be an inner anonymous struct/union rather than the outer
passed type) via its last parameter. However, the parameter was named
@member_type, which easily confuses readers and static analyzers into
thinking it returns the member field's own datatype.
Rename @member_type to @container_type in the prototype, definition,
and callers to make the relationship explicit. Also convert the comment
above btf_find_struct_member() to standard kernel-doc format, documenting
all parameters and return values.
Link: https://lore.kernel.org/all/179030954365.77418.17193777972248047975.stgit@devnote2/
Assisted-by: LLM
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
KCSAN reports pid_alive() reading task->thread_pid while it gets cleared
under tasklist_lock. The check only cares about NULL, so READ_ONCE() and
WRITE_ONCE() are enough.
Reported-by: syzbot+c382ee653fd70f5cf1bb@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=c382ee653fd70f5cf1bb
Signed-off-by: Babanpreet Singh <bbnpreetsingh@gmail.com>
Link: https://patch.msgid.link/20261002012141.7-1-bbnpreetsingh@gmail.com
Acked-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
map_write() parses each line written to uid_map, gid_map or projid_map
with simple_strtoul() and stores the results in the u32 fields of
struct uid_gid_extent. On 64-bit kernels a value above U32_MAX is
silently truncated modulo 2^32. The coding assistant flagged this
truncation site while reading map_write(), and I confirmed it by hand
on a running kernel, where writing to the uid_map of a fresh user
namespace installs a different mapping than the one requested:
"4294967296 1000 1" -> installed as "0 1000 1"
"4294967297 1000 1" -> installed as "1 1000 1"
"4294967301 1000 1" -> installed as "5 1000 1"
The same happens to the lower_first and count fields. Only a literal of
2^64 or more is rejected, and only because simple_strtoul() saturates
to ULLONG_MAX, which truncates onto the (u32)-1 value map_write()
already refuses. On 32-bit kernels unsigned long is itself 32 bits
wide, so the same truncation happens inside simple_strtoul().
The write succeeds, so a runtime that computes ranges and gets one
wrong ends up with a different map than it asked for and no error to
tell it so.
Parse each field with kstrtou32() instead. A new helper,
map_parse_u32(), terminates the field at the next whitespace or NUL,
parses it and puts the original character back, leaving pos where
simple_strtoul() used to leave it. The checks after each field, the
trailing junk check and every later check are unchanged and run in the
same order. A field that is not a decimal number fitting in u32 now
fails the write with -EINVAL, like any other malformed line.
Besides the range check, the one visible difference is that kstrtou32()
accepts a single leading '+', so "+5 1000 1" is now installed as
"5 1000 1" where it used to be rejected.
This makes the parser more correct; it is not a security fix: the
truncated values are valid ids and every later check still applies to
them.
Suggested-by: Eric W. Biederman <ebiederm@xmission.com>
Signed-off-by: Natasha Klaus <natalie.klaus@runtimeverification.com>
Link: https://patch.msgid.link/20260930112854.373184-2-natalie.klaus@runtimeverification.com
Acked-by: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
check_dup_ids() rejects equal BTF ids, but different ids can resolve to
the same address. Aliases:
ffffffff8161b6b0 t __do_sys_munlockall
ffffffff8161b6b0 T __ia32_sys_munlockall
ffffffff8161b6b0 T __x64_sys_munlockall
or several BTF_KIND_FUNC with the same name that vmlinux BTF has when
CONFIG_DEBUG_INFO_BTF_INLINE is set.
Each id gets its own trampoline and register_fentry_multi() adds them to
data->reg with the same ip. update_ftrace_direct_add() checks that new
entries are not in direct_functions, but doesn't compare them with each
other, so the attach succeeds. At detach update_ftrace_direct_del() sees
that 'direct' of one of the entries doesn't match and returns -EINVAL:
WARNING: kernel/bpf/trampoline.c:1845 at bpf_trampoline_multi_detach+0x24c/0x280
Call Trace:
bpf_link_free+0x51/0x110
bpf_link_release+0x18/0x20
__fput+0x103/0x2d0
__x64_sys_close+0x3d/0x80
The images are freed anyway, while ftrace still calls them:
Oops: int3: 0000 [#1] SMP NOPTI
RIP: 0010:0xffffffffc042b201
Call Trace:
? __audit_syscall_exit+0x9/0x100
? do_syscall_64+0x5ab/0x5e0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Kernel panic - not syncing: Fatal exception in interrupt
Return -EINVAL from register_fentry_multi() when ip is already in
data->reg. When such ids are attached via different links the second
attach already fails with -EINVAL in update_ftrace_direct_add().
Move the declaration of ftrace_lookup_ip() to include/linux/ftrace.h.
The lookup adds 60 msec to the attach of 64480 functions in
tracing_multi_bench_attach: 1.34 -> 1.40 sec.
Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions")a
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20261011164836.2233874-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
format_disasm_line() strips trailing newlines from the disassembly it
stores, as diag_print_insn_context() adds its own. Since
commit 483a1bb0b6cf8 ("bpf: Do not print a newline after disassembly in
bpf_verbose_insn()"), print_bpf_insn() does not emit one, so there is
nothing left to strip.
The strnlen() resync of the seq_buf length only existed so the loop
could index the buffer safely after an overflow, and the second
termination only restored the NUL that seq_buf_pop() does not write.
Remove all of it, keeping a single seq_buf_terminate().
Build tested ARCH=x86_64 defconfig with GCC 16.2.0 and
CONFIG_BPF_SYSCALL=y.
Assisted-by: LLM
Acked-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261005155708.1471260-8-kees@kernel.org
Signed-off-by: Kees Cook <kees@kernel.org>
|
|
Seven callers call seq_buf_str() only to NUL-terminate the buffer,
discarding the returned pointer. Two of them need a comment to say so.
Add seq_buf_terminate(), wrapping the __seq_buf_terminate() helper that
seq_buf_str() and seq_buf_strlen() already use, and convert those
callers. It returns void: returning the offset would just be
seq_buf_strlen() under another name. A zero-sized seq_buf is left
untouched, as in the other accessors.
Add tests for the three cases: room for the NUL after the data, an
overflowed buffer where it lands in the last byte, and a zero-sized
buffer that must not be written to.
Build tested ARCH=x86_64 defconfig with GCC 16.2.0, plus
CONFIG_HIST_TRIGGERS=y and CONFIG_BPF_SYSCALL=y to reach the converted
call sites in kernel/trace/trace_events_hist.c and
kernel/bpf/diagnostics.c. Tests run 24/24 passing on ARCH=um.
Assisted-by: LLM
Acked-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261005155708.1471260-7-kees@kernel.org
Signed-off-by: Kees Cook <kees@kernel.org>
|
|
Module .BTF.inline is split relative to its module BTF. For external
modules, module BTF can instead be split relative to a distilled
.BTF.base and is relocated against vmlinux when the module loads.
Parse ordinary module inline BTF relative to the pre-relocation
module BTF. Retain the module relocation ID and string maps, then
rebase inline type IDs and string offsets onto the final
vmlinux/module BTF space before exposing the result through sysfs.
Malformed or unrelocatable optional inline BTF is warned about and
omitted without preventing normal module BTF registration. The
btf_vmlinux_inline carrier remains on its existing hash-checked,
vmalloc-backed publishing path, preserving mmap support for
vmlinux.inline when delivered by a module.
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009112324.1442319-6-alan.maguire@oracle.com
|
|
Make btf_relocate() optionally return the string-offset map it builds
while relocating a BTF relative to a distilled base. A subsequent split
BTF object can use that map to rebase references to base strings.
Also account for the accumulated base string offset when assigning a
new BTF base. This is required when the base is itself split.
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009112324.1442319-5-alan.maguire@oracle.com
|
|
Expose BTF inline-location information from vmlinux .BTF.inline
sections in /sys/kernel/btf as vmlinux.inline. Embedded vmlinux
inline BTF supports read-only mmap().
Provide runtime support for a module-delivered vmlinux inline BTF
carrier. Its link record binds the carrier module name, BTF size, and
SHA-256 digest to the running kernel before its data is made available
through a lazy vmlinux.inline sysfs attribute. The carrier is handled
before ordinary module BTF registration, so its incidental .BTF data is
not exposed.
Make CONFIG_DEBUG_INFO_BTF_INLINE tristate. When configured as a
module, retain vmlinux final split inline BTF for
btf_vmlinux_inline.ko instead of embedding it in vmlinux.
Reserve .BTF.inline.link in vmlinux and extend the final
resolve_btfids patch pass to bind btf_vmlinux_inline to the raw
vmlinux.BTF.inline payload. This preserves the normal built-in
configuration while allowing the large inline BTF payload to load on
demand.
Normal module inline BTF is handled separately, after its
references have been relocated against the final module BTF.
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009112324.1442319-4-alan.maguire@oracle.com
|
|
For whatever reason gcc built-ins implementing 64-bit division are not
wired into 32-bit kernel builds:
ld: kernel/bpf/scev.c:2074:(.text+0x4512): undefined reference to `__udivdi3'
ld: kernel/bpf/scev.c:2141:(.text+0x475b): undefined reference to `__udivmoddi4'
Instead, one has to use div64_u64() and div64_u64_rem() explicitly.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Reported-by: Tejun Heo <tj@kernel.org>
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20261010205828.3366892-1-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Since commit d3539347022a ("serial: 8250: Switch to nbcon console, take
2"), several KCSAN tests fail with CONFIG_KCSAN_VERBOSE=y when using the
8250 console.
With CONFIG_KCSAN_VERBOSE=y, the report prints the racing tasks' IRQ
trace events via print_irqtrace_events(), which marks its output as a
printk emergency section. Within an emergency section, printk() flushes
all pending messages to nbcon consoles synchronously. With a slow
console, the reporting thread can therefore stall for over a second
while holding report_lock, i.e. before the remaining lines of the report
are stored.
The test, however, stops checking for reports after a fixed deadline,
which is easily exceeded by a single slow report:
| [ 5.360278] BUG: KCSAN: data-race in test_kernel_read / test_kernel_write
| [ 5.360303] write to 0xffffffffab95a478 of 8 bytes by task 237 on cpu 7:
| ...
| [ 5.360411] irq event stamp: 23
| [ 6.059836] # test_basic: EXPECTATION FAILED at kernel/kcsan/kcsan_test.c:735
| ...
| [ 6.616416] read to 0xffffffffab95a478 of 8 bytes by task 238 on cpu 9:
Waiting for in-progress reports to complete is insufficient: a slow
report that does not match the expected report still uses up the time
to observe the expected report.
Fix it by pausing the test's time budget while a test-related report is
being printed, bounded by an upper limit to avoid stalling the test
forever. Once a report completes, always check it at least once.
Assisted-by: Antigravity
Signed-off-by: Marco Elver <elver@google.com>
|
|
Pull to receive 069403dff337 ("sched_ext: Fix use-after-free of a
destroyed user DSQ by a deferred reenqueue") and point its new cancel
sweep in free_dsq_irq_workfn() at dsq->pcpu_user, which 92344fad5b54
("sched_ext: Skip per-CPU data allocation for built-in DSQs") renamed
here, so that for-next builds.
Signed-off-by: Tejun Heo <tj@kernel.org>
|