| Age | Commit message (Collapse) | Author |
|
The signature of a program covers its insns as submitted, along with the
frozen metadata maps of a signed loader, and bpf_prog_verify_signature()
checks it before the verifier does anything else with the program. CO-RE
relocations are the one input besides the insns that changes them: the
loader hands them in with the program, and bpf_check_core_relo() applies
them after the signature check, rewriting the immediate or the offset of
every insn a record names with a value the loader chooses, a field
offset, a type size or id or an enum value of the kernel BTF, or a type
id of the BTF the loader supplies itself. Whoever holds a signed program
can so change its constants once the signature has passed, the offsets a
signed loader reads its metadata map with for instance, and run a
program the signer never saw. The exclusive metadata map of a signed
loader does not make up for it: the hash that binds the map to a program
is chosen by whoever creates the map, so it can name the relocated
program as well.
Refuse a signed load that carries relocations, before the signature is
examined, like the other checks on the shape of a signed load. A signed
loader has none to lose: the programs it embeds are relocated by the
loader itself through bpf_core_apply(), and nothing relocates the loader.
The BTF of the program itself can stay: without relocations it steers
only the verification, through func_info, line_info and the annotations
of global function arguments, and rewrites nothing.
Fixes: 349271568303 ("bpf: Implement signature verification for BPF programs")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://patch.msgid.link/20261019144851.4052056-2-memxor@gmail.com
|
|
kernel/bpf/Makefile builds arena.o only for CONFIG_MMU && CONFIG_64BIT,
but special_kfunc_list references bpf_arena_alloc_pages() and
bpf_arena_free_pages() unconditionally, so on 32-bit and !MMU kernels
resolve_btfids can't resolve them:
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
WARN: resolve_btfids: unresolved symbol bpf_arena_alloc_pages
Since commit 201a454df948 ("resolve_btfids: Fix the --fatal_warnings
exit status") these warnings fail the build with CONFIG_WERROR=y, which
breaks i386 builds with CONFIG_DEBUG_INFO_BTF=y:
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
Only reference them under the condition arena.o is built with, and use
BTF_ID_UNUSED otherwise to keep the list in sync with enum
special_kfunc_type. An unresolved ID is zero, like BTF_ID_UNUSED, so the
contents of the list don't change.
Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019211006.2574350-3-sashal@kernel.org
|
|
Commit 6f033615ef8f ("bpf: Trim special_kfunc_list in verifier") dropped
bpf_arena_reserve_pages, bpf_stream_vprintk and bpf_stream_print_stack
from special_kfunc_list once nothing used them anymore, but commit
ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path")
brought all three back without adding any users.
Besides being dead, the bpf_arena_reserve_pages entry can't be resolved
on 32-bit and !MMU kernels, which don't build arena.o, and since commit
201a454df948 ("resolve_btfids: Fix the --fatal_warnings exit status")
the resulting resolve_btfids warning fails the build with
CONFIG_WERROR=y:
WARN: resolve_btfids: unresolved symbol bpf_arena_reserve_pages
Drop the three entries again.
Fixes: ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019211006.2574350-2-sashal@kernel.org
|
|
Building kernel/trace/trace_osnoise.o with W=1 triggers the following
compiler warning:
kernel/trace/trace_osnoise.c:96:17: warning: function 'osnoise_print'
might be a candidate for 'gnu_printf' format attribute
[-Wsuggest-attribute=format]
osnoise_print() forwards its format string and va_list arguments to
trace_array_vprintk(), which is annotated with __printf(3, 0).
Add the __printf(1, 2) attribute to osnoise_print() to silence the W=1
warning and enable compile-time format string checking for its callers.
Link: https://patch.msgid.link/20261019205344.190022-1-adriangarciacasado42@gmail.com
Signed-off-by: Adrián García Casado <adriangarciacasado42@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The SCX_KICK_WAIT wait is deferred to rq->scx.kick_sync_bal_cb, which a
final pick queues on the local rq. Under core scheduling, a pick that
returns a task isn't final: pick_next_task() goes on to pick the SMT
siblings, and their dispatch can release the shared rq lock with the
callback still queued. A CPU acquiring the rq lock in that window then
finds a pending balance callback and triggers the warning in
rq_pin_lock().
Fix this by queuing the callback from set_next_task_scx() on SNT_PICK
and SNT_REPICK. put_prev_set_next_task() calls it once the pick is
final, after the core-wide selection, so the rq lock can't be dropped
with the callback queued. If the CPU ends up running idle or a higher
class task, no SCX task runs there and there's nothing to wait for:
kick_sync_pending stays set and the next pick of an SCX task waits
before running it.
Also, proxy exec zaps the balance callbacks and picks again when
find_proxy_task() fails, so clear kick_sync_pending when the callback
runs rather than when it's queued.
Fixes: 415cb193bb97 ("sched_ext: Fix SCX_KICK_WAIT deadlock by deferring wait to balance callback")
Cc: stable@vger.kernel.org # v7.0+: c72945693b90: sched: Restart fair hrtick after same-task repicks
Cc: stable@vger.kernel.org # v7.0+
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
cgroup.kill does not work in case a process' thread group leader is gone
but other threads are still running. This is because __cgroup_kill()
only walks the thread group leaders and sends SIGKILL to each of them
with send_sig(), which uses PIDTYPE_PID:
1. The exited leader is still listed by CSS_TASK_ITER_PROCS as long as
the process has live threads, but a signal directed at a single
exiting task never takes effect: complete_signal() does not look for
another thread to deliver it to, and __send_signal_locked() now drops
it altogether. The process keeps running.
2. The live threads of such a process can be migrated to another cgroup
by writing any of their IDs to cgroup.procs, while the exited leader
stays behind. The destination cgroup has no thread group leader to
iterate, so writing to its cgroup.kill does nothing at all.
In both cases the cgroup stays populated after cgroup.kill, and a
process inside a container can use this to survive the container being
killed. This also works with a private PID namespace when the init of
the container is the process, as the PID namespace is not torn down
while init has live threads.
Case 1 can be fixed by using do_send_sig_info(..., PIDTYPE_TGID) instead
of send_sig(), so that the signal is taken by a live thread even if the
task found by the iterator is exiting. Note it was pointed out by Eric
in [1] while reviewing v2 of cgroup.kill patchset, and v3 was meant to
switch to it, but the change was only made in cgroup_post_fork(),
leaving __cgroup_kill() with send_sig().
Case 2 can only be fixed by iterating over all threads, so this is what
the fix also does. Once a thread group is being killed, its other
threads are skipped since they already have a fatal signal pending.
I ran some performance checks and the added cost is ~60 ns per every
thread on my laptop (and less than a millisecond for 10K threads),
which seems low. The time to "populated 0" seems unchanged.
The issue was found in crun by Anthropic's OSS Scanner, and crun is
getting a workaround for it [2] (which does what we needed to do
before cgroup.kill was added).
[1] https://lore.kernel.org/all/m1v97x6niq.fsf@fess.ebiederm.org/
[2] https://github.com/containers/crun/pull/2351
Fixes: 661ee6280931 ("cgroup: introduce cgroup.kill")
Cc: stable@vger.kernel.org # v5.14+
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Although it is not typical for real users to create a complicated nested
partition hierarchy, it is currently allowed to create nested root and
isolated partitions. In addition, disabling or invalidating a partition
should always be allowed especially when the cpuset is being deleted.
With a nested root and isolated partition hierarchy, it is possible
that all active housekeeping CPUs can be exhausted when disabling or
invalidating a root partition and returning its CPUs to the isolated
parent.
One way to avoid this problem is to disallow the creation of a root
partition under an isolated parent. It is indeed a new limitation imposed
on the existing cpuset partition functionality. However an isolated
partition is usually a left node where scheduling and some other kernel
noises are disabled. It doesn't make much sense to create a child root
partition under an isolated parent and remove some CPUs from the set
of isolated CPUs again. Creation of a child isolated partition under a
root or isolated parent is still allowed.
If a partition with child root partitions is converted to isolated,
the child root partitions will be invalidated.
The cgroup-v2 documentation is also updated to reflect this new
limitation. New test cases are added to test_cpuset_prs.sh to verify
this new behavior. Some existing test cases are also updated accordingly.
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Before commit 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full
don't leave any housekeeping"), the partition state can be freely switched
from "root" to "isolated" and vice versa. After that commit, the switch
from "root" to "isolated" can fail if it exhausts all the housekeeping
CPUs. Later on, the switch from "isolated" to "root" can also fail if
some of the partition CPUs are boot-time isolated by "isolcpus".
A local or remote partition is made invalid when the switch fails.
However there are 2 problems with the existing invalidation code.
1) Both the subpartitions_cpus and isolated_cpus are not properly
updated.
2) In the case of remote partition with remote_partition flag set, it
is not cleared.
Fix this by calling the proper partition disabling function in this case.
On an x86 test system with boot option "isolcpus=10 cgroup_debug" set
and more than 16 cores, the following commands were executed after boot.
# cd /sys/fs/cgroup
# echo +cpuset > cgroup.subtree_control
# cat cpuset.cpus.isolated
10
# mkdir A
# echo 10-12 > A/cpuset.cpus
# echo isolated > A/cpuset.cpus.partition
# cat A/cpuset.cpus.partition
isolated
Before the patch:
# echo root > A/cpuset.cpus.partition
# cat A/cpuset.cpus.partition
root invalid (partition config conflicts with housekeeping setup)
# cat cpuset.cpus.isolated .__DEBUG__.cpuset.cpus.subpartitions
10-12
10-12
After the patch:
# echo root > A/cpuset.cpus.partition
# cat A/cpuset.cpus.partition
root invalid (partition config conflicts with housekeeping setup)
# cat cpuset.cpus.isolated .__DEBUG__.cpuset.cpus.subpartitions
10
Reported-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/lkml/7f4c57b26ad120ab30adf35653f1dd94@kernel.org/
Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup")
Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping")
Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
valid
When a hotplug event happens or when update_cpumasks_hier() is called, an
invalid partition may be converted back to valid. However check against
housekeeping conflict isn't being done which can lead to a partition
state that violates the housekeeping rules. Update the code to allow
reviving an invalid partition only if it can pass the housekeeping check.
Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup")
Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When prstate_housekeeping_conflict() is called to perform housekeeping
check with cpumask changes, the cpumask changes are passed in via add_cpus
and del_cpus parameters. With this arrangement, the exclusive CPUs owned
by the child partitions and not affected by the cpumask change, if any,
are excluded in the analysis. The only places where there are separate
add and delete masks are when the cpuset.cpus or cpuset.cpus.exclusive
is being changed. In both cases, partition_cpus_change() is a good place
to perform the housekeeping check.
If we allow a root partition to be created under an isolated partition,
it is possible the cpumask change will still leave active housekeeping
CPUs but then no housekeeping CPUs will be left when the child partitions
are disabled or invalidated.
As a first step to protect against this possibility, all the exclusive
CPUs of the cpuset should be considered to be owned by the current cpuset
when doing housekeeping check. At the same time, we do want to know if
some housekeeping CPUs are going to be removed from the isolated partition
that there will be no danger of losing all the active housekeeping CPUs.
So the current add_cpus paremeter is renamed as new_cpus to indicate
that it includes the complete set of effective xcpus.
With the new check in partition_cpus_change(), the housekeeping check
in update_parent_effective_cpumask() with partcmd_update and newmask
is now duplicative and can be removed. The housekeeping check in
remote_cpus_update(), however, will still be useful as this function can
be called directly from update_cpumasks_hier() without going through
partition_cpus_change(). The update_parent_effective_cpumask() call
from update_cpumasks_hier() is partcmd_update with no cpumask which
still have the housekeeping check and so is covered.
Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
prstate_housekeeping_conflict()
The isolated_cpus_can_update() and prstate_housekeeping_conflict() are
checking different aspects of upcoming cpumask changes that may conflict
with the current setting of the housekeeping cpumasks. There are places
where both are called together. There are also places where only one
of them is called. That inconsistency can contribute to missing check
where invalid cpumask changes may be allowed to move forward.
Fix that by consolidating isolated_cpus_can_update() into
prstate_housekeeping_conflict() and call prstate_housekeeping_conflict()
in all the places where either one of them or both are called. The
exception is the validate_partition() function where the
prstate_housekeeping_conflict() call is removed. It is because
validate_partition() is called only from partition_cpus_change()
where prstate_housekeeping_conflict() will be called from either
remote_cpus_update() or update_parent_effective_cpumask() with
partcmd_update if not for partition invalidatation or disablement.
Now prstate_housekeeping_conflict() will be called in the following
locations:
- when a partition is enabled in remote_partition_enable() or in
update_parent_effective_cpumask() with partcmd_enable*.
- when a cpumask is updated in remote_cpus_update() or in
update_parent_effective_cpumask() with partcmd_update.
- when a partition state changes from root to isolated or vice versa
in update_prstate().
Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup")
Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping")
Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
It is found that when a local isolated partition is being invalidated
because its parent partition is no longer valid, the isolated_cpus
cpumask may not be properly updated. Those isolated CPUs may or may
not needed to be removed from isolated_cpus depending on if there is
a local isolated partition ancestor that will now own those isolated
CPUs instead.
Fix that by calling a new ancestor_partition() helper to return the
closeset valid partition root ancestor. If that ancestor is PRS_ROOT,
the isolated CPUs will have to be removed from isolated_cpus. For
a PRS_ISOLATED ancestor, the isolated CPUs are checked against
the ancestor's ones to see if any of them should be removed as it is
possible that the invalidation of the parent may be caused by change in
the effective xcpus of that ancestor. If any of the isolated CPUs should
be removed, isolated_cpus_update() will be called under callback_lock.
An existing test case in test_cpuset_prs.sh is modifed and a new one
added to test that.
Fixes: 11e5f407b64a ("cgroup/cpuset: Keep track of CPUs in isolated partitions")
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A cpuset update changes the task's mems_allowed and rebinds its
mempolicy in separate steps under mems_allowed_seq, but the check
reads both without it. This can cause a spurious numa placement
miss under some cpuset/mempolicy combinations.
A reader that loads one mask before the update and the other after
it can see a spurious empty intersection and drop the nodemask.
For example, a task with cpuset.mems={0} and MPOL_BIND={0}. If the
cpuset moves to mems={1,2}, the policy is rebound to {1}.
The allocator can race and compare the old mems_allowed {0} against
the new policy {1} - which will result in a dropped nodemask and a
misplaced allocation on node 2.
Simply retry the intersection until mems_allowed_seq is stable.
This covers the task policy, which is rebound under mems_allowed_seq.
VMA policies are rebound afterwards by mpol_rebind_mm(), outside
mems_allowed_seq, and are not covered by this change.
Fixes: cc9a6c877661 ("cpuset: mm: reduce large amounts of memory barrier related damage v3")
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The watchdog measures how long a task has been waiting from
p->scx.runnable_at, which set_task_runnable() resets when the task becomes
runnable after it last ran. ops_dequeue() never asks for a reset, as the
dequeue of a waiting task is temporary, e.g. for a migration or an
attribute change.
Leaving the scheduler isn't temporary. A task which is switched out of
SCHED_EXT while it's waiting keeps its runnable_at. When it's switched back
and becomes runnable, its wait appears to have started before it left, and
the time it spent in another class counts towards the timeout. If the
watchdog checks before the task runs, it aborts the scheduler:
sched_ext: class_switch_stall: runner[418] failed to run for 3.010s
Here, the task waited for a few milliseconds before it was switched to
SCHED_OTHER, slept outside of SCHED_EXT for three seconds, and was charged
for all of it as soon as it woke up on SCHED_EXT again.
The same happens when scx_sub_enable_workfn() hands a waiting task over to
a new sub-scheduler, which can then be blamed for time the task waited
under the parent. scx_rehome_task() isn't affected, as it moves the task
through SCX_TASK_INIT, which resets runnable_at. The stale runnable_at also
skews the default core-sched ordering in scx_prio_less().
Reset runnable_at in scx_disable_task(), when the task leaves the control
of its scheduler.
Fixes: 8a010b81b3a5 ("sched_ext: Implement runnable task stall watchdog")
Cc: stable@vger.kernel.org # v6.12+
Assisted-by: LLM
Signed-off-by: Michal Blaszczyk <michalblk@google.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Suspend and hibernation take down every non-boot cpu and bring them back. A
loss and a regain per cpu to every sub-scheduler would be noise. Leave them
out: a cpu they take down reports no loss and keeps the caps last delivered,
and the reseed after resume reports only the net change. Their downs are the
ones with cpuhp_tasks_frozen set between the PM notifications. A kexec jump
sets the flag too but sends no PM notification.
Two things need care. A cpu that suspend or hibernation takes down doesn't
wait for its syncs, and no ecaps_report() mark can be pending on it then.
The attach report runs while the attaching task waits uninterruptibly, which
holds off the freezer, and the detach report holds cpus_read_lock().
The detach report reaches only active cpus, and the loss on a cpu that
suspend or hibernation took down was never delivered. The detach waits for
those cpus to come back, so every loss is delivered. They come back before
the tasks thaw, so the wait doesn't hold up the freezer. If one fails to
come back, the hierarchy is aborted at the end of the PM cycle.
v2: Fix why no report mark is pending at a PM down, add a data_race().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
When a cpu goes offline, a sub-scheduler's effective caps on its cid are
cleared, but neither the sub-scheduler nor its parent is notified. The
sub-scheduler keeps treating the cid as usable, and the parent can't act on
the loss, e.g. reset the cpuperf target.
Report the loss before the root's ops.cpu_offline() and the regain after its
ops.cpu_online(). The zeroing at offline was the last change to a sched's
ecaps outside a delivery, so they now change only with one. Syncs on an
inactive cpu are processed instead of discarded, and the reseed reschedules
the cpu itself, as they got no resched when queued. Suspend and hibernation
now report their cpu downs and ups as well. This is temporary, and a later
patch leaves them out.
Two things need care. A cpu going down marks a sync for each sub-scheduler,
and its hotplug thread waits until each marked sync has been processed. The
cpu's own ext picks process them, and the ext server is started in case a
fair task holds the cpu. Only the marked syncs count, so a flood of changes
on the cid can't hold up the hotplug.
The detach report reaches only active cpus. It holds off cpu hotplug, so a
cpu it skips has reported its loss as it went down, and no unreported loss
is left to drop.
Drop the cpuperf reset from scx_qmap's ops.sub_detach(), as its
ops.sub_child_ecaps_updated() now resets the targets on the loss reports.
v2: Reschedule from the irq work and document the regain order.
v3: Pack nr_ecaps_offline with kick_sync_pending to avoid a hole (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler's ecaps ops run in dispatch context, and ops.cpu_online()
and ops.cpu_offline() document that the cpu doesn't call ops.dispatch()
outside them. The reseed of a cpu coming back is queued before the root's
ops.cpu_online(), so the ops of a sub-scheduler and its parent can run for a
cpu the root still considers offline.
Add SCX_RQ_OPS_ONLINE, set while the root considers the cpu online, from its
ops.cpu_online() or its enable until its ops.cpu_offline(). A sync outside
that span leaves the ecaps alone, and the reseed, now queued after
ops.cpu_online(), reports the net change. SCX_RQ_ONLINE can't be used, as
the core changes it ahead of the root's view on both edges.
A cpu going offline still drops its caps without a report.
While at it, fix the SCX_RQ_ONLINE comments, which claimed that the flag
tracks ops.cpu_online() and ops.cpu_offline().
v2: Fix the SCX_RQ_ONLINE comments.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(),
but the caps granted from ops.sub_attach() reach it only after its enable
bypass lifts, when it is already scheduling its tasks.
Report the caps at the end of the enable, before bypass lifts. On disable,
report their loss before any task goes back to the parent, so that the
parent hears of the loss of every cap it was told of on an active cpu.
ops.sub_detach() remains where the parent cleans up after the sub-scheduler.
Deliveries no longer depend on bypass, so ops.sub_ecaps_updated() and
ops.sub_child_ecaps_updated() also run while a scheduler bypasses. The
replay at unbypass goes, and the enable's bypass returns to after
ops.sub_attach().
A sub-scheduler's ecaps now change only with the deliveries that report
them, so that whenever its ops run under the rq lock, its caps are the ones
it was last told of. A sync before the attach report leaves them alone, and
the detach report zeroes them as it delivers the loss. A cpu going offline
still drops its caps without a report, and those are the only changes not
delivered. clear_all_caps() drops that unreported loss on inactive cpus.
The following things need care. A report reschedules only the cpus the
sub-scheduler was ever granted a cap on. Rescheduling every cpu would
disrupt a large machine split many ways, and which cpus still owe a delivery
can't be read off the caps.
The report and a queued sync reschedule the cpu from hard irq work on that
cpu, not with a kick or a direct resched. dispatch_one() processes the syncs
only at its start, and a kick only guarantees that a scheduling event
completes after it, which can be the pick in progress. That pick has already
processed the syncs and clears the resched, and a cpu that then idles or
runs a task with an infinite slice leaves the report waiting. A pick runs
with IRQs off, so the irq work runs after any pick in progress on that cpu.
Under core scheduling, a sibling's committed pick can switch the cpu without
dispatch_one(), and a sync queued after that pick is processed only at the
cpu's next scheduling event. A cpu running a task of a higher class isn't
rescheduled, as its pick would return that task again. While the root
bypasses, a queued sync doesn't reschedule the cpu, as a grant or revoke can
run in the PM syscore phase, where irq work can malfunction. The report
never runs there and reschedules the cpu itself.
The report waits on an rcuwait woken from hard irq work. Under core
scheduling, a cpu going down can process a sibling's marked sync after
hotplug has parked its irq_work thread. On PREEMPT_RT, a regular irq_work
queued there would be left behind, and a hard irq can't take a wait queue's
lock.
v2: Reschedule from hard irq work on the cpu instead of a kick (sashiko).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The free path removes a sub-scheduler's ecaps sync node from each cpu's list
and then cycles the rq lock until the node reads off-list, as a sync batch
can hold the node privately. Cycling the lock is correct only because the
sched is bypassing and a bypassing sched's sync calls no op, so the batch
touches the pcpu within one rq lock hold.
Tying the free path to what the sync does for a bypassing sched is unusual
and brittle, and it gets in the way of changing the sync. Delivering ecaps
to a bypassing sched or its parent would break the wait, as a delivery drops
the rq lock.
Wait for the batches with RCU instead, the usual way to wait for lockless
users before a free. Discard the nodes after the first grace period as
before, and free the sched after a second one. The first grace period waits
for the paths that found the sched before it was unlinked, such as the
sub-cap kfuncs and the cpu hotplug hooks. Until they finish, they can queue
its syncs, and a cpu can take a queued node into a sync batch that the first
grace period doesn't wait for. After it, nothing queues the syncs, and the
discards remove the nodes still queued.
A batch that took a node keeps irqs off from llist_del_all() to its last
pcpu write, even across the rq lock drops of a delivery. The second grace
period waits for it whether or not it called an op. The exit and disable
irq_works, the helper kthread and the bypass load balancing timer are
stopped ahead of the per-cpu checks as before, so they move into
scx_sched_quiesce_rcu_work() with the checks.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the cid to pshard lookup out of the ecaps sync paths into
cid_pshard() so that new sync paths can share it. No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A JIT sets up the arena base in the prologue of every function of a
program that has an arena, a callee-saved register loaded with a 64-bit
immediate and spilled around the function on x86. That is paid for by
every program with an arena, whether it touches the arena or not, and
now that libbpf binds the arena of an object to every program in it, by
every program of an object that happens to contain an arena.
Decide per function, on its final insns, right before it is JITed.
bpf_prog_has_arena_insns() looks for an insn that works with the base:
an access through an arena pointer, or through a number in a program
loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32
form by then, a cast to the user address space keeps
BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which
the JIT rebases, has a function model that says so. A load of an
__arena struct member is rewritten with an immediate and needs none;
dereferencing the result does.
Looking at the final insns rather than recording the uses during
verification keeps the answer in step with what the JIT sees. A
constant NULL passed to a nullable __arena kfunc argument takes the
early exit for NULL arguments in the verifier, and the JIT rebases the
call all the same; an insn that is dead code has disappeared by then.
bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single
function program, jit_subprogs() for each subprogram once it has its
kfunc table and arena. It implies that the program has an arena, as the
scan says no without one. The JITs with arena support pick up the arena
base and user address only when bpf_prog_uses_arena_insns() says so,
and so generate the prologue and epilogue of a program bound to an
arena it never touches as if it had none.
The other users of the arena address of a program stay as they are on
purpose. The struct_ops trampoline converts __arena arguments on entry
and arena_member_fixup() converts loaded __arena members after the load,
both with the base as an immediate. Both have to happen whether or not
the program dereferences the result, it may compare, store or pass it
on, so neither depends on the base register nor on the flag. The arena
fault report reads the user address of the arena the faulting insn
belongs to, and such an insn sets the flag anyway.
check_map_prog_compatibility() checks the user address when the arena
is bound, before any use is known. The rest, in arena.c and sched_ext,
works with the map and not with a program.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com
|
|
A kernel struct shared with BPF programs may want to point at arena
memory, e.g. a sched_ext argument struct that the kernel builds in the
scheduler's arena and that refers to another arena object. A pointer
member reaches the program typed as a kernel pointer, so sched_ext
publishes a plain u64 holding a pre-translated arena address that the
program casts before use.
Translate in the verifier instead. When a load through a kernel struct
pointer lands on a member tagged __arena, btf_struct_walk() reports it
and the destination register becomes PTR_TO_ARENA. The member holds the
kernel address of the arena memory; the fixup pass appends the
conversion to the arena pointer form that the struct_ops trampoline
applies to __arena arguments, after a plain load or a load-acquire of
the member:
r1 = *(u64 *)(r2 + off)
if r1 == 0 goto +1 // __nullable members and probed loads
w1 -= lo32(kern_vm_start)
A probed load, through a plain or an untrusted pointer, keeps the NULL
test as well: it reads as 0 when it faults, and the subtraction must
not turn that into an arena pointer. An acquire load only comes through
a pointer that cannot fault.
Stores to kernel structs are rejected as before, and with them the
atomics that write such a member. A member
inside a multi-member union is rejected too: an offset into a union
resolves to its first member, so a program reading another member there
would receive an arena pointer it did not ask for. Loads through
program-allocated objects and program BTF are unaffected, there the tag
is what the compiler emits for the arena address space.
The rewrite is per insn while verification is per path, so the aux data
records what an insn loaded on any path: an insn that loaded an arena
pointer member on one path and a different member on another is
rejected by the fixup pass, while paths disagreeing only on __nullable
keep the NULL test, which a never-NULL member can afford. As for
__arena arguments, the program needs an associated arena, which also
supplies the base to subtract.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-8-memxor@gmail.com
|
|
A pointer member of a kernel struct that the verifier derives as trusted
or RCU-protected is assumed to be non-NULL, so a load through it is a
plain dereference. The BTF_TYPE_SAFE_*_OR_NULL lists exist for members
known to be NULL at times, but they live in the verifier and have to be
extended by hand.
Teach btf_struct_walk() to report the "nullable" BTF type tag on a
pointer member as PTR_MAYBE_NULL. The tag combines with any other tag
on the member; it only says that the pointer may be NULL. The verifier
keeps the flag where NULL would otherwise be assumed away: on trusted
and RCU results, which a program dereferences as they are, and on
per-CPU results, which the per-CPU helpers shift without a check. Such
a pointer has to be tested before it is dereferenced or passed on.
Plain and untrusted results are probed anyway and lose the flag, as the
BTF_TYPE_SAFE_*_OR_NULL lists never applied to them either, so a tagged
member of an existing struct asks no more of the programs that read it
than a list entry does, and the tag can take the place of the lists.
The next patch uses the tag on __arena members to decide whether the
injected address translation has to preserve NULL.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-7-memxor@gmail.com
|
|
An exclusive map is bound to the digest of its program, which the
verifier compared against prog->digest in check_map_prog_compatibility()
as the map was added to the program. For a map an insn refers to that
happens in check_and_resolve_insns(), after the digest is computed at
the top of that function. For a map bound through a continuous fd_array
it happens in process_fd_array(), which runs earlier, so for an unsigned
program the comparison saw a zeroed digest and the load failed with
-EACCES, "program's hash doesn't match map's excl_prog_hash", for the
very program the map is exclusive to.
Computing the digest earlier would not do: it has to cover the insns as
the kernel verifies and runs them, after the CO-RE relocations it
applies, or a loader could submit the insns of the exclusive program
together with relocations that change them and pass the check with a
different program. Leave the digest where it is and move the comparison
instead: check every used map against the digest at the end of
check_and_resolve_insns(), whether an insn referred to it or the array
bound it.
The claims a program makes on its maps move behind the check as well:
bpf_map_claim(), which records the program as the user of the map for
resolve_func_ptrs(), and the takeover of an insn array by
bpf_insn_array_init(). Both ran as the map was added, so a program that
is not the owner of an exclusive map could have flipped the owner's
single-user state to shared, which a failed load does not undo, or held
the owner's insn array while it was being verified. Now they run once
every map has passed the check, and a rejected program leaves the maps
as it found them. Nothing consults a claim before resolve_func_ptrs()
and the fixup passes, which run later.
The early digest of a signed program goes with the early comparison.
Commit b707068e0ed9 ("bpf: Verify signed loader metadata at load time")
computed it only so that the maps of a signed loader passed the check in
process_fd_array(); the signature itself is verified over the submitted
insns and the frozen maps by bpf_prog_verify_signature(), which does not
read the digest. The maps of a signed program are now checked against
the digest of the final insns like everyone else's, which covers the
relocations the kernel applies and the signature does not; a loader
program carries none.
Fixes: baefdbdf6812 ("bpf: Implement exclusive map creation")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-2-memxor@gmail.com
|
|
refcount
In event_hist_trigger_parse() and hist_unreg_all(), the histogram
registration and unregistration paths look up find_synth_event() using
trace_event_name(file->event_call) and adjust se->ref if a match is
found.
Because find_synth_event() only compares the event basename, registering
or removing a histogram on a non-synthetic trace event whose basename
matches a synthetic event erroneously modifies the synthetic event's
reference count. If a histogram is registered on the non-synthetic
event before the synthetic event is created and removed afterwards,
se->ref is decremented without a matching increment, allowing the
synthetic event to be freed while an active histogram action still holds
a pointer to it.
Introduce find_synth_event_call() to verify that &se->call matches
file->event_call before adjusting se->ref.
Fixes: 4b147936fa50 ("tracing: Add support for 'synthetic' events")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261019120730.2342739-1-lee@kernel.org
Signed-off-by: Lee Jones <lee@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
print_rec() leaves 'modname' and 'offset' uninitialized and
kallsyms_lookup() only sets them on success. When the lookup fails like
when a module is being loaded or unloaded, 'modname' holds a garbage
value, passes the 'if (modname)' check, and is dereferenced by
seq_printf().
Oops: general protection fault
RIP: 0010:string+0x48/0xe0
Call Trace:
vsnprintf+0x1f5/0x650
seq_printf+0x8e/0xb0
print_rec+0x6c/0xc0
t_show+0x4c/0x440
seq_read_iter+0x119/0x480
This can be triggered by reading available_filter_functions while a
module is being loaded or unloaded.
Initialize both variables so a failed lookup skips the module name.
Cc: stable@vger.kernel.org
Fixes: b39181f7c690 ("ftrace: Add FTRACE_MCOUNT_MAX_OFFSET to avoid adding weak function")
Link: https://patch.msgid.link/20261016230403.4916-1-aarnahmd@amazon.com
Tested-by: Gavin Inglis <giinglis@amazon.com>
Signed-off-by: Aaron Ahmed <aarnahmd@amazon.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When ftrace_call_adjust() returns 0 for a patch site,
ftrace_process_locs() does not skip the site. The NULL test runs before
the adjustment since commit 6eeca746fa5f ("ftrace: Test mcount_loc
addr before calling ftrace_call_addr()"), so the 0 is stored as the
record's ip and ftrace_init_nop() later hits the BUG_ON() in arm64's
patch_map(). Test the adjusted address too and count the site as
skipped.
Link: https://patch.msgid.link/20260924-b4-arm64-callops-kcfi-v2-2-587865b6d991@linux.dev
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When every mcount_loc entry of a module is skipped,
ftrace_process_locs() leaves the module's first page group linked
with no records. ftrace_release_mod() matches a module's groups by
records[0].ip, which is 0 here, so the group is never freed. While
it stays linked, ftrace_free_mem() reads pg->records[pg->index - 1]
with pg->index == 0 on every later module load, as lookup_rec() did
before commit ee92fa443358f ("ftrace: Fix invalid address access in
lookup_rec() when index is 0").
Unlink and free the new page groups when none of them got a record.
Link: https://patch.msgid.link/20260924-b4-arm64-callops-kcfi-v2-1-587865b6d991@linux.dev
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260904013350.5141B1F000E9@smtp.kernel.org/
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Merge updates related to system sleep, an energy model handling update,
a generic CPU PM driver update, a clock framework PM update, and a
pm-graph utility update for 7.4:
- Ignore page-zero RAM in E820 checksum during hibernation on x86 to
avoid rejecting images created after kexec() (Matthias Goergens)
- Add resume event mapping for PMSG_POWEROFF to the PM core to allow it
to carry out device shutdown in the future (Mario Limonciello)
- Add DPM watchdog to prepare/late/early/noirq/complete phases of
device suspend and resume (Mayank Rungta)
- Update swsusp.txt references to swsusp.rst in hibernation
documentation (Li Jun)
- Fix use-after-free of perf domain in netlink doit handlers for energy
models (Fan Wu)
- Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it
was removed inadvertently in previous development cycles to avoid
use-after-free (Bradley Morgan)
- Unregister CPU PM notifier outside the timer lock in the timer-ti-dm
clocksource driver (Bradley Morgan)
- Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc
comment and typo "hierachy" in a comment in pm-graph (Hemanth Selam)
* pm-sleep:
x86/hibernate: Ignore page-zero RAM in E820 checksum
PM: sleep: Add resume event mapping for PMSG_POWEROFF
PM: sleep: Add DPM watchdog to prepare/late/early/noirq/complete phases
PM: hibernate: docs: update swsusp.txt references to swsusp.rst
* pm-em:
PM: EM: Fix use-after-free of perf domain in netlink doit handlers
* pm-cpu:
PM: CPU: Restore synchronize_rcu() to cpu_pm_unregister_notifier()
clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock
* pm-misc:
PM: clk: fix typo "sucessfully" in comment
* pm-tools:
tools: power: pm-graph: fix typo "hierachy" in comment
|
|
Merge cpufreq updates for 7.4:
- Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel
Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy,
Sumeet Pawnikar, and Xueqin Luo)
- Fix wrong behaviors on multi-node servers in the Loongson cpufreq
driver (Huacai Chen)
- Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu)
- Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue)
- Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq
driver (Mario Limoncielloi, Wentao Guan)
- Initial support for maximum frequency capacity handling on Zen6
client in amd-pstate (Mario Limonciello)
- Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option
(K Prateek Nayak)
- New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello,
Vishal Badole)
- amd-pstate documentation fix (Karl Mehltretter)
- Restrict deferred downscaling to low load for shared policies in the
conservative cpufreq governor to avoid undesirable CPU frequency
reduction in some cases (Shengming Hu)
- Serialize sugov_start() against rate limit updates in the schedutil
cpufreq governor to avoid using a stale rate limit value (Hui Su)
* pm-cpufreq: (34 commits)
cpufreq: loongson3: Replace IOCSR read/write with MMIO ones
cpufreq: loongson3: Use global physical CPU ID in get/target callbacks
cpufreq: loongson3: Replace per-package mutex with per-node
cpufreq: loongson3: Adjust the width of id and val in smc_message
cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64
cpufreq: schedutil: Serialize sugov_start() against rate limit updates
arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend
cpufreq: mediatek: Enter system sleep at the suspend OPP
cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing
cpufreq: qcom-nvmem: Add IPQ9650 support
cpufreq: Add Milos to cpufreq-dt-platdev blocklist
cpufreq: Use %pe to print error pointers symbolically
rust: rcpufreq_dt: add module alias
cpufreq: sti: avoid NULL dereference in dev_err()
cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv
cpufreq: sparc-us2e: fix frequency table index copy-paste error
rust: cpufreq: reject NULL from cpufreq_cpu_get()
cpufreq: amd-pstate-ut: Fix max_freq and EPP test failures on Zen6
cpufreq: amd-pstate: Propagate cppc_set_auto_sel() errors on mode change
cpufreq: amd-pstate: Restore previous mode when changing driver mode fails
...
|
|
An extension that writes to the packet of a tc, lwt_xmit or sk_skb program
starts with bpf_unclone_prologue(). If the skb is cloned, that gives it
a new head.
It does not count as changing packet data. changes_pkt_data goes by the
helpers and kfuncs that a program calls, and whether it gets a prologue
is known only after it has been compared with its target. So the extension
can replace a global function that leaves the packet alone, and the callers
of that function keep their packet pointers across the call. These point
into the old head, which is left to the other clone. Once that is freed,
by the reader of a packet socket for example:
BUG: KASAN: slab-use-after-free in bpf_prog_53bc21b9810cadbc_t_read+0x2e8/0x36e
Read of size 1 at addr ffff888107bfe73c by task tt11/102
Freed by task 112:
kfree+0x161/0x500
skb_release_data+0x2dc/0x340
consume_skb+0xa6/0x1c0
packet_recvmsg+0x4b3/0xa20
Nothing is wrong if the target program writes to the packet too: its own
prologue has uncloned the skb, and that of the extension does nothing.
Nor if the callers hold no packet pointers, like a dispatcher, or if the
main function is replaced.
Note for every global function that leaves the packet alone if it is
called while packet pointers, or slices of the dynptr of a packet, are
held. What the callers of a global function hold is not known while it
is verified, so the functions that it calls are taken for such as well.
Refuse an extension with a prologue for such a function, unless the target
program has a prologue, both when the extension is loaded and when it
is attached to another target.
Functions can share a type id in func_info. The first of them is the one
that is replaced, so look it up in one place.
Fixes: 7e40781cc8b7 ("bpf: verifier: Use target program's type for access verifications")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261019011328.1843764-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A global subprogram parameter tagged __arg_trusted is verified with
PTR_TRUSTED, but the common call-site type check accepts bare PTR_TO_BTF_ID
and MEM_RCU registers. The callee can consequently pass a pointer with
weaker provenance to a kfunc that does not accept RCU-protected pointers.
RCU protection only guarantees that the object remains allocated through a
grace period. It does not guarantee that an object's refcount is nonzero,
so an RCU-protected pointer cannot satisfy the trusted-pointer contract.
For example, an RCU-protected bpf_cpumask loaded from a map may have a
zero refcount. Passing it through __arg_trusted lets the callee pass it
to bpf_cpumask_acquire(), whose plain refcount_inc() can retain an object
that will be freed after the grace period.
Make the existing provenance check contract-driven. Mark generated BTF ID
kfunc argument types with PTR_TRUSTED, and add MEM_RCU only for KF_RCU
kfuncs. Normalize PTR_MAYBE_NULL when the argument contract permits it.
Then use the argument flags to require trusted or referenced provenance
and allow MEM_RCU only when the contract says so.
Split is_trusted_reg() so the normalized type is used for type and modifier
checks while reference lookup still uses the original register. Conversion
of scalar-struct BTF ID arguments to fixed-size memory continues to
construct a new ARG_PTR_TO_MEM type without PTR_TRUSTED.
Fixes: e2b3c4ff5d18 ("bpf: add __arg_trusted global func arg tag")
Suggested-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20261007-a3-arg-trusted-v4-v4-1-253ef3a1a69a@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps,
htab_map_check_update_flags() and check_flags() are not considering
the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags
with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the
(BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.
For example, when using (BPF_F_CPU | BPF_EXIST) or
(BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem()
on a percpu hash map, the BPF_EXIST flag does not prevent new
insertions as expected and BPF_NOEXIST flag does not prevent
modification of existing entries as expected.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, htab_map_check_update_flags() rejected
(BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with
-EINVAL. After the patch those combinations are accepted.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes: c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
Signed-off-by: Masoud Aghasi <maghasi@disroot.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20261008120734.1565972-3-maghasi@disroot.org
|
|
For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update()
is not considering the possibility of a combination of
(BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags.
This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect
in some cases.
For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination
with bpf_map_update_elem() results in an incorrect EINVAL error
response, even though the flag combination is valid.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, bpf_percpu_array_update() rejected
BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with
-EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted.
After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and
BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST
result in -EEXIST.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes: 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")
Signed-off-by: Masoud Aghasi <maghasi@disroot.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20261008120734.1565972-2-maghasi@disroot.org
|
|
bpf_check_attach_target() rejects a freplace whose target is a static
function, using the subprog "unreliable" flag as a proxy for "static".
That flag is raised whenever a subprogram's BTF prototype disagrees with
the registers at its call site.
`147400e9c27c ("bpf: Compute parameters liveness in
bpf_compute_live_registers()")` made that work for static callees: a
static function with an unused argument no longer matches its BTF
prototype at the call site and is marked unreliable.
Since LLVM commit `29b36a16f111 ("[BPF] Emit ABI-accurate BTF
prototypes for DW_CC_nocall")` in 23.1 [1] that premise is gone:
BTFDebug may now drop arguments removed by the optimizer, so for
__noinline static int subprog(struct __sk_buff *ctx)
{
return bpf_ktime_get_ns();
}
compiled with clang 23 the BTF describes subprog() as
[75] FUNC_PROTO '(anon)' ret_type_id=2 vlen=0
[76] FUNC 'subprog' type_id=75 linkage=static
There is then no argument left to compare, the subprog stays reliable,
and the check in bpf_check_attach_target() is skipped. The freplace is
then rejected by the later linkage check instead, reporting "subprog()
is not a global function", and the selftest "exceptions" fails.
Determine staticness from the BTF linkage instead, and keep the
unreliable check for global functions whose BTF does not match their
call site.
There is no functional change: freplacing a static function was rejected
before and is still rejected with the same -EINVAL. Selftest "exceptions"
is updated to match.
[1] https://github.com/llvm/llvm-project/pull/198426
Fixes: 147400e9c27c ("bpf: Compute parameters liveness in bpf_compute_live_registers()")
Signed-off-by: KaFai Wan <kafai.wan@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261018005348.2195130-2-kafai.wan@linux.dev
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix __sched_setscheduler() class change race causing
lost reschedules, exposed by recent RCU changes (K Prateek Nayak)
- Fix s64 multiplication overflow in place_entity()'s vlag
scaling code that can corrupt an entity's placement (Hui Su)
* tag 'sched-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/fair: Avoid overflow in place_entity()
sched/core: Call wakeup_preempt() for queued tasks during class change
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf events fixes from Ingo Molnar:
- Fix x86 Intel uncore driver bug on Diamond Rapids CPUs
that erroneously always-cleared the invert bit (Guixiong Wei)
- Fix accounting bug in the perf mlock 'gift' code causing
unrelated RLIMIT_MEMLOCK functionality to fail (Lorenzo Stoakes)
- Fix perf_pmu_migrate_context() bug leaving events behind during
CPU hot-unplug (Haris Okanovic)
- Update cgroup time when rotating multiplexed events, to fix
time_running & scaled counts accounting bug (Antoine Huret)
* tag 'perf-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf: Update cgroup time when rotating multiplexed events
perf: Fix perf_pmu_migrate_context() leaving events behind
perf: Track the perf mlock gift separately
perf/x86/intel/uncore: Fix DMR CXL CM invert filter
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fixes from Ingo Molnar:
- Fix FUT_OFF_MMSHARED futex hashing bug that can cause
missed wakeups or spurious wakeups on futex-waiting
tasks (Jann Horn)
- Fix lockdep context tracking bug in the deferred hrtimer
rearming entry code, which causes a lockdep chain leak
that will eventually turn lockdep off (Mikhail Gavrilov)
* tag 'locking-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
entry: Account the deferred hrtimer rearm as hardirq context for lockdep
futex: Fix mm reuse handling for FUT_OFF_MMSHARED
|
|
bpf_get_func_arg() takes a u32 index, but the inlined expansion compared
only the low 32 bits of R2 with the argument count and then shifted the
full 64-bit register to compute the context address. R2 is ARG_ANYTHING,
so nothing guarantees that its upper bits are zero: an index of
1ULL << 32 passes the bounds check as index zero while the load is
performed at ctx + (1ULL << 35).
Shift the index as a 32-bit value. When the 32-bit bounds check passes,
the low 32 bits of R2 are below R0, and R0 is at most 255: the trampoline
arm masks the argument count with 0xFF and the raw_tp arm uses the BTF
argument count minus one, which the verifier caps at MAX_BPF_FUNC_ARGS.
The 3-bit shift therefore cannot overflow and it clears the upper bits of
the register. Widening the comparison instead would reject indices whose
low 32 bits are valid. The check itself is unchanged, so this cannot
change which programs are accepted, only the address that is computed.
Fixes: f92c1e183604 ("bpf: Add get_func_[arg|ret|arg_cnt] helpers")
Signed-off-by: Woojin Ji <random6.xyz@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261018-get-func-arg-v2-1-048848b3eddf@gmail.com
|
|
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every
SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips
unmarked tasks without dropping either, so a long run of unmarked tasks is
walked in one hold of both locks.
Count the visited tasks instead of the re-enqueued ones and drop both locks
every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as
often as before.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The helpers which may change packet data do not take a packet pointer as a
memory argument: they have no pkt_access. They make the skb writable, or
make room in it, before they read what the argument points to, and that
can reallocate the head which it points into.
A slice from bpf_dynptr_slice() of the dynptr of the skb points into the
same head, but it is a PTR_TO_MEM, and the verifier lets it pass.
bpf_lwt_push_encap(), bpf_lwt_seg6_store_bytes() and bpf_lwt_seg6_action()
then read the old head, which has been freed. bpf_skb_store_bytes() follows
the head by now, but a slice is no better than a packet pointer there.
Refuse a slice of a packet where a packet pointer is refused for that
reason. All other helpers take slices as before. bpf_dynptr_write() and
bpf_dynptr_copy() can be used to copy inside of a packet.
Fixes: 2012c867c800 ("bpf: verifier: Accept dynptr mem as mem in helpers")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-17-alexei.starovoitov@gmail.com
|
|
bpf_dynptr_copy(), bpf_dynptr_memset() and the kfuncs that read user or
kernel memory into a dynptr ask bpf_dynptr_slice_rdwr() for a slice and
write to it. For the linear part of an skb that is a pointer into its
head. bpf_dynptr_slice_rdwr() counts on the verifier to have uncloned the
skb, see bpf_unclone_prologue().
The verifier does so if the program itself calls bpf_dynptr_slice_rdwr()
or writes to the packet. A program that only uses these kfuncs gets no
prologue, and changes the data of all the clones of the skb. A packet
socket, whose clone is taken before tc ingress runs, sees what
bpf_dynptr_memset() has written there.
Treat a call that may write to an skb like a direct write.
Fixes: daec295a7094 ("bpf/helpers: Introduce bpf_dynptr_copy kfunc")
Fixes: a498ee7576de ("bpf: Implement dynptr copy kfuncs")
Fixes: 5fc5d8fded57 ("bpf: Add bpf_dynptr_memset() kfunc")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-13-alexei.starovoitov@gmail.com
|
|
A tc, lwt_xmit or sk_skb program that writes to packet data starts with
bpf_unclone_prologue(). It takes the skb from R1, and if the skb is cloned
it calls bpf_skb_pull_data() and restores R1 only.
That suits the main function of a program. An extension gets the prologue
too, but it starts with the arguments of the function that it replaces.
If the first one is not the context, something else is taken for an skb:
BUG: KASAN: stack-out-of-bounds in skb_ensure_writable+0x84/0x1a0
Read of size 8 at addr ffffc90000270ad0 by task uncl/105
skb_ensure_writable+0x84/0x1a0
bpf_skb_pull_data+0x32/0xf0
bpf_prog_853eb1991d35e093_ext_slice+0x4f/0x321
bpf_prog_9eb18752b5d0959a_t_glob+0x2ac/0x2de
BUG: unable to handle page fault for address: 0000000082298821
And whatever R2-R5 held is lost, while the verifier goes on trusting
pointers in them.
Refuse an extension that needs a prologue unless it gets the context as
its first argument and scalars beside it. A scalar is still clobbered if
the skb is cloned, which the verifier can cope with. Extensions of XDP,
sk_msg and sockopt programs get no prologue and load as before.
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-12-alexei.starovoitov@gmail.com
|
|
A global function is verified on its own. A dynptr which it gets as an
argument is assumed to be of type BPF_DYNPTR_TYPE_LOCAL, but the callers
may pass one of any type, that of an skb for example.
The type decides whether a write through the dynptr invalidates packet
pointers, and whether a slice of it is invalidated together with them.
Inside of the function neither happens. It can take a packet pointer from
its context argument, or a slice from the dynptr, write through the dynptr
and go on using the pointer, although the write may have reallocated the
head of the skb.
Assume the type of an skb dynptr instead. Nothing becomes forbidden for the
dynptr itself, and a clone inherits the caution. The dynptr which the
callback of bpf_user_ringbuf_drain() gets is known to be local, and stays
so.
Fixes: a64bfe618665 ("bpf: add support for passing dynptr pointer to global subprog")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-11-alexei.starovoitov@gmail.com
|
|
The caller of a global function loses its packet pointers if the function
may change packet data. That is found out by a scan for calls of the
helpers which bpf_helper_changes_pkt_data() lists.
bpf_dynptr_write() is not on the list, because it depends on the type of
the dynptr: check_helper_call() looks at it. The scan cannot do that, and
so a global function which makes a dynptr from the skb, or gets one as an
argument, and writes through it counts as harmless. The write can
reallocate the head of the skb, and the caller goes on with pointers into
the old one, which has been freed.
Let every call of bpf_dynptr_write() mark the function, as a tail call
does. The same mark decides whether a program may replace a function.
Fixes: 51081a3f25c7 ("bpf: track changes_pkt_data property for global functions")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-10-alexei.starovoitov@gmail.com
|
|
bpf_dynptr_write() to the dynptr of an skb ends in
__bpf_skb_store_bytes(), which makes the range writable first. That pulls
data into the head or unclones it, and either can reallocate the head. So
check_helper_call() invalidates all packet pointers and all slices of the
skb behind the helper.
Several kfuncs take the same road: bpf_dynptr_copy() and
bpf_dynptr_memset(), if they get no writable slice of the destination
because it is not linear, and bpf_probe_read_{user,kernel}[_str]_dynptr()
and bpf_copy_from_user[_task][_str]_dynptr() likewise. The verifier does
not know. A program can keep a packet pointer or a slice across such a
call and read and write the old head, which has been freed.
Treat a kfunc of that kind as bpf_dynptr_write() is treated, by the type
of the dynptr it writes to, which is the first argument of all of them.
Where a global function is only scanned for what it may do, the type is
not known yet. There any call of these kfuncs counts, as a tail call does.
Fixes: daec295a7094 ("bpf/helpers: Introduce bpf_dynptr_copy kfunc")
Fixes: a498ee7576de ("bpf: Implement dynptr copy kfuncs")
Fixes: 5fc5d8fded57 ("bpf: Add bpf_dynptr_memset() kfunc")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-9-alexei.starovoitov@gmail.com
|
|
A frame that is redirected to another CPU waits in the queue of the cpumap
entry with a plain pointer to the device it came from, and an skb from
generic XDP with skb->dev. Nothing keeps that device. Closing and
unregistering it waits for its NAPI and for RCU, but not for the kthread
of the entry. If the kthread is late by more than that, since its CPU is
busy with something of a higher priority, it builds skbs for a device that
has been freed and hands them to the stack:
BUG: KASAN: use-after-free in dev_gro_receive+0x1f51/0x2b30
Read of size 8 at addr ffff88800a7340b0 by task cpumap/1/map:1/54
Take a reference to the device for each entry of the queue. The kthread
cannot drop it at the end of the batch, since its GRO node may keep the skb
for longer. Let it note the devices and drop the references behind a full
GRO flush, which it does whenever the queue is empty, and from now on also
when it has noted about 2048 of them.
A device whose frames are stuck in a queue is now unregistered only when
the kthread has got to them. RTNL is not held meanwhile, but if the device
goes with its network namespace, the cleanup of all namespaces waits.
Fixes: 1c601d829ab0 ("bpf: cpumap xdp_buff to skb conversion and allocation")
Fixes: 11941f8a8536 ("bpf: cpumap: Implement generic cpumap")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-8-alexei.starovoitov@gmail.com
|
|
When the last reference to a program goes, __bpf_prog_put_noref() takes
its symbols out of the kallsyms tree and frees its line info at once, and
waits for the grace periods only with the rest. But the program may still
be running: the link or the array it was called from is gone, the call is
not over.
search_bpf_extables() finds the exception table of a program by its
symbol. A load that faults, and that the table is there to fix up, is
fatal from then on:
Oops: general protection fault, probably for non-canonical address
0xdead0000000006f8: 0000 [#1] SMP NOPTI
RIP: 0010:0xffffffffc06008f6
It takes a program that is detached and closed on one CPU while it runs on
another. Unwinders and bpf_prog_find_from_stack() do not know the program
any more either, and the stack dump of a BPF stream reads the freed line
info.
Free the line info behind the grace periods the program needs, as it is
done with the BTF that its text comes from. Remove the symbols in
bpf_prog_free_deferred(), which runs behind them as well. The symbols are
looked up under rcu_read_lock(), so the images and the program itself are
freed one more grace period later. All that the program refers to is
released when it was before.
Fixes: 3dec541b2e63 ("bpf: Add support for BTF pointers to x86 JIT")
Fixes: d7c431cafcb4 ("bpf: Add dump_stack() analogue to print to BPF stderr")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-6-alexei.starovoitov@gmail.com
|
|
An extension cannot be sleepable, and it is verified as a program that
runs under the RCU read lock. But it runs in the context of the function
that it replaces. For a sleepable program, or a function that is called
from a bpf_wq or bpf_task_work callback, that is rcu_read_lock_trace().
The verifier lets such an extension use maps and pointers that rely on
RCU. Its maps are not counted as maps with sleepable users either, so
that what it got from them is freed behind a plain RCU grace period:
WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0xe8/0xf0
BUG: KASAN: slab-use-after-free in bpf_prog_b3bad285227d855b_ext_storage+0x245/0x297
Write of size 8 at addr ffff888108717a50 by task fr/955
The verifier knows the contexts that a global function is called in.
Refuse to replace one that is called without the RCU read lock, and the
main function of a sleepable program. A function that sleepable code only
calls under bpf_rcu_read_lock() can still be replaced.
The summarization test has replaced functions of sleepable programs by
ones that do not sleep. Expect that to fail.
Fixes: 1e6c62a88215 ("bpf: Introduce sleepable BPF programs")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-5-alexei.starovoitov@gmail.com
|
|
bpf_local_storage_destroy() frees the elements of an owner behind a plain
RCU grace period, on the ground that no program can refer to an owner that
goes away. But the pointer that bpf_task_storage_get() and its like
return stays valid for the verifier after the program has released the
owner, or has left the bpf_rcu_read_lock() section it had the owner from.
A sleepable program is in no RCU read side critical section, so it can
write to the value after the owner has died and the element was freed:
BUG: KASAN: slab-use-after-free in _copy_from_user+0x42/0xa0
Write of size 8 at addr ffff8880074404a8 by task lsuaf/55
Free the element behind an RCU Tasks Trace grace period if sleepable code
may use the map. A program that a map is bound to was loaded, and counted
in map->sleepable_refcnt, before it got a value from the live owner, and
stays counted until a grace period behind its last run.
A program can also get at a storage map by a lookup in an outer map. Then
only the outer map is counted, its programs can come after the inner map
was put into it, and can hold on to the inner map after it was taken out.
So note in a map that it has been an inner map, and treat it like one with
sleepable users from then on.
The same goes for what the special fields of the value hold. The page of
a uptr has to stay pinned, and a kptr which the program stores after the
owner has died has to be released. Leave the fields, and the smap that it
takes to free them, to bpf_selem_free_trace_rcu(), as bpf_selem_unlink()
does. bpf_local_storage_map_free() waits for that callback, and leaves
smap alone if it comes across such an element.
Elements of such maps now wait for sleepable programs that are running
when their owner dies. Other maps are freed as before.
Fixes: dcf456c9a095 ("bpf: Fix usage of trace RCU in local storage.")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-4-alexei.starovoitov@gmail.com
|