summaryrefslogtreecommitdiff
path: root/kernel
AgeCommit message (Collapse)Author
2026-10-20bpf: Refuse CO-RE relocations for a signed programKumar Kartikeya Dwivedi
The signature of a program covers its insns as submitted, along with the frozen metadata maps of a signed loader, and bpf_prog_verify_signature() checks it before the verifier does anything else with the program. CO-RE relocations are the one input besides the insns that changes them: the loader hands them in with the program, and bpf_check_core_relo() applies them after the signature check, rewriting the immediate or the offset of every insn a record names with a value the loader chooses, a field offset, a type size or id or an enum value of the kernel BTF, or a type id of the BTF the loader supplies itself. Whoever holds a signed program can so change its constants once the signature has passed, the offsets a signed loader reads its metadata map with for instance, and run a program the signer never saw. The exclusive metadata map of a signed loader does not make up for it: the hash that binds the map to a program is chosen by whoever creates the map, so it can name the relocated program as well. Refuse a signed load that carries relocations, before the signature is examined, like the other checks on the shape of a signed load. A signed loader has none to lose: the programs it embeds are relocated by the loader itself through bpf_core_apply(), and nothing relocates the loader. The BTF of the program itself can stay: without relocations it steers only the verification, through func_info, line_info and the annotations of global function arguments, and rewrites nothing. Fixes: 349271568303 ("bpf: Implement signature verification for BPF programs") Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://patch.msgid.link/20261019144851.4052056-2-memxor@gmail.com
2026-10-20bpf: Don't reference arena kfuncs when arena isn't builtSasha Levin
kernel/bpf/Makefile builds arena.o only for CONFIG_MMU && CONFIG_64BIT, but special_kfunc_list references bpf_arena_alloc_pages() and bpf_arena_free_pages() unconditionally, so on 32-bit and !MMU kernels resolve_btfids can't resolve them: WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages WARN: resolve_btfids: unresolved symbol bpf_arena_alloc_pages Since commit 201a454df948 ("resolve_btfids: Fix the --fatal_warnings exit status") these warnings fail the build with CONFIG_WERROR=y, which breaks i386 builds with CONFIG_DEBUG_INFO_BTF=y: Failed to generate BTF for vmlinux Try to disable CONFIG_DEBUG_INFO_BTF Only reference them under the condition arena.o is built with, and use BTF_ID_UNUSED otherwise to keep the list in sync with enum special_kfunc_type. An unresolved ID is zero, like BTF_ID_UNUSED, so the contents of the list don't change. Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe") Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019211006.2574350-3-sashal@kernel.org
2026-10-20bpf: Drop unused entries from special_kfunc_list againSasha Levin
Commit 6f033615ef8f ("bpf: Trim special_kfunc_list in verifier") dropped bpf_arena_reserve_pages, bpf_stream_vprintk and bpf_stream_print_stack from special_kfunc_list once nothing used them anymore, but commit ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path") brought all three back without adding any users. Besides being dead, the bpf_arena_reserve_pages entry can't be resolved on 32-bit and !MMU kernels, which don't build arena.o, and since commit 201a454df948 ("resolve_btfids: Fix the --fatal_warnings exit status") the resulting resolve_btfids warning fails the build with CONFIG_WERROR=y: WARN: resolve_btfids: unresolved symbol bpf_arena_reserve_pages Drop the three entries again. Fixes: ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path") Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019211006.2574350-2-sashal@kernel.org
2026-10-19tracing/osnoise: Add __printf attribute to osnoise_print()Adrián García Casado
Building kernel/trace/trace_osnoise.o with W=1 triggers the following compiler warning: kernel/trace/trace_osnoise.c:96:17: warning: function 'osnoise_print' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format] osnoise_print() forwards its format string and va_list arguments to trace_array_vprintk(), which is annotated with __printf(3, 0). Add the __printf(1, 2) attribute to osnoise_print() to silence the W=1 warning and enable compile-time format string checking for its callers. Link: https://patch.msgid.link/20261019205344.190022-1-adriangarciacasado42@gmail.com Signed-off-by: Adrián García Casado <adriangarciacasado42@gmail.com> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-10-19sched_ext: Queue the kick_sync wait from set_next_task_scx()Andrea Righi
The SCX_KICK_WAIT wait is deferred to rq->scx.kick_sync_bal_cb, which a final pick queues on the local rq. Under core scheduling, a pick that returns a task isn't final: pick_next_task() goes on to pick the SMT siblings, and their dispatch can release the shared rq lock with the callback still queued. A CPU acquiring the rq lock in that window then finds a pending balance callback and triggers the warning in rq_pin_lock(). Fix this by queuing the callback from set_next_task_scx() on SNT_PICK and SNT_REPICK. put_prev_set_next_task() calls it once the pick is final, after the core-wide selection, so the rq lock can't be dropped with the callback queued. If the CPU ends up running idle or a higher class task, no SCX task runs there and there's nothing to wait for: kick_sync_pending stays set and the next pick of an SCX task waits before running it. Also, proxy exec zaps the balance callbacks and picks again when find_proxy_task() fails, so clear kick_sync_pending when the callback runs rather than when it's queued. Fixes: 415cb193bb97 ("sched_ext: Fix SCX_KICK_WAIT deadlock by deferring wait to balance callback") Cc: stable@vger.kernel.org # v7.0+: c72945693b90: sched: Restart fair hrtick after same-task repicks Cc: stable@vger.kernel.org # v7.0+ Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup: fix cgroup.kill for exited thread group leaderKir Kolyshkin
cgroup.kill does not work in case a process' thread group leader is gone but other threads are still running. This is because __cgroup_kill() only walks the thread group leaders and sends SIGKILL to each of them with send_sig(), which uses PIDTYPE_PID: 1. The exited leader is still listed by CSS_TASK_ITER_PROCS as long as the process has live threads, but a signal directed at a single exiting task never takes effect: complete_signal() does not look for another thread to deliver it to, and __send_signal_locked() now drops it altogether. The process keeps running. 2. The live threads of such a process can be migrated to another cgroup by writing any of their IDs to cgroup.procs, while the exited leader stays behind. The destination cgroup has no thread group leader to iterate, so writing to its cgroup.kill does nothing at all. In both cases the cgroup stays populated after cgroup.kill, and a process inside a container can use this to survive the container being killed. This also works with a private PID namespace when the init of the container is the process, as the PID namespace is not torn down while init has live threads. Case 1 can be fixed by using do_send_sig_info(..., PIDTYPE_TGID) instead of send_sig(), so that the signal is taken by a live thread even if the task found by the iterator is exiting. Note it was pointed out by Eric in [1] while reviewing v2 of cgroup.kill patchset, and v3 was meant to switch to it, but the change was only made in cgroup_post_fork(), leaving __cgroup_kill() with send_sig(). Case 2 can only be fixed by iterating over all threads, so this is what the fix also does. Once a thread group is being killed, its other threads are skipped since they already have a fatal signal pending. I ran some performance checks and the added cost is ~60 ns per every thread on my laptop (and less than a millisecond for 10K threads), which seems low. The time to "populated 0" seems unchanged. The issue was found in crun by Anthropic's OSS Scanner, and crun is getting a workaround for it [2] (which does what we needed to do before cgroup.kill was added). [1] https://lore.kernel.org/all/m1v97x6niq.fsf@fess.ebiederm.org/ [2] https://github.com/containers/crun/pull/2351 Fixes: 661ee6280931 ("cgroup: introduce cgroup.kill") Cc: stable@vger.kernel.org # v5.14+ Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> Acked-by: Oleg Nesterov <oleg@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Don't allow a child root partition under an isolated parentWaiman Long
Although it is not typical for real users to create a complicated nested partition hierarchy, it is currently allowed to create nested root and isolated partitions. In addition, disabling or invalidating a partition should always be allowed especially when the cpuset is being deleted. With a nested root and isolated partition hierarchy, it is possible that all active housekeeping CPUs can be exhausted when disabling or invalidating a root partition and returning its CPUs to the isolated parent. One way to avoid this problem is to disallow the creation of a root partition under an isolated parent. It is indeed a new limitation imposed on the existing cpuset partition functionality. However an isolated partition is usually a left node where scheduling and some other kernel noises are disabled. It doesn't make much sense to create a child root partition under an isolated parent and remove some CPUs from the set of isolated CPUs again. Creation of a child isolated partition under a root or isolated parent is still allowed. If a partition with child root partitions is converted to isolated, the child root partitions will be invalidated. The cgroup-v2 documentation is also updated to reflect this new limitation. New test cases are added to test_cpuset_prs.sh to verify this new behavior. Some existing test cases are also updated accordingly. Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Properly disabling partition when partition state switching failsWaiman Long
Before commit 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping"), the partition state can be freely switched from "root" to "isolated" and vice versa. After that commit, the switch from "root" to "isolated" can fail if it exhausts all the housekeeping CPUs. Later on, the switch from "isolated" to "root" can also fail if some of the partition CPUs are boot-time isolated by "isolcpus". A local or remote partition is made invalid when the switch fails. However there are 2 problems with the existing invalidation code. 1) Both the subpartitions_cpus and isolated_cpus are not properly updated. 2) In the case of remote partition with remote_partition flag set, it is not cleared. Fix this by calling the proper partition disabling function in this case. On an x86 test system with boot option "isolcpus=10 cgroup_debug" set and more than 16 cores, the following commands were executed after boot. # cd /sys/fs/cgroup # echo +cpuset > cgroup.subtree_control # cat cpuset.cpus.isolated 10 # mkdir A # echo 10-12 > A/cpuset.cpus # echo isolated > A/cpuset.cpus.partition # cat A/cpuset.cpus.partition isolated Before the patch: # echo root > A/cpuset.cpus.partition # cat A/cpuset.cpus.partition root invalid (partition config conflicts with housekeeping setup) # cat cpuset.cpus.isolated .__DEBUG__.cpuset.cpus.subpartitions 10-12 10-12 After the patch: # echo root > A/cpuset.cpus.partition # cat A/cpuset.cpus.partition root invalid (partition config conflicts with housekeeping setup) # cat cpuset.cpus.isolated .__DEBUG__.cpuset.cpus.subpartitions 10 Reported-by: Tejun Heo <tj@kernel.org> Link: https://lore.kernel.org/lkml/7f4c57b26ad120ab30adf35653f1dd94@kernel.org/ Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup") Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping") Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn> Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Do housekeeping check before converting invalid partition to ↵Waiman Long
valid When a hotplug event happens or when update_cpumasks_hier() is called, an invalid partition may be converted back to valid. However check against housekeeping conflict isn't being done which can lead to a partition state that violates the housekeeping rules. Update the code to allow reviving an invalid partition only if it can pass the housekeeping check. Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup") Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn> Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Consider all the exclusive CPUs when doing housekeeping checkWaiman Long
When prstate_housekeeping_conflict() is called to perform housekeeping check with cpumask changes, the cpumask changes are passed in via add_cpus and del_cpus parameters. With this arrangement, the exclusive CPUs owned by the child partitions and not affected by the cpumask change, if any, are excluded in the analysis. The only places where there are separate add and delete masks are when the cpuset.cpus or cpuset.cpus.exclusive is being changed. In both cases, partition_cpus_change() is a good place to perform the housekeeping check. If we allow a root partition to be created under an isolated partition, it is possible the cpumask change will still leave active housekeeping CPUs but then no housekeeping CPUs will be left when the child partitions are disabled or invalidated. As a first step to protect against this possibility, all the exclusive CPUs of the cpuset should be considered to be owned by the current cpuset when doing housekeeping check. At the same time, we do want to know if some housekeeping CPUs are going to be removed from the isolated partition that there will be no danger of losing all the active housekeeping CPUs. So the current add_cpus paremeter is renamed as new_cpus to indicate that it includes the complete set of effective xcpus. With the new check in partition_cpus_change(), the housekeeping check in update_parent_effective_cpumask() with partcmd_update and newmask is now duplicative and can be removed. The housekeeping check in remote_cpus_update(), however, will still be useful as this function can be called directly from update_cpumasks_hier() without going through partition_cpus_change(). The update_parent_effective_cpumask() call from update_cpumasks_hier() is partcmd_update with no cpumask which still have the housekeeping check and so is covered. Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping") Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Consolidate isolated_cpus_can_update() into ↵Waiman Long
prstate_housekeeping_conflict() The isolated_cpus_can_update() and prstate_housekeeping_conflict() are checking different aspects of upcoming cpumask changes that may conflict with the current setting of the housekeeping cpumasks. There are places where both are called together. There are also places where only one of them is called. That inconsistency can contribute to missing check where invalid cpumask changes may be allowed to move forward. Fix that by consolidating isolated_cpus_can_update() into prstate_housekeeping_conflict() and call prstate_housekeeping_conflict() in all the places where either one of them or both are called. The exception is the validate_partition() function where the prstate_housekeeping_conflict() call is removed. It is because validate_partition() is called only from partition_cpus_change() where prstate_housekeeping_conflict() will be called from either remote_cpus_update() or update_parent_effective_cpumask() with partcmd_update if not for partition invalidatation or disablement. Now prstate_housekeeping_conflict() will be called in the following locations: - when a partition is enabled in remote_partition_enable() or in update_parent_effective_cpumask() with partcmd_enable*. - when a cpumask is updated in remote_cpus_update() or in update_parent_effective_cpumask() with partcmd_update. - when a partition state changes from root to isolated or vice versa in update_prstate(). Fixes: 4a74e418881f ("cgroup/cpuset: Check partition conflict with housekeeping setup") Fixes: 103b08709e8a ("cgroup/cpuset: Fail if isolated and nohz_full don't leave any housekeeping") Reviewed-by: Guopeng Zhang <zhangguopeng@kylinos.cn> Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: Fix a missing isolated_cpus update in update_cpumasks_hier()Waiman Long
It is found that when a local isolated partition is being invalidated because its parent partition is no longer valid, the isolated_cpus cpumask may not be properly updated. Those isolated CPUs may or may not needed to be removed from isolated_cpus depending on if there is a local isolated partition ancestor that will now own those isolated CPUs instead. Fix that by calling a new ancestor_partition() helper to return the closeset valid partition root ancestor. If that ancestor is PRS_ROOT, the isolated CPUs will have to be removed from isolated_cpus. For a PRS_ISOLATED ancestor, the isolated CPUs are checked against the ancestor's ones to see if any of them should be removed as it is possible that the invalidation of the parent may be caused by change in the effective xcpus of that ancestor. If any of the isolated CPUs should be removed, isolated_cpus_update() will be called under callback_lock. An existing test case in test_cpuset_prs.sh is modifed and a new one added to test that. Fixes: 11e5f407b64a ("cgroup/cpuset: Keep track of CPUs in isolated partitions") Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19cgroup/cpuset: use mems_allowed_seq when checking a policy nodemaskGregory Price (Meta)
A cpuset update changes the task's mems_allowed and rebinds its mempolicy in separate steps under mems_allowed_seq, but the check reads both without it. This can cause a spurious numa placement miss under some cpuset/mempolicy combinations. A reader that loads one mask before the update and the other after it can see a spurious empty intersection and drop the nodemask. For example, a task with cpuset.mems={0} and MPOL_BIND={0}. If the cpuset moves to mems={1,2}, the policy is rebound to {1}. The allocator can race and compare the old mems_allowed {0} against the new policy {1} - which will result in a dropped nodemask and a misplaced allocation on node 2. Simply retry the intersection until mems_allowed_seq is stable. This covers the task policy, which is rebound under mems_allowed_seq. VMA policies are rebound afterwards by mpol_rebind_mm(), outside mems_allowed_seq, and are not covered by this change. Fixes: cc9a6c877661 ("cpuset: mm: reduce large amounts of memory barrier related damage v3") Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Waiman Long <longman@redhat.com> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19sched_ext: Reset runnable_at when a task leaves its schedulerMichal Blaszczyk
The watchdog measures how long a task has been waiting from p->scx.runnable_at, which set_task_runnable() resets when the task becomes runnable after it last ran. ops_dequeue() never asks for a reset, as the dequeue of a waiting task is temporary, e.g. for a migration or an attribute change. Leaving the scheduler isn't temporary. A task which is switched out of SCHED_EXT while it's waiting keeps its runnable_at. When it's switched back and becomes runnable, its wait appears to have started before it left, and the time it spent in another class counts towards the timeout. If the watchdog checks before the task runs, it aborts the scheduler: sched_ext: class_switch_stall: runner[418] failed to run for 3.010s Here, the task waited for a few milliseconds before it was switched to SCHED_OTHER, slept outside of SCHED_EXT for three seconds, and was charged for all of it as soon as it woke up on SCHED_EXT again. The same happens when scx_sub_enable_workfn() hands a waiting task over to a new sub-scheduler, which can then be blamed for time the task waited under the parent. scx_rehome_task() isn't affected, as it moves the task through SCX_TASK_INIT, which resets runnable_at. The stale runnable_at also skews the default core-sched ordering in scx_prio_less(). Reset runnable_at in scx_disable_task(), when the task leaves the control of its scheduler. Fixes: 8a010b81b3a5 ("sched_ext: Implement runnable task stall watchdog") Cc: stable@vger.kernel.org # v6.12+ Assisted-by: LLM Signed-off-by: Michal Blaszczyk <michalblk@google.com> Reviewed-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-19sched_ext: Leave suspend and hibernation out of the ecaps reportsTejun Heo
Suspend and hibernation take down every non-boot cpu and bring them back. A loss and a regain per cpu to every sub-scheduler would be noise. Leave them out: a cpu they take down reports no loss and keeps the caps last delivered, and the reseed after resume reports only the net change. Their downs are the ones with cpuhp_tasks_frozen set between the PM notifications. A kexec jump sets the flag too but sends no PM notification. Two things need care. A cpu that suspend or hibernation takes down doesn't wait for its syncs, and no ecaps_report() mark can be pending on it then. The attach report runs while the attaching task waits uninterruptibly, which holds off the freezer, and the detach report holds cpus_read_lock(). The detach report reaches only active cpus, and the loss on a cpu that suspend or hibernation took down was never delivered. The detach waits for those cpus to come back, so every loss is delivered. They come back before the tasks thaw, so the wait doesn't hold up the freezer. If one fails to come back, the hierarchy is aborted at the end of the PM cycle. v2: Fix why no report mark is pending at a PM down, add a data_race(). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19sched_ext: Report the ecaps loss when a cpu goes offlineTejun Heo
When a cpu goes offline, a sub-scheduler's effective caps on its cid are cleared, but neither the sub-scheduler nor its parent is notified. The sub-scheduler keeps treating the cid as usable, and the parent can't act on the loss, e.g. reset the cpuperf target. Report the loss before the root's ops.cpu_offline() and the regain after its ops.cpu_online(). The zeroing at offline was the last change to a sched's ecaps outside a delivery, so they now change only with one. Syncs on an inactive cpu are processed instead of discarded, and the reseed reschedules the cpu itself, as they got no resched when queued. Suspend and hibernation now report their cpu downs and ups as well. This is temporary, and a later patch leaves them out. Two things need care. A cpu going down marks a sync for each sub-scheduler, and its hotplug thread waits until each marked sync has been processed. The cpu's own ext picks process them, and the ext server is started in case a fair task holds the cpu. Only the marked syncs count, so a flood of changes on the cid can't hold up the hotplug. The detach report reaches only active cpus. It holds off cpu hotplug, so a cpu it skips has reported its loss as it went down, and no unreported loss is left to drop. Drop the cpuperf reset from scx_qmap's ops.sub_detach(), as its ops.sub_child_ecaps_updated() now resets the targets on the loss reports. v2: Reschedule from the irq work and document the regain order. v3: Pack nr_ecaps_offline with kick_sync_pending to avoid a hole (Andrea). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19sched_ext: Deliver ecaps only while the root has the cpu onlineTejun Heo
A sub-scheduler's ecaps ops run in dispatch context, and ops.cpu_online() and ops.cpu_offline() document that the cpu doesn't call ops.dispatch() outside them. The reseed of a cpu coming back is queued before the root's ops.cpu_online(), so the ops of a sub-scheduler and its parent can run for a cpu the root still considers offline. Add SCX_RQ_OPS_ONLINE, set while the root considers the cpu online, from its ops.cpu_online() or its enable until its ops.cpu_offline(). A sync outside that span leaves the ecaps alone, and the reseed, now queued after ops.cpu_online(), reports the net change. SCX_RQ_ONLINE can't be used, as the core changes it ahead of the root's view on both edges. A cpu going offline still drops its caps without a report. While at it, fix the SCX_RQ_ONLINE comments, which claimed that the flag tracks ops.cpu_online() and ops.cpu_offline(). v2: Fix the SCX_RQ_ONLINE comments. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19sched_ext: Report a sub-scheduler's ecaps on attach and detachTejun Heo
A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(), but the caps granted from ops.sub_attach() reach it only after its enable bypass lifts, when it is already scheduling its tasks. Report the caps at the end of the enable, before bypass lifts. On disable, report their loss before any task goes back to the parent, so that the parent hears of the loss of every cap it was told of on an active cpu. ops.sub_detach() remains where the parent cleans up after the sub-scheduler. Deliveries no longer depend on bypass, so ops.sub_ecaps_updated() and ops.sub_child_ecaps_updated() also run while a scheduler bypasses. The replay at unbypass goes, and the enable's bypass returns to after ops.sub_attach(). A sub-scheduler's ecaps now change only with the deliveries that report them, so that whenever its ops run under the rq lock, its caps are the ones it was last told of. A sync before the attach report leaves them alone, and the detach report zeroes them as it delivers the loss. A cpu going offline still drops its caps without a report, and those are the only changes not delivered. clear_all_caps() drops that unreported loss on inactive cpus. The following things need care. A report reschedules only the cpus the sub-scheduler was ever granted a cap on. Rescheduling every cpu would disrupt a large machine split many ways, and which cpus still owe a delivery can't be read off the caps. The report and a queued sync reschedule the cpu from hard irq work on that cpu, not with a kick or a direct resched. dispatch_one() processes the syncs only at its start, and a kick only guarantees that a scheduling event completes after it, which can be the pick in progress. That pick has already processed the syncs and clears the resched, and a cpu that then idles or runs a task with an infinite slice leaves the report waiting. A pick runs with IRQs off, so the irq work runs after any pick in progress on that cpu. Under core scheduling, a sibling's committed pick can switch the cpu without dispatch_one(), and a sync queued after that pick is processed only at the cpu's next scheduling event. A cpu running a task of a higher class isn't rescheduled, as its pick would return that task again. While the root bypasses, a queued sync doesn't reschedule the cpu, as a grant or revoke can run in the PM syscore phase, where irq work can malfunction. The report never runs there and reschedules the cpu itself. The report waits on an rcuwait woken from hard irq work. Under core scheduling, a cpu going down can process a sibling's marked sync after hotplug has parked its irq_work thread. On PREEMPT_RT, a regular irq_work queued there would be left behind, and a hard irq can't take a wait queue's lock. v2: Reschedule from hard irq work on the cpu instead of a kick (sashiko). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19sched_ext: Free a sub-scheduler a grace period after discarding its ecaps syncsTejun Heo
The free path removes a sub-scheduler's ecaps sync node from each cpu's list and then cycles the rq lock until the node reads off-list, as a sync batch can hold the node privately. Cycling the lock is correct only because the sched is bypassing and a bypassing sched's sync calls no op, so the batch touches the pcpu within one rq lock hold. Tying the free path to what the sync does for a bypassing sched is unusual and brittle, and it gets in the way of changing the sync. Delivering ecaps to a bypassing sched or its parent would break the wait, as a delivery drops the rq lock. Wait for the batches with RCU instead, the usual way to wait for lockless users before a free. Discard the nodes after the first grace period as before, and free the sched after a second one. The first grace period waits for the paths that found the sched before it was unlinked, such as the sub-cap kfuncs and the cpu hotplug hooks. Until they finish, they can queue its syncs, and a cpu can take a queued node into a sync batch that the first grace period doesn't wait for. After it, nothing queues the syncs, and the discards remove the nodes still queued. A batch that took a node keeps irqs off from llist_del_all() to its last pcpu write, even across the rq lock drops of a delivery. The second grace period waits for it whether or not it called an op. The exit and disable irq_works, the helper kthread and the bypass load balancing timer are stopped ahead of the per-cpu checks as before, so they move into scx_sched_quiesce_rcu_work() with the checks. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19sched_ext: Factor out cid_pshard()Tejun Heo
Factor the cid to pshard lookup out of the ecaps sync paths into cid_pshard() so that new sync paths can share it. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-19bpf: Set up the arena base only in functions that use itKumar Kartikeya Dwivedi
A JIT sets up the arena base in the prologue of every function of a program that has an arena, a callee-saved register loaded with a 64-bit immediate and spilled around the function on x86. That is paid for by every program with an arena, whether it touches the arena or not, and now that libbpf binds the arena of an object to every program in it, by every program of an object that happens to contain an arena. Decide per function, on its final insns, right before it is JITed. bpf_prog_has_arena_insns() looks for an insn that works with the base: an access through an arena pointer, or through a number in a program loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32 form by then, a cast to the user address space keeps BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which the JIT rebases, has a function model that says so. A load of an __arena struct member is rewritten with an immediate and needs none; dereferencing the result does. Looking at the final insns rather than recording the uses during verification keeps the answer in step with what the JIT sees. A constant NULL passed to a nullable __arena kfunc argument takes the early exit for NULL arguments in the verifier, and the JIT rebases the call all the same; an insn that is dead code has disappeared by then. bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single function program, jit_subprogs() for each subprogram once it has its kfunc table and arena. It implies that the program has an arena, as the scan says no without one. The JITs with arena support pick up the arena base and user address only when bpf_prog_uses_arena_insns() says so, and so generate the prologue and epilogue of a program bound to an arena it never touches as if it had none. The other users of the arena address of a program stay as they are on purpose. The struct_ops trampoline converts __arena arguments on entry and arena_member_fixup() converts loaded __arena members after the load, both with the base as an immediate. Both have to happen whether or not the program dereferences the result, it may compare, store or pass it on, so neither depends on the base register nor on the flag. The arena fault report reads the user address of the arena the faulting insn belongs to, and such an insn sets the flag anyway. check_map_prog_compatibility() checks the user address when the arena is bound, before any use is known. The rest, in arena.c and sched_ext, works with the map and not with a program. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com
2026-10-19bpf: Turn loads of __arena pointer members into arena pointersKumar Kartikeya Dwivedi
A kernel struct shared with BPF programs may want to point at arena memory, e.g. a sched_ext argument struct that the kernel builds in the scheduler's arena and that refers to another arena object. A pointer member reaches the program typed as a kernel pointer, so sched_ext publishes a plain u64 holding a pre-translated arena address that the program casts before use. Translate in the verifier instead. When a load through a kernel struct pointer lands on a member tagged __arena, btf_struct_walk() reports it and the destination register becomes PTR_TO_ARENA. The member holds the kernel address of the arena memory; the fixup pass appends the conversion to the arena pointer form that the struct_ops trampoline applies to __arena arguments, after a plain load or a load-acquire of the member: r1 = *(u64 *)(r2 + off) if r1 == 0 goto +1 // __nullable members and probed loads w1 -= lo32(kern_vm_start) A probed load, through a plain or an untrusted pointer, keeps the NULL test as well: it reads as 0 when it faults, and the subtraction must not turn that into an arena pointer. An acquire load only comes through a pointer that cannot fault. Stores to kernel structs are rejected as before, and with them the atomics that write such a member. A member inside a multi-member union is rejected too: an offset into a union resolves to its first member, so a program reading another member there would receive an arena pointer it did not ask for. Loads through program-allocated objects and program BTF are unaffected, there the tag is what the compiler emits for the arena address space. The rewrite is per insn while verification is per path, so the aux data records what an insn loaded on any path: an insn that loaded an arena pointer member on one path and a different member on another is rejected by the fixup pass, while paths disagreeing only on __nullable keep the NULL test, which a never-NULL member can afford. As for __arena arguments, the program needs an associated arena, which also supplies the base to subtract. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-8-memxor@gmail.com
2026-10-19bpf: Recognize __nullable on pointer members of kernel structsKumar Kartikeya Dwivedi
A pointer member of a kernel struct that the verifier derives as trusted or RCU-protected is assumed to be non-NULL, so a load through it is a plain dereference. The BTF_TYPE_SAFE_*_OR_NULL lists exist for members known to be NULL at times, but they live in the verifier and have to be extended by hand. Teach btf_struct_walk() to report the "nullable" BTF type tag on a pointer member as PTR_MAYBE_NULL. The tag combines with any other tag on the member; it only says that the pointer may be NULL. The verifier keeps the flag where NULL would otherwise be assumed away: on trusted and RCU results, which a program dereferences as they are, and on per-CPU results, which the per-CPU helpers shift without a check. Such a pointer has to be tested before it is dereferenced or passed on. Plain and untrusted results are probed anyway and lose the flag, as the BTF_TYPE_SAFE_*_OR_NULL lists never applied to them either, so a tagged member of an existing struct asks no more of the programs that read it than a list entry does, and the tag can take the place of the lists. The next patch uses the tag on __arena members to decide whether the injected address translation has to preserve NULL. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-7-memxor@gmail.com
2026-10-19bpf: Check exclusive maps once the program digest is finalKumar Kartikeya Dwivedi
An exclusive map is bound to the digest of its program, which the verifier compared against prog->digest in check_map_prog_compatibility() as the map was added to the program. For a map an insn refers to that happens in check_and_resolve_insns(), after the digest is computed at the top of that function. For a map bound through a continuous fd_array it happens in process_fd_array(), which runs earlier, so for an unsigned program the comparison saw a zeroed digest and the load failed with -EACCES, "program's hash doesn't match map's excl_prog_hash", for the very program the map is exclusive to. Computing the digest earlier would not do: it has to cover the insns as the kernel verifies and runs them, after the CO-RE relocations it applies, or a loader could submit the insns of the exclusive program together with relocations that change them and pass the check with a different program. Leave the digest where it is and move the comparison instead: check every used map against the digest at the end of check_and_resolve_insns(), whether an insn referred to it or the array bound it. The claims a program makes on its maps move behind the check as well: bpf_map_claim(), which records the program as the user of the map for resolve_func_ptrs(), and the takeover of an insn array by bpf_insn_array_init(). Both ran as the map was added, so a program that is not the owner of an exclusive map could have flipped the owner's single-user state to shared, which a failed load does not undo, or held the owner's insn array while it was being verified. Now they run once every map has passed the check, and a rejected program leaves the maps as it found them. Nothing consults a claim before resolve_func_ptrs() and the fixup passes, which run later. The early digest of a signed program goes with the early comparison. Commit b707068e0ed9 ("bpf: Verify signed loader metadata at load time") computed it only so that the maps of a signed loader passed the check in process_fd_array(); the signature itself is verified over the submitted insns and the frozen maps by bpf_prog_verify_signature(), which does not read the digest. The maps of a signed program are now checked against the digest of the final insns like everyone else's, which covers the relocations the kernel applies and the signature does not; a loader program carries none. Fixes: baefdbdf6812 ("bpf: Implement exclusive map creation") Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-2-memxor@gmail.com
2026-10-19tracing/histograms: Verify event_call identity when updating synth_event ↵Lee Jones
refcount In event_hist_trigger_parse() and hist_unreg_all(), the histogram registration and unregistration paths look up find_synth_event() using trace_event_name(file->event_call) and adjust se->ref if a match is found. Because find_synth_event() only compares the event basename, registering or removing a histogram on a non-synthetic trace event whose basename matches a synthetic event erroneously modifies the synthetic event's reference count. If a histogram is registered on the non-synthetic event before the synthetic event is created and removed afterwards, se->ref is decremented without a matching increment, allowing the synthetic event to be freed while an active histogram action still holds a pointer to it. Introduce find_synth_event_call() to verify that &se->call matches file->event_call before adjusting se->ref. Fixes: 4b147936fa50 ("tracing: Add support for 'synthetic' events") Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20261019120730.2342739-1-lee@kernel.org Signed-off-by: Lee Jones <lee@kernel.org> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-10-19ftrace: Initialize modname and offset in print_rec()Aaron Ahmed
print_rec() leaves 'modname' and 'offset' uninitialized and kallsyms_lookup() only sets them on success. When the lookup fails like when a module is being loaded or unloaded, 'modname' holds a garbage value, passes the 'if (modname)' check, and is dereferenced by seq_printf(). Oops: general protection fault RIP: 0010:string+0x48/0xe0 Call Trace: vsnprintf+0x1f5/0x650 seq_printf+0x8e/0xb0 print_rec+0x6c/0xc0 t_show+0x4c/0x440 seq_read_iter+0x119/0x480 This can be triggered by reading available_filter_functions while a module is being loaded or unloaded. Initialize both variables so a failed lookup skips the module name. Cc: stable@vger.kernel.org Fixes: b39181f7c690 ("ftrace: Add FTRACE_MCOUNT_MAX_OFFSET to avoid adding weak function") Link: https://patch.msgid.link/20261016230403.4916-1-aarnahmd@amazon.com Tested-by: Gavin Inglis <giinglis@amazon.com> Signed-off-by: Aaron Ahmed <aarnahmd@amazon.com> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-10-19ftrace: Let ftrace_call_adjust() reject a patch siteJose Fernandez (Anthropic)
When ftrace_call_adjust() returns 0 for a patch site, ftrace_process_locs() does not skip the site. The NULL test runs before the adjustment since commit 6eeca746fa5f ("ftrace: Test mcount_loc addr before calling ftrace_call_addr()"), so the 0 is stored as the record's ip and ftrace_init_nop() later hits the BUG_ON() in arm64's patch_map(). Test the adjusted address too and count the site as skipped. Link: https://patch.msgid.link/20260924-b4-arm64-callops-kcfi-v2-2-587865b6d991@linux.dev Reviewed-by: Ben Cressey <ben@cressey.dev> Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev> Assisted-by: LLM Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-10-19ftrace: Do not leak a module's empty page groupJose Fernandez (Anthropic)
When every mcount_loc entry of a module is skipped, ftrace_process_locs() leaves the module's first page group linked with no records. ftrace_release_mod() matches a module's groups by records[0].ip, which is 0 here, so the group is never freed. While it stays linked, ftrace_free_mem() reads pg->records[pg->index - 1] with pg->index == 0 on every later module load, as lookup_rec() did before commit ee92fa443358f ("ftrace: Fix invalid address access in lookup_rec() when index is 0"). Unlink and free the new page groups when none of them got a record. Link: https://patch.msgid.link/20260924-b4-arm64-callops-kcfi-v2-1-587865b6d991@linux.dev Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/all/20260904013350.5141B1F000E9@smtp.kernel.org/ Assisted-by: LLM Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2026-10-19Merge branches 'pm-sleep', 'pm-em', 'pm-cpu', 'pm-misc' and 'pm-tools'Rafael J. Wysocki
Merge updates related to system sleep, an energy model handling update, a generic CPU PM driver update, a clock framework PM update, and a pm-graph utility update for 7.4: - Ignore page-zero RAM in E820 checksum during hibernation on x86 to avoid rejecting images created after kexec() (Matthias Goergens) - Add resume event mapping for PMSG_POWEROFF to the PM core to allow it to carry out device shutdown in the future (Mario Limonciello) - Add DPM watchdog to prepare/late/early/noirq/complete phases of device suspend and resume (Mayank Rungta) - Update swsusp.txt references to swsusp.rst in hibernation documentation (Li Jun) - Fix use-after-free of perf domain in netlink doit handlers for energy models (Fan Wu) - Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it was removed inadvertently in previous development cycles to avoid use-after-free (Bradley Morgan) - Unregister CPU PM notifier outside the timer lock in the timer-ti-dm clocksource driver (Bradley Morgan) - Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc comment and typo "hierachy" in a comment in pm-graph (Hemanth Selam) * pm-sleep: x86/hibernate: Ignore page-zero RAM in E820 checksum PM: sleep: Add resume event mapping for PMSG_POWEROFF PM: sleep: Add DPM watchdog to prepare/late/early/noirq/complete phases PM: hibernate: docs: update swsusp.txt references to swsusp.rst * pm-em: PM: EM: Fix use-after-free of perf domain in netlink doit handlers * pm-cpu: PM: CPU: Restore synchronize_rcu() to cpu_pm_unregister_notifier() clocksource/drivers/timer-ti-dm: Unregister CPU PM notifier outside of the timer lock * pm-misc: PM: clk: fix typo "sucessfully" in comment * pm-tools: tools: power: pm-graph: fix typo "hierachy" in comment
2026-10-19Merge branch 'pm-cpufreq'Rafael J. Wysocki
Merge cpufreq updates for 7.4: - Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy, Sumeet Pawnikar, and Xueqin Luo) - Fix wrong behaviors on multi-node servers in the Loongson cpufreq driver (Huacai Chen) - Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu) - Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue) - Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq driver (Mario Limoncielloi, Wentao Guan) - Initial support for maximum frequency capacity handling on Zen6 client in amd-pstate (Mario Limonciello) - Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option (K Prateek Nayak) - New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello, Vishal Badole) - amd-pstate documentation fix (Karl Mehltretter) - Restrict deferred downscaling to low load for shared policies in the conservative cpufreq governor to avoid undesirable CPU frequency reduction in some cases (Shengming Hu) - Serialize sugov_start() against rate limit updates in the schedutil cpufreq governor to avoid using a stale rate limit value (Hui Su) * pm-cpufreq: (34 commits) cpufreq: loongson3: Replace IOCSR read/write with MMIO ones cpufreq: loongson3: Use global physical CPU ID in get/target callbacks cpufreq: loongson3: Replace per-package mutex with per-node cpufreq: loongson3: Adjust the width of id and val in smc_message cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64 cpufreq: schedutil: Serialize sugov_start() against rate limit updates arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend cpufreq: mediatek: Enter system sleep at the suspend OPP cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing cpufreq: qcom-nvmem: Add IPQ9650 support cpufreq: Add Milos to cpufreq-dt-platdev blocklist cpufreq: Use %pe to print error pointers symbolically rust: rcpufreq_dt: add module alias cpufreq: sti: avoid NULL dereference in dev_err() cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv cpufreq: sparc-us2e: fix frequency table index copy-paste error rust: cpufreq: reject NULL from cpufreq_cpu_get() cpufreq: amd-pstate-ut: Fix max_freq and EPP test failures on Zen6 cpufreq: amd-pstate: Propagate cppc_set_auto_sel() errors on mode change cpufreq: amd-pstate: Restore previous mode when changing driver mode fails ...
2026-10-19bpf: Refuse the prologue for an extension if callers keep packet pointersAlexei Starovoitov
An extension that writes to the packet of a tc, lwt_xmit or sk_skb program starts with bpf_unclone_prologue(). If the skb is cloned, that gives it a new head. It does not count as changing packet data. changes_pkt_data goes by the helpers and kfuncs that a program calls, and whether it gets a prologue is known only after it has been compared with its target. So the extension can replace a global function that leaves the packet alone, and the callers of that function keep their packet pointers across the call. These point into the old head, which is left to the other clone. Once that is freed, by the reader of a packet socket for example: BUG: KASAN: slab-use-after-free in bpf_prog_53bc21b9810cadbc_t_read+0x2e8/0x36e Read of size 1 at addr ffff888107bfe73c by task tt11/102 Freed by task 112: kfree+0x161/0x500 skb_release_data+0x2dc/0x340 consume_skb+0xa6/0x1c0 packet_recvmsg+0x4b3/0xa20 Nothing is wrong if the target program writes to the packet too: its own prologue has uncloned the skb, and that of the extension does nothing. Nor if the callers hold no packet pointers, like a dispatcher, or if the main function is replaced. Note for every global function that leaves the packet alone if it is called while packet pointers, or slices of the dynptr of a packet, are held. What the callers of a global function hold is not known while it is verified, so the functions that it calls are taken for such as well. Refuse an extension with a prologue for such a function, unless the target program has a prologue, both when the extension is loaded and when it is attached to another target. Functions can share a type id in func_info. The first of them is the one that is replaced, so look it up in one place. Fixes: 7e40781cc8b7 ("bpf: verifier: Use target program's type for access verifications") Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20261019011328.1843764-1-alexei.starovoitov@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-10-19bpf: Enforce trusted BTF ID argument provenanceYiyang Chen
A global subprogram parameter tagged __arg_trusted is verified with PTR_TRUSTED, but the common call-site type check accepts bare PTR_TO_BTF_ID and MEM_RCU registers. The callee can consequently pass a pointer with weaker provenance to a kfunc that does not accept RCU-protected pointers. RCU protection only guarantees that the object remains allocated through a grace period. It does not guarantee that an object's refcount is nonzero, so an RCU-protected pointer cannot satisfy the trusted-pointer contract. For example, an RCU-protected bpf_cpumask loaded from a map may have a zero refcount. Passing it through __arg_trusted lets the callee pass it to bpf_cpumask_acquire(), whose plain refcount_inc() can retain an object that will be freed after the grace period. Make the existing provenance check contract-driven. Mark generated BTF ID kfunc argument types with PTR_TRUSTED, and add MEM_RCU only for KF_RCU kfuncs. Normalize PTR_MAYBE_NULL when the argument contract permits it. Then use the argument flags to require trusted or referenced provenance and allow MEM_RCU only when the contract says so. Split is_trusted_reg() so the normalized type is used for type and modifier checks while reference lookup still uses the original register. Conversion of scalar-struct BTF ID arguments to fixed-size memory continues to construct a new ARG_PTR_TO_MEM type without PTR_TRUSTED. Fixes: e2b3c4ff5d18 ("bpf: add __arg_trusted global func arg tag") Suggested-by: Amery Hung <ameryhung@gmail.com> Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn> Reviewed-by: Amery Hung <ameryhung@gmail.com> Link: https://lore.kernel.org/bpf/20261007-a3-arg-trusted-v4-v4-1-253ef3a1a69a@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
2026-10-18bpf: Fix incorrect handling of user flags by percpu hash map updatesMasoud Aghasi
For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps, htab_map_check_update_flags() and check_flags() are not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases. For example, when using (BPF_F_CPU | BPF_EXIST) or (BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem() on a percpu hash map, the BPF_EXIST flag does not prevent new insertions as expected and BPF_NOEXIST flag does not prevent modification of existing entries as expected. This patch fixes the bug by adding proper flag validations and checks. Before this patch, htab_map_check_update_flags() rejected (BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with -EINVAL. After the patch those combinations are accepted. This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted. Fixes: c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps") Signed-off-by: Masoud Aghasi <maghasi@disroot.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://patch.msgid.link/20261008120734.1565972-3-maghasi@disroot.org
2026-10-18bpf: Fix incorrect handling of user flags in bpf_percpu_array_updateMasoud Aghasi
For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update() is not considering the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases. For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination with bpf_map_update_elem() results in an incorrect EINVAL error response, even though the flag combination is valid. This patch fixes the bug by adding proper flag validations and checks. Before this patch, bpf_percpu_array_update() rejected BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with -EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted. After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST result in -EEXIST. This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST), with or without BPF_F_CPU, which was previously accepted. Fixes: 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps") Signed-off-by: Masoud Aghasi <maghasi@disroot.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Leon Hwang <leon.hwang@linux.dev> Link: https://patch.msgid.link/20261008120734.1565972-2-maghasi@disroot.org
2026-10-18bpf: Determine freplace staticness from BTF linkageKaFai Wan
bpf_check_attach_target() rejects a freplace whose target is a static function, using the subprog "unreliable" flag as a proxy for "static". That flag is raised whenever a subprogram's BTF prototype disagrees with the registers at its call site. `147400e9c27c ("bpf: Compute parameters liveness in bpf_compute_live_registers()")` made that work for static callees: a static function with an unused argument no longer matches its BTF prototype at the call site and is marked unreliable. Since LLVM commit `29b36a16f111 ("[BPF] Emit ABI-accurate BTF prototypes for DW_CC_nocall")` in 23.1 [1] that premise is gone: BTFDebug may now drop arguments removed by the optimizer, so for __noinline static int subprog(struct __sk_buff *ctx) { return bpf_ktime_get_ns(); } compiled with clang 23 the BTF describes subprog() as [75] FUNC_PROTO '(anon)' ret_type_id=2 vlen=0 [76] FUNC 'subprog' type_id=75 linkage=static There is then no argument left to compare, the subprog stays reliable, and the check in bpf_check_attach_target() is skipped. The freplace is then rejected by the later linkage check instead, reporting "subprog() is not a global function", and the selftest "exceptions" fails. Determine staticness from the BTF linkage instead, and keep the unreliable check for global functions whose BTF does not match their call site. There is no functional change: freplacing a static function was rejected before and is still rejected with the same -EINVAL. Selftest "exceptions" is updated to match. [1] https://github.com/llvm/llvm-project/pull/198426 Fixes: 147400e9c27c ("bpf: Compute parameters liveness in bpf_compute_live_registers()") Signed-off-by: KaFai Wan <kafai.wan@linux.dev> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261018005348.2195130-2-kafai.wan@linux.dev
2026-10-18Merge tag 'sched-urgent-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler fixes from Ingo Molnar: - Fix __sched_setscheduler() class change race causing lost reschedules, exposed by recent RCU changes (K Prateek Nayak) - Fix s64 multiplication overflow in place_entity()'s vlag scaling code that can corrupt an entity's placement (Hui Su) * tag 'sched-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: sched/fair: Avoid overflow in place_entity() sched/core: Call wakeup_preempt() for queued tasks during class change
2026-10-18Merge tag 'perf-urgent-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull perf events fixes from Ingo Molnar: - Fix x86 Intel uncore driver bug on Diamond Rapids CPUs that erroneously always-cleared the invert bit (Guixiong Wei) - Fix accounting bug in the perf mlock 'gift' code causing unrelated RLIMIT_MEMLOCK functionality to fail (Lorenzo Stoakes) - Fix perf_pmu_migrate_context() bug leaving events behind during CPU hot-unplug (Haris Okanovic) - Update cgroup time when rotating multiplexed events, to fix time_running & scaled counts accounting bug (Antoine Huret) * tag 'perf-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: perf: Update cgroup time when rotating multiplexed events perf: Fix perf_pmu_migrate_context() leaving events behind perf: Track the perf mlock gift separately perf/x86/intel/uncore: Fix DMR CXL CM invert filter
2026-10-18Merge tag 'locking-urgent-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull locking fixes from Ingo Molnar: - Fix FUT_OFF_MMSHARED futex hashing bug that can cause missed wakeups or spurious wakeups on futex-waiting tasks (Jann Horn) - Fix lockdep context tracking bug in the deferred hrtimer rearming entry code, which causes a lockdep chain leak that will eventually turn lockdep off (Mikhail Gavrilov) * tag 'locking-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: entry: Account the deferred hrtimer rearm as hardirq context for lockdep futex: Fix mm reuse handling for FUT_OFF_MMSHARED
2026-10-18bpf: Zero-extend the index in inlined bpf_get_func_arg()Woojin Ji
bpf_get_func_arg() takes a u32 index, but the inlined expansion compared only the low 32 bits of R2 with the argument count and then shifted the full 64-bit register to compute the context address. R2 is ARG_ANYTHING, so nothing guarantees that its upper bits are zero: an index of 1ULL << 32 passes the bounds check as index zero while the load is performed at ctx + (1ULL << 35). Shift the index as a 32-bit value. When the 32-bit bounds check passes, the low 32 bits of R2 are below R0, and R0 is at most 255: the trampoline arm masks the argument count with 0xFF and the raw_tp arm uses the BTF argument count minus one, which the verifier caps at MAX_BPF_FUNC_ARGS. The 3-bit shift therefore cannot overflow and it clears the upper bits of the register. Widening the comparison instead would reject indices whose low 32 bits are valid. The check itself is unchanged, so this cannot change which programs are accepted, only the address that is computed. Fixes: f92c1e183604 ("bpf: Add get_func_[arg|ret|arg_cnt] helpers") Signed-off-by: Woojin Ji <random6.xyz@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261018-get-func-arg-v2-1-048848b3eddf@gmail.com
2026-10-17sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueueTejun Heo
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips unmarked tasks without dropping either, so a long run of unmarked tasks is walked in one hold of both locks. Count the visited tasks instead of the re-enqueued ones and drop both locks every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as often as before. Suggested-by: Andrea Righi <arighi@nvidia.com> Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-18bpf: Refuse packet slices for packet-changing helpersJosef Bacik
The helpers which may change packet data do not take a packet pointer as a memory argument: they have no pkt_access. They make the skb writable, or make room in it, before they read what the argument points to, and that can reallocate the head which it points into. A slice from bpf_dynptr_slice() of the dynptr of the skb points into the same head, but it is a PTR_TO_MEM, and the verifier lets it pass. bpf_lwt_push_encap(), bpf_lwt_seg6_store_bytes() and bpf_lwt_seg6_action() then read the old head, which has been freed. bpf_skb_store_bytes() follows the head by now, but a slice is no better than a packet pointer there. Refuse a slice of a packet where a packet pointer is refused for that reason. All other helpers take slices as before. bpf_dynptr_write() and bpf_dynptr_copy() can be used to copy inside of a packet. Fixes: 2012c867c800 ("bpf: verifier: Accept dynptr mem as mem in helpers") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-17-alexei.starovoitov@gmail.com
2026-10-18bpf: Unclone the skb for kfuncs that write through a dynptrAlexei Starovoitov
bpf_dynptr_copy(), bpf_dynptr_memset() and the kfuncs that read user or kernel memory into a dynptr ask bpf_dynptr_slice_rdwr() for a slice and write to it. For the linear part of an skb that is a pointer into its head. bpf_dynptr_slice_rdwr() counts on the verifier to have uncloned the skb, see bpf_unclone_prologue(). The verifier does so if the program itself calls bpf_dynptr_slice_rdwr() or writes to the packet. A program that only uses these kfuncs gets no prologue, and changes the data of all the clones of the skb. A packet socket, whose clone is taken before tc ingress runs, sees what bpf_dynptr_memset() has written there. Treat a call that may write to an skb like a direct write. Fixes: daec295a7094 ("bpf/helpers: Introduce bpf_dynptr_copy kfunc") Fixes: a498ee7576de ("bpf: Implement dynptr copy kfuncs") Fixes: 5fc5d8fded57 ("bpf: Add bpf_dynptr_memset() kfunc") Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-13-alexei.starovoitov@gmail.com
2026-10-18bpf: Refuse the prologue for an extension that cannot take itAlexei Starovoitov
A tc, lwt_xmit or sk_skb program that writes to packet data starts with bpf_unclone_prologue(). It takes the skb from R1, and if the skb is cloned it calls bpf_skb_pull_data() and restores R1 only. That suits the main function of a program. An extension gets the prologue too, but it starts with the arguments of the function that it replaces. If the first one is not the context, something else is taken for an skb: BUG: KASAN: stack-out-of-bounds in skb_ensure_writable+0x84/0x1a0 Read of size 8 at addr ffffc90000270ad0 by task uncl/105 skb_ensure_writable+0x84/0x1a0 bpf_skb_pull_data+0x32/0xf0 bpf_prog_853eb1991d35e093_ext_slice+0x4f/0x321 bpf_prog_9eb18752b5d0959a_t_glob+0x2ac/0x2de BUG: unable to handle page fault for address: 0000000082298821 And whatever R2-R5 held is lost, while the verifier goes on trusting pointers in them. Refuse an extension that needs a prologue unless it gets the context as its first argument and scalars beside it. A scalar is still clobbered if the skb is cloned, which the verifier can cope with. Extensions of XDP, sk_msg and sockopt programs get no prologue and load as before. Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions") Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-12-alexei.starovoitov@gmail.com
2026-10-18bpf: Don't take a dynptr argument for a local dynptrJosef Bacik
A global function is verified on its own. A dynptr which it gets as an argument is assumed to be of type BPF_DYNPTR_TYPE_LOCAL, but the callers may pass one of any type, that of an skb for example. The type decides whether a write through the dynptr invalidates packet pointers, and whether a slice of it is invalidated together with them. Inside of the function neither happens. It can take a packet pointer from its context argument, or a slice from the dynptr, write through the dynptr and go on using the pointer, although the write may have reallocated the head of the skb. Assume the type of an skb dynptr instead. Nothing becomes forbidden for the dynptr itself, and a clone inherits the caution. The dynptr which the callback of bpf_user_ringbuf_drain() gets is known to be local, and stays so. Fixes: a64bfe618665 ("bpf: add support for passing dynptr pointer to global subprog") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-11-alexei.starovoitov@gmail.com
2026-10-18bpf: Let bpf_dynptr_write() mark a global functionJosef Bacik
The caller of a global function loses its packet pointers if the function may change packet data. That is found out by a scan for calls of the helpers which bpf_helper_changes_pkt_data() lists. bpf_dynptr_write() is not on the list, because it depends on the type of the dynptr: check_helper_call() looks at it. The scan cannot do that, and so a global function which makes a dynptr from the skb, or gets one as an argument, and writes through it counts as harmless. The write can reallocate the head of the skb, and the caller goes on with pointers into the old one, which has been freed. Let every call of bpf_dynptr_write() mark the function, as a tail call does. The same mark decides whether a program may replace a function. Fixes: 51081a3f25c7 ("bpf: track changes_pkt_data property for global functions") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-10-alexei.starovoitov@gmail.com
2026-10-18bpf: Drop packet pointers behind kfuncs writing an skbJosef Bacik
bpf_dynptr_write() to the dynptr of an skb ends in __bpf_skb_store_bytes(), which makes the range writable first. That pulls data into the head or unclones it, and either can reallocate the head. So check_helper_call() invalidates all packet pointers and all slices of the skb behind the helper. Several kfuncs take the same road: bpf_dynptr_copy() and bpf_dynptr_memset(), if they get no writable slice of the destination because it is not linear, and bpf_probe_read_{user,kernel}[_str]_dynptr() and bpf_copy_from_user[_task][_str]_dynptr() likewise. The verifier does not know. A program can keep a packet pointer or a slice across such a call and read and write the old head, which has been freed. Treat a kfunc of that kind as bpf_dynptr_write() is treated, by the type of the dynptr it writes to, which is the first argument of all of them. Where a global function is only scanned for what it may do, the type is not known yet. There any call of these kfuncs counts, as a tail call does. Fixes: daec295a7094 ("bpf/helpers: Introduce bpf_dynptr_copy kfunc") Fixes: a498ee7576de ("bpf: Implement dynptr copy kfuncs") Fixes: 5fc5d8fded57 ("bpf: Add bpf_dynptr_memset() kfunc") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-9-alexei.starovoitov@gmail.com
2026-10-18bpf, cpumap: Hold the device of frames in the queueJosef Bacik
A frame that is redirected to another CPU waits in the queue of the cpumap entry with a plain pointer to the device it came from, and an skb from generic XDP with skb->dev. Nothing keeps that device. Closing and unregistering it waits for its NAPI and for RCU, but not for the kthread of the entry. If the kthread is late by more than that, since its CPU is busy with something of a higher priority, it builds skbs for a device that has been freed and hands them to the stack: BUG: KASAN: use-after-free in dev_gro_receive+0x1f51/0x2b30 Read of size 8 at addr ffff88800a7340b0 by task cpumap/1/map:1/54 Take a reference to the device for each entry of the queue. The kthread cannot drop it at the end of the batch, since its GRO node may keep the skb for longer. Let it note the devices and drop the references behind a full GRO flush, which it does whenever the queue is empty, and from now on also when it has noted about 2048 of them. A device whose frames are stuck in a queue is now unregistered only when the kthread has got to them. RTNL is not held meanwhile, but if the device goes with its network namespace, the cleanup of all namespaces waits. Fixes: 1c601d829ab0 ("bpf: cpumap xdp_buff to skb conversion and allocation") Fixes: 11941f8a8536 ("bpf: cpumap: Implement generic cpumap") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-8-alexei.starovoitov@gmail.com
2026-10-18bpf: Keep the symbols of a program until it has endedJosef Bacik
When the last reference to a program goes, __bpf_prog_put_noref() takes its symbols out of the kallsyms tree and frees its line info at once, and waits for the grace periods only with the rest. But the program may still be running: the link or the array it was called from is gone, the call is not over. search_bpf_extables() finds the exception table of a program by its symbol. A load that faults, and that the table is there to fix up, is fatal from then on: Oops: general protection fault, probably for non-canonical address 0xdead0000000006f8: 0000 [#1] SMP NOPTI RIP: 0010:0xffffffffc06008f6 It takes a program that is detached and closed on one CPU while it runs on another. Unwinders and bpf_prog_find_from_stack() do not know the program any more either, and the stack dump of a BPF stream reads the freed line info. Free the line info behind the grace periods the program needs, as it is done with the BTF that its text comes from. Remove the symbols in bpf_prog_free_deferred(), which runs behind them as well. The symbols are looked up under rcu_read_lock(), so the images and the program itself are freed one more grace period later. All that the program refers to is released when it was before. Fixes: 3dec541b2e63 ("bpf: Add support for BTF pointers to x86 JIT") Fixes: d7c431cafcb4 ("bpf: Add dump_stack() analogue to print to BPF stderr") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-6-alexei.starovoitov@gmail.com
2026-10-18bpf: Don't take an extension for a function that runs sleepableAlexei Starovoitov
An extension cannot be sleepable, and it is verified as a program that runs under the RCU read lock. But it runs in the context of the function that it replaces. For a sleepable program, or a function that is called from a bpf_wq or bpf_task_work callback, that is rcu_read_lock_trace(). The verifier lets such an extension use maps and pointers that rely on RCU. Its maps are not counted as maps with sleepable users either, so that what it got from them is freed behind a plain RCU grace period: WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0xe8/0xf0 BUG: KASAN: slab-use-after-free in bpf_prog_b3bad285227d855b_ext_storage+0x245/0x297 Write of size 8 at addr ffff888108717a50 by task fr/955 The verifier knows the contexts that a global function is called in. Refuse to replace one that is called without the RCU read lock, and the main function of a sleepable program. A function that sleepable code only calls under bpf_rcu_read_lock() can still be replaced. The summarization test has replaced functions of sleepable programs by ones that do not sleep. Expect that to fail. Fixes: 1e6c62a88215 ("bpf: Introduce sleepable BPF programs") Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-5-alexei.starovoitov@gmail.com
2026-10-18bpf: Keep values of a dead owner for sleepable programsJosef Bacik
bpf_local_storage_destroy() frees the elements of an owner behind a plain RCU grace period, on the ground that no program can refer to an owner that goes away. But the pointer that bpf_task_storage_get() and its like return stays valid for the verifier after the program has released the owner, or has left the bpf_rcu_read_lock() section it had the owner from. A sleepable program is in no RCU read side critical section, so it can write to the value after the owner has died and the element was freed: BUG: KASAN: slab-use-after-free in _copy_from_user+0x42/0xa0 Write of size 8 at addr ffff8880074404a8 by task lsuaf/55 Free the element behind an RCU Tasks Trace grace period if sleepable code may use the map. A program that a map is bound to was loaded, and counted in map->sleepable_refcnt, before it got a value from the live owner, and stays counted until a grace period behind its last run. A program can also get at a storage map by a lookup in an outer map. Then only the outer map is counted, its programs can come after the inner map was put into it, and can hold on to the inner map after it was taken out. So note in a map that it has been an inner map, and treat it like one with sleepable users from then on. The same goes for what the special fields of the value hold. The page of a uptr has to stay pinned, and a kptr which the program stores after the owner has died has to be released. Leave the fields, and the smap that it takes to free them, to bpf_selem_free_trace_rcu(), as bpf_selem_unlink() does. bpf_local_storage_map_free() waits for that callback, and leaves smap alone if it comes across such an element. Elements of such maps now wait for sleepable programs that are running when their owner dies. Other maps are freed as before. Fixes: dcf456c9a095 ("bpf: Fix usage of trace RCU in local storage.") Signed-off-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261017230950.802052-4-alexei.starovoitov@gmail.com