summaryrefslogtreecommitdiff
path: root/kernel
AgeCommit message (Collapse)Author
2026-10-09sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueueTejun Heo
A deferred reenqueue can read a freed user DSQ. The request stays on its CPU's deferred list until run_deferred() takes it, and the RCU callback that frees the DSQ assumes that the grace period has flushed it, so exit_dsq() checks the per-CPU node without the deferred list lock. A grace period waits for readers in flight, not for a pending run_deferred(). A run_deferred() which starts after the grace period began can detach the request right before the callback looks and then read the freed DSQ's id. Cancel the pending requests before the grace period instead. The irq work that frees the DSQ unlinks every CPU's request under the per-CPU deferred list lock, and schedule_dsq_reenq() ignores a new request under the same lock once the id is invalid. Only a run_deferred() that started before the grace period can hold a request, and it runs with IRQs disabled, so the grace period waits for it and nothing touches the DSQ after the callback. Fixes: 84b1a0ea0b7c ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs") Cc: stable@vger.kernel.org # v7.1+ Reported-and-tested-by: Hui Su <sh_def@163.com> Link: https://lore.kernel.org/all/20261008024753.4096008-1-sh_def@163.com/ Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-09sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on ↵Tejun Heo
its CPU's local DSQ With SCX_OPS_ENQ_LAST, the last runnable task on a CPU is passed to ops.enqueue() instead of being kept running. Exiting tasks without SCX_OPS_ENQ_EXITING, migration disabled tasks without SCX_OPS_ENQ_MIGRATION_DISABLED and tasks on an offline rq skip ops.enqueue() and go straight to the local DSQ. This happens in put_prev_task_scx() after the pick has settled on idle, so nothing reschedules the CPU, which idles with the task queued until an unrelated wakeup lands on it or the watchdog fires. scx_mitosis, which sets SCX_OPS_ENQ_LAST without SCX_OPS_ENQ_EXITING, hit this during mass cgroup teardown: exiting tasks stuck in exit_mmap() for over 40 seconds. A task put on the local DSQ runs, as anywhere else. If the SCX_ENQ_LAST enqueue left the task on this CPU's local DSQ, set need_resched on the task being switched in, as proxy_resched_idle() does. resched_curr() would flag the task being put instead, and __schedule() clears that right after the pick. Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class") Cc: stable@vger.kernel.org # v6.12+ Reported-by: Xiangyu Bu <xbu@meta.com> Link: https://github.com/sched-ext/scx/pull/3870 Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Dan Schatzberg <schatzberg.dan@gmail.com>
2026-10-10Merge tag 'dma-mapping-7.3-2026-10-09' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux Pull dma-mapping fixes from Marek Szyprowski: "Two more fixes for the corner cases in the DMA-mapping SWIOTLB code (Peng Fan and Marek Szyprowski)" * tag 'dma-mapping-7.3-2026-10-09' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux: swiotlb: fix default_swiotlb_limit() for non-growable default pool iommu/dma: skip swiotlb bounce for DMA_ATTR_MMIO in iommu_dma_map_phys
2026-10-09Merge tag 'vfs-7.3-rc7.fixes' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs Pull vfs fixes from Christian Brauner: "This contains fixes for the current development cycle. All of them came out of a review of the mount code that started with a bug report. The review modeled the corner cases of mount propagation, unmounting and mount reference counting and turned up a lot of bugs. Most of them years old. Most fixes come with a selftest. - Rework connected mounts. A mount that is unmounted together with its parent can stay attached to the parent to keep its mountpoint covered. That happens when the mountpoint is removed with rmdir(), unlink() or rename(), when a detached tree is dissolved, and for locked mounts in any umount that isn't synchronous, including the teardown of their mount namespace. The parent then owns the child and drops it on its own final mntput(). So any reference from the child's superblock back to one of its ancestors becomes a cycle that is never freed. A loop device backed by an image on a tmpfs and mounted on that same tmpfs is enough. Remove the directory the tmpfs is mounted on from the host, let the container's mount namespace exit, and the loop device, the tmpfs and the filesystem on the loop device are leaked for good. The same works with autofs, zram, ecryptfs, binfmt_misc, fuse passthrough, zloop, a mass storage gadget and md, and the selftests have reproducers for them. This has been possible since v4.1. It is also why "put_mnt_ns(): leave mounts connected" was reverted in -rc5. Keeping every mount of a dying mount namespace connected made these cycles trivial to create. Every unmounted mount is now detached from its parent. Where the mountpoint has to stay covered the mount leaves a cover on the parent instead, allocated together with the mount. A lookup on the unmounted parent that hits a cover finds an empty immutable directory or file on the private nullfs instance. Nothing leads from a cover to another mount, so no unmounted mount owns another one and no cycle can form. This is visible to userspace. A formerly connected mount can no longer be reached through its unmounted parent and ".." inside it leads nowhere, as for every other lazily unmounted mount. With that the private nullfs instance becomes reachable from userspace, so it now refuses mounts on top, is mounted read-only, and refuses fsnotify marks and file locks. Its inodes are shared by every holder and a watch or a lock would otherwise reach across users. may_decode_fh() now also decides its subtree check under a single mount_lock hold, as a racing umount could otherwise let it decode into what a locked child covered. - umount: * Don't silently unmount busy mounts. Since v4.13 propagate_umount() takes down propagated copies of the victim with children as long as each child is an overmount or another copy of the victim, but propagate_mount_busy() only ever checked copies without children or with just an overmount. A container that moved a tree beneath its copy of a host mount lost that tree from under its open file descriptor to a plain umount() on the host. propagate_mount_busy() now applies the same rules, walking each chain of copies once. * Don't let a migrating task hide its reference from umount(). mnt_get_count() sums the per-cpu counters under mount_lock but the mntget() and mntput() fast paths don't take it. A task that takes a reference on a cpu the sum has already passed and drops it after migrating to one the sum hasn't reached yet hides the reference it held to begin with, and umount() succeeds with the file still open. Gets and puts now live in separate per-cpu counters and all puts are summed before all gets with a full barrier in between, the way srcu_readers_active_idx_check() does it. mntget() is unchanged and mntput() gains an smp_wmb(). * Check each submount for references right before unmounting it. shrink_submounts() and mark_mounts_for_expiry() checked all their victims up front. Unmounting the first could move a busy overmount to where the next victim's propagated copy is looked up and it was then unmounted without a check. * Never expire a locked mount. A shrinkable mount moved beneath a locked mount with MOVE_MOUNT_BENEATH takes over the lock, and umount() of an unlocked ancestor expired it and revealed what it covered. That umount() now fails with EBUSY as it does for any other locked child. A lazy umount still takes the whole tree. - Overmounts and locked mounts: * Unhash a dentry before detaching the mounts on it. unlink(), rmdir() and rename() detach the mounts on the victim but only d_delete() it once its inode is unlocked, a window that includes an expedited RCU grace period. In between, a lookup from a mount namespace in which the dentry is a mountpoint found it hashed, positive and uncovered. Drop the dentry first, as d_invalidate() already does. * Don't reveal overmounted entries in refwalk. A refwalk that had grabbed the dentry before the unlink never rechecked it the way rcuwalk does with d_seq and mount_lock. Without any artificial widening three walkers read the covered file 27 times in a minute. step_into() now fails an unhashed dentry marked DCACHE_CANT_MOUNT with -ESTALE and the walk is retried. * Keep covered mounts covered in OPEN_TREE_NAMESPACE. Creating such a mount namespace only takes a user namespace and the copy followed bind mount rules: no children without AT_RECURSIVE and no unbindable mounts with it. An unprivileged user could see what mounts covered in the source, such as the parts of /proc and /sys that container runtimes mask. If the caller doesn't own the source mount namespace a non-recursive copy of a mount with something mounted below the requested directory is now refused and a recursive copy includes unbindable mounts, the way unshare() copies. * Keep the lock on a mount that a propagated copy is moved beneath. MNT_LOCKED moved to any mount that ended up beneath a locked mount, propagated copies included. A host mount and umount on a directory covered by a locked mount in a less privileged mount namespace left that cover unlocked for the namespace's owner to remove. Only mounts the caller places beneath take over the lock now. * Handle mount locking for automounts correctly. Which copies to lock was decided by the mount namespace of the task that triggered the automount. A task in a user namespace that triggered one on a host mount through a file descriptor got the host's own automount locked while its own copy stayed unlocked and could have nosuid, nodev and noexec cleared. Use the owner of the mount namespace the mount lands in. - Use-after-free and crashes: * Refuse an automount below a mount that is in no namespace. The private clones overlayfs uses for its layers have the MNT_NS_INTERNAL error pointer as their namespace, which finish_automount() let through and count_mounts() dereferenced. A fanotify filesystem mark on an overlayfs lower layer hands out file descriptors on such a clone. With debugfs as the lower layer opening "tracing" oopses with namespace_sem held for writing and every mount operation on the system blocks from then on. * Reset the old parent's ->overmount in mnt_change_mountpoint(). When propagate_umount() moved an overmount off a mount that a file descriptor kept alive, MOVE_MOUNT_BENEATH through that descriptor later followed the stale pointer into the freed overmount. * statmount() with STATMOUNT_BY_FD and pivot_root() read the parent of a mount that may be unmounted and only held by a file descriptor, while the parent's final mntput() can free it. statmount() now reads it under mount_lock and pivot_root() first checks that both mounts are in the caller's mount namespace. * Queue a mount only once for mount notifications. A mount reparented by one umount_tree() and taken down by the next under the same namespace_sem hold, as in shrink_submounts(), was queued twice. That cut the mounts queued in between out of notify_list while it still pointed at them, and once they were freed every later mount operation walked freed memory. * Don't let a pseudo dentry become the root of a mount. A bind mount of a bpf token file did that with a DCACHE_NORCU dentry, which is freed without an RCU grace period while lockless path walks may still look at it. Refuse to clone such a mount. * Don't inherit MNT_UMOUNT in clone_mnt(). A bind mount of a lazily unmounted nsfs or pidfs mount through its file descriptor started out flagged as unmounted. Among other things __detach_mounts() then dropped the namespace's reference on it, the mount outlived its namespace and mount_setattr() through the descriptor read the freed namespace. A recursive bind mount of such a mount also copied the unmounted stack still attached to it. That now fails with EINVAL, copying the mount itself still works. * Remove the fsnotify marks of a mount namespace in free_mnt_ns() instead of the RCU callback that frees the namespace, where taking the group mutexes meant sleeping in softirq context. - Propagation and copies: * Keep a copied mount unbindable. Since v6.17 clone_mnt() didn't copy the unbindable flag, so every mount namespace created with CLONE_NEWNS had bindable copies of all unbindable mounts. This had been fixed once before. * Refuse MOVE_MOUNT_SET_GROUP on an unbindable mount. It made the mount an unbindable slave, a state nothing else can produce, or silently dropped the unbindable flag. CRIU applies MS_UNBINDABLE after restoring sharing and isn't affected. * Check a recursive bind mount for mount namespace loops. Recursively bind mounting a tree from another mount namespace could put a mount of a namespace's file inside that same namespace, which then pins itself and all its mounts. Repeating it leaks without limit, the reproducer took Shmem from 380 kB to 65916 kB. The copy is now checked with check_for_nsfs_mounts() before it is grafted, as move_mount() does. * Look at the topmost mount for a mount namespace file. attach_recursive_mnt() never looked at the topmost mount of the source's chain of overmounts. If that was the chain's only mount namespace file an existing mount at a propagated destination got buried below the root of the nsfs file where no path walk reaches it. * Don't put a mountpoint on a dentry that's being removed. attach_recursive_mnt() makes a mountpoint of the source's root without its inode lock, so a racing rmdir() of that directory could leave a mount on it that nothing ever detaches. d_set_mounted() now checks cant_mount() as well. - nullfs: * Take no inode lock for readdir of an immutable directory. The root of every empty mount namespace is the same nullfs directory and iterate_dir() held its i_rwsem across ->iterate_shared(). A reader whose buffer faults on a FUSE mount of its own holds it for as long as its server wants, and with an exclusive locker queued behind it every lookup that misses the dcache, every create and every mount in that directory waits. One user of an empty mount namespace stalls all others. Directories with the new FOP_IMMUTABLE flag skip the lock. * Refuse to reconfigure internal superblocks through fspick(), MS_REMOUNT or the read-only remount that a synchronous umount() of the root does. For nullfs only root in the initial user namespace could do it, but the superblock is shared by every mount namespace and the flags showed up in statfs() for all of them. * Don't update the access time on nullfs and refuse F_SET_RW_HINT on an immutable inode. - unshare: Free an nsproxy that was never installed with nsproxy_free() when set_cred_ucounts() fails. put_nsproxy() dropped active references that were never taken, which triggered a warning and hid the caller's own namespaces from listns(). - Smaller changes: mount_setattr() checks the target before it walks the tree to allocate peer group ids, unshare() puts the old fs_struct before the old namespaces, dissolve_on_fput() drops the file's reference to the tree itself, disconnect_mount() is simplified and the documentation of the propagated unmount rule is brought up to date" * tag 'vfs-7.3-rc7.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (58 commits) namespace: simplify disconnect_mount() selftests/filesystems: test covered mounts namespace: rework connected mounts nullfs: add an empty immutable regular file selftests/filesystems: check that reading the root of an empty mount namespace stalls nobody selftests/filesystems: add a helper that holds a readdir in a page fault readdir: take no inode lock on an immutable directory nullfs: refuse file locks fsnotify: let a filesystem refuse marks on its objects namespace: nothing is mounted on or written through knullfs namespace: keep the private nullfs instance in knullfs fhandle: decide the subtree check under mount_lock selftests/filesystems: check that an automount below an overlay layer is refused selftests/filesystems: check the atime of the empty mount namespace root selftests/filesystems: check that a lock lands on the right mount and stays namespace: keep the lock on a mount that a propagated copy is moved beneath namespace: never expire a locked mount nullfs: don't update the access time namespace: handle mount locking for automounts correctly namespace: refuse an automount below a mount that is in no namespace ...
2026-10-09sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective ↵Tejun Heo
cap changes A grant or revoke only records the target caps. They take effect on a cid at its next dispatch, and nothing tells the parent when. Without that the parent cannot act on a revoke: a child that ran a cid at a low cpuperf target leaves the target there when PERF is revoked, as the kernel resets targets only at root enable. Nor can the parent tell when it may schedule on the cid again. Add ops.sub_child_ecaps_updated(), delivered to the direct parent right after the child's own ops.sub_ecaps_updated() with the child's cgroup id and the same before and after caps, in the same dispatch context. Both deliveries are suppressed while the child is bypassing and replayed together afterwards. A disabled child reports nothing: ops.sub_detach() is where the parent restores what it had delegated. A sub is now bypassed before it is linked, so that the grants queued for it from ops.sub_attach() are consumed while it is bypassed and replayed to it and the parent once it is enabled. Otherwise a sync consumed before the sub's ops are registered would reach the parent but never the sub. v2: Drop the disable report with its sleep lock and nested-dispatch gate, the parent restores from ops.sub_detach() instead. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-09bpf: Bpf_func_state size optimizationEduard Zingerman
Pack bpf_reg_state to bring it's size from 88 bytes to 80. Consequently, bpf_func_state changes from 1048 to 960. Thus bpf_func_state moves from 2K to 1K allocator bucket. 1K bucket is the one it fit before the SCEV series. With this optimization SCEV increases memory consumption on a subset of Meta internal programs by 3%, without this optimization the increase is 48%. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-36-bcba9bc53ac9@gmail.com
2026-10-09bpf: Avoid widening registers that hinder exact stack-slot trackingEduard Zingerman
For loops like this: int arr[10]; for (int i = 0; i < 10; i++) arr[i] = i; use(arr[i]); SCEV-based loop widening converts `i` from being enumerated 10 times as a constant value 0, 1, ..., 9 to being enumerated once as a range [0..9]. From the point of view of the main verification pass, this replaces constant-offset stack access with varying-offset stack access. The verifier tracks stack values accessed through varying offsets much less precisely than those accessed via constant offsets. For example, the read at use(arr[i]) would yield STACK_MISC. Precision might be lost for the following operations: - spills (BPF_ST/STX to the stack): stored value can no longer be recovered when read back, e.g. after the loop; - fills (BPF_LDX from the stack): a varying-offset read of a spilled pointer yields a SCALAR_VALUE, so the pointer is lost. - calls that construct objects on the stack (dynptr, iter, irq_flag, res_spin_lock): the stack argument is required to have a constant offset. This commit adds logic to avoid widening registers used by such instructions: - During the main SCEV computation stage, collect_store_base_regs(): - inspects instructions accessing the stack (spill, fill, call to a function listed in bpf_needs_fixed_stack_off), and - if there is a SCEV expression for a stack base address register, records which registers the expression depends on in the bpf_loop->store_base_reg field. - `store_base_reg`s accumulated for nested loops are propagated to outer loops as well. - During the loop-widening stage, bpf_compute_loop_iters() refuses to widen a loop when some of the registers the loop changes appear in store_base_regs. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-35-bcba9bc53ac9@gmail.com
2026-10-09bpf: Use SCEV to widen bounded loopsEduard Zingerman
Loop-bound computation and register widening =========================================== During the main verification pass, use SCEV expressions and register values at the entry to the loop to estimate the number of iterations the loop might execute. Use this number to widen loop induction variables. For example: 0: r7 = 5; 1: r6 = 0; 2: while (r6 < 3) { // latch 3: if (r6 == 1) 4: r7 = 10; 5: r6 += 1; 6: } 8: use(r7); - Scalar evolution expressions (SCEVs) at the latch are: - r6 = (linear r6 1) - r7 = (any r7 10) - For this pre-condition loop, this corresponds to the linear inequality `r6_entry + n < 3`. With r6_entry = 0, the inequality is `n < 3`. - Hence infer that r6's range at line 2 is [0,3], including the final failed test. The body starts with r6 in [0,2]. - r7's (any r7 10) expression means that its initial value 5 from line 0 is joined with 10 from line 4: it is widened to [5,10] within and after the loop. Currently such inference is supported for: - Reducible loops with one backedge and complete backedge/exit lists. - A linear latch dominating the backedge, with one successor exiting the loop. Additional exits are allowed; they can shorten execution. - 64-bit arithmetic and comparisons whose continuation condition normalizes to signed/unsigned < or <=, or !=. - Initial counter, latch offset, slope and bound evaluable as constants at loop entry, yielding a finite, representable iteration bound. - Values with an eligible type at loop entry, e.g. SCALAR_VALUE or PTR_TO_STACK, but not PTR_TO_CTX. In order for widening to proceed, every register live at the loop entry has to have a SCEV that can be widened. Currently, the following expressions are supported: - (linear <self> <step>): entry value + iteration# * constant step. - (reg <self>): loop invariant; keep its entry value unchanged. - (any ...): join the entry value with immediates or invariant registers; types must match, and immediates require SCALAR_VALUE. Main verification loop integration ================================== Before proceeding with the usual instruction processing, do_check() takes the following steps at loop entries (H): - Checks whether the loop can be widened and obtains its iteration bound (scev.c:bpf_compute_loop_iters()). - If it can, saves an unwidened checkpoint E0, which will be used as a base for deriving widened values throughout this invocation of the loop. - Widens registers in the current state using E0 and SCEV expressions computed for H (scev.c:bpf_widen_scev_regs()). - Pushes a per-frame loop-stack record to the current verifier state. For terminating loops, the record stores E0 and the iteration bounds. - Lets is_state_visited() create a new checkpoint W at the same header, now containing the widened values, and proceeds with normal checking. - On an exit edge, pops the loop record and verifies the exit path normally. is_state_visited() treats a loop proven to terminate like an iterator-based loop: RANGE_WITHIN comparison attempts to establish convergence against a checkpoint whose exploration is still in progress. For example: r0 = 0; H: r0 += 1; if (r0 != 3) goto H; exit; first H: prove H_count = 3; save E0(r0 = 0) widen r0 to [0,2]; push the loop record save W(r0 = [0,2]) body: r0 becomes [1,3]; fork at the conditional exit: r0 = 3; pop the loop record and check exit backedge: r0 = [1,2]; clamp, then match with W and prune Clamping -------- Independent register ranges lose correlations between induction variables. For example: r6 = 0; r7 = 0; do { r6 += 1; r7 += 2; } while (r6 < 3); use(r7); header: r6 in [0,2], r7 in {0,2,4} body: r6 in [1,3], r7 in {2,4,6} backedge: r6 in [1,2], r7 still in {2,4,6} The latch narrows r6 but does not propagate this refinement to r7. To account for this, before comparing states on a backedge, scev.c:bpf_clamp_scev_regs() recomputes r7's range from E0 and the saved iteration bound, then intersects the current range {2,4,6} with the recomputed {0,2,4}, arriving at {2,4}. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-34-bcba9bc53ac9@gmail.com
2026-10-09bpf: Compute scalar evolution expressions for loopsEduard Zingerman
Assign algebraic expressions to loop variables (registers and stack spills), describing how their values evolve across iterations. These summaries provide symbolic input for the subsequent widening patch. The algorithm draws on ideas from the following papers: - "Symbolic Evaluation of Chains of Recurrences for Loop Optimization" Robert A. van Engelen, 2000 - "The CR# Algebra and its Application in Loop Analysis and Optimization" Robert A. van Engelen, 2004 The analysis proceeds in two phases: - Compute expressions for one symbolic iteration: start each variable with a reference to its input value, apply instruction effects and join expressions where control-flow paths merge. - Convert the resulting backedge updates to recurrences at the header, then substitute these recurrences into expressions at the latch. Analyze loops innermost first. Within each loop, visit blocks in topological order (reverse postorder, with backedges ignored and nested loops collapsed). Treat each nested loop as an opaque operation: preserve its invariant values, forget the others and continue at its exits, without expanding its iterations. For example, using 64-bit arithmetic: 0: r7 = 5; 1: r6 = 0; 2: do { // header 3: if (r6 == 1) 4: r7 = 10; 5: r6 += 1; 6: } while (r6 < 3); // latch Expressions use symbolic inputs r6 and r7; the concrete values from lines 0-1 are supplied later by the main verifier. First phase: ------------ At each instruction, an environment maps register numbers to their current expressions. Transfer functions update this mapping, while joins merge the environments arriving along different paths. For illustration, follow each branch separately to the backedge: - At 3, take the path skipping 4: r6 = r6, r7 = r7. - At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1). - At 2, the first path contributes r6 = (+ r6 1), r7 = r7. - At 4, on the other path from 3, transfer(r7 = 10): r7 = r7 -> r7 = 10. - At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1). - At 2, join the two paths' contributions: join((+ r6 1), (+ r6 1)) = (+ r6 1) for r6; join(r7, 10) = (any r7 10) for r7. Equal expressions stay unchanged; different expressions become ANY alternatives. These summarize one iteration, without unrolling the loop. Second phase: ------------- The update r6 = (+ r6 1) says that each iteration adds 1 to r6. Repeating this update n times gives r6_entry + n, a linear recurrence. - At header 2, r6 = (+ r6 1) becomes (linear r6 1), meaning r6_entry + n, where n counts iterations from zero. - At latch 6, substitute this recurrence into (+ r6 1): (+ (linear r6 1) 1) simplifies to (linear (+ r6 1) 1), meaning r6_entry + 1 + n. - At both points, r7 retains (any r7 10): either its entry value or 10. ANY records alternatives, not numeric ranges or the path conditions selecting them. Unchanged variables retain their inputs. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-33-bcba9bc53ac9@gmail.com
2026-10-09bpf: Allow precision backtracking between overlapping checkpointsEduard Zingerman
SCEV integration logic might create two consecutive checkpoints at the same instruction w/o processing any instructions in between. This happens for loop-entry checkpoint -> regular checkpoint sequences, where the loop-entry checkpoint has to remain unchanged but the regular checkpoint is used for widening. This commit adapts bpf_mark_chain_precision() to support this. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-32-bcba9bc53ac9@gmail.com
2026-10-09bpf: Record basic-block ends in insn_aux_dataEduard Zingerman
SCEV needs information about the program's basic-block structure. Piggyback on compute_predecessors() and set insn_aux_data->bb_end for an instruction if: - it has multiple successors, or - its sole successor has multiple predecessors, or - it is a jump instruction. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-31-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_spill_size()Eduard Zingerman
A utility function counting the number of STACK_SPILL slots in a bpf_stat_state instance. For use by SCEV widening logic. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-30-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_same_memory_origin()Eduard Zingerman
When dealing with loops like: for (ptr = ...; ptr < end_ptr; ptr += 8) ... Scalar evolution needs to know if ptr and end_ptr identify a same memory location (possibly with different offsets). If they do, it is sound to attempt to infer iteration bounds for such a loop. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-29-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_split_cur_state()Eduard Zingerman
SCEV widening needs an explicit checkpoint preserving the original loop-entry state, independently of ordinary state-cache heuristics. This commit extracts checkpoint creation logic from bpf_is_state_visited() into a separate utility function. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-28-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_reg_union() and bpf_reg_union_compatible()Eduard Zingerman
Utility functions to take a union of two registers' scalar values: merge the circular 32-bit and 64-bit bounds, tnums and base/step of two registers, then synchronize and validate the result. Leave id invalidation and type validation to the caller. Caller should use bpf_reg_union_compatible() to ensure that final union result is sensible. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-27-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_mark_reg_known_scalar()Eduard Zingerman
Provide a helper to set a register to a known scalar value for use outside verifier.c. To be used by SCEV widening logic. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-26-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add bpf_set_reg_range()Eduard Zingerman
A utility function for setting a register's range and step information. Used by SCEV widening and clamping logic further in the series. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-25-bcba9bc53ac9@gmail.com
2026-10-09bpf: Compute loop hierarchyEduard Zingerman
SCEV needs to process loops from innermost to outermost, summarizing results for nested loops. Add a pass to compute the loop structure. Use a non-recursive adaptation of the algorithm from: "A New Algorithm for Identifying Loops in Decompilation" by Wei et al. Record the loop hierarchy per instruction: - The field insn_aux_data->loop_header records the innermost loop header containing it. - The field insn_aux_data->loop records additional information about the loop if this instruction is a loop header (exits, backedges, irreducibility). Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-24-bcba9bc53ac9@gmail.com
2026-10-09bpf: Compute immediate dominatorsEduard Zingerman
SCEV needs to find an exit condition that dominates a loop's backedge; such conditions bound every loop iteration. Compute the immediate dominator tree as a prerequisite for identifying such conditions. Use the iterative reverse-postorder algorithm from: "A Simple, Fast Dominance Algorithm" by Cooper et al. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-23-bcba9bc53ac9@gmail.com
2026-10-09bpf: Move the live-register and SCC printout to a standalone functionEduard Zingerman
Tests for multiple analyses performed by the verifier need the verifier log to contain analysis results alongside the program disassembly. This patch moves the log-level-2 program dump from bpf_compute_live_registers() to a standalone function called from bpf_check(), in order to provide a common logging function for such analyses (and thus avoid printing program disassembly multiple times). Rename the dump header from "Live regs before insn:" to "Program dump (scc? insn#: live_regs_before):" and print available source-line information before the corresponding instructions. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-22-bcba9bc53ac9@gmail.com
2026-10-09bpf: Save DFS postorder numbers for program instructionsEduard Zingerman
Dominator intersections and SCEV worklist scheduling need to compare instructions by their DFS postorder rank. Save per-instruction postorder numbers alongside the existing postorder sequence. This commit modifies bpf_compute_postorder() to actually do the DFS traversal, instead of scheduling traversal of all siblings at once. Consider a graph: A -> B A -> C C -> B Old result: C, B, A DFS result: B, C, A Fixes: efcda22aa541 ("bpf: compute instructions postorder per subprogram") Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-21-bcba9bc53ac9@gmail.com
2026-10-09bpf: Varying offset access support for PTR_TO_BTF_ID pointersEduard Zingerman
Relax the requirements for reads like `p->arr[i].field`, where `p` is a PTR_TO_BTF_ID, by allowing `i` to be a non-constant value. This is useful for loops over arrays accessed from a PTR_TO_BTF_ID when the verifier widens the loop induction variable. Check that the access stays within array bounds based on the access register's min and max bounds. Check that the access is aligned with array elements using the register's `base` and `step`. Technically, modify btf_struct_access() such that: - it identifies `field` by walking BTF from the minimum offset (the register's reg_smin + static instruction offset); - it remembers which arrays contain `field`; - it checks that offsets `k * step` stay within the array and point to the same `field`; - it does not check upper bounds for flexible arrays (as is already the case for constant-offset access). Support for flexible arrays nested inside trailing struct members (e.g. p->nested.data[i], with data declared as int data[]) is left for future work. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-20-bcba9bc53ac9@gmail.com
2026-10-09bpf: Representation for intervals with stepsEduard Zingerman
Extend scalar register tracking with a linear "base + step * k" description: each scalar carries two u16 fields, base and step (invariant: base < step), recording that its value is some point on the line base + step * k. A fresh scalar starts as base=0, step=1, i.e. any integer. This lets the verifier reason about strided values, e.g. an index multiplied by an element size, which neither the min/max bounds nor the tnum can capture for a non-power-of-2 stride. Such reasoning is necessary to allow array access with a variable offset through PTR_TO_BTF_ID, e.g. for expressions like p->arr[i].field. reg_bounds_sync() uses the description in two ways: - __reg_bound_offset() clears the low bits of var_off implied by the number of trailing zeros shared by base and step. - __update_bounds_from_step() intersects the 64-bit range with the line via cnum64_intersect_linear(), snapping smin/smax to actual points on it. This is the last step of the sync, as refining the range from var_off can move the bounds off the line. range_within() is extended to check whether the cur register's equation describes a subset of the points described by the old register's equation. Scalar ALU operations update base/step as follows: - ADD of a constant (including pointer + scalar): keep step, shift base; - MUL by a constant: scale step and base by the constant; - LSH by a constant: scale step and base by the corresponding power of two; - sign extension: keep the register unchanged if all values fit the target type, reset to base=0, step=1 otherwise; - All other operations reset to base=0, step=1. Branch (eq/neq) checks do not take into account base/step yet. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-19-bcba9bc53ac9@gmail.com
2026-10-09bpf: Allow subrange relations for PTR_TO_STACK in regsafe()Eduard Zingerman
regsafe() currently requires an exact match for PTR_TO_STACK registers. This prevents an explored state from pruning a current state whose variable-offset range is a subset of the explored range. This change allows a checkpoint containing a widened stack pointer to cover narrower offsets reached on subsequent loop iterations, for example [-56, -8] within [-64, -8]. Merge the PTR_TO_STACK case with PTR_TO_MEM and others, as the set of necessary checks is almost identical. The only difference is a map_uid check, which is not needed for PTR_TO_STACK. All locations that produce PTR_TO_STACK reset map_uid to zero. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-18-bcba9bc53ac9@gmail.com
2026-10-09bpf: Expose bpf_verbose_snum()Eduard Zingerman
For use in SCEV logging to make large widened bounds more readable. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-17-bcba9bc53ac9@gmail.com
2026-10-09bpf: Expose bpf_grow_stack_state()Eduard Zingerman
When windening SCEV expressions like *fp-8 -> (any *fp-8 *fp-16), fp-8 might be not yet allocated. Expose a utility function such that widening logic can handle this. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-16-bcba9bc53ac9@gmail.com
2026-10-09bpf: Expose bpf_rev_opcode() and bpf_flip_opcode()Eduard Zingerman
Expose the existing helpers verifier.c:rev_opcode() and verifier.c:flip_opcode() as bpf_rev_opcode() and bpf_flip_opcode() and declare them in bpf_verifier.h. The helpers are used by SCEV logic while analyzing loop conditions. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-15-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add cnum64_intersect_linear()Eduard Zingerman
Add cnum64_intersect_linear(), which intersects a cnum64 interval with the set of integers congruent to 'base' modulo 'step', i.e. the points base + step * k. It tightens the interval by rounding its signed minimum up and its signed maximum down to the nearest such point, and returns an empty cnum when no point falls within the interval. The arithmetic is done on the signed bounds [smin, smax]. The line is defined over the integers, while a cnum64 is an arc on the u64 circle, so a u64-modular computation would misplace the line for negative values whenever 'step' does not divide 2^64. Working from the signed bounds keeps the residues consistent, e.g. it can reason about a range like [-9, 3] with step 3. Also add imod() to bpf_verifier.h: a mathematical modulo returning an always-non-negative residue (unlike C's '%', which follows the sign of the dividend). It is used by cnum64_intersect_linear() and by the base/step tracking added in subsequent patches. This is a building block for tracking scalar registers whose value is known to lie on a line base + step * k. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-14-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add cnum{32,64}_union()Eduard Zingerman
Add cnum32_union() and cnum64_union() to compute a smallest circular range containing both inputs. Do so by enumerating the following configurations in a rotated frame (a.base at the origin): 0 UT_MAX |---------------------------------------------------| [= a ==============================] | [= b tail =] [= b main ===================> [= union ===========================================] 0 UT_MAX |---------------------------------------------------| [= a =====================] | [= b tail ======] [= b main =====> [= union tail ============] [= union main => 0 UT_MAX |---------------------------------------------------| [= a =======] | [= b tail =============] [= b main =====> [= union tail =========] [= union main => 0 UT_MAX |---------------------------------------------------| [= a =========================] | | [= b ====================] | [= union ==================================] | 0 UT_MAX |---------------------------------------------------| [= a =====================================] | | [= b ========] | [= union =================================] | 0 UT_MAX |---------------------------------------------------| [= a ============] | | [= b =======] | Two possible covering arcs: [= ab ======================================] | [= ba tail ======] [= ba main =========> Pick the smaller one. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-13-bcba9bc53ac9@gmail.com
2026-10-09bpf: Add tnum_alignment()Eduard Zingerman
Used by the SCEV widening logic later in the series. The verifier checks alignment for some memory accesses. When SCEV widens a pointer or an index variable, it must retain alignment information in order to allow array access. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-12-bcba9bc53ac9@gmail.com
2026-10-09bpf: Compute parameters liveness in bpf_compute_live_registers()Eduard Zingerman
This computes exact liveness for the arguments of a call to a subprogram instead of assuming that r1..r5 are read: - For static subprograms compute use registers at the bpf_compute_live_registers() fixpoint phase. - For global subprograms use BTF information. Add a separate pass doing try_btf_prepare_func_args() in order for bpf_subprog_info->arg_slot_cnt to be available during bpf_compute_live_registers(). The conservative liveness marks hindered SCEV widening logic on a set of production programs. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-11-bcba9bc53ac9@gmail.com
2026-10-09bpf: Derive global subprog stack liveness from BTF signaturesEduard Zingerman
Stack liveness currently analyzes global callees in the context of their callers. The resulting read/write marks describe only the current body, but freplace can replace it after the caller has been verified. A no-op callee taking a stack pointer can be replaced with one writing through that pointer, leaving the call's may_write marks incomplete. SCEV could then preserve a stack counter recurrence across a call that resets it, invalidating a loop termination proof. Instead, derive global call effects from BTF argument types instead, same is it done on the main verification pass. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-10-bcba9bc53ac9@gmail.com
2026-10-09bpf: Fix arg_track_xfer() for atomic stack spills and fillsEduard Zingerman
Track spilled stack pointers through 64-bit atomic accesses: - BPF_STORE_REL spills the source register. - BPF_LOAD_ACQ restores the spilled value into dst_reg. - BPF_XCHG and BPF_{ADD,AND,OR,XOR} | BPF_FETCH restore it into src_reg. - BPF_CMPXCHG restores it into R0. This prevents missing stack accesses through pointers recovered by these instructions, matching the main verifier's spill/fill behavior. Fixes: bf0c571f7feb ("bpf: introduce forward arg-tracking dataflow analysis") Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-9-bcba9bc53ac9@gmail.com
2026-10-09bpf: Record write effects for atomic operations in liveness.cEduard Zingerman
Atomic read-modify-write instructions are recorded as reads only, omitting their stack write effects from the may_write masks needed by SCEV. Record both read and write accesses for atomic RMW operations. Keep LOAD_ACQ read-only and STORE_REL write-only. Share the precise/imprecise stack-access dispatch so both accesses use the same address handling. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-8-bcba9bc53ac9@gmail.com
2026-10-09bpf: Summarize regs that may hold a frame pointer in insn_aux_dataEduard Zingerman
SCEV needs to know which registers might be stack pointers at a particular instruction. This information is used to avoid widening registers if that would hinder further verification. See the patch [1] for additional information. liveness.c:compute_subprog_args() already tracks this information. This commit modifies it to save the information in insn_aux_data for further usage. [1] "bpf: avoid widening registers that hinder exact stack-slot tracking" Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-7-bcba9bc53ac9@gmail.com
2026-10-09bpf: Summarize live stack slots in insn_aux_dataEduard Zingerman
SCEV needs live stack information in order to skip tracking dead slots. Liveness tracks this per function instance, but SCEV is not callchain-sensitive and needs a summary across calling contexts. For each instruction, union the current-frame liveness masks from all analyzed function instances. Represent 8-byte slots as a summary of constituent halves. Store the summary in insn_aux_data. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-6-bcba9bc53ac9@gmail.com
2026-10-09bpf: Summarize may write stack slots in insn_aux_dataEduard Zingerman
SCEV needs may-write information to invalidate stack expressions after indirect writes. Liveness tracks this per function instance, but SCEV is not callchain-sensitive and needs a summary across calling contexts. For each instruction, union the current-frame may_write masks from all analyzed instances. Represent 8-byte slots as a summary of constituent halves. Store the summary in insn_aux_data and expose it via a helper. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-5-bcba9bc53ac9@gmail.com
2026-10-09bpf: Track may_write flags in livenessEduard Zingerman
Track stack slots that instructions may modify, in order to account for such writes when constructing SCEV expressions. Consume the may_write flag in arg_access_info separately from reads and definite writes. Include partially covered slots and all candidate offsets; imprecise writes mark whole candidate frames. Possible writes do not kill liveness. Merge analyzed instances as: may_write(dst) |= may_write(src) must_write(dst) &= must_write(src) For each ancestor frame f, summarize callee writes at the callsite: may_write(callsite, f) |= OR_{i in callee} may_write(i, f) Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-4-bcba9bc53ac9@gmail.com
2026-10-09bpf: Describe kfunc stack accesses with arg_access_infoEduard Zingerman
Return kfunc stack effects through arg_access_info while preserving the existing BTF size lookup and read/def classification. Retain __sz/__szk handling and conservatively mark pointer arguments as possible writes. Remove the temporary signed-size adapter from liveness. Add liveness assertions for privileged and unprivileged __uninit outputs. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-3-bcba9bc53ac9@gmail.com
2026-10-09bpf: Describe helper stack accesses with arg_access_infoEduard Zingerman
Adjust bpf_helper_stack_access_bytes() to return results as a struct arg_access_info. The result is based on bpf_func_proto->argX_type value as in the following table: | Argument flags | allow_uninit_stack | Meaning | |----------------------+--------------------+----------------------| | MEM_WRITE | true | may_read, must_write | | MEM_WRITE+MEM_UNINIT | true | must_write | | MEM_WRITE | false | may_read, must_write | | MEM_WRITE+MEM_UNINIT | false | may_read, may_write | | 0 | true/false | may_read | MEM_WRITE now produces must_write when allow_uninit_stack is true: the verifier discards previous stack knowledge over the affected range. Previously, liveness only treated MEM_UNINIT this way. Also use resolve_map_arg_type() to account for bpf_map_peek_elem() treating its buffer as an input for bloom filters. Annotate ringbuf dynptr submit and discard arguments with MEM_WRITE. Their existing OBJ_RELEASE handling already invalidates the dynptr, but the prototypes need to describe the write as well as the release. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-2-bcba9bc53ac9@gmail.com
2026-10-09bpf: Represent stack access effects with arg_access_infoEduard Zingerman
The following patches in the series add support for may_write property inference. To properly track this property there is a need to distinguish three ways the helpers/kfuncs affect stack: - read; - write and destroy all previous stack tracking information for affected slots; - both read and write the stack slots, potentially preserving some of the tracking information. The later property can't be encoded by signed access_byte convention. Hence, introduce arg_access_info with a size and separate may_read, may_write and must_write bits. Convert the liveness access recording paths to use it. Proper may_write calculation and support will be added in the following patches. Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-1-bcba9bc53ac9@gmail.com
2026-10-09kho: rename kho_scratch= commandline parameter to kho_bootmem=Pratyush Yadav (Google)
KHO now no longer uses "scratch" to describe the memory areas passed by previous kernel. Instead, it calls them boot memory, or bootmem in short. Rename the kho_scratch= commandline parameter to reflect this terminology change. Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org> Link: https://patch.msgid.link/20260922041321.233986-5-pratyush@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
2026-10-09kho: rename KHO scratch to KHO bootmemPratyush Yadav (Google)
The term "KHO scratch" is vague and overloaded. For one, it does not accurately describe what the memory is for. From KHO's perspective, it is memory passed by the previous kernel that is guaranteed to have no allocations and hence it is safe to allocate from. For another, the term is overloaded. KHO also knows to discover other ranges of memory that don't have any preservations and are safe to allocate from. This is done by kho_extend_scratch(). These areas, while called "scratch", are completely different from the chunk of memory passed by the previous kernel for early boot allocations. Rename "KHO scratch" to "KHO boot memory", or "KHO bootmem" in short. Update all function names, variable names, comments, and documentation to use this. Call the discovered areas "noprsrv", matching what memblock would see them as. Rename kho_extend_scratch() to reflect this. This patch largely has no functional changes. The only functional changes are renames of debugfs files from "scratch_phys" and "scratch_len" to "bootmem_phys" and "bootmem_len" respectively. Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org> Link: https://patch.msgid.link/20260922041321.233986-4-pratyush@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
2026-10-09memblock: rename KHO_SCRATCH to KHO_NOPRSRVPratyush Yadav (Google)
The term "KHO scratch" is vague. It does not accurately describe what the memory is for. Generally, scratch is for things that don't contain anything useful. This is not true. For memblock, KHO_SCRATCH is the _only_ useful memory during a KHO boot. It is memory passed by the previous kernel that is guaranteed to have no preserved pages, and so is the only thing safe to allocate from early in boot. In addition, KHO now has the ability to discover other "scratch" areas. These aren't directly passed by the previous kernel. Instead, they are found by searching the KHO preserved memory map. Memblock does not care about the difference between the two. From memblock's perspective, both are memory areas with no preservations and so they are safe to allocate from before KHO fully initializes. Rename MEMBLOCK_KHO_SCRATCH to MEMBLOCK_KHO_NOPRSRV. This name clearly shows what the memory is for from memblock's perspective. Update memblock functions and comments that refer to scratch to use noprsrv. There are no functional changes apart from the flag name change in debugfs. Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org> Link: https://patch.msgid.link/20260922041321.233986-3-pratyush@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
2026-10-09memblock: get rid of CONFIG_MEMBLOCK_KHO_SCRATCHPratyush Yadav (Google)
This config is only a proxy for CONFIG_KEXEC_HANDOVER. It is only enabled by KHO. No point in the indirection. Use CONFIG_KEXEC_HANDOVER directly. Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org> Link: https://patch.msgid.link/20260922041321.233986-2-pratyush@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
2026-10-09dma: swiotlb: Track whether the pool size was explicitly setAneesh Kumar K.V (Arm)
swiotlb_adjust_size() must not override a pool size supplied through the swiotlb= command-line parameter. Currently, it determines whether a size was supplied by comparing default_nslabs with IO_TLB_DEFAULT_SIZE. This fails when the user explicitly requests the compiled-in default size, allowing the request to be overwritten by automatic pool sizing. Record whether swiotlb= supplied a numeric size in swiotlb_cmdline_size_set and check that flag directly before adjusting the pool size. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org> Link: https://lore.kernel.org/r/20260924060756.1325156-4-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
2026-10-09dma: swiotlb: Consolidate slab roundingAneesh Kumar K.V (Arm)
Pass the area count explicitly to swiotlb_calc_nslabs() and keep the calculation separate from updates to the default_nslabs . Move the default_nslabs update into swiotlb_adjust_nareas(). Round slabs to a power of two even before the area count is initialized. For early swiotlb_adjust_size() calls, this moves rounding from area initialization to size adjustment, so early queries see the rounded size. Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org> Link: https://lore.kernel.org/r/20260924060756.1325156-3-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
2026-10-09dma: swiotlb: Rename swiotlb_size_or_default()Aneesh Kumar K.V (Arm)
Rename swiotlb_size_or_default() to swiotlb_default_pool_size() to make clear that it returns the current size of the default SWIOTLB pool. This includes changes made by command-line options or sizing heuristics, not just the compiled-in default. No functional change. Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org> Link: https://lore.kernel.org/r/20260924060756.1325156-2-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
2026-10-09dma: swiotlb: use KiB in restricted pool allocation logFrancesco Valla
Update the allocation logs for restricted memory pools to use KiBs instead of MiBs, to avoid truncation when the size of the memory area is not a multiple of a MiB. Without this modification, pools smaller than 1MiB are logged as 0MiB. The same modification has already been performed for coherent memory pools [1]. Link: https://lore.kernel.org/all/20260629223759.2637162-1-sharaienko@google.com/ [1] Signed-off-by: Francesco Valla <francesco@valla.it> Link: https://lore.kernel.org/r/20261008-swiotlb-size-print-v1-1-374493b5c5a6@valla.it Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
2026-10-09dma-direct: Restore arch_dma_alloc() for the DMA_ATTR_NO_KERNEL_MAPPING caseMarek Szyprowski
Commit 849facea92fa ("dma-direct: simplify the DMA_ATTR_NO_KERNEL_MAPPING handling") moved the DMA_ATTR_NO_KERNEL_MAPPING shortcut in dma_direct_alloc() in front of the arch_dma_alloc() call. Architectures providing their own DMA coherent allocator (CONFIG_ARCH_HAS_DMA_ALLOC) therefore no longer see such requests. ARM 32bit is one of such. Lack of the call to arch_dma_alloc() results in the lack of proper cache management on HighMem pages. Take the dma-direct shortcut only for coherent devices or when the architecture doesn't provide arch_dma_alloc(). While touching this, move that check into separate function to avoid duplication in alloc and free paths. Fixes: 849facea92fa ("dma-direct: simplify the DMA_ATTR_NO_KERNEL_MAPPING handling") Reported-by: Aneesh Kumar K.V <aneesh.kumar@kernel.org> Closes: https://lore.kernel.org/all/yq5ams2akznl.fsf@kernel.org/ Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Link: https://lore.kernel.org/r/20260930135734.1013905-1-m.szyprowski@samsung.com Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>