Refresh the tree with run of the existing Coccinelle rules in scripts/coccinelle/api/kmalloc_objs.cocci, now that they also convert arrays of multi-byte integers. The conversions break down as: 435 kzalloc_objs() 169 kmalloc_objs() 52 kvzalloc_objs() 47 kzalloc_obj() 42 kvmalloc_objs() 8 kmalloc_obj() 2 kzalloc_flex() 2 kvmalloc_flex() and 37 existing kmalloc_obj() family calls drop a redundant GFP_KERNEL. Build tested with allmodconfig on x86_64 with GCC 16.2.0, and, together with the devm_kmalloc_obj() conversion that follows, on x86_64, arm64, i386, arm, riscv, s390, m68k, mips, parisc, sparc, loongarch, arc, xtensa, and openrisc with GCC 16.2.0 with no new warnings. Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Outgoing stack arguments retain scalar verifier state separately from
registers and ordinary spills. State comparison uses their precision
marks, but propagate_precision() does not carry those dependencies from
a reused state to the current path's ancestors. A path can consequently
reuse a state without making earlier checkpoints distinguish scalar
values that will later be used as memory offsets.
Seed the existing backtracking batch with precise scalar argument slots
from every frame. Incoming arguments live in the caller's outgoing
slots, so include caller frames as well as the current frame.
Also cover scalar argument slots in bpf_mark_all_scalars_precise(), so
conservative fallback preserves the same invariant. Keep its existing
ancestor-only traversal and leave pointer and uninitialized slots alone.
AI assistance was used in preparing this patch.
Fixes: 0f6bd5e7a804 ("bpf: Support stack arguments for bpf functions")
Signed-off-by: Xu Yunxiang <xyx2021@mail.ustc.edu.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261102190350.788433-2-xyx2021@mail.ustc.edu.cn
When the time of a prog is up all loops with may_goto end, but bpf_loop()
goes on. It calls its callback up to 1 << 23 times. The loops in the
callback make no iteration, but the rest of the callback runs, and that can
be another bpf_loop(). Two test progs with bpf_loop(1 << 23, cb) and some
work in cb return after 4 sec and 10 sec. With this patch they return after
250 ms.
bpf_loop() and bpf_for_each_map_elem() end their loop when the callback
returns 1. So return 1 in the first insns of the callback when the time is
up:
r0 = address of the count
r0 = &(void __percpu *)(r0)
r0 = *(u64 *)(r0 +0)
if r0 != 0x0 goto pc+2
r0 = 1
exit
The verifier has seen what follows: the loop of the helper can end after
each call of the callback.
A function that BPF code calls too, or that is the callback of another
helper or kfunc, stays as it is: there the return value has another
meaning. The callback of bpf_user_ringbuf_drain() stays as it is too. The
helper takes a sample out of the buffer for each call, so a call that
does nothing loses the sample. Its loop ends after 128K samples.
This patch does not limit bpf_loop() itself. Only may_goto counts and reads
the clock. So nested bpf_loop() are not stopped when the prog has no
may_goto, or has none in the callbacks: bpf_loop(1 << 20, cb) with
bpf_loop(4096, cb2) in cb takes 5 sec. Nested open coded iterators have the
same problem. A follow up will fix these holes.
A call of such a callback executes 4 more insns. The cost on x86-64 (Zen 5)
in nsec:
before after
callback makes 1 iteration, inlined bpf_loop() 1.16 1.54
callback makes 1 iteration, bpf_loop() 6.52 6.74
callback without may_goto, inlined bpf_loop() 1.16 1.16
callback without may_goto, bpf_loop() 4.95 4.95
callback without may_goto, nested bpf_loop() 1.13 1.38
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-9-alexei.starovoitov@gmail.com
The count and the timestamp of may_goto are in the stack frame. So the
limit of 250 ms is for one call of a subprog, not for one invocation of the
prog, and loops that nest across calls multiply it. These 9 insns cause a
soft lockup:
main: sub:
1: call sub 1: may_goto 2f
may_goto 2f goto 1b
goto 1b 2: r0 = 0
2: r0 = 1 exit
exit
Use one count and one timestamp for all may_goto of a prog, in per-cpu
memory of the prog. The main prog sets them in its first insns. When the
time is up each loop ends at its next may_goto, in any function.
rX = address of the count
rX = &(void __percpu *)(rX)
r12 = *(u64 *)(rX +0)
if r12 == 0x0 goto target
r12 -= 1
if r12 != 0x0 goto pc+2
r12 = rX
call arch_bpf_timed_may_goto
*(u64 *)(rX +0) = r12
rX is a register that is dead at the may_goto. When all ten are live R0 is
saved in the stack.
An async callback can interrupt the prog on the same cpu. So its caller
sets count and timestamp, and puts the old values back at the end. A second
invocation of the prog on the same cpu gives more time to the first one.
x86-64 (Zen 5), nsec before after
iteration, empty loop 0.36 0.44
iteration, the body takes 4 cycles 0.89 0.89
call of a subprog and 1 iteration 1.21 1.16
The 56 progs of scx with may_goto get 1.7% more JITed code and use 25% less
stack.
Fixes: e723608bf428 ("bpf: Add verifier support for timed may_goto")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-8-alexei.starovoitov@gmail.com
arch_bpf_timed_may_goto() gets the offset of count and timestamp in the
stack frame in BPF_REG_AX. Arch code adds the frame pointer to it, and
calls bpf_check_timed_may_goto() with the result. On arm64, riscv,
loongarch, powerpc and s390 the asm function does it. On x86-64 the JIT
does it, because the frame pointer is in another register when the prog has
a private stack.
The next patch moves count and timestamp out of the stack frame. So pass
the pointer in BPF_REG_AX, and let the verifier emit the addition:
before after
r12 = -16 r12 = r10
call arch_bpf_timed_may_goto r12 += -16
call arch_bpf_timed_may_goto
The JITs know which register r10 is. The asm functions have a move in place
of the addition, and the special case in the x86-64 JIT goes away.
These insns run when the count is used up: in each 65535th iteration. The
cost of an iteration does not change.
Three tests have the insns of may_goto in them. Update them.
Tested on x86-64, and on arm64 and s390 in QEMU. riscv, loongarch and
powerpc are compile tested only.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-7-alexei.starovoitov@gmail.com
The timed form of may_goto calls arch_bpf_timed_may_goto(). This function
takes its argument and returns its result in BPF_REG_AX, in a register of
the cpu. Only JITed code can call it.
The verifier emits the timed form when the arch has a JIT that supports it.
It does not look at the prog. With CONFIG_BPF_JIT_ALWAYS_ON=n the
interpreter can run such a prog: the JIT is off, or it fails. The
interpreter calls the function like a helper. The function takes what the C
code left in that register for a pointer, and bpf_check_timed_may_goto()
reads and writes through it.
Emit the timed form only when the prog asks for a JIT, and make the JIT
mandatory for such a prog. All other progs get the form that counts down
from BPF_MAX_LOOPS.
One test of verifier_may_goto_1 shows the insns of the timed form. Load it
only when progs are JITed.
Fixes: e723608bf428 ("bpf: Add verifier support for timed may_goto")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-2-alexei.starovoitov@gmail.com
bpf_dynptr_slice(&ptr, off, buffer, size) and bpf_dynptr_slice_rdwr()
return a pointer to the data of a dynptr. When the data is not contiguous,
for example in a non-linear skb, they copy it to 'buffer' and return
'buffer'. Usually the buffer is on the stack. For the verifier the result
is plain memory (PTR_TO_MEM), and an access through it is not an access to
the stack:
- A store makes wrong what the verifier tracks in the buffer: a known
scalar, a spilled pointer, a dynptr, an iterator, irq flags.
- A load gives the bytes of a spilled pointer or of a dynptr as a scalar.
- Both are possible after the frame of the buffer returned.
The accesses are safe while the buffer contains only unknown bytes, as it
does after the call. Thus keep this rule: the buffer contains no tracked
data while a slice that can be the buffer exists. The slice remembers the
frame, the offset and the size of its buffer. The verifier invalidates the
slice, read-only or not, at each event that breaks the rule:
event slices of the buffer
------------------------------------------------- --------------------
the prog stores to bytes of the buffer invalid
a dynptr, an iterator or irq flags are made there invalid
the frame of the buffer returns invalid
a helper or a kfunc fills the buffer no change
a store next to the buffer, a read of the buffer no change
Tracked data gets into a stack slot in check_stack_write(), also the bytes
of a new dynptr, iterator or irq flags. Thus one call there, and one call
in prepare_func_exit(), are sufficient. Each is one walk over the
registers, only in progs that have such a slice. Helpers and kfuncs leave
only unknown bytes: mark_raw_stack() goes around check_stack_write().
The range of the buffer is in the part of bpf_reg_state that the verifier
copies with the register and compares for pruning. A slice that is the
buffer of a second call passes its range on. Local, ringbuf and skb
metadata dynptrs never return the buffer: no range. A dynptr argument of a
global function counts as an skb dynptr.
The usual pattern only reads the buffer, and it continues to load:
if (p == buffer)
bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
A prog that stores to the buffer and then uses the old slice is rejected:
"R7 invalid mem access 'scalar'". The diagnostics name the reason, and the
log shows the buffer of a slice: R7=mem(sz=8,buf=fp[0]-8+8).
Fixes: 66e3a13e7c2c ("bpf: Add bpf_dynptr_slice and bpf_dynptr_slice_rdwr")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261102170958.3586893-2-alexei.starovoitov@gmail.com
arg_track follows pointers to the stack through registers and spills.
Scalar evolution and the removal of dead slots rely on it to know which
slots an insn reads and writes.
arg_track assumes that a call does not change the pointers that are spilled
in the frame of the caller. But a static subprog or a callback can store
a pointer to other memory there, and the caller can use it:
*(u64 *)(r10 - 16) = fp-8
r1 = fp-16, r2 = map_value
call sub ; *(u64 *)(r1 + 0) = r2
r7 = *(u64 *)(r10 - 16) ; verifier: map_value. arg_track: fp-8
*(u64 *)(r10 - 8) = fp-24
*(u64 *)(r7 + 0) = 0 ; arg_track: fp-8 is overwritten
r8 = *(u64 *)(r10 - 8) ; verifier: fp-24. arg_track: none
loop:
r1 = *(u64 *)(r10 - 24) ; the counter of the loop
...
*(u64 *)(r8 + 0) = 4096 ; arg_track: not a write to fp-24
Scalar evolution does not see the last store. It gives the counter the
range of a loop that only increments it. The prog loads, and it accesses
a map value out of bounds.
Valid progs are rejected as well. The store through r7 makes fp-8 dead, and
a later read of fp-8 fails with "slot poisoned by dead code elimination".
When a static subprog or a callback can get a pointer to the frame of its
caller, join every slot of that frame with "not a pointer to the stack".
Reads and possible writes through a filled pointer are still seen, but no
store through it is a certain write. It has to be every slot: the callee
reaches the whole frame from one pointer, and callees are analyzed after
the caller. The price is precision: an 8-byte load through a filled
pointer gives a value that can point to any frame.
For a spilled copy of r10 this needs "bpf: Fix the join of fp+0 with
a value that is not a pointer to stack".
In selftests, Cilium and sched_ext the stack marks change in 4 of 12522
progs. Verdicts, insns and states do not change in any prog.
one_fp_clear_stack_threshold and two_fp_clear_stack_threshold expect the
fake offset fp+0 in the log now.
Fixes: bf0c571f7feb ("bpf: introduce forward arg-tracking dataflow analysis")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261102055913.3598414-4-alexei.starovoitov@gmail.com
arg_track follows pointers to the stack through registers and spills.
Scalar evolution and the removal of dead slots rely on it to know which
slots an insn reads and writes.
Where two paths meet, a register can be fp-N on one path and not a pointer
to the stack on the other path. There is no value for that. The join adds
a fake second offset, fp+0: with two offsets no store through the register
is a certain write. But for fp+0 the fake offset is the offset itself:
join(fp-8, none) = {fp-8, fp+0} two offsets
join(fp+0, none) = {fp+0} one offset: "exactly fp+0"
Then a store counts as a certain write to the stack, but at run time it can
go to other memory:
r7 = r10, or r7 = map_value + 8 ; arg_track: exactly fp+0
*(u64 *)(r10 - 8) = fp-24
*(u64 *)(r7 - 8) = map_value ; arg_track: fp-8 is overwritten
r8 = *(u64 *)(r10 - 8) ; fp-24 or map_value. arg_track: none
loop:
r1 = *(u64 *)(r10 - 24) ; the counter of the loop
...
*(u64 *)(r8 + 0) = 4096 ; arg_track: not a write to fp-24
Scalar evolution does not see the last store. It gives the counter the
range of a loop that only increments it. The prog loads, and it accesses
a map value out of bounds.
Valid progs are rejected as well. The store through r7 makes fp-8 dead, and
a later read of fp-8 fails with "slot poisoned by dead code elimination".
When the single offset is 0, drop the offsets. The result is a pointer to
an unknown slot of the frame, and all updates through it are weak.
Veristat sees no difference in selftests, Cilium and sched_ext.
Fixes: bf0c571f7feb ("bpf: introduce forward arg-tracking dataflow analysis")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261102055913.3598414-2-alexei.starovoitov@gmail.com
The user range of an arena must not cross a 4 GiB boundary. Thus
arena_get_unmapped_area() asks the mm for 2 * len bytes and rounds the
result up if necessary. This is correct only if the mm finds a free range.
With MAP_FIXED the mm does no search and returns addr. For
MAP_FIXED_NOREPLACE, do_mmap() also sets MAP_FIXED.
With MAP_FIXED the arena is not at addr after round_up(). Thus mmap()
removes mappings outside the range from addr to addr + len. It keeps a part
of those in this range. Also, round_up() can wrap to address 0, and the
test for address 0 is before it.
The arch code gets twice the length of the mapping. Thus it can refuse a
valid MAP_FIXED range. On s390 the arch code also grows the page table for
the range. With the double length it can omit this step for a valid range.
The test for a crossing is not exact. Because ret is a long, ret >> 32
extends the sign, but the other side is unsigned. The sides are never equal
for an address with bit 63 set, and round_up() always runs. Such user
addresses exist on s390. arena_map_alloc() does the same test with a u64.
With MAP_FIXED, pass len to the mm. Return -EINVAL if the range crosses a
boundary, as for address 0. Make ret unsigned, so that only these ranges
get the error. On x86-64, if round_up() gives address 0 today, mmap() now
fails with -EINVAL or with -ENOMEM from the mm. Today mmap() can also fail
for the address after round_up(), for example with -ENOMEM, -EEXIST, -EPERM
or -EBUSY.
Without MAP_FIXED the mm must still find a free range of 2 * len bytes
inside the address space of the task. The range after round_up() is inside
it, and round_up() cannot wrap. An address with bit 63 set now moves only
if its range crosses a boundary.
For an arena, libbpf uses MAP_FIXED only with map_extra, and
arena_map_alloc() refuses a map_extra range that crosses a boundary. A
subsequent mmap() must use the range of the first mmap(). For mmap() of the
map fd, this range comes from arena_get_unmapped_area(), thus it crosses no
boundary. arena_map_alloc() still accepts a map_extra above TASK_SIZE, and
mmap() of such an arena fails.
Fixes: 317460317a02 ("bpf: Introduce bpf_arena.")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261102064231.2153879-2-alexei.starovoitov@gmail.com