| Age | Commit message (Collapse) | Author |
|
How large the context of a raw tracepoint, tracepoint or syscall program
is depends on what the program is attached to, or is run with.
bpf_probe_register(), perf_event_set_bpf_prog() and the test runs compare
that with max_ctx_offset of the program.
An extension of the program, or of one of its global functions, gets the
same context. How far it reaches is noted in its own max_ctx_offset, which
nobody looks at. On a tracepoint with one argument it reads the stack
behind the arguments, in a test run the memory behind the context, or
from NULL if there is none:
BUG: KASAN: stack-out-of-bounds in bpf_prog_954b5d2051843629_ts28_ext_far+0x7e/0x113
Read of size 8 at addr ffffc90000def7b8 by task ts28/104
BUG: KASAN: slab-out-of-bounds in bpf_prog_954b5d2051843629_ts28_ext_far+0x7e/0x113
Read of size 8 at addr ffff88810712c3e8 by task ts28/104
Refuse to attach an extension that reaches further than the program which
it extends. A program that leaves the context to its extensions has to
touch as much of it as they do.
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-3-alexei.starovoitov@gmail.com
|
|
bpf_iter_prog_supported() goes by the name of the attach function. Commit
7cc639dc45c5 ("bpf: Don't take an extension for an iterator program")
keeps out extensions, for which that is the name of a function of another
program. A tracing program with BPF_TRACE_ITER that is loaded with
attach_prog_fd gets there as well: unlike for BPF_TRACE_RAW_TP,
bpf_check_attach_target() does not mind the target program.
If the function is called like that of an iterator target, bpf_iter_bpf_map
for example, its id in the BTF of the program is cached as the id of the
target, even if the load fails later. Programs of that target which are
loaded already cannot be attached any more. And the program itself can be
attached as an iterator of that target, with a context that the verifier
has taken from the prototype of that function:
BUG: KASAN: stack-out-of-bounds in bpf_prog_66d4234e2e50f5b6_ts26_forged+0x12a/0x20b
Read of size 8 at addr ffffc90000d9fc48 by task ts26/104
Refuse every program that has a target program. That covers extensions.
Fixes: 15d83c4d7cef ("bpf: Allow loading of a bpf_iter program")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-2-alexei.starovoitov@gmail.com
|
|
DSQ iteration
scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the
cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only
the stranded tasks placed again.
Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When
the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and
re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ
like scx_bpf_dsq_move() and is callable from the same contexts. The mark is
cleared when the task leaves the DSQ, so a task consumed before the walk is
not re-enqueued.
An iteration's DSQ can be destroyed before the iteration ends, so
schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the
scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same
path.
Declare the kfunc for the tools and add a compat probe.
v2: Note in the kerneldoc that another iteration's re-enqueue may take a
marked task early, and that a task on another scheduler returns false
(Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the rq lock switch on entry to scx_dsq_move() and its undo on exit
out into kf_switch_rq_lock() and kf_restore_rq_lock() so that other kfuncs
operating on a task from a DSQ iteration can share them. No functional
change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive 680edc7f23ad ("sched_ext: Don't queue balance callbacks from
a core-sched pick") and resolve its conflict with the proxy execution work
here so that for-next merges clean.
Conflicts:
- do_pick_task_scx(): 680edc7f23ad samples sched_core_enabled() into a new
local next to the prev declaration, which 3d6577ea93c9 ("sched_ext: Split
curr|donor references properly") changed to rq->donor. Kept rq->donor and
added the local.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Commit ff0545e0c263 ("sched_ext: Don't leave balance callbacks queued
across a pick retry") queues the deferred work as a balance callback
only when do_pick_task_scx() returns a final pick. However, under core
scheduling, a pick that returns a task isn't final either:
pick_next_task() goes on to pick the SMT siblings, and their dispatch
can release the shared rq lock with the callback still queued,
triggering the rq_pin_lock() warning.
Moreover, dispatch_core_pick() returns early when another dispatch is
in flight on the rq, and do_pick_task_scx() then consumes
%SCX_RQ_BAL_CB_PENDING set by that dispatch while it's still running.
This can be reproduced running scx_qmap -I on SMT siblings with
coresched-tagged workloads that frequently sleep, mixed with untagged
CPU-bound tasks.
Fix this by always deferring the pending work to the irq_work path from
dispatch_core_pick() under core scheduling, for both the local and the
sibling picks, and by leaving %SCX_RQ_BAL_CB_PENDING alone in
do_pick_task_scx(), so that it's handled by the dispatch that set it.
Sample the core scheduling state once at the start of the pick, since
core scheduling can get enabled while a non-core dispatch has the rq lock
released, and the follow-up work must match the dispatch path that ran.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
bpf_local_storage_map_free() takes the elements of the map off the lists
of their owners, which are alive, and frees them, and the storage of an
owner with its last element, behind a plain RCU grace period. Nobody uses
the map any more, but the other maps of these owners are in use. A
sleepable program calls the get and delete helpers under
rcu_read_lock_trace() only. While it walks the list of an owner for its
own map, or is about to lock the storage, the element it stands on, or the
storage, can be freed under it.
Whatever else takes an element off the list of a live owner frees it
behind an RCU Tasks Trace grace period, which implies a plain one. Do so
in bpf_local_storage_map_free() as well. It waits for these callbacks
before it frees the map.
This does not help a sleepable program that hands an owner to the helpers
which nobody holds, and which may be gone by then.
Fixes: dcf456c9a095 ("bpf: Fix usage of trace RCU in local storage.")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-31-alexei.starovoitov@gmail.com
|
|
The bpf_map_elem iterator, for all but the per-cpu maps, and the
bpf_sk_storage_map iterator hand the live value of an element to the
program as a writable buffer. The verifier does not know the map when it
checks the program, so nothing keeps the program from writing over a
bpf_spin_lock, a bpf_timer, a kptr or another special field, which neither
a map update nor a pointer to a map value can touch. With CAP_BPF and
CAP_PERFMON, or a BPF token, a program can so make the kernel follow a
pointer of its choice: after a write over a bpf_timer the kernel crashes
on the written value when the element is removed.
Offsets into a buffer are constants. Let the verifier record the range of
bytes that a program may write through a read-write buffer, and refuse the
program with -EACCES, at attach and at a link update, if a special field
of the map lies in that range.
A program that only reads, or that writes beside the special fields, works
as before. One that writes on both sides of a special field is refused,
since the range covers the field.
Fixes: d6c4503cc296 ("bpf: Implement bpf iterator for hash maps")
Fixes: 5ce6e77c7edf ("bpf: Implement bpf iterator for sock local storage map")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-30-alexei.starovoitov@gmail.com
|
|
bpf_probe_register() compares max_tp_access of the program that is
attached with the size of the buffer that the tracepoint lets programs
write to. An extension of that program, or of one of its global
functions, gets the same context and the same pointer to the buffer. What
it accesses is noted in its own max_tp_access, which nobody looks at. So
it reads and writes behind the buffer, of a tracepoint that may have none:
BUG: KASAN: stack-out-of-bounds in bpf_prog_5843f2b820eb4929_tr26_ext_far+0xf4/0x10c
Write of size 4 at addr ffffc90000d8f894 by task wext/103
Refuse such extensions.
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-29-alexei.starovoitov@gmail.com
|
|
A writable raw tracepoint hands a buffer to the program. The verifier
notes how far the program reaches into it, and bpf_probe_register() refuses
a program that reaches further than the tracepoint allows. A program that
does not touch the buffer passes, and can tail-call one of its type that
reads and writes behind it.
Take a BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE program with a tail call to
reach as far as can be, as for the stack, the packet and the buffers of
iterators. It cannot be attached any more. Plain raw tracepoint programs,
for which the limit is zero, cannot get at the buffer, and a tail call
stays within one type: they are left alone.
Fixes: 9df1c28bb752 ("bpf: add writable context for raw tracepoints")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-28-alexei.starovoitov@gmail.com
|
|
An extension inherits the expected_attach_type of the program it extends.
If that is BPF_TRACE_ITER, check_attach_btf_id() hands the extension to
bpf_iter_prog_supported(), which goes by the name of the attach function.
For an extension that is the name of the subprogram it replaces, so the
load fails with -EINVAL, unless that subprogram is called like the function
of a target, bpf_iter_bpf_map_elem for example.
Then the extension gets the context of that target. If the subprogram
has as many arguments, it can use the key and the value as buffers, but
how far it reaches is noted in the extension, where the test of the
iterator link does not look: it reads and writes behind the elements of
the map. And the id of the subprogram in the BTF of the program is cached
as the id of the target, so that programs of that target which are loaded
already cannot be attached.
Refuse all extensions here.
Fixes: 15d83c4d7cef ("bpf: Allow loading of a bpf_iter program")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-27-alexei.starovoitov@gmail.com
|
|
The bpf_map_elem, bpf_sk_storage_map and sockmap iterators hand the key
and the value of an element to the program as buffers. The verifier notes
how far a program reaches into them, and the iterator compares that with
the map when the program is attached. A program that does not touch the
buffers passes. It can tail-call another program of the same target,
which gets the same context, and whose reach nobody compares with the
map:
BUG: KASAN: slab-out-of-bounds in ___bpf_prog_run+0xa46a/0xae80
The program array can be filled at any time, so there is nothing to look
at when the link is made. For the same reason a program with a tail call
is taken to use the whole stack and the whole packet. Take it to reach
as far as can be into the buffers as well, so that these iterators refuse
it with -EACCES.
Fixes: d6c4503cc296 ("bpf: Implement bpf iterator for hash maps")
Fixes: 5ce6e77c7edf ("bpf: Implement bpf iterator for sock local storage map")
Fixes: 0365351524d7 ("net: Allow iterating sockmap and sockhash")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-26-alexei.starovoitov@gmail.com
|
|
Sleepable programs can only use the kinds of map that do without an RCU
read side critical section. check_map_prog_compatibility() sees to that
when a map is bound to the program, and asks whether the program is
sleepable. That misses the bpf_wq or bpf_task_work callback of a program
that is not, which runs sleepable all the same, and it misses an inner
map, which is not bound.
The bpf_wq callback of a tc program that looks a key up in a sockhash
gets as far as
WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0x18f/0x1e0
Ask the same question for the map that is handed to a helper in sleepable
context, unless the RCU read lock is held: by bpf_rcu_read_lock(), or by
the helper that runs the callback, as bpf_for_each_map_elem() does for a
hash map. Programs that take the lock go on working. Disabled preemption
or interrupts are not enough, since the maps ask for rcu_read_lock_held().
The other parts of the program may use such a map as before.
A global function that is called with preemption or interrupts disabled is
verified for the non-sleepable context, in which nothing is known of all
that. Verify it for a context of its own: not sleepable, and without the
RCU read lock. It differs in what maps can be used, and in nothing else.
What is refused from now on runs into the warning above, or uses what the
map frees behind a plain RCU grace period: an entry of a devmap, of a perf
event array or of a cgroup array.
A bloom filter does without RCU. Add it to the list, so that callbacks
which use one go on loading. As a side effect sleepable programs can use
bloom filters as well.
A pointer to the value of a devmap which a program keeps beyond
bpf_rcu_read_unlock() is not dealt with here.
Fixes: 638e4b825d52 ("bpf: Allows per-cpu maps and map-in-map in sleepable programs")
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-25-alexei.starovoitov@gmail.com
|
|
map->sleepable_refcnt tells bpf_map_fd_put_ptr() whether an inner map that
is taken out of its outer map has to wait for an RCU Tasks Trace grace
period as well. It counts the sleepable programs that use the outer map.
The bpf_wq or bpf_task_work callback of a program that is not sleepable
runs under rcu_read_lock_trace() only, and is not counted.
Such a callback can look an inner map up and keep it while user space
deletes it from the outer map. The inner map is freed behind a plain RCU
grace period, under the callback:
BUG: KASAN: slab-use-after-free in array_map_update_elem+0x6bc/0x7d0
Count a program with such a callback. The maps are bound before the
verifier meets the callback, so raise the count of those that are bound
when it does.
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-24-alexei.starovoitov@gmail.com
|
|
bpf_wq_work() runs the callback under rcu_read_lock_trace() only, since
the callback may sleep. The program it belongs to need not be sleepable,
and such a program is freed behind a plain RCU grace period. The
reference that the bpf_wq holds is dropped when the map element goes, or
when another callback is set, no matter whether the work is running.
If that was the last reference the program is freed under the running
callback, and the worker goes on in the freed JIT image:
Oops: int3: 0000 [#1] SMP NOPTI
RIP: 0010:0xffffffffc0600947
All it takes is to close the map and the program while the callback runs.
Note in the program that it has a callback that runs sleepable, and wait
for an RCU Tasks Trace grace period, which implies a plain one, before
such a program is freed. Let bpf_wq_work() enter its read side critical
section before it reads callback_fn.
bpf_task_work holds a reference to the program until the callback is done.
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-23-alexei.starovoitov@gmail.com
|
|
bpf_prog_dev_bound_init() looks the device up, takes bpf_devs_lock and
registers the device for the program. The reference it holds does not
keep the device registered. If bpf_dev_bound_netdev_unregister() has run
by the time the lock is taken, nobody takes the program off the device any
more: the bpf_offload_netdev and offload->netdev point to a net_device
that is freed as soon as the reference is dropped.
bpf_dev_bound_resolve_kfunc() dereferences it while the program is
verified, and a new device at the same address finds the old entry.
unregister_netdevice_many_notify() leaves NETREG_REGISTERED before it
calls bpf_dev_bound_netdev_unregister(), which takes bpf_devs_lock. Look
at reg_state under that lock, and treat a device that goes like one that
is not there.
Fixes: 2b3486bc2d23 ("bpf: Introduce device-bound XDP programs")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-22-alexei.starovoitov@gmail.com
|
|
bpf_prog_realloc() moves a program that the rewrites of the verifier make
outgrow its pages. It updates aux->prog, but not the offload->prog of a
device-bound program. When the device is unregistered,
__bpf_offload_dev_netdev_unregister() hands the stale pointer to
__bpf_prog_offload_destroy(), which faults on the freed program:
BUG: unable to handle page fault for address: ffffc90000071050
RIP: 0010:__bpf_prog_offload_destroy+0x3f/0x310
It takes the right to load a device-bound XDP program, CAP_BPF and
CAP_NET_ADMIN or a BPF token, and a device that can be removed.
Commit a1aa9ef47c29 ("bpf: Fix stale offload->prog pointer after constant
blinding") repaired the other place that replaces a program. Do the same
here.
Nothing keeps the device while a program is loaded, so it can also be
unregistered at that very moment. That frees the offload, and it may have
read offload->prog already when the old copy of the program is freed. In
both places update the pointer under bpf_devs_lock, before the old copy
goes.
Fixes: ab3f0063c48c ("bpf: offload: add infrastructure for loading programs for a specific netdev")
Fixes: 2b3486bc2d23 ("bpf: Introduce device-bound XDP programs")
Fixes: a1aa9ef47c29 ("bpf: Fix stale offload->prog pointer after constant blinding")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-21-alexei.starovoitov@gmail.com
|
|
The bpf_map_elem, bpf_sk_storage_map and sockmap iterators hand the key
and the value of an element to the program as buffers. The verifier only
records how far a program reaches into them, and ->attach_target()
compares that with the key and value size of the map.
bpf_iter_link_replace() does not. BPF_LINK_UPDATE installs a program that
an attach refuses with -EACCES, and the program then reads, and for a
value writes, behind every element that the iterator visits. It takes
CAP_BPF and CAP_PERFMON, or a BPF token.
Split the test out of the three ->attach_target() and ask it, as
->check_prog(), for the new program of an update as well. Targets without
such a test take every program of their kind, as before.
Fixes: d6c4503cc296 ("bpf: Implement bpf iterator for hash maps")
Fixes: 5ce6e77c7edf ("bpf: Implement bpf iterator for sock local storage map")
Fixes: 0365351524d7 ("net: Allow iterating sockmap and sockhash")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-20-alexei.starovoitov@gmail.com
|
|
btf_struct_show() and btf_array_show() recurse for every nested struct
and array. MAX_RESOLVE_DEPTH bounds one resolve walk when BTF is loaded,
not the nesting: a member whose type is resolved already is not walked
again, so BTF that defines its types from the inside out can nest them as
deep as it has types.
Whoever may load BTF, with CAP_BPF or a BPF token, can give such a type
to the value of a map. Reading the file of that map in bpffs then
overruns the kernel stack: with a few hundred levels the stack guard page
is hit and the kernel panics.
Stop at 32 levels. That is MAX_RESOLVE_DEPTH, and __btf_show_indent()
has as many columns. Nothing is shown for a member that lies deeper, and
the status becomes -E2BIG, which bpf_snprintf_btf() and
bpf_seq_printf_btf() return. The types of an x86_64 defconfig kernel nest
14 deep.
Fixes: b00b8daec828 ("bpf: btf: Add pretty print capability for data with BTF type info")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-19-alexei.starovoitov@gmail.com
|
|
bpf_local_storage_alloc() links the first element of a new storage to the
bucket of its map, and then publishes the storage in the owner with
cmpxchg(). If another cpu has been faster, the element is taken out of
the bucket again and is freed together with the storage.
In between the iterator over a sk_storage map can find the element. It
stays linked to a storage that points to the socket, but the socket knows
nothing of that storage: bpf_sk_storage_free() neither unlinks the element
nor waits for anybody who uses the storage. So nothing tells the iterator
that the socket has gone away.
Three cpus that create the first storage of a new AF_UNIX socket at the
same time, and a fourth that reads the iterator, are enough: in 20 seconds
its program was shown some 4000 elements of storages that had lost. If the
socket is closed in time, the program is handed a freed socket.
Link the element once the storage is published. Both happens under the
lock of the bucket, which holds off whoever finds the element through the
owner and wants to unlink it.
Fixes: c69d2ddb2072 ("bpf: Using rcu_read_lock for bpf_sk_storage_map iterator")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-17-alexei.starovoitov@gmail.com
|
|
bpf_local_storage_map_free() pins the owner with refcount_inc_not_zero()
while it uncharges an element. The pin counts when
bpf_local_storage_destroy() could not get local_storage->lock: destroy()
then goes on, drops the owner's reference and spins until the count is
zero. So the pin is the last reference whenever it is needed, and
refcount_dec() takes that for a bug: it warns "decrement hit 0; leaking
memory" and saturates the count. Unless destroy() happens to read the
count between the two stores it spins for ever, under rcu_read_lock().
Drop the pin with refcount_dec_and_test().
Fixes: 5d800f87d0a5 ("bpf: Support lockless unlink when freeing map or local storage")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-11-alexei.starovoitov@gmail.com
|
|
For a perf event on a kprobe or uprobe event which was made in tracefs,
bpf_get_kprobe_info() and bpf_get_uprobe_info() look the probe up by
name in the list of dynamic events, and hand a pointer to its symbol or
file name back to the caller. None of the callers holds event_mutex.
The perf event keeps its trace event alive, nothing else. Other dynamic
events are removed from the list and freed during the walk. And a probe
which shares its event with other probes can be deleted while the event
is in use, so even the probe which is found can be freed, together with
the string, before the caller has measured and copied it. The result is
a use-after-free in the walk, in strlen() and in the copy to user space,
or a NULL pointer dereference.
The readers are BPF_OBJ_GET_INFO_BY_FD and fdinfo of a perf link, for
whoever holds a descriptor of the link, and BPF_TASK_FD_QUERY. The other
side is anybody who may write to dynamic_events, kprobe_events or
uprobe_events.
Take event_mutex in bpf_get_perf_event_info() and give the caller a copy
of the name, which it has to free.
Fixes: 41bdc4b40ed6 ("bpf: introduce bpf subcommand BPF_TASK_FD_QUERY")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-10-alexei.starovoitov@gmail.com
|
|
In the branch of a BPF_JSET which is taken, the verifier can learn
something about the register only if the mask has a single bit: then this
bit is set. regs_refine_cond_op() tests this with is_power_of_2(val) and
then ORs val into var_off.
val is a u64, but is_power_of_2() takes an unsigned long. On a 32-bit
kernel it only sees the lower half of the mask. For
if r1 & 0xffffffff80000000 goto l
which is a BPF_JSET | BPF_K with an immediate of 0x80000000, the lower
half is a power of two. So the verifier takes all 33 bits for set at l,
while the branch is taken if any of them is. A mask of 0x100000001 from a
register does the same for bit 0.
The log of an i386 kernel, for r6 = bpf_get_prandom_u32() << 31:
3: (45) if r6 & 0x80000000 goto pc+2
from 3 to 6: R6=0xffffffff80000000 R10=fp0
6: (77) r6 >>= 63 ; R6=1
When the program runs, r6 >> 63 is 0.
>From then on the verifier works with known bits which the register does
not have at run time. It accepts, for example, an offset into a map value
which it derives from these bits and believes to be constant, and the
program can access memory outside of the map value.
Count the bits of the whole value.
Only 32-bit kernels are affected. On a 64-bit kernel unsigned long holds
the whole mask, and this patch changes nothing.
Fixes: 960ea056561a ("bpf: verifier: teach the verifier to reason about the BPF_JSET instruction")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-8-alexei.starovoitov@gmail.com
|
|
A sleepable program can call bpf_find_vma() and use a sleepable helper
such as bpf_copy_from_user() or bpf_copy_from_user_task() in the
callback. The verifier accepts this, but bpf_find_vma() runs the callback
with the mmap_lock of the mm held for read. If the user copy faults on
that mm, the fault handler has to take mmap_lock again. Lockdep reports
recursive locking, and if a writer queued up in between, the fault blocks
behind the writer, which waits for the read lock the helper already
holds:
sleepable prog other thread of the same mm
bpf_find_vma(current, ...)
mmap_read_trylock(mm)
mmap_write_lock(mm) <- waits
callback: copy_from_user()
page fault
mmap_read_lock(mm) <- queued behind the writer
This is the same problem as with sleepable task_vma iterator programs.
The verifier checks synchronous callbacks in the caller's context, so
in_sleepable_context() is true in the callback of a sleepable program.
Verify the callback like a hash map for_each callback, as running in an
RCU read-side critical section where sleeping is not allowed, and actually
hold rcu_read_lock() around it, so that global functions the callback
calls, which are then verified for a non-sleepable context, see the RCU
protection they are verified with.
Non-sleepable programs already run the callback under RCU, nothing changes
for them. A sleepable program whose bpf_find_vma() callback calls any
helper, kfunc or global function that may sleep is now rejected by the
verifier, including ones that would be harmless under mmap_lock as they
don't touch user memory.
Fixes: 7c7e3d31e785 ("bpf: Introduce helper bpf_find_vma")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-7-alexei.starovoitov@gmail.com
|
|
A sleepable "iter.s/task_vma" program can be attached, but its ->show()
callback runs with the mmap_lock of the iterated mm held for read. As soon
as the program runs, bpf_iter_run_prog() calls might_fault(), and any
bpf_copy_from_user() or bpf_copy_from_user_task() it does may really fault
or take mmap_lock again. With lockdep this is reported as recursive
locking of mmap_lock. Without lockdep, a fault on the reader's own mm, or
access_process_vm() on the iterated mm, has to take mmap_lock for read
again. If a writer queued up in the meantime that blocks behind the
writer, which in turn waits for the read lock the iterator already holds,
and the read() hangs:
reader (iter.s/task_vma) other thread of the same mm
task_vma_seq_get_next()
mmap_read_lock(mm)
mmap_write_lock(mm) <- waits
bpf prog: copy_from_user()
page fault
mmap_read_lock(mm) <- queued behind the writer
The reader waits killable, so the hang lasts until it gets a fatal signal.
bpf_iter_link_attach() allows sleepable programs for the targets that set
BPF_ITER_RESCHED, and task_vma sets it because cond_resched() between VMAs
is fine with mmap_lock held. Rescheduling between objects and running a
sleepable program inside ->show() are different properties, though, and
task_vma only has the first one. The open-coded task_vma iterator does not
have this problem since it hands out a snapshot of the VMA and holds no
lock when it returns to the program.
The vma handed to the program is only stable under mmap_lock, so the lock
can't be dropped around ->show(). Add a feature flag BPF_ITER_NOSLEEP for
targets which can't run sleepable programs although they can reschedule,
and set it for task_vma.
The check at attach time is not enough. BPF_LINK_UPDATE replaces the
program of a link, and bpf_iter_link_replace() only compares the type, the
attach type and the BTF id of the two programs, which are the same for a
sleepable and a non-sleepable one. Do the check there as well. That also
closes this way around the existing rule: so far a sleepable program could
be put on a link of a target that doesn't set BPF_ITER_RESCHED, like the
hash map element iterator, which runs it under rcu_read_lock().
BPF_LINK_CREATE of a sleepable program for the task_vma target, and
BPF_LINK_UPDATE to one, now fail with -EINVAL, also for programs that
don't touch user memory and have been working since sleepable iterator
programs exist. The programs can still be loaded. They have to be
loaded as non-sleepable programs, or use the open-coded iterator.
Fixes: b77fb25dcb34 ("bpf: Add support for bpf iterator programs to use sleepable helpers")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-6-alexei.starovoitov@gmail.com
|
|
When htab_map_update_elem() replaces an element of a preallocated map,
alloc_htab_elem() takes the spare element of this cpu and puts the old
element into the per-cpu slot right away, long before the old element is
unlinked from its bucket. The bucket lock disables interrupts, but it
does not keep out an NMI, or a tracing program on the return of
alloc_htab_elem(). bpf_prog_active only helps if both programs are
kprobe, tracepoint or perf_event programs. A program that updates an
existing key in another bucket of the same map gets to use the slot:
htab_map_update_elem(k1) tc, xdp, fentry, raw_tp, ...
alloc_htab_elem()
l_new = *pl_new
*pl_new = l_old still linked in bucket 1
<NMI>
htab_map_update_elem(k2) perf_event program
l_new = *pl_new the l_old of the outer update
hlist_nulls_add_head_rcu(l_new, bucket 2)
hlist_nulls_del_rcu(k2's old element)
</NMI>
hlist_nulls_add_head_rcu(l_new, bucket 1)
hlist_nulls_del_rcu(l_old)
If l_old was the first element of bucket 1, the outer add points its pprev
at l_new, and the delete takes it out of bucket 1 only. l_new->next now
leads into bucket 2, where l_old stays with a poisoned pprev, on which a
delete of k2 crashes. If it was not the first, the delete takes it out of
bucket 2: k2 is gone from the map, and the chain of bucket 1 still contains
l_old, whose next pointer leads into bucket 2.
Either way a walk of bucket 1 ends on the nulls value of bucket 2.
lookup_nulls_elem_raw() restarts whenever that happens, so every lookup in
bucket 1 which does not find its key spins forever:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
__htab_map_lookup_elem+0x354/0x4d0
BUG: KASAN: wild-memory-access in htab_map_delete_elem+0x30c/0x670
Write of size 8 at addr dead000000000122 by task extra/102
An NMI between the load and the store of the slot makes both updates use
the same element, with much the same result.
Empty the slot when the spare element is taken, in a way that is atomic
on this cpu, and put the old element in only after it has been unlinked
and its timers and the like have been cancelled. A nested update that
finds the slot empty, which is the case that corrupted the map, fails with
-EBUSY.
Fixes: 8c290e60fa2a ("bpf: fix hashmap extra_elems logic")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-5-alexei.starovoitov@gmail.com
|
|
bpf_token_create() allocates a bpffs inode and hands it to
alloc_file_pseudo(). The inode reference only moves to the new file when
that call succeeds: alloc_file_pseudo() leaves the caller's reference alone
on failure, and FD_PREPARE() does not even evaluate the file expression
when it cannot get a descriptor.
Before the conversion to FD_PREPARE() the error path did iput(inode). The
conversion dropped it, so each BPF_TOKEN_CREATE that fails with -EMFILE,
-ENFILE or -ENOMEM at this point leaks an inode. A task that is allowed to
create tokens can hit this at will by lowering RLIMIT_NOFILE, and the
leaked inodes make umount of that bpffs instance complain about busy
inodes.
Put the inode again if no file was created.
Fixes: 981bec8f698b ("bpf: convert bpf_token_create() to FD_PREPARE()")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-4-alexei.starovoitov@gmail.com
|
|
A sleepable program (e.g. SEC("syscall"), lsm.s, fentry.s) can pass a
callback to bpf_for_each_map_elem() on a hash map and call a sleepable
helper or kfunc such as bpf_copy_from_user() from it. The verifier
accepts this, but at run time the callback is invoked from inside an RCU
read-side critical section:
bpf_for_each_hash_elem()
rcu_read_lock();
callback_fn(map, key, val, ctx, 0);
bpf_copy_from_user() <- may fault and sleep
rcu_read_unlock();
bpf_each_rhash_elem() does the same for BPF_MAP_TYPE_RHASH. Sleeping
there triggers the "Voluntary context switch within RCU read-side
critical section!" warning, delays grace periods, and with
CONFIG_PREEMPT_RCU=n silently ends the read-side critical section that
the bucket walk relies on.
Synchronous callbacks are verified with the caller's state, so
kernel/bpf/verifier.c:in_sleepable_context() only looks at the program's
sleepability and the locks the program took itself, and does not know
that the helper holds rcu_read_lock() around the callback.
With a SEC("syscall") program whose callback reads a page that has to be
read from disk first:
BUG: sleeping function called from invalid context at include/linux/uaccess.h:169
RCU nest depth: 1, expected: 0
__might_fault+0x29/0x80
_copy_from_user+0x26/0x90
bpf_copy_from_user+0x23/0x60
bpf_prog_17d44fa39eebd6cc_cb+0x35/0x46
bpf_for_each_hash_elem+0xdb/0x280
Fix by marking the callback frame of hash map iteration as running under
RCU and making in_sleepable_context() false while such a frame is on the
call stack, so sleepable helpers, kfuncs and global functions are
rejected there as inside bpf_rcu_read_lock(). The rules for trusting RCU
protected pointers are left alone, which is conservative and keeps
callbacks that verify today and do not sleep working. The flag is
compared in func_states_equal() so that pruning never mixes the two
contexts. Array map callbacks run without any lock and keep the caller's
context.
Fixes: 314ee05e2fc6 ("bpf: Add hashtab support for bpf_for_each_map_elem() helper")
Fixes: 818e00848227 ("bpf: Implement iteration ops for resizable hashtab")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-3-alexei.starovoitov@gmail.com
|
|
A devmap egress program that removes a whole fragment of a redirected
xdp_frame with bpf_xdp_adjust_tail() or bpf_xdp_pull_data() frees it
with page_frag_free() whatever the frame's memory type is. For a
MEM_TYPE_PAGE_POOL frame from a page_pool based driver this drops the
page reference owned by the pool while the pool still accounts the
fragment ("page_pool leak" bad page, leaked pp reference).
kernel/bpf/devmap.c:dev_map_bpf_prog_run() points xdp.rxq at an
on-stack
struct xdp_rxq_info rxq = { .dev = rx_dev };
whose mem.type is 0, i.e. MEM_TYPE_PAGE_SHARED, and
net/core/filter.c:bpf_xdp_shrink_data() releases frags according to
rxq->mem.type. cpumap, which runs programs on xdp_frames the same way,
already copies xdpf->mem_type into its private rxq for each frame.
With 9000 byte frames from virtio_net and an egress program that cuts the
fragments off:
BUG: Bad page state in process swapper/3 pfn:10660b
page dumped because: page_pool leak
bad_page+0xbd/0x190
__free_frozen_pages+0x5c7/0x900
bpf_xdp_adjust_tail+0x3be/0x480
bpf_prog_dad43f894a3faff7_egress+0x15c/0x1cd
dev_map_bpf_prog_run+0x288/0x610
The pool waits for these pages for ever when its device is gone. That is
the harmless case. A page of the pool can hold more than one fragment
while its refcount is 1: the first page_frag_free() frees it under the
fragment which is still in use, and the second one runs into
page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0)
kernel BUG at include/linux/mm.h:1846!
veth is not affected. It takes references of its own on the pages and
redirects the frame as MEM_TYPE_PAGE_SHARED.
Fix by setting rxq.mem.type from each frame before running the devmap
program, as cpu_map_bpf_prog_run_xdp() does.
Fixes: ca9984c5f0ab ("bpf: devmap: provide rxq after redirect")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017163745.3341787-2-alexei.starovoitov@gmail.com
|
|
The dispatch path can request deferred work to be run from a balance
callback and pick_task_scx() queues rq->scx.deferred_bal_cb right after
dispatching. However, a balance callback must be consumed before the rq
lock is released and pick_task_scx() is not necessarily the end of the
pick:
- it returns RETRY_TASK when a higher priority class task has been
enqueued while the dispatch had the rq lock released;
- the dl_server pick can return NULL, in which case the lower classes
are picked next.
In both cases the callback stays queued while the core keeps picking,
and the rq lock can be released again, either by prev_balance() or by
another dispatch. For example, with an RT task as prev:
__schedule()
__pick_next_task()
pick_task_scx()
dispatch_pick()
maybe_queue_balance_callback() <- deferred_bal_cb queued
return RETRY_TASK
prev_balance()
balance_rt()
pull_rt_task()
double_lock_balance() <- rq lock released
Any CPU acquiring the rq lock in that window finds a pending balance
callback and triggers the following warning in rq_pin_lock():
WARNING: kernel/sched/sched.h:1919 at rq_lock+0x7f/0xe0, CPU#10: stress-ng-sleep/533
CPU: 10 UID: 0 PID: 533 Comm: stress-ng-sleep Not tainted 7.2.0-rc6-virtme #61 PREEMPT(full)
Sched_ext: qmap (enabled+all)
Call Trace:
<IRQ>
try_to_wake_up+0x2c1/0x5c0
hrtimer_wakeup+0x22/0x30
__hrtimer_run_queues+0x1d5/0x350
hrtimer_interrupt+0x106/0x1e0
__sysvec_apic_timer_interrupt+0x6f/0x210
sysvec_apic_timer_interrupt+0x71/0x90
</IRQ>
This can be reproduced running scx_qmap -I together with a SCHED_FIFO
workload that frequently sleeps, e.g.:
$ vng -v --cpus 16 --memory 4G -- \
'tools/sched_ext/build/bin/scx_qmap -I >/dev/null & \
stress-ng --sleep 4 --sched fifo --sched-prio 1 -t 30 -q'
Fix this by queueing the balance callbacks only when the pick is final.
Otherwise, defer the pending work to the irq_work path, which doesn't
depend on the current rq lock section, and leave the kick sync wait
pending for the next pick.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 30fb2b1e4d41 ("sched_ext: Fix the running task keeping its
protection into bypass") and 5809f749fbdb ("sched_ext: Fix a bypassed task
stalling on a cpu its scheduler has no caps on") and resolve their conflicts
with the proxy execution work here.
Conflicts:
- scx_bypass(): 30fb2b1e4d41 drops the protection strip in the runnable_list
walk, which never sees the running task, and strips rq->curr ahead of the
walk instead. 3d6577ea93c9 ("sched_ext: Split curr|donor references
properly") moved the walk's strip to task_current_donor(), as protection
belongs to the scheduling context. set_next_task_scx() takes the donor off
the runnable_list, so that strip is dead too. Dropped it, and the new
strip takes rq->donor.
- scx_bypass(), merged without a marker: 5809f749fbdb evicts the running
task when bypass lifts if it can't stay on the cpu, on rq->curr.
3d6577ea93c9 moved the cap revoke eviction of the running task in
reenq_local() to rq->donor. The lift eviction takes rq->donor too, and
both comments in the block say the current donor.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A bypassing sub-scheduler's tasks are run by its nearest ancestor that isn't
bypassing. The ancestor puts them on the local DSQ under its own caps.
Whether such a task may stay on the cpu and whether a cap revoke reenqueues
it are still checked against the caps of the task's own scheduler.
The bypass load balancer moves tasks between cpus without looking at caps,
so a bypassed task can run on a cpu where its own scheduler holds none. Once
its slice runs out, it can't stay. It goes to the ancestor's bypass DSQ
without a kick, and the cpu can go idle with it queued until something else
reschedules the cpu.
Make both checks against the caps of the scheduler running the task. When
bypass lifts, the running task keeps its slice without a cap check, so evict
it then if it can't stay on the cpu.
Fixes: 75a8c8202c91 ("sched_ext: Add reject DSQ for cap-rejected dispatches")
Fixes: 46a85ae6fe5b ("sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Protection ends when the scheduler enters bypass, so that bypass scheduling
can cut every task's slice. scx_bypass() clears it for the queued tasks by
cycling them, and strips the running task's in the same walk over
rq->scx.runnable_list. The running task is never on that list.
set_next_task_scx() takes it off and put_prev_task_scx() puts it back, so
the strip never runs. A protected running task then keeps its slice into
bypass, where the tick's attempts to cut it are refused.
Strip the running task's protection outside the walk.
Fixes: 9cfc6ab34a31 ("sched_ext: Add SCX_TASK_PROTECTED")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Commit ea2e6467ac36 ("bpf: Return hashes of maps in BPF_OBJ_GET_INFO_BY_FD")
started copying struct bpf_map_info in from userspace in order to read
the new hash and hash_size input fields. Before that, the kernel ignored
the contents of the user buffer and returned a zeroed structure with only
the output fields filled in.
Some output fields are only set for certain maps: btf_id, btf_key_type_id
and btf_value_type_id when the map has BTF (and btf_vmlinux_id for
struct_ops maps, ifindex, netns_dev and netns_ino when the map is
offloaded). For all other maps, these fields now return whatever
userspace passed in before.
This breaks userspace which reuses one bpf_map_info for several queries.
Restore the original behavior by zeroing the structure and copying in
only hash and hash_size from userspace before the output fields are
filled in.
Fixes: ea2e6467ac36 ("bpf: Return hashes of maps in BPF_OBJ_GET_INFO_BY_FD")
Reported-by: Quentin Monnet <qmo@kernel.org>
Reported-by: Tianyi Chen <hi@tychen.cc>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016191635.662690-1-daniel@iogearbox.net
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux
Pull printk fixes from Petr Mladek:
- Switch NBCON consoles to the emergency mode in emergency_restart() to
see messages even in this shutdown code path
- Postpone the start of printk kthreads after smp_init(). It prevents
boot stalls on some HW
- Typo fix in a previous hot-fix patch
* tag 'printk-for-7.3-hotfix' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux:
printk: Postpone kthread startup
printk: fix kernel-doc name for nbcon_write_context_is_braille
nbcon/reboot: Switch NBCON consoles into the emergency mode in emergency_restart()
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt:
- Fix NULL dereference when copying histogram field variable
If event_hist_trigger_parse() creates an action that uses one of
pseudo fields for its keys, the hist_data field element will have a
NULL for its field member. When looking to match that key, the lookup
uses key_field->field->name which will dereference a NULL pointer.
Instead use the helper function that handles such cases:
expr_field_str().
- Take the trace_types_lock when dropping the buffer_meta reference
The tracing_buffer_meta_open() error return drops the trace_array
reference with __trace_array_put() instead of trace_array_put() where
the former doesn't take the lock and the latter does.
- Fix NULL pointer dereference when reading per_cpu/cpuN/buffer_meta
These files are created for every possible CPU. But it only contains
data for CPUs that have come online. If one is read for a CPU that
hasn't come online yet, it triggers a NULL pointer dereference.
Have reading of CPUs that have not come on line yet return -ENODEV.
- Fix an out-of-bounds shift in the log2 histogram modifier
Histograms can take a log2 modifier to have the buckets defined by
log2 increments. If a shift is greater than ULONG_MAX or above 2^63,
the key falls into a lower order bucket.
Use order_base_2() instead of "ilog2(n - 1) + 1" to have all
overflowed keys go into the top bucket.
- Fix the persistent ring buffer rb_meta_subbuf_idx() accounting
The algorithm for finding the index of a sub-buffer in a persistent
ring buffer buffer did not take into account the meta data at the
beginning of the buffer causing the wrong index to be returned.
As rb_subbufs_from_meta() does the proper accounting to find the
first sub-buffer, use that function instead of repeating the logic
(incorrectly) and make the calculations based off of that.
- Fix the symbol name checks for the remote buffer code
The remote ring buffer has a build time check to make sure its not
using any code that will not be in the hypervisor. For various
architectures, there has been false positive reports from this
checker.
Fix them.
- Fix false warning in ring_buffer_map_get_reader()
There is a WARN_ON() after rb_get_reader_page() if no reader is
returned. But there are legitimate cases where this can happen.
Remove the WARN_ON() and also a check that covers one of the
legitimate cases that tries to exit out early. There's no need for
the check if it just simply allows the return of NULL from
rb_get_reader_page().
- Fix the ignored return value from resize_buffer_duplicate_size()
Some clean up of the code in tracing_snapshot_write() removed
checking the return value of resize_buffer_duplicate_size() and if
that failed it would not be detected.
- Fix the histogram timestamps for buffered events
The histogram timestamps are taken from the events that are recorded.
It currently reads ring_buffer_event_time_stamp() but if the event is
buffered it does not read it properly. Use tracing_event_time_stamp()
which handles both buffered and normal events.
- Do not initialize tracefs if tracing is disabled at boot
If tracing fails to allocate or some other error is detected,
tracing_disabled is set. Do not initialize tracefs in this case as it
may expect data to be initialized but is not.
- Free eventfs inode after an RCU grace period
Currently eventfs_inodes are freed after a SRCU grace period as the
iterations in the code use SRCU. There are locations that the inode
may still be accessed from the VFS that uses RCU which does not have
the same synchronization as SRCU. Add an RCU wait before free eventfs
inodes.
- Do not fail persistent and remote buffers under memory pressure
As persistent and remote buffers have their own pre-allocated memory,
there's no reason to check how much memory is available before
created them. By checking the memory pressure, these buffers may fail
to be created even though their memory is already present.
- Account for subbuffer size when checking how much memory is available
Before allocating a ring buffer, the size is checked against the
current memory pressure of the system. But it assumes every
sub-buffer is of page size. This is no longer the case. Use the
actual sub-buffer size when calculating how much memory will be used.
- Reject non-stack fields for synthetic stacktrace fields
Stack traces are special fields and the checks currently do not do
enough to make sure that the fields being processed as stack traces
are indeed stack traces. Update the logic to properly detect what is
or is not a stack trace.
- Fix a crash in print_graph_function_event()
When a tracer does not define static or default flags the
tr->current_trace_flags is set to NULL. If the ring buffer is not
cleared when switching tracers, the previous tracers events could be
read. The function_graph entries will dereference the
tr->current_trace_flags and cause a crash.
Set the tr->current_trace_flags to a empty set of flags to prevent it
from being NULL.
- Separate tracing filter predicate count from allocation
The amount allocated for the predicate program is the number of
predicates defined plus two for adding "true" and "false" at the end.
The nr_preds is incremented by two to accommodate the booleans but is
then used as the limit check for number of predicates. This is
incorrect as it is now two higher than what is expected.
Use a separate variable for the size of the predicate array.
- Fix the filter operator stack check
The operator stack increments the pointer and then writes the content
onto the stack. The check for overflow compared the current pointer
to see if it was greater than the stack size before doing so. This is
an off by 2 error as the write will be at the size - 1 and the
compare is done before the increment. Adjust the check accordingly.
* tag 'trace-v7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing: Check operator stack capacity before pushing
tracing: Keep the predicate limit separate from program capacity
tracing: Fix crash in print_graph_function_event()
tracing/histograms: Reject non-stacktrace fields for synthetic stacktrace fields
ring-buffer: Use subbuf_order in available memory estimation
ring-buffer: Skip si_mem_available() for persistent and remote buffers
eventfs: Free the eventfs_inode after an rcu grace period too
tracing: Do not initialize tracefs work if tracing is disabled
tracing: Fix hist trigger timestamps for buffered events
tracing: Fix ignored return value of resize_buffer_duplicate_size()
ring-buffer: Fix false warning in ring_buffer_map_get_reader()
tracing: Fix simple_ring_buffer undef check on sparc64
tracing: Fix simple_ring_buffer undef check on parisc
tracing: Fix empty symbol names in simple_ring_buffer undef check
ring-buffer: Fix the sub-buffer array base in rb_meta_subbuf_idx()
tracing: Fix the shift out of bounds in the log2 histogram modifier
ring-buffer: Do not show buffer_meta for CPUs without a buffer
tracing: Take trace_types_lock when dropping the buffer_meta reference
tracing: Fix NULL dereference when copying keys for a field variable
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fix from Masami Hiramatsu:
- error out if BTF is required but unavailable
Fix to return an error instead of NULL if BTF is not available when
user specify '$args*' meta fetch argument for probe events.
* tag 'probes-fixes-v7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
tracing: probes: error if BTF is required but unavailable
|
|
Manual merge fixup in ns_common_types.h:ns_common to work around a conflict
with the nsfs RCU grace period fix.
* lsm/dev:
lsm: remove redundant NULL check in security_init()
selftests/landlock: Check full ioctl commands in audit records
lsm: Preserve full ioctl commands in audit records
doc: fix LSM ordering description for /sys/kernel/security/lsm
doc: describe CONFIG_LSM and lsm= as the selection mechanism
selftests/bpf: verify mount idmaps reach inode hooks
lsm: expose mount idmaps to inode hooks
rust: lsm: replace `core::mem::zeroed` with `pin_init::zeroed`
cred: prevent slab cache merging for cred_jar
lsm: don't call security_backing_file_free() multiple times
lsm: delete duplicate assignment in lsm_prepare()
lsm: update the BUILD_BUG_ON() in audit_log_lsm_data()
lsm: add LSM_AUDIT_DATA_NS for namespace audit records
lsm: add LSM blob and hooks for namespaces
ns: Free anonymous mount namespaces via ns_common_free()
cred: clarify that task_struct::cred is only for the current task
|
|
When the BTF of a function in the target program is unreliable, the
model of the function given to its trampoline is MAX_BPF_FUNC_REG_ARGS
u64 arguments. The verifier takes a static function's stack arguments
from its BTF before it finds the BTF unreliable, though, so such a
function can still take more: one with six or more arguments, one of
them a plain pointer, for example. The trampoline saves and restores
five registers then. After an fentry program, the rest of the register
arguments, r9 on x86-64 and x5 to x7 on arm64, hold whatever
__bpf_prog_exit() left there, and for fexit the stack arguments are not
copied either, so the function goes on with garbage in arguments the
verifier checked, pointers included.
Refuse the attach when such a function has more argument slots in its
BTF than fit in registers. The main program is left alone: it takes only
its context, and the verifier gives it no stack arguments when its BTF
is unreliable.
Fixes: 0f6bd5e7a804 ("bpf: Support stack arguments for bpf functions")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016183120.1136008-1-yonghong.song@linux.dev
|
|
A trampoline on a BPF function, fentry, fexit or fsession attached
through attach_prog_fd, takes the function's arguments from where BPF
code left them. A call from one BPF function to another passes slot N
where the kernel convention passes its Nth u64 argument, in a register
or a stack slot, whatever the types of the arguments: the x86-64, arm64
and riscv JITs all place a stack argument that way. The model of the
function, distilled from its BTF, still carries those types, so each
arch places the arguments by its own rules for them. For a 16-byte
argument at an odd slot, such as s in
u64 f(u64 a, struct { __int128 v; } s, u64 b)
the arm64 trampoline reads s from x2 and x3, where the caller left it in
x1 and x2, and b from x4. And an x86-64 trampoline puts a 16-byte
argument at slot 6 wholly on the stack, while x86-64 BPF code splits it
between r9 and the stack. The s390 trampoline, which takes a 16-byte
argument by reference, loads s through its first half as if that were a
pointer, and saves and restores only three of the four registers the
caller passed.
Rewrite the model of a target program's function in
bpf_check_attach_target() as one u64 argument per argument slot.
Every convention places N u64 arguments in slot order, so all of the
above take them from where BPF code put them, and no arch has to know
that the function is a BPF one.
The model no longer gives the size and flags of each argument, so an
arch that sign-extends a narrow signed argument, or clears the unused
bytes of a narrow stack argument, for the program does not do that for a
BPF function any more: the program gets the eightbyte BPF code passed,
as it does on x86-64, and fexit hands it back unchanged.
Fixes: afdbb4cdd3d2 ("bpf, arm64: Place trampoline arguments by the arm64 calling convention")
Fixes: 4dd676ca5b7a ("bpf: Support by-value struct arguments up to 16 bytes")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016183100.1134528-1-yonghong.song@linux.dev
|
|
irq_work_is_busy() cannot see the per-CPU send_signal_work while
it is being filled: the check only matches after irq_work_queue()
has claimed the work. A concurrent caller therefore passes it, both
callers race for the same irq_work, and the loser's signal is
silently lost along with its task reference while the queued work
runs with a mix of both callers' fields.
kprobe, tracepoint and perf_event programs exclude each other on
the same CPU through bpf_prog_active, so for two callers to meet,
one of the programs has to be a raw tracepoint or an fentry one.
And since commit 87c544108b61 ("bpf: Send signals asynchronously
if !preemptible") this path runs with IRQs enabled too, so a hard
IRQ can interrupt the fill as well, not only an NMI. Found by code
inspection and reproduced with a perf_event NMI program racing an
fentry one.
Claim the work through work->task itself: cmpxchg() it from NULL
to the target task before touching the other fields, and set it
back to NULL once the callback has consumed them. A context
finding the work claimed returns the documented -EBUSY.
Fixes: 1bc7896e9ef4 ("bpf: Fix deadlock with rq_lock in bpf_send_signal()")
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009022427.968304-1-chenyuan@kylinos.cn
|
|
create_field_var_hist() creates a histogram on the matched event to
supply a field variable, and records it on the target so it can be torn
down later:
hist_data = find_compatible_hist(target_hist_data, file);
...
/* Save the compatible histogram information */
var_hist->hist_data = hist_data;
hist_data is not the histogram it creates. It is one that was already
there, whose keys the new one copies. The saved pointer has a single
user, unregister_field_var_hists(), which runs when the target is
removed:
file = hist_data->field_var_hists[i]->hist_data->event_file;
find_compatible_hist() matches on keys alone, so it can pick one with no
variables of its own. check_var_refs() then returns false and the user
can remove it while the target still points at it:
BUG: KASAN: slab-use-after-free in event_hist_trigger_free+0x2b2/0x320
Read of size 8 at addr ffff8880093b90e0 by task init/1
Freed by task 1:
kfree+0x154/0x420
event_hist_trigger_free+0x1d5/0x320
event_hist_trigger_parse+0x35f3/0x69e0
trigger_process_regex+0x1a6/0x250
Save the event file instead. It is all the dereference ever wanted, and
it does not go away when the user removes a trigger.
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/all/20260914110753.221E61F000FF@smtp.kernel.org/
Link: https://patch.msgid.link/20260914234754.3676905-1-donggeunyoo.kernel@gmail.com
Fixes: 02205a6752f2 ("tracing: Add support for 'field variables'")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Acked-by: Tom Zanussi <zanussi@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The jump tables which compute_subprog_jts() builds are only read through
bpf_insn_successors() once CFG discovery is done. Its last caller is
bpf_compute_scev(), which runs after bpf_compute_live_registers(), so
free the tables right after it instead of keeping them until the verifier
environment is torn down. With large insn_array maps, this also drops them
before do_check() starts to allocate states.
Dead code removal runs after do_check(), so a removed subprogram no longer
owns a table and adjust_subprog_starts_after_remove() can go back to not
caring about the table pointers.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016155559.652461-1-daniel@iogearbox.net
|
|
register/unregister/modify_fentry_multi()
Since 'tr->ip' is already a 'ftrace_location()'-ed IP in
bpf_trampoline_lookup(), it is unnecessary to call ftrace_location() again
in register/unregister/modify_fentry_multi().
No functional changes intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://patch.msgid.link/20261016142014.12420-3-leon.hwang@linux.dev
|
|
A trampoline managed by ftrace is determined by its target, that said
ftrace_managed is already known when the trampoline is initialized. so,
set 'ftrace_managed = tr->ip != 0' in bpf_trampoline_get() instead of
relying on the first regular or tracing_multi attachment to set it.
This makes the backend selection independent of attachment order and keeps
the registration callbacks focused on installing the trampoline.
No functional change intended.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://patch.msgid.link/20261016142014.12420-2-leon.hwang@linux.dev
|
|
update_last_data_if_empty() is called from tracing_open() with O_TRUNC,
tracing_read_pipe() and tracing_buffers_read() without trace_types_lock.
Two readers can both pass the LAST_BOOT test and free module_delta
twice, and a module notifier can update module_delta while it is being
freed.
Take trace_types_lock in those paths and keep the callers that already
hold it on an unlocked variant.
Cc: stable@vger.kernel.org
Fixes: 32dc0042528d ("tracing: Reset last-boot buffers when reading out all cpu buffers")
Link: https://patch.msgid.link/20261016061638.214847-2-devnexen@gmail.com
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
trace_module_record() walks ftrace_trace_arrays from the module notifier
without trace_types_lock. A concurrent rmdir of an instance removes it
from the list and frees it under that lock, so a module load or unload
can walk into a freed trace_array.
Cc: stable@vger.kernel.org
Fixes: 5f3719f697c3 ("tracing: Update modules to persistent instances when loaded")
Link: https://patch.msgid.link/20261016061638.214847-1-devnexen@gmail.com
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
|
|
syscall_copy_user_array() copies the buffer argument of faultable
syscalls (write, pwrite64, ...) with __copy_from_user(), which skips
access_ok(). Use copy_from_user() instead so a non-user address is
rejected and the entry is recorded as faulted.
It could be a security concern for machines that trust perf a bit too
much.
Cc: stable@vger.kernel.org
Fixes: baa031b7bd2ce ("tracing: Allow syscall trace events to read more than one user parameter")
Link: https://patch.msgid.link/6ad19afd.a938ce27.16851e.3440@mx.google.com
Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
There have been various bug reports related to SMP systems hanging
on boot. The reports vary but seem to have some overlap with regard
to missed IPI's during SMP setup. The reports also show that
reverting NBCON console usage allows the bugs to disappear.
The NBCON kthreads for console printing are started very early
during boot using early_initcall(). Since these kthreads are
asynchronously enabling/disabling interrupts during early boot, they
may be interfering with early SMP setup on some hardware platforms.
Starting the console printing kthreads later has shown to avoid the
hanging on systems that could previously reproduce the issue.
Postpone kthread starting until after all arch initcalls have
completed.
Reported-by: Derek Barbosa <debarbos@redhat.com>
Closes: https://lore.kernel.org/linux-rt-users/arPmYPbFLf2EldIl@debarbos-thinkpadt14gen5.rmtusma.csb
Fixes: 76f258bf3f2a ("printk: nbcon: Introduce printer kthreads")
Signed-off-by: John Ogness <john.ogness@linutronix.de>
Tested-by: Derek Barbosa <debarbos@redhat.com>
Link: https://patch.msgid.link/20261016085441.857695-1-john.ogness@linutronix.de
Signed-off-by: Petr Mladek <pmladek@suse.com>
|