diff options
| author | Alexei Starovoitov <ast@kernel.org> | 2026-09-24 05:39:50 +0000 |
|---|---|---|
| committer | Alexei Starovoitov <ast@kernel.org> | 2026-09-24 05:39:50 +0000 |
| commit | 24629aac43d2884109ae47a993fd51b504e2b09b (patch) | |
| tree | f439f52a67096c92d639302ddc74d28b7aee472a /kernel/bpf/arraymap.c | |
| parent | b4d936547f5a3ca5563f48ee81f153c928336027 (diff) | |
| parent | fd318eab4b0905ffeb2fe1e93ad0d2e217f1f584 (diff) | |
Merge branch 'bpf-indirect-calls-of-bpf-subprogs-callx'
Alexei Starovoitov says:
====================
bpf: Indirect calls of bpf subprogs (callx)
From: Alexei Starovoitov <ast@kernel.org>
v1->v2:
- Dropped "bpf: Don't fold loads from insn_array maps into constants".
bpf_map_direct_read() rejects insn_array already.
- patch 1: emit the same diag as sort_subprogs_topo() (bpf-ci).
- patch 3 is new: taking the address of a function that is never called
hit verifier_bug() in jit_subprogs(). Not related to callx, but callx
makes it likely.
- patch 4: BPF_OP() == BPF_CALL in liveness like in const_fold, explained
in the comment why slots are live before callx callsite (Eduard).
- patch 5: moved !bpf_is_callx() into nested 'if' (Eduard). JIT support
is checked before the type of the register.
- patch 6: sync callbacks are in the call graph too. Stack depth of
a callback whose address is read from .rodata or taken in another
subprog was not accounted.
- patch 10:
. check_cfg() explores functions of a table one at a time. It was
losing changes_pkt_data and other effects of the functions in
a table that call each other.
. no cmpxchg64(), it's not available on all 32-bit archs.
. don't reject the prog that reads a pointer to a function that was
removed as dead code (sashiko).
. load of the pointer sets jit_required, so the prog that passes it to
a helper as a callback can't run in the interpreter (bpf-ci).
. tbl[i].data at variable offset is a regular load when none of
the possible offsets overlaps with a pointer (bpf-ci).
- patch 12: append the functions and copy the map only for progs that
have callx. The rest use .rodata as-is (bpf-ci). Pass token_fd to map
create. Dropped dead bpf_map_lookup_elem() fallback.
- patch 14: light skeleton reserves fd_array slots only when the object
has callx.
- selftests: anchored callx_retval_is_tracked to the caller, the tests
of global func in a table and of const folding of the pointer test what
they claim now, skip without JIT, fixed ASSERT_GT() of u32 and
the comment, added tests for the above (bpf-ci). The table in
callx_rodata_lskel test is volatile. clang was turning it into
direct calls, so the test didn't test light skeleton.
- Added acks.
v1: https://lore.kernel.org/bpf/20260922011323.1298619-1-alexei.starovoitov@gmail.com/
Introduce BPF_JMP | BPF_CALL | BPF_X 'callx dst_reg' instruction: indirect
call of bpf subprog with address in a register. LLVM emits it for calls
via function pointer. The verifier rejected it as unknown opcode.
Only static subprogs can be called. The callee is verified in the context
of the caller like a direct call of static subprog. No callx into global
subprogs, helpers or kfuncs.
All callees are known before the main verifier pass. The address of
the callee comes from:
- ld_imm64 BPF_PSEUDO_FUNC, which is allowed for static subprogs only.
add_subprogs() and check_cfg() see them already.
- 64-bit load from read-only data: tables of functions, struct ops,
vtables, where pointers are mixed with other data. Rust needs that:
r6 = vtable ll
r1 = *(u64 *)(r6 + 0) // data
r2 = *(u64 *)(r6 + 8) // pointer to a function
callx r2
libbpf keeps the data in a frozen read-only array map, one per prog
that has callx, and stores byte offset of static function in the prog
into the pointer. Progs without callx use .rodata as before.
The kernel guesses pointers by that value before check_cfg() (patch 10).
Wrong guess is safe. Functions referenced by data only are found via
func_info (patch 9). After JIT the kernel replaces the offsets in
the map with addresses. The prog must be the only user of the map,
since other progs would have been verified with its old content. The
prog reads addresses as data, so CAP_PERFMON is required.
Both produce PTR_TO_FUNC.
The passes before the main one don't know the callee and treat callx
conservatively (patch 4). The main pass records caller -> callee edges.
Recursion and stack depth checks use them afterwards (patch 6).
In C:
static const struct shape_ops square_ops = { 1, area, 10, perimeter };
...
return ops->area(x) * ops->scale + ops->perimeter(ops->id);
and
static int (* const handlers[])(struct xdp_md *) = { foo, bar, baz };
...
if (i < ARRAY_SIZE(handlers))
return handlers[i](ctx);
plus what clang compiles without data: picking one of several functions,
passing a function pointer into another function.
Patches 1-3 are fixes I hit along the way: a prog without callx that
hangs bpf_prog_load() in check_max_stack_depth(), its test, and
verifier_bug() when the address of a function that is never called is
taken.
Not supported:
- JITs other than x86-64 and arm64. No interpreter support.
- x86-64 with FineIBT: bpf_jit_supports_callx() returns false.
- tail calls in callees of callx.
- pointers to functions in writable data, misaligned pointers.
====================
Link: https://patch.msgid.link/20260924031042.1690890-1-alexei.starovoitov@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Diffstat (limited to 'kernel/bpf/arraymap.c')
0 files changed, 0 insertions, 0 deletions
