summaryrefslogtreecommitdiff
path: root/kernel/bpf/arraymap.c
diff options
context:
space:
mode:
authorAlexei Starovoitov <ast@kernel.org>2026-09-24 05:39:50 +0000
committerAlexei Starovoitov <ast@kernel.org>2026-09-24 05:39:50 +0000
commit24629aac43d2884109ae47a993fd51b504e2b09b (patch)
treef439f52a67096c92d639302ddc74d28b7aee472a /kernel/bpf/arraymap.c
parentb4d936547f5a3ca5563f48ee81f153c928336027 (diff)
parentfd318eab4b0905ffeb2fe1e93ad0d2e217f1f584 (diff)
Merge branch 'bpf-indirect-calls-of-bpf-subprogs-callx'
Alexei Starovoitov says: ==================== bpf: Indirect calls of bpf subprogs (callx) From: Alexei Starovoitov <ast@kernel.org> v1->v2: - Dropped "bpf: Don't fold loads from insn_array maps into constants". bpf_map_direct_read() rejects insn_array already. - patch 1: emit the same diag as sort_subprogs_topo() (bpf-ci). - patch 3 is new: taking the address of a function that is never called hit verifier_bug() in jit_subprogs(). Not related to callx, but callx makes it likely. - patch 4: BPF_OP() == BPF_CALL in liveness like in const_fold, explained in the comment why slots are live before callx callsite (Eduard). - patch 5: moved !bpf_is_callx() into nested 'if' (Eduard). JIT support is checked before the type of the register. - patch 6: sync callbacks are in the call graph too. Stack depth of a callback whose address is read from .rodata or taken in another subprog was not accounted. - patch 10: . check_cfg() explores functions of a table one at a time. It was losing changes_pkt_data and other effects of the functions in a table that call each other. . no cmpxchg64(), it's not available on all 32-bit archs. . don't reject the prog that reads a pointer to a function that was removed as dead code (sashiko). . load of the pointer sets jit_required, so the prog that passes it to a helper as a callback can't run in the interpreter (bpf-ci). . tbl[i].data at variable offset is a regular load when none of the possible offsets overlaps with a pointer (bpf-ci). - patch 12: append the functions and copy the map only for progs that have callx. The rest use .rodata as-is (bpf-ci). Pass token_fd to map create. Dropped dead bpf_map_lookup_elem() fallback. - patch 14: light skeleton reserves fd_array slots only when the object has callx. - selftests: anchored callx_retval_is_tracked to the caller, the tests of global func in a table and of const folding of the pointer test what they claim now, skip without JIT, fixed ASSERT_GT() of u32 and the comment, added tests for the above (bpf-ci). The table in callx_rodata_lskel test is volatile. clang was turning it into direct calls, so the test didn't test light skeleton. - Added acks. v1: https://lore.kernel.org/bpf/20260922011323.1298619-1-alexei.starovoitov@gmail.com/ Introduce BPF_JMP | BPF_CALL | BPF_X 'callx dst_reg' instruction: indirect call of bpf subprog with address in a register. LLVM emits it for calls via function pointer. The verifier rejected it as unknown opcode. Only static subprogs can be called. The callee is verified in the context of the caller like a direct call of static subprog. No callx into global subprogs, helpers or kfuncs. All callees are known before the main verifier pass. The address of the callee comes from: - ld_imm64 BPF_PSEUDO_FUNC, which is allowed for static subprogs only. add_subprogs() and check_cfg() see them already. - 64-bit load from read-only data: tables of functions, struct ops, vtables, where pointers are mixed with other data. Rust needs that: r6 = vtable ll r1 = *(u64 *)(r6 + 0) // data r2 = *(u64 *)(r6 + 8) // pointer to a function callx r2 libbpf keeps the data in a frozen read-only array map, one per prog that has callx, and stores byte offset of static function in the prog into the pointer. Progs without callx use .rodata as before. The kernel guesses pointers by that value before check_cfg() (patch 10). Wrong guess is safe. Functions referenced by data only are found via func_info (patch 9). After JIT the kernel replaces the offsets in the map with addresses. The prog must be the only user of the map, since other progs would have been verified with its old content. The prog reads addresses as data, so CAP_PERFMON is required. Both produce PTR_TO_FUNC. The passes before the main one don't know the callee and treat callx conservatively (patch 4). The main pass records caller -> callee edges. Recursion and stack depth checks use them afterwards (patch 6). In C: static const struct shape_ops square_ops = { 1, area, 10, perimeter }; ... return ops->area(x) * ops->scale + ops->perimeter(ops->id); and static int (* const handlers[])(struct xdp_md *) = { foo, bar, baz }; ... if (i < ARRAY_SIZE(handlers)) return handlers[i](ctx); plus what clang compiles without data: picking one of several functions, passing a function pointer into another function. Patches 1-3 are fixes I hit along the way: a prog without callx that hangs bpf_prog_load() in check_max_stack_depth(), its test, and verifier_bug() when the address of a function that is never called is taken. Not supported: - JITs other than x86-64 and arm64. No interpreter support. - x86-64 with FineIBT: bpf_jit_supports_callx() returns false. - tail calls in callees of callx. - pointers to functions in writable data, misaligned pointers. ==================== Link: https://patch.msgid.link/20260924031042.1690890-1-alexei.starovoitov@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Diffstat (limited to 'kernel/bpf/arraymap.c')
0 files changed, 0 insertions, 0 deletions