bpf, x86: Support fetching AND/OR/XOR atomics in arena

x86-64 has no single instruction for a fetching AND/OR/XOR, so the JIT
lowers them to a CMPXCHG loop. That loop could not be used against arena
memory: it contains two memory accesses, the load of the old value and
the CMPXCHG itself, either of which can fault when the arena page goes
away, while the verifier reserves only one exception table entry per
instruction. bpf_jit_supports_insn() therefore rejected the three opcodes
and such programs failed to load with

  BPF_ATOMIC stores into R1 arena is not allowed

Emit the same loop with R12-indexed addressing and give each of the two
accesses its own exception table entry. The verifier still reserves one
per instruction, so the JIT accounts for the extra entry itself by
rescanning the instruction stream in bpf_int_jit_compile() before the
extable is sized, as s390 does in bpf_jit_alloc() for its BPF_XCHG
lowering. The rescan sits after the extra-pass early exit, so a program
is counted exactly once. Drop the bpf_jit_supports_insn() rejection.

A loop without the load would need only one entry, since a CMPXCHG that
loses the comparison reloads RAX, but its first CMPXCHG would then almost
always lose. Every operation would pay for a second locked CMPXCHG, which
makes the fetching insn more expensive than the load and CMPXCHG loop
programs write by hand today. The second entry is paid for once, in the
JIT.

Both entries resume past the whole loop rather than past the faulting
instruction alone, with the fetch destination cleared, so a fault cannot
re-enter the loop, and both are reported as writes since the BPF insn is
a read-modify-write. This repurposes the INSN_LEN field of the fixup as a
resume distance rather than an instruction length; a sequence long enough
to overflow its 8 bits would first have to exceed BPF_MAX_INSN_SIZE, which
do_jit() rejects before the image is used.

The loop needs RAX for CMPXCHG and saves R0 in BPF_REG_AX, which it also
substitutes for R0 when either operand is R0. ex_handler_bpf() may then
have to name r10 as the register holding the arena address or the one to
clear on fault; reg2pt_regs[] already has that entry.

An access through a number, made by a program loaded with
BPF_F_ARENA_SCALAR, keeps its address in BPF_REG_AX itself (see
arena_scalar_access()), which is where the loop would save R0. Save R0 on
the stack in that case, as the div and mod lowering does. An R0 operand
stays in that stack slot: the new value is built from it, RAX ends up
holding the old value that R0 is to return, and the slot is dropped
instead of popped. Both entries resume before that pop or drop, so a
fault leaves the stack balanced.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20261016171319.1623447-2-puranjay@kernel.org

0 files changed, 0 insertions, 0 deletions