bpf: Make unwind_exit of an exception callback exit in JITs

For the verifier unwind_exit of an exception callback is exit. For JITs
the callback is a subprog, so they emit the lookup of the return address.
The idea was that the kernel calls the callback and the lookup finds
nothing.

But an exception callback has no frame of its own. It returns from
the frame of the main prog that threw, and the kernel isn't always
the caller of that one:
- An extension returns to the caller of the subprog that it replaced.
- After a tail call in a subprog the main prog returns to the caller of
  that subprog.

When unwind_goto follows that call, the callback goes to the pad, and
the pad wasn't verified with it:

  r6 = 1
  call a             // can unwind
  unwind_goto pad
  r6 = 2
  call b             // cannot unwind
  unwind_goto pad    // stays, since the pad is not dead code
  exit
pad:
  r0 = r6            // r6 is 1 for the verifier

An extension without unwind_exit can replace b(). When it throws and its
exception callback ends with unwind_exit, the pad runs with r6 = 2.

Both JITs ask bpf_prog_can_unwind(). Let it say no for an exception
callback.

An async callback has a frame of its own and the kernel is the caller.

Fixes: a2d274771321 ("bpf: Add unwind_goto and unwind_exit insns")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/5f24cbed1ec4a6a0ecd0b834d738c48dc9eb64362be4fa50dfb0eaddfad928b3@mail.kernel.org/
Link: https://patch.msgid.link/20261020231942.2568163-2-alexei.starovoitov@gmail.com
Closes: https://lore.kernel.org/bpf/sashiko-outbox-179498@kernel.org/

3 files changed, 6 insertions, 2 deletions

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index b3efe0668e86..326e9e1dea0a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1273,7 +1273,7 @@ static long bpf_unwind_fixup(u64 lr)
 	e = search_bpf_extables(ptrauth_strip_kernel_insn_pac(lr) - 1);
 	/*
 	 * The verifier made sure that unwind_goto follows a call in a prog. When
-	 * the kernel called an async or an exception callback, it's exit.
+	 * the kernel called an async callback, it's exit.
 	 */
 	return e ? e->fixup : 0;
 }
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d3ab36281144..6a2fe3d0039d 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2552,7 +2552,7 @@ static u64 bpf_unwind_target(u64 ret_addr)
 
 	/*
 	 * The verifier made sure that unwind_goto follows a call in a prog. When
-	 * the kernel called an async or an exception callback, it's exit.
+	 * the kernel called an async callback, it's exit.
 	 */
 	return e ? ret_addr + e->fixup : ret_addr;
 }
diff --git a/include/linux/filter.h b/include/linux/filter.h
index fccd8db33647..e0218e677a1d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -267,9 +267,13 @@ static inline bool bpf_is_unwind_exit(const struct bpf_insn *insn)
 /*
  * The caller of the main prog doesn't tell unwind_exit from exit. An extension
  * is different: its caller is the caller of the subprog that it replaced.
+ * An exception callback returns from the frame of the main prog. After freplace
+ * or a tail call it returns to a prog that wasn't verified with the callback.
  */
 static inline bool bpf_prog_can_unwind(const struct bpf_prog *prog)
 {
+	if (prog->aux->exception_cb)
+		return false;
 	return bpf_is_subprog(prog) || prog->type == BPF_PROG_TYPE_EXT;
 }