Merge tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Pull bpf updates from Alexei Starovoitov:
 "Major changes:

   - Introduce scalar evolution (SCEV) analysis in the verifier.

     It infers how registers and stack slots change in a loop and lets
     the verifier check bounded loops with induction variables as ranges
     instead of walking every iteration. The number of processed insns
     drops ten fold in programs with loops (Eduard Zingerman)

   - Add 'callx' instruction: indirect call of a static BPF subprogram.

     The address is taken with ld_imm64 or loaded from read-only data.
     Supported by x86-64, arm64 and riscv64 JITs (Alexei Starovoitov, Pu
     Lehui)

   - Add unwind_goto and unwind_exit instructions, so that Rust panic or
     C++ exception can run destructors of every frame it passes through.
     The normal path is free of unwind code. Supported by x86-64 and
     arm64 JITs (Alexei Starovoitov)

   - Support programs compiled by rust-bpf: treat load and store through
     a number as arena access with BPF_F_ARENA_SCALAR, accept rust names
     in BTF (Alexei Starovoitov)

   - Raise BPF program stack size from 512 bytes to 2 KiB on x86-64 and
     arm64 (Kumar Kartikeya Dwivedi)

   - Support struct, union and __int128 up to 16 bytes passed and
     returned by value in BPF subprograms and kfuncs, and fix x86-64 and
     arm64 trampolines for such arguments (Yonghong Song)

   - Introduce a common way to attach struct_ops to a cgroup via
     bpf_link. bpf_tcp_ops is the first user and mirrors TCP sock_ops
     callbacks (Martin KaFai Lau, Amery Hung)

   - Describe inlined functions in BTF. New kinds tell where the inline
     sites are and how to fetch the arguments there.

     CONFIG_DEBUG_INFO_BTF_INLINE keeps it apart from the main BTF in
     .BTF.inline section or in btf_vmlinux_inline.ko (Alan Maguire)

   - Fix many memory safety, lifetime and locking bugs in BPF core,
     verifier, iterators, local storage, dynptr, devmap, cpumap and
     seg6local (Josef Bacik, Alexei Starovoitov)

   - Diff stat is dominated by selftests and by removal of
     lib/test_bpf.c. Its 16k lines of instruction tests were converted
     to BPF selftests that BPF CI runs on every architecture it covers
     (Daniel Borkmann)

  Other features and fixes:

   - Add KASAN checks to memory accesses of JITed programs on x86-64
     (Alexis Lothoré)

   - Complete unification of helper, kfunc and subprog argument checks
     in the verifier (Amery Hung)

   - Add bpf_copy_from_user_mm[_str]() kfuncs (Anastasios Papagiannis)

   - Various features and fixes (Daniel Borkmann):
       - Add .bpf keyring for program signature verification, modelled
         after dm-verity keyring, and support ML-DSA in signing tools
       - Emit shorter and faster code in x86-64 JIT for patterns that
         LLVM generates
       - Fix various issues in gotox handling

   - Let BPF LSM set xattrs at inode creation with bpf_inode_init_xattr()
     (David Windsor, Daniel Borkmann)

   - Use sleepable page allocation in sleepable arena paths. Fix races
     between arena free, allocation and user page fault (Emil Tsalapatis)

   - Switch BPF crypto from lskcipher to AES-CBC and AES-ECB libraries
     (Eric Biggers)

   - Add bpf_proactive_reclaim() kfunc (Hui Zhu)

   - Render decl_tags in libbpf btf_dump and add tests for bpftool C
     dump (Ihor Solodrai)

   - Speed up tracing_multi attach on x86 (Jiri Olsa)

   - Various features and fixes (Kumar Kartikeya Dwivedi):
       - Add BPF_PROG_STREAM_OPEN command to read BPF streams through a
         file descriptor with poll support
       - Let kernel structs publish arena pointers with __arena type tag
       - Fix __uninit kfunc output buffers and fastcall rewrite with
         indirect stack accesses
       - Refuse CO-RE relocations for signed programs

   - Add enqueue_rcvq and dequeue_rcvq hooks to bpf_tcp_ops to adjust
     sk_rcvlowat and avoid unnecessary wakeups (Kuniyuki Iwashima)

   - Add bpf_ksock_sendto() kfunc and allow ksock connect and send from
     bpf_wq (Mahe Tardy)

   - Fix racy plt target update in arm64 bpf_arch_text_poke()
     (Matthew Wood)

   - Restructure BPF selftests Makefile as a layered build
     (Mykola Lysenko)

   - Speed up hash map lookups for u32 and u64 keys (Mykyta Yatsenko)

   - Add signed div/mod, movsx, bswap and gotol to MIPS JITs, signed
     div/mod to parisc JITs (Nicholas Dudar, Johan Almbladh)

   - Fix stack overflow in build ID parser on 32-bit (Nickolai
     Zeldovich)

   - Add stack arguments, signed arena loads and gotox to riscv64 JIT.
     Pu Lehui is now co-maintainer of riscv BPF JITs (Pu Lehui, Chen
     Pei, Feng Jiang)

   - Add bpf_call_rcu() and bpf_call_rcu_tasks_trace() kfuncs. Support
     fetching and/or/xor atomics in arena on x86-64 (Puranjay Mohan)

   - Fix flaky selftests, selftests with clang 23 and bpf-gcc
     (Qiliang Yuan, KaFai Wan, Vineet Gupta)

   - Fix builds broken by fatal resolve_btfids warnings, BTF generation
     on ARC, and define arch_cmpxchg_local() on arc, hexagon and
     openrisc (Sasha Levin)

   - Reduce memory use of hash maps by up to 50% per element
     (T.J. Mercier)

   - Mark pointer fields of sched_ext ops arguments as trusted (Tejun
     Heo)

   - Remove legacy bpf_asm, bpf_dbg and bpf_jit_disasm tools (Xia Yan)

   - Fix index overflow in queue and stack maps, arena range tree after
     allocation failure, tail call loop after freplace detach, lost
     signal in bpf_send_signal() (Yuan Chen)

   - Many more fixes and cleanups"

* tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (660 commits)
  selftests/bpf: Time tc_edt from its first packet with data
  selftests/bpf: Test unwind_exit of an exception callback
  selftests/bpf: Tests for 32-bit scalar evolution bounds computation
  bpf: Reject unwind_goto and unwind_exit in offloaded progs
  bpf: Support 32-bit linear scalar evolution
  bpf: Make unwind_exit of an exception callback exit in JITs
  selftests/bpf: Test forbidden bpf_ksock_sendto() LSM attach
  selftests/bpf: Test ksock sendto()
  bpf: Add bpf_ksock_sendto() kfunc
  selftests/bpf: Test unwinding from R0: R2 subprog
  selftests/bpf: Add a C++ prog with exceptions
  selftests/bpf: Add tests for unwind_goto and unwind_exit
  bpf, arm64: Support unwind_goto and unwind_exit
  bpf, x86: Support unwind_goto and unwind_exit
  bpf: Add unwind_goto and unwind_exit insns
  tools/bpf: Remove legacy BPF tools
  bpf: Annotate data races on bpf_lru_node type
  selftests/bpf: Add arena allocation race tests
  bpf: Fix arena race between page free and alloc leading to incoherency
  bpf: Revalidate faulted arena pages before installing user mappings
  ...

3 files changed, 193 insertions, 17 deletions

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 732b35cc08d1..f29b0aad440c 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -56,6 +56,19 @@
 
 enum bpf_cond_pseudo_jmp {
 	BPF_MAY_GOTO = 0,
+	/*
+	 * Follows a call of a bpf subprog. Taken when the subprog left with
+	 * BPF_EXIT that has src_reg == BPF_UNWIND_EXIT.
+	 */
+	BPF_UNWIND_GOTO = 1,
+};
+
+enum bpf_pseudo_exit {
+	/*
+	 * When bpf_exit->src_reg == BPF_UNWIND_EXIT, it's the same as bpf_exit,
+	 * except that BPF_UNWIND_GOTO after the call is taken.
+	 */
+	BPF_UNWIND_EXIT = 1,
 };
 
 /* Register numbers */
@@ -936,6 +949,33 @@ union bpf_iter_link_info {
  * 		0 on success or -1 if an error occurred (in which case,
  * 		*errno* is set appropriately).
  *
+ * BPF_PROG_STREAM_OPEN
+ *	Description
+ *		Open a file descriptor for one of the BPF streams associated
+ *		with the program identified by *prog_fd*. The stream is selected
+ *		by *stream_id*.
+ *
+ *		The returned file descriptor supports **read**\ (2) and
+ *		**poll**\ (2). Reads block while the stream is empty unless
+ *		**BPF_F_STREAM_NONBLOCK** is specified in *flags*. A non-blocking
+ *		read of an empty stream fails with **EAGAIN**.
+ *
+ *		**poll**\ (2) reports **POLLIN** when data is available and
+ *		**POLLHUP** once the program has been freed, that is, after every
+ *		reference to it, including links and other file descriptors, has
+ *		been dropped. Hangup may lag the final release because program
+ *		teardown is deferred. Buffered data remains readable after
+ *		**POLLHUP** and a read returns zero after all such data has been
+ *		consumed.
+ *
+ *		The file descriptor is read-only and has the close-on-exec flag
+ *		set. It is not seekable and **lseek**\ (2) fails with **ESPIPE**.
+ *		*flags* may only contain **BPF_F_STREAM_NONBLOCK**.
+ *
+ *	Return
+ *		A new file descriptor (a nonnegative integer), or -1 if an
+ *		error occurred (in which case, *errno* is set appropriately).
+ *
  * NOTES
  *	eBPF objects (maps and programs) can be shared between processes.
  *
@@ -993,6 +1033,7 @@ enum bpf_cmd {
 	BPF_TOKEN_CREATE,
 	BPF_PROG_STREAM_READ_BY_FD,
 	BPF_PROG_ASSOC_STRUCT_OPS,
+	BPF_PROG_STREAM_OPEN,
 	__MAX_BPF_CMD,
 	BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
 };
@@ -1316,6 +1357,12 @@ enum bpf_perf_event_type {
 /* The verifier internal test flag. Behavior is undefined */
 #define BPF_F_TEST_REG_INVARIANTS	(1U << 7)
 
+/*
+ * Load and store through a number is an access to the arena of the program
+ * at the low 32 bits of the number. It's for programs written in Rust.
+ */
+#define BPF_F_ARENA_SCALAR	(1U << 8)
+
 /* link_create.kprobe_multi.flags used in LINK_CREATE command for
  * BPF_TRACE_KPROBE_MULTI attach type to create return probe.
  */
@@ -1524,6 +1571,11 @@ enum {
 	BPF_STREAM_STDERR = 2,
 };
 
+/* flags for BPF_PROG_STREAM_OPEN command */
+enum {
+	BPF_F_STREAM_NONBLOCK = (1U << 0),
+};
+
 union bpf_attr {
 	struct { /* anonymous struct used by BPF_MAP_CREATE command */
 		__u32	map_type;	/* one of enum bpf_map_type */
@@ -1756,7 +1808,7 @@ union bpf_attr {
 			__u32	prog_cnt;
 			__u32	count;
 		};
-		__u32		:32;
+		__u32		type_id;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
@@ -1950,6 +2002,12 @@ union bpf_attr {
 		__u32		flags;
 	} prog_assoc_struct_ops;
 
+	struct {
+		__u32		prog_fd;
+		__u32		stream_id;
+		__u32		flags;
+	} prog_stream_open;
+
 } __attribute__((aligned(8)));
 
 /* The description below is an attempt at providing documentation to eBPF
@@ -4867,15 +4925,18 @@ union bpf_attr {
  * 		The non-negative copied *buf* length equal to or less than
  * 		*size* on success, or a negative error in case of failure.
  *
- * long bpf_load_hdr_opt(struct bpf_sock_ops *skops, void *searchby_res, u32 len, u64 flags)
+ * long bpf_load_hdr_opt(void *ctx, void *searchby_res, u32 len, u64 flags)
  *	Description
  *		Load header option.  Support reading a particular TCP header
- *		option for bpf program (**BPF_PROG_TYPE_SOCK_OPS**).
+ *		option for bpf program (**BPF_PROG_TYPE_SOCK_OPS**).  For the
+ *		**bpf_tcp_ops** struct_ops, this helper can be called from the
+ *		**parse_hdr**\ () and **write_hdr_opt**\ () operators.
  *
- *		If *flags* is 0, it will search the option from the
- *		*skops*\ **->skb_data**.  The comment in **struct bpf_sock_ops**
- *		has details on what skb_data contains under different
- *		*skops*\ **->op**.
+ *		If *flags* is 0, it will search the option from the packet
+ *		associated with the current operation.  For
+ *		**BPF_PROG_TYPE_SOCK_OPS**, the comment in
+ *		**struct bpf_sock_ops** has details on what skb_data
+ *		contains under different *op*.
  *
  *		The first byte of the *searchby_res* specifies the
  *		kind that it wants to search.
@@ -4908,6 +4969,8 @@ union bpf_attr {
  *
  *		* **BPF_LOAD_HDR_OPT_TCP_SYN** to search from the
  *		  saved_syn packet or the just-received syn packet.
+ *		  Not supported by the **bpf_tcp_ops** struct_ops, which
+ *		  rejects all flags.
  *
  *	Return
  *		> 0 when found, the header option is copied to *searchby_res*.
@@ -4928,9 +4991,9 @@ union bpf_attr {
  *		packet.
  *
  *		**-EPERM** if the helper cannot be used under the current
- *		*skops*\ **->op**.
+ *		operation.
  *
- * long bpf_store_hdr_opt(struct bpf_sock_ops *skops, const void *from, u32 len, u64 flags)
+ * long bpf_store_hdr_opt(void *ctx, const void *from, u32 len, u64 flags)
  *	Description
  *		Store header option.  The data will be copied
  *		from buffer *from* with length *len* to the TCP header.
@@ -4946,7 +5009,9 @@ union bpf_attr {
  *		by searching the same option in the outgoing skb.
  *
  *		This helper can only be called during
- *		**BPF_SOCK_OPS_WRITE_HDR_OPT_CB**.
+ *		**BPF_SOCK_OPS_WRITE_HDR_OPT_CB**, or from the
+ *		**write_hdr_opt**\ () operator of the **bpf_tcp_ops**
+ *		struct_ops.
  *
  *	Return
  *		0 on success, or negative error in case of failure:
@@ -4961,9 +5026,9 @@ union bpf_attr {
  *		**-EFAULT** on failure to parse the existing header options.
  *
  *		**-EPERM** if the helper cannot be used under the current
- *		*skops*\ **->op**.
+ *		operation.
  *
- * long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
+ * long bpf_reserve_hdr_opt(void *ctx, u32 len, u64 flags)
  *	Description
  *		Reserve *len* bytes for the bpf header option.  The
  *		space will be used by **bpf_store_hdr_opt**\ () later in
@@ -4973,7 +5038,9 @@ union bpf_attr {
  *		the total number of bytes will be reserved.
  *
  *		This helper can only be called during
- *		**BPF_SOCK_OPS_HDR_OPT_LEN_CB**.
+ *		**BPF_SOCK_OPS_HDR_OPT_LEN_CB**, or from the
+ *		**hdr_opt_len**\ () operator of the **bpf_tcp_ops**
+ *		struct_ops.
  *
  *	Return
  *		0 on success, or negative error in case of failure:
@@ -4983,7 +5050,7 @@ union bpf_attr {
  *		**-ENOSPC** if there is not enough space in the header.
  *
  *		**-EPERM** if the helper cannot be used under the current
- *		*skops*\ **->op**.
+ *		operation.
  *
  * void *bpf_inode_storage_get(struct bpf_map *map, void *inode, void *value, u64 flags)
  *	Description
@@ -6890,6 +6957,8 @@ struct bpf_link_info {
 		} xdp;
 		struct {
 			__u32 map_id;
+			__u32 :32;
+			__u64 cgroup_id;
 		} struct_ops;
 		struct {
 			__u32 pf;
@@ -7284,6 +7353,31 @@ enum {
 					 */
 };
 
+/*
+ * Most callbacks of struct bpf_tcp_ops are placed in the slow
+ * path (e.g., one-shot connection setup or unlikely events like
+ * timers) and are invoked simply by defining non-NULL callbacks.
+ *
+ * Callbacks in the fast path, however, would incur noticeable
+ * overhead even when set to NULL, so they are disabled by default
+ * and must be explicitly enabled per socket via bpf_tcp_ops_set_flags().
+ *
+ * The flags are per-socket and shared by all effective bpf_tcp_ops
+ * programs.
+ */
+enum {
+	/* .rtt() */
+	BPF_TCP_OPS_FLAG_RTT			= (1 << 0),
+	/* .parse_hdr() */
+	BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_ALL	= (1 << 1),
+	BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_UNKNOWN	= (1 << 2),
+	/* .hdr_opt_len() and .write_hdr_opt() */
+	BPF_TCP_OPS_FLAG_WRITE_HDR_OPT		= (1 << 3),
+	/* .enqueue_rcvq() and .dequeue_rcvq() */
+	BPF_TCP_OPS_FLAG_RCVQ			= (1 << 4),
+	BPF_TCP_OPS_FLAG_ALL			= (1 << 5) - 1,
+};
+
 /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
  * changes between the TCP and BPF versions. Ideally this should never happen.
  * If it does, we need to add code to convert them before calling
@@ -7600,6 +7694,10 @@ struct bpf_task_work {
 	__u64 __opaque;
 } __attribute__((aligned(8)));
 
+struct bpf_rcu_head {
+	__u64 __opaque[6];
+} __attribute__((aligned(8)));
+
 struct bpf_wq {
 	__u64 __opaque[2];
 } __attribute__((aligned(8)));
diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h
index 618167cab4e6..11d65871d163 100644
--- a/include/uapi/linux/btf.h
+++ b/include/uapi/linux/btf.h
@@ -54,8 +54,8 @@ struct btf_type {
 	 *             decl_tag and type_tag
 	 */
 	__u32 info;
-	/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
-	 * "size" tells the size of the type it is describing.
+	/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC, ENUM64
+	 * and LOC_PARAM. "size" tells the size of the type it is describing.
 	 *
 	 * "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
 	 * FUNC, FUNC_PROTO, VAR, DECL_TAG and TYPE_TAG.
@@ -92,7 +92,9 @@ enum {
 	BTF_KIND_DECL_TAG	= 17,	/* Decl Tag */
 	BTF_KIND_TYPE_TAG	= 18,	/* Type Tag */
 	BTF_KIND_ENUM64		= 19,	/* Enumeration up to 64-bit values */
-
+	BTF_KIND_LOC_PARAM	= 20,	/* Location parameter information */
+	BTF_KIND_LOC_PROTO	= 21,	/* Location prototype for site */
+	BTF_KIND_LOCSEC		= 22,	/* Location section */
 	NR_BTF_KINDS,
 	BTF_KIND_MAX		= NR_BTF_KINDS - 1,
 };
@@ -212,4 +214,79 @@ struct btf_enum64 {
 	__u32	val_hi32;
 };
 
+/*
+ * BTF_KIND_LOC_PARAM is followed by a single "struct btf_loc_param"
+ * that contains flags specifying the contents of the vlen-specified
+ * number of 4-byte values that follow.
+ */
+struct btf_loc_param {
+	__u32 flags;
+	__u32 values[];
+};
+
+/*
+ * The combination of size, vlen and flags gives us the means to interpret
+ * the following vlen-specified set of 4-byte values:
+ *
+ * - a BTF_LOC_PARAM_CONST is a constant value; combination
+ *   of size, vlen and _SIGNED flag determines it. If the value requires
+ *   64 bits it is stored in {lo,hi} order.
+ * - a BTF_LOC_PARAM_ADDR|BTF_LOC_PARAM_CONST is an address that should be
+ *   normalized with respect to kernel/module base address.
+ * - a BTF_LOC_PARAM_REG with vlen 1 is a simple register number;
+ *   with vlen 2 it is a multi-register parameter.  Register numbers are
+ *   numbers derived from DW_OP_reg values, i.e. 0 is DW_OP_reg0; since
+ *   DW_OP_fbreg has its own special DW_OP_value and DW_OP_regx can refer
+ *   to an arbitrary register number, we reserve BTF_LOC_PARAM_FBREG for
+ *   the frame base register to avoid collisions.
+ * - a _REG | OFFSET describes an address without dereferencing it.
+ * - a _REG | DEREF with vlen 1 dereferences the value in the register
+ *   number specified.
+ * - a REG | DEREF | OFFSET with vlen > 1 specifies the register number
+ *   in the first 4-byte value and the offset in the remainder.
+ *   In the case of REG and OFFSET combinations, the OFFSET has the width
+ *   of the value words while the type size describes the represented parameter,
+ *   so for example a REG | DEREF | OFFSET with size 8 and vlen 2 would be
+ *   an 8-byte register dereference with signed 4-byte offset, since the vlen 2
+ *   values consist of a register value and the signed value.
+ */
+enum btf_loc_param_flags {
+	BTF_LOC_PARAM_SIGNED		=	0x1,
+	BTF_LOC_PARAM_CONST		=	0x2,
+	BTF_LOC_PARAM_ADDR		=	0x4,
+	BTF_LOC_PARAM_REG		=	0x8,
+	BTF_LOC_PARAM_DEREF		=	0x10,
+	BTF_LOC_PARAM_OFFSET		=	0x20,
+};
+
+enum {
+	BTF_LOC_PARAM_FBREG		=	0xffffffff
+};
+
+/*
+ * BTF_KIND_LOC_PROTO specifies location prototypes; i.e. how locations relate
+ * to parameters; a struct btf_type of BTF_KIND_LOC_PROTO is followed by a
+ * vlen-specified number of __u32 BTF type ids which specify the associated
+ * BTF_KIND_LOC_PARAM for each function parameter associated with the
+ * location.  The type should either be 0 (no location info) or point at
+ * a BTF_KIND_LOC_PARAM.
+ */
+
+/*
+ * BTF_KIND_LOCSEC consists of vlen-specified number of "struct btf_loc"
+ * containing location site-specific information for a specific ELF section;
+ * for example locations in ".text" are in a LOCSEC named "inline.text".
+ *
+ * - function (func)
+ * - location prototype type id (loc_proto)
+ * - address offset (offset) relative to the runtime base address of the
+ *   ELF section associated with the LOCSEC
+ */
+
+struct btf_loc {
+	__u32 func;
+	__u32 loc_proto;
+	__u32 offset;
+};
+
 #endif /* _UAPI__LINUX_BTF_H__ */
diff --git a/include/uapi/linux/keyctl.h b/include/uapi/linux/keyctl.h
index 4c8884eea808..fa85b9760391 100644
--- a/include/uapi/linux/keyctl.h
+++ b/include/uapi/linux/keyctl.h
@@ -24,6 +24,7 @@
 #define KEY_SPEC_GROUP_KEYRING		-6	/* - key ID for GID-specific keyring */
 #define KEY_SPEC_REQKEY_AUTH_KEY	-7	/* - key ID for assumed request_key auth key */
 #define KEY_SPEC_REQUESTOR_KEYRING	-8	/* - key ID for request_key() dest keyring */
+#define KEY_SPEC_BPF_KEYRING		-9	/* - key ID for the BPF-specific keyring */
 
 /* request-key default keyrings */
 #define KEY_REQKEY_DEFL_NO_CHANGE		-1