From 14eba6d98cabe625218fae0bfed8959d97226bc3 Mon Sep 17 00:00:00 2001 From: Tao Cui Date: Tue, 25 Aug 2026 10:35:57 +0800 Subject: sched_ext: Rename tg->scx.idle to tg->scx.sched_idle A bare "idle" reads as CPU idle state in sched_ext (ops.update_idle(), idle cpumasks, scx_bpf_pick_idle_cpu()). cpu.idle is the cgroup analog of the SCHED_IDLE policy, so name the task_group field sched_idle to match scx_cgroup_init_args.sched_idle. Pure rename, no behavior change. Signed-off-by: Tao Cui Reviewed-by: Andrea Righi Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 582d7cd4a983..7b9cdb65deef 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -323,7 +323,7 @@ struct scx_task_group { u64 bw_period_us; u64 bw_quota_us; u64 bw_burst_us; - bool idle; + bool sched_idle; #endif }; -- cgit v1.2.3 From 92344fad5b54d1c20ea19f882a4f086ca68c11c0 Mon Sep 17 00:00:00 2001 From: fangqiurong Date: Tue, 1 Sep 2026 09:32:58 +0800 Subject: sched_ext: Skip per-CPU data allocation for built-in DSQs Only user DSQs use the per-DSQ deferred reenqueue state: schedule_dsq_reenq() serves the local DSQ through sch->pcpu->deferred_reenq_local and rejects all other built-in DSQ ids. Every DSQ, built-in or not, still allocates a struct scx_dsq_pcpu per possible CPU. As the local, reject, rescue and bypass DSQs are themselves per-CPU, the wasted memory grows with the square of the CPU count, most of it at boot before any scheduler is loaded. Skip the allocation for built-in DSQ ids. While at it, rename ->pcpu to ->pcpu_user so that the per-CPU data reads as user-DSQ only. Signed-off-by: fangqiurong Reviewed-by: Zhan Xusheng Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 7b9cdb65deef..8de69843c215 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -91,7 +91,7 @@ struct scx_dispatch_q { struct rhash_head hash_node; struct llist_node free_node; struct scx_sched *sched; - struct scx_dsq_pcpu __percpu *pcpu; + struct scx_dsq_pcpu __percpu *pcpu_user; struct rcu_head rcu; }; -- cgit v1.2.3 From f8e5a4e3f3bea6b389ae8dbeafe50fdb8ab543b2 Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Fri, 18 Sep 2026 19:13:27 +0200 Subject: sched_ext: Add lazy preemption support The fair scheduling class can request lazy rescheduling, deferring an in-kernel scheduling boundary until returning to user space or until the next scheduler tick. sched_ext only exposes immediate preemption, preventing BPF schedulers from making the same trade-off. Add SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY. Both expire the current sched_ext task slice but request lazy rescheduling. Immediate preemption, WAIT and plain kicks take precedence when requests are combined, while a lazy enqueue to a non-local DSQ retains the head-insertion semantics of SCX_ENQ_PREEMPT. Add SCX_OPS_LAZY_RESCHED as the default slice-expiry policy for newly enabled tasks and initialize it before ops.enable(). Add scx_bpf_task_set_lazy_resched() so the owning scheduler can override the persistent policy per task from any callback while preserving sub-scheduler task ownership boundaries. Bypass continues to force immediate rescheduling. For an infinite-slice task whose tick is stopped, set the lazy reschedule request before restoring the scheduler tick dependency. This lets the dependency update's IPI serve the request on return to user space and guarantees that the next tick promotes it otherwise. Use pinned rq locking for remote kick delivery so RQCF_UPDATED reflects clock updates since lock acquisition. Accumulate kick requests independently and resolve precedence while holding the target rq lock. Reject invalid SCX_KICK_IDLE combinations and unknown kick flags, which were previously silently ignored. Tested-by: Tao Cui Signed-off-by: Andrea Righi Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 8de69843c215..1e1fc3312bc4 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -278,6 +278,14 @@ struct sched_ext_entity { */ bool disallow; /* reject switching into SCX */ + /* + * If set, depletion of this task's slice at the scheduler tick requests + * lazy instead of immediate rescheduling. Initialized from + * %SCX_OPS_LAZY_RESCHED immediately before ops.enable() and may be + * modified afterwards with scx_bpf_task_set_lazy_resched(). + */ + bool lazy_resched; + /* cold fields */ #ifdef CONFIG_EXT_GROUP_SCHED struct cgroup *cgrp_moving_from; -- cgit v1.2.3 From a87743bd9e9bef4b69d6fcce992cdaaa8edb78e3 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Thu, 24 Sep 2026 09:14:46 -0700 Subject: sched_ext: Place dsq_vtime next to dsq_priq scx_dispatch_enqueue() calls rb_add() for a vtime-ordered DSQ while holding dsq->lock. At each level of the descent, scx_dsq_priq_less() loads the visited task's dsq_vtime, and that comparison selects rb_left or rb_right from the same task's dsq_priq. In the x86-64 benchmark configuration, these accesses fell on different 64-byte cache lines before this change, so a cold visited task could require a second cache line fill on the dependent descent path. Move dsq_vtime immediately before dsq_priq, keeping dsq_seq and dsq_flags next to dsq_list. In the two x86-64 layouts inspected, where scx starts at offsets 776 and 840 in task_struct, the fields from dsq_list through dsq_priq occupy one cache line. The exact cache line placement depends on the containing task_struct layout and is not guaranteed for every configuration or architecture. Where they share a line, avoiding the dependent cache line fill helps reduce dsq->lock hold time during vtime insertion and therefore contention on the lock. In a 16-vCPU KVM guest, the instrumented enqueue interval was 4-6% shorter with scx_lavd and scx_layered. scx_mitosis and end-to-end workload time showed no consistent change. This only reorders fields; no scheduling behavior change is intended. Signed-off-by: Usama Arif Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 1e1fc3312bc4..b85676871624 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -197,9 +197,19 @@ struct sched_ext_entity { u64 ddsp_slice; u64 ddsp_vtime; struct scx_dsq_list_node dsq_list; /* dispatch order */ - struct rb_node dsq_priq; /* p->scx.dsq_vtime order */ u32 dsq_seq; u32 dsq_flags; /* protected by DSQ lock */ + + /* + * Used to order tasks when dispatching to the vtime-ordered priority + * queue of a dsq. This is usually set through + * scx_bpf_dsq_insert_vtime() but can also be modified directly by the + * BPF scheduler. Modifying it while a task is queued on a dsq may + * mangle the ordering and is not recommended. Kept next to @dsq_priq + * as rbtree insertion reads both on every visited node. + */ + u64 dsq_vtime; + struct rb_node dsq_priq; /* p->scx.dsq_vtime order */ u32 flags; /* protected by rq lock */ u32 weight; u32 reenq_cnt; /* reenqueues since last run */ @@ -225,7 +235,7 @@ struct sched_ext_entity { u64 tid; struct rhash_head tid_hash_node; /* see SCX_OPS_TID_TO_TASK */ - /* BPF scheduler modifiable fields */ + /* BPF scheduler modifiable fields, along with @dsq_vtime above */ /* * Runtime budget in nsecs - how long the task may hold its cpu. Owned @@ -240,15 +250,6 @@ struct sched_ext_entity { */ u64 slice; - /* - * Used to order tasks when dispatching to the vtime-ordered priority - * queue of a dsq. This is usually set through - * scx_bpf_dsq_insert_vtime() but can also be modified directly by the - * BPF scheduler. Modifying it while a task is queued on a dsq may - * mangle the ordering and is not recommended. - */ - u64 dsq_vtime; - /* * Out-of-band slice request from scx_bpf_task_set_slice() when the * caller does not hold the rq lock, applied under the rq lock at the -- cgit v1.2.3 From 5f882051efabaf19f9b4dbe4b50cee1d250ac19e Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Tue, 22 Sep 2026 18:51:46 +0200 Subject: sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors With proxy execution, pick_next_task() can select a blocked task as the scheduling context before find_proxy_task() resolves the execution context. From the BPF scheduler perspective, that donor is running while its scheduling context drives the lock owner; ops.tick() and other accounting must therefore remain enclosed by a matching ops.running()/ops.stopping() session. In this scenario, the session boundaries do not always match physical task switches. Keep the "running" session open when the same donor continues on the same CPU. When proxy execution migrates a donor's scheduling context to another CPU, end its running session on the source CPU and start a new session on the destination CPU only after proxy resolution succeeds. This prevents a failed resolution from exposing a provisional ops.running() event. Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit running-state tracking is also required by later donor-based accounting: it prevents an EXT owner executing for a non-EXT donor from being treated as the active EXT scheduling context when it is dequeued. This is a preparatory change for enabling proxy execution together with sched_ext. Acked-by: John Stultz Signed-off-by: Andrea Righi Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 1 + 1 file changed, 1 insertion(+) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index b85676871624..77bc767fdf30 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -104,6 +104,7 @@ enum scx_ent_flags { SCX_TASK_SUB_INIT = 1 << 4, /* task being initialized for a sub sched */ SCX_TASK_IMMED = 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */ SCX_TASK_PROTECTED = 1 << 6, /* slice and DSQ head position protected */ + SCX_TASK_RUN_TRACKED = 1 << 7, /* task is in an ops.running()/stopping() session */ /* * Bits 8 to 10 are used to carry task state: -- cgit v1.2.3 From 13ae184e4b8279dcc7b84bf58e987a60d9aad69a Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Tue, 22 Sep 2026 18:51:49 +0200 Subject: sched_ext: Handle proxy-exec races in remote DSQ transfers Without proxy execution, the DSQ lock and holding_cpu handshake ensure that a task cannot be dequeued or start running during an rq-lock handoff without clearing holding_cpu. Proxy execution is an exception: a task can start physically executing as a lock owner while its scheduling context remains on a DSQ; its on-CPU or migration-disabled state can therefore change without clearing holding_cpu. Recheck these states after acquiring the source rq lock. If the transfer can no longer proceed, park the task on the source rq's reject DSQ and reenqueue it through its owning scheduler. This preserves the BPF scheduler's placement policy and keeps descendant tasks within their sub-scheduler's cap grants. Queue a deferred drain for every rejection. If a drain finds a proxy-rejected task still running or donating, leave it parked and record a retry request on the rq. scx_proxy_reenqueue_retry() consumes the request and schedules another drain when proxy state changes. Without this change and proxy execution enabled, stress-ng --pipeherd can trigger this race and migrate an active execution context, leading to sleeping-while-atomic warnings and subsequent lockdep corruption. This is a preparatory change to support proxy execution with sched_ext. Suggested-by: Tejun Heo Signed-off-by: Andrea Righi Signed-off-by: Tejun Heo --- include/linux/sched/ext.h | 2 ++ 1 file changed, 2 insertions(+) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index 77bc767fdf30..23f9e178bc5a 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -137,6 +137,7 @@ enum scx_ent_flags { * IMMED reenqueued due to failed ENQ_IMMED * PREEMPTED preempted while running * CAP sub-sched cap miss, see p->scx.reenq_reason_* + * PROXY proxy state prevented a remote DSQ transfer */ SCX_TASK_REENQ_REASON_SHIFT = 12, SCX_TASK_REENQ_REASON_BITS = 3, @@ -147,6 +148,7 @@ enum scx_ent_flags { SCX_TASK_REENQ_IMMED = 2 << SCX_TASK_REENQ_REASON_SHIFT, SCX_TASK_REENQ_PREEMPTED = 3 << SCX_TASK_REENQ_REASON_SHIFT, SCX_TASK_REENQ_CAP = 4 << SCX_TASK_REENQ_REASON_SHIFT, + SCX_TASK_REENQ_PROXY = 5 << SCX_TASK_REENQ_REASON_SHIFT, /* iteration cursor, not a task */ SCX_TASK_CURSOR = 1 << 31, -- cgit v1.2.3 From b3768d058c98d4e8114a246d3267d7dae1c123fb Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 17 Oct 2026 13:09:28 -1000 Subject: sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only the stranded tasks placed again. Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ like scx_bpf_dsq_move() and is callable from the same contexts. The mark is cleared when the task leaves the DSQ, so a task consumed before the walk is not re-enqueued. An iteration's DSQ can be destroyed before the iteration ends, so schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same path. Declare the kfunc for the tools and add a compat probe. v2: Note in the kerneldoc that another iteration's re-enqueue may take a marked task early, and that a task on another scheduler returns false (Andrea). Signed-off-by: Tejun Heo Reviewed-by: Andrea Righi --- include/linux/sched/ext.h | 1 + 1 file changed, 1 insertion(+) (limited to 'include/linux') diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h index ae696d11dac9..efd635a99309 100644 --- a/include/linux/sched/ext.h +++ b/include/linux/sched/ext.h @@ -157,6 +157,7 @@ enum scx_ent_flags { /* scx_entity.dsq_flags */ enum scx_ent_dsq_flags { SCX_TASK_DSQ_ON_PRIQ = 1 << 0, /* task is queued on the priority queue of a dsq */ + SCX_TASK_DSQ_REENQ = 1 << 1, /* marked by scx_bpf_dsq_reenq_task() */ }; enum scx_dsq_lnode_flags { -- cgit v1.2.3