summaryrefslogtreecommitdiff
path: root/kernel/sched
AgeCommit message (Collapse)Author
5 daysMerge tag 'sched_ext-for-7.4-rc2-fixes' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - With proxy execution or core scheduling, a CPU could run a task that had been requeued on another CPU while dispatch had the rq lock dropped. - Unregistering a scheduler could return while its teardown was still running, letting the teardown call into BPF code that was already freed. - A CPU coming online could call into the BPF scheduler before the scheduler was told that the CPU is online. - A parent scheduler could be asked to attach a sub-scheduler to a cgroup that was already removed. - Hardening: more kfuncs that take a queue lock are rejected in tracing programs, where they could deadlock the CPU, and a queue iterator no longer restarts after reaching the end, which the BPF verifier assumes can't happen. - Smaller fixes: tasks needlessly sent back to the BPF scheduler on idle CPUs, stale scheduling state carried into a task's next scheduler, and uneven CPU shards with partially populated SMT cores. * tag 'sched_ext-for-7.4-rc2-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Reject DSQ iterator kfuncs in tracing programs sched_ext: make DSQ iterator stay drained after NULL sched_ext: Fix partial-SMT core counting in calc_shard_layout() sched_ext: Reset task fields on exit of an SCX_TASK_READY task sched_ext: Read exit_kind before flushing in scx_flush_disable_work() sched_ext: Reject scx_bpf_destroy_dsq() in tracing programs sched_ext: Fix ops.enqueue/dispatch() calls before ops.cpu_online() sched_ext: Factor out drain_offline_rq() sched_ext: Fix stale pick when the donor changes during dispatch sched_ext: Call ops.sub_attach() only while the cgroup is online sched_ext: Fix scx_bpf_dsq_reenq() kerneldoc claiming NMI calls are allowed sched_ext: Avoid unnecessary IMMED reenqueues on idle CPUs
9 dayssched_ext: Reject DSQ iterator kfuncs in tracing programsTao Cui
bpf_iter_scx_dsq_next() and bpf_iter_scx_dsq_destroy() take the target user DSQ's raw spinlock, but they are in scx_kfunc_ids_any, which is also registered for BPF_PROG_TYPE_TRACING. For an active root scheduler without ops.sub_attach, the compatibility path in scx_prog_sched() lets an unassociated tracing program resolve to that scheduler. The tracing program can then call the iterator kfuncs while the target DSQ lock is already held. For example, a scheduler which mixes FIFO and priority-queue insertions into a user DSQ triggers scx_error() from scx_dispatch_enqueue() while holding that DSQ's lock. scx_claim_exit() emits sched_ext_exit before the lock is released: scx_dispatch_enqueue() raw_spin_lock(&dsq->lock) scx_error() scx_claim_exit() trace_sched_ext_exit() tracing BPF program bpf_iter_scx_dsq_new(dsq_id) bpf_for_each(scx_dsq, p, dsq_id, 0) bpf_iter_scx_dsq_next(it) raw_spin_lock_irqsave(&dsq->lock) The tracing program recursively acquires the same raw spinlock and locks up the CPU. Because the tracepoint runs before scx_finish_exit(), disable_irq_work is not queued and the scheduler cannot enter its normal disable path. Reject the two iterator kfuncs from tracing programs at verification time. The kfuncs remain available to SCX struct_ops and syscall programs. bpf_iter_scx_dsq_destroy() has no independent live reproduction: its cursor only links into the DSQ after at least one successful next(), which deadlocks in the same window first. It is included for symmetry. This extends the coverage of commit b07a7450f6ef ("sched_ext: Reject rq-lock-taking kfuncs in tracing programs") and commit 4c2e0751d5e6 ("sched_ext: Reject scx_bpf_destroy_dsq() in tracing programs"). Fixes: 650ba21b131e ("sched_ext: Implement DSQ iterator") Signed-off-by: Tao Cui <cuitao@kylinos.cn> Signed-off-by: Tejun Heo <tj@kernel.org>
11 dayssched_ext: make DSQ iterator stay drained after NULLKang Chen
nldsq_cursor_next_task() unlinks the iteration cursor when it reaches the end of the DSQ. A later call sees list_empty(&cursor->node), starts from the DSQ head again, and returns the first task of the same iteration. The verifier relies on the iterator contract that once next() returns NULL it keeps returning NULL: process_iter_next_call() marks the iterator DRAINED and treats the result of later next() calls as the constant zero. A restarted DSQ iteration therefore hands the BPF program a task pointer while the verifier believes R0 == 0. Remember the drained state in the cursor flags and return NULL on every later next() call. The cursor has spare flag bits and the BPF-visible iterator layout is unchanged. Fixes: 650ba21b131e ("sched_ext: Implement DSQ iterator") Signed-off-by: Kang Chen <k.chen@smail.nju.edu.cn> Signed-off-by: Tejun Heo <tj@kernel.org>
11 dayssched_ext: Fix partial-SMT core counting in calc_shard_layout()Zhang Qiao
calc_shard_layout() counts the cores of an LLC to spread them evenly across shards. A core is counted when its cpu is the first cpu of topology_sibling_cpumask(), but the cores scx_cid_init() builds are the siblings intersected with @llc_cpus. The sibling mask may contain cpus outside @llc_cpus - siblings which landed in another node or LLC unit, or offlined siblings on archs that keep them in the mask (mips, sparc64). A core whose first sibling is such a cpu is never counted, so cores_per_shard and nr_large come out too small and the shards end up uneven. Count a core at its first sibling in @llc_cpus, matching the cores the build loop constructs. Fixes: e2c841b91bf9 ("sched_ext: Add CID sharding") Signed-off-by: Zhang Qiao <zhangqiao22@huawei.com> Signed-off-by: Tejun Heo <tj@kernel.org>
12 dayssched/core: Do not touch SMT siblings that have not joined the core yetSeiji Nishikawa
When a CPU comes online, it is added to cpu_smt_mask() of its siblings before sched_core_cpu_starting() sets its rq->core to the core leader. For example, on x86, ap_starting() calls set_cpu_sibling_map() first, and calls notify_cpu_starting() after that. This order is intended. Before sched_core_cpu_starting() runs, rq->core of the new CPU still points to its own rq. But rq->core_enabled of the new CPU is already set. __sched_core_flip() sets it for all possible CPUs, also for offline CPUs, and CPU offline does not clear it. So __rq_lockp() of the new CPU returns its own rq->__lock. It does not return the core wide lock. If a sibling CPU runs pick_next_task() for SCHED_CORE at this time, it checks all CPUs in cpu_smt_mask(). It calls update_rq_clock() and pick_task() for the rq of the new CPU. But it holds only its own core lock, not the lock of the new CPU. Then lockdep shows this warning in update_rq_clock(). debug_locks && !(lock_is_held(&(__rq_lockp(rq))->dep_map) != 0) WARNING: kernel/sched/sched.h:1666 at update_rq_clock+0x40a/0xd20 kernel/sched/core.c:875, CPU#0: syz-executor251/5712 Call Trace: <TASK> pick_next_task kernel/sched/core.c:6375 [inline] __schedule+0x1c69/0x6920 kernel/sched/core.c:7199 preempt_schedule_irq+0x4e/0x90 kernel/sched/core.c:7606 The next loops in pick_next_task() also change rq_i->core_pick of this rq, and can call resched_curr() for it, without its lock. resched_curr() has the same lockdep_assert_rq_held() check. __sched_core_account_forceidle() also checks all CPUs in the same mask. CPU offline does not have this problem. remove_siblinginfo() and sched_core_cpu_dying() both run in take_cpu_down() under stop_machine. So no sibling CPU can be in pick_next_task() at that time. Fix this. In these loops, skip a sibling CPU if its rq->core is different from our rq->core. At run time, rq->core of a CPU is changed by sched_core_cpu_starting(), sched_core_cpu_deactivate() and sched_core_cpu_dying(). The first two hold sched_core_lock(), and sched_core_lock() takes the rq locks of all CPUs in the SMT mask. The last one runs under stop_machine, as written above. So while we hold our core lock, the result of this check does not change. A CPU that is not in the core yet will pick its own tasks. The comment in the reschedule loop of pick_next_task() already expects this case. I tested this with the syzbot C reproducer in QEMU. The two vCPUs were SMT siblings (-smp 2,sockets=1,cores=1,threads=2). Without this patch, the warning came after about 100 seconds. Just before the warning, a debug print showed that the loop on CPU 0 saw CPU 1 with rq->core pointing to CPU 1 itself, and cpu_online() was false for CPU 1. With this patch, the same state still happened, but there was no warning in more than 2 hours. Fixes: 3c474b3239f1 ("sched: Fix Core-wide rq->lock for uninitialized CPUs") Closes: https://syzkaller.appspot.com/bug?extid=681e729b2cf0d4860042 Reported-by: syzbot+681e729b2cf0d4860042@syzkaller.appspotmail.com Signed-off-by: Seiji Nishikawa <snishika@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20261104022004.3810619-1-snishika@redhat.com
12 dayssched/cache: Don't use fits_capacity() to compare thread and CPU countsZhan Xusheng
invalid_llc_nr() uses fits_capacity() to compare a thread count with a CPU count, only for its 20% margin. Qais's series changes fits_capacity() to take a CPU number as its second argument, and this call would then pass the CPU count as a CPU number without a warning. Give the check its own helper. x * 5 < y * 4 is the same comparison as x * 1280 < y * 1024, so there is no functional change. Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Qais Yousef <qyousef@layalina.io> Reviewed-by: Chen Yu <yu.c.chen@intel.com> Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com> Link: https://lore.kernel.org/all/20260504020003.71306-5-qyousef@layalina.io/ Link: https://patch.msgid.link/20261102021536.1026143-1-zhanxusheng1024@gmail.com
12 dayssched/core: Fix cpuset_mutex double unlock in __sched_setscheduler()Andrea Parri
A sched_setattr() with SCHED_FLAG_KEEP_POLICY on a SCHED_DEADLINE task unlocks cpuset_mutex twice if the task's policy changes concurrently: WARNING: bad unlock balance detected! 7.4.0-rc1 #1 Not tainted ------------------------------------- repro/126 is trying to release lock (cpuset_mutex) at: [<ffffffff9a17cacc>] __sched_setscheduler+0x9cc/0xc40 but there are no more locks to release! ... Call Trace: print_unlock_imbalance_bug.part.0+0xfb/0x110 lock_release+0x286/0x2e0 __mutex_unlock_slowpath+0x1a/0x400 __sched_setscheduler+0x9cc/0xc40 __x64_sys_sched_setattr+0xa3/0x1b0 do_syscall_64+0x106/0x5f0 entry_SYSCALL_64_after_hwframe+0x77/0x7f __sched_setscheduler() samples p->policy before taking any lock, and restarts from the recheck label if p->policy differs under the rq lock. The restart drops cpuset_mutex but leaves cpuset_locked set, so a pass that no longer needs the mutex still unlocks it on its way out: CPU 1 CPU 2 __sched_setscheduler(KEEP_POLICY) policy = p->policy /* DEADLINE */ __sched_setscheduler(SCHED_OTHER) p->policy = SCHED_OTHER cpuset_locked = true cpuset_lock() rq = task_rq_lock(p) oldpolicy != p->policy cpuset_unlock() goto recheck /* no cpuset_lock() this time */ ... if (cpuset_locked) cpuset_unlock() Without CONFIG_DEBUG_MUTEXES, or CONFIG_DEBUG_RT_MUTEXES on PREEMPT_RT, nothing warns. The second unlock still clears the owner field regardless of who holds the mutex, so another task can lose cpuset_mutex inside its critical section. Entering SCHED_DEADLINE needs CAP_SYS_NICE, but a racing caller that owns the task does not need it if it samples a policy other than SCHED_DEADLINE. Clear cpuset_locked when dropping the mutex for the restart. This should be safe because the flag is only tested on the paths leaving the next pass, after that pass has set it again if it took the mutex. The following program, run as root, reproduces the splat, which no longer appears with this applied. On PREEMPT_RT it crashes the kernel within seconds, which this also fixes: #define _GNU_SOURCE #include <pthread.h> #include <stdint.h> #include <sys/syscall.h> #include <unistd.h> struct attr { uint32_t size, policy; uint64_t flags; int32_t nice; uint32_t prio; uint64_t runtime, deadline, period; }; static volatile int stop; static volatile pid_t target; static void set(struct attr a) { a.size = sizeof(a); syscall(SYS_sched_setattr, target, &a, 0); } static void *sleeper(void *arg) { target = syscall(SYS_gettid); while (!stop) usleep(1000); return arg; } static void *switcher(void *arg) { while (!stop) { /* SCHED_DEADLINE, 2us every 1s */ set((struct attr){ .policy = 6, .runtime = 2000, .deadline = 1000000000, .period = 1000000000 }); set((struct attr){ .policy = 0 }); /* SCHED_OTHER */ } return arg; } static void *racer(void *arg) { /* KEEP_POLICY | KEEP_PARAMS */ while (!stop) set((struct attr){ .flags = 0x18 }); return arg; } int main(void) { pthread_t t[5]; int i; pthread_create(&t[0], NULL, sleeper, NULL); while (!target) usleep(100); pthread_create(&t[1], NULL, switcher, NULL); for (i = 2; i < 5; i++) pthread_create(&t[i], NULL, racer, NULL); sleep(30); stop = 1; for (i = 0; i < 5; i++) pthread_join(t[i], NULL); return 0; } An LLM helped find the PREEMPT_RT crash and write this patch and its test. Fixes: 111cd11bbc54 ("sched/cpuset: Bring back cpuset_mutex") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Andrea Parri <parri.andrea@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Waiman Long <longman@redhat.com> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Reviewed-by: Juri Lelli <juri.lelli@redhat.com> Cc: stable@vger.kernel.org Link: https://sashiko.dev/#/patchset/20261016062629.2102883-1-sh_def%40163.com Link: https://patch.msgid.link/20261102134606.242743-1-parri.andrea@gmail.com
12 dayssched/uclamp: Reject clamp values above SCHED_CAPACITY_SCALEAndrea Parri
sched_setattr() accepts sched_util_min and sched_util_max values that do not fit in uclamp_se::value. A single thread can do: sched_setattr(max = 800) returns 0 sched_setattr(min = 0xfffffffe) returns 0 sched_getattr() min=2046 max=800 which leaves the minimum above both SCHED_CAPACITY_SCALE and the task's own maximum. The range check in uclamp_validate() relies on the + 1 wrapping -1, the reset value, to 0: if (util_min + 1 > SCHED_CAPACITY_SCALE + 1) That only works for an unsigned type. util_min and util_max are int and SCHED_CAPACITY_SCALE is a long, so every negative value passes the check, and so does INT_MAX, whose + 1 wraps. uclamp_se_set() then truncates what got through to the 11 bits of uclamp_se::value, but computes the bucket from the full value. A minimum of 0xfffff800 is stored as 0 in the top bucket, where uclamp_rq_max_value() takes it for the highest active clamp once a higher value leaves the rq. The rq clamp falls below what the remaining tasks requested, and emptying their bucket trips the WARN_ON_ONCE() in uclamp_rq_dec_id(), without any privilege: WARNING: kernel/sched/core.c:1862 at dequeue_task+0x2a3/0x2b0, CPU#0: repro/126 Call Trace: __schedule+0x933/0x1320 schedule+0x36/0x100 do_nanosleep+0x61/0x190 hrtimer_nanosleep+0xb5/0x1d0 Make util_min and util_max unsigned int, so that -1 is the only value that wraps. The pair check below is unaffected, since it skips -1 explicitly and anything else is at most SCHED_CAPACITY_SCALE by then. Requests that were only accepted because of this now fail with -EINVAL. The following program triggers the warning when run by any user outside a child cpu cgroup. With this applied, its 0xfffff800 request fails with -EINVAL and the warning no longer appears: #define _GNU_SOURCE #include <pthread.h> #include <sched.h> #include <stdint.h> #include <sys/syscall.h> #include <time.h> #include <unistd.h> struct attr { uint32_t size, policy; uint64_t flags; int32_t nice; uint32_t prio; uint64_t runtime, deadline, period; uint32_t util_min, util_max; }; static volatile int stop; static void set_min(uint32_t v) { /* KEEP_POLICY | KEEP_PARAMS | UTIL_CLAMP_MIN */ struct attr a = { .size = sizeof(a), .flags = 0x38, .util_min = v }; syscall(SYS_sched_setattr, 0, &a, 0); } static void burn(long us) { struct timespec s, n; clock_gettime(CLOCK_MONOTONIC, &s); do clock_gettime(CLOCK_MONOTONIC, &n); while ((n.tv_sec - s.tv_sec) * 1000000 + (n.tv_nsec - s.tv_nsec) / 1000 < us); } static void *bogus(void *arg) { set_min(0xfffff800); while (!stop) burn(1000); return arg; } static void *worker(void *arg) { long v = (long)arg; set_min(v); while (!stop) { burn(v == 800 ? 300 : 1500); usleep(100); } return arg; } int main(void) { pthread_t t[3]; cpu_set_t c; int i; CPU_ZERO(&c); CPU_SET(0, &c); sched_setaffinity(0, sizeof(c), &c); pthread_create(&t[0], NULL, bogus, NULL); usleep(50000); pthread_create(&t[1], NULL, worker, (void *)500L); pthread_create(&t[2], NULL, worker, (void *)800L); sleep(15); stop = 1; for (i = 0; i < 3; i++) pthread_join(t[i], NULL); return 0; } An LLM helped find the warning and write this patch and its tests. Fixes: 480a6ca2dc6e ("sched/uclamp: Allow to reset a task uclamp constraint value") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Andrea Parri <parri.andrea@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Cc: stable@vger.kernel.org Link: https://sashiko.dev/#/patchset/20261016062629.2102883-1-sh_def%40163.com Link: https://patch.msgid.link/20261102134831.243255-1-parri.andrea@gmail.com
12 dayssched/deadline: Fix spurious -EPERM for migration-disabled tasksAndrea Parri
sched_setattr() can refuse to make a task SCHED_DEADLINE with -EPERM, even though the task's affinity covers the whole root domain. A retry can succeed, but the error blames an affinity the task does not have. __sched_setscheduler() tests p->cpus_ptr, which is not always the task's affinity. When a task is scheduled out inside a migrate_disable() section, migrate_disable_switch() points p->cpus_ptr at the current CPU until migrate_enable(). On PREEMPT_RT every spinlock_t holder is in such a section, so ordinary kernel work opens the window. Elsewhere, seccomp filters run in one. Test p->cpus_mask instead, which keeps the task's affinity and is what sched_getaffinity() reports. This should be safe because an admitted task's cpus_mask covers the root domain, so set_cpus_allowed_dl() moves no bandwidth when migrate_enable() restores p->cpus_ptr. The following program, run as root, had 553679, 453720 and 424623 of its 1000000 requests refused on three boots of a 4-CPU PREEMPT_RT guest, and 236182, 217174 and 237183 without PREEMPT_RT. With this applied the count is 0, also with lockdep, and a task pinned to one CPU is still refused: #define _GNU_SOURCE #include <errno.h> #include <linux/filter.h> #include <linux/seccomp.h> #include <pthread.h> #include <stddef.h> #include <stdint.h> #include <stdio.h> #include <sys/syscall.h> #include <unistd.h> #define REQUESTS 1000000 #define INSNS 300 struct attr { uint32_t size, policy; uint64_t flags; int32_t nice; uint32_t prio; uint64_t runtime, deadline, period; }; static volatile int stop; static volatile pid_t target; static long set(struct attr a) { a.size = sizeof(a); return syscall(SYS_sched_setattr, target, &a, 0); } static void *worker(void *arg) { struct sock_filter f[INSNS + 1]; struct sock_fprog prog = { .len = INSNS + 1, .filter = f }; int i; /* Reading the arguments keeps seccomp from caching the result */ for (i = 0; i < INSNS; i++) f[i] = (struct sock_filter) BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, args)); f[INSNS] = (struct sock_filter) BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW); syscall(SYS_seccomp, SECCOMP_SET_MODE_FILTER, 0, &prog); target = syscall(SYS_gettid); while (!stop) syscall(SYS_getppid); return arg; } static void *hog(void *arg) { while (!stop) ; return arg; } int main(void) { pthread_t t[9]; long refused = 0; int i; pthread_create(&t[0], NULL, worker, NULL); while (!target) usleep(100); for (i = 1; i < 9; i++) pthread_create(&t[i], NULL, hog, NULL); for (i = 0; i < REQUESTS; i++) { /* SCHED_DEADLINE, 2us every 1s */ if (set((struct attr){ .policy = 6, .runtime = 2000, .deadline = 1000000000, .period = 1000000000 }) && errno == EPERM) refused++; set((struct attr){ .policy = 0 }); /* SCHED_OTHER */ } stop = 1; for (i = 0; i < 9; i++) pthread_join(t[i], NULL); printf("%ld of %d requests refused\n", refused, REQUESTS); return 0; } An LLM helped find this bug and write this patch and its test. Fixes: af449901b84c ("sched: Add migrate_disable()") Signed-off-by: Andrea Parri <parri.andrea@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Acked-by: Juri Lelli <juri.lelli@redhat.com> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20261102135420.244745-1-parri.andrea@gmail.com
2026-11-06sched_ext: Reset task fields on exit of an SCX_TASK_READY taskRahad Bhuiya
Tasks in SCX_TASK_READY are attached to a scheduler but outside the ext class, such as RT/DL tasks, or non-SCHED_EXT tasks under a partial switch. The scheduler's BPF code can still update their dsq_vtime and slice (for instance, via ops.cgroup_move() as done by scx_flatcg). Because __scx_disable_and_exit_task() currently breaks out of the state switch for SCX_TASK_READY without calling scx_reset_task_fields(), these fields are not cleared when the task leaves the scheduler. If the task is subsequently moved to another cgroup/sub-scheduler and later switches to SCHED_NORMAL, the new scheduler's ops.enable() sees stale dsq_vtime from the previous scheduler's clock. Call scx_reset_task_fields() in the SCX_TASK_READY case of __scx_disable_and_exit_task() to ensure SCX-managed fields are properly reset when leaving the scheduler. Fixes: 5771e79e461e ("sched_ext: Reset dsq_vtime and slice when a task leaves SCX") Signed-off-by: Rahad Bhuiya <rahadbhuiya2021@gmail.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-11-06sched/cache: Ignore non-fair current in alb_break_llc()Kayra Cizmeci
Commit d6b9afab44e23 ("sched/cache: Fix checking active load balance by only considering the CFS task") stopped using task_util() for non-fair cur. __active_load_balance_cpu_stop() calls detach_one_task(), that only looks to cfs_tasks list. So if the rq->curr is not fair, doing a decision according to it is wrong, since we only migrate tasks that is fair. Commit f0d243a96f26 ("sched/fair: Avoid creating misfits during cache-aware balancing") passes cur to task_misfits_asym_cpu() without checking if the cur is in fair class or not. Fix this. Fixes: f0d243a96f26 ("sched/fair: Avoid creating misfits during cache-aware balancing") Signed-off-by: Kayra Cizmeci <kayracizmeci@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com> Link: https://patch.msgid.link/20261020185402.57017-1-kayracizmeci@gmail.com
2026-11-06sched/topology: Balance cluster static-key updatesHui Su
build_sched_domains() increments sched_cluster_active once per newly built partition that contains an SD_CLUSTER domain. detach_destroy_domains() instead tested the global static key before decrementing it, so destroying a cluster-less partition could drop a reference owned by another partition and disable cluster-aware scheduling while SD_CLUSTER domains remained installed. Check the installed domains before detaching them, only while the key is enabled, and decrement only when the removed partition contains an SD_CLUSTER domain. With two cluster-bearing partitions installed, removing a singleton partition left the baseline count at 1 where 2 partitions still carried SD_CLUSTER; the fixed kernel correctly reports 2. On an asymmetric cluster topology where the parent partition gains no cluster domain, the baseline count drops to 0 while a cluster partition remains installed; the fixed kernel keeps it at 1. Fixes: 8881e1639f1f ("sched/fair: Scan cluster before scanning LLC in wake-up path") Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Link: https://patch.msgid.link/20261021043834.1140951-1-sh_def@163.com
2026-11-06sched/pelt: Fix u32 truncation of the PELT period countHui Su
accumulate_sum() computes the number of elapsed PELT periods in a u64 but returns u32. When the count is a multiple of 2^32, it is truncated to zero even though the PELT sums have been updated. The caller interprets zero as no period boundary crossed and skips recomputing the averages, leaving *_avg stale relative to *_sum. Return the period count as u64 so the caller can test it without truncation. Fixes: a481db34b9be ("sched/fair: Optimize ___update_sched_avg()") Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Link: https://patch.msgid.link/20261023025207.2967903-1-sh_def@163.com
2026-11-06sched/fair: Clear throttle state when leaving the fair classDaehyeon Ko
Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations. DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives. Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set. After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG(). Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq. A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash. An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions. Fixes: e1fad12dcb66 ("sched/fair: Switch to task based throttle model") Signed-off-by: Daehyeon Ko <4ncienth@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Aaron Lu <ziqianlu@bytedance.com> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20261022235234.150957-1-4ncienth@gmail.com
2026-11-06sched/core: Account I/O waits independently of delayed dequeueChristian Loehle
Starting blkio delay accounting and incrementing rq->nr_iowait in __block_task() ties both to removal from the runqueue. With EEVDF delayed dequeue, this misses the initial part of an I/O sleep, or the entire sleep if the task wakes before it is dequeued. Such tasks are also missing from procs_blocked while their dequeue is delayed. Final dequeue can run in another task's context, causing delayacct_blkio_start() to timestamp the wrong task. Start both forms of accounting in try_to_block_task(), when the current task commits to blocking, before delayed dequeue or proxy execution can retain it on the runqueue. End both when clearing is_blocked, including queued wakeups and the proxy execution paths that let the current task continue. Checking is_blocked excludes wakeups that precede blocking. Save the CPU charged at the start of the wait. Delayed tasks can migrate while still queued, and proxy execution can also move blocked tasks. Decrement the saved CPU's counter on unblock, regardless of the task's current placement, and remove the old dequeue-based accounting. CPU iowait continues to classify idle time using the counter on the charged CPU; it does not accumulate the full per-task blocked duration. Closes: https://lore.kernel.org/r/20261014081731.1068794-1-zhanxusheng@xiaomi.com Fixes: 152e11f6df29 ("sched/fair: Implement delayed dequeue") Reported-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Christian Loehle <christian.loehle@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Link: https://patch.msgid.link/fd8578a1-e60b-4993-ab51-2b5d63abcf77@arm.com
2026-11-06sched/deadline: Fix this_bw on class change of a dying taskNickolai Zeldovich
Moving a SCHED_DEADLINE task out of the class with sched_setscheduler(2) after it stored TASK_DEAD in do_task_dead(), but before __schedule() dequeued it, subtracts its bandwidth from the runqueue's this_bw twice. this_bw then stays short until reboot: __sub_rq_bw() warns about the underflow or about running_bw > this_bw, and GRUB reclaim for SCHED_FLAG_RECLAIM tasks on that CPU is miscomputed. The class change dequeues the still queued task with DEQUEUE_SAVE, which removes its bandwidth from running_bw and this_bw. switched_from_dl() then calls task_non_contending(), which, if the 0-lag time has passed, removes it from this_bw again because the task is TASK_DEAD. That removal is meant for the dying task's own DEQUEUE_SLEEP dequeue, which has removed nothing before it. Do the TASK_DEAD removal only on that dequeue path. Reproduced under QEMU on x86_64 and riscv64 by moving exiting deadline tasks to SCHED_OTHER in a loop; with this patch there is no warning. Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too") Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Juri Lelli <juri.lelli@redhat.com> Tested-by: Juri Lelli <juri.lelli@redhat.com> Link: https://patch.msgid.link/20261025124657.836229-1-nickolai@csail.mit.edu
2026-11-05sched_ext: Read exit_kind before flushing in scx_flush_disable_work()Kuba Piecuch
scx_disable_workfn() sets exit_kind to SCX_EXIT_DONE before it tears the scheduler down. scx_flush_disable_work() reads exit_kind after syncing disable_irq_work and flushing disable_work, so it can observe DONE from a disable_work that was queued after its flush and return while the teardown is still running: CPU A CPU B ----- ----- scx_vexit() scx_claim_exit() -> true bpf_struct_ops_map_link_detach() bpf_scx_unreg() scx_disable() scx_claim_exit() -> false scx_flush_disable_work() irq_work_sync() // not queued kthread_flush_work() // not queued irq_work_queue() scx_disable_irq_workfn() kthread_queue_work() scx_disable_workfn() exit_kind = SCX_EXIT_DONE scx_root_disable() atomic_read() -> SCX_EXIT_DONE bpf_map_put() Once bpf_scx_unreg() returns, its caller drops the struct_ops map reference, and the struct_ops trampolines can be freed while scx_root_disable() is still running and can still call into the BPF scheduler, e.g. ops.exit(). Read exit_kind before syncing and flushing, with acquire ordering paired with the atomic_try_cmpxchg() in scx_disable_workfn(). DONE is only set from disable_work, so once it is observed, the following kthread_flush_work() sees the queued work and waits for the teardown to finish. This patch was created with the help of an AI coding assistant. Fixes: d99f7a32f09d ("sched_ext: Fix scx_flush_disable_work() UAF race") Cc: stable@vger.kernel.org # v7.1+ Reported-by: Sashiko <sashiko-bot@kernel.org> Closes: https://lore.kernel.org/all/sashiko-outbox-195908@kernel.org/ Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Reviewed-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-11-02sched_ext: Reject scx_bpf_destroy_dsq() in tracing programsShubhang Kaushik (Ampere)
scx_bpf_destroy_dsq() takes the target user DSQ's raw spinlock. However, it is in scx_kfunc_ids_any, which is also registered for BPF_PROG_TYPE_TRACING. For an active root scheduler without ops.sub_attach, the compatibility path in scx_prog_sched() lets an unassociated tracing program resolve to that scheduler. The tracing program can then call scx_bpf_destroy_dsq() while the target DSQ lock is already held. For example, a scheduler which mixes FIFO and priority-queue insertions into a user DSQ triggers scx_error() from scx_dispatch_enqueue() while holding that DSQ's lock. scx_claim_exit() emits sched_ext_exit before the lock is released: scx_dispatch_enqueue() raw_spin_lock(&dsq->lock) scx_error() scx_claim_exit() trace_sched_ext_exit() tracing BPF program scx_bpf_destroy_dsq(dsq_id) destroy_dsq() raw_spin_lock_irqsave(&dsq->lock) The tracing program recursively acquires the same raw spinlock and locks up the CPU. Because the tracepoint runs before scx_finish_exit(), disable_irq_work is not queued and the scheduler cannot enter its normal disable path. Reject scx_bpf_destroy_dsq() from tracing programs at verification time. The kfunc remains available to SCX struct_ops and syscall programs. Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class") Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-11-02sched_ext: Fix ops.enqueue/dispatch() calls before ops.cpu_online()Tejun Heo
ops.cpu_online() documents that the cpu doesn't call ops.enqueue() or ops.dispatch() before that op. Both are gated on SCX_RQ_ONLINE, which follows the core's rq online state through the ext class's rq_online and rq_offline hooks. On a runtime cpu up, sched_cpu_activate() runs: 1. set_cpu_active() marks the cpu active. 2. cpuset_cpu_active() rebuilds the sched domains synchronously, and rq_attach_root() puts the rq online as the cpu is active now. The rq_online hook sets SCX_RQ_ONLINE. 3. scx_rq_activate() calls the root's ops.cpu_online(). Between 2 and 3, ops.enqueue() can be called for tasks woken or migrated onto the cpu, and ops.dispatch() can run on it. The hooks can't call the ops. They run under the rq lock with IRQs off while the ops can be sleepable BPF programs, and handle_hotplug() needs the rebuilt sched domains for its idle topology update. Track SCX_RQ_ONLINE in handle_hotplug() alone. Set it right after the root's ops.cpu_online(). Clear it and move the cpu's tasks to its local DSQ when the cpu goes down. Sched domain rebuilds, which take a live cpu's rq offline and online again, no longer touch the flag, so a remote dispatch can't see it cleared by a rebuild and bounce to the global DSQ. Between 2 and 3, tasks that land on the cpu now go to its local DSQ without ops.enqueue(), and with sub-schedulers a task whose scheduler lacks caps on the cpu is admitted anyway and counted in SCX_EV_SUB_FORCED_ADMIT. Both already happen between 1 and 2. Fixes: 60c27fb59f6c ("sched_ext: Implement sched_ext_ops.cpu_online/offline()") Cc: stable@vger.kernel.org # v6.12+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-11-02sched_ext: Factor out drain_offline_rq()Tejun Heo
Factor drain_offline_rq() out of rq_offline_scx() so that the drain can move to handle_hotplug() in the following patch, which removes the rq_online and rq_offline hooks. The drain is the rescue flush and the re-enqueue of an offline cpu's tasks onto its local DSQ. No functional change. Cc: stable@vger.kernel.org # v6.12+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-11-01sched_ext: Fix stale pick when the donor changes during dispatchTejun Heo
Dispatch takes rq->donor as @prev and can drop the rq lock. With proxy execution, @prev may not be the running task, and another CPU can reset the donor to rq->curr meanwhile, through proxy_needs_return() on a wakeup or sched_proxy_block_task() on a class or scheduler change. The wakeup case: 1. dispatch_one() drops the rq lock, e.g. to move a task from another CPU. 2. Another CPU wakes the donor, which proxy execution migrated to this CPU. proxy_needs_return() dequeues it and resets the donor to rq->curr, and the wakeup enqueues it, possibly on another CPU. 3. dispatch_one() sees @prev queued and returns SCX_DSP_PREV. 4. do_pick_task_scx() returns @prev, and this CPU runs a task that is queued on another rq. Under core scheduling, the core pick runs dispatch for the sibling rqs with each sibling's donor as @prev, and a sibling can switch @prev out while dispatch has the lock dropped. Whatever dispatch concluded, and whatever it does with @prev after the drop, is about a task the rq no longer owns, whether it keeps @prev running or hands it to ops.dispatch(). rq_modified_above() catches the change only when rq->curr is in a class above ext, and the dl_server pick skips that check. dispatch_core_pick() retries after a lock drop, but by then dispatch has used @prev. Restart the pick when @prev is no longer the donor. dispatch_one() tests after each step that can drop the lock, and scx_dispatch_sched() decides per ops.dispatch() call whether to hand @prev over, which also covers a class or scheduler change across a drop. With proxy execution alone, the donor only ever changes to rq->curr during a pick, so the retried pick can't go stale again. Under core scheduling, the restart goes through the core pick, which already retries after any lock drop. 7a3a6bfbd62a ("sched: Rework prev_balance() to avoid stale prev references") closed the same window in prev_balance() by reading rq->donor instead of taking @prev. That doesn't work here, as dispatch never evaluated the new donor, which may not even be a sched_ext task. Verified with synthetic reproducers for both cases. v2: Also cover rq lock drops inside dispatch (sashiko). v3: Don't keep a stale @prev after ops.dispatch() (sashiko). v4: Also cover a sibling switching @prev out under core scheduling. Retitle. Fixes: 7b0888b7cc19 ("sched_ext: Implement core-sched support") Fixes: c6fe97c34a1a ("sched: Allow enabling proxy exec with sched_ext") Cc: stable@vger.kernel.org # v6.12+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-11-01Merge tag 'sched-urgent-2026-11-01' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler fix from Ingo Molnar: - Fix a false positive cputime related warning in kcpustat_dyntick_stop() that can trigger sporadically after a clocksource switch (Stian Halseth) * tag 'sched-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: sched/cputime: Don't warn when the tick period starts after idle exit
2026-11-01Merge tag 'timers-urgent-2026-11-01' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull timer fixes from Ingo Molnar: - Fix on-stack hrtimer cleanup in play_idle_precise(), and add the hrtimer_on_stack cleanup class to facilitate the fix (Hui Su) - Fix clock_getres() system call's handling of NULL output pointer in the !CONFIG_POSIX_TIMERS case (Hungyu Lin) - Fix get_cpu_itimer() to return TICK_NSEC when the sampled CPU time is exactly equal to the nonzero expiry (Hungyu Lin) - Fix do_timens_ktime_to_host() time offset subtraction underflow check (Hungyu Lin) - Report zero remaining time for disarmed SIGEV_NONE posix timers, and add a self-test to detect this bug in the future (Hung-Jui Lin) - Fix NOHZ 'full kick' handling bug that can cause missed tick updates (Peng Wang) - Fix field access bug in the vdso/gettimeofday code that works only by accident (Thomas Weißschuh) - Fix & prevent immediate clocksource watchdog rerun after a delay (Karl Mehltretter) * tag 'timers-urgent-2026-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: clocksource: Prevent immediate watchdog rerun after a delay vdso/gettimeofday: Correct the read of tz_minuteswest and tz_dsttime tick/nohz: Reevaluate the tick for kicks flushed outside hardirq selftests: timers: Add a test for disarmed SIGEV_NONE timers posix-timers: Report zero remaining time for disarmed SIGEV_NONE timers time/namespace: Fix overflow when converting absolute deadlines timers/itimer: Keep a nonzero value at CPU timer expiry posix-timers: Accept NULL clock_getres() output in stubs sched/idle: Use hrtimer cleanup class in play_idle_precise() hrtimer: Add cleanup class for on-stack timers
2026-11-01sched/cputime: Don't warn when the tick period starts after idle exitStian Halseth
kcpustat_dyntick_stop() warns if idle ends before the tick period that the next tick will account has started. That happens on x86 in a CPU's first idle period with the tick stopped, right after the switch from the jiffies clocksource to tsc-early. On the jiffies clocksource ktime only moves at the tick. When the clocksource switches some time d after a tick, that d is lost: ktime continues from the value of the last tick. The next tick comes TICK_NSEC after the last one, but ktime has only moved TICK_NSEC - d, while tick_periodic() still adds a full TICK_NSEC to tick_next_period, which is now d ahead of ktime. When NOHZ starts, tick_init_jiffy_update() rounds it up to a multiple of TICK_NSEC and the sched timer is armed there, up to nearly two ticks after now. A tick stop saves that as ts->last_tick, and an idle exit before now reaches ts->last_tick - TICK_NSEC sees the tick period in the future. Nothing is accounted twice then. The first tick covers a period that starts after the idle exit, and the code already handles it. Drop the warning. Fixes: 96a494c9cfd8 ("sched/cputime: Don't account idle time twice after dyntick-idle") Reported-by: Paul Menzel <pmenzel@molgen.mpg.de> Reported-by: Breno Leitao <leitao@debian.org> Reported-by: Linus Torvalds <torvalds@linux-foundation.org> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Reviewed-by: Frederic Weisbecker <frederic@kernel.org> Signed-off-by: Stian Halseth <stian@itx.no> Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: https://lore.kernel.org/all/auTCQauaU733PY5z@gmail.com/ Link: https://lore.kernel.org/all/CAHk-=winC2u2D1FAX5grfq9F=bxmARxhuiQ4MvHAiJFu3iF6gw@mail.gmail.com/ Link: https://patch.msgid.link/20261031081408.3378884-1-stian@itx.no Closes: https://lore.kernel.org/all/bfe9b14a-6e65-41c5-b638-26f6e033788b@molgen.mpg.de/ -- kernel/sched/cputime.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
2026-10-30sched_ext: Call ops.sub_attach() only while the cgroup is onlineTejun Heo
The parent's ops.sub_attach() runs before the enable work checks that the cgroup is still online, so the parent can get an attach for a cgroup whose rmdir already returned. That complicates parents where cgroup cleanup and sub-scheduler attach race. Check CSS_ONLINE and call ops.sub_attach() under cgroup_lock(), so that the parent sees an attach only for a cgroup that is online. Fixes: ebeca1f930ea ("sched_ext: Introduce cgroup sub-sched support") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-30sched_ext: Fix scx_bpf_dsq_reenq() kerneldoc claiming NMI calls are allowedTejun Heo
c659e506f9a7 ("sched_ext: Reject NMI calls to lock-taking kfuncs") made scx_bpf_dsq_reenq() abort the scheduler when called from NMI. The kerneldocs of scx_bpf_dsq_reenq() and of its wrapper scx_bpf_reenqueue_local___v2() still say the kfuncs can be called from anywhere. Update both to exclude NMI. While at it, fix the wrapper's kerneldoc, which says it re-enqueues the calling CPU's local DSQ rather than the one SCX_DSQ_LOCAL resolves to, and the SCX_RENQ_* typo in the scx_bpf_dsq_reenq() kerneldoc. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Andrea Righi <arighi@nvidia.com>
2026-10-29sched_ext: Merge branch 'master' into for-7.4-fixesTejun Heo
Pull to receive 1832696b2207 ("bpf: Add bpf_call_rcu() kfunc") as a dependency for the upcoming tools/sched_ext/include sync, whose common.bpf.h declares bpf_call_rcu() using bpf_rcu_callback_t. Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-29Merge tag 'sched_ext-for-7.4-fixes' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: "This is a big pull for fixes, but much of it was in the works before the merge window. Most of these fixes are in sub-scheduler support, which nests BPF schedulers under a parent that hands them CPUs. It was heavily reworked over recent cycles and is now functionally complete. Porting real schedulers on it drove the latest additions and keeps finding corner cases. For example, once CPUs are shared and can be taken back, the kernel must keep every task moving when its scheduler can't run it or is failing, and those fallbacks interact subtly with each other and with other new core features. - The fallback paths meant to keep a task moving when its scheduler can't run it or is failing could themselves leave it stuck - The stall watchdog could blame a scheduler for time it wasn't responsible for, even taking down all of sched_ext - Sub-schedulers were told late, or not at all, which CPUs they could use - Deferred callbacks could stay queued or be dropped under core scheduling and proxy execution, with a sched/core fix to the duplicate check - Hardening: tracing programs can no longer call kfuncs that could deadlock the CPU" * tag 'sched_ext-for-7.4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (34 commits) sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programs sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypass sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while running sched_ext: Fix a lone IMMED task waiting behind a kept prev sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE sched_ext: Let the bypass load balancer move tasks homed on other cpus sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlocked sched_ext: Fix the bypass load balancer stranding tasks on cpus where the host lacks SCX_CAP_BASE sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASE sched_ext: Expire prev's slice in dispatch_one() under bypass sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the local DSQ sched_ext: Refuse extending a kernel slice sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICE sched_ext: Fix a cpu idling with a runnable rescuee sched_ext: Restore a rescuee's slice after ops.stopping() sched_ext: Fix rescue served time disagreeing with the slice under proxy execution sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime() sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQ sched_ext: Fix bypassed tasks stranded by the reenqueue limit sched_ext: Clear holding_cpu where the claim is confirmed ...
2026-10-29sched_ext: Avoid unnecessary IMMED reenqueues on idle CPUsAndrea Righi
When two IMMED tasks wake on an idle CPU, the kernel can unnecessarily send both back to BPF instead of keeping the first task ready to run. The first wakeup inserts a task into the local DSQ and raises rq->next_class to ext before the CPU leaves idle. The second insertion queues a reenqueue scan. rq_is_open() sees next_class == ext and reports the CPU unavailable, although it is still idle. The scan then reenqueues the head along with the task behind it, losing a placement that could have been honored on the next pick. Report the rq as open while the CPU is idle, even if a local insertion has already raised next_class to ext. This keeps the local DSQ head through the reenqueue scan while IMMED tasks behind it are returned to BPF. A local reproducer runs in a four-vCPU virtme-ng guest: two futex waiters pinned to CPU 0 are woken together from CPU 2, and the BPF scheduler inserts their wakeups into the local DSQ with SCX_ENQ_IMMED. Across 10000 pairs, kernel instrumentation counts 9776 unnecessary head reenqueues while the CPU is idle and next_class is ext. Keeping the idle CPU open eliminates these head reenqueues. Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED") Reported-by: Tejun Heo <tj@kernel.org> Link: https://lore.kernel.org/r/1548cc9309a6e0894e358262e0b3d4ee@kernel.org Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-27Merge tag 'mm-stable-2026-10-25-19-27' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Pull more MM updates from Andrew Morton: - "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo Stoakes): Replace VM_SPECIAL with helpers describing mapping ownership and behavior. Enforce the mmap flag rules and convert affected drivers, removing ambiguous tests shared by driver mappings and ordinary memory. - "support kselftest on nommu platform" (Hajime Tazaki): Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner to avoid its fork-dependent timeout command and document the build and runtime requirements. - "mm/collapse: separate a collapse from its callers" (Kiryl Shutsemau): Separate huge-page collapse from caller policy. Replace khugepaged identity checks with explicit permissions, and separate scanning from collapse with the mmap_lock drop in one place. - "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo Stoakes): Provide a common helper for testing whether a VMA has anonymous reverse mappings. Document the locking rules, annotate intentional concurrent reads for KCSAN, and convert the core MM callers. - "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups" (David Hildenbrand): Remove SECTION_MARKED_PRESENT in favor of the early-section flag. Simplify sparse memory initialization, section metadata handling and related PFN validity checks. - "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport): Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have been unusable since the NO_BOOTMEM conversion, leaving only the code and data overhead. - "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun Song): Unify the generic device DAX and HugeTLB vmemmap population paths and shared tail-page handling. Remove the duplicate DAX machinery while retaining the powerpc radix implementation. - "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum Yun): Fix huge-page split tests racing with khugepaged recollapse. Use pagemap and page flags instead of VMA-boundary assumptions which fail when glibc has already enabled THP. - "memcg: move memcgid refcount to objcg to unpin dying memcgs" (Bingfang Guo): Let dying memcgs be freed while swapped-out shmem still holds their IDs. Move ID references to reparentable objcgs, with the necessary swap accounting and RCU lifetime changes. - "mm/truncate: fix data loss when truncating straddling large folios" (Zhang Yi): Preserve out-of-range data when truncating straddling large folios. Fix a lookup race and split-failure handling, and handle minimum folio orders for the forthcoming ext4 iomap buffered I/O path. - "mm/damon: introduce damos quota goal target metric complement flag" (SJ Park): Let DAMOS quota goals use a metric's complement, so utilization can also express free-memory goals without adding separate metrics. Add the sysfs interface, documentation and tests. - "arm64/mm: Standardize printing for pgtable entries" (Anshuman Khandual): Use the generic formatter for arm64 page-table entries, ready for future wider entries. Move the safe-PTE-update check out of the header to avoid an include cycle. - "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk): Fix KMSAN metadata leaks on iounmap and failed ioremap allocations. Free backing pages before removing their mappings, and share the teardown with partial-allocation cleanup. - "mm/damon/core: preserve quota state when constructing schemes" (Karl Mehltretter): Preserve running DAMOS quota state while validating parameter updates. Even rejected updates could reset accounting and exceed the quota window; add tests for both accepted and rejected updates. - "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu): Fix vmalloc_dump_obj() lookups for last-page addresses and allocations spanning vmap zones. These allocations were being missed in diagnostic output. - "mm: various (urgent?) fixes" (David Hildenbrand): Fix mmap charge leaks and hugetlb use-after-free, wrong-address unmapping, stuck locks and endless faults. Also preserve reservations on failed mremap and handle stale userfaultfd bits. - "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris Charisis): Fix pointer and metadata confusion during RCU destruction of maple_range_64 nodes, avoiding double frees and invalid accesses. Add a regression test; no in-tree user currently uses this combination. - "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu Li): Fix hugetlb migration and preallocation accounting so /proc/vmstat retains the correct count and memory.numa_stat follows the folio to its new node. - "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park): Fix 32-bit DAMON age overflow which made old regions look young. Share hugetlb folio acquisition code and add a regression test for the previously fixed last-region skipping bug. - "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ Park): Reduce kdamond stack usage and remove unused enums. Add deterministic PSI quota-goal tests which do not depend on the machine's current memory pressure. - "mm/damon: fix the temporal goal tuner's size quota conversion" (Donggeun Yoo): Bound the temporal tuner's quota conversion to prevent overflow and zero effective quotas, which could stop schemes making progress. Add boundary tests. - "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He): Remove redundant GFP_DMA flags from coherent DMA allocation calls. The DMA API already selects the zone from device limits and ignores the caller's flag. - "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum Yun): Propagate child-process failures in the khugepaged tests and prevent background collapse from racing their checks. This removes false passes and intermittent failures. - "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM" (Enze Li): Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds the right maintainers. Also label the closing hugetlb_cgroup.h include guard. Summary of the 65 singletons in this merge: The singletons include a few worthwhile performance fixes. Avoiding unnecessary reverse-map tree updates gave a reported 15.9% improvement in UnixBench's execl test on a 192-core x86 system. A reclaim retry fix stops defrag_mode from spinning under memory.low protection; a reported file-creation stress test ran 5.7 times faster, with far less memory pressure. Narrower hugetlb invalidations also spare KVM guests needless refaults: the reported Redis-in-VM P99 QPS penalty while punching holes fell from 7.09% to 1.45%. Several fixes prevent lost data or corrupted memory. Lazyfree huge-page races with mremap()/munmap() could lose data or access freed memory, and MADV_COLD could lose a concurrent write's dirty bit. Another fix prevents stale migration entries from outliving large folios. On filesystems with blocks smaller than a page, extending a file by 4 GiB or more could expose bytes written past the old EOF through mmap instead of zeroes. That is fixed, along with use-after-free races in PAGEMAP_SCAN during process exit and in memory-cgroup accounting on PREEMPT_RT kernels. Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios, which could wedge other memory operations, and stop mremap() leaking commit charges until allocations fail under strict overcommit. A further fix keeps userfaultfd tracking intact on neighboring mappings. NUMA binding no longer prevents long-term pins when the chosen node has only movable memory, fixing QEMU/VFIO startup and io_uring buffer registration failures. MGLRU also gets a fix for early-boot crashes with memory cgroups disabled at boot, notably in small crash kernels. * tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits) mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root mm/memcontrol: publish null pointer before dropping css reference test_maple_tree: test maple_range_64 node in RCU mode maple_tree: fix maple_range_64 crashes in RCU mode mm/vma: avoid broken uffd merge on mremap tools/testing/shared: catch a failed posix_memalign() mm/damon/core: remove unreachable damos charge_{target,addr}_from mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache riscv: mm: exclude invalid THP PMDs from page table check mm/damon/core: filter out damos address filter-split failed region fs/proc/task_mmu: pin mm_users in do_pagemap_scan() mm/mempolicy: apply bind policies against the scoped allocation zone mm/vmscan: reclaim every zone in per-node proactive reclaim mm/mremap: correctly uncharge accountable VMAs in error path mm/mremap: populate calculated fields in vrm state correctly MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP mm: bdi: reset min_ratio after the sysfs attributes are gone mm/page_vma_mapped: continue past nonmatching huge PMDs selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers ...
2026-10-27sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programsTejun Heo
A tracing program can call scx_bpf_sub_grant() and scx_bpf_sub_revoke() through scx_kfunc_ids_any, which is registered on the tracing kfunc hook. Before returning, both invoke ops.sub_caps_updated() of the sub-schedulers whose caps changed, so those ops run in the tracing program's context. No SCX op expects to run from an arbitrary tracing context. With an rq lock held, this leads to the same deadlock that b07a7450f6ef ("sched_ext: Reject rq-lock-taking kfuncs in tracing programs") fixed for direct calls: 1. A tp_btf/sched_switch program fires from __schedule() with the rq lock held. scx_locked_rq() only tracks rq locks held across SCX callbacks, so it returns NULL there. 2. The program calls scx_bpf_sub_grant() on a child, which invokes the child's ops.sub_caps_updated() before returning. 3. That op calls scx_bpf_cidperf_set() on this cpu, which sees no locked rq and takes the rq lock the cpu already holds. Add both kfuncs to scx_kfunc_ids_no_tracing and rewrite the set's comment to cover both reasons for membership: taking an rq lock and running SCX ops. Fixes: 5f2a9a4c2e6c ("sched_ext: Add coalescing sub_caps_updated() notifier for sub-schedulers") Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-27sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypassTejun Heo
An earlier patch sends a bypassed task to rescue when its host lacks SCX_CAP_BASE on the cpu. The task's rescue time is charged to its owner, the bypassing sub, which is usually exiting. The overload pick skips exiting subs and ejects an unrelated sub or nobody. The host runs the task and lacks BASE, but carries none of the charge. Charge the sched running the rescuee, the host while the owner bypasses. SCX_EV_SUB_RESCUE already counts on that sched and the stall watchdog blames it. Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while runningTejun Heo
Since an earlier patch sends a bypassed task to rescue when its host lacks SCX_CAP_BASE, a task scx_fail_parent() punted to its bypassing parent at NONE can be a rescuee. When a cgroup migration into a root-owned cgroup re-homes it while running, nothing resets it: the DEQUEUE_SAVE dequeue and put_prev_task_scx() keep a running task's slice, and __scx_disable_and_exit_task() skips a NONE task. rq->scx.rescue.curr keeps pointing at the task, now on the root. An RT task, for example, can keep the task from serving its grant until the last sub is freed. From then on scx_rescuee() returns NULL and the rescue never ends. scx_rescue_timerfn() keeps accessing the task after it is freed. Move the reset out of scx_disable_task() into scx_reset_task_fields() and call it on every path that takes a task out of its sched's control, the exit of a NONE task included. Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix a lone IMMED task waiting behind a kept prevTejun Heo
While dispatch is in progress, rq_is_open() returns true and dsq_inc_nr() schedules no reenqueue for an IMMED task inserted on an empty local DSQ. The check at the end of dispatch_one() schedules the reenqueue only when more than one task is queued, so a dispatch that keeps prev leaves a lone IMMED task behind: 1. a parent's ops.dispatch() sub-dispatches into a child, which consumes a task with SCX_ENQ_IMMED onto the empty local DSQ 2. the parent extends prev's slice and dispatch keeps prev 3. the local DSQ holds one task, so no reenqueue is scheduled The IMMED task waits on the local DSQ behind prev. Schedule the reenqueue when dispatch keeps prev and an IMMED task is queued, as even a lone IMMED task can't run right away behind prev. v2: The description and dispatch_one()'s comment are reworded. Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASETejun Heo
A task already queued on a bypass DSQ stalls when the host loses SCX_CAP_BASE on that cpu, as the cpu no longer consumes the DSQ. Sending a bypassed task to rescue at enqueue time covers only a task enqueued after the loss, and the balancer no longer donates to a capless cpu. Have the bypass load balancer drain such a DSQ: a cpu where the host lacks BASE is a donor with a target of zero, and the donees take its tasks under their ordinary fill. A drained task no donee can take is reenqueued at its home: to rescue when the host lacks BASE there and rescue is enabled, and to the home's bypass DSQ otherwise. A balancing donor leaves a task that can't move to its donee where it is. Reenqueueing it at home would put a task homed on the donor cpu at the tail of its own DSQ, and a blocked task that proxy execution migrated there, which every remote cpu refuses, would go to the tail on every pass. Without rescue, a task homed on the drained cpu goes back onto the drained DSQ, once per pass, and waits for the host to regain BASE, the bypass to end, or the stall watchdog to eject the host. Its reenqueues are not counted against the reenqueue limit, as its scheduler is bypassing. A BASE loss kicks the balancer so that the drain doesn't wait for the next interval. With a zero interval the kick is the only pass, and a task that lands on the drained DSQ after it waits the same way. v2: The description drops an in-flight affinity change from the causes of a refused balancer move. v3: - The description says that a stuck task's reenqueues are not counted against the reenqueue limit (sashiko). - The description names the stall watchdog as the way out for a task stuck without rescue, and the tasks a zero interval leaves on a drained DSQ. - bypass_lb_reenq_home()'s kerneldoc and the description give the placement with rescue disabled and for a task that must run at home, and the other two comments on that placement point to the kerneldoc. v4: - Skip a task the donee would refuse on a balancing donor instead of reenqueueing it at home (Andrea). - bypass_lb_reenq_home()'s kerneldoc is corrected on a migration-disabled task homed on a donee and on the local DSQ admission. - The description is condensed. Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Let the bypass load balancer move tasks homed on other cpusTejun Heo
The following patch uses the bypass load balancer to drain a cpu's bypass DSQ when the host loses SCX_CAP_BASE there. The host no longer consumes that DSQ, so the drain has to move every task on it. The balancer skips the ones homed on another cpu. An earlier pass moved such a task in from its home cpu's DSQ, and the donor cpu has not consumed it yet. A task's DSQ membership may only change under the task's rq lock, which the balancer does not hold. Take such a task off the donor DSQ with unlink_dsq_and_switch_rq_lock(), the handshake a remote consumer uses. It drops the DSQ lock and switches to the task's rq lock. A dequeue that gets in between takes the task, and the balancer moves on. The insert then runs under the task's home rq lock, as it already does for a donor-homed task. A task whose donee is its own cpu can always run there. task_can_run_on_remote_rq() warns when asked about a task's own cpu, so the balancer skips the test for such a task. The walk holds the donor rq lock, which keeps the donor cpu's dispatch and its enqueuers away from the donor DSQ. A move of a task homed elsewhere switches to the task's rq lock, and both can reach the DSQ meanwhile. The parked cursor keeps the walk's place through a consume, and the sequence bound keeps a task enqueued meanwhile out of the pass. v2: The description says "the following patch", not "patches", uses the balancer to drain a DSQ. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlockedTejun Heo
Prepare the bypass load balancer for moving tasks homed on other cpus. Such a move takes the task's own rq lock through the remote consumer's handshake, which drops the donor DSQ lock on the way, so its insert runs with the donor DSQ unlocked and under the task's rq lock. The balancer's own move holds the donor DSQ lock throughout, with the donee DSQ lock nested inside. Switch the donor-homed move to the same scheme so that one body can serve both: factor it out of bypass_lb_cpu() into bypass_lb_move(), which takes the task off the donor DSQ and inserts with the DSQ unlocked. A task that can't run on the donee goes back to the head of the donor DSQ instead of being skipped in place. The walk parks its cursor behind the task before each move and resumes from it afterwards. It is bounded to the tasks queued when it began, by their DSQ sequence numbers, so a task it put back is not visited again in the same pass. Before, the walk held both the donor rq lock and the donor DSQ lock across every move. The donor DSQ can be very long and contended, a walk competing with the cpu's dispatch and the enqueues for the DSQ lock would crawl, and the walk is what relieves that contention by moving tasks off the DSQ. The walk now cycles the DSQ lock around each insert but still holds the rq lock throughout, and that alone keeps dispatch and enqueues away. The DSQ lock is uncontended when retaken, so cycling it is cheap. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix the bypass load balancer stranding tasks on cpus where the ↵Tejun Heo
host lacks SCX_CAP_BASE The bypass load balancer spreads a host's bypassed tasks over the cpus of a node by moving them between its per-cpu bypass DSQs. It counts every cpu of the node as capacity and takes any cpu below the target as a donee. A cpu's dispatch only descends into schedulers holding SCX_CAP_BASE on it, so a bypass DSQ on a cpu where the host lacks BASE is never consumed. A task moved there sits until the host regains the cap or the bypass window ends. Count only cpus where the host holds BASE as capacity and pick donees from them. The target rises to spread the node's tasks over the cpus that can run them, and a capless cpu receives nothing. A capless cpu with tasks is still an ordinary donor, offloaded down to the donor target when its queue grows past the threshold; draining it is left to the following patches. Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASETejun Heo
A bypassed task stalls on a cpu where its host lacks SCX_CAP_BASE. The host, the nearest ancestor that is not bypassing, runs a bypassing sub-scheduler's tasks through its per-cpu bypass DSQs. A cpu's dispatch only descends into schedulers holding BASE on it, so the host's bypass DSQ on such a cpu is never consumed until the host regains the cap. Resolve a bypassed task against its host at enqueue time. If the host lacks BASE on the cpu, send the task to rescue on that cpu, which runs it at bounded bandwidth without the host's dispatch having to reach it. The enqueue does not look above the host for an ancestor that holds BASE. That ancestor would run the task on cpus it never granted the host. With rescue disabled the task goes to the host's bypass DSQ as before and stalls there until the host regains BASE or the bypass ends, or until the stall watchdog ejects the host, which it blames for a task stuck on its DSQ. A task that must run on the cpu, migration-disabled or with a migration pending, is admitted to the local DSQ instead, as any insert of such a task is. Bypass also used to end a rescue, sending a preempted rescuee to the bypass DSQ with every other preempted task. Now that rescue is how a bypassed task runs while its host lacks BASE, keep a preempted rescuee on the local DSQ under bypass too. dispatch_one() also keeps such a rescuee running, as an earlier patch's bypass expiry leaves a kernel slice alone. The grant stays bounded by served time. A task scx_fail_parent() punted to its bypassing parent at NONE can be a rescuee too. switched_from_scx() ends its slice as it does for an enabled task, so that a NONE rescuee leaving sched_ext while running doesn't leave rq->scx.rescue.curr pointing at it. While at it, the rescue test in __scx_resolve_local_dsq() goes through the new scx_rescue_enabled() helper. v2: - The description notes that dispatch_one() keeps a bypassed rescuee too, through the new patch before this one (sashiko). - scx_rescue_keep()'s kerneldoc drops dispatch_one() declining a rescuee with slice left, which the new patch removes. v3: - With rescue disabled, a task that must run on the cpu is admitted to the local DSQ instead of waiting on the host's bypass DSQ. - The description names the stall watchdog as the way out for a task stuck without rescue. v4: - switched_from_scx() ends the slice of a NONE task, which can now be a rescuee (sashiko). - scx_punt_task()'s kerneldoc no longer says that switched_from_scx() skips a NONE task. - scx_bypass()'s kerneldoc and the ops.sub_ecaps_updated() doc say where a bypassed task can be besides the bypass DSQs. Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Expire prev's slice in dispatch_one() under bypassTejun Heo
The next patch keeps a bypassed rescuee on the local DSQ and needs dispatch_one() to keep it running too. dispatch_one()'s keep test refuses any prev whose scheduler is bypassing, so another local task could run ahead of the rescuee, even an escalated SCX_TASK_PROTECTED one. The test exists because a cpu-form scheduler can write p->scx.slice directly from BPF, so the slice can't be trusted under bypass. The refusal doesn't hold anyway. Since 025b1bd41965 ("sched_ext: Implement hierarchical bypass mode"), the host's ops.dispatch() runs while prev's scheduler bypasses, and the post-dispatch test in scx_dispatch_sched() keeps a queued prev with slice left without checking bypass. A task dispatched in between waits behind prev until the next bypass tick zeroes the slice, at most a tick. Expire the slice before the keep test instead when prev's scheduler is bypassing, as the bypass tick would. Neither test keeps prev on a zeroed slice, so prev rotates. A kernel slice survives the expiry and keeps prev. Only a sub-scheduler's task can hold one, and sub-schedulers are cid-form, which rejects direct slice writes, so it can be trusted. Drop the stale pointer to scx_disable_workfn(), update scx_bypass()'s kerneldoc, and add a TODO noting that the bypass slice handling can be cleaned up once the cpu form is deprecated and the kfuncs refuse slice raises from a bypassing scheduler. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the ↵Tejun Heo
local DSQ put_prev_task_scx() enqueues a still-runnable task when the pick has settled on idle. The insert can land the task on the cpu's own local DSQ, and then the cpu must not go idle over it. 7a9df7667963 ("sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ") reschedules the idle task in that case, but only after the SCX_ENQ_LAST insert. The plain insert, taken when the task can't stay on the cpu, can land on the local DSQ too: rescue can admit a task there when its scheduler lacks SCX_CAP_BASE on the cpu. Without the resched the cpu idles with the task queued until an unrelated wakeup or the rescue timer's escalation. Reschedule after either insert. In the following patches, the plain insert becomes the common path for bypassed tasks under rescue. v2: The description's forward reference to the plain insert becoming the common path no longer names the following patch. Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Refuse extending a kernel sliceTejun Heo
A rescuee can keep running after its grant is served, on a cpu its scheduler has lost. The write rules deny a slice extension only where the scheduler lacks SCX_CAP_BASE on the cpu, and a scheduler holding BASE but not ENQ can have a task under rescue there: 1. the scheduler extends the rescuee's slice 2. BASE is revoked. The cap-revoke eviction skips the rescuee. 3. the grant is served and scx_rescue_charge() ends the rescue 4. the task runs out the extension on the cpu its scheduler lost, outside the rescue's bandwidth A scheduler shortens or zeroes a rescuee's slice to preempt it, and scx_rescue_keep() resumes the grant afterwards. An extension serves no such purpose. Refuse extending a kernel slice in both scheduler write paths, the direct write in scx_bpf_task_set_slice() and the out-of-band application, counted in SCX_EV_SLICE_DENIED like an extension without BASE. Shortening stays allowed. scx_rescue_keep()'s restore of the remainder raises the slice from zero. Like the kernel's other slice writes, it goes through scx_set_task_slice() and not the scheduler write paths. v2: - scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot. - SCX_EV_SLICE_DENIED's comment lists every refusal it counts. - scx_bpf_task_set_slice()'s kerneldoc says that shortening is refused on a protected slice. Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICETejun Heo
In the following patches, bypassed tasks run under rescue when their host lacks SCX_CAP_BASE on the cpu. Under bypass the kernel zeroes the running task's slice at every tick, as it does not trust the scheduler's slices. A rescue grant has to survive that, or it is served a tick at a time with the rescuee sent to the tail of the local DSQ each time. The generic slice handling can't tell a kernel grant from a scheduler's slice. Mark the grant as a kernel slice with SCX_TASK_KERNEL_SLICE, set at rescue admission and cleared when the slice ends. The generic slice handling leaves it alone. The default refill does nothing for it, and the three zeroings go through the new expire_task_slice(), which skips it: - the bypass tick's zeroing of the running task - the eviction of a running task that can't stay on the cpu when bypass lifts - the cap-revoke eviction of a running task Schedulers keep their control. Preemption and scx_bpf_task_set_slice() still apply, and the rescue stays bounded by served time. The sites that recognized the rescuee by identity, the reenqueue scan and put_prev_task_scx()'s reinsert, test the flag instead. scx_task_has_kernel_slice() hides the test behind the scx_has_subs() static key, as only a sub-scheduler's task can carry the flag. The flag's entries join the enum tables in tools/sched_ext. No kernel slice reaches a zeroing until bypassed tasks run under rescue. The cap-revoke eviction skipped the rescuee already, and bypass entry ends an open rescue, so the bypass tick and the eviction at bypass lift see none. The refill is the one behavior change. The pick refills a zero slice only for a task that can stay on the cpu, so it concerns a rescuee whose scheduler holds BASE there. A scheduler may zero such a queued rescuee's slice, and the next pick used to refill it to the default. Now the rescuee runs on the zero slice until the next tick. There put_prev_task_scx() restores the remainder when another task is runnable. When nothing else is runnable, dispatch_one() keeps the rescuee as the last task. The pick then ends its zero slice, which ends the rescue, and refills the default. A scheduler with SCX_OPS_ENQ_LAST skips the keep-last test. scx_rescue_keep() restores the remainder there instead and the rescue goes on. v2: - scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot. - The description covers a scheduler with SCX_OPS_ENQ_LAST, whose rescuee dispatch_one() keeps on the remainder. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix a cpu idling with a runnable rescueeTejun Heo
A scheduler preempts a running rescuee like any other task by zeroing its slice. The rescue grant survives the preemption. put_prev_task_scx() restores the unserved remainder through scx_rescue_keep() and queues the rescuee at the tail of the local DSQ, behind whatever the scheduler wanted to run first. That breaks down when the scheduler zeroes the slice but dispatches nothing. dispatch_one() keeps the previous task running only with slice left or, when nothing else is runnable, with its scheduler holding SCX_CAP_BASE on the cpu and not taking the last task through SCX_OPS_ENQ_LAST. A rescuee fails both once its slice is zeroed if its scheduler lacks BASE there or takes the last task: 1. ops.tick() or scx_bpf_task_set_slice() zeroes the rescuee's slice 2. the pick finds nothing else to run and settles on idle 3. put_prev_task_scx() restores the remainder and queues the rescuee on the local DSQ 4. nothing reschedules the cpu, as the pick has settled and __schedule() clears need_resched right after it The cpu idles with the rescuee queued until an unrelated wakeup or the rescue timer's escalation. Restore the remainder from dispatch_one() too. When nothing else is runnable, its last keep now keeps the rescuee running on the remainder, so the pick never settles on idle with a rescuee runnable. put_prev_task_scx() keeps handling a displaced rescuee, where another task won the pick. A rescuee that passes the existing keep-last test is kept there instead and continues as an ordinary task, its rescue ended with the zeroed slice. That is fine as its scheduler holds SCX_CAP_BASE on the cpu. The new keep also covers bypassed tasks run under rescue in the following patches. v2: The dispatch_one() comment and description now say that a rescuee whose scheduler holds SCX_CAP_BASE on the cpu passes the keep-last test and continues as an ordinary task (sashiko). Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Restore a rescuee's slice after ops.stopping()Tejun Heo
put_prev_task_scx() restores a preempted rescuee's unserved slice through scx_rescue_keep() before it calls ops.stopping(). A rescuee whose slice the scheduler zeroes there is left to the reinsert instead: 1. ops.stopping() zeroes the rescuee's slice 2. the reinsert condition special-cases the rescuee and keeps it on the local DSQ with a zero slice 3. the pick's default refill supplies a slice, or the task runs on a zero slice until the tick when its scheduler lacks BASE Move the keep after the stopping call so that scx_rescue_keep() restores the remainder whether a preemption or ops.stopping() zeroed the slice, and drop the special case from the reinsert condition as a rescuee reaching it always has a slice. A rescuee whose slice ops.stopping() zeroed now goes to the tail of the local DSQ with its remainder restored, behind what the scheduler wanted to run first, instead of staying at the head on a zero slice. The slice end in the same block moves with it, which changes nothing for a protected task. Its slice is the unserved remainder, and scx_rescue_charge() ends the protection in the same update_curr_scx() call that zeroes the slice, so the block never sees a protected task in either position. v2: The description no longer says that a protected task's protection outlasts ops.stopping(). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix rescue served time disagreeing with the slice under proxy ↵Tejun Heo
execution Rescue measures served time as the rescuee's sum_exec_runtime minus a snapshot taken at admission. Under proxy execution that measure follows the wrong task. update_se() credits sum_exec_runtime to rq->curr, the mutex owner that executes, while update_curr_scx() drains the slice of rq->donor and charges its rescue. A rescuee can be either, so its served time and its slice disagree in both directions. As the owner running for another donor, the rescuee accrues served time that drained no slice and charged no budget, and its rescue ends early. As the donor, its rescue runs long: 1. the rescuee blocks on a mutex and the owner runs in its place 2. the owner's run drains the rescuee's slice and the budget, but not the served time 3. the slice runs out with the rescue still open, and scx_rescue_keep() restores a remainder that the owner's run didn't reduce 4. the owner keeps running on the rescue until it releases the mutex, past the admitted slice and overdrawing the budget Count served time in scx_rescue_charge() from the delta that drains the slice. Charging the owner's run to the donor's rescue matches proxy execution, which drains the donor's slice and accounts the cgroup time to the donor for the owner's run. scx_rescue_keep() also ignored the return of scx_set_task_slice(). The call refuses the restore on a rescuee that escalation marked SCX_TASK_PROTECTED, and the rescuee was kept with a zero slice and its protection set. That refusal is now unreachable. Escalation sets the slice to the unserved remainder before it protects the task. From then on every slice write a scheduler can reach is refused. Only a sub-scheduler's task is rescued, and sub-schedulers are cid-form, which rejects direct slice writes. The slice then moves only by what update_curr_scx() drains, and that drain charges the rescue by the same amount, so the slice runs out exactly when the charge ends the rescue. WARN and end the rescue if the restore is refused anyway. Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime()Tejun Heo
Move refill_task_slice_dfl() below apply_slice_vtime(), with the other slice writers, to prepare for the kernel-slice handling the following patches add to them. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQTejun Heo
Bypass exists so that runnable tasks make progress without trusting the BPF scheduler. A host with SCX_CAP_ENQ_IMMED (BASE) but not SCX_CAP_ENQ on a cpu cannot run a single bypassed task there. The task loops instead: 1. a bypassed task lands on the host's bypass DSQ for the cpu 2. the cpu's dispatch descends into the host, which holds BASE there 3. the host consumes the task into the local DSQ as a plain insert 4. the insert needs ENQ and the host lacks it, so the task is rejected 5. the reject drain reenqueues the task and it is back at step 1 A bypass consume puts a task on the cpu being dispatched for, to run right away. That is the IMMED contract, and BASE is the cap it needs. Consume with SCX_ENQ_IMMED. BASE is then the one cap a bypassed task ever needs. Dispatch descends into a host only on a cpu where the host holds BASE, so a consume never lacks it. A consumed task is first on the local DSQ when dispatch enters the host with the DSQ empty. That is always the case for the root and normally for a parent dispatching into a child. A consumed task that cannot run right away is reenqueued like any IMMED task and goes back to the bypass DSQ. Waiting on the local DSQ would need ENQ. The task cannot run right away when a higher class or a HEAD insert took the cpu meanwhile, or when the parent queued its own task before dispatching into the host. Each such task waits on the bypass DSQ for the next dispatch. The reenqueue is not counted against the reenqueue limit, as the task's scheduler is bypassing. v2: The description says that the reenqueue of a consumed task is not counted against the reenqueue limit (sashiko). Fixes: 701b7bcad868 ("sched_ext: Add the SCX_CAP_ENQ cap") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Fix bypassed tasks stranded by the reenqueue limitTejun Heo
scx_do_enqueue_task() counts a task's reenqueues until the task runs. Past SCX_REENQ_MAX_REPEAT it ejects the owning scheduler and leaves the task off every DSQ for the scheduler's exit to requeue. The count comes before the bypass test, so it also counts the reenqueues of a task whose scheduler is bypassing. Those reenqueues are not the scheduler's placements, and ejecting the scheduler for them fixes nothing. Under bypass the kernel queues the scheduler's tasks on the bypass DSQs. A host calling scx_bpf_reenqueue_local() reenqueues a bypassed child's tasks on the cpu's local DSQ along with its own, and the reject drain reenqueues a consumed task that a cap check rejected. In the following patches, the consume itself reenqueues a task it can't run right away, and the bypass load balancer reenqueues a task at its home. The count is reset only when the task runs or leaves sched_ext, so an ejection can lead to a stranding the exit does not recover: 1. the task's reenqueues pass the limit, which ejects its scheduler and strands the task 2. the exit's scx_bypass() cycles the task through dequeue and enqueue, which requeues it on a bypass DSQ with the count still past the limit 3. the task's next reenqueue under bypass strands it again 4. scx_bypass() requeues stranded tasks only when the scheduler wasn't bypassing yet, so the task stays off every DSQ until the disable walks the tasks 5. the disable takes scx_enable_mutex, the cgroup lock and scx_fork_rwsem before the walk, and a sub-scheduler's disable the cpu hotplug lock too. It hangs if the stranded task holds one of them, forking, attaching to a cgroup or doing a cpu hotplug Test bypass before the count. A task reenqueued under bypass stays queued until a later dispatch or balancer pass, so there is no tight loop to bound, and the task's first run resets the count. The offline test moves up with it to stay ahead of the bypass test. A reenqueue on an offline cpu goes to the local DSQ and isn't counted either. A host that keeps calling scx_bpf_reenqueue_local() could get a bypassed child ejected, as the child's tasks on the local DSQ counted against the child. The reenqueues are the host's, and its own tasks still count against it. v2: The description lists the cpu hotplug lock among the locks a sub-scheduler's disable takes before the walk. Fixes: 7706d6e4f2e3 ("sched_ext: Bound per-task reenqueues and eject the owning scheduler") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-27sched_ext: Clear holding_cpu where the claim is confirmedTejun Heo
The bypass load balancer will take tasks off a remote cpu's bypass DSQ with unlink_dsq_and_switch_rq_lock() in the following patches and insert them into another bypass DSQ. Clearing holding_cpu after the claim is up to each claimant, so the balancer would need a clear of its own. unlink_dsq_and_switch_rq_lock() marks a task with holding_cpu for the gap between dropping the DSQ lock and taking the task's rq lock, so that a dequeue in that gap can tell the claimant it lost. Once the claimant holds the rq lock and finds the marker intact, nothing can take the task away under it and the marker has no reader left. Each claimant clears it at its own site after that point: - dispatch_to_local_dsq(), which runs the handshake inline, in its same-rq and fallback branches - scx_proxy_reject_task() for the proxy reject - the dequeue inside deactivate_task() for both remote moves Clear it where the claim is confirmed instead and drop the explicit clears. No behavior change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>