Schedulers on the default cid mapping treat [0, nr_online_cids) as the
online set and restart on hotplug. Schedulers that install their own mapping
with scx_bpf_cid_override() have no way to learn which cids are online: the
count no longer identifies members and the CPU-form cpumask is unusable from
cid programs. This is an obvious hole in the cid API.

Add scx_bpf_online_cmask(), a kernel-maintained cmask in the scheduler's
arena, allocated alongside the per-CPU scratch masks and populated after the
cid mapping is finalized and before ops.init(), for child schedulers too.
The pointer stays valid through ops.exit() with no reference to take. It is
the arena offset as a void pointer, the same form struct_ops arena arguments
arrive in. The verifier types the void return as a scalar for the program's
arena cast.

The mask follows the SCX hotplug notifications: seeded from cpu_active_mask
and updated before ops.cid_online/offline() runs, so it lags cpu_online_mask
only inside a hotplug transition. Updates walk the scheduler list under the
lock that also serializes unlinking. Reads are live, not atomic snapshots.
Root initialization excludes hotplug.

v2: Reworded the getter kerneldoc (Andrea Righi).

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>

scx_qmap tracks idle cids itself. pick_direct_dispatch_cid() claims a cid by
clearing its bit and the task is inserted into that cid's local DSQ, which
kicks the CPU. When the task does not arrive, for example because the insert
fell back to the global DSQ after an affinity change, the CPU wakes, finds
nothing and picks idle again. That is not an idle transition, so
ops.update_idle() is not called and the cid stays marked busy until an
unrelated task runs on it.

Restore the claim from ops.dispatch(). The kick guarantees a dispatch on the
kicked CPU, and when it finds nothing to run with a NULL @prev, the CPU is
going back to idle. Document the pattern in ops.update_idle(), which reports
only actual transitions.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Cc: Andrea Righi <arighi@nvidia.com>

scx_cgroup_init_args carries the initial weight and bandwidth control
parameters of a cgroup to ops.cgroup_init(), but not its cpu.idle
state. A cgroup that was already configured idle before the scheduler
was loaded (or before it was onlined under it) is presented as
non-idle, and the BPF scheduler only learns about it if cpu.idle is
written again later.

Add the sched_idle state to scx_cgroup_init_args and fill it in all
four places that build the args: scx_tg_online() for cgroups onlined
under the scheduler, scx_cgroup_init() for cgroups that already exist
when the scheduler is loaded, and the sub-scheduler handover paths
scx_cgroup_claim_subtree() and scx_cgroup_return_subtree().

Verified in a VM with a probe scheduler printing the init args: a
cgroup configured cpu.idle=1 before loading shows sched_idle=1 in
ops.cgroup_init(), the default shows 0, and later cpu.idle writes
still come through ops.cgroup_set_idle(). The sub-scheduler paths
are compile-tested only.

Fixes: 347ed2d566da ("sched/ext: Implement cgroup_set_idle() callback")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

Fix several comment issues found during review:
__setschduler_prio() -> __setscheduler_class()
scx_iter_scx_dsq_new() -> bpf_iter_scx_dsq_new()
scx_next_task_scx() -> set_next_task_scx()

Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>

ops.cgroup_set_bandwidth() is delivered from scx_group_set_bandwidth(),
which runs from the cpu.max cgroup interface write path (tg_set_bandwidth())
in process context. scx_group_set_bandwidth() holds
percpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep.
The call site is therefore sleepable, like ops.cgroup_init().

bpf_scx_check_member() rejects a sleepable program on any member not on its
allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when
a cgroup gains a cpu.max limit at runtime; it must instead pre-reserve memory
for a callback that cannot allocate. Add cgroup_set_bandwidth() to the
allow-list so the callback can allocate on demand, and document that it may
block.

A scheduler must decide at load time whether to mark the callback sleepable,
but the allow-list entry is a verifier property with no symbol to probe. Add
a compatibility marker whose presence in the kernel's BTF lets userspace detect
this support: DEFINE_SCX_COMPAT_MARKER() emits an empty, callerless function,
here scx_compat_marker_cgroup_set_bandwidth_may_sleep(). It is __used
__retain so neither the compiler nor the linker (under
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) drops it. The markers share the
scx_compat_marker_ prefix and are collected near the end of ext.c so more
can be added as further capabilities appear.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>

Pull to receive the __arena argument conversion:

 67f1f4a48c24 ("sched_ext: Pass kernel arena pointers to ops_cid callbacks")
 a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")
 a05c5b5cb5cf ("sched_ext: Convert scx_bpf_cid_override() to __arena array arguments")

along with the bpf-next branch carrying the __arena argument support they
depend on.

Conflict in kernel/sched/ext/ext.c between:

 c384ab8a0b13 ("sched_ext: Move the config-off sub-cap kfunc stubs into sub.c")

and:

 a8dc810968af ("sched_ext: Convert sub-cap kfuncs to __arena cmask arguments")

which updated the stubs in their old ext.c location. Resolved by keeping
ext.c without the stubs and applying the prototype conversion to the
relocated stubs in sub.c.

Signed-off-by: Tejun Heo <tj@kernel.org>

With sub-schedulers, tasks of different schedulers routinely share rqs and
SMT siblings, but scx_prio_less() consults ops.core_sched_before() only when
both tasks belong to the same scheduler. Every pair spanning two schedulers
falls back to the default ordering, so no scheduler can express ordering
across a scheduler boundary, including a root over its sub-schedulers'
tasks.

Order a pair spanning schedulers by the nearest common ancestor that
implements ops.core_sched_before(): both tasks are in its subtree, making
this the one op where a scheduler is called on tasks it delegated to its
sub-schedulers and may not be scheduling anymore. Same-scheduler pairs keep
using the owning scheduler's op so a parent never orders inside a subtree it
delegated. The op is skipped when the deciding scheduler is bypassing on
either task's CPU.

Update scx_qmap to fall back to the kernel's default ordering when handed a
delegated task it has no task_ctx for.

Signed-off-by: Tejun Heo <tj@kernel.org>

- inlines.h: scx_bpf_dispatch() doesn't exist; the comment means
  scx_bpf_sub_dispatch()
- internal.h: name %SCX_DEQ_SCHED_CHANGE instead of the never-defined
  %SCX_DEQ_SAVE
- internal.h: @name shows up in the ops file in the scheduler's sysfs
  directory, not a "kernel.sched_ext_ops" sysctl

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>

The cid-form set_cmask() and sub_caps_updated() callbacks receive cmasks
that the kernel builds in the arena, and the kernel converts the kernel
addresses to the BPF arena pointer form by hand before each call.

BPF now translates between BPF and kernel arena addresses for __arena
arguments. Tag the arguments __arena in the cfi stubs and the ops_cid member
declarations and pass the kernel arena addresses directly, dropping the
manual scx_kaddr_to_arena() conversions and the now-unused helper. The
delivered value is unchanged and existing BPF-side code works as before.

The arena argument address translation is currently implemented only on
x86-64. cid-form schedulers implementing these callbacks load only there for
now.

Signed-off-by: Tejun Heo <tj@kernel.org>

When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The
watchdog blames the waiting task's owner, but the misbehaving party is
whoever floods the queue, not whoever happens to time out.

Track each sched's recent rescue consumption per cpu as a decaying average.
Once the oldest waiter on a cpu's rescue DSQ has been queued past a
threshold derived from the rescue knobs (4s at the defaults), the rescue
timer ejects the sub with the highest recent consumption on that cpu with
SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and
nothing is ejected - the generic stall watchdog eventually blames the
waiter's owner instead. Ejections on a cpu are spaced one threshold apart so
the freed bandwidth can drain the backlog before another sub is judged.

The overload check only wins the race against the stall watchdog when the
watchdog timeout clears the threshold, and a single in-budget wait must not
cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit
and on knobs whose funding period exceeds half the threshold.

v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
      wraps 2^31 ticks after the last ejection and suppresses ejections.
      (sashiko AI)

    - Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay
      delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage
      average in the victim pick.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>