diff options
| author | Cinq Prairie <notprairie@gmail.com> | 2026-10-27 21:48:40 +0100 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2026-11-09 14:48:18 -0700 |
| commit | 39c7210e970200f8b17de3767c73efe64173a95f (patch) | |
| tree | f81832d640000256b25c05cf297b45ccd1437cc4 /kernel/debug | |
| parent | cf6542493e35ab9e72ac07da10a178ac78db5a16 (diff) | |
io_uring/napi: don't end the single NAPI busy poll early on need_resched()
With a single registered NAPI, io_napi_blocking_busy_loop() hands
io_napi_busy_loop_should_end() to napi_busy_loop_rcu() as loop_end and
runs its own loop once, relying on __napi_busy_loop() to spin until
that predicate fires. napi_busy_loop_rcu() sets NAPI_F_END_ON_RESCHED,
so __napi_busy_loop() also returns on the first need_resched(). io_uring
treats the return as the end of the busy poll period and sleeps in
schedule_hrtimeout() for the rest of the wait timeout.
With two or more NAPIs the inner loop is single-shot and io_uring's own
loop runs until its predicate is met, so busy_poll_to is honoured. The
same registration therefore behaves differently for one NAPI than for
several.
Make the busy poll deadline absolute, stored in io_wait_queue, so that
io_napi_busy_loop_should_end() no longer depends on the start_time the
napi loop snapshots on every entry. The single NAPI loop can then be
re-entered after an early need_resched() exit if there is budget left,
and the loop condition drops the "run once" special case. The fast path
that holds the NAPI across polls is kept.
Measured on io_uring-7.4 at 9c207aa58e54, x86-64, HZ=1000,
PREEMPT_DYNAMIC (lazy), no CPU isolation, low RX traffic, r8169 with one
RX queue, prefer_busy_poll, busy_poll_to = wait timeout = 10 ms.
"waker" is a SCHED_FIFO task pinned to the polling CPU waking every 1 ms:
1 NAPI, waker, before: busy 0.5-1.5 ms per 10 ms wait, ~0.8M polls/s
1 NAPI, waker, after: busy 10 ms per 10 ms wait, ~16.2M polls/s
1 NAPI, idle, before: busy 10 ms per 10 ms wait, ~15.3M polls/s
1 NAPI, idle, after: busy 10 ms per 10 ms wait, ~16.2M polls/s
2 NAPIs, waker, before: busy 10 ms per 10 ms wait, ~7.2M polls/s
2 NAPIs, waker, after: busy 10 ms per 10 ms wait, ~7.1M polls/s
The held-NAPI fast path polls about twice as often as the single-shot
path used for several NAPIs, which is why it is kept.
Fixes: 8d0c12a80cde ("io-uring: add napi busy poll support")
Cc: stable@vger.kernel.org
Signed-off-by: Cinq Prairie <notprairie@gmail.com>
Link: https://patch.msgid.link/20261027204840.7823-1-notprairie@gmail.com
[axboe: trim comments, drop __always_unused]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'kernel/debug')
0 files changed, 0 insertions, 0 deletions
