io_uring/wait: don't sleep on an already expired timeout

When io_cqring_wait() gets to the point of sleeping and the wait's timeout
has already passed, io_cqring_schedule_timeout() still arms an hrtimer
for it and calls schedule(). The timer fires right away, but only after
the task has switched to idle, so getting -ETIME back costs a round trip
through idle plus a timer interrupt.

NAPI busy polling hits this on every wait that times out.
__io_napi_busy_loop() caps the busy poll at the remaining timeout, so when
the loop runs to its end without a completion, the timeout has expired by
the time we get here. The task then stops polling for several usec, and
packets that arrive meanwhile are left to the device interrupt. This
affects one of our production workloads, which currently works around it
by having another thread signal an eventfd on the ring before the timeout
expires.

A wait with a zero timeout, or one that runs out before the task gets to
sleep, pays the same cost. io_uring_wait_cqe_timeout() with nothing to
reap and a zero timeout takes ~11 usec and a voluntary context switch on
a Zen 5 desktop.

Return -ETIME without arming the timer once the timeout has passed, as
ep_poll() already does with ep_schedule_timeout(). A passed timeout ends
the wait with -ETIME whether or not a min_timeout is set, so the check
covers both. It costs one io_get_time() per wait that is about to sleep
with a timeout.

Signed-off-by: Ioannis Coward <ioanniscrd@gmail.com>
Link: https://patch.msgid.link/20261104104707.2031280-1-ioanniscrd@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

0 files changed, 0 insertions, 0 deletions