diff options
| author | Kyle Zeng <kylebot@openai.com> | 2026-10-25 02:43:10 +0000 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2026-10-28 10:50:34 -0600 |
| commit | 3de16279b43dc285ed2222aba06cebdc82fdb28b (patch) | |
| tree | b3bdc8fa304efa66ab68568b4ac4d08e8d744c24 /io_uring | |
| parent | 37e425460b7c41687e51e3254b1eb1d799da9614 (diff) | |
io_uring: defer foreign io-wq enqueues to task_work
An IOPOLL ring without DEFER_TASKRUN can be reaped by a task other than
the one that submitted the I/O. If a polled request is marked
REQ_F_REISSUE, io_free_batch_list() calls io_queue_iowq() with the
original tctx. Ordinary task exit can tear down tctx->io_wq after
draining only the tracked requests. The live request pins the task, but
not its io-wq, so the foreign task can race io_wq_put_and_exit() and
enqueue through a freed or NULL queue.
Forward foreign userspace enqueues to the owning task before accessing
tctx->io_wq. Use a dedicated task_work callback so reissues still go
straight to io-wq, and fail them if task_work is running in cancellation
mode. Preserve the existing PF_KTHREAD cancellation path. This also
covers the other callers of io_queue_iowq() without changing the
worker-owned enqueue path.
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'io_uring')
| -rw-r--r-- | io_uring/io_uring.c | 25 |
1 files changed, 23 insertions, 2 deletions
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 615933a6e8ff..03724419c4e1 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -423,6 +423,17 @@ static __always_inline bool io_assign_file(struct io_kiocb *req, return !!req->file; } +static void io_req_queue_iowq_tw(struct io_tw_req tw_req, io_tw_token_t tw) +{ + struct io_kiocb *req = tw_req.req; + + io_tw_lock(req->ctx, tw); + if (unlikely(tw.cancel)) + io_req_defer_failed(req, -ECANCELED); + else + io_queue_iowq(req); +} + void io_queue_iowq(struct io_kiocb *req) { struct io_uring_task *tctx = req->tctx; @@ -430,10 +441,17 @@ void io_queue_iowq(struct io_kiocb *req) BUG_ON(!tctx); lockdep_assert_held(&req->ctx->uring_lock); - if ((current->flags & PF_KTHREAD) || !tctx->io_wq) { - io_req_task_queue_fail(req, -ECANCELED); + if (current->flags & PF_KTHREAD) + goto cancel; + + /* Only the submitter can safely access its io-wq during task exit. */ + if (unlikely(current != tctx->task)) { + req->io_task_work.func = io_req_queue_iowq_tw; + io_req_task_work_add(req); return; } + if (!tctx->io_wq) + goto cancel; /* * The file is looked up as late as possible, as it may be the result @@ -462,6 +480,9 @@ void io_queue_iowq(struct io_kiocb *req) trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work)); io_wq_enqueue(tctx->io_wq, &req->work); + return; +cancel: + io_req_task_queue_fail(req, -ECANCELED); } unsigned io_linked_nr(struct io_kiocb *req) |