From 495f22c5a99ac38e527294cdc4f3972718b9bee0 Mon Sep 17 00:00:00 2001 From: Mani Tofigh Date: Fri, 6 Nov 2026 11:53:21 -0500 Subject: blk-mq: take quiesce_lock when removing a queue from the tag set Unquiescing a tag set can run the hw queues of a queue whose hctxs were already freed. blk_mq_unquiesce_tagset() walks the tag set's queues under quiesce_lock and rcu_read_lock(). Commit 71ec3c7d6db8 ("blk-mq: quiesce a queue that joins a quiesced tag set") added quiesce_lock to the walks and to adding a queue, but blk_mq_del_queue_tag_set() still removes a queue without it. The hctxs are then freed right away since RCU only keeps the request_queue itself alive: Task A (controller reset) Task B (namespace removal) blk_mq_unquiesce_tagset() /* finds q */ blk_mq_del_queue_tag_set(q) blk_mq_release(q) /* frees the hctxs */ blk_mq_run_hw_queues(q) /* reads freed hctxs */ The window is short. On PREEMPT_RT quiesce_lock is a sleeping lock and the walk can be preempted inside it, which is how it's hit here: [ 399.851277] BUG: KASAN: slab-use-after-free in blk_mq_run_hw_queues+0x4bd/0x550 [ 399.851328] Read of size 8 at addr ff11000003014148 by task kworker/u9:2/116 [ 399.851352] CPU: 1 UID: 0 PID: 116 Comm: kworker/u9:2 Not tainted 7.3.0-16299-g81cb9aa43246 #1 PREEMPT_RT [ 399.851425] Workqueue: nvme-reset-wq nvme_loop_reset_ctrl_work [ 399.851441] Call Trace: [ 399.851457] [ 399.851551] blk_mq_run_hw_queues+0x4bd/0x550 [ 399.851564] blk_mq_unquiesce_tagset+0x128/0x210 [ 399.851576] nvme_loop_reset_ctrl_work+0x78/0x4c0 [ 399.851600] process_one_work+0x92b/0x16d0 [ 399.851660] worker_thread+0x4e6/0xd60 [ 399.851707] kthread+0x300/0x3e0 [ 399.851727] ret_from_fork+0x4f0/0x6f0 [ 399.851776] ret_from_fork_asm+0x1a/0x30 [ 399.851793] ... [ 399.851988] Freed by task 111: [ 399.852040] kfree+0x1b3/0x4b0 [ 399.852050] blk_mq_release+0x400/0x550 [ 399.852060] blk_put_queue+0x94/0x120 [ 399.852068] disk_release+0x1bd/0x3f0 [ 399.852102] nvme_free_ns+0x39/0x270 [ 399.852113] nvme_remove_nsid_range+0x2d1/0x4a0 [ 399.852123] nvme_scan_ns_list+0x2cc/0x390 [ 399.852134] nvme_scan_work+0x421/0x9f0 ... With BLK_CGROUP the disk's root blkg holds the queue until a grace period later, which hides this for NVMe namespaces. It's still open without BLK_CGROUP, and from reading the code also for queues freed without a disk (e.g., SCSI device queues on a UFS host). Take quiesce_lock around the list_del_rcu() as well so that the walk either finishes with a queue before it leaves the list or never sees it, and say so in the comment of quiesce_lock. With the fix, KASAN doesn't report the use-after-free anymore. This bug was identified and confirmed with the help of a deterministic hypervisor currently under development. The C reproducer below however hits it about once every 20 minutes on a PREEMPT_RT kernel. Fixes: 59e25ef2b413 ("block: Use RCU in blk_mq_[un]quiesce_tagset() instead of set->tag_list_lock") Cc: stable@vger.kernel.org Reviewed-by: Mohamed Khalfella Signed-off-by: Mani Tofigh Reviewed-by: Christoph Hellwig Reviewed-by: Nilay Shroff Link: https://patch.msgid.link/20261106-blk-mq-del-quiesce-lock-v2-1-cd4f72a6eb03@columbia.edu Signed-off-by: Jens Axboe --- include/linux/blk-mq.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'include/linux') diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 1a2f8d48adc4..3c9667854c46 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -524,7 +524,8 @@ enum hctx_type { * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. * @quiesce_lock: Serializes blk_mq_[un]quiesce_tagset() against each other - * and against a request queue being added to @tag_list. + * and against a request queue being added to or removed from + * @tag_list. * @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without * blk_mq_unquiesce_tagset(). * @srcu: Use as lock when type of the request queue is blocking -- cgit v1.2.3 From 6bd856cc99963b786cb530141745ef733de76cc3 Mon Sep 17 00:00:00 2001 From: Md Haris Iqbal Date: Wed, 11 Nov 2026 22:58:37 +0100 Subject: block: allow error injection rules to delay bios Error injection can only fail a bio today. Add a delay_us option so that a rule can hold a bio back first, to model a slow device. If a matching rule has delay_us set, the bio is held for that long. It is then failed if the rule also has a status, or resubmitted otherwise. The resubmission is done through submit_bio_noacct_nocheck(), which is where the injection hook sits. To avoid the same delay rule being applied to it again, mark the bio with the BIO_ERROR_INJECTED flag as soon as a rule matches. This also prevents bios that are split later and resubmitted from being evaluated for delay (or error injection) again. __bio_clone() does not propagate the flag, so a clone a stacking driver aims at a lower device is still evaluated. The bio is submitted from a workqueue rather than from the timer, because submitting a bio can sleep. A REQ_NOWAIT bio is failed with BLK_STS_AGAIN instead of being delayed. Values above 600 seconds are rejected. Cc: Christoph Hellwig Signed-off-by: Md Haris Iqbal Link: https://patch.msgid.link/20261111215838.103356-3-haris.iqbal@linux.dev Signed-off-by: Jens Axboe --- include/linux/blk_types.h | 1 + 1 file changed, 1 insertion(+) (limited to 'include/linux') diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 0cc09b975d8f..1a5812fa4052 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -328,6 +328,7 @@ enum { BIO_ZONE_WRITE_PLUGGING, /* bio handled through zone write plugging */ BIO_EMULATES_ZONE_APPEND, /* bio emulates a zone append operation */ BIO_COMPLETE_IN_TASK, /* complete bi_end_io() in task context */ + BIO_ERROR_INJECTED, /* error injection rules already applied */ BIO_FLAG_LAST }; -- cgit v1.2.3