Set new allocated bfqq to bic or remove freed bfqq from bic are both
protected by bfqd->lock, however bfq_limit_depth() is deferencing bfqq
from bic without the lock, this can lead to UAF if the io_context is
shared by multiple tasks.
For example, test bfq with io_uring can trigger following UAF in v6.6:
==================================================================
BUG: KASAN: slab-use-after-free in bfqq_group+0x15/0x50
Call Trace:
<TASK>
dump_stack_lvl+0x47/0x80
print_address_description.constprop.0+0x66/0x300
print_report+0x3e/0x70
kasan_report+0xb4/0xf0
bfqq_group+0x15/0x50
bfqq_request_over_limit+0x130/0x9a0
bfq_limit_depth+0x1b5/0x480
__blk_mq_alloc_requests+0x2b5/0xa00
blk_mq_get_new_requests+0x11d/0x1d0
blk_mq_submit_bio+0x286/0xb00
submit_bio_noacct_nocheck+0x331/0x400
__block_write_full_folio+0x3d0/0x640
writepage_cb+0x3b/0xc0
write_cache_pages+0x254/0x6c0
write_cache_pages+0x254/0x6c0
do_writepages+0x192/0x310
filemap_fdatawrite_wbc+0x95/0xc0
__filemap_fdatawrite_range+0x99/0xd0
filemap_write_and_wait_range.part.0+0x4d/0xa0
blkdev_read_iter+0xef/0x1e0
io_read+0x1b6/0x8a0
io_issue_sqe+0x87/0x300
io_wq_submit_work+0xeb/0x390
io_worker_handle_work+0x24d/0x550
io_wq_worker+0x27f/0x6c0
ret_from_fork_asm+0x1b/0x30
</TASK>
Allocated by task 808602:
kasan_save_stack+0x1e/0x40
kasan_set_track+0x21/0x30
__kasan_slab_alloc+0x83/0x90
kmem_cache_alloc_node+0x1b1/0x6d0
bfq_get_queue+0x138/0xfa0
bfq_get_bfqq_handle_split+0xe3/0x2c0
bfq_init_rq+0x196/0xbb0
bfq_insert_request.isra.0+0xb5/0x480
bfq_insert_requests+0x156/0x180
blk_mq_insert_request+0x15d/0x440
blk_mq_submit_bio+0x8a4/0xb00
submit_bio_noacct_nocheck+0x331/0x400
__blkdev_direct_IO_async+0x2dd/0x330
blkdev_write_iter+0x39a/0x450
io_write+0x22a/0x840
io_issue_sqe+0x87/0x300
io_wq_submit_work+0xeb/0x390
io_worker_handle_work+0x24d/0x550
io_wq_worker+0x27f/0x6c0
ret_from_fork+0x2d/0x50
ret_from_fork_asm+0x1b/0x30
Freed by task 808589:
kasan_save_stack+0x1e/0x40
kasan_set_track+0x21/0x30
kasan_save_free_info+0x27/0x40
__kasan_slab_free+0x126/0x1b0
kmem_cache_free+0x10c/0x750
bfq_put_queue+0x2dd/0x770
__bfq_insert_request.isra.0+0x155/0x7a0
bfq_insert_request.isra.0+0x122/0x480
bfq_insert_requests+0x156/0x180
blk_mq_dispatch_plug_list+0x528/0x7e0
blk_mq_flush_plug_list.part.0+0xe5/0x590
__blk_flush_plug+0x3b/0x90
blk_finish_plug+0x40/0x60
do_writepages+0x19d/0x310
filemap_fdatawrite_wbc+0x95/0xc0
__filemap_fdatawrite_range+0x99/0xd0
filemap_write_and_wait_range.part.0+0x4d/0xa0
blkdev_read_iter+0xef/0x1e0
io_read+0x1b6/0x8a0
io_issue_sqe+0x87/0x300
io_wq_submit_work+0xeb/0x390
io_worker_handle_work+0x24d/0x550
io_wq_worker+0x27f/0x6c0
ret_from_fork+0x2d/0x50
ret_from_fork_asm+0x1b/0x30
Fix the problem by protecting bic_to_bfqq() with bfqd->lock.
CC: Jan Kara <jack@suse.cz>
Fixes: 76f1df88bbc2 ("bfq: Limit number of requests consumed by each cgroup")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20241129091509.2227136-1-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This reverts commit bc3b1e9e7c50e1de0f573eea3871db61dd4787de.
The bic is associated with sync_bfqq, and bfq_release_process_ref cannot
be put into bfq_put_cooperator.
kasan report:
[ 400.347277] ==================================================================
[ 400.347287] BUG: KASAN: slab-use-after-free in bic_set_bfqq+0x200/0x230
[ 400.347420] Read of size 8 at addr ffff88881cab7d60 by task dockerd/5800
[ 400.347430]
[ 400.347436] CPU: 24 UID: 0 PID: 5800 Comm: dockerd Kdump: loaded Tainted: G E 6.12.0 #32
[ 400.347450] Tainted: [E]=UNSIGNED_MODULE
[ 400.347454] Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.20192059.B64.2207280713 07/28/2022
[ 400.347460] Call Trace:
[ 400.347464] <TASK>
[ 400.347468] dump_stack_lvl+0x5d/0x80
[ 400.347490] print_report+0x174/0x505
[ 400.347521] kasan_report+0xe0/0x160
[ 400.347541] bic_set_bfqq+0x200/0x230
[ 400.347549] bfq_bic_update_cgroup+0x419/0x740
[ 400.347560] bfq_bio_merge+0x133/0x320
[ 400.347584] blk_mq_submit_bio+0x1761/0x1e20
[ 400.347625] __submit_bio+0x28b/0x7b0
[ 400.347664] submit_bio_noacct_nocheck+0x6b2/0xd30
[ 400.347690] iomap_readahead+0x50c/0x680
[ 400.347731] read_pages+0x17f/0x9c0
[ 400.347785] page_cache_ra_unbounded+0x366/0x4a0
[ 400.347795] filemap_fault+0x83d/0x2340
[ 400.347819] __xfs_filemap_fault+0x11a/0x7d0 [xfs]
[ 400.349256] __do_fault+0xf1/0x610
[ 400.349270] do_fault+0x977/0x11a0
[ 400.349281] __handle_mm_fault+0x5d1/0x850
[ 400.349314] handle_mm_fault+0x1f8/0x560
[ 400.349324] do_user_addr_fault+0x324/0x970
[ 400.349337] exc_page_fault+0x76/0xf0
[ 400.349350] asm_exc_page_fault+0x26/0x30
[ 400.349360] RIP: 0033:0x55a480d77375
[ 400.349384] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 3b 66 10 0f 86 ae 02 00 00 55 48 89 e5 48 83 ec 58 48 8b 10 <83> 7a 10 00 0f 84 27 02 00 00 44 0f b6 42 28 44 0f b6 4a 29 41 80
[ 400.349392] RSP: 002b:00007f18c37fd8b8 EFLAGS: 00010216
[ 400.349401] RAX: 00007f18c37fd9d0 RBX: 0000000000000000 RCX: 0000000000000000
[ 400.349407] RDX: 000055a484407d38 RSI: 000000c000e8b0c0 RDI: 0000000000000000
[ 400.349412] RBP: 00007f18c37fd910 R08: 000055a484017f60 R09: 000055a484066f80
[ 400.349417] R10: 0000000000194000 R11: 0000000000000005 R12: 0000000000000008
[ 400.349422] R13: 0000000000000000 R14: 000000c000476a80 R15: 0000000000000000
[ 400.349430] </TASK>
[ 400.349452]
[ 400.349454] Allocated by task 5800:
[ 400.349459] kasan_save_stack+0x30/0x50
[ 400.349469] kasan_save_track+0x14/0x30
[ 400.349475] __kasan_slab_alloc+0x89/0x90
[ 400.349482] kmem_cache_alloc_node_noprof+0xdc/0x2a0
[ 400.349492] bfq_get_queue+0x1ef/0x1100
[ 400.349502] __bfq_get_bfqq_handle_split+0x11a/0x510
[ 400.349511] bfq_insert_requests+0xf55/0x9030
[ 400.349519] blk_mq_flush_plug_list+0x446/0x14c0
[ 400.349527] __blk_flush_plug+0x27c/0x4e0
[ 400.349534] blk_finish_plug+0x52/0xa0
[ 400.349540] _xfs_buf_ioapply+0x739/0xc30 [xfs]
[ 400.350246] __xfs_buf_submit+0x1b2/0x640 [xfs]
[ 400.350967] xfs_buf_read_map+0x306/0xa20 [xfs]
[ 400.351672] xfs_trans_read_buf_map+0x285/0x7d0 [xfs]
[ 400.352386] xfs_imap_to_bp+0x107/0x270 [xfs]
[ 400.353077] xfs_iget+0x70d/0x1eb0 [xfs]
[ 400.353786] xfs_lookup+0x2ca/0x3a0 [xfs]
[ 400.354506] xfs_vn_lookup+0x14e/0x1a0 [xfs]
[ 400.355197] __lookup_slow+0x19c/0x340
[ 400.355204] lookup_one_unlocked+0xfc/0x120
[ 400.355211] ovl_lookup_single+0x1b3/0xcf0 [overlay]
[ 400.355255] ovl_lookup_layer+0x316/0x490 [overlay]
[ 400.355295] ovl_lookup+0x844/0x1fd0 [overlay]
[ 400.355351] lookup_one_qstr_excl+0xef/0x150
[ 400.355357] do_unlinkat+0x22a/0x620
[ 400.355366] __x64_sys_unlinkat+0x109/0x1e0
[ 400.355375] do_syscall_64+0x82/0x160
[ 400.355384] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 400.355393]
[ 400.355395] Freed by task 5800:
[ 400.355400] kasan_save_stack+0x30/0x50
[ 400.355407] kasan_save_track+0x14/0x30
[ 400.355413] kasan_save_free_info+0x3b/0x70
[ 400.355422] __kasan_slab_free+0x4f/0x70
[ 400.355429] kmem_cache_free+0x176/0x520
[ 400.355438] bfq_put_queue+0x67e/0x980
[ 400.355447] bfq_bic_update_cgroup+0x407/0x740
[ 400.355454] bfq_bio_merge+0x133/0x320
[ 400.355460] blk_mq_submit_bio+0x1761/0x1e20
[ 400.355467] __submit_bio+0x28b/0x7b0
[ 400.355473] submit_bio_noacct_nocheck+0x6b2/0xd30
[ 400.355480] iomap_readahead+0x50c/0x680
[ 400.355490] read_pages+0x17f/0x9c0
[ 400.355498] page_cache_ra_unbounded+0x366/0x4a0
[ 400.355505] filemap_fault+0x83d/0x2340
[ 400.355514] __xfs_filemap_fault+0x11a/0x7d0 [xfs]
[ 400.356204] __do_fault+0xf1/0x610
[ 400.356213] do_fault+0x977/0x11a0
[ 400.356221] __handle_mm_fault+0x5d1/0x850
[ 400.356230] handle_mm_fault+0x1f8/0x560
[ 400.356238] do_user_addr_fault+0x324/0x970
[ 400.356248] exc_page_fault+0x76/0xf0
[ 400.356258] asm_exc_page_fault+0x26/0x30
[ 400.356266]
[ 400.356269] The buggy address belongs to the object at ffff88881cab7bc0
which belongs to the cache bfq_queue of size 576
[ 400.356276] The buggy address is located 416 bytes inside of
freed 576-byte region [ffff88881cab7bc0, ffff88881cab7e00)
[ 400.356285]
[ 400.356287] The buggy address belongs to the physical page:
[ 400.356292] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff88881cab0b00 pfn:0x81cab0
[ 400.356300] head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 400.356323] flags: 0x50000000000040(head|node=1|zone=2)
[ 400.356331] page_type: f5(slab)
[ 400.356340] raw: 0050000000000040 ffff88880a00c280 dead000000000122 0000000000000000
[ 400.356347] raw: ffff88881cab0b00 00000000802e0025 00000001f5000000 0000000000000000
[ 400.356354] head: 0050000000000040 ffff88880a00c280 dead000000000122 0000000000000000
[ 400.356359] head: ffff88881cab0b00 00000000802e0025 00000001f5000000 0000000000000000
[ 400.356365] head: 0050000000000003 ffffea002072ac01 ffffffffffffffff 0000000000000000
[ 400.356370] head: 0000000000000008 0000000000000000 00000000ffffffff 0000000000000000
[ 400.356378] page dumped because: kasan: bad access detected
[ 400.356381]
[ 400.356383] Memory state around the buggy address:
[ 400.356387] ffff88881cab7c00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 400.356392] ffff88881cab7c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 400.356397] >ffff88881cab7d00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 400.356400] ^
[ 400.356405] ffff88881cab7d80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 400.356409] ffff88881cab7e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 400.356413] ==================================================================
Cc: stable@vger.kernel.org
Fixes: bc3b1e9e7c50 ("block, bfq: merge bfq_release_process_ref() into bfq_put_cooperator()")
Signed-off-by: Zach Wade <zachwade.k@gmail.com>
Cc: Ding Hui <dinghui@sangfor.com.cn>
Reviewed-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20241119153410.2546-1-zachwade.k@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Make code cleaner, there are no functional changes. Signed-off-by: Yu Kuai <yukuai3@huawei.com> Link: https://lore.kernel.org/r/20240909134154.954924-8-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
Now that 'bfqq_already_existing' is only used in one branch, it can be removed. There are no functional changes. Signed-off-by: Yu Kuai <yukuai3@huawei.com> Link: https://lore.kernel.org/r/20240909134154.954924-7-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
The local variable is used to call bfq_bfqq_resume_state() later, since 'bfqd->lock' is held, and bfqq status will not change between setting 'split' and calling bfq_bfqq_resume_state(), move forward bfq_bfqq_resume_state() so that 'split' can be removed. There are no functional chagnes. Signed-off-by: Yu Kuai <yukuai3@huawei.com> Link: https://lore.kernel.org/r/20240909134154.954924-6-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
Because bfq_put_cooperator() is always followed by bfq_release_process_ref(). Signed-off-by: Yu Kuai <yukuai3@huawei.com> Link: https://lore.kernel.org/r/20240909134154.954924-4-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
Original state:
Process 1 Process 2 Process 3 Process 4
(BIC1) (BIC2) (BIC3) (BIC4)
Λ | | |
\--------------\ \-------------\ \-------------\|
V V V
bfqq1--------->bfqq2---------->bfqq3----------->bfqq4
ref 0 1 2 4
After commit 0e456dba86c7 ("block, bfq: choose the last bfqq from merge
chain in bfq_setup_cooperator()"), if P1 issues a new IO:
Without the patch:
Process 1 Process 2 Process 3 Process 4
(BIC1) (BIC2) (BIC3) (BIC4)
Λ | | |
\------------------------------\ \-------------\|
V V
bfqq1--------->bfqq2---------->bfqq3----------->bfqq4
ref 0 0 2 4
bfqq3 will be used to handle IO from P1, this is not expected, IO
should be redirected to bfqq4;
With the patch:
-------------------------------------------
| |
Process 1 Process 2 Process 3 | Process 4
(BIC1) (BIC2) (BIC3) | (BIC4)
| | | |
\-------------\ \-------------\|
V V
bfqq1--------->bfqq2---------->bfqq3----------->bfqq4
ref 0 0 2 4
IO is redirected to bfqq4, however, procress reference of bfqq3 is still
2, while there is only P2 using it.
Fix the problem by calling bfq_merge_bfqqs() for each bfqq in the merge
chain. Also change bfqq_merge_bfqqs() to return new_bfqq to simplify
code.
Fixes: 0e456dba86c7 ("block, bfq: choose the last bfqq from merge chain in bfq_setup_cooperator()")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20240909134154.954924-3-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
After commit 42c306ed7233 ("block, bfq: don't break merge chain in
bfq_split_bfqq()"), if the current procress is the last holder of bfqq,
the bfqq can be freed after bfq_split_bfqq(). Hence recored the bfqq and
then access bfqq->waker_bfqq may trigger UAF. What's more, the waker_bfqq
may in the merge chain of bfqq, hence just recored waker_bfqq is still
not safe.
Fix the problem by adding a helper bfq_waker_bfqq() to check if
bfqq->waker_bfqq is in the merge chain, and current procress is the only
holder.
Fixes: 42c306ed7233 ("block, bfq: don't break merge chain in bfq_split_bfqq()")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20240909134154.954924-2-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Instead of open coding it, there are no functional changes. Signed-off-by: Yu Kuai <yukuai3@huawei.com> Link: https://lore.kernel.org/r/20240902130329.3787024-5-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe <axboe@kernel.dk>
Consider the following scenario:
Process 1 Process 2 Process 3 Process 4
(BIC1) (BIC2) (BIC3) (BIC4)
Λ | | |
\-------------\ \-------------\ \--------------\|
V V V
bfqq1--------->bfqq2---------->bfqq3----------->bfqq4
ref 0 1 2 4
If Process 1 issue a new IO and bfqq2 is found, and then bfq_init_rq()
decide to spilt bfqq2 by bfq_split_bfqq(). Howerver, procress reference
of bfqq2 is 1 and bfq_split_bfqq() just clear the coop flag, which will
break the merge chain.
Expected result: caller will allocate a new bfqq for BIC1
Process 1 Process 2 Process 3 Process 4
(BIC1) (BIC2) (BIC3) (BIC4)
| | |
\-------------\ \--------------\|
V V
bfqq1--------->bfqq2---------->bfqq3----------->bfqq4
ref 0 0 1 3
Since the condition is only used for the last bfqq4 when the previous
bfqq2 and bfqq3 are already splited. Fix the problem by checking if
bfqq is the last one in the merge chain as well.
Fixes: 36eca8948323 ("block, bfq: add Early Queue Merge (EQM)")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20240902130329.3787024-4-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>