We had kernel panic, it is caused by unload module and last
close confirmation.
call trace:
[1196029.743127] free_sess+0x15/0x50 [rtrs_client]
[1196029.743128] rtrs_clt_close+0x4c/0x70 [rtrs_client]
[1196029.743129] ? rnbd_clt_unmap_device+0x1b0/0x1b0 [rnbd_client]
[1196029.743130] close_rtrs+0x25/0x50 [rnbd_client]
[1196029.743131] rnbd_client_exit+0x93/0xb99 [rnbd_client]
[1196029.743132] __x64_sys_delete_module+0x190/0x260
And in the crashdump confirmation kworker is also running.
PID: 6943 TASK: ffff9e2ac8098000 CPU: 4 COMMAND: "kworker/4:2"
#0 [ffffb206cf337c30] __schedule at ffffffff9f93f891
#1 [ffffb206cf337cc8] schedule at ffffffff9f93fe98
#2 [ffffb206cf337cd0] schedule_timeout at ffffffff9f943938
#3 [ffffb206cf337d50] wait_for_completion at ffffffff9f9410a7
#4 [ffffb206cf337da0] __flush_work at ffffffff9f08ce0e
#5 [ffffb206cf337e20] rtrs_clt_close_conns at ffffffffc0d5f668 [rtrs_client]
#6 [ffffb206cf337e48] rtrs_clt_close at ffffffffc0d5f801 [rtrs_client]
#7 [ffffb206cf337e68] close_rtrs at ffffffffc0d26255 [rnbd_client]
#8 [ffffb206cf337e78] free_sess at ffffffffc0d262ad [rnbd_client]
#9 [ffffb206cf337e88] rnbd_clt_put_dev at ffffffffc0d266a7 [rnbd_client]
The problem is both code path try to close same session, which lead to
panic.
To fix it, just skip the sess if the refcount already drop to 0.
Fixes: f7a7a5c228d4 ("block/rnbd: client: main functionality")
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Reviewed-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Adding name to the Contributors List Signed-off-by: Swapnil Ingle <ingleswapnil@gmail.com> Acked-by: Jack Wang <jinpu.wang@cloud.ionos.com> Acked-by: Danil Kipnis <danil.kipnis@cloud.ionos.com> Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
Since dynamically allocate sglist is used for rnbd_iu, we can't free sg
table after send_usr_msg since the callback function (cqe.done) could
still access the sglist.
Otherwise KASAN reports UAF issue:
[ 4856.600257] BUG: KASAN: use-after-free in dma_direct_unmap_sg+0x53/0x290
[ 4856.600772] Read of size 4 at addr ffff888206af3a98 by task swapper/1/0
[ 4856.601729] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Tainted: G W 5.10.0-pserver #5.10.0-1+feature+linux+next+20201214.1025+0910d71
[ 4856.601748] Hardware name: Supermicro Super Server/X11DDW-L, BIOS 3.3 02/21/2020
[ 4856.601766] Call Trace:
[ 4856.601785] <IRQ>
[ 4856.601822] dump_stack+0x99/0xcb
[ 4856.601856] ? dma_direct_unmap_sg+0x53/0x290
[ 4856.601888] print_address_description.constprop.7+0x1e/0x230
[ 4856.601913] ? freeze_kernel_threads+0x73/0x73
[ 4856.601965] ? mark_held_locks+0x29/0xa0
[ 4856.602019] ? dma_direct_unmap_sg+0x53/0x290
[ 4856.602039] ? dma_direct_unmap_sg+0x53/0x290
[ 4856.602079] kasan_report.cold.9+0x37/0x7c
[ 4856.602188] ? mlx5_ib_post_recv+0x430/0x520 [mlx5_ib]
[ 4856.602209] ? dma_direct_unmap_sg+0x53/0x290
[ 4856.602256] dma_direct_unmap_sg+0x53/0x290
[ 4856.602366] complete_rdma_req+0x188/0x4b0 [rtrs_client]
[ 4856.602451] ? rtrs_clt_close+0x80/0x80 [rtrs_client]
[ 4856.602535] ? mlx5_ib_poll_cq+0x48b/0x16e0 [mlx5_ib]
[ 4856.602589] ? radix_tree_insert+0x3a0/0x3a0
[ 4856.602610] ? do_raw_spin_lock+0x119/0x1d0
[ 4856.602647] ? rwlock_bug.part.1+0x60/0x60
[ 4856.602740] rtrs_clt_rdma_done+0x3f7/0x670 [rtrs_client]
[ 4856.602804] ? rtrs_clt_rdma_cm_handler+0xda0/0xda0 [rtrs_client]
[ 4856.602857] ? check_flags.part.31+0x6c/0x1f0
[ 4856.602927] ? rcu_read_lock_sched_held+0xaf/0xe0
[ 4856.602963] ? rcu_read_lock_bh_held+0xc0/0xc0
[ 4856.603137] __ib_process_cq+0x10a/0x350 [ib_core]
[ 4856.603309] ib_poll_handler+0x41/0x1c0 [ib_core]
[ 4856.603358] irq_poll_softirq+0xe6/0x280
[ 4856.603392] ? lockdep_hardirqs_on_prepare+0x111/0x210
[ 4856.603446] __do_softirq+0x10d/0x646
[ 4856.603540] asm_call_irq_on_stack+0x12/0x20
[ 4856.603563] </IRQ>
[ 4856.605096] Allocated by task 8914:
[ 4856.605510] kasan_save_stack+0x19/0x40
[ 4856.605532] __kasan_kmalloc.constprop.7+0xc1/0xd0
[ 4856.605552] __kmalloc+0x155/0x320
[ 4856.605574] __sg_alloc_table+0x155/0x1c0
[ 4856.605594] sg_alloc_table+0x1f/0x50
[ 4856.605620] send_msg_sess_info+0x119/0x2e0 [rnbd_client]
[ 4856.605646] remap_devs+0x71/0x210 [rnbd_client]
[ 4856.605676] init_sess+0xad8/0xe10 [rtrs_client]
[ 4856.605706] rtrs_clt_reconnect_work+0xd6/0x170 [rtrs_client]
[ 4856.605728] process_one_work+0x521/0xa90
[ 4856.605748] worker_thread+0x65/0x5b0
[ 4856.605769] kthread+0x1f2/0x210
[ 4856.605789] ret_from_fork+0x22/0x30
[ 4856.606159] Freed by task 8914:
[ 4856.606559] kasan_save_stack+0x19/0x40
[ 4856.606580] kasan_set_track+0x1c/0x30
[ 4856.606601] kasan_set_free_info+0x1b/0x30
[ 4856.606622] __kasan_slab_free+0x108/0x150
[ 4856.606642] slab_free_freelist_hook+0x64/0x190
[ 4856.606661] kfree+0xe2/0x650
[ 4856.606681] __sg_free_table+0xa4/0x100
[ 4856.606707] send_msg_sess_info+0x1d6/0x2e0 [rnbd_client]
[ 4856.606733] remap_devs+0x71/0x210 [rnbd_client]
[ 4856.606763] init_sess+0xad8/0xe10 [rtrs_client]
[ 4856.606792] rtrs_clt_reconnect_work+0xd6/0x170 [rtrs_client]
[ 4856.606813] process_one_work+0x521/0xa90
[ 4856.606833] worker_thread+0x65/0x5b0
[ 4856.606853] kthread+0x1f2/0x210
[ 4856.606872] ret_from_fork+0x22/0x30
The solution is to free iu's sgtable after the iu is not used anymore.
And also move sg_alloc_table into rnbd_get_iu accordingly.
Fixes: 5a1328d0c3a7 ("block/rnbd-clt: Dynamically allocate sglist for rnbd_iu")
Signed-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
KASAN detect following BUG:
[ 778.215311] ==================================================================
[ 778.216696] BUG: KASAN: use-after-free in rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.219037] Read of size 8 at addr ffff88b1d6516c28 by task tee/8842
[ 778.220500] CPU: 37 PID: 8842 Comm: tee Kdump: loaded Not tainted 5.10.0-pserver #5.10.0-1+feature+linux+next+20201214.1025+0910d71
[ 778.220529] Hardware name: Supermicro Super Server/X11DDW-L, BIOS 3.3 02/21/2020
[ 778.220555] Call Trace:
[ 778.220609] dump_stack+0x99/0xcb
[ 778.220667] ? rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.220715] print_address_description.constprop.7+0x1e/0x230
[ 778.220750] ? freeze_kernel_threads+0x73/0x73
[ 778.220896] ? rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.220932] ? rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.220994] kasan_report.cold.9+0x37/0x7c
[ 778.221066] ? kobject_put+0x80/0x270
[ 778.221102] ? rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.221184] rnbd_srv_sess_dev_force_close+0x38/0x60 [rnbd_server]
[ 778.221240] rnbd_srv_dev_session_force_close_store+0x6a/0xc0 [rnbd_server]
[ 778.221304] ? sysfs_file_ops+0x90/0x90
[ 778.221353] kernfs_fop_write+0x141/0x240
[ 778.221451] vfs_write+0x142/0x4d0
[ 778.221553] ksys_write+0xc0/0x160
[ 778.221602] ? __ia32_sys_read+0x50/0x50
[ 778.221684] ? lockdep_hardirqs_on_prepare+0x13d/0x210
[ 778.221718] ? syscall_enter_from_user_mode+0x1c/0x50
[ 778.221821] do_syscall_64+0x33/0x40
[ 778.221862] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 778.221896] RIP: 0033:0x7f4affdd9504
[ 778.221928] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 778.221956] RSP: 002b:00007fffebb36b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 778.222011] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4affdd9504
[ 778.222038] RDX: 0000000000000002 RSI: 00007fffebb36c50 RDI: 0000000000000003
[ 778.222066] RBP: 00007fffebb36c50 R08: 0000556a151aa600 R09: 00007f4affeb1540
[ 778.222094] R10: fffffffffffffc19 R11: 0000000000000246 R12: 0000556a151aa520
[ 778.222121] R13: 0000000000000002 R14: 00007f4affea6760 R15: 0000000000000002
[ 778.222764] Allocated by task 3212:
[ 778.223285] kasan_save_stack+0x19/0x40
[ 778.223316] __kasan_kmalloc.constprop.7+0xc1/0xd0
[ 778.223347] kmem_cache_alloc_trace+0x186/0x350
[ 778.223382] rnbd_srv_rdma_ev+0xf16/0x1690 [rnbd_server]
[ 778.223422] process_io_req+0x4d1/0x670 [rtrs_server]
[ 778.223573] __ib_process_cq+0x10a/0x350 [ib_core]
[ 778.223709] ib_cq_poll_work+0x31/0xb0 [ib_core]
[ 778.223743] process_one_work+0x521/0xa90
[ 778.223773] worker_thread+0x65/0x5b0
[ 778.223802] kthread+0x1f2/0x210
[ 778.223833] ret_from_fork+0x22/0x30
[ 778.224296] Freed by task 8842:
[ 778.224800] kasan_save_stack+0x19/0x40
[ 778.224829] kasan_set_track+0x1c/0x30
[ 778.224860] kasan_set_free_info+0x1b/0x30
[ 778.224889] __kasan_slab_free+0x108/0x150
[ 778.224919] slab_free_freelist_hook+0x64/0x190
[ 778.224947] kfree+0xe2/0x650
[ 778.224982] rnbd_destroy_sess_dev+0x2fa/0x3b0 [rnbd_server]
[ 778.225011] kobject_put+0xda/0x270
[ 778.225046] rnbd_srv_sess_dev_force_close+0x30/0x60 [rnbd_server]
[ 778.225081] rnbd_srv_dev_session_force_close_store+0x6a/0xc0 [rnbd_server]
[ 778.225111] kernfs_fop_write+0x141/0x240
[ 778.225140] vfs_write+0x142/0x4d0
[ 778.225169] ksys_write+0xc0/0x160
[ 778.225198] do_syscall_64+0x33/0x40
[ 778.225227] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 778.226506] The buggy address belongs to the object at ffff88b1d6516c00
which belongs to the cache kmalloc-512 of size 512
[ 778.227464] The buggy address is located 40 bytes inside of
512-byte region [ffff88b1d6516c00, ffff88b1d6516e00)
The problem is in the sess_dev release function we call
rnbd_destroy_sess_dev, and could free the sess_dev already, but we still
set the keep_id in rnbd_srv_sess_dev_force_close, which lead to use
after free.
To fix it, move the keep_id before the sysfs removal, and cache the
rnbd_srv_session for lock accessing,
Fixes: 786998050cbc ("block/rnbd-srv: close a mapped device from server side.")
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Reviewed-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
lkp reboot following build error:
drivers/block/rnbd/rnbd-clt.c: In function 'rnbd_softirq_done_fn':
>> drivers/block/rnbd/rnbd-clt.c:387:2: error: implicit declaration of function 'sg_free_table_chained' [-Werror=implicit-function-declaration]
387 | sg_free_table_chained(&iu->sgt, RNBD_INLINE_SG_CNT);
| ^~~~~~~~~~~~~~~~~~~~~
The reason is CONFIG_SG_POOL is not enabled in the config, to
avoid such failure, select SG_POOL in Kconfig for RNBD_CLIENT.
Fixes: 5a1328d0c3a7 ("block/rnbd-clt: Dynamically allocate sglist for rnbd_iu")
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Without crc32, the driver fails to link:
arm-linux-gnueabi-ld: drivers/block/rsxx/config.o: in function `rsxx_load_config':
config.c:(.text+0x124): undefined reference to `crc32_le'
Fixes: 8722ff8cdbfa ("block: IBM RamSan 70/80 device driver")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Pull block fixes from Jens Axboe:
"A few stragglers in here, but mostly just straight fixes. In
particular:
- Set of rnbd fixes for issues around changes for the merge window
(Gioh, Jack, Md Haris Iqbal)
- iocost tracepoint addition (Baolin)
- Copyright/maintainers update (Christoph)
- Remove old blk-mq fast path CPU warning (Daniel)
- loop max_part fix (Josh)
- Remote IPI threaded IRQ fix (Sebastian)
- dasd stable fixes (Stefan)
- bcache merge window fixup and style fixup (Yi, Zheng)"
* tag 'block-5.11-2020-12-23' of git://git.kernel.dk/linux-block:
md/bcache: convert comma to semicolon
bcache:remove a superfluous check in register_bcache
block: update some copyrights
block: remove a pointless self-reference in block_dev.c
MAINTAINERS: add fs/block_dev.c to the block section
blk-mq: Don't complete on a remote CPU in force threaded mode
s390/dasd: fix list corruption of lcu list
s390/dasd: fix list corruption of pavgroup group list
s390/dasd: prevent inconsistent LCU device data
s390/dasd: fix hanging device offline processing
blk-iocost: Add iocg idle state tracepoint
nbd: Respect max_part for all partition scans
block/rnbd-clt: Does not request pdu to rtrs-clt
block/rnbd-clt: Dynamically allocate sglist for rnbd_iu
block/rnbd: Set write-back cache and fua same to the target device
block/rnbd: Fix typos
block/rnbd-srv: Protect dev session sysfs removal
block/rnbd-clt: Fix possible memleak
block/rnbd-clt: Get rid of warning regarding size argument in strlcpy
blk-mq: Remove 'running from the wrong CPU' warning
Pull more xen updates from Juergen Gross: "Some minor cleanup patches and a small series disentangling some Xen related Kconfig options" * tag 'for-linus-5.11-rc1b-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip: xen: Kconfig: remove X86_64 depends from XEN_512GB xen/manage: Fix fall-through warnings for Clang xen-blkfront: Fix fall-through warnings for Clang xen: remove trailing semicolon in macro definition xen: Kconfig: nest Xen guest options xen: Remove Xen PVH/PVHVM dependency on PCI x86/xen: Convert to DEFINE_SHOW_ATTRIBUTE
Pull powerpc updates from Michael Ellerman: - Switch to the generic C VDSO, as well as some cleanups of our VDSO setup/handling code. - Support for KUAP (Kernel User Access Prevention) on systems using the hashed page table MMU, using memory protection keys. - Better handling of PowerVM SMT8 systems where all threads of a core do not share an L2, allowing the scheduler to make better scheduling decisions. - Further improvements to our machine check handling. - Show registers when unwinding interrupt frames during stack traces. - Improvements to our pseries (PowerVM) partition migration code. - Several series from Christophe refactoring and cleaning up various parts of the 32-bit code. - Other smaller features, fixes & cleanups. Thanks to: Alan Modra, Alexey Kardashevskiy, Andrew Donnellan, Aneesh Kumar K.V, Ard Biesheuvel, Athira Rajeev, Balamuruhan S, Bill Wendling, Cédric Le Goater, Christophe Leroy, Christophe Lombard, Colin Ian King, Daniel Axtens, David Hildenbrand, Frederic Barrat, Ganesh Goudar, Gautham R. Shenoy, Geert Uytterhoeven, Giuseppe Sacco, Greg Kurz, Harish, Jan Kratochvil, Jordan Niethe, Kaixu Xia, Laurent Dufour, Leonardo Bras, Madhavan Srinivasan, Mahesh Salgaonkar, Mathieu Desnoyers, Nathan Lynch, Nicholas Piggin, Oleg Nesterov, Oliver O'Halloran, Oscar Salvador, Po-Hsu Lin, Qian Cai, Qinglang Miao, Randy Dunlap, Ravi Bangoria, Sachin Sant, Sandipan Das, Sebastian Andrzej Siewior , Segher Boessenkool, Srikar Dronamraju, Tyrel Datwyler, Uwe Kleine-König, Vincent Stehlé, Youling Tang, and Zhang Xiaoxu. * tag 'powerpc-5.11-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux: (304 commits) powerpc/32s: Fix cleanup_cpu_mmu_context() compile bug powerpc: Add config fragment for disabling -Werror powerpc/configs: Add ppc64le_allnoconfig target powerpc/powernv: Rate limit opal-elog read failure message powerpc/pseries/memhotplug: Quieten some DLPAR operations powerpc/ps3: use dma_mapping_error() powerpc: force inlining of csum_partial() to avoid multiple csum_partial() with GCC10 powerpc/perf: Fix Threshold Event Counter Multiplier width for P10 powerpc/mm: Fix hugetlb_free_pmd_range() and hugetlb_free_pud_range() KVM: PPC: Book3S HV: Fix mask size for emulated msgsndp KVM: PPC: fix comparison to bool warning KVM: PPC: Book3S: Assign boolean values to a bool variable powerpc: Inline setup_kup() powerpc/64s: Mark the kuap/kuep functions non __init KVM: PPC: Book3S HV: XIVE: Add a comment regarding VP numbering powerpc/xive: Improve error reporting of OPAL calls powerpc/xive: Simplify xive_do_source_eoi() powerpc/xive: Remove P9 DD1 flag XIVE_IRQ_FLAG_EOI_FW powerpc/xive: Remove P9 DD1 flag XIVE_IRQ_FLAG_MASK_FW powerpc/xive: Remove P9 DD1 flag XIVE_IRQ_FLAG_SHIFT_BUG ...
Pull ceph updates from Ilya Dryomov:
"The big ticket item here is support for msgr2 on-wire protocol, which
adds the option of full in-transit encryption using AES-GCM algorithm
(myself).
On top of that we have a series to avoid intermittent errors during
recovery with recover_session=clean and some MDS request encoding work
from Jeff, a cap handling fix and assorted observability improvements
from Luis and Xiubo and a good number of cleanups.
Luis also ran into a corner case with quotas which sadly means that we
are back to denying cross-quota-realm renames"
* tag 'ceph-for-5.11-rc1' of git://github.com/ceph/ceph-client: (59 commits)
libceph: drop ceph_auth_{create,update}_authorizer()
libceph, ceph: make use of __ceph_auth_get_authorizer() in msgr1
libceph, ceph: implement msgr2.1 protocol (crc and secure modes)
libceph: introduce connection modes and ms_mode option
libceph, rbd: ignore addr->type while comparing in some cases
libceph, ceph: get and handle cluster maps with addrvecs
libceph: factor out finish_auth()
libceph: drop ac->ops->name field
libceph: amend cephx init_protocol() and build_request()
libceph, ceph: incorporate nautilus cephx changes
libceph: safer en/decoding of cephx requests and replies
libceph: more insight into ticket expiry and invalidation
libceph: move msgr1 protocol specific fields to its own struct
libceph: move msgr1 protocol implementation to its own file
libceph: separate msgr1 protocol implementation
libceph: export remaining protocol independent infrastructure
libceph: export zero_page
libceph: rename and export con->flags bits
libceph: rename and export con->state states
libceph: make con->state an int
...