linux-toradex.git/drivers/infiniband/hw/irdma, branch master Linux kernel for Apalis and Colibri modules RDMA/irdma: Bound in-flight CQP ops across deferred completions 2026-10-20T10:21:15+00:00 Sunny Tiwari sunnytiwari@google.com 2026-10-17T05:28:56+00:00 c3e8e9451e6f1b69f685b75e0fbf14f3d0fb2539 On IRDMA_GEN_3+ devices, CQP operations can complete in two stages: Stage-1 on the CCQ acknowledges receipt and moves an entry from cqp->ooo_avail to cqp->ooo_pnd, while Stage-2 signals final completion via an AEQ event or a subsequent CCQ CQE. In irdma_sc_ccq_get_cqe_info(), Stage-1 processing advances cqp->sq_ring.tail so the SQ descriptor slot can be reused once consumed by hardware, whereas cqp->completed_ops is only incremented on Stage-2. Because irdma_cqp_ring_full() only checks cqp->sq_ring, a burst of backlogged commands in dev->cqp_cmd_head allows irdma_process_bh() to submit more concurrent in-flight CQP commands than the maximum supported limit (cqp->sq_size - 1) while earlier commands await Stage-2. This causes CQP completions to be lost and synchronous waiters to time out with -ETIMEDOUT (-110). Fix this by: - Updating irdma_cqp_ring_full() to bound total in-flight CQP operations (cqp->requested_ops - cqp->completed_ops) to (cqp->sq_size - 1). - Calling irdma_process_bh() at the end of irdma_process_ae_def_cmpl() so backlogged commands are drained after Stage-2 AEQ completions increment cqp->completed_ops and free in-flight capacity. Fixes: c7db0abe5f2b ("RDMA/irdma: Add GEN3 CQP support with deferred completions") Assisted-by: Gemini Signed-off-by: Sunny Tiwari <sunnytiwari@google.com> Link: https://patch.msgid.link/20261017052856.3570063-1-sunnytiwari@google.com Reviewed-by: Jacob Moroni <jmoroni@google.com> Signed-off-by: Leon Romanovsky <leon@kernel.org> 
On IRDMA_GEN_3+ devices, CQP operations can complete in two stages:
Stage-1 on the CCQ acknowledges receipt and moves an entry from
cqp->ooo_avail to cqp->ooo_pnd, while Stage-2 signals final completion
via an AEQ event or a subsequent CCQ CQE.

In irdma_sc_ccq_get_cqe_info(), Stage-1 processing advances
cqp->sq_ring.tail so the SQ descriptor slot can be reused once consumed
by hardware, whereas cqp->completed_ops is only incremented on Stage-2.
Because irdma_cqp_ring_full() only checks cqp->sq_ring, a burst of
backlogged commands in dev->cqp_cmd_head allows irdma_process_bh() to
submit more concurrent in-flight CQP commands than the maximum supported
limit (cqp->sq_size - 1) while earlier commands await Stage-2. This
causes CQP completions to be lost and synchronous waiters to time out
with -ETIMEDOUT (-110).

Fix this by:
  - Updating irdma_cqp_ring_full() to bound total in-flight CQP
    operations (cqp->requested_ops - cqp->completed_ops) to
    (cqp->sq_size - 1).
  - Calling irdma_process_bh() at the end of irdma_process_ae_def_cmpl()
    so backlogged commands are drained after Stage-2 AEQ completions
    increment cqp->completed_ops and free in-flight capacity.

Fixes: c7db0abe5f2b ("RDMA/irdma: Add GEN3 CQP support with deferred completions")
Assisted-by: Gemini
Signed-off-by: Sunny Tiwari <sunnytiwari@google.com>
Link: https://patch.msgid.link/20261017052856.3570063-1-sunnytiwari@google.com
Reviewed-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Fix PBLE_LEVEL_2 FRWR generation 2026-10-18T12:11:08+00:00 Jacob Moroni jmoroni@google.com 2026-10-16T18:09:01+00:00 9ffd2aa8b53b044a61f939e8fba5ec250c5fddd9 When an FRWR is posted with only 1 page mapped, it uses a direct mapping, with the page-aligned DMA address provided in the WQE (reg_addr_pa). This address is obtained by reading it from the PBLE that was previously populated during ib_map_mr_sg(), but it did not account for whether the original allocation was PBLE_LEVEL_1 or PBLE_LEVEL_2 and just assumed it was always level 1. As a result, it would program incorrect addresses if the MR was originally allocated as PBLE_LEVEL_2. Additionally, multi-page PBLE_LEVEL_2 FRWRs were also broken because it was using the wrong chunk_size. Fix these issues by correctly populating the stag_info fields depending on whether it's a PBLE_LEVEL_1 or PBLE_LEVEL_2 MR and whether more than 1 page has been mapped or not. Tested with NFS over RDMA running various fio scenarios, with and without a modified kernel to force it to take the PBLE_LEVEL_2 path. Fixes: 137d264c6f63 ("RDMA/irdma: Add 2 level PBLE support for FMR") Suggested-by: Stefan Metzmacher <metze@samba.org> Link: https://lore.kernel.org/linux-rdma/20261012161135.494544-1-metze@samba.org/ Signed-off-by: Jacob Moroni <jmoroni@google.com> Link: https://patch.msgid.link/20261016180901.3292775-1-jmoroni@google.com Signed-off-by: Leon Romanovsky <leon@kernel.org> 
When an FRWR is posted with only 1 page mapped, it uses a
direct mapping, with the page-aligned DMA address provided
in the WQE (reg_addr_pa). This address is obtained by reading
it from the PBLE that was previously populated during ib_map_mr_sg(),
but it did not account for whether the original allocation was
PBLE_LEVEL_1 or PBLE_LEVEL_2 and just assumed it was always level 1.

As a result, it would program incorrect addresses if the MR was
originally allocated as PBLE_LEVEL_2.

Additionally, multi-page PBLE_LEVEL_2 FRWRs were also broken
because it was using the wrong chunk_size.

Fix these issues by correctly populating the stag_info fields
depending on whether it's a PBLE_LEVEL_1 or PBLE_LEVEL_2 MR and
whether more than 1 page has been mapped or not.

Tested with NFS over RDMA running various fio scenarios, with
and without a modified kernel to force it to take the PBLE_LEVEL_2
path.

Fixes: 137d264c6f63 ("RDMA/irdma: Add 2 level PBLE support for FMR")
Suggested-by: Stefan Metzmacher <metze@samba.org>
Link: https://lore.kernel.org/linux-rdma/20261012161135.494544-1-metze@samba.org/
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20261016180901.3292775-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Fix FRWR IRDMAQPSQ_PBLADDR field population 2026-10-15T18:44:18+00:00 Jacob Moroni jmoroni@google.com 2026-10-15T16:17:55+00:00 59df57262e6fa9e6f4aa76d6d324134777c702a0 Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated using a mask that was right-shifted, whereas it should have been the address being right-shifted. This ended up working fine in many cases because the lowermost 12 bits are always zero (the addresses are page aligned) and the uppermost 12 bits are usually zero too, but it could break for DMA addresses that are wider than 52 bits. Fix this by right-shifting the address instead of the mask, similar to how it was done in the original i40iw driver. Tested with NFS over RDMA with some added debug logs to ensure it was hitting both the direct (single page) path as well as multi page (< 512). Fixes: 3f49d6842569 ("RDMA/irdma: Implement HW Admin Queue OPs") Signed-off-by: Jacob Moroni <jmoroni@google.com> Link: https://patch.msgid.link/20261015161756.3078300-1-jmoroni@google.com Signed-off-by: Leon Romanovsky <leon@kernel.org> 
Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated
using a mask that was right-shifted, whereas it should have been the
address being right-shifted.

This ended up working fine in many cases because the lowermost 12 bits
are always zero (the addresses are page aligned) and the uppermost 12
bits are usually zero too, but it could break for DMA addresses that
are wider than 52 bits.

Fix this by right-shifting the address instead of the mask, similar
to how it was done in the original i40iw driver.

Tested with NFS over RDMA with some added debug logs to ensure it
was hitting both the direct (single page) path as well as multi
page (< 512).

Fixes: 3f49d6842569 ("RDMA/irdma: Implement HW Admin Queue OPs")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20261015161756.3078300-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Check MR contiguity before PBLE alloc 2026-10-15T15:38:16+00:00 Jacob Moroni jmoroni@google.com 2026-10-14T16:21:44+00:00 9165b3f7372bc1547549687f3a113c047ed94a36 The original logic would attempt to allocate PBLEs, then would check whether the region was contiguous _after_, and if so, would free the PBLEs and use a direct mapping instead. These unnecessary temporary PBLE allocations can technically fail (PBLEs are a finite resource), which can cause an MR registration failure even if the region is contiguous and should have succeeded. Fix this by checking whether the region is contiguous before allocating any PBLEs. This also provides an opportunity to consolidate some of the rereg_mr logic and use the existing ib_umem_is_contiguous() helper. Tested on GEN3 HW with a test program that does an extensive sweep of MR size/alignment/hugepage combinations, but most importantly, first/last byte reachable and -1/+1 unreachable when using contig regions with both aligned and unaligned start addresses, plus a kprobe on irdma_get_pble() to ensure we're actually hitting the new path. Ring MRs still have their own logic. That will be addressed next. Signed-off-by: Jacob Moroni <jmoroni@google.com> Link: https://patch.msgid.link/20261014162144.4053279-1-jmoroni@google.com Signed-off-by: Leon Romanovsky <leon@kernel.org> 
The original logic would attempt to allocate PBLEs, then would
check whether the region was contiguous _after_, and if so,
would free the PBLEs and use a direct mapping instead.

These unnecessary temporary PBLE allocations can technically
fail (PBLEs are a finite resource), which can cause an MR
registration failure even if the region is contiguous and
should have succeeded.

Fix this by checking whether the region is contiguous before
allocating any PBLEs. This also provides an opportunity to
consolidate some of the rereg_mr logic and use the existing
ib_umem_is_contiguous() helper.

Tested on GEN3 HW with a test program that does an extensive
sweep of MR size/alignment/hugepage combinations, but most
importantly, first/last byte reachable and -1/+1 unreachable
when using contig regions with both aligned and unaligned start
addresses, plus a kprobe on irdma_get_pble() to ensure we're
actually hitting the new path.

Ring MRs still have their own logic. That will be addressed next.

Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20261014162144.4053279-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Do not move a fresh RoCE QP to ERROR on RESET 2026-09-30T13:48:40+00:00 Nguyễn Văn Cao Nguyên nguyennvc@greennode.ai 2026-09-30T07:51:42+00:00 6de34bfb56ad8c24bf63b243b5f32b345d9cdd4d irdma_modify_qp_roce() handles IB_QPS_RESET like IB_QPS_ERR and moves the QP to ERROR. On a QP that was never moved to INIT, this makes the following RESET->INIT fail with -EINVAL, while ibv_query_qp() still reports RESET. RESET->RESET is a valid transition. The Mooncake transfer engine resets every new QP before INIT, so it cannot connect over an E810, while mlx5 accepts the same sequence: ibv_create_qp(RC) -> RESET ibv_modify_qp(RESET) -> 0 ibv_modify_qp(INIT, ...) -> -EINVAL Skip the hardware modify when the QP has not been moved to INIT yet (iwarp_state is IRDMA_QP_STATE_INVALID), as there is nothing to flush. RESET after INIT still goes to ERROR. Fixes: b48c24c2d710 ("RDMA/irdma: Implement device supported verb APIs") Assisted-by: LLM Signed-off-by: Nguyễn Văn Cao Nguyên <nguyennvc@greennode.ai> Link: https://patch.msgid.link/20260930-irdma-reset-fresh-qp-v1-1-065327274e2b@greennode.ai Signed-off-by: Leon Romanovsky <leon@kernel.org> 
irdma_modify_qp_roce() handles IB_QPS_RESET like IB_QPS_ERR and moves
the QP to ERROR. On a QP that was never moved to INIT, this makes the
following RESET->INIT fail with -EINVAL, while ibv_query_qp() still
reports RESET.

RESET->RESET is a valid transition. The Mooncake transfer engine
resets every new QP before INIT, so it cannot connect over an E810,
while mlx5 accepts the same sequence:

  ibv_create_qp(RC)        -> RESET
  ibv_modify_qp(RESET)     -> 0
  ibv_modify_qp(INIT, ...) -> -EINVAL

Skip the hardware modify when the QP has not been moved to INIT yet
(iwarp_state is IRDMA_QP_STATE_INVALID), as there is nothing to flush.
RESET after INIT still goes to ERROR.

Fixes: b48c24c2d710 ("RDMA/irdma: Implement device supported verb APIs")
Assisted-by: LLM
Signed-off-by: Nguyễn Văn Cao Nguyên <nguyennvc@greennode.ai>
Link: https://patch.msgid.link/20260930-irdma-reset-fresh-qp-v1-1-065327274e2b@greennode.ai
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Merge remote-tracking branch 'ionic/ionic-shared' into wip/leon-for-next 2026-09-30T13:07:27+00:00 Leon Romanovsky leon@kernel.org 2026-09-30T13:07:27+00:00 82b748439324b6364cc7fc71f0ca5f886c779553 Preparatory patch to support SRQ in RDMA ionic driver. Signed-off-by: Leon Romanovsky <leon@kernel.org> * ionic/ionic-shared: (1790 commits) net: ionic: Fetch qid allocation and SRQ capability from firmware Linux 7.3-rc4 ... 
Preparatory patch to support SRQ in RDMA ionic driver.

Signed-off-by: Leon Romanovsky <leon@kernel.org>

* ionic/ionic-shared: (1790 commits)
  net: ionic: Fetch qid allocation and SRQ capability from firmware
  Linux 7.3-rc4
  ...
RDMA/irdma: Free IRQ when CEQ vector mapping fails 2026-09-28T12:06:47+00:00 Serhat Kumral serhatkumral1@gmail.com 2026-09-18T11:04:28+00:00 952f18e04fa9790e347e09263c25318afe723d86 irdma_cfg_ceq_vector() registers an IRQ handler before requesting the CEQ-to-vector mapping through the virtual channel. If the mapping fails, the function returns without releasing the IRQ resources. The caller destroys the CEQ, leaving the IRQ handler registered and potentially referencing freed memory. Call irdma_destroy_irq() on mapping failure to release the IRQ resources and drain the associated tasklet before the caller destroys the CEQ. Use the same dev_id passed to request_irq(), accounting for CEQ0 sharing its interrupt vector with the AEQ. Fixes: b800e82feba7 ("RDMA/irdma: Add GEN3 support for AEQ and CEQ") Assisted-by: Claude:claude-opus-5 Signed-off-by: Serhat Kumral <serhatkumral1@gmail.com> Link: https://patch.msgid.link/20260918110428.36586-1-serhatkumral1@gmail.com Signed-off-by: Leon Romanovsky <leon@kernel.org> 
irdma_cfg_ceq_vector() registers an IRQ handler before requesting the
CEQ-to-vector mapping through the virtual channel. If the mapping
fails, the function returns without releasing the IRQ resources.
The caller destroys the CEQ, leaving the IRQ handler registered
and potentially referencing freed memory.

Call irdma_destroy_irq() on mapping failure to release
the IRQ resources and drain the associated tasklet
before the caller destroys the CEQ. Use the same dev_id
passed to request_irq(), accounting for CEQ0 sharing
its interrupt vector with the AEQ.

Fixes: b800e82feba7 ("RDMA/irdma: Add GEN3 support for AEQ and CEQ")
Assisted-by: Claude:claude-opus-5
Signed-off-by: Serhat Kumral <serhatkumral1@gmail.com>
Link: https://patch.msgid.link/20260918110428.36586-1-serhatkumral1@gmail.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: check vport device allocation 2026-09-23T07:37:07+00:00 Slavin Liu bolin.liu@seu.edu.cn 2026-09-11T06:09:09+00:00 2a770e6a64d66fc3a7317669ecca9773d8339ead An existing RDMA function does not guarantee allocation of the new vport device. Return -ENOMEM before initializing a NULL iwdev. Detected by static analysis and reviewed with AI-assisted source auditing. Fixes: 2ad49ae7330b ("RDMA/irdma: Introduce GEN3 vPort driver support") Assisted-by: LLM Signed-off-by: Slavin Liu <bolin.liu@seu.edu.cn> Link: https://patch.msgid.link/20260911060909.94219-1-bolin.liu@seu.edu.cn Signed-off-by: Leon Romanovsky <leon@kernel.org> 
An existing RDMA function does not guarantee allocation of the new
vport device. Return -ENOMEM before initializing a NULL iwdev.

Detected by static analysis and reviewed with AI-assisted source auditing.

Fixes: 2ad49ae7330b ("RDMA/irdma: Introduce GEN3 vPort driver support")
Assisted-by: LLM
Signed-off-by: Slavin Liu <bolin.liu@seu.edu.cn>
Link: https://patch.msgid.link/20260911060909.94219-1-bolin.liu@seu.edu.cn
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Use kvzalloc for lvl2 leafmem allocation 2026-09-22T14:16:16+00:00 Jeremy Bongio jbongio@google.com 2026-09-16T17:32:03+00:00 9c40f13a63a4b87177de3e8c318af25068fc6ab4 During VM live migration or when registering large memory regions (e.g., guest memory >= 256 GB backed by 4 KiB pages), get_lvl2_pble() calculates total leaves exceeding ~131,072. The memory required for lvl2->leafmem.va exceeds 4 MiB (sizeof(*leaf) * total > 5.24 MiB). Because kzalloc() requires physically contiguous pages, requests larger than MAX_PAGE_ORDER (4 MiB on x86_64) fail and trigger a warning: WARNING: CPU: 123 PID: 119906 at mm/page_alloc.c:5997 __alloc_frozen_pages_noprof+0x50c/0x1050 Call Trace: <TASK> alloc_frozen_pages_noprof+0xc0/0x240 ___kmalloc_large_node+0x3d/0xd0 __kmalloc_large_node_noprof+0x1b/0x90 __kmalloc_noprof+0x254/0x660 get_lvl1_lvl2_pble+0xbc/0x270 [irdma] irdma_get_pble+0x44/0xc0 [irdma] irdma_setup_pbles+0x46/0x1d0 [irdma] irdma_reg_user_mr_type_mem+0x41/0x230 [irdma] irdma_reg_user_mr+0x194/0x1c0 [irdma] ib_uverbs_reg_mr+0x1b0/0x2c0 [ib_uverbs] The leafmem array is purely host software tracking structures (struct irdma_pble_info) and does not require physically contiguous memory. Switch the allocation from kzalloc() to kvzalloc() and free it with kvfree() so that requests exceeding MAX_PAGE_ORDER fall back to virtually contiguous memory via vmalloc. Fixes: e8c4dbc2fcac ("RDMA/irdma: Add PBLE resource manager") Assisted-by: LLM Signed-off-by: Jeremy Bongio <jbongio@google.com> Link: https://patch.msgid.link/20260916173203.2555458-1-jbongio@google.com Tested-by: Jacob Moroni <jmoroni@google.com> Signed-off-by: Leon Romanovsky <leon@kernel.org> 
During VM live migration or when registering large memory regions
(e.g., guest memory >= 256 GB backed by 4 KiB pages), get_lvl2_pble()
calculates total leaves exceeding ~131,072. The memory required for
lvl2->leafmem.va exceeds 4 MiB (sizeof(*leaf) * total > 5.24 MiB).

Because kzalloc() requires physically contiguous pages, requests larger
than MAX_PAGE_ORDER (4 MiB on x86_64) fail and trigger a warning:

  WARNING: CPU: 123 PID: 119906 at mm/page_alloc.c:5997 __alloc_frozen_pages_noprof+0x50c/0x1050
  Call Trace:
   <TASK>
   alloc_frozen_pages_noprof+0xc0/0x240
   ___kmalloc_large_node+0x3d/0xd0
   __kmalloc_large_node_noprof+0x1b/0x90
   __kmalloc_noprof+0x254/0x660
   get_lvl1_lvl2_pble+0xbc/0x270 [irdma]
   irdma_get_pble+0x44/0xc0 [irdma]
   irdma_setup_pbles+0x46/0x1d0 [irdma]
   irdma_reg_user_mr_type_mem+0x41/0x230 [irdma]
   irdma_reg_user_mr+0x194/0x1c0 [irdma]
   ib_uverbs_reg_mr+0x1b0/0x2c0 [ib_uverbs]

The leafmem array is purely host software tracking structures (struct
irdma_pble_info) and does not require physically contiguous memory.

Switch the allocation from kzalloc() to kvzalloc() and free it with
kvfree() so that requests exceeding MAX_PAGE_ORDER fall back to
virtually contiguous memory via vmalloc.

Fixes: e8c4dbc2fcac ("RDMA/irdma: Add PBLE resource manager")
Assisted-by: LLM
Signed-off-by: Jeremy Bongio <jbongio@google.com>
Link: https://patch.msgid.link/20260916173203.2555458-1-jbongio@google.com
Tested-by: Jacob Moroni <jmoroni@google.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
RDMA/irdma: Fix erroneous -ENOMEM for large MR registration 2026-09-22T13:22:57+00:00 Jacob Moroni jmoroni@google.com 2026-09-15T17:03:17+00:00 5bb710151753fe53047b1d34b1b54d0f179cec68 Commit cc8997c94bf3 ("RDMA/irdma: Refactor PBLE functions") replaced the explicit level1_only boolean with a bit mask but did not properly convert the "if (level1_only)" check in irdma_get_pble() to be "lvl == PBLE_LEVEL_1" and instead left it as "if (lvl)". User MR registration passes "lvl = PBLE_LEVEL_1 | PBLE_LEVEL_2" so this check always evaluates to true and causes the loop to bail after grabbing a single SD even if the region actually requires more than one SD (as is the case for regions larger than 1 GiB with 4k pages). As a result, these registrations can fail with -ENOMEM even if there are sufficient PBLE resources. Fixes: cc8997c94bf3 ("RDMA/irdma: Refactor PBLE functions") Signed-off-by: Jacob Moroni <jmoroni@google.com> Link: https://patch.msgid.link/20260915170317.2901672-1-jmoroni@google.com Signed-off-by: Leon Romanovsky <leon@kernel.org> 
Commit cc8997c94bf3 ("RDMA/irdma: Refactor PBLE functions")
replaced the explicit level1_only boolean with a bit mask
but did not properly convert the "if (level1_only)" check
in irdma_get_pble() to be "lvl == PBLE_LEVEL_1" and instead
left it as "if (lvl)".

User MR registration passes "lvl = PBLE_LEVEL_1 | PBLE_LEVEL_2"
so this check always evaluates to true and causes the
loop to bail after grabbing a single SD even if the region
actually requires more than one SD (as is the case for
regions larger than 1 GiB with 4k pages). As a result, these
registrations can fail with -ENOMEM even if there are
sufficient PBLE resources.

Fixes: cc8997c94bf3 ("RDMA/irdma: Refactor PBLE functions")
Signed-off-by: Jacob Moroni <jmoroni@google.com>
Link: https://patch.msgid.link/20260915170317.2901672-1-jmoroni@google.com
Signed-off-by: Leon Romanovsky <leon@kernel.org>