From e11a07856ba06867a4c0c8aa91770e1b209dea98 Mon Sep 17 00:00:00 2001 From: Hisam Mehboob Date: Tue, 25 Aug 2026 03:28:47 +0500 Subject: vfio: selftests: Include for basename() on non-glibc builds sysfs.c calls basename() but does not include its declaration. Glibc exposes basename() through with _GNU_SOURCE, while musl declares it in . With musl, this results in an implicit-function-declaration error and subsequent type-mismatch errors at the callers. Include so the declaration is available on both glibc and musl. This is safe at both call sites: the arguments are local writable char[PATH_MAX] buffers filled by readlink_safe(). Link: https://lore.kernel.org/kvm/CALzav=fs4bqMM7BXNPXqhtbqGzA+LW5fgx2-1Q5=sTN5+krYhw@mail.gmail.com/ Fixes: a262fc49e0aa ("KVM: selftests: Build and link selftests/vfio/lib into KVM selftests") Reviewed-by: David Matlack Signed-off-by: Hisam Mehboob Link: https://lore.kernel.org/r/20260824222845.459181-3-hisamshar@gmail.com Signed-off-by: Alex Williamson --- tools/testing/selftests/vfio/lib/sysfs.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c index 98a46a2543cd..d88e348a1e0d 100644 --- a/tools/testing/selftests/vfio/lib/sysfs.c +++ b/tools/testing/selftests/vfio/lib/sysfs.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include -- cgit v1.2.3 From 573f39cdf714a129d2d8cee16d3e04a5bb5d5ce5 Mon Sep 17 00:00:00 2001 From: David Matlack Date: Thu, 3 Sep 2026 18:38:14 +0000 Subject: vfio: selftests: Add documentation Add documentation for VFIO selftests in driver-api/vfio-selftests.rst, include it in the driver-api index, and update MAINTAINERS. Since VFIO selftests was introduced it has expanded to include a helper library (libvfio), driver-level abstractions for triggering hardware DMA and interrupts, multiple IOMMU execution modes, dynamic IOVA allocation, and native integration into KVM selftests. Dedicated documentation should be helpful to guide kernel developers on writing tests, setting up devices, running tests, and understanding the helper library. Assisted-by: Gemini:3.1-Pro Signed-off-by: David Matlack Link: https://lore.kernel.org/r/20260903183814.2111748-1-dmatlack@google.com Signed-off-by: Alex Williamson --- Documentation/driver-api/index.rst | 1 + Documentation/driver-api/vfio-selftests.rst | 252 ++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 254 insertions(+) create mode 100644 Documentation/driver-api/vfio-selftests.rst diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 6601a258690f..53fb1786fab8 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -47,6 +47,7 @@ of interest to most developers working on device drivers. vfio-mediated-device vfio vfio-pci-device-specific-driver-acceptance + vfio-selftests Bus-level documentation ======================= diff --git a/Documentation/driver-api/vfio-selftests.rst b/Documentation/driver-api/vfio-selftests.rst new file mode 100644 index 000000000000..9d39a22519d6 --- /dev/null +++ b/Documentation/driver-api/vfio-selftests.rst @@ -0,0 +1,252 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +VFIO Selftests +============== + +VFIO selftests are built on top of the kernel's selftests framework and are +located in ``tools/testing/selftests/vfio``. + +VFIO selftests enable kernel developers to write and run tests that take the +form of userspace programs that interact with VFIO and IOMMUFD uAPIs. VFIO +selftests can be used to write functional tests for new features, regression +tests for bugs, and performance tests for optimizations. + +These tests are designed to interact with real PCI devices, i.e. they do not +rely on mocking out or faking any behavior in the kernel. This allows the tests +to exercise not only VFIO but also IOMMUFD, the IOMMU driver, interrupt +remapping, IRQ handling, etc. + +Running Tests +============= + +Running VFIO selftests requires a device bound to the ``vfio-pci`` driver. +There are scripts in ``tools/testing/selftests/vfio/scripts/`` to help with +setting one up. + +Picking a Device +---------------- + +Some VFIO selftests require a device with a supported driver implementation in +the selftests library (see `Driver Framework`_), while other tests (such as +basic PCI configuration space or reset tests) can use any arbitrary PCI device. + +This section will be expanded in the future to guide developers on how to find +a device with a supported driver. In the meantime, the selftests library +includes driver support for the following devices: + +- Intel IGB Gigabit Ethernet controllers +- Intel IOAT (I/O Acceleration Technology) DMA engines +- Intel DSA (Data Streaming Accelerator) +- NVIDIA Falcon engines (found in many NVIDIA GPUs, e.g., GTX 1080 or RTX 2080) + +Note that using Intel DSA devices requires setting the module parameter +``vfio_pci.disable_denylist=Y`` to allow ``vfio-pci`` to bind to DSA devices. + +Setting up Devices +------------------ + +.. code-block:: sh + + tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0 + +The script handles unbinding the device from its current driver and binding it +to ``vfio-pci``. Metadata about this device is stored in +``/tmp/vfio-selftests-devices`` so that the device can be cleaned up later +(bound back to its original driver). + +Execution +--------- + +VFIO selftests expect the PCI device BDF string as their final command-line +argument. e.g. + +.. code-block:: sh + + tools/testing/selftests/vfio/vfio_pci_device_test 0000:01:00.0 + +Alternatively, the PCI device BDF can be passed via the ``VFIO_SELFTESTS_BDF`` +environment variable, which can be useful when running all tests: + +.. code-block:: sh + + export VFIO_SELFTESTS_BDF=0000:01:00.0 + make -C tools/testing/selftests TARGETS=vfio install + tools/testing/selftests/kselftest_install/run_kselftest.sh -c vfio + +Cleanup +------- + +.. code-block:: sh + + tools/testing/selftests/vfio/scripts/cleanup.sh + +This script handles unbinding all previously setup devices from ``vfio-pci`` +and binding them back to their original driver. Specific devices can be cleaned +up by passing their BDF string to the cleanup script. + +Writing Tests +============= + +VFIO selftests leverage the ``kselftest_harness.h`` macros to structure setup +and teardown scenarios. A basic test involves: + +1. Declaring a test FIXTURE containing the device and IOMMU state. +2. Generating test variants that span the different IOMMU modes utilizing + ``FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES()``. +3. Initializing the device and binding the default driver ops. + +For a complete example of writing a VFIO selftest, refer to +``tools/testing/selftests/vfio/vfio_pci_device_test.c``. + +VFIO Selftests Library (libvfio) +================================ + +The selftests directory contains a helper library compiled from the ``lib/`` +subdirectory. It provides structures and helper functions to handle standard +operations like opening VFIO devices, creating VFIO containers, and interacting +with VFIO and IOMMUFD. + +All VFIO selftests have libvfio linked in by default. Tests can access the +library by including ````. + +Core Objects +------------ + +The VFIO selftests framework revolves around two distinct core objects: +``struct iommu`` and ``struct vfio_pci_device``. It is important to understand +the distinction and relationship between them: + +- ``struct iommu`` represents an IOMMU domain and the API used to program it + (e.g. legacy VFIO Type1 vs IOMMUFD). It is responsible for managing the I/O + address space, handling DMA mappings, and translating between host virtual + addresses (HVA) and IO virtual addresses (IOVA). + +- ``struct vfio_pci_device`` represents a single PCI device. It encapsulates + the VFIO device file descriptor, handles access to the PCI configuration + space, manages memory-mapped BARs, and drives device interrupts. + +A typical test initializes an IOMMU instance first, then passes it when +initializing one or more PCI devices, linking the devices to the shared IOMMU +domain. + +``struct iommu`` +^^^^^^^^^^^^^^^^ + +The ``struct iommu`` abstracts away the complexity of managing VFIO containers, +IOMMU groups, and IOMMUFD contexts. A test allocates its IOMMU handle using +``iommu_init()`` passing one of the supported IOMMU modes: + +.. code-block:: c + + struct iommu *iommu = iommu_init(MODE_IOMMUFD); + ... + iommu_cleanup(iommu); + +The selftest framework is designed to work uniformly across all standard kernel +IOMMU APIs. The defined modes (in ``iommu.h``) include: + +- ``MODE_VFIO_TYPE1_IOMMU`` +- ``MODE_VFIO_TYPE1V2_IOMMU`` +- ``MODE_IOMMUFD_COMPAT_TYPE1`` +- ``MODE_IOMMUFD_COMPAT_TYPE1V2`` +- ``MODE_IOMMUFD`` + +VFIO selftests can replicate their test cases across all possible IOMMU modes +using FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES() to ensure kernel IOMMU APIs remain +compatible. + +``struct vfio_pci_device`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A PCI device is represented as a ``struct vfio_pci_device``. The device +representation is initialized using ``vfio_pci_device_init()``, which takes a +target BDF string, opens the device in VFIO, and binds it to an existing +``struct iommu`` instance: + +.. code-block:: c + + struct vfio_pci_device *device = vfio_pci_device_init(device_bdf, iommu); + ... + vfio_pci_device_cleanup(device); + +``struct iova_allocator`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To facilitate DMA mappings without hardcoding IOVAs (IO virtual addresses) that +might conflict with reserved platform address ranges, the library provides a +``struct iova_allocator``. + +The IOVA allocator queries the underlying IOMMU (via ``struct iommu``) for its +valid target IOVA ranges. Tests can safely carve out unique contiguous chunks +of IOVA space for device DMA by repeatedly calling: + +.. code-block:: c + + struct iova_allocator *allocator = iova_allocator_init(iommu); + iova_t addr = iova_allocator_alloc(allocator, size); + +Driver Framework +---------------- + +The primary goal of VFIO selftests is to verify the correctness of the kernel +code handling VFIO devices, including VFIO core, IOMMUFD, IOMMU drivers, page +pinning, DMA mapping/unmapping, interrupt remapping, and IRQ delivery. + +Because different physical devices have vastly different programming interfaces +for triggering work, the selftests library implements a standard driver +framework. This framework abstracts diverse hardware endpoints (such as Intel +IGB, IOAT, or DSA) behind a uniform interface, allowing tests to provoke a +broad range of devices to generate DMA operations and interrupts in a +standardized manner. + +By exercising DMA and interrupts with real endpoints, tests can thoroughly +stress and validate the underlying kernel paths, such as IOMMU page table +management, dirty page tracking, Device TLB and IOTLB invalidations, and +interrupt delivery under authentic hardware conditions. + +All endpoint driver implementations populate a common set of operations defined +in ``struct vfio_pci_driver_ops``: + +- ``probe()``: Validate whether the driver supports the current PCI device + (e.g. matching vendor/device IDs). +- ``init()``: Set up specific control registers or ring-buffer data structures. +- ``remove()``: Clean up structures before destruction. +- ``memcpy_start()``: Initiate DMA copies via the device. +- ``memcpy_wait()``: Poll and wait until previously initiated copies complete + safely. +- ``send_msi()``: Provoke the device to signal an MSI interrupt. + +Tests are required to set up a dedicated DMA region for the driver to use +within ``struct vfio_pci_driver`` to serve as the driver's main memory +allocation region (e.g. used for in-memory descriptor rings and hardware +buffers). Tests must provision and map this DMA region before completing driver +initialization. + +For a complete example of interacting with the driver framework from a test +perspective, refer to ``tools/testing/selftests/vfio/vfio_pci_driver_test.c``. + +When contributing a new driver implementation to the selftests library, the +primary acceptance criteria is that ``vfio_pci_driver_test`` passes using the +new driver and target device. + +Integration with KVM Selftests +------------------------------ + +libvfio is not strictly limited to VFIO selftests, it is also leveraged within +KVM selftests (``tools/testing/selftests/kvm/``). KVM selftests build their own +copy of libvfio and link against it, enabling libvfio to be used in KVM +selftests. + +This integration enables testing the interaction between KVM and VFIO and +IOMMUFD. For example, testing the end-to-end delivery of device interrupts into +running vCPUs. + +Contribution Guidelines +======================= + +- Prefer the shortlog prefix ``vfio: selftests: ...`` for commits that modify + files in ``tools/testing/selftests/vfio/``. +- Follow the Coding Style, Comments, Changelogs, and Function References + guidelines from the + :doc:`KVM x86 maintainers handbook `. diff --git a/MAINTAINERS b/MAINTAINERS index 3a19da74d00c..a19f1de16f56 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -28727,6 +28727,7 @@ VFIO SELFTESTS M: David Matlack L: kvm@vger.kernel.org S: Maintained +F: Documentation/driver-api/vfio-selftests.rst F: tools/testing/selftests/vfio/ VFIO VIRTIO PCI DRIVER -- cgit v1.2.3 From b1d0f0d97a7d94e5b94dc4d28251002fb17dfbaa Mon Sep 17 00:00:00 2001 From: Hemanth Selam Date: Fri, 4 Sep 2026 17:21:08 +0530 Subject: vfio: fix typo "stucture" in comment Correct "stucture" to "structure", reported by scripts/checkpatch.pl using the misspelling list in scripts/spelling.txt. Only touches comments, no code changes. Assisted-by: Cursor:claude-opus-5 Signed-off-by: Hemanth Selam Link: https://lore.kernel.org/r/20260904115109.6170-1-hemanth.selam@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/mdev/mdev_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c index 3b4aa0632fcc..34337292a9b1 100644 --- a/drivers/vfio/mdev/mdev_core.c +++ b/drivers/vfio/mdev/mdev_core.c @@ -50,7 +50,7 @@ static int mdev_device_remove_cb(struct device *dev, void *data) * @types: Array of supported mdev types * @nr_types: Number of entries in @types * - * Registers the @parent stucture as a parent for mdev types and thus mdev + * Registers the @parent structure as a parent for mdev types and thus mdev * devices. The caller needs to hold a reference on @dev that must not be * released until after the call to mdev_unregister_parent(). * -- cgit v1.2.3 From 258ba46543abdd36844f04bba8b4022a5731d873 Mon Sep 17 00:00:00 2001 From: Alex Williamson Date: Fri, 11 Sep 2026 11:04:24 -0600 Subject: vfio: Reject a second cdev open before mutating shared device state The cdev single-open check lives in vfio_df_open(), which runs at the end of the bind ioctl, after vfio_df_ioctl_bind_iommufd() has already updated state shared across all opens: vfio_df_check_token() can set the PF vf_token and vfio_df_get_kvm_safe() records the caller's KVM pointer in device->kvm and takes a reference. A second cdev bind of an already-open device runs both, only to be rejected in vfio_df_open(). The error path clears device->kvm and drops the reference, tearing down the current opener's KVM association and potentially resulting in an unbalanced reference on close or premature release, while the vf_token remains clobbered. Move the single-open check into vfio_df_ioctl_bind_iommufd() ahead of both mutations, so a bind that cannot complete leaves the current opener's state untouched. df->group is NULL on this path, so a non-zero open_count is exactly what vfio_df_open() rejected. The test in vfio_df_open() becomes redundant and is removed. Return -EBUSY rather than -EINVAL here. The arguments are not invalid, the device is in use, which could be a transient condition due to a delayed fput if the prior user is terminated. This provides compatibility with the group path, where a group open returns -EBUSY, and users may choose bounded polling to detect such a transient condition. Fixes: 839e692fa4eb ("vfio: Make vfio_df_open() single open for device cdev path") Fixes: 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD") Fixes: 86624ba3b522 ("vfio/pci: Do vf_token checks for VFIO_DEVICE_BIND_IOMMUFD") Assisted-by: LLM Reviewed-by: Jason Gunthorpe Signed-off-by: Alex Williamson Reviewed-by: Kevin Tian Link: https://lore.kernel.org/r/20260911170429.1642480-2-alex.williamson@nvidia.com Signed-off-by: Alex Williamson --- drivers/vfio/device_cdev.c | 12 ++++++++++++ drivers/vfio/vfio_main.c | 7 ------- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c index 1d9515c967b0..30362936c7b5 100644 --- a/drivers/vfio/device_cdev.c +++ b/drivers/vfio/device_cdev.c @@ -130,6 +130,18 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, goto out_unlock; } + /* + * The cdev path allows only a single open. Reject a second open here, + * before the VF token and device->kvm updates below would clobber the + * current opener's state on a bind that cannot complete. Return -EBUSY + * rather than -EINVAL since a delayed release of the prior opener can + * make this transient. + */ + if (device->open_count) { + ret = -EBUSY; + goto out_unlock; + } + ret = vfio_df_check_token(device, &bind); if (ret) goto out_unlock; diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 423ead48aafe..cb3deb5a4857 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -603,13 +603,6 @@ int vfio_df_open(struct vfio_device_file *df) lockdep_assert_held(&device->dev_set->lock); - /* - * Only the group path allows the device to be opened multiple - * times. The device cdev path doesn't have a secure way for it. - */ - if (device->open_count != 0 && !df->group) - return -EINVAL; - device->open_count++; if (device->open_count == 1) { ret = vfio_df_device_first_open(df); -- cgit v1.2.3 From 24acae9894b96624a69f07b708d58e591d11df98 Mon Sep 17 00:00:00 2001 From: Alex Williamson Date: Fri, 11 Sep 2026 11:04:26 -0600 Subject: vfio: selftests: Extend mix_and_match timeout to 90s Systems with firmware first error handling can see heavy SMM load as a result of the mix_and_match induced IOMMU errors. For devices like igb that get wedged on these errors, the resulting FLR is delayed waiting for the transaction pending register to clear, which never occurs. However, due to the background SMM, the backoff delays in PCI code wildly exceed their timeouts. As a result, the whole test can report failure. Extend the mix_and_match timeout to a value shown to complete reliably on such systems, with ~2x margin. Assisted-by: LLM Reviewed-by: David Matlack Signed-off-by: Alex Williamson Link: https://lore.kernel.org/r/20260911170429.1642480-4-alex.williamson@nvidia.com Signed-off-by: Alex Williamson --- tools/testing/selftests/vfio/vfio_pci_driver_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/vfio/vfio_pci_driver_test.c b/tools/testing/selftests/vfio/vfio_pci_driver_test.c index 761bf117d624..1d532f944cee 100644 --- a/tools/testing/selftests/vfio/vfio_pci_driver_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_driver_test.c @@ -190,7 +190,7 @@ TEST_F(vfio_pci_driver_test, send_msi) ASSERT_EQ(1, value); } -TEST_F(vfio_pci_driver_test, mix_and_match) +TEST_F_TIMEOUT(vfio_pci_driver_test, mix_and_match, 90) { u64 value; int i; -- cgit v1.2.3 From e281aa76bc22865052dc7b7250403ff1258856a0 Mon Sep 17 00:00:00 2001 From: Alex Williamson Date: Fri, 11 Sep 2026 11:04:27 -0600 Subject: vfio: selftests: Extend timeout for runner executions When executed via the runner, ie. run_kselftest.sh, tests are subject to a default 45s timeout. None of the igb, ioatdma, or nv_falcon drivers can run the vfio_pci_driver_test within this window. The igb driver is the worst due to the device getting wedged from bad DMAs in mix_and_match and requiring the full transaction pending delay prior to FLR. This is even further exacerbated when the system implements firmware-first error handling and the transaction pending scheduled delays are preempted beyond expectations by SMIs. Worst case observed runtime for the whole vfio_pci_driver_test on such a setup is 450s. Round up to 600s. Assisted-by: LLM Reviewed-by: David Matlack Signed-off-by: Alex Williamson Link: https://lore.kernel.org/r/20260911170429.1642480-5-alex.williamson@nvidia.com Signed-off-by: Alex Williamson --- tools/testing/selftests/vfio/.gitignore | 1 + tools/testing/selftests/vfio/settings | 5 +++++ 2 files changed, 6 insertions(+) create mode 100644 tools/testing/selftests/vfio/settings diff --git a/tools/testing/selftests/vfio/.gitignore b/tools/testing/selftests/vfio/.gitignore index 7fadc19d3bca..fc14bc18bcd8 100644 --- a/tools/testing/selftests/vfio/.gitignore +++ b/tools/testing/selftests/vfio/.gitignore @@ -8,3 +8,4 @@ !*.mk !.gitignore !Makefile +!settings diff --git a/tools/testing/selftests/vfio/settings b/tools/testing/selftests/vfio/settings new file mode 100644 index 000000000000..c9abf1c19f19 --- /dev/null +++ b/tools/testing/selftests/vfio/settings @@ -0,0 +1,5 @@ +# Extend kselftest default 45s timeout based on worst case test runtime, +# currently vfio:vfio_pci_driver_test running on physical NIC, battling +# SMIs due to invoked firmware-first error handling from bad DMAs and +# device wedging. Observed worst case ~450s, extend to 600s. +timeout=600 -- cgit v1.2.3 From bc78c90728cd74d1c7335fef786fa51968bdebad Mon Sep 17 00:00:00 2001 From: Samuel Crossley Date: Sat, 19 Sep 2026 15:13:03 -0700 Subject: vfio/type1: avoid walking reserved-only mappings while unpinning Tearing down a large device-passthrough DMA mapping can process tens of millions of reserved pfns in a single VFIO_IOMMU_UNMAP_DMA. A 128 GiB reserved mapping holds about 33.6 million 4KiB pfns, and vfio_unpin_pages_remote() walks every one of them whenever dma->has_rsvd is set, because reserved and non-reserved pfns might be mixed. For a mapping that contains only reserved pfns every put_pfn() is a no-op, but each one still pays for the pfn_valid()/PageReserved() classification. Seen on GPU-passthrough hosts running 6.13 with the v6.18 type1 series backported, where a single VFIO_IOMMU_UNMAP_DMA held a CPU past the soft-lockup watchdog (host identification, register dump and module list trimmed): watchdog: BUG: soft lockup - CPU#261 stuck for 22s! [qemu-system-x86:134503] RIP: 0010:vfio_unpin_pages_remote+0x139/0x2a0 [vfio_iommu_type1] Code: 90 4c 89 e1 48 c1 e9 34 75 74 4c 89 e1 48 c1 e9 22 75 6b 48 8b 0d f7 6f c2 e3 48 85 c9 74 5f 4c 89 e2 48 c1 ea 16 48 8b 0c d1 <48> 85 c9 74 4f 41 8b 55 30 4c 89 e6 48 c1 ee 0f 83 e6 7f c1 e6 05 Call Trace: ? watchdog_timer_fn+0x3d6/0x440 ? running_clock+0x10/0x10 ? hrtimer_interrupt+0x185/0x580 ? __sysvec_apic_timer_interrupt+0x44/0xe0 ? sysvec_apic_timer_interrupt+0x6b/0x80 ? asm_sysvec_apic_timer_interrupt+0x16/0x20 ? vfio_unpin_pages_remote+0x139/0x2a0 [vfio_iommu_type1] vfio_sync_unpin+0x96/0xf0 [vfio_iommu_type1] vfio_unmap_unpin+0x324/0x3d0 [vfio_iommu_type1] vfio_remove_dma+0x25/0xa0 [vfio_iommu_type1] vfio_iommu_type1_ioctl+0xcd5/0x1760 [vfio_iommu_type1] ? amd_pmu_v2_enable_all+0xa/0x30 ? perf_pmu_sched_task+0xbf/0xf0 x64_sys_call+0x282/0x1ac0 ? syscall_trace_enter+0x1e5/0x1f0 do_syscall_64+0x68/0x130 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP is the mem_section root test in pfn_valid(), inlined into put_pfn() through is_invalid_reserved_pfn(), so the stall is in the dma->has_rsvd per-page loop. mem_section[] has no root for that pfn, so pfn_valid() returns false and put_pfn() does nothing - for every pfn in the mapping. Track whether the mapping also contains non-reserved pfns and skip the walk entirely for reserved-only mappings. Mixed mappings keep the existing per-pfn walk, and ordinary mappings keep the existing batched unpin path. Tested on an affected host: reserved-only mappings up to 128 GiB take the new path and complete in microseconds. The ordinary-page path was exercised unchanged. Suggested-by: Alex Williamson Link: https://lore.kernel.org/all/20260803154707.71cc0d6b@shazbot.org/ Signed-off-by: Samuel Crossley Link: https://lore.kernel.org/r/20260919-vfio-v2-1-68e7ed355972@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index c8151ba54de3..f7addfbe1aab 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -94,6 +94,7 @@ struct vfio_dma { bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; bool has_rsvd; /* has 1 or more rsvd pfns */ + bool has_non_rsvd; /* has 1 or more !rsvd pfns */ struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; @@ -791,6 +792,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, out: dma->has_rsvd |= rsvd; + dma->has_non_rsvd |= !rsvd; ret = vfio_lock_acct(dma, lock_acct, false); unpin_out: @@ -821,11 +823,13 @@ static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, long unlocked = 0, locked = vpfn_pages(dma, iova, npage); if (dma->has_rsvd) { - unsigned long i; + if (dma->has_non_rsvd) { + unsigned long i; - for (i = 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + for (i = 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } } else { put_valid_unreserved_pfns(pfn, npage, dma->prot); unlocked = npage; -- cgit v1.2.3 From ddbf67d2480293aed66450c7367a596c4856c7a2 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:41 -0300 Subject: net/mlx5: Add IFC structures for CQE and WQE Building the WQE and CQE using the IFC system is easier than with a C layout struct. Add definitions for their layout. Structs for: - wqe_ctrl_seg_bits (16B): WQE control segment - wqe_raddr_seg_bits (16B): RDMA remote address segment - wqe_data_seg_bits (16B): data segment - cqe64_bits (64B): 64-byte CQE with error syndrome union The VFIO mlx5 selftest will use these. Assisted-by: Claude:claude-opus-4.6 Signed-off-by: Jason Gunthorpe Acked-by: Leon Romanovsky Link: https://lore.kernel.org/r/1-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- include/linux/mlx5/mlx5_ifc.h | 55 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 12b9a81e5621..a3edc5ba1cd4 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -5979,6 +5979,61 @@ struct mlx5_ifc_cqe_error_syndrome_bits { u8 syndrome[0x8]; }; +struct mlx5_ifc_wqe_ctrl_seg_bits { + u8 opmod[0x8]; + u8 wqe_index[0x10]; + u8 opcode[0x8]; + + u8 qp_or_sq[0x18]; + u8 reserved_at_38[0x2]; + u8 ds[0x6]; + + u8 signature[0x8]; + u8 reserved_at_48[0x10]; + u8 fm[0x3]; + u8 reserved_at_5b[0x1]; + u8 ce[0x2]; + u8 se[0x1]; + u8 reserved_at_5f[0x1]; + + u8 imm[0x20]; +}; + +struct mlx5_ifc_wqe_raddr_seg_bits { + u8 raddr[0x40]; + + u8 rkey[0x20]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_wqe_data_seg_bits { + u8 reserved_at_0[0x1]; + u8 byte_count[0x1f]; + + u8 lkey[0x20]; + + u8 addr[0x40]; +}; + +struct mlx5_ifc_cqe64_bits { + u8 reserved_at_0[0x1a0]; + + union { + u8 reserved_at_1a0[0x20]; + struct mlx5_ifc_cqe_error_syndrome_bits error_syndrome; + }; + + u8 send_wqe_opcode[0x8]; + u8 qpn_or_dctn_or_flow_tag[0x18]; + + u8 wqe_counter[0x10]; + u8 signature[0x8]; + u8 opcode[0x4]; + u8 cqe_format[0x2]; + u8 se[0x1]; + u8 owner[0x1]; +}; + struct mlx5_ifc_qp_context_extension_bits { u8 reserved_at_0[0x60]; -- cgit v1.2.3 From a2327ba82f93240e1501e52e2dfbd8d2c1c1a5cf Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:42 -0300 Subject: net/mlx5: Move HW constant groups from device.h/cq.h to mlx5_ifc.h Generally the IFC file should contain enums for the fields it defines. Move the ones in device.h that are relevant to the VFIO selftests over to the IFC file: wqe_ctrl_seg_bits.opcode: all MLX5_OPCODE_* WQE opcodes wqe_ctrl_seg_bits.ce: new MLX5_WQE_CE_* completion/event modes cqe64_bits.opcode: all MLX5_CQE_* CQE opcode values eqe_bits.event_type: enum mlx5_event (all event types) cmd_out_bits.status: all MLX5_CMD_STAT_* status codes query_hca_cap_in_bits.op_mod: enum mlx5_cap_mode Tidy MLX5_PCI_CMD_XPORT which is an alias of MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT. No functional change. All existing users of device.h and cq.h continue to see these constants through the include chain. Assisted-by: Claude:claude-opus-4.6 Signed-off-by: Jason Gunthorpe Acked-by: Leon Romanovsky Link: https://lore.kernel.org/r/2-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- include/linux/mlx5/cq.h | 10 ---- include/linux/mlx5/device.h | 114 +-------------------------------------- include/linux/mlx5/mlx5_ifc.h | 123 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 123 deletions(-) diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h index 9d47cdc727ad..a1c14479e462 100644 --- a/include/linux/mlx5/cq.h +++ b/include/linux/mlx5/cq.h @@ -81,16 +81,6 @@ enum { enum { MLX5_CQE_OWNER_MASK = 1, - MLX5_CQE_REQ = 0, - MLX5_CQE_RESP_WR_IMM = 1, - MLX5_CQE_RESP_SEND = 2, - MLX5_CQE_RESP_SEND_IMM = 3, - MLX5_CQE_RESP_SEND_INV = 4, - MLX5_CQE_RESIZE_CQ = 5, - MLX5_CQE_SIG_ERR = 12, - MLX5_CQE_REQ_ERR = 13, - MLX5_CQE_RESP_ERR = 14, - MLX5_CQE_INVALID = 15, }; enum { diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 8cb321a9fb3d..8646303b029d 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -172,7 +172,7 @@ enum mlx5_inline_modes { enum { MLX5_MAX_COMMANDS = 32, MLX5_CMD_DATA_BLOCK_SIZE = 512, - MLX5_PCI_CMD_XPORT = 7, + MLX5_PCI_CMD_XPORT = MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT, MLX5_MKEY_BSF_OCTO_SIZE = 4, MLX5_MAX_PSVS = 4, }; @@ -308,63 +308,6 @@ enum { MLX5_EVENT_QUEUE_TYPE_DCT = 6, }; -/* mlx5 components can subscribe to any one of these events via - * mlx5_eq_notifier_register API. - */ -enum mlx5_event { - /* Special value to subscribe to any event */ - MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0, - /* HW events enum start: comp events are not subscribable */ - MLX5_EVENT_TYPE_COMP = 0x0, - /* HW Async events enum start: subscribable events */ - MLX5_EVENT_TYPE_PATH_MIG = 0x01, - MLX5_EVENT_TYPE_COMM_EST = 0x02, - MLX5_EVENT_TYPE_SQ_DRAINED = 0x03, - MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13, - MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14, - - MLX5_EVENT_TYPE_CQ_ERROR = 0x04, - MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05, - MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07, - MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10, - MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11, - MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12, - MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27, - - MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08, - MLX5_EVENT_TYPE_PORT_CHANGE = 0x09, - MLX5_EVENT_TYPE_GPIO_EVENT = 0x15, - MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16, - MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17, - MLX5_EVENT_TYPE_XRQ_ERROR = 0x18, - MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19, - MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22, - MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24, - MLX5_EVENT_TYPE_PPS_EVENT = 0x25, - - MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a, - MLX5_EVENT_TYPE_STALL_EVENT = 0x1b, - - MLX5_EVENT_TYPE_CMD = 0x0a, - MLX5_EVENT_TYPE_PAGE_REQUEST = 0xb, - - MLX5_EVENT_TYPE_PAGE_FAULT = 0xc, - MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0xd, - - MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0xe, - MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0xf, - - MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c, - MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d, - - MLX5_EVENT_TYPE_FPGA_ERROR = 0x20, - MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21, - - MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26, - - MLX5_EVENT_TYPE_MAX = 0x100, -}; - enum mlx5_driver_event { MLX5_DRIVER_EVENT_TYPE_TRAP = 0, MLX5_DRIVER_EVENT_UPLINK_NETDEV, @@ -420,22 +363,6 @@ enum { }; enum { - MLX5_OPCODE_NOP = 0x00, - MLX5_OPCODE_SEND_INVAL = 0x01, - MLX5_OPCODE_RDMA_WRITE = 0x08, - MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, - MLX5_OPCODE_SEND = 0x0a, - MLX5_OPCODE_SEND_IMM = 0x0b, - MLX5_OPCODE_LSO = 0x0e, - MLX5_OPCODE_RDMA_READ = 0x10, - MLX5_OPCODE_ATOMIC_CS = 0x11, - MLX5_OPCODE_ATOMIC_FA = 0x12, - MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, - MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, - MLX5_OPCODE_BIND_MW = 0x18, - MLX5_OPCODE_CONFIG_CMD = 0x1f, - MLX5_OPCODE_ENHANCED_MPSW = 0x29, - MLX5_RECV_OPCODE_RDMA_WRITE_IMM = 0x00, MLX5_RECV_OPCODE_SEND = 0x01, MLX5_RECV_OPCODE_SEND_IMM = 0x02, @@ -443,19 +370,6 @@ enum { MLX5_CQE_OPCODE_ERROR = 0x1e, MLX5_CQE_OPCODE_RESIZE = 0x16, - - MLX5_OPCODE_SET_PSV = 0x20, - MLX5_OPCODE_GET_PSV = 0x21, - MLX5_OPCODE_CHECK_PSV = 0x22, - MLX5_OPCODE_DUMP = 0x23, - MLX5_OPCODE_RGET_PSV = 0x26, - MLX5_OPCODE_RCHECK_PSV = 0x27, - - MLX5_OPCODE_UMR = 0x25, - - MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c, - - MLX5_OPCODE_ACCESS_ASO = 0x2d, }; enum { @@ -1224,12 +1138,6 @@ enum mlx5_flex_parser_protos { /* MLX5 DEV CAPs */ -/* TODO: EAT.ME */ -enum mlx5_cap_mode { - HCA_CAP_OPMOD_GET_MAX = 0, - HCA_CAP_OPMOD_GET_CUR = 1, -}; - /* Any new cap addition must update mlx5_hca_caps_alloc() to allocate * capability memory. */ @@ -1507,26 +1415,6 @@ enum mlx5_qcam_feature_groups { #define MLX5_CAP_PSP(mdev, cap)\ MLX5_GET(psp_cap, (mdev)->caps.hca[MLX5_CAP_PSP]->cur, cap) -enum { - MLX5_CMD_STAT_OK = 0x0, - MLX5_CMD_STAT_INT_ERR = 0x1, - MLX5_CMD_STAT_BAD_OP_ERR = 0x2, - MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3, - MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4, - MLX5_CMD_STAT_BAD_RES_ERR = 0x5, - MLX5_CMD_STAT_RES_BUSY = 0x6, - MLX5_CMD_STAT_NOT_READY = 0x7, - MLX5_CMD_STAT_LIM_ERR = 0x8, - MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9, - MLX5_CMD_STAT_IX_ERR = 0xa, - MLX5_CMD_STAT_NO_RES_ERR = 0xf, - MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50, - MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51, - MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10, - MLX5_CMD_STAT_BAD_PKT_ERR = 0x30, - MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40, -}; - enum { MLX5_IEEE_802_3_COUNTERS_GROUP = 0x0, MLX5_RFC_2863_COUNTERS_GROUP = 0x1, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index a3edc5ba1cd4..8247cb0f8dfd 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -5999,6 +5999,39 @@ struct mlx5_ifc_wqe_ctrl_seg_bits { u8 imm[0x20]; }; +/* Values for wqe_ctrl_seg_bits.opcode */ +enum { + MLX5_OPCODE_NOP = 0x00, + MLX5_OPCODE_SEND_INVAL = 0x01, + MLX5_OPCODE_RDMA_WRITE = 0x08, + MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, + MLX5_OPCODE_SEND = 0x0a, + MLX5_OPCODE_SEND_IMM = 0x0b, + MLX5_OPCODE_LSO = 0x0e, + MLX5_OPCODE_RDMA_READ = 0x10, + MLX5_OPCODE_ATOMIC_CS = 0x11, + MLX5_OPCODE_ATOMIC_FA = 0x12, + MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, + MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, + MLX5_OPCODE_BIND_MW = 0x18, + MLX5_OPCODE_CONFIG_CMD = 0x1f, + MLX5_OPCODE_SET_PSV = 0x20, + MLX5_OPCODE_GET_PSV = 0x21, + MLX5_OPCODE_CHECK_PSV = 0x22, + MLX5_OPCODE_DUMP = 0x23, + MLX5_OPCODE_UMR = 0x25, + MLX5_OPCODE_RGET_PSV = 0x26, + MLX5_OPCODE_RCHECK_PSV = 0x27, + MLX5_OPCODE_ENHANCED_MPSW = 0x29, + MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c, + MLX5_OPCODE_ACCESS_ASO = 0x2d, +}; + +/* Values for wqe_ctrl_seg_bits.ce */ +enum { + MLX5_WQE_CE_CQE_ALWAYS = 2, +}; + struct mlx5_ifc_wqe_raddr_seg_bits { u8 raddr[0x40]; @@ -6034,6 +6067,20 @@ struct mlx5_ifc_cqe64_bits { u8 owner[0x1]; }; +/* Values for cqe64_bits.opcode */ +enum { + MLX5_CQE_REQ = 0, + MLX5_CQE_RESP_WR_IMM = 1, + MLX5_CQE_RESP_SEND = 2, + MLX5_CQE_RESP_SEND_IMM = 3, + MLX5_CQE_RESP_SEND_INV = 4, + MLX5_CQE_RESIZE_CQ = 5, + MLX5_CQE_SIG_ERR = 12, + MLX5_CQE_REQ_ERR = 13, + MLX5_CQE_RESP_ERR = 14, + MLX5_CQE_INVALID = 15, +}; + struct mlx5_ifc_qp_context_extension_bits { u8 reserved_at_0[0x60]; @@ -6530,6 +6577,12 @@ struct mlx5_ifc_query_hca_cap_in_bits { u8 reserved_at_60[0x20]; }; +/* Values for query_hca_cap_in_bits.op_mod */ +enum mlx5_cap_mode { + HCA_CAP_OPMOD_GET_MAX = 0, + HCA_CAP_OPMOD_GET_CUR = 1, +}; + struct mlx5_ifc_other_hca_cap_bits { u8 roce[0x1]; u8 reserved_at_1[0x27f]; @@ -11325,6 +11378,55 @@ struct mlx5_ifc_eqe_bits { u8 owner[0x1]; }; +/* Values for eqe_bits.event_type */ +enum mlx5_event { + /* Special value to subscribe to any event */ + MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0, + /* HW events enum start: comp events are not subscribable */ + MLX5_EVENT_TYPE_COMP = 0x0, + /* HW Async events enum start: subscribable events */ + MLX5_EVENT_TYPE_PATH_MIG = 0x01, + MLX5_EVENT_TYPE_COMM_EST = 0x02, + MLX5_EVENT_TYPE_SQ_DRAINED = 0x03, + MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13, + MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14, + + MLX5_EVENT_TYPE_CQ_ERROR = 0x04, + MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05, + MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07, + MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10, + MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11, + MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12, + MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27, + + MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08, + MLX5_EVENT_TYPE_PORT_CHANGE = 0x09, + MLX5_EVENT_TYPE_CMD = 0x0a, + MLX5_EVENT_TYPE_PAGE_REQUEST = 0x0b, + MLX5_EVENT_TYPE_PAGE_FAULT = 0x0c, + MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0x0d, + MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0x0e, + MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0x0f, + MLX5_EVENT_TYPE_GPIO_EVENT = 0x15, + MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16, + MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17, + MLX5_EVENT_TYPE_XRQ_ERROR = 0x18, + MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19, + MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a, + MLX5_EVENT_TYPE_STALL_EVENT = 0x1b, + MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c, + MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d, + MLX5_EVENT_TYPE_FPGA_ERROR = 0x20, + MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21, + MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22, + MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24, + MLX5_EVENT_TYPE_PPS_EVENT = 0x25, + MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26, + + MLX5_EVENT_TYPE_MAX = 0x100, +}; + +/* Values for cmd_queue_entry_bits.type */ enum { MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT = 0x7, }; @@ -11367,6 +11469,27 @@ struct mlx5_ifc_cmd_out_bits { u8 command_output[0x20]; }; +/* Values for cmd_out_bits.status */ +enum { + MLX5_CMD_STAT_OK = 0x0, + MLX5_CMD_STAT_INT_ERR = 0x1, + MLX5_CMD_STAT_BAD_OP_ERR = 0x2, + MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3, + MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4, + MLX5_CMD_STAT_BAD_RES_ERR = 0x5, + MLX5_CMD_STAT_RES_BUSY = 0x6, + MLX5_CMD_STAT_NOT_READY = 0x7, + MLX5_CMD_STAT_LIM_ERR = 0x8, + MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9, + MLX5_CMD_STAT_IX_ERR = 0xa, + MLX5_CMD_STAT_NO_RES_ERR = 0xf, + MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10, + MLX5_CMD_STAT_BAD_PKT_ERR = 0x30, + MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40, + MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50, + MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51, +}; + struct mlx5_ifc_cmd_in_bits { u8 opcode[0x10]; u8 reserved_at_10[0x10]; -- cgit v1.2.3 From e601522f287be247771cdfa7381c96ed1aa3b533 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:43 -0300 Subject: net/mlx5: Extract MLX5_SET/GET macros into mlx5_ifc_macros.h Extract the entire MLX5_SET/GET macro family and their internal helpers from device.h into a new lightweight header (include/linux/mlx5/mlx5_ifc_macros.h). device.h cannot be included by the VFIO selftest because it pulls in rdma/ib_verbs.h; the macros themselves depend only on endian helpers, BUILD_BUG_ON, and basic C types. The moved macros include the internal helpers (__mlx5_nullp through __mlx5_st_sz_bits), all size/address macros (MLX5_ST_SZ_BYTES, MLX5_BYTE_OFF, MLX5_ADDR_OF, etc.), the 32-bit accessors (MLX5_SET, MLX5_GET, MLX5_SET_TO_ONES, MLX5_ARRAY_SET, MLX5_GET_PR), the 64-bit accessors (MLX5_SET64, MLX5_GET64, MLX5_ARRAY_SET64, MLX5_GET64_PR), the 16-bit accessors (MLX5_GET16, MLX5_SET16), and the big-endian getters (MLX5_GET64_BE, MLX5_GET_BE). device.h includes the new header so existing kernel code is unchanged. Assisted-by: Claude:claude-opus-4.6 Signed-off-by: Jason Gunthorpe Acked-by: Leon Romanovsky Link: https://lore.kernel.org/r/3-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- include/linux/mlx5/device.h | 117 +----------------------------- include/linux/mlx5/mlx5_ifc_macros.h | 133 +++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 116 deletions(-) create mode 100644 include/linux/mlx5/mlx5_ifc_macros.h diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 8646303b029d..37727b5ab78f 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -36,6 +36,7 @@ #include #include #include +#include #include #if defined(__LITTLE_ENDIAN) @@ -46,122 +47,6 @@ #error Host endianness not defined #endif -/* helper macros */ -#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0) -#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld) -#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld)) -#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16) -#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32) -#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64) -#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf)) -#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f)) -#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) -#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld)) -#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) -#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld)) -#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) - -#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) -#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) -#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) -#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) -#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8) -#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32) -#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8) -#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld))) - -/* insert a value to a struct */ -#define MLX5_SET(typ, p, fld, v) do { \ - u32 _v = v; \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ - *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ - cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ - (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \ - << __mlx5_dw_bit_off(typ, fld))); \ -} while (0) - -#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \ - MLX5_SET(typ, p, fld[idx], v); \ -} while (0) - -#define MLX5_SET_TO_ONES(typ, p, fld) do { \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ - *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ - cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ - (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \ - << __mlx5_dw_bit_off(typ, fld))); \ -} while (0) - -#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\ -__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \ -__mlx5_mask(typ, fld)) - -#define MLX5_GET_PR(typ, p, fld) ({ \ - u32 ___t = MLX5_GET(typ, p, fld); \ - pr_debug(#fld " = 0x%x\n", ___t); \ - ___t; \ -}) - -#define __MLX5_SET64(typ, p, fld, v) do { \ - BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \ - *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \ -} while (0) - -#define MLX5_SET64(typ, p, fld, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ - __MLX5_SET64(typ, p, fld, v); \ -} while (0) - -#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ - __MLX5_SET64(typ, p, fld[idx], v); \ -} while (0) - -#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld))) - -#define MLX5_GET64_PR(typ, p, fld) ({ \ - u64 ___t = MLX5_GET64(typ, p, fld); \ - pr_debug(#fld " = 0x%llx\n", ___t); \ - ___t; \ -}) - -#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\ -__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \ -__mlx5_mask16(typ, fld)) - -#define MLX5_SET16(typ, p, fld, v) do { \ - u16 _v = v; \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \ - *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \ - cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \ - (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \ - << __mlx5_16_bit_off(typ, fld))); \ -} while (0) - -/* Big endian getters */ -#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\ - __mlx5_64_off(typ, fld))) - -#define MLX5_GET_BE(type_t, typ, p, fld) ({ \ - type_t tmp; \ - switch (sizeof(tmp)) { \ - case sizeof(u8): \ - tmp = (__force type_t)MLX5_GET(typ, p, fld); \ - break; \ - case sizeof(u16): \ - tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \ - break; \ - case sizeof(u32): \ - tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \ - break; \ - case sizeof(u64): \ - tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \ - break; \ - } \ - tmp; \ - }) - enum mlx5_inline_modes { MLX5_INLINE_MODE_NONE, MLX5_INLINE_MODE_L2, diff --git a/include/linux/mlx5/mlx5_ifc_macros.h b/include/linux/mlx5/mlx5_ifc_macros.h new file mode 100644 index 000000000000..d357acfd351d --- /dev/null +++ b/include/linux/mlx5/mlx5_ifc_macros.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * Copyright (c) 2013-2026, Mellanox Technologies. All rights reserved. + * + * Accessor macros for mlx5 IFC structures. + * + * Extracted from device.h so that code which cannot include device.h + * (e.g. selftests) can still use the MLX5_SET/GET family directly. + */ + +#ifndef MLX5_IFC_MACROS_H +#define MLX5_IFC_MACROS_H + +/* Internal helpers -- 32-bit */ +#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0) +#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld) +#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld)) +#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16) +#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32) +#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64) +#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf)) +#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f)) +#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) +#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld)) +#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) +#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld)) +#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) + +/* Size and address macros */ +#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) +#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) +#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) +#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) +#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8) +#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32) +#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8) +#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld))) + +/* insert a value to a struct */ +#define MLX5_SET(typ, p, fld, v) do { \ + u32 _v = v; \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ + cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ + (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld))); \ +} while (0) + +#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \ + MLX5_SET(typ, p, fld[idx], v); \ +} while (0) + +#define MLX5_SET_TO_ONES(typ, p, fld) do { \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ + cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ + (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld))); \ +} while (0) + +#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\ +__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \ +__mlx5_mask(typ, fld)) + +#define MLX5_GET_PR(typ, p, fld) ({ \ + u32 ___t = MLX5_GET(typ, p, fld); \ + pr_debug(#fld " = 0x%x\n", ___t); \ + ___t; \ +}) + +/* 64-bit field accessors */ +#define __MLX5_SET64(typ, p, fld, v) do { \ + BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \ + *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \ +} while (0) + +#define MLX5_SET64(typ, p, fld, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + __MLX5_SET64(typ, p, fld, v); \ +} while (0) + +#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + __MLX5_SET64(typ, p, fld[idx], v); \ +} while (0) + +#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld))) + +#define MLX5_GET64_PR(typ, p, fld) ({ \ + u64 ___t = MLX5_GET64(typ, p, fld); \ + pr_debug(#fld " = 0x%llx\n", ___t); \ + ___t; \ +}) + +/* 16-bit field accessors */ +#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\ +__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \ +__mlx5_mask16(typ, fld)) + +#define MLX5_SET16(typ, p, fld, v) do { \ + u16 _v = v; \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \ + *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \ + cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \ + (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \ + << __mlx5_16_bit_off(typ, fld))); \ +} while (0) + +/* Big endian getters */ +#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\ + __mlx5_64_off(typ, fld))) + +#define MLX5_GET_BE(type_t, typ, p, fld) ({ \ + type_t tmp; \ + switch (sizeof(tmp)) { \ + case sizeof(u8): \ + tmp = (__force type_t)MLX5_GET(typ, p, fld); \ + break; \ + case sizeof(u16): \ + tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \ + break; \ + case sizeof(u32): \ + tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \ + break; \ + case sizeof(u64): \ + tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \ + break; \ + } \ + tmp; \ + }) + +#endif /* MLX5_IFC_MACROS_H */ -- cgit v1.2.3 From 5f8e94a75a11661184de9e2d8aaeca095554c6f7 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:44 -0300 Subject: net/mlx5: Add ONCE and MMIO accessor variants to mlx5_ifc_macros.h Add MLX5_GET_ONCE / MLX5_SET_ONCE which include READ_ONCE/WRITE_ONCE semantics for touching ownership bits that hardware may read or write at any time. The kernel driver doesn't use the IFC struct for these ownership bits, but the VFIO driver will and needs these helpers. Add MLX5_GET_MMIO / MLX5_SET_MMIO for accessing the init seg using the IFC offsets. They embed a readl/writel using the IFC struct to generate the addressing. Add MLX5_ARRAY_GET64(), the missing counterpart to MLX5_ARRAY_SET64(). Assisted-by: Claude:claude-opus-4.6 Signed-off-by: Jason Gunthorpe Acked-by: Leon Romanovsky Link: https://lore.kernel.org/r/4-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- include/linux/mlx5/mlx5_ifc_macros.h | 52 ++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/include/linux/mlx5/mlx5_ifc_macros.h b/include/linux/mlx5/mlx5_ifc_macros.h index d357acfd351d..1b9e39a50949 100644 --- a/include/linux/mlx5/mlx5_ifc_macros.h +++ b/include/linux/mlx5/mlx5_ifc_macros.h @@ -85,6 +85,13 @@ __mlx5_mask(typ, fld)) __MLX5_SET64(typ, p, fld[idx], v); \ } while (0) +#define MLX5_ARRAY_GET64(typ, p, fld, idx) \ + ({ \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + be64_to_cpu( \ + *((__be64 *)(p) + __mlx5_64_off(typ, fld) + (idx))); \ + }) + #define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld))) #define MLX5_GET64_PR(typ, p, fld) ({ \ @@ -130,4 +137,49 @@ __mlx5_mask16(typ, fld)) tmp; \ }) +/* + * Use READ_ONCE/WRITE_ONCE for a single field that hardware may read/write + * unpredictably, mostly owner bits. All other bits in the DW must be stable. + * Usually a dma_wmb() will be required before a write and a dma_rmb() after a + * read. + */ +#define MLX5_GET_ONCE(typ, p, fld) \ + ((be32_to_cpu(READ_ONCE(*((__be32 *)(p) + __mlx5_dw_off(typ, fld)))) >> \ + __mlx5_dw_bit_off(typ, fld)) & \ + __mlx5_mask(typ, fld)) + +#define MLX5_SET_ONCE(typ, p, fld, v) \ + do { \ + u32 _v = v; \ + __be32 *_dw = (__be32 *)(p) + __mlx5_dw_off(typ, fld); \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + WRITE_ONCE(*_dw, \ + cpu_to_be32((be32_to_cpu(READ_ONCE(*_dw)) & \ + (~__mlx5_dw_mask(typ, fld))) | \ + (((_v) & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld)))); \ + } while (0) + +/* Access MMIO registers, usually the init segment, using IFC structs. */ +#define MLX5_GET_MMIO(typ, p, fld) \ + ((ioread32be(((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld))) >> \ + __mlx5_dw_bit_off(typ, fld)) & \ + __mlx5_mask(typ, fld)) + +/* The set is not relaxed so there is an integrated dma_wmb(). */ +#define MLX5_SET_MMIO(typ, p, fld, v) \ + do { \ + u32 _v = v; \ + void __iomem *_dw = \ + ((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld)); \ + if (__mlx5_bit_sz(typ, fld) == 32) \ + iowrite32be(_v, _dw); \ + else \ + iowrite32be((ioread32be(_dw) & \ + (~__mlx5_dw_mask(typ, fld))) | \ + ((_v & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld)), \ + _dw); \ + } while (0) + #endif /* MLX5_IFC_MACROS_H */ -- cgit v1.2.3 From be0237a5f35df3518953075c375cc07fe09aef38 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:45 -0300 Subject: selftests: Add additional kernel functions to tools/include/ These are needed by the VFIO mlx5 selftest in the following patches, which includes some headers from mlx5 and also needs a few more MMIO-related features. - DECLARE_FLEX_ARRAY in new tools/include/linux/stddef.h (wraps existing __DECLARE_FLEX_ARRAY from uapi/linux/stddef.h) - dma_wmb/dma_rmb barriers: x86 uses compiler barrier (DMA-coherent), arm64 uses dmb oshst/oshld (outer-shareable for device visibility), generic fallback uses wmb/rmb - ioread32be/iowrite32be in tools/include/asm-generic/io.h for big-endian MMIO register access Assisted-by: Claude:claude-opus-4.6 Reviewed-by: David Matlack Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/5-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- tools/arch/arm64/include/asm/barrier.h | 4 ++++ tools/arch/x86/include/asm/barrier.h | 5 ++++ tools/include/asm-generic/io.h | 28 ++++++++++++++++++++++ tools/include/asm/barrier.h | 8 +++++++ tools/include/linux/stddef.h | 10 ++++++++ .../selftests/kvm/include/arm64/processor.h | 4 +--- 6 files changed, 56 insertions(+), 3 deletions(-) create mode 100644 tools/include/linux/stddef.h diff --git a/tools/arch/arm64/include/asm/barrier.h b/tools/arch/arm64/include/asm/barrier.h index 3b9b41331c4f..abdc64fc3c70 100644 --- a/tools/arch/arm64/include/asm/barrier.h +++ b/tools/arch/arm64/include/asm/barrier.h @@ -24,6 +24,10 @@ #define smp_wmb() asm volatile("dmb ishst" ::: "memory") #define smp_rmb() asm volatile("dmb ishld" ::: "memory") +/* DMA barriers use outer-shareable (osh) for device visibility */ +#define dma_rmb() asm volatile("dmb oshld" ::: "memory") +#define dma_wmb() asm volatile("dmb oshst" ::: "memory") + #define smp_store_release(p, v) \ do { \ union { typeof(*p) __val; char __c[1]; } __u = \ diff --git a/tools/arch/x86/include/asm/barrier.h b/tools/arch/x86/include/asm/barrier.h index 0adf295dd5b6..0b51431fa530 100644 --- a/tools/arch/x86/include/asm/barrier.h +++ b/tools/arch/x86/include/asm/barrier.h @@ -43,4 +43,9 @@ do { \ ___p1; \ }) #endif /* defined(__x86_64__) */ + +/* x86 is DMA-coherent so DMA barriers are just compiler barriers */ +#define dma_rmb() barrier() +#define dma_wmb() barrier() + #endif /* _TOOLS_LINUX_ASM_X86_BARRIER_H */ diff --git a/tools/include/asm-generic/io.h b/tools/include/asm-generic/io.h index e5a0b07ad452..0d89decdafb8 100644 --- a/tools/include/asm-generic/io.h +++ b/tools/include/asm-generic/io.h @@ -479,4 +479,32 @@ static inline void writesq(volatile void __iomem *addr, const void *buffer, } #endif +/* + * ioread/iowrite for big-endian MMIO registers. + */ + +#ifndef ioread32be +#define ioread32be ioread32be +static inline u32 ioread32be(const volatile void __iomem *addr) +{ + return bswap_32(readl(addr)); +} +#endif + +#ifndef iowrite32be +#define iowrite32be iowrite32be +static inline void iowrite32be(u32 value, volatile void __iomem *addr) +{ + writel(bswap_32(value), addr); +} +#endif + +#ifndef iowrite64be +#define iowrite64be iowrite64be +static inline void iowrite64be(u64 value, volatile void __iomem *addr) +{ + writeq(bswap_64(value), addr); +} +#endif + #endif /* _TOOLS_ASM_GENERIC_IO_H */ diff --git a/tools/include/asm/barrier.h b/tools/include/asm/barrier.h index 0c21678ac5e6..e7e0c7de5a2f 100644 --- a/tools/include/asm/barrier.h +++ b/tools/include/asm/barrier.h @@ -47,6 +47,14 @@ # define smp_mb() mb() #endif +#ifndef dma_rmb +# define dma_rmb() rmb() +#endif + +#ifndef dma_wmb +# define dma_wmb() wmb() +#endif + #ifndef smp_store_release # define smp_store_release(p, v) \ do { \ diff --git a/tools/include/linux/stddef.h b/tools/include/linux/stddef.h new file mode 100644 index 000000000000..99182ea4a141 --- /dev/null +++ b/tools/include/linux/stddef.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TOOLS_LINUX_STDDEF_H +#define _TOOLS_LINUX_STDDEF_H + +#include_next + +#define DECLARE_FLEX_ARRAY(TYPE, NAME) \ + __DECLARE_FLEX_ARRAY(TYPE, NAME) + +#endif /* _TOOLS_LINUX_STDDEF_H */ diff --git a/tools/testing/selftests/kvm/include/arm64/processor.h b/tools/testing/selftests/kvm/include/arm64/processor.h index b8a902ba8573..5878a6abdf20 100644 --- a/tools/testing/selftests/kvm/include/arm64/processor.h +++ b/tools/testing/selftests/kvm/include/arm64/processor.h @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -191,11 +192,8 @@ static inline void cpu_relax(void) #define dsb(opt) asm volatile("dsb " #opt : : : "memory") #define dmb(opt) asm volatile("dmb " #opt : : : "memory") -#define dma_wmb() dmb(oshst) #define __iowmb() dma_wmb() -#define dma_rmb() dmb(oshld) - #define __iormb(v) \ ({ \ unsigned long tmp; \ -- cgit v1.2.3 From 53e5fe5f28df72dd5917b74665465d42b2f0e90e Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:46 -0300 Subject: vfio: selftests: Allow drivers to specify required region size Add a region_size field to struct vfio_pci_driver_ops so drivers can declare how much DMA-mapped region they need. The mlx5 driver will need ~18MB for firmware pages. Existing drivers pass in the sizeof their state struct. Since drivers all use only their declared state we can use the value directly and remove the SZ_2M hard coding. Assisted-by: Claude:claude-opus-4.6 Signed-off-by: Jason Gunthorpe Acked-by: Sean Christopherson Reviewed-by: David Matlack Link: https://lore.kernel.org/r/6-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- tools/testing/selftests/kvm/irq_test.c | 2 -- tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c | 1 + tools/testing/selftests/vfio/lib/drivers/igb/igb.c | 1 + tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c | 1 + tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c | 1 + tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h | 6 ++++++ tools/testing/selftests/vfio/lib/iova_allocator.c | 7 ++++++- tools/testing/selftests/vfio/lib/vfio_pci_driver.c | 4 ++++ tools/testing/selftests/vfio/vfio_pci_driver_test.c | 3 ++- 9 files changed, 22 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/kvm/irq_test.c b/tools/testing/selftests/kvm/irq_test.c index 168c52734fef..791e576c334c 100644 --- a/tools/testing/selftests/kvm/irq_test.c +++ b/tools/testing/selftests/kvm/irq_test.c @@ -6,7 +6,6 @@ #include "proc_util.h" #include -#include #include #include #include @@ -101,7 +100,6 @@ static int vfio_setup_msi(struct vfio_pci_device *device) allocator = iova_allocator_init(device->iommu); region = &device->driver.region; - region->size = SZ_2M; region->iova = iova_allocator_alloc(allocator, region->size); region->vaddr = kvm_mmap(region->size, prot, flags, -1); TEST_ASSERT(region->vaddr != MAP_FAILED, "mmap() failed\n"); diff --git a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c index 19d9630b24c2..40b8541b588e 100644 --- a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c +++ b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c @@ -418,6 +418,7 @@ static void dsa_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops dsa_ops = { .name = "dsa", + .region_size = sizeof(struct dsa_state), .probe = dsa_probe, .init = dsa_init, .remove = dsa_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c index fd9e05d77ea4..2fafcc9f51c0 100644 --- a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c +++ b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c @@ -576,6 +576,7 @@ static void igb_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops igb_ops = { .name = "igb", + .region_size = sizeof(struct igb), .probe = igb_probe, .init = igb_init, .remove = igb_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c index a871b935542b..c9b28365c5eb 100644 --- a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c +++ b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c @@ -226,6 +226,7 @@ static void ioat_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops ioat_ops = { .name = "ioat", + .region_size = sizeof(struct ioat_state), .probe = ioat_probe, .init = ioat_init, .remove = ioat_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c index c08aa81c44f4..6958877ad463 100644 --- a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c +++ b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c @@ -775,6 +775,7 @@ static int nv_falcon_memcpy_wait(struct vfio_pci_device *device) const struct vfio_pci_driver_ops nv_falcon_ops = { .name = "nv_falcon", + .region_size = sizeof(struct gpu_device), .probe = nv_falcon_probe, .init = nv_falcon_init, .remove = nv_falcon_remove, diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h index e5ada209b1d1..547369c5cff9 100644 --- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h +++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h @@ -9,6 +9,12 @@ struct vfio_pci_device; struct vfio_pci_driver_ops { const char *name; + /* + * Size of the driver's state structure overlaid on + * device->driver.region.vaddr + */ + u64 region_size; + /** * @probe() - Check if the driver supports the given device. * diff --git a/tools/testing/selftests/vfio/lib/iova_allocator.c b/tools/testing/selftests/vfio/lib/iova_allocator.c index 4a660f636f49..a8e5815a1b16 100644 --- a/tools/testing/selftests/vfio/lib/iova_allocator.c +++ b/tools/testing/selftests/vfio/lib/iova_allocator.c @@ -13,8 +13,10 @@ #include #include +#include #include #include +#include #include #include @@ -50,7 +52,10 @@ void iova_allocator_cleanup(struct iova_allocator *allocator) iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size) { VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n"); - VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n"); + + VFIO_ASSERT_LE(size, rounddown_pow_of_two(SIZE_MAX), + "Invalid size arg, too large (%zu)\n", size); + size = roundup_pow_of_two(size); for (;;) { struct iommu_iova_range *range; diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c index 5e65434d2318..64135ac43877 100644 --- a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c +++ b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c @@ -32,6 +32,10 @@ void vfio_pci_driver_probe(struct vfio_pci_device *device) continue; device->driver.ops = ops; + + VFIO_ASSERT_NE(ops->region_size, 0); + device->driver.region.size = + round_up(ops->region_size, getpagesize()); } } diff --git a/tools/testing/selftests/vfio/vfio_pci_driver_test.c b/tools/testing/selftests/vfio/vfio_pci_driver_test.c index 761bf117d624..2bd949aafd73 100644 --- a/tools/testing/selftests/vfio/vfio_pci_driver_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_driver_test.c @@ -87,7 +87,8 @@ FIXTURE_SETUP(vfio_pci_driver_test) driver = &self->device->driver; region_setup(self->iommu, self->iova_allocator, &self->memcpy_region, SZ_1G); - region_setup(self->iommu, self->iova_allocator, &driver->region, SZ_2M); + region_setup(self->iommu, self->iova_allocator, &driver->region, + driver->region.size); /* Any IOVA that doesn't overlap memcpy_region and driver->region. */ self->unmapped_iova = iova_allocator_alloc(self->iova_allocator, SZ_1G); -- cgit v1.2.3 From 4fef48b937f74571cad394dbc2c60c87756cdbaf Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:47 -0300 Subject: vfio: selftests: Add dev_dbg Enable it with a #define DEBUG at the top of the file. Allows leaving behind debugging prints that are useful in case future changes are required. Assisted-by: Claude:claude-opus-4.6 Reviewed-by: David Matlack Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/7-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- .../selftests/vfio/lib/include/libvfio/vfio_pci_device.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h index e19bd94b8dd2..007adc322c1e 100644 --- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h @@ -40,6 +40,17 @@ struct vfio_pci_device { struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu); void vfio_pci_device_free(struct vfio_pci_device *device); + +#ifdef DEBUG +#define dev_dbg dev_info +#else +#define dev_dbg(_dev, _fmt, ...) \ + do { \ + if (0) \ + dev_info(_dev, _fmt, ##__VA_ARGS__); \ + } while (0) +#endif + struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu); void vfio_pci_device_cleanup(struct vfio_pci_device *device); -- cgit v1.2.3 From 3eaeaf318a08fb881c17e519b0db6c5a7b1be586 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:48 -0300 Subject: vfio: selftests: Add mlx5 driver - HW init and command interface Add an mlx5 ConnectX selftest driver that programs VFs and PFs through the command interface. Create the driver skeleton with probe(), the command interface, HCA boot sequence, and all resource allocation up through ALLOC_PD + CREATE_MKEY. The driver implements vfio_pci_driver_ops (probe/init/remove) and registers with the VFIO selftest framework. It matches the same mlx5 ConnectX device IDs that the kernel matches. init() brings the HCA to a running state: - Command interface setup (two-slot: regular + async pages) - ENABLE_HCA -> SET_ISSI -> QUERY/MANAGE_PAGES -> SET_HCA_CAP -> INIT_HCA - EQ creation (CMD + PAGE_REQUEST events for PF support) - ALLOC_PD, CREATE_MKEY (PA-mode, covers all IOVAs) - Force-loopback capability check remove() tears down in reverse order through TEARDOWN_HCA and DISABLE_HCA, including async pages slot drain and FW page reclaim. The driver region (~18MB) holds all HW-visible DMA buffers (command queue, mailboxes, EQ, FW pages) overlaid as a single struct on device->driver.region.vaddr. Data path ops (memcpy_start/memcpy_wait) are left as stubs for the next patch. Assisted-by: Claude:claude-opus-4.6 Acked-by: David Matlack Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/8-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- .../testing/selftests/vfio/lib/drivers/mlx5/mlx5.c | 1414 ++++++++++++++++++++ .../selftests/vfio/lib/drivers/mlx5/mlx5_hw.h | 108 ++ .../selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h | 1 + .../vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h | 1 + .../vfio/lib/drivers/mlx5/mlx5_ifc_macros.h | 1 + tools/testing/selftests/vfio/lib/libvfio.mk | 1 + tools/testing/selftests/vfio/lib/vfio_pci_driver.c | 2 + 7 files changed, 1528 insertions(+) create mode 100644 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c create mode 100644 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h create mode 120000 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h create mode 120000 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h create mode 120000 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c new file mode 100644 index 000000000000..5a87d070dc42 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c @@ -0,0 +1,1414 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * mlx5 VFIO selftest driver + * + * Programs mlx5 ConnectX VFs and PFs through the bare-metal command interface + * and RDMA WRITE self-loopback to perform DMA. Implements vfio_pci_driver_ops + * (probe/init/remove) and plugs into the VFIO selftest framework. + */ +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +#include "mlx5_hw.h" + +/* + * Driver state — overlaid on device->driver.region.vaddr. + * + * Contains both software-only state and HW-visible DMA buffers. HW buffers need + * strict IOVA alignment. + */ +struct mlx5st_device { + /* Back pointer */ + struct vfio_pci_device *device; + + /* BAR0 */ + struct mlx5st_initial_seg __iomem *init_seg; + void __iomem *bar0; + + /* Command interface */ + struct mlx5st_cmd_queue_entry *cmd_lay; + struct mlx5st_cmd_queue_entry *pages_cmd_lay; + u8 cmd_log_stride; + unsigned int pages_slot; + u8 cmd_token; + bool cmd_sig_enabled; + + /* PD */ + u32 pdn; + + /* Global PA-mode MKEY */ + u32 global_lkey; + u32 global_rkey; + u32 mkey_index; + + /* CQ */ + u32 cqn; + u32 cq_ci; + + /* UAR */ + u32 uar_page; + void __iomem *uar_base; + unsigned int uar_bf_offset; + + /* EQ */ + u32 eqn; + u32 eq_cons_index; + bool have_eq; + + /* Async pages slot state */ + bool pages_slot_in_use; + bool pages_slot_is_reclaim; + unsigned int pages_reclaim_npages; + unsigned int pages_pending_give; + unsigned int pages_pending_reclaim; + u16 pages_pending_func_id; + bool pages_func_id_seen; + + /* QP */ + u32 qpn; + u32 sq_pi; + u32 sq_ci; + + /* FW pages bitmap */ + u64 fw_pages_bitmap[MAX_FW_PAGES / 64]; + u32 fw_pages_given; + u16 fw_func_id; + + /* Capabilities */ + bool fl_supported; + u8 log_max_msg; + + /* + * HW-visible DMA buffers below — device reads/writes via DMA. + */ + struct mlx5st_cmd_queue_entry cmd_queue + [MLX5_HW_PAGE_SIZE / sizeof(struct mlx5st_cmd_queue_entry)] + __aligned(MLX5_HW_PAGE_SIZE); + struct mlx5st_send_wqe sq_buf[SQ_WQE_CNT]; + struct mlx5st_dbrec cq_dbrec; + struct mlx5st_dbrec qp_dbrec; + struct mlx5st_cqe64 cq_buf[CQ_CQE_CNT]; + + /* Slot 0 mailboxes (regular commands) */ + struct mlx5st_mbox_entry cmd_in_mbox[CMD_MBOX_NENT]; + struct mlx5st_mbox_entry cmd_out_mbox[CMD_MBOX_NENT]; + + /* Pages slot mailboxes (async MANAGE_PAGES) */ + struct mlx5st_mbox_entry pages_in_mbox[CMD_MBOX_NENT]; + struct mlx5st_mbox_entry pages_out_mbox[CMD_MBOX_NENT]; + + /* EQ does not support page_offset */ + struct mlx5st_eqe eq_buf[EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE); + + u8 fw_pages[MAX_FW_PAGES][MLX5_HW_PAGE_SIZE] + __aligned(MLX5_HW_PAGE_SIZE); +}; + +/* Check against HW limits on IOVA alignment */ +static_assert(offsetof(struct mlx5st_device, cmd_in_mbox) % + CMD_MBOX_STRIDE == 0, + "cmd_in_mbox must be stride-aligned"); +static_assert(offsetof(struct mlx5st_device, pages_in_mbox) % + CMD_MBOX_STRIDE == 0, + "pages_in_mbox must be stride-aligned"); +static_assert(offsetof(struct mlx5st_device, cq_buf) % 64 == 0, + "cq_buf must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, sq_buf) % 64 == 0, + "sq_buf must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, cq_dbrec) % 64 == 0, + "cq_dbrec must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, qp_dbrec) % 64 == 0, + "qp_dbrec must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, eq_buf) % + MLX5_HW_PAGE_SIZE == 0, + "eq_buf must be page-aligned"); +static_assert(offsetof(struct mlx5st_device, fw_pages) % + MLX5_HW_PAGE_SIZE == 0, + "fw_pages must be page-aligned"); + +static struct mlx5st_device *to_mlx5st(struct vfio_pci_device *device) +{ + return device->driver.region.vaddr; +} + +/* + * Fill a PAS (Physical Address Segment) for a buffer in the driver region. + * Sets pas[0] to the page-aligned IOVA and returns the page_offset (the + * buffer's byte offset within that page, in units of 64 bytes). + */ +static unsigned int mlx5st_fill_pas(struct vfio_pci_device *device, void *buf, + __be64 *pas) +{ + u64 iova = to_iova(device, buf); + + pas[0] = cpu_to_be64(iova & ~(u64)(MLX5_HW_PAGE_SIZE - 1)); + return (iova & (MLX5_HW_PAGE_SIZE - 1)) / 64; +} + +/* + * Probe — match mlx5 devices by PCI vendor/device ID. + */ + +static int mlx5st_probe(struct vfio_pci_device *device) +{ + static const u16 mlx5st_pci_ids[] = { + 0x1011, /* Connect-IB */ + 0x1012, /* Connect-IB VF */ + 0x1013, /* ConnectX-4 */ + 0x1014, /* ConnectX-4 VF */ + 0x1015, /* ConnectX-4LX */ + 0x1016, /* ConnectX-4LX VF */ + 0x1017, /* ConnectX-5 */ + 0x1018, /* ConnectX-5 VF */ + 0x1019, /* ConnectX-5 Ex */ + 0x101a, /* ConnectX-5 Ex VF */ + 0x101b, /* ConnectX-6 */ + 0x101c, /* ConnectX-6 VF */ + 0x101d, /* ConnectX-6 Dx */ + 0x101e, /* ConnectX-6 Dx VF */ + 0x101f, /* ConnectX-6 LX */ + 0x1021, /* ConnectX-7 */ + 0x1023, /* ConnectX-8 */ + 0x1025, /* ConnectX-9 */ + 0x1027, /* ConnectX-10 */ + 0x2101, /* ConnectX-10 NVLink-C2C */ + 0xa2d2, /* BlueField integrated ConnectX-5 */ + 0xa2d3, /* BlueField integrated ConnectX-5 VF */ + 0xa2d6, /* BlueField-2 integrated ConnectX-6 Dx */ + 0xa2dc, /* BlueField-3 integrated ConnectX-7 */ + 0xa2df, /* BlueField-4 integrated ConnectX-8 */ + }; + unsigned int i; + u16 did; + + if (vfio_pci_config_readw(device, PCI_VENDOR_ID) != + PCI_VENDOR_ID_MELLANOX) + return -ENODEV; + + did = vfio_pci_config_readw(device, PCI_DEVICE_ID); + for (i = 0; i < ARRAY_SIZE(mlx5st_pci_ids); i++) { + if (mlx5st_pci_ids[i] == did) + return 0; + } + + return -ENODEV; +} + +/* + * Command interface + */ + +static u8 xor8_buf(const void *buf, size_t offset, size_t len) +{ + const u8 *p = buf; + u8 sum = 0; + size_t i; + + for (i = offset; i < offset + len; i++) + sum ^= p[i]; + return sum; +} + +#define CMD_IF_BOX_CTRL_OFF MLX5_BYTE_OFF(cmd_if_box, reserved_at_1000) +#define CMD_IF_BOX_CTRL_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, ctrl_signature) +#define CMD_IF_BOX_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, signature) + +static void mlx5st_cmd_calc_block_sig(struct mlx5st_cmd_if_box *blk) +{ + MLX5_SET(cmd_if_box, blk, ctrl_signature, + ~xor8_buf(blk, CMD_IF_BOX_CTRL_OFF, + CMD_IF_BOX_CTRL_SIG_OFF - CMD_IF_BOX_CTRL_OFF)); + MLX5_SET(cmd_if_box, blk, signature, + ~xor8_buf(blk, 0, CMD_IF_BOX_SIG_OFF)); +} + +static int mlx5st_cmd_verify_block_sig(struct mlx5st_cmd_if_box *blk) +{ + if (xor8_buf(blk, CMD_IF_BOX_CTRL_OFF, + CMD_IF_BOX_SIG_OFF - CMD_IF_BOX_CTRL_OFF) != 0xff) + return -1; + if (xor8_buf(blk, 0, sizeof(struct mlx5st_cmd_if_box)) != 0xff) + return -1; + return 0; +} + +static unsigned int mlx5st_cmd_setup_mbox_chain(struct vfio_pci_device *device, + struct mlx5st_mbox_entry *mbox, + unsigned int nblocks, u8 token) +{ + unsigned int i; + + for (i = 0; i < nblocks; i++) { + struct mlx5st_cmd_if_box *blk = &mbox[i].block; + u64 next_iova; + + memset(blk, 0, sizeof(struct mlx5st_cmd_if_box)); + MLX5_SET(cmd_if_box, blk, block_number, i); + MLX5_SET(cmd_if_box, blk, token, token); + if (i < nblocks - 1) { + next_iova = to_iova(device, &mbox[i + 1]); + MLX5_SET(cmd_if_box, blk, next_pointer_63_32, + next_iova >> 32); + MLX5_SET(cmd_if_box, blk, next_pointer_31_10, + (u32)next_iova >> 10); + } + } + return nblocks; +} + +static void mlx5st_cmd_copy_to_mbox(struct mlx5st_mbox_entry *mbox, + const void *data, unsigned int len) +{ + const u8 *src = data; + unsigned int i = 0; + + while (len > 0) { + unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ? + len : + MLX5_CMD_DATA_BLOCK_SIZE; + + memcpy(MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data), + src, chunk); + src += chunk; + len -= chunk; + i++; + } +} + +static void mlx5st_cmd_copy_from_mbox(void *data, + const struct mlx5st_mbox_entry *mbox, + unsigned int len) +{ + unsigned int i = 0; + u8 *dst = data; + + while (len > 0) { + unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ? + len : + MLX5_CMD_DATA_BLOCK_SIZE; + + memcpy(dst, + MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data), + chunk); + dst += chunk; + len -= chunk; + i++; + } +} + +/* Forward declaration — cmd_exec polls events during command wait */ +static void mlx5st_process_events(struct mlx5st_device *dev); + +static const char *mlx5st_cmd_name(u16 opcode) +{ + switch (opcode) { + case MLX5_CMD_OP_QUERY_HCA_CAP: return "QUERY_HCA_CAP"; + case MLX5_CMD_OP_INIT_HCA: return "INIT_HCA"; + case MLX5_CMD_OP_TEARDOWN_HCA: return "TEARDOWN_HCA"; + case MLX5_CMD_OP_ENABLE_HCA: return "ENABLE_HCA"; + case MLX5_CMD_OP_DISABLE_HCA: return "DISABLE_HCA"; + case MLX5_CMD_OP_QUERY_PAGES: return "QUERY_PAGES"; + case MLX5_CMD_OP_MANAGE_PAGES: return "MANAGE_PAGES"; + case MLX5_CMD_OP_SET_HCA_CAP: return "SET_HCA_CAP"; + case MLX5_CMD_OP_SET_ISSI: return "SET_ISSI"; + case MLX5_CMD_OP_CREATE_MKEY: return "CREATE_MKEY"; + case MLX5_CMD_OP_DESTROY_MKEY: return "DESTROY_MKEY"; + case MLX5_CMD_OP_CREATE_EQ: return "CREATE_EQ"; + case MLX5_CMD_OP_DESTROY_EQ: return "DESTROY_EQ"; + case MLX5_CMD_OP_CREATE_CQ: return "CREATE_CQ"; + case MLX5_CMD_OP_DESTROY_CQ: return "DESTROY_CQ"; + case MLX5_CMD_OP_CREATE_QP: return "CREATE_QP"; + case MLX5_CMD_OP_DESTROY_QP: return "DESTROY_QP"; + case MLX5_CMD_OP_RST2INIT_QP: return "RST2INIT_QP"; + case MLX5_CMD_OP_INIT2RTR_QP: return "INIT2RTR_QP"; + case MLX5_CMD_OP_RTR2RTS_QP: return "RTR2RTS_QP"; + case MLX5_CMD_OP_ALLOC_PD: return "ALLOC_PD"; + case MLX5_CMD_OP_DEALLOC_PD: return "DEALLOC_PD"; + case MLX5_CMD_OP_ALLOC_UAR: return "ALLOC_UAR"; + case MLX5_CMD_OP_DEALLOC_UAR: return "DEALLOC_UAR"; + default: return "UNKNOWN"; + } +} + +/* + * Post a command on a given slot: fill the cmd_queue_entry, set up mailbox + * chains, compute signatures, hand ownership to FW, and ring the doorbell. + */ +static void mlx5st_cmd_post(struct mlx5st_device *dev, + struct mlx5st_cmd_queue_entry *cmd, + struct mlx5st_mbox_entry *in_mbox, + struct mlx5st_mbox_entry *out_mbox, + void *in, unsigned int ilen, unsigned int olen, + u32 doorbell) +{ + struct vfio_pci_device *device = dev->device; + unsigned int in_remain, out_remain, in_nblk, out_nblk; + unsigned int i; + void *cin, *cout; + u8 token; + + /* Rotating non-zero token ties cmd entry to its mailbox blocks */ + token = ++dev->cmd_token; + if (!token) + token = ++dev->cmd_token; + + in_remain = ilen > MLX5_CMD_INLINE_SZ ? ilen - MLX5_CMD_INLINE_SZ : 0; + out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0; + in_nblk = (in_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + + /* Set up mailbox chains */ + if (in_nblk > 0) { + mlx5st_cmd_setup_mbox_chain(device, in_mbox, in_nblk, token); + mlx5st_cmd_copy_to_mbox(in_mbox, + (u8 *)in + MLX5_CMD_INLINE_SZ, + in_remain); + } + if (out_nblk > 0) + mlx5st_cmd_setup_mbox_chain(device, out_mbox, out_nblk, token); + + /* Copy inline input */ + cin = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_input_inline_data); + memset(cin, 0, MLX5_CMD_INLINE_SZ); + memcpy(cin, in, ilen < MLX5_CMD_INLINE_SZ ? ilen : MLX5_CMD_INLINE_SZ); + MLX5_SET(cmd_queue_entry, cmd, input_length, ilen); + MLX5_SET(cmd_queue_entry, cmd, token, token); + + /* Zero inline output */ + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + memset(cout, 0, MLX5_CMD_INLINE_SZ); + MLX5_SET(cmd_queue_entry, cmd, output_length, olen); + + /* + * Compute signatures: mailbox blocks first, then cmd_queue_entry. + * The sig must cover the final state including ownership=0x1, but + * we must not set ownership until after the sig is in place - + * XOR in the 0x1 without storing it to memory. + */ + for (i = 0; i < in_nblk; i++) + mlx5st_cmd_calc_block_sig(&in_mbox[i].block); + for (i = 0; i < out_nblk; i++) + mlx5st_cmd_calc_block_sig(&out_mbox[i].block); + MLX5_SET(cmd_queue_entry, cmd, signature, 0); + MLX5_SET(cmd_queue_entry, cmd, signature, + ~(xor8_buf(cmd, 0, sizeof(struct mlx5st_cmd_queue_entry)) ^ + 0x1)); + + /* Ensure all cmd data (including sig) is visible, then hand to FW */ + dma_wmb(); + MLX5_SET_ONCE(cmd_queue_entry, cmd, ownership, 1); + + /* Ring doorbell */ + MLX5_SET_MMIO(initial_seg, dev->init_seg, command_doorbell_vector, + doorbell); +} + +static void mlx5st_cmd_exec(struct mlx5st_device *dev, void *in, + unsigned int ilen, void *out, unsigned int olen) +{ + struct mlx5st_cmd_queue_entry *cmd = dev->cmd_lay; + unsigned int out_remain, out_nblk; + struct timespec start, now; + unsigned int elapsed; + unsigned int i; + void *cout; + + mlx5st_cmd_post(dev, cmd, dev->cmd_in_mbox, dev->cmd_out_mbox, in, + ilen, olen, 1); + + out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0; + out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + + /* Poll for completion — also process EQ events for PF page requests */ + clock_gettime(CLOCK_MONOTONIC, &start); + for (;;) { + if (!MLX5_GET_ONCE(cmd_queue_entry, cmd, ownership)) + break; + if (dev->have_eq) + mlx5st_process_events(dev); + sched_yield(); + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > MLX5_CMD_TIMEOUT_MS) + VFIO_FAIL("cmd timeout after %d ms", elapsed); + } + /* Ensure output data reads happen after ownership is seen clear */ + dma_rmb(); + + /* Verify output signatures when FW has checksums enabled */ + if (dev->cmd_sig_enabled) { + if (xor8_buf(cmd, 0, + sizeof(struct mlx5st_cmd_queue_entry)) != 0xff) + VFIO_FAIL("cmd output signature mismatch"); + for (i = 0; i < out_nblk; i++) { + if (mlx5st_cmd_verify_block_sig( + &dev->cmd_out_mbox[i].block)) + VFIO_FAIL("cmd output mailbox block %d signature mismatch", + i); + } + } + + /* Copy output: inline first */ + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + memcpy(out, cout, olen < MLX5_CMD_INLINE_SZ ? olen : MLX5_CMD_INLINE_SZ); + + /* Copy remaining from output mailbox chain */ + if (out_remain > 0) + mlx5st_cmd_copy_from_mbox((u8 *)out + MLX5_CMD_INLINE_SZ, + dev->cmd_out_mbox, out_remain); + + /* Check command status */ + if (MLX5_GET(enable_hca_out, out, status) != MLX5_CMD_STAT_OK) + VFIO_FAIL("%s: status=0x%x syndrome=0x%x", + mlx5st_cmd_name(MLX5_GET(enable_hca_in, in, opcode)), + MLX5_GET(enable_hca_out, out, status), + MLX5_GET(enable_hca_out, out, syndrome)); +} + +static struct mlx5st_cmd_queue_entry * +mlx5st_cmd_slot_init(struct mlx5st_device *dev, unsigned int slot, + struct mlx5st_mbox_entry *in_mbox, + struct mlx5st_mbox_entry *out_mbox) +{ + struct vfio_pci_device *device = dev->device; + struct mlx5st_cmd_queue_entry *cmd = + &dev->cmd_queue[(slot << dev->cmd_log_stride) / + sizeof(struct mlx5st_cmd_queue_entry)]; + u64 iova; + + MLX5_SET(cmd_queue_entry, cmd, type, + MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT); + iova = to_iova(device, in_mbox); + MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_63_32, + iova >> 32); + MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_31_9, iova >> 9); + iova = to_iova(device, out_mbox); + MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_63_32, + iova >> 32); + MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_31_9, + iova >> 9); + return cmd; +} + +static void mlx5st_cmd_init(struct mlx5st_device *dev) +{ + struct mlx5st_initial_seg __iomem *seg = dev->init_seg; + struct vfio_pci_device *device = dev->device; + u16 cmdif_rev; + u8 log_sz; + u64 iova; + + cmdif_rev = MLX5_GET_MMIO(initial_seg, seg, cmd_interface_rev); + VFIO_ASSERT_EQ(cmdif_rev, 5); + + /* Read command queue geometry from BAR */ + log_sz = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_size); + dev->cmd_log_stride = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_stride); + dev->pages_slot = (1 << log_sz) - 1; + + VFIO_ASSERT_LE((unsigned int)(1 << log_sz), 32u); + VFIO_ASSERT_GE((unsigned int)(1 << dev->cmd_log_stride), + (unsigned int)sizeof(struct mlx5st_cmd_queue_entry)); + VFIO_ASSERT_LE((unsigned int)((dev->pages_slot + 1) << + dev->cmd_log_stride), + (unsigned int)sizeof(dev->cmd_queue)); + + /* Set up slot 0 — regular commands */ + dev->cmd_lay = mlx5st_cmd_slot_init(dev, 0, dev->cmd_in_mbox, + dev->cmd_out_mbox); + + /* Set up pages slot — async MANAGE_PAGES */ + dev->pages_cmd_lay = mlx5st_cmd_slot_init(dev, dev->pages_slot, + dev->pages_in_mbox, + dev->pages_out_mbox); + + /* Write command queue page address to BAR0 */ + iova = to_iova(device, dev->cmd_queue); + MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_63_32, iova >> 32); + MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_31_12, iova >> 12); + + dev_dbg(device, + "Command interface initialized (cmdif_rev=5, log_sz=%u, log_stride=%u, pages_slot=%u)\n", + log_sz, dev->cmd_log_stride, dev->pages_slot); +} + +/* + * FW pages: bitmap allocator + MANAGE_PAGES + */ + +static void mlx5st_fw_pages_alloc(struct mlx5st_device *dev, + unsigned int npages, u64 *iovas) +{ + struct vfio_pci_device *device = dev->device; + unsigned int found = 0; + unsigned int w, b; + u64 word; + + for (w = 0; w < MAX_FW_PAGES / 64 && found < npages; w++) { + word = dev->fw_pages_bitmap[w]; + + for (b = 0; b < 64 && found < npages; b++) { + if (!(word & (1ULL << b))) { + unsigned int idx = w * 64 + b; + + dev->fw_pages_bitmap[w] |= (1ULL << b); + iovas[found++] = to_iova(device, + dev->fw_pages[idx]); + } + } + } + VFIO_ASSERT_EQ(found, npages); + dev->fw_pages_given += npages; +} + +static void mlx5st_fw_pages_free(struct mlx5st_device *dev, + unsigned int npages, const u64 *iovas) +{ + struct vfio_pci_device *device = dev->device; + unsigned int i, idx; + u64 off; + + for (i = 0; i < npages; i++) { + off = iovas[i] - to_iova(device, dev->fw_pages); + idx = off / MLX5_HW_PAGE_SIZE; + + VFIO_ASSERT_TRUE(idx < MAX_FW_PAGES); + dev->fw_pages_bitmap[idx / 64] &= ~(1ULL << (idx % 64)); + } + dev->fw_pages_given -= npages; +} + +static void *mlx5st_build_manage_pages_give(u16 func_id, unsigned int npages, + const u64 *iovas, + unsigned int *out_inlen) +{ + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in) + npages * 8; + unsigned int i; + void *in; + + in = calloc(1, inlen); + VFIO_ASSERT_NOT_NULL(in); + + MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_ALLOCATION_SUCCESS); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, npages); + + for (i = 0; i < npages; i++) + MLX5_ARRAY_SET64(manage_pages_in, in, pas, i, iovas[i]); + + *out_inlen = inlen; + return in; +} + +static void mlx5st_fw_pages_give_one(struct mlx5st_device *dev, u16 func_id, + unsigned int npages, u64 *iovas) +{ + u32 out[MLX5_ST_SZ_DW(manage_pages_out)] = {}; + unsigned int inlen; + void *in; + + in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen); + mlx5st_cmd_exec(dev, in, inlen, out, sizeof(out)); + free(in); +} + +static void mlx5st_fw_pages_give(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int remaining = npages; + u64 *iovas; + + if (!npages) + return; + + iovas = calloc(npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(iovas); + + mlx5st_fw_pages_alloc(dev, npages, iovas); + + /* Batch into chunks that fit in one mailbox */ + for (unsigned int off = 0; remaining > 0;) { + unsigned int batch = remaining < MAX_FW_PAGES_PER_CMD ? + remaining : + MAX_FW_PAGES_PER_CMD; + + mlx5st_fw_pages_give_one(dev, func_id, batch, iovas + off); + off += batch; + remaining -= batch; + } + + dev_dbg(dev->device, "MANAGE_PAGES GIVE: %d pages to func_id=%u\n", + npages, func_id); + free(iovas); +} + +static void mlx5st_fw_pages_satisfy(struct mlx5st_device *dev, int boot) +{ + u32 qo[MLX5_ST_SZ_DW(query_pages_out)] = {}; + u32 qi[MLX5_ST_SZ_DW(query_pages_in)] = {}; + u16 func_id; + int npages; + + MLX5_SET(query_pages_in, qi, opcode, MLX5_CMD_OP_QUERY_PAGES); + MLX5_SET(query_pages_in, qi, op_mod, boot ? 0x01 : 0x02); + mlx5st_cmd_exec(dev, qi, sizeof(qi), qo, sizeof(qo)); + + npages = MLX5_GET(query_pages_out, qo, num_pages); + func_id = MLX5_GET(query_pages_out, qo, function_id); + dev_dbg(dev->device, "QUERY_PAGES (%s): %d pages (func_id=%u)\n", + boot ? "boot" : "init", npages, func_id); + + if (npages > 0) { + dev->fw_func_id = func_id; + mlx5st_fw_pages_give(dev, func_id, npages); + } +} + +/* + * Async MANAGE_PAGES on the pages command slot. + * + * On PFs, firmware sends PAGE_REQUEST events via the EQ during command + * execution. We must respond with MANAGE_PAGES on a second command slot + * before the first (regular) command can complete. + */ + +static void mlx5st_pages_slot_post(struct mlx5st_device *dev, void *in, + unsigned int ilen, unsigned int olen) +{ + mlx5st_cmd_post(dev, dev->pages_cmd_lay, dev->pages_in_mbox, + dev->pages_out_mbox, in, ilen, olen, + 1 << dev->pages_slot); +} + +static void mlx5st_pages_slot_give(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int inlen; + u64 *iovas; + void *in; + + iovas = calloc(npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(iovas); + + mlx5st_fw_pages_alloc(dev, npages, iovas); + + in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen); + free(iovas); + + mlx5st_pages_slot_post(dev, in, inlen, + MLX5_ST_SZ_BYTES(manage_pages_out)); + dev->pages_slot_in_use = true; + dev->pages_slot_is_reclaim = false; + free(in); + + dev_dbg(dev->device, + "PAGE_REQUEST: %d pages given async to func_id=%u\n", + npages, func_id); +} + +static void mlx5st_pages_slot_reclaim(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in); + unsigned int outlen = + MLX5_ST_SZ_BYTES(manage_pages_out) + npages * 8; + void *in; + + in = calloc(1, inlen); + VFIO_ASSERT_NOT_NULL(in); + + MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, npages); + + mlx5st_pages_slot_post(dev, in, inlen, outlen); + dev->pages_slot_in_use = true; + dev->pages_slot_is_reclaim = true; + dev->pages_reclaim_npages = npages; + free(in); + + dev_dbg(dev->device, + "PAGE_REQUEST: reclaim %d pages async from func_id=%u\n", + npages, func_id); +} + +static void mlx5st_pages_slot_kick(struct mlx5st_device *dev) +{ + unsigned int batch; + + if (dev->pages_slot_in_use) + return; + + if (dev->pages_pending_give) { + batch = dev->pages_pending_give < MAX_FW_PAGES_PER_CMD ? + dev->pages_pending_give : + MAX_FW_PAGES_PER_CMD; + dev->pages_pending_give -= batch; + mlx5st_pages_slot_give(dev, dev->pages_pending_func_id, batch); + } else if (dev->pages_pending_reclaim) { + batch = dev->pages_pending_reclaim < MAX_FW_PAGES_PER_CMD ? + dev->pages_pending_reclaim : + MAX_FW_PAGES_PER_CMD; + dev->pages_pending_reclaim -= batch; + mlx5st_pages_slot_reclaim(dev, dev->pages_pending_func_id, + batch); + } +} + +static void mlx5st_fw_pages_give_async(struct mlx5st_device *dev, + u16 func_id, unsigned int npages) +{ + if (!npages) + return; + + dev->pages_pending_give += npages; + dev->pages_pending_func_id = func_id; + mlx5st_pages_slot_kick(dev); +} + +static void mlx5st_fw_pages_reclaim_async(struct mlx5st_device *dev, + u16 func_id, unsigned int npages) +{ + dev->pages_pending_reclaim += npages; + dev->pages_pending_func_id = func_id; + mlx5st_pages_slot_kick(dev); +} + +static void mlx5st_pages_slot_complete(struct mlx5st_device *dev) +{ + struct mlx5st_cmd_queue_entry *cmd = dev->pages_cmd_lay; + void *cout; + + dma_rmb(); + + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + if (MLX5_GET(enable_hca_out, cout, status) != MLX5_CMD_STAT_OK) + VFIO_FAIL("async MANAGE_PAGES failed: status=0x%x syndrome=0x%x", + MLX5_GET(enable_hca_out, cout, status), + MLX5_GET(enable_hca_out, cout, syndrome)); + + if (dev->pages_slot_is_reclaim) { + unsigned int outlen = MLX5_ST_SZ_BYTES(manage_pages_out) + + dev->pages_reclaim_npages * 8; + unsigned int num_claimed; + unsigned int i; + void *out; + u64 *iovas; + + out = calloc(1, outlen); + iovas = calloc(dev->pages_reclaim_npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(out); + VFIO_ASSERT_NOT_NULL(iovas); + + /* Copy inline output */ + memcpy(out, cout, MLX5_CMD_INLINE_SZ); + if (outlen > MLX5_CMD_INLINE_SZ) + mlx5st_cmd_copy_from_mbox( + (u8 *)out + MLX5_CMD_INLINE_SZ, + dev->pages_out_mbox, + outlen - MLX5_CMD_INLINE_SZ); + + num_claimed = + MLX5_GET(manage_pages_out, out, output_num_entries); + for (i = 0; i < num_claimed; i++) + iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas, + i); + + mlx5st_fw_pages_free(dev, num_claimed, iovas); + dev_dbg(dev->device, "PAGE_REQUEST: reclaimed %d pages\n", + num_claimed); + + free(iovas); + free(out); + } + + dev->pages_slot_in_use = false; + mlx5st_pages_slot_kick(dev); +} + +/* + * UAR alloc/dealloc + */ + +static void mlx5st_alloc_uar(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(alloc_uar_out)] = {}; + u32 in[MLX5_ST_SZ_DW(alloc_uar_in)] = {}; + + MLX5_SET(alloc_uar_in, in, opcode, MLX5_CMD_OP_ALLOC_UAR); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->uar_page = MLX5_GET(alloc_uar_out, out, uar); + dev->uar_base = (u8 __iomem*)dev->bar0 + dev->uar_page * MLX5_HW_PAGE_SIZE; + dev->uar_bf_offset = MLX5_BF_OFFSET; + + dev_dbg(dev->device, + "Allocated UAR page_id=%u, doorbell offset=0x%x\n", + dev->uar_page, + dev->uar_page * MLX5_HW_PAGE_SIZE + MLX5_BF_OFFSET); +} + +static void mlx5st_dealloc_uar(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(dealloc_uar_out)] = {}; + u32 in[MLX5_ST_SZ_DW(dealloc_uar_in)] = {}; + + MLX5_SET(dealloc_uar_in, in, opcode, MLX5_CMD_OP_DEALLOC_UAR); + MLX5_SET(dealloc_uar_in, in, uar, dev->uar_page); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * EQ infrastructure + */ + +static struct mlx5st_eqe *mlx5st_eq_get_eqe(struct mlx5st_device *dev, u32 cc) +{ + u32 ci = dev->eq_cons_index + cc; + struct mlx5st_eqe *eqe = &dev->eq_buf[ci % EQ_NENT]; + u8 owner = MLX5_GET_ONCE(eqe, eqe, owner); + u8 expected = !!(ci & EQ_NENT); + + if (owner != expected) + return NULL; + dma_rmb(); + return eqe; +} + +static void mlx5st_eq_update_ci(struct mlx5st_device *dev, u32 cc, bool arm) +{ + u32 val; + + dev->eq_cons_index += cc; + val = (dev->eq_cons_index & 0xffffff) | (dev->eqn << 24); + iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET + + (arm ? 0 : 8)); +} + +static void mlx5st_create_eq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {}; + struct mlx5_ifc_eqc_bits *eqc; + unsigned int i; + __be64 *pas; + + /* Initialize EQE owner bits */ + for (i = 0; i < EQ_NENT; i++) { + struct mlx5st_eqe *eqe = &dev->eq_buf[i]; + + MLX5_SET_ONCE(eqe, eqe, owner, 1); + } + + MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ); + + /* Subscribe to CMD completions and PAGE_REQUEST events */ + MLX5_ARRAY_SET64(create_eq_in, in, event_bitmask, 0, + (1ULL << MLX5_EVENT_TYPE_CMD) | + (1ULL << MLX5_EVENT_TYPE_PAGE_REQUEST)); + + eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry); + MLX5_SET(eqc, eqc, log_eq_size, LOG_EQ_SIZE); + MLX5_SET(eqc, eqc, uar_page, dev->uar_page); + pas = MLX5_ADDR_OF(create_eq_in, in, pas); + VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->eq_buf, pas), 0u); + MLX5_SET(eqc, eqc, log_page_size, 0); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->eqn = MLX5_GET(create_eq_out, out, eq_number); + dev->eq_cons_index = 0; + mlx5st_eq_update_ci(dev, 0, 0); + dev->have_eq = true; + + dev_dbg(device, "Created EQ: eqn=%u, %d entries (CMD+PAGE_REQUEST)\n", + dev->eqn, EQ_NENT); +} + +static void mlx5st_destroy_eq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {}; + + MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ); + MLX5_SET(destroy_eq_in, in, eq_number, dev->eqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * Drain all pending EQ events. Dispatches PAGE_REQUEST to the async pages + * slot and CMD completions to the pages slot completion handler. + */ +static void mlx5st_process_events(struct mlx5st_device *dev) +{ + struct mlx5st_eqe *eqe; + u32 cc = 0; + + while ((eqe = mlx5st_eq_get_eqe(dev, cc))) { + u8 type = MLX5_GET(eqe, eqe, event_type); + + switch (type) { + case MLX5_EVENT_TYPE_PAGE_REQUEST: { + void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data); + u16 func_id = MLX5_GET(pages_req_event, evdata, + function_id); + s32 npages = (s32)MLX5_GET(pages_req_event, evdata, + num_pages); + + /* + * The selftest doesn't use more than one func_id so a + * simple counter approach is possible. + */ + if (dev->pages_func_id_seen) + VFIO_ASSERT_EQ(func_id, + dev->pages_pending_func_id); + dev->pages_func_id_seen = true; + + if (npages > 0) + mlx5st_fw_pages_give_async(dev, func_id, + npages); + else if (npages < 0) + mlx5st_fw_pages_reclaim_async(dev, func_id, + -npages); + break; + } + case MLX5_EVENT_TYPE_CMD: { + void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data); + u32 vector = MLX5_GET(cmd_inter_comp_event, evdata, + command_completion_vector); + + if (vector & (1U << dev->pages_slot)) + mlx5st_pages_slot_complete(dev); + break; + } + default: + break; + } + cc++; + } + + if (cc) + mlx5st_eq_update_ci(dev, cc, 0); +} + +/* + * HCA init / teardown + */ + +#define FW_INIT_TIMEOUT_MS 120000 +#define FW_INIT_WAIT_MS 200 + +static void mlx5st_wait_fw_init(struct mlx5st_device *dev) +{ + struct timespec start, now; + unsigned int elapsed; + + clock_gettime(CLOCK_MONOTONIC, &start); + while (MLX5_GET_MMIO(initial_seg, dev->init_seg, initializing)) { + usleep(FW_INIT_WAIT_MS * 1000); + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > FW_INIT_TIMEOUT_MS) + VFIO_FAIL("FW init timeout after %d ms", elapsed); + } +} + +static void mlx5st_set_issi(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(set_issi_out)] = {}; + u32 in[MLX5_ST_SZ_DW(set_issi_in)] = {}; + + MLX5_SET(set_issi_in, in, opcode, MLX5_CMD_OP_SET_ISSI); + MLX5_SET(set_issi_in, in, current_issi, 1); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "SET_ISSI: OK (issi=1)\n"); +} + +static void mlx5st_set_hca_caps(struct mlx5st_device *dev) +{ + u32 qout[MLX5_ST_SZ_DW(query_hca_cap_out)] = {}; + u32 qin[MLX5_ST_SZ_DW(query_hca_cap_in)] = {}; + u32 sout[MLX5_ST_SZ_DW(set_hca_cap_out)] = {}; + u32 sin[MLX5_ST_SZ_DW(set_hca_cap_in)] = {}; + struct mlx5_ifc_cmd_hca_cap_bits *set_hca_cap; + u32 max_checksum; + + /* Query max caps to learn cmdif_checksum support */ + MLX5_SET(query_hca_cap_in, qin, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, qin, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1)); + mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout)); + + max_checksum = MLX5_GET( + cmd_hca_cap, + MLX5_ADDR_OF(query_hca_cap_out, qout, capability), + cmdif_checksum); + + /* Query current caps as base for SET */ + memset(qout, 0, sizeof(qout)); + MLX5_SET(query_hca_cap_in, qin, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) | + HCA_CAP_OPMOD_GET_CUR); + mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout)); + + set_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, sin, capability); + memcpy(set_hca_cap, + MLX5_ADDR_OF(query_hca_cap_out, qout, capability), + MLX5_ST_SZ_BYTES(cmd_hca_cap)); + + MLX5_SET(cmd_hca_cap, set_hca_cap, cmdif_checksum, max_checksum); + MLX5_SET(cmd_hca_cap, set_hca_cap, log_uar_page_sz, 0); + + MLX5_SET(set_hca_cap_in, sin, opcode, MLX5_CMD_OP_SET_HCA_CAP); + MLX5_SET(set_hca_cap_in, sin, op_mod, + MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1); + + mlx5st_cmd_exec(dev, sin, sizeof(sin), sout, sizeof(sout)); + + dev->cmd_sig_enabled = max_checksum == 0x3; + dev_dbg(dev->device, "SET_HCA_CAP: OK (cmdif_checksum=%u)\n", + max_checksum); +} + +static void mlx5st_hca_init(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(enable_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(enable_hca_in)] = {}; + + mlx5st_wait_fw_init(dev); + dev_dbg(dev->device, "Firmware ready\n"); + + MLX5_SET(enable_hca_in, in, opcode, MLX5_CMD_OP_ENABLE_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "ENABLE_HCA: OK\n"); + + mlx5st_set_issi(dev); + mlx5st_fw_pages_satisfy(dev, 1); + + mlx5st_set_hca_caps(dev); + mlx5st_fw_pages_satisfy(dev, 0); + + memset(in, 0, sizeof(in)); + memset(out, 0, sizeof(out)); + MLX5_SET(init_hca_in, in, opcode, MLX5_CMD_OP_INIT_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "INIT_HCA: OK\n"); + + /* + * Create EQ immediately after INIT_HCA so PAGE_REQUEST events + * are captured during all subsequent commands. + */ + mlx5st_alloc_uar(dev); + mlx5st_create_eq(dev); +} + +static void mlx5st_disable_hca(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(disable_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(disable_hca_in)] = {}; + + MLX5_SET(disable_hca_in, in, opcode, MLX5_CMD_OP_DISABLE_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +static void mlx5st_fw_pages_reclaim(struct mlx5st_device *dev, u16 func_id) +{ + unsigned int npages = dev->fw_pages_given; + unsigned int total_claimed = 0; + + while (npages > 0) { + unsigned int batch = npages < MAX_FW_PAGES_PER_CMD ? + npages : + MAX_FW_PAGES_PER_CMD; + unsigned int outlen = + MLX5_ST_SZ_BYTES(manage_pages_out) + batch * 8; + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in); + unsigned int num_claimed; + unsigned int i; + void *in, *out; + u64 *iovas; + + in = calloc(1, inlen); + out = calloc(1, outlen); + iovas = calloc(batch, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(in); + VFIO_ASSERT_NOT_NULL(out); + VFIO_ASSERT_NOT_NULL(iovas); + + MLX5_SET(manage_pages_in, in, opcode, + MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, batch); + + mlx5st_cmd_exec(dev, in, inlen, out, outlen); + + num_claimed = + MLX5_GET(manage_pages_out, out, output_num_entries); + for (i = 0; i < num_claimed; i++) + iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas, + i); + + mlx5st_fw_pages_free(dev, num_claimed, iovas); + total_claimed += num_claimed; + npages -= num_claimed; + + free(iovas); + free(in); + free(out); + + if (!num_claimed && !dev->fw_pages_given) + break; + if (!num_claimed) + VFIO_FAIL("MANAGE_PAGES RECLAIM: FW returned 0 but %d pages still given", + dev->fw_pages_given); + } + + dev_dbg(dev->device, + "MANAGE_PAGES RECLAIM: %d pages (%d still given)\n", + total_claimed, dev->fw_pages_given); +} + +static void mlx5st_hca_teardown(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(teardown_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(teardown_hca_in)] = {}; + + /* Drain async pages slot, then stop EQ processing */ + while (dev->pages_slot_in_use) { + if (!MLX5_GET_ONCE(cmd_queue_entry, dev->pages_cmd_lay, + ownership)) + mlx5st_pages_slot_complete(dev); + else + sched_yield(); + } + dev->have_eq = false; + + if (dev->eqn) { + mlx5st_destroy_eq(dev); + dev->eqn = 0; + } + if (dev->uar_page) { + mlx5st_dealloc_uar(dev); + dev->uar_page = 0; + } + + dev_dbg(dev->device, " hca_teardown: TEARDOWN_HCA\n"); + MLX5_SET(teardown_hca_in, in, opcode, MLX5_CMD_OP_TEARDOWN_HCA); + MLX5_SET(teardown_hca_in, in, profile, + MLX5_TEARDOWN_HCA_IN_PROFILE_GRACEFUL_CLOSE); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + if (dev->fw_pages_given > 0) { + dev_dbg(dev->device, " hca_teardown: reclaim %d pages\n", + dev->fw_pages_given); + mlx5st_fw_pages_reclaim(dev, dev->fw_func_id); + } + + dev_dbg(dev->device, " hca_teardown: DISABLE_HCA\n"); + mlx5st_disable_hca(dev); +} + +/* + * Query capabilities + */ +static void mlx5st_query_fl_caps(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {}; + u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {}; + bool fl_roce_en, fl_roce_dis; + + /* Query RoCE capabilities */ + MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, in, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_ROCE << 1) | HCA_CAP_OPMOD_GET_CUR); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + fl_roce_en = MLX5_GET(query_hca_cap_out, out, + capability.roce_cap.fl_rc_qp_when_roce_enabled); + fl_roce_dis = MLX5_GET(query_hca_cap_out, out, + capability.roce_cap.fl_rc_qp_when_roce_disabled); + + /* Also check general caps */ + memset(in, 0, sizeof(in)); + memset(out, 0, sizeof(out)); + MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, in, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) | + HCA_CAP_OPMOD_GET_CUR); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + fl_roce_dis |= + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.fl_rc_qp_when_roce_disabled); + + dev->fl_supported = fl_roce_en || fl_roce_dis; + dev->log_max_msg = MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.log_max_msg); + dev_dbg(dev->device, + "HCA capabilities: fl_roce_enabled=%d fl_roce_disabled=%d log_max_msg=%u\n", + fl_roce_en, fl_roce_dis, dev->log_max_msg); + + VFIO_ASSERT_TRUE(dev->fl_supported, + "Force-loopback not supported on this device"); +} + +/* + * Resource allocation + */ + +static void mlx5st_alloc_pd(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(alloc_pd_out)] = {}; + u32 in[MLX5_ST_SZ_DW(alloc_pd_in)] = {}; + + MLX5_SET(alloc_pd_in, in, opcode, MLX5_CMD_OP_ALLOC_PD); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->pdn = MLX5_GET(alloc_pd_out, out, pd); + dev_dbg(dev->device, "Allocated PD pdn=%u\n", dev->pdn); +} + +static void mlx5st_dealloc_pd(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(dealloc_pd_out)] = {}; + u32 in[MLX5_ST_SZ_DW(dealloc_pd_in)] = {}; + + MLX5_SET(dealloc_pd_in, in, opcode, MLX5_CMD_OP_DEALLOC_PD); + MLX5_SET(dealloc_pd_in, in, pd, dev->pdn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +static void mlx5st_create_mkey(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(create_mkey_out)] = {}; + u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {}; + struct mlx5_ifc_mkc_bits *mkc; + + MLX5_SET(create_mkey_in, in, opcode, MLX5_CMD_OP_CREATE_MKEY); + + mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); + MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA); + MLX5_SET(mkc, mkc, length64, 1); + MLX5_SET(mkc, mkc, pd, dev->pdn); + MLX5_SET(mkc, mkc, qpn, 0xffffff); + MLX5_SET(mkc, mkc, lr, 1); + MLX5_SET(mkc, mkc, lw, 1); + MLX5_SET(mkc, mkc, rw, 1); + MLX5_SET(mkc, mkc, rr, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->mkey_index = MLX5_GET(create_mkey_out, out, mkey_index); + dev->global_lkey = mlx5st_idx_to_mkey(dev->mkey_index); + dev->global_rkey = dev->global_lkey; + + dev_dbg(dev->device, "Created global PA-mode MKEY: lkey=0x%x\n", + dev->global_lkey); +} + +static void mlx5st_destroy_mkey(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_mkey_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_mkey_in)] = {}; + + MLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY); + MLX5_SET(destroy_mkey_in, in, mkey_index, dev->mkey_index); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * Driver ops callbacks + */ + +static void mlx5st_init(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + iova_t iova_align = + device->driver.region.iova % __alignof__(struct mlx5st_device); + + VFIO_ASSERT_GE(device->driver.region.size, sizeof(*dev)); + VFIO_ASSERT_EQ(iova_align, 0); + memset(dev, 0, sizeof(*dev)); + + dev->device = device; + dev->bar0 = device->bars[0].vaddr; + dev->init_seg = dev->bar0; + + vfio_pci_config_writew(device, PCI_COMMAND, + vfio_pci_config_readw(device, PCI_COMMAND) | + PCI_COMMAND_MASTER); + + mlx5st_wait_fw_init(dev); + + mlx5st_cmd_init(dev); + mlx5st_hca_init(dev); + mlx5st_query_fl_caps(dev); + mlx5st_alloc_pd(dev); + mlx5st_create_mkey(dev); + + dev_dbg(device, "mlx5 driver initialized\n"); +} + +static void mlx5st_remove(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + + dev_dbg(device, "teardown: destroy_mkey\n"); + if (dev->mkey_index) { + mlx5st_destroy_mkey(dev); + dev->mkey_index = 0; + } + + dev_dbg(device, "teardown: dealloc_pd\n"); + if (dev->pdn) { + mlx5st_dealloc_pd(dev); + dev->pdn = 0; + } + + dev_dbg(device, "teardown: hca_teardown\n"); + mlx5st_hca_teardown(dev); + + vfio_pci_config_writew(device, PCI_COMMAND, + vfio_pci_config_readw(device, PCI_COMMAND) & + ~(u16)PCI_COMMAND_MASTER); + + dev_dbg(device, "Teardown complete\n"); +} + +struct vfio_pci_driver_ops mlx5st_ops = { + .name = "mlx5", + .region_size = sizeof(struct mlx5st_device), + .probe = mlx5st_probe, + .init = mlx5st_init, + .remove = mlx5st_remove, + .memcpy_start = NULL, + .memcpy_wait = NULL, + .send_msi = NULL, +}; diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h new file mode 100644 index 000000000000..a2506ec8a195 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h @@ -0,0 +1,108 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * mlx5 VFIO selftest driver - HW definitions + * + * Typed wrappers, constants, and helpers for programming mlx5 hardware + * via the VFIO selftest framework. Most HW constants and all MLX5_SET/GET + * macros come from the kernel headers (mlx5_ifc.h, mlx5_ifc_macros.h). + */ +#ifndef SELFTESTS_VFIO_MLX5_HW_H +#define SELFTESTS_VFIO_MLX5_HW_H + +#include +#include +#include + +#include "mlx5_ifc.h" +#include "mlx5_ifc_macros.h" + +/* + * Typed HW object wrappers for driver region arrays. + * + * The IFC _bits structs have sizeof == num_bits (not bytes), so they cannot + * be used as array elements. These wrappers provide byte-sized types. + */ +#define MLX5ST_MAKE_DATA32(name) \ + struct mlx5st_##name { \ + u32 data[MLX5_ST_SZ_DW(name)]; \ + } +#define MLX5ST_MAKE_DATA64(name) \ + struct mlx5st_##name { \ + u64 data[MLX5_ST_SZ_QW(name)]; \ + } + +MLX5ST_MAKE_DATA32(initial_seg); +MLX5ST_MAKE_DATA64(cmd_queue_entry); +MLX5ST_MAKE_DATA64(cmd_if_box); +MLX5ST_MAKE_DATA64(wqe_ctrl_seg); +MLX5ST_MAKE_DATA64(wqe_raddr_seg); +MLX5ST_MAKE_DATA64(wqe_data_seg); +MLX5ST_MAKE_DATA64(cqe64) __aligned(64); +MLX5ST_MAKE_DATA64(eqe); + +/* + * Mailbox blocks: 512 data + 64 header = 576 bytes, but the + * next_pointer field stores bits [31:10], requiring 1024-byte alignment. + */ +#define CMD_MBOX_SIZE (2 * MLX5_HW_PAGE_SIZE) +#define CMD_MBOX_STRIDE 1024 +#define CMD_MBOX_NENT (CMD_MBOX_SIZE / CMD_MBOX_STRIDE) +/* Stride-aligned mailbox entry — block + padding to 1024 bytes */ +struct mlx5st_mbox_entry { + struct mlx5st_cmd_if_box block; +} __aligned(CMD_MBOX_STRIDE); + +#define MLX5_CMD_INLINE_SZ \ + MLX5_FLD_SZ_BYTES(cmd_queue_entry, command_input_inline_data) + +/* Command interface mailbox block (512 data + 64 header) */ +#define MLX5_CMD_DATA_BLOCK_SIZE MLX5_FLD_SZ_BYTES(cmd_if_box, mailbox_data) + +/* RDMA Write WQE — one basic block: ctrl + raddr + data + padding */ +struct mlx5st_send_wqe { + struct mlx5st_wqe_ctrl_seg ctrl; + struct mlx5st_wqe_raddr_seg raddr; + struct mlx5st_wqe_data_seg data; +} __aligned(64); +static_assert(sizeof(struct mlx5st_send_wqe) == 64, + "send WQE segments must fit in one BB"); + +/* DS = number of 16-byte segments in the WQE (ctrl + raddr + data) */ +#define MLX5_RDMA_WRITE_DS 3 + +/* Doorbell record — two __be32 in a 64-byte aligned pair */ +struct mlx5st_dbrec { + __be32 recv_counter; + __be32 send_counter; +} __aligned(64); + +/* UAR BlueFlame buffer offsets within a UAR page */ +#define MLX5_BF_OFFSET 0x800 +#define MLX5_BF_SIZE 0x100 + +/* EQ doorbell offset within UAR page */ +#define MLX5_EQ_DOORBELL_OFFSET 0x40 + +#define MLX5_HW_PAGE_SIZE 4096 + +/* + * Test parameters + */ +#define SQ_WQE_CNT 16 +#define LOG_SQ_SIZE 4 +#define CQ_CQE_CNT 16 +#define LOG_CQ_SIZE 4 +#define EQ_NENT 64 +#define LOG_EQ_SIZE 6 + +#define MAX_FW_PAGES 8192 +#define MAX_FW_PAGES_PER_CMD 512 + +#define MLX5_CMD_TIMEOUT_MS 5000 + +static inline u32 mlx5st_idx_to_mkey(u32 mkey_idx) +{ + return mkey_idx << 8; +} + +#endif /* SELFTESTS_VFIO_MLX5_HW_H */ diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h new file mode 120000 index 000000000000..7dcbb79e1af0 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc.h \ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h new file mode 120000 index 000000000000..865d99e2aeec --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc_fpga.h \ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h new file mode 120000 index 000000000000..97408c247f06 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc_macros.h \ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk index bcfa74ae040e..eff95828719e 100644 --- a/tools/testing/selftests/vfio/lib/libvfio.mk +++ b/tools/testing/selftests/vfio/lib/libvfio.mk @@ -15,6 +15,7 @@ LIBVFIO_C += drivers/ioat/ioat.c LIBVFIO_C += drivers/dsa/dsa.c endif +LIBVFIO_C += drivers/mlx5/mlx5.c LIBVFIO_C += drivers/nv_falcon/nv_falcon.c LIBVFIO_C += drivers/igb/igb.c diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c index 64135ac43877..b31abfe2c7ff 100644 --- a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c +++ b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c @@ -6,6 +6,7 @@ extern struct vfio_pci_driver_ops dsa_ops; extern struct vfio_pci_driver_ops ioat_ops; #endif +extern struct vfio_pci_driver_ops mlx5st_ops; extern struct vfio_pci_driver_ops nv_falcon_ops; extern struct vfio_pci_driver_ops igb_ops; @@ -14,6 +15,7 @@ static struct vfio_pci_driver_ops *driver_ops[] = { &dsa_ops, &ioat_ops, #endif + &mlx5st_ops, &nv_falcon_ops, &igb_ops, }; -- cgit v1.2.3 From 4fdbdaf53fc8a47ef7bb3d97c0ab7afb0cf75ae2 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:49 -0300 Subject: vfio: selftests: Add mlx5 driver - data path and memcpy ops Complete the mlx5 driver by adding CQ/QP creation, QP state transitions, WQE posting, CQ polling, and the memcpy_start/memcpy_wait callbacks. After this patch the driver is functional for DMA tests. The data path implements RDMA Write self-loopback via an RC QP with force-loopback. WQEs are posted to a 16-entry send queue with an NC doorbell, and completions are polled from a 16-entry CQ. Assisted-by: Claude:claude-opus-4.6 Acked-by: David Matlack Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/9-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- .../testing/selftests/vfio/lib/drivers/mlx5/mlx5.c | 359 ++++++++++++++++++++- 1 file changed, 357 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c index 5a87d070dc42..fff3babef5e7 100644 --- a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c @@ -1343,6 +1343,354 @@ static void mlx5st_destroy_mkey(struct mlx5st_device *dev) mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } +/* + * CQ create/destroy + */ + +static void mlx5st_create_cq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_cq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_cq_out)] = {}; + struct mlx5_ifc_cqc_bits *cqc; + unsigned int i; + __be64 *pas; + + /* Initialize CQEs before CREATE_CQ: opcode=0xF, owner=1 */ + for (i = 0; i < CQ_CQE_CNT; i++) { + struct mlx5st_cqe64 *cqe = &dev->cq_buf[i]; + + MLX5_SET(cqe64, cqe, opcode, 0xF); + MLX5_SET_ONCE(cqe64, cqe, owner, 1); + } + + MLX5_SET(create_cq_in, in, opcode, MLX5_CMD_OP_CREATE_CQ); + + cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context); + MLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE); + MLX5_SET(cqc, cqc, uar_page, dev->uar_page); + MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->eqn); + MLX5_SET(cqc, cqc, cqe_sz, 0); + pas = MLX5_ADDR_OF(create_cq_in, in, pas); + MLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev->cq_buf, pas)); + MLX5_SET(cqc, cqc, log_page_size, 0); + MLX5_SET64(cqc, cqc, dbr_addr, to_iova(device, &dev->cq_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->cqn = MLX5_GET(create_cq_out, out, cqn); + dev->cq_ci = 0; + dev_dbg(device, "Created CQ: cqn=%u, %d entries\n", dev->cqn, + CQ_CQE_CNT); +} + +static void mlx5st_destroy_cq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_cq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {}; + + MLX5_SET(destroy_cq_in, in, opcode, MLX5_CMD_OP_DESTROY_CQ); + MLX5_SET(destroy_cq_in, in, cqn, dev->cqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * QP create/destroy + */ + +static void mlx5st_create_qp(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_qp_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_qp_out)] = {}; + struct mlx5_ifc_qpc_bits *qpc; + __be64 *pas; + + MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP); + + qpc = MLX5_ADDR_OF(create_qp_in, in, qpc); + MLX5_SET(qpc, qpc, st, MLX5_QPC_ST_RC); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, pd, dev->pdn); + MLX5_SET(qpc, qpc, uar_page, dev->uar_page); + MLX5_SET(qpc, qpc, cqn_snd, dev->cqn); + MLX5_SET(qpc, qpc, cqn_rcv, dev->cqn); + MLX5_SET(qpc, qpc, log_sq_size, LOG_SQ_SIZE); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, rq_type, 0x3); + MLX5_SET(qpc, qpc, ts_format, 1); + pas = MLX5_ADDR_OF(create_qp_in, in, pas); + MLX5_SET(qpc, qpc, page_offset, + mlx5st_fill_pas(device, dev->sq_buf, pas)); + MLX5_SET(qpc, qpc, log_page_size, 0); + MLX5_SET64(qpc, qpc, dbr_addr, to_iova(device, &dev->qp_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->qpn = MLX5_GET(create_qp_out, out, qpn); + dev->sq_pi = 0; + dev_dbg(device, "Created QP: qpn=%u, RC, sq=%d wqes\n", dev->qpn, + SQ_WQE_CNT); +} + +static void mlx5st_destroy_qp(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_qp_in)] = {}; + + MLX5_SET(destroy_qp_in, in, opcode, MLX5_CMD_OP_DESTROY_QP); + MLX5_SET(destroy_qp_in, in, qpn, dev->qpn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * QP state transitions + */ + +static void mlx5st_qp_rst2init(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rst2init_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc); + + MLX5_SET(rst2init_qp_in, in, opcode, MLX5_CMD_OP_RST2INIT_QP); + MLX5_SET(rst2init_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, rre, 1); + MLX5_SET(qpc, qpc, rwe, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RST->INIT\n"); +} + +static void mlx5st_qp_init2rtr(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(init2rtr_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc); + + MLX5_SET(init2rtr_qp_in, in, opcode, MLX5_CMD_OP_INIT2RTR_QP); + MLX5_SET(init2rtr_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, mtu, 3); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, remote_qpn, dev->qpn); + MLX5_SET(qpc, qpc, min_rnr_nak, 12); + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, primary_address_path.fl, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP INIT->RTR (fl=1)\n"); +} + +static void mlx5st_qp_rtr2rts(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rtr2rts_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc); + + MLX5_SET(rtr2rts_qp_in, in, opcode, MLX5_CMD_OP_RTR2RTS_QP); + MLX5_SET(rtr2rts_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, log_ack_req_freq, 0); + MLX5_SET(qpc, qpc, retry_count, 7); + MLX5_SET(qpc, qpc, rnr_retry, 7); + MLX5_SET(qpc, qpc, primary_address_path.ack_timeout, 14); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RTR->RTS\n"); +} + +/* + * Post RDMA Write WQE + */ +static void mlx5st_post_rdma_write(struct mlx5st_device *dev, u64 src_addr, + u32 src_lkey, u64 dst_addr, u32 dst_rkey, + u32 length, bool signaled) +{ + struct mlx5st_send_wqe *wqe; + unsigned int idx; + + idx = dev->sq_pi % SQ_WQE_CNT; + wqe = &dev->sq_buf[idx]; + + memset(wqe, 0, sizeof(*wqe)); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, opcode, MLX5_OPCODE_RDMA_WRITE); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, wqe_index, dev->sq_pi); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, qp_or_sq, dev->qpn); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ds, MLX5_RDMA_WRITE_DS); + if (signaled) + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ce, MLX5_WQE_CE_CQE_ALWAYS); + + MLX5_SET64(wqe_raddr_seg, &wqe->raddr, raddr, dst_addr); + MLX5_SET(wqe_raddr_seg, &wqe->raddr, rkey, dst_rkey); + + MLX5_SET(wqe_data_seg, &wqe->data, byte_count, length); + MLX5_SET(wqe_data_seg, &wqe->data, lkey, src_lkey); + MLX5_SET64(wqe_data_seg, &wqe->data, addr, src_addr); + + dev->sq_pi++; + + /* Ensure WQE is visible to device before doorbell record */ + dma_wmb(); + + WRITE_ONCE(dev->qp_dbrec.send_counter, + cpu_to_be32(dev->sq_pi & 0xffff)); + + /* + * Ring doorbell: write first 8 bytes of ctrl to UAR BF register, + * iowrite has an internal dma_wmb() so the doorbell record will be + * visible. + */ + iowrite64be(be64_to_cpu(*(__be64 *)wqe), + (u8 __iomem *)dev->uar_base + dev->uar_bf_offset); + dev->uar_bf_offset ^= MLX5_BF_SIZE; +} + +/* + * Poll CQ + */ +static int mlx5st_poll_cq_batch(struct mlx5st_device *dev, + unsigned int max_cqe) +{ + unsigned int polled = 0; + + while (polled < max_cqe) { + unsigned int idx = dev->cq_ci % CQ_CQE_CNT; + struct mlx5st_cqe64 *cqe = &dev->cq_buf[idx]; + u8 owner, opcode; + + owner = MLX5_GET_ONCE(cqe64, cqe, owner); + if (owner != ((dev->cq_ci >> LOG_CQ_SIZE) & 1)) + break; + + dma_rmb(); + + opcode = MLX5_GET(cqe64, cqe, opcode); + + dev->cq_ci++; + WRITE_ONCE(dev->cq_dbrec.recv_counter, + cpu_to_be32(dev->cq_ci & 0xffffff)); + + if (opcode == MLX5_CQE_REQ) { + dev->sq_ci = + (u16)(MLX5_GET(cqe64, cqe, wqe_counter) + 1); + polled++; + continue; + } + if (opcode == MLX5_CQE_REQ_ERR || + opcode == MLX5_CQE_RESP_ERR) { + dev_dbg(dev->device, + "CQE error: opcode=0x%x syndrome=0x%x vendor=0x%x\n", + opcode, + MLX5_GET(cqe64, cqe, error_syndrome.syndrome), + MLX5_GET(cqe64, cqe, + error_syndrome.vendor_error_syndrome)); + return -1; + } + dev_err(dev->device, "CQE unexpected opcode=0x%x\n", opcode); + return -1; + } + + return polled; +} + +static int mlx5st_poll_cq(struct mlx5st_device *dev, unsigned int timeout_ms) +{ + struct timespec start, now; + unsigned int elapsed; + int ret; + + clock_gettime(CLOCK_MONOTONIC, &start); + for (;;) { + ret = mlx5st_poll_cq_batch(dev, 1); + if (ret < 0) + return -1; + if (ret > 0) + return 0; + + if (dev->have_eq) + mlx5st_process_events(dev); + + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > timeout_ms) { + dev_err(dev->device, "CQ poll timeout after %u ms\n", + timeout_ms); + return -1; + } + } +} + +/* + * Data path setup/teardown helpers + */ + +static void mlx5st_setup_datapath(struct mlx5st_device *dev) +{ + mlx5st_create_cq(dev); + mlx5st_create_qp(dev); + mlx5st_qp_rst2init(dev); + mlx5st_qp_init2rtr(dev); + mlx5st_qp_rtr2rts(dev); +} + +static void mlx5st_teardown_datapath(struct mlx5st_device *dev) +{ + if (dev->qpn) { + mlx5st_destroy_qp(dev); + dev->qpn = 0; + } + if (dev->cqn) { + mlx5st_destroy_cq(dev); + dev->cqn = 0; + } + dev->sq_pi = 0; + dev->sq_ci = 0; + memset(&dev->qp_dbrec, 0, sizeof(dev->qp_dbrec)); + memset(&dev->cq_dbrec, 0, sizeof(dev->cq_dbrec)); +} + +/* + * memcpy callbacks + */ + +#define MLX5ST_MEMCPY_TIMEOUT_MS 60000 + +static void mlx5st_memcpy_start(struct vfio_pci_device *device, + iova_t src, iova_t dst, u64 size, u64 count) +{ + struct mlx5st_device *dev = to_mlx5st(device); + u64 i; + + for (i = 0; i < count; i++) { + bool signaled = (i == count - 1); + + mlx5st_post_rdma_write(dev, src, dev->global_lkey, dst, + dev->global_rkey, size, signaled); + } +} + +static int mlx5st_memcpy_wait(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + int ret; + + ret = mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS); + if (ret) { + /* + * CQE error puts the QP in error state. Rebuild the data path + * so subsequent operations can succeed. + */ + mlx5st_teardown_datapath(dev); + mlx5st_setup_datapath(dev); + } + return ret; +} + /* * Driver ops callbacks */ @@ -1373,6 +1721,11 @@ static void mlx5st_init(struct vfio_pci_device *device) mlx5st_alloc_pd(dev); mlx5st_create_mkey(dev); + mlx5st_setup_datapath(dev); + + device->driver.max_memcpy_size = 1ULL << dev->log_max_msg; + device->driver.max_memcpy_count = SQ_WQE_CNT - 1; + dev_dbg(device, "mlx5 driver initialized\n"); } @@ -1380,6 +1733,8 @@ static void mlx5st_remove(struct vfio_pci_device *device) { struct mlx5st_device *dev = to_mlx5st(device); + mlx5st_teardown_datapath(dev); + dev_dbg(device, "teardown: destroy_mkey\n"); if (dev->mkey_index) { mlx5st_destroy_mkey(dev); @@ -1408,7 +1763,7 @@ struct vfio_pci_driver_ops mlx5st_ops = { .probe = mlx5st_probe, .init = mlx5st_init, .remove = mlx5st_remove, - .memcpy_start = NULL, - .memcpy_wait = NULL, + .memcpy_start = mlx5st_memcpy_start, + .memcpy_wait = mlx5st_memcpy_wait, .send_msi = NULL, }; -- cgit v1.2.3 From eeb2cc76b0b4b7cbf06ff383f755a93d7747b302 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Mon, 21 Sep 2026 19:35:50 -0300 Subject: vfio: selftests: mlx5 driver - add send_msi support Wire an MSI-X vector to a dedicated EQ so the mlx5 driver supports send_msi(). Each EQ can be linked to an MSI-X vector, and the CQ can be set up to deliver an event to the EQ. Thus, when everything is armed, an RDMA WRITE posted to the QP generates a CQE, which generates an EQE, which generates an MSI-X. To keep things simple this just re-uses all the existing QPs and CQs, so they generate single MSIs during memcpy. send_msi() drains any accumulated MSI EQ events from prior memcpy completions, posts a small signaled RDMA Write, then polls the CQ to consume the resulting CQE (avoiding stale completions on subsequent test cycles). Assisted-by: Claude:claude-opus-4.6 Acked-by: David Matlack Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/10-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com Signed-off-by: Alex Williamson --- .../testing/selftests/vfio/lib/drivers/mlx5/mlx5.c | 165 ++++++++++++++++++++- .../selftests/vfio/lib/drivers/mlx5/mlx5_hw.h | 6 + 2 files changed, 168 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c index fff3babef5e7..a65b52e183bc 100644 --- a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c @@ -57,17 +57,23 @@ struct mlx5st_device { /* CQ */ u32 cqn; u32 cq_ci; + u32 cq_arm_sn; /* UAR */ u32 uar_page; void __iomem *uar_base; unsigned int uar_bf_offset; - /* EQ */ + /* EQ (cmd/pages events — polled, not interrupt-driven) */ u32 eqn; u32 eq_cons_index; bool have_eq; + /* MSI EQ (CQ completion events — fires MSI-X) */ + u32 msi_eqn; + u32 msi_eq_cons_index; + bool have_msi_eq; + /* Async pages slot state */ bool pages_slot_in_use; bool pages_slot_is_reclaim; @@ -91,6 +97,10 @@ struct mlx5st_device { bool fl_supported; u8 log_max_msg; + /* Buffers used by send_msi() to trigger an interrupt */ + u64 send_msi_src; + u64 send_msi_dst; + /* * HW-visible DMA buffers below — device reads/writes via DMA. */ @@ -113,6 +123,9 @@ struct mlx5st_device { /* EQ does not support page_offset */ struct mlx5st_eqe eq_buf[EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE); + /* MSI EQ buffer — CQ completions generate EQEs here -> MSI-X */ + struct mlx5st_eqe msi_eq_buf[MSI_EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE); + u8 fw_pages[MAX_FW_PAGES][MLX5_HW_PAGE_SIZE] __aligned(MLX5_HW_PAGE_SIZE); }; @@ -135,6 +148,9 @@ static_assert(offsetof(struct mlx5st_device, qp_dbrec) % 64 == 0, static_assert(offsetof(struct mlx5st_device, eq_buf) % MLX5_HW_PAGE_SIZE == 0, "eq_buf must be page-aligned"); +static_assert(offsetof(struct mlx5st_device, msi_eq_buf) % + MLX5_HW_PAGE_SIZE == 0, + "msi_eq_buf must be page-aligned"); static_assert(offsetof(struct mlx5st_device, fw_pages) % MLX5_HW_PAGE_SIZE == 0, "fw_pages must be page-aligned"); @@ -1013,6 +1029,85 @@ static void mlx5st_process_events(struct mlx5st_device *dev) mlx5st_eq_update_ci(dev, cc, 0); } +/* + * MSI EQ — dedicated EQ for CQ completion events that fires MSI-X. + * Separate from the cmd/pages EQ so that only CQ completions (from + * send_msi or memcpy) trigger the interrupt vector. + */ + +static void mlx5st_msi_eq_drain(struct mlx5st_device *dev) +{ + u32 cc = 0; + u32 val; + + while (cc < MSI_EQ_NENT) { + u32 ci = dev->msi_eq_cons_index + cc; + struct mlx5st_eqe *eqe = + &dev->msi_eq_buf[ci % MSI_EQ_NENT]; + + if (MLX5_GET_ONCE(eqe, eqe, owner) != !!(ci & MSI_EQ_NENT)) + break; + cc++; + } + + /* Update consumer index and re-arm for next interrupt */ + dev->msi_eq_cons_index += cc; + val = (dev->msi_eq_cons_index & 0xffffff) | (dev->msi_eqn << 24); + iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET); +} + +static void mlx5st_create_msi_eq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {}; + struct mlx5_ifc_eqc_bits *eqc; + unsigned int i; + __be64 *pas; + + /* Initialize EQE owner bits */ + for (i = 0; i < MSI_EQ_NENT; i++) { + struct mlx5st_eqe *eqe = &dev->msi_eq_buf[i]; + + MLX5_SET_ONCE(eqe, eqe, owner, 1); + } + + MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ); + + /* + * No event_bitmask — completion events are routed to this EQ via + * the CQ's c_eqn field, not through CREATE_EQ subscription. + */ + eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry); + MLX5_SET(eqc, eqc, log_eq_size, LOG_MSI_EQ_SIZE); + MLX5_SET(eqc, eqc, uar_page, dev->uar_page); + MLX5_SET(eqc, eqc, intr, MSI_VECTOR); + pas = MLX5_ADDR_OF(create_eq_in, in, pas); + VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->msi_eq_buf, pas), 0u); + MLX5_SET(eqc, eqc, log_page_size, 0); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->msi_eqn = MLX5_GET(create_eq_out, out, eq_number); + dev->msi_eq_cons_index = 0; + dev->have_msi_eq = true; + mlx5st_msi_eq_drain(dev); + + dev_dbg(device, + "Created MSI EQ: eqn=%u, %d entries (COMP), vector=%d\n", + dev->msi_eqn, MSI_EQ_NENT, MSI_VECTOR); +} + +static void mlx5st_destroy_msi_eq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {}; + + MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ); + MLX5_SET(destroy_eq_in, in, eq_number, dev->msi_eqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + /* * HCA init / teardown */ @@ -1369,7 +1464,7 @@ static void mlx5st_create_cq(struct mlx5st_device *dev) cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context); MLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE); MLX5_SET(cqc, cqc, uar_page, dev->uar_page); - MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->eqn); + MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->msi_eqn); MLX5_SET(cqc, cqc, cqe_sz, 0); pas = MLX5_ADDR_OF(create_cq_in, in, pas); MLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev->cq_buf, pas)); @@ -1394,6 +1489,30 @@ static void mlx5st_destroy_cq(struct mlx5st_device *dev) mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } +/* + * Arm CQ for event generation. The CQ event delivery state machine is + * single-shot: after generating one EQE the CQ enters "Fired" state and + * won't generate another until re-armed via ARM_NEXT. Both the CQ doorbell + * record and the UAR CQ doorbell register must be written. + */ +static void mlx5st_arm_cq(struct mlx5st_device *dev) +{ + u32 sn = dev->cq_arm_sn & 3; + u32 ci = dev->cq_ci & 0xffffff; + u64 doorbell; + + /* Update CQ doorbell record arm word */ + WRITE_ONCE(dev->cq_dbrec.send_counter, + cpu_to_be32(sn << 28 | ci)); + + /* Ring CQ doorbell register, iowrite has an internal dma_wmb() */ + doorbell = ((u64)(sn << 28 | ci) << 32) | dev->cqn; + iowrite64be(doorbell, + (u8 __iomem *)dev->uar_base + MLX5_CQ_DOORBELL_OFFSET); + + dev->cq_arm_sn++; +} + /* * QP create/destroy */ @@ -1650,6 +1769,7 @@ static void mlx5st_teardown_datapath(struct mlx5st_device *dev) } dev->sq_pi = 0; dev->sq_ci = 0; + dev->cq_arm_sn = 0; memset(&dev->qp_dbrec, 0, sizeof(dev->qp_dbrec)); memset(&dev->cq_dbrec, 0, sizeof(dev->cq_dbrec)); } @@ -1691,6 +1811,34 @@ static int mlx5st_memcpy_wait(struct vfio_pci_device *device) return ret; } +/* + * send_msi callback — trigger CQE -> EQE -> MSI-X via a small RDMA Write. + * + * Both the CQ and MSI EQ use single-shot arming: the CQ must be armed so the + * CQE generates an EQE, and the MSI EQ must be armed so the EQE fires MSI-X. + */ +static void mlx5st_send_msi(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + + /* Drain accumulated MSI EQ events and re-arm for next interrupt */ + mlx5st_msi_eq_drain(dev); + + /* Arm CQ so the next CQE generates an EQE on the MSI EQ */ + mlx5st_arm_cq(dev); + + /* Post a signaled RDMA Write to trigger CQE -> EQE -> MSI-X */ + mlx5st_post_rdma_write(dev, + to_iova(device, &dev->send_msi_src), + dev->global_lkey, + to_iova(device, &dev->send_msi_dst), + dev->global_rkey, + sizeof(dev->send_msi_src), true); + + /* Consume the CQE to avoid stale completions */ + VFIO_ASSERT_EQ(mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS), 0); +} + /* * Driver ops callbacks */ @@ -1721,8 +1869,13 @@ static void mlx5st_init(struct vfio_pci_device *device) mlx5st_alloc_pd(dev); mlx5st_create_mkey(dev); + /* MSI EQ must be created before CQ so CQ can reference its eqn */ + mlx5st_create_msi_eq(dev); mlx5st_setup_datapath(dev); + vfio_pci_msix_enable(device, MSI_VECTOR, 1); + device->driver.msi = MSI_VECTOR; + device->driver.max_memcpy_size = 1ULL << dev->log_max_msg; device->driver.max_memcpy_count = SQ_WQE_CNT - 1; @@ -1733,8 +1886,14 @@ static void mlx5st_remove(struct vfio_pci_device *device) { struct mlx5st_device *dev = to_mlx5st(device); + vfio_pci_msix_disable(device); mlx5st_teardown_datapath(dev); + if (dev->have_msi_eq) { + mlx5st_destroy_msi_eq(dev); + dev->have_msi_eq = false; + } + dev_dbg(device, "teardown: destroy_mkey\n"); if (dev->mkey_index) { mlx5st_destroy_mkey(dev); @@ -1765,5 +1924,5 @@ struct vfio_pci_driver_ops mlx5st_ops = { .remove = mlx5st_remove, .memcpy_start = mlx5st_memcpy_start, .memcpy_wait = mlx5st_memcpy_wait, - .send_msi = NULL, + .send_msi = mlx5st_send_msi, }; diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h index a2506ec8a195..2c451e411ec1 100644 --- a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h @@ -80,6 +80,9 @@ struct mlx5st_dbrec { #define MLX5_BF_OFFSET 0x800 #define MLX5_BF_SIZE 0x100 +/* CQ doorbell offset within UAR page */ +#define MLX5_CQ_DOORBELL_OFFSET 0x20 + /* EQ doorbell offset within UAR page */ #define MLX5_EQ_DOORBELL_OFFSET 0x40 @@ -94,6 +97,9 @@ struct mlx5st_dbrec { #define LOG_CQ_SIZE 4 #define EQ_NENT 64 #define LOG_EQ_SIZE 6 +#define MSI_EQ_NENT 16 +#define LOG_MSI_EQ_SIZE 4 +#define MSI_VECTOR 0 #define MAX_FW_PAGES 8192 #define MAX_FW_PAGES_PER_CMD 512 -- cgit v1.2.3 From a1c1731c24e9852af872ddb0c394b8006d04af77 Mon Sep 17 00:00:00 2001 From: Alex Williamson Date: Fri, 25 Sep 2026 14:51:32 -0600 Subject: vfio: selftests: Verify a failed second open preserves the vf_token The cdev path enforces a single open per device and rejects a second bind of an already open device. That rejection must happen before the bind can mutate state shared across opens, notably the PF vf_token, so that a bind which cannot complete leaves the current opener's state untouched. Before the fix in commit 258ba46543ab ("vfio: Reject a second cdev open before mutating shared device state"), the second bind mutated the vf_token and only then failed with -EINVAL. It is now rejected early with -EBUSY. Add a regression test that binds a PF with one token, attempts a second bind of the same PF with a different token, then initializes a VF with the original token. The VF init succeeds only if the rejected second bind left the PF vf_token intact; a regression that clobbered it to the second token would make the VF init fail. The -EBUSY errno is checked with EXPECT_EQ() so the clobber assertion still runs if the rejection returns a different error. This hazard is specific to the cdev/iommufd single-open path, so the test runs only in iommufd mode. Suggested-by: David Matlack Assisted-by: LLM Signed-off-by: Alex Williamson Reviewed-by: David Matlack Link: https://lore.kernel.org/r/20260925205134.3505611-1-alex.williamson@nvidia.com Signed-off-by: Alex Williamson --- .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 35 ++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c index 19d657d00b75..f3a018b8699d 100644 --- a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c @@ -157,6 +157,41 @@ TEST_F(vfio_pci_sriov_uapi_test, override_token) ASSERT_COND_VF_CREATION(ret); } +TEST(failed_second_open_does_not_clobber_token) +{ + struct vfio_pci_device *pf = NULL, *pf_second_fd = NULL, *vf = NULL; + struct iommu *iommu; + int ret; + + iommu = iommu_init(MODE_IOMMUFD); + + /* Create and bind PF using UUID_1 */ + ret = device_init(pf_bdf, iommu, UUID_1, &pf); + ASSERT_EQ(ret, 0); + + /* + * Attempt to open the same PF again and bind it with a *different* + * token (UUID_2). This must fail with -EBUSY because the cdev path + * only supports a single open per device. Enforce it with EXPECT_EQ() + * so the clobber assertion below still runs if the errno differs. + */ + ret = device_init(pf_bdf, iommu, UUID_2, &pf_second_fd); + EXPECT_EQ(ret, -EBUSY); + + /* + * Attempt to initialize a VF using the original PF token (UUID_1). + * If the failed open above clobbered the PF's token (i.e. updated it to + * UUID_2), this VF initialization will fail. + */ + ret = device_init(vf_bdf, iommu, UUID_1, &vf); + ASSERT_EQ(ret, 0); + + device_cleanup(vf); + device_cleanup(pf_second_fd); + device_cleanup(pf); + iommu_cleanup(iommu); +} + static void vf_teardown(void) { /* -- cgit v1.2.3 From 420bed9131dbbe37abe9d6b32af51219875811e0 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:42 +0300 Subject: vfio/platform: prevent read-only region mappings from becoming writable vfio_platform_mmap() rejects writable mappings of regions without the WRITE flag, but leaves VM_MAYWRITE set. Userspace can map such a region read-only and then upgrade the mapping to writable with mprotect(). Clear VM_MAYWRITE for regions without the WRITE flag, as i915 does for its read-only objects and as fixed in drm/vc4 (CVE-2026-68445), drm/panthor (CVE-2024-53071) and commit a5edadbae57e ("ptp: vmclock: prevent read-only mappings from becoming writable"). Note this is defensive hardening: no in-tree platform driver currently publishes a region without the WRITE flag. The guard costs nothing and keeps the mmap() interface honest if a read-only region ever appears. Fixes: fad4d5b1f042 ("vfio/platform: support MMAP of MMIO regions") Reviewed-by: Mostafa Saleh Reviewed-by: Eric Auger Reviewed-by: Pranjal Shrivastava Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-2-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/platform/vfio_platform_common.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c index c72db5a99ebd..ab531318f170 100644 --- a/drivers/vfio/platform/vfio_platform_common.c +++ b/drivers/vfio/platform/vfio_platform_common.c @@ -591,9 +591,13 @@ int vfio_platform_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma && (vma->vm_flags & VM_READ)) return -EINVAL; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) - && (vma->vm_flags & VM_WRITE)) - return -EINVAL; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EINVAL; + + vm_flags_clear(vma, VM_MAYWRITE); + } vma->vm_private_data = vdev; -- cgit v1.2.3 From 25fd8b27ee8d4f59cc2ad99cf67aef4570b82096 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:43 +0300 Subject: vfio/fsl-mc: prevent read-only region mappings from becoming writable vfio_fsl_mc_mmap() rejects writable mappings of regions without the WRITE flag, but leaves VM_MAYWRITE set. Userspace can map such a region read-only and then upgrade the mapping to writable with mprotect(). Clear VM_MAYWRITE for regions without the WRITE flag, as i915 does for its read-only objects and as fixed in drm/vc4 (CVE-2026-68445), drm/panthor (CVE-2024-53071) and commit a5edadbae57e ("ptp: vmclock: prevent read-only mappings from becoming writable"). Note this is defensive hardening: the fsl-mc bus publishes all device regions with the WRITE flag set, so no device can currently reach the read-only path. The guard costs nothing and keeps the mmap() interface honest if a read-only region ever appears. Fixes: 67247289688d4 ("vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions") Reviewed-by: Ioana Ciornei Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-3-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index b4c3958201b2..5a6cd78dcc06 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -403,9 +403,13 @@ static int vfio_fsl_mc_mmap(struct vfio_device *core_vdev, && (vma->vm_flags & VM_READ)) return -EINVAL; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) - && (vma->vm_flags & VM_WRITE)) - return -EINVAL; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EINVAL; + + vm_flags_clear(vma, VM_MAYWRITE); + } vma->vm_private_data = mc_dev; -- cgit v1.2.3 From 9c5cd385d295fa8621dff0e9f6063e3f6ff24159 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:44 +0300 Subject: vfio/cdx: prevent read-only region mappings from becoming writable vfio_cdx_mmap() rejects writable mappings of regions without the WRITE flag, but leaves VM_MAYWRITE set. Userspace can map such a region read-only and then upgrade the mapping to writable with mprotect(), writing to MMIO regions the device marks read-only. Clear VM_MAYWRITE for regions without the WRITE flag, as i915 does for its read-only objects and as fixed in drm/vc4 (CVE-2026-68445), drm/panthor (CVE-2024-53071) and commit a5edadbae57e ("ptp: vmclock: prevent read-only mappings from becoming writable"). Note this is defensive hardening: no in-tree cdx device currently publishes a region without the WRITE flag. The guard costs nothing and keeps the mmap() interface honest if a read-only region ever appears. Fixes: 234489ac56130 ("vfio/cdx: add support for CDX bus") Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-4-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/cdx/main.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/cdx/main.c b/drivers/vfio/cdx/main.c index b31ed4be7bdc..28386fdb01f2 100644 --- a/drivers/vfio/cdx/main.c +++ b/drivers/vfio/cdx/main.c @@ -281,9 +281,13 @@ static int vfio_cdx_mmap(struct vfio_device *core_vdev, (vma->vm_flags & VM_READ)) return -EPERM; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) && - (vma->vm_flags & VM_WRITE)) - return -EPERM; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EPERM; + + vm_flags_clear(vma, VM_MAYWRITE); + } return vfio_cdx_mmap_mmio(vdev->regions[index], vma); } -- cgit v1.2.3 From a9ce237e1a1b3123ca75bd94308572f088fa43d8 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:45 +0300 Subject: vfio/platform: keep logical vm_pgoff in MMIO region mmap vfio_platform_mmap_mmio() overwrites vma->vm_pgoff with the physical frame number of the MMIO region and passes it to remap_pfn_range(). The VMA is inserted into the device file's mapping->i_mmap interval tree keyed by vm_pgoff, which VFIO expects to be the logical file offset: the core links every device mmap to the device inode's i_mapping precisely so that unmap_mapping_range() can revoke all mappings associated with a device (see vfio_device_cdev_open()). With a raw PFN in vm_pgoff the interval tree entry lands in the wrong coordinate space and unmap_mapping_range() cannot find the VMA, leaving stale MMIO mappings behind any revocation attempt. Keep vm_pgoff in the logical VFIO offset space, as vfio-pci does, and pass the physical PFN to remap_pfn_range() explicitly. This is defensive hygiene: none of vfio-platform, vfio/fsl-mc and vfio/cdx call unmap_mapping_range() today, so no reachable stale-mapping issue exists. Keeping vm_pgoff logical preserves the VFIO core contract for any future revocation path. Reviewed-by: Eric Auger Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-5-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/platform/vfio_platform_common.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c index ab531318f170..e8cfa76ad7f8 100644 --- a/drivers/vfio/platform/vfio_platform_common.c +++ b/drivers/vfio/platform/vfio_platform_common.c @@ -559,9 +559,8 @@ static int vfio_platform_mmap_mmio(struct vfio_platform_region region, return -EINVAL; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; - - return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, req_len, vma->vm_page_prot); } -- cgit v1.2.3 From c9e9ecb94f09aadc813c9640512495ba0c314726 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:46 +0300 Subject: vfio/fsl-mc: keep logical vm_pgoff in MMIO region mmap vfio_fsl_mc_mmap_mmio() overwrites vma->vm_pgoff with the physical frame number of the MMIO region and passes it to remap_pfn_range(). The VMA is inserted into the device file's mapping->i_mmap interval tree keyed by vm_pgoff, which VFIO expects to be the logical file offset: the core links every device mmap to the device inode's i_mapping precisely so that unmap_mapping_range() can revoke all mappings associated with a device (see vfio_device_cdev_open()). With a raw PFN in vm_pgoff the interval tree entry lands in the wrong coordinate space and unmap_mapping_range() cannot find the VMA, leaving stale MMIO mappings behind any revocation attempt. Keep vm_pgoff in the logical VFIO offset space, as vfio-pci does, and pass the physical PFN to remap_pfn_range() explicitly. This is defensive hygiene: none of vfio-platform, vfio/fsl-mc and vfio/cdx call unmap_mapping_range() today, so no reachable stale-mapping issue exists. Keeping vm_pgoff logical preserves the VFIO core contract for any future revocation path. Reviewed-by: Ioana Ciornei Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-6-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 5a6cd78dcc06..15b05b8c0c90 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -369,9 +369,8 @@ static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region, if (!region_cacheable) vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; - - return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, size, vma->vm_page_prot); } -- cgit v1.2.3 From 614634f262d83f367c993c1efd26230303f2e725 Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:47 +0300 Subject: vfio/cdx: keep logical vm_pgoff in MMIO region mmap vfio_cdx_mmap_mmio() overwrites vma->vm_pgoff with the physical frame number of the MMIO region and passes it to io_remap_pfn_range(). The VMA is inserted into the device file's mapping->i_mmap interval tree keyed by vm_pgoff, which VFIO expects to be the logical file offset: the core links every device mmap to the device inode's i_mapping precisely so that unmap_mapping_range() can revoke all mappings associated with a device (see vfio_device_cdev_open()). With a raw PFN in vm_pgoff the interval tree entry lands in the wrong coordinate space and unmap_mapping_range() cannot find the VMA, leaving stale MMIO mappings behind any revocation attempt. Keep vm_pgoff in the logical VFIO offset space, as vfio-pci does, and pass the physical PFN to io_remap_pfn_range() explicitly. This is defensive hygiene: none of vfio-platform, vfio/fsl-mc and vfio/cdx call unmap_mapping_range() today, so no reachable stale-mapping issue exists. Keeping vm_pgoff logical preserves the VFIO core contract for any future revocation path. Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-7-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/cdx/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/cdx/main.c b/drivers/vfio/cdx/main.c index 28386fdb01f2..ea55b7767672 100644 --- a/drivers/vfio/cdx/main.c +++ b/drivers/vfio/cdx/main.c @@ -254,10 +254,10 @@ static int vfio_cdx_mmap_mmio(struct vfio_cdx_region region, if (base + size > region.size) return -EINVAL; - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; vma->vm_page_prot = pgprot_device(vma->vm_page_prot); - return io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return io_remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, size, vma->vm_page_prot); } -- cgit v1.2.3 From 8b25b35840c15ecca14251f68079792531f1758f Mon Sep 17 00:00:00 2001 From: Abdifatah Suruur Date: Tue, 29 Sep 2026 12:48:48 +0300 Subject: vfio/cdx: reject non-shared MMIO mmaps vfio_cdx_mmap() accepts MAP_PRIVATE mappings. A non-shared mapping with VM_MAYWRITE is a COW mapping, and remap_pfn_range()'s COW special case in get_remap_pgoff() then overwrites vma->vm_pgoff with the physical frame number: if (is_cow) { if (addr != vm_start || end != vm_end) return -EINVAL; *vm_pgoff_p = pfn; } The VMA is inserted into the device file's mapping->i_mmap interval tree keyed by vm_pgoff, which VFIO expects to be the logical file offset: the core links every device mmap to the device inode's i_mapping precisely so that unmap_mapping_range() can revoke all mappings associated with a device (see vfio_device_open_file() and vfio_device_fops_cdev_open()). A raw PFN in vm_pgoff lands the interval tree entry in the wrong coordinate space, so unmap_mapping_range() cannot find the VMA and a stale MMIO mapping survives any revocation attempt. Without this check, a MAP_PRIVATE mapping reintroduces that corruption even with the logical-vm_pgoff fix in place. Require VM_SHARED as vfio-pci, vfio-fsl-mc and vfio-platform already do. This is defensive hygiene: vfio-cdx does not call unmap_mapping_range() today, so no reachable stale-mapping issue exists. Signed-off-by: Abdifatah Suruur Link: https://lore.kernel.org/r/20260929094848.7439-8-suruurism@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/cdx/main.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/vfio/cdx/main.c b/drivers/vfio/cdx/main.c index ea55b7767672..28dc8c825600 100644 --- a/drivers/vfio/cdx/main.c +++ b/drivers/vfio/cdx/main.c @@ -271,6 +271,9 @@ static int vfio_cdx_mmap(struct vfio_device *core_vdev, index = vma->vm_pgoff >> (VFIO_CDX_OFFSET_SHIFT - PAGE_SHIFT); + if (!(vma->vm_flags & VM_SHARED)) + return -EINVAL; + if (index >= cdx_dev->res_count) return -EINVAL; -- cgit v1.2.3 From b30b52c2fb82e6d38ee2845a4d85e8254b44f3d8 Mon Sep 17 00:00:00 2001 From: Li RongQing Date: Tue, 22 Sep 2026 12:00:13 +0800 Subject: vfio/pci: Restore 8-byte ioeventfd support Commit b44a06bd28f2 ("vfio/pci: Remove #ifdef iowrite64 and #ifdef ioread64") inadvertently converted a conditional 8-byte rejection into an unconditional one. Originally, 'count == 8' was rejected only if iowrite64 was undefined (#ifndef iowrite64). When iowrite64 became globally available via generic fallbacks, this rejection became dead code, allowing 8-byte ioeventfds. However, b44a06bd28f2 removed the #ifndef wrappers but left the nested 'if (count == 8) return -EINVAL;' intact, breaking all 8-byte registrations. Drop the unconditional check to restore 8-byte ioeventfd support, matching the original intent. Fixes: b44a06bd28f2 ("vfio/pci: Remove #ifdef iowrite64 and #ifdef ioread64") Cc: stable@vger.kernel.org Signed-off-by: Li RongQing Link: https://lore.kernel.org/r/20260922040013.2188-1-lirongqing@baidu.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/vfio_pci_rdwr.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c index 7f14dd46de17..df8848087681 100644 --- a/drivers/vfio/pci/vfio_pci_rdwr.c +++ b/drivers/vfio/pci/vfio_pci_rdwr.c @@ -423,9 +423,6 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset, pos >= vdev->msix_offset + vdev->msix_size)) return -EINVAL; - if (count == 8) - return -EINVAL; - io = vfio_pci_core_get_iomap(vdev, bar); if (IS_ERR(io)) return PTR_ERR(io); -- cgit v1.2.3 From 93d41424caf7569ab50092a922477e2a881232b4 Mon Sep 17 00:00:00 2001 From: Dave Airlie Date: Tue, 22 Sep 2026 16:29:37 +1000 Subject: PCI/VGA: pass a data pointer to be used in the decode callback (v2) This changes the vgaarb client API so that the user can pass a private data pointer into the register that will get used in the decode callback. This allows a bunch of pdev conversions in the drivers, and lets some future vfio cleanups be nicer. I'd like to merge this via the drm next tree but also fine with it going via pci. Signed-off-by: Dave Airlie Reviewed-by: Jani Nikula Acked-by: Alex Williamson Acked-by: Bjorn Helgaas Cc: Rodrigo Vivi Acked-by: Alex Deucher Link: https://lore.kernel.org/dri-devel/20260922071807.2533884-1-airlied@gmail.com/ --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 ++++----- drivers/gpu/drm/i915/display/intel_vga.c | 7 ++++--- drivers/gpu/drm/loongson/lsdc_drv.c | 4 ++-- drivers/gpu/drm/nouveau/nouveau_vga.c | 6 +++--- drivers/gpu/drm/radeon/radeon_device.c | 9 ++++----- drivers/pci/vgaarb.c | 13 +++++++++---- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/linux/vgaarb.h | 8 +++++--- 8 files changed, 36 insertions(+), 29 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 9269e780feb7..ef9ff30c0a7b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -1429,16 +1429,15 @@ bool amdgpu_device_should_use_aspm(struct amdgpu_device *adev) /** * amdgpu_device_vga_set_decode - enable/disable vga decode * - * @pdev: PCI device pointer + * @data: driver private data (adev) * @state: enable/disable vga decode * * Enable/disable vga decode (all asics). * Returns VGA resource flags. */ -static unsigned int amdgpu_device_vga_set_decode(struct pci_dev *pdev, - bool state) +static unsigned int amdgpu_device_vga_set_decode(void *data, bool state) { - struct amdgpu_device *adev = drm_to_adev(pci_get_drvdata(pdev)); + struct amdgpu_device *adev = data; amdgpu_asic_set_vga_state(adev, state); if (state) @@ -4199,7 +4198,7 @@ fence_driver_init: * ignore it */ if ((adev->pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) - vga_client_register(adev->pdev, amdgpu_device_vga_set_decode); + vga_client_register(adev->pdev, amdgpu_device_vga_set_decode, adev); px = amdgpu_device_supports_px(adev); diff --git a/drivers/gpu/drm/i915/display/intel_vga.c b/drivers/gpu/drm/i915/display/intel_vga.c index 4966fd7eea92..4c292fae7b92 100644 --- a/drivers/gpu/drm/i915/display/intel_vga.c +++ b/drivers/gpu/drm/i915/display/intel_vga.c @@ -283,9 +283,10 @@ reset_vgacntr: intel_de_posting_read(display, vga_reg); } -static unsigned int intel_vga_set_decode(struct pci_dev *pdev, bool enable_decode) +static unsigned int intel_vga_set_decode(void *data, bool enable_decode) { - struct intel_display *display = to_intel_display(pdev); + struct intel_display *display = data; + struct pci_dev *pdev = to_pci_dev(display->drm->dev); unsigned int decodes = VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; drm_dbg_kms(display->drm, "%s VGA decode due to VGA arbitration\n", @@ -331,7 +332,7 @@ void intel_vga_register(struct intel_display *display) * then we do not take part in VGA arbitration and the * vga_client_register() fails with -ENODEV. */ - ret = vga_client_register(pdev, intel_vga_set_decode); + ret = vga_client_register(pdev, intel_vga_set_decode, display); drm_WARN_ON(display->drm, ret && ret != -ENODEV); } diff --git a/drivers/gpu/drm/loongson/lsdc_drv.c b/drivers/gpu/drm/loongson/lsdc_drv.c index 3fd828d23271..5a13df31eb90 100644 --- a/drivers/gpu/drm/loongson/lsdc_drv.c +++ b/drivers/gpu/drm/loongson/lsdc_drv.c @@ -254,7 +254,7 @@ lsdc_create_device(struct pci_dev *pdev, /* For multiple GPU driver instance co-exixt in the system */ -static unsigned int lsdc_vga_set_decode(struct pci_dev *pdev, bool state) +static unsigned int lsdc_vga_set_decode(void *data, bool state) { return VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; } @@ -291,7 +291,7 @@ static int lsdc_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) pci_set_drvdata(pdev, ddev); - vga_client_register(pdev, lsdc_vga_set_decode); + vga_client_register(pdev, lsdc_vga_set_decode, NULL); drmm_kms_helper_poll_init(ddev); diff --git a/drivers/gpu/drm/nouveau/nouveau_vga.c b/drivers/gpu/drm/nouveau/nouveau_vga.c index a6c375a24154..17464c846ffd 100644 --- a/drivers/gpu/drm/nouveau/nouveau_vga.c +++ b/drivers/gpu/drm/nouveau/nouveau_vga.c @@ -9,9 +9,9 @@ #include "nouveau_vga.h" static unsigned int -nouveau_vga_set_decode(struct pci_dev *pdev, bool state) +nouveau_vga_set_decode(void *data, bool state) { - struct nouveau_drm *drm = pci_get_drvdata(pdev); + struct nouveau_drm *drm = data; struct nvif_object *device = &drm->client.device.object; if (drm->client.device.info.family == NV_DEVICE_INFO_V0_CURIE && @@ -95,7 +95,7 @@ nouveau_vga_init(struct nouveau_drm *drm) return; pdev = to_pci_dev(dev->dev); - vga_client_register(pdev, nouveau_vga_set_decode); + vga_client_register(pdev, nouveau_vga_set_decode, drm); /* don't register Thunderbolt eGPU with vga_switcheroo */ if (pci_is_thunderbolt_attached(pdev)) diff --git a/drivers/gpu/drm/radeon/radeon_device.c b/drivers/gpu/drm/radeon/radeon_device.c index 705c012fcf9e..d21fdecd163f 100644 --- a/drivers/gpu/drm/radeon/radeon_device.c +++ b/drivers/gpu/drm/radeon/radeon_device.c @@ -1070,16 +1070,15 @@ void radeon_combios_fini(struct radeon_device *rdev) /** * radeon_vga_set_decode - enable/disable vga decode * - * @pdev: PCI device + * @data: driver private data (rdev) * @state: enable/disable vga decode * * Enable/disable vga decode (all asics). * Returns VGA resource flags. */ -static unsigned int radeon_vga_set_decode(struct pci_dev *pdev, bool state) +static unsigned int radeon_vga_set_decode(void *data, bool state) { - struct drm_device *dev = pci_get_drvdata(pdev); - struct radeon_device *rdev = dev->dev_private; + struct radeon_device *rdev = data; radeon_vga_set_state(rdev, state); if (state) return VGA_RSRC_LEGACY_IO | VGA_RSRC_LEGACY_MEM | @@ -1424,7 +1423,7 @@ int radeon_device_init(struct radeon_device *rdev, /* if we have > 1 VGA cards, then disable the radeon VGA resources */ /* this will fail for cards that aren't VGA class devices, just * ignore it */ - vga_client_register(rdev->pdev, radeon_vga_set_decode); + vga_client_register(rdev->pdev, radeon_vga_set_decode, rdev); if (rdev->flags & RADEON_IS_PX) runtime = true; diff --git a/drivers/pci/vgaarb.c b/drivers/pci/vgaarb.c index 3de05aee7859..3e58b2c05c85 100644 --- a/drivers/pci/vgaarb.c +++ b/drivers/pci/vgaarb.c @@ -51,7 +51,8 @@ struct vga_device { unsigned int mem_norm_cnt; /* normal MEM count */ bool bridge_has_one_vga; bool is_firmware_default; /* device selected by firmware */ - unsigned int (*set_decode)(struct pci_dev *pdev, bool decode); + unsigned int (*set_decode)(void *data, bool decode); + void *set_decode_data; }; static LIST_HEAD(vga_list); @@ -935,6 +936,7 @@ EXPORT_SYMBOL(vga_set_legacy_decoding); * vga_client_register - register or unregister a VGA arbitration client * @pdev: PCI device of the VGA client * @set_decode: VGA decode change callback + * @data: private data to pass to callback * * Clients have two callback mechanisms they can use. * @@ -956,15 +958,18 @@ EXPORT_SYMBOL(vga_set_legacy_decoding); * Returns: 0 on success, -ENODEV on failure */ int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool decode)) + unsigned int (*set_decode)(void *data, bool decode), + void *data) { unsigned long flags; struct vga_device *vgadev; spin_lock_irqsave(&vga_lock, flags); vgadev = vgadev_find(pdev); - if (vgadev) + if (vgadev) { vgadev->set_decode = set_decode; + vgadev->set_decode_data = data; + } spin_unlock_irqrestore(&vga_lock, flags); if (!vgadev) return -ENODEV; @@ -1464,7 +1469,7 @@ static void vga_arbiter_notify_clients(void) spin_lock_irqsave(&vga_lock, flags); list_for_each_entry(vgadev, &vga_list, list) { if (vgadev->set_decode) { - new_decodes = vgadev->set_decode(vgadev->pdev, + new_decodes = vgadev->set_decode(vgadev->set_decode_data, new_state); vga_update_device_decodes(vgadev, new_decodes); } diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 6757054e9d87..b04ce3ded08c 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -162,9 +162,10 @@ static inline void vfio_pci_core_debugfs_init(struct vfio_pci_core_device *vdev) * has no way to get to it and routing can be disabled externally at the * bridge. */ -static unsigned int vfio_pci_set_decode(struct pci_dev *pdev, bool single_vga) +static unsigned int vfio_pci_set_decode(void *data, bool single_vga) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); + struct vfio_pci_core_device *vdev = data; + struct pci_dev *pdev = vdev->pdev; struct pci_dev *tmp = NULL; unsigned char max_busnr; unsigned int decodes; @@ -2159,10 +2160,10 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev) if (ret) return ret; - ret = vga_client_register(pdev, vfio_pci_set_decode); + ret = vga_client_register(pdev, vfio_pci_set_decode, vdev); if (ret) return ret; - vga_set_legacy_decoding(pdev, vfio_pci_set_decode(pdev, false)); + vga_set_legacy_decoding(pdev, vfio_pci_set_decode(vdev, false)); return 0; } diff --git a/include/linux/vgaarb.h b/include/linux/vgaarb.h index 97129a1bbb7d..71a364669eaf 100644 --- a/include/linux/vgaarb.h +++ b/include/linux/vgaarb.h @@ -33,7 +33,8 @@ struct pci_dev *vga_default_device(void); void vga_set_default_device(struct pci_dev *pdev); int vga_remove_vgacon(struct pci_dev *pdev); int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool state)); + unsigned int (*set_decode)(void *data, bool state), + void *data); #else /* CONFIG_VGA_ARB */ static inline void vga_set_legacy_decoding(struct pci_dev *pdev, unsigned int decodes) @@ -59,7 +60,8 @@ static inline int vga_remove_vgacon(struct pci_dev *pdev) return 0; } static inline int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool state)) + unsigned int (*set_decode)(void *data, bool state), + void *data) { return 0; } @@ -97,7 +99,7 @@ static inline int vga_get_uninterruptible(struct pci_dev *pdev, static inline void vga_client_unregister(struct pci_dev *pdev) { - vga_client_register(pdev, NULL); + vga_client_register(pdev, NULL, NULL); } #endif /* LINUX_VGA_H */ -- cgit v1.2.3 From 3b2e9b05011ee74543b6d17401bb02f1df3b93ff Mon Sep 17 00:00:00 2001 From: Jiale Yao Date: Sat, 3 Oct 2026 17:38:06 +0800 Subject: vfio/pci: Drain eventfd RCU callbacks on module exit vfio_pci_eventfd_replace_locked() defers freeing replaced eventfds to vfio_pci_eventfd_rcu_free(). A callback can remain queued after close_device() returns and the device driver module reference is dropped. Once the dependent driver is removed, vfio-pci-core can be unloaded while the callback still points into its module text. Wait for outstanding RCU callbacks before vfio-pci-core exits. Fixes: 98693e0897f7 ("vfio/pci: Use RCU for error/request triggers to avoid circular locking") Signed-off-by: Jiale Yao Link: https://lore.kernel.org/r/20261003093806.544631-1-yaojiale02@163.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/vfio_pci_core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 6757054e9d87..9bce419c52c8 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2675,6 +2675,7 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set) static void vfio_pci_core_cleanup(void) { vfio_pci_uninit_perm_bits(); + rcu_barrier(); } static int __init vfio_pci_core_init(void) -- cgit v1.2.3 From 37b2ee6747b7857920ebc71eabc25208a39164a4 Mon Sep 17 00:00:00 2001 From: Seongjun Hong Date: Mon, 5 Oct 2026 16:16:29 +0000 Subject: vfio/mlx5: Fix boundary check in set_report_output() Use > instead of >= on boundary check of recv_buf to avoid the bug which the full message in the last slot of recv_buf is ignored even though the received data size does not overflow the recv_buf. Fixes: 9c7c5430bca3 ("vfio/mlx5: Align the page tracking max message size with the device capability") Cc: stable@vger.kernel.org Reviewed-by: Yishai Hadas Signed-off-by: Seongjun Hong Link: https://lore.kernel.org/r/20261005-vfio-mlx5-fix-boundary-check-v2-1-75599d2f4497@snu.ac.kr Signed-off-by: Alex Williamson --- drivers/vfio/pci/mlx5/cmd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 5fe0621b5fbd..04180035a7fe 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -1644,7 +1644,7 @@ set_report_output(u32 size, int index, struct mlx5_vhca_qp *qp, int i; buf_offset = index * qp->max_msg_size; - if (WARN_ON(buf_offset + size >= qp->recv_buf.npages * PAGE_SIZE || + if (WARN_ON(buf_offset + size > qp->recv_buf.npages * PAGE_SIZE || (nent > qp->max_msg_size / entry_size))) return; -- cgit v1.2.3 From 6a5f427bee12c82beaf13efbd133416da1541c39 Mon Sep 17 00:00:00 2001 From: Simon Song Date: Mon, 5 Oct 2026 11:42:36 -0700 Subject: vfio/pci: Remove the core dependency on driver data vfio-pci-core currently has runtime functions that interpret pci driver_data as a pointer to vfio_pci_core_device, and enforce vfio variant drivers must set vfio_pci_core_device to their pci driver_data. This constrains variant drivers' private-data layout, including the typed driver data used by the Rust PCI infrastructure. Add VFIO_PCI_CORE_DEFINE_CALLBACKS macro to generate wrapper code for each vfio variant driver, update vfio variant drivers to store their private structures in driver_data and use generated wrapper to pass the vfio core device to helpers. Assisted-by: LLM Suggested-by: Jason Gunthorpe Suggested-by: Alex Williamson Suggested-by: Danilo Krummrich Link: https://lore.kernel.org/all/DLFD2ZDSK9YQ.3A4R66G8UJMD8@kernel.org/ Signed-off-by: Simon Song Reviewed-by: Jason Gunthorpe Reviewed-by: Danilo Krummrich Link: https://lore.kernel.org/r/20261005184236.1946167-2-xinmengs@nvidia.com [VGA chunks and commit log reference dropped with rebase to commit 93d41424caf7 ("PCI/VGA: pass a data pointer to be used in the decode callback (v2)")] Signed-off-by: Alex Williamson --- drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 27 ++++++++--------- drivers/vfio/pci/ism/main.c | 19 +++++++----- drivers/vfio/pci/mlx5/main.c | 13 ++++---- drivers/vfio/pci/nvgrace-gpu/main.c | 19 ++++++------ drivers/vfio/pci/pds/pci_drv.c | 8 +++-- drivers/vfio/pci/pds/vfio_dev.c | 5 +--- drivers/vfio/pci/qat/main.c | 12 ++++---- drivers/vfio/pci/vfio_pci.c | 33 ++++++++++++++++++++- drivers/vfio/pci/vfio_pci_core.c | 36 ++++------------------ drivers/vfio/pci/virtio/main.c | 8 +++-- drivers/vfio/pci/xe/main.c | 8 +++-- include/linux/vfio_pci_core.h | 41 ++++++++++++++++++++++++-- 12 files changed, 141 insertions(+), 88 deletions(-) diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index 86362ec424a5..a5ff79b968d9 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -16,6 +16,9 @@ #include "hisi_acc_vfio_pci.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(hisi_acc_vf, struct hisi_acc_vf_core_device, + core_device) + /* Return 0 on VM acc device ready, -ETIMEDOUT hardware timeout */ static int qm_wait_dev_not_ready(struct hisi_qm *qm) { @@ -588,10 +591,7 @@ static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, static struct hisi_acc_vf_core_device *hisi_acc_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct hisi_acc_vf_core_device, - core_device); + return pci_get_drvdata(pdev); } /* Check the PF's RAS state and Function INT state */ @@ -1416,9 +1416,8 @@ static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device *vde static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; u64 value; int ret; @@ -1445,9 +1444,8 @@ static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); struct acc_vf_data *vf_data; int ret; @@ -1492,9 +1490,7 @@ mutex_release: static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev->debug_migf; @@ -1687,7 +1683,7 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device if (IS_ERR(hisi_acc_vdev)) return PTR_ERR(hisi_acc_vdev); - dev_set_drvdata(&pdev->dev, &hisi_acc_vdev->core_device); + dev_set_drvdata(&pdev->dev, hisi_acc_vdev); ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); if (ret) goto out_put_vdev; @@ -1721,7 +1717,7 @@ MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table); static const struct pci_error_handlers hisi_acc_vf_err_handlers = { .reset_prepare = hisi_acc_vf_pci_reset_prepare, .reset_done = hisi_acc_vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = hisi_acc_vf_aer_err_detected, }; static struct pci_driver hisi_acc_vfio_pci_driver = { @@ -1729,6 +1725,7 @@ static struct pci_driver hisi_acc_vfio_pci_driver = { .id_table = hisi_acc_vfio_pci_table, .probe = hisi_acc_vfio_pci_probe, .remove = hisi_acc_vfio_pci_remove, + .driver = { .pm = pm_ptr(&hisi_acc_vf_pm_ops) }, .err_handler = &hisi_acc_vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/ism/main.c b/drivers/vfio/pci/ism/main.c index f83e09b915ed..cce3bdf1f070 100644 --- a/drivers/vfio/pci/ism/main.c +++ b/drivers/vfio/pci/ism/main.c @@ -47,6 +47,9 @@ struct ism_vfio_pci_core_device { struct kmem_cache *store_block_cache; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(ism_vfio, struct ism_vfio_pci_core_device, + core_device) + static int ism_vfio_pci_open_device(struct vfio_device *core_vdev) { struct ism_vfio_pci_core_device *ivpcd; @@ -363,7 +366,7 @@ static int ism_vfio_pci_probe(struct pci_dev *pdev, if (IS_ERR(ivpcd)) return PTR_ERR(ivpcd); - dev_set_drvdata(&pdev->dev, &ivpcd->core_device); + dev_set_drvdata(&pdev->dev, ivpcd); ret = vfio_pci_core_register_device(&ivpcd->core_device); if (ret) @@ -374,12 +377,7 @@ static int ism_vfio_pci_probe(struct pci_dev *pdev, static void ism_vfio_pci_remove(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device; - struct ism_vfio_pci_core_device *ivpcd; - - core_device = dev_get_drvdata(&pdev->dev); - ivpcd = container_of(core_device, struct ism_vfio_pci_core_device, - core_device); + struct ism_vfio_pci_core_device *ivpcd = pci_get_drvdata(pdev); vfio_pci_core_unregister_device(&ivpcd->core_device); vfio_put_device(&ivpcd->core_device.vdev); @@ -392,12 +390,17 @@ static const struct pci_device_id ism_device_table[] = { }; MODULE_DEVICE_TABLE(pci, ism_device_table); +static const struct pci_error_handlers ism_vfio_err_handlers = { + .error_detected = ism_vfio_aer_err_detected, +}; + static struct pci_driver ism_vfio_pci_driver = { .name = KBUILD_MODNAME, .id_table = ism_device_table, .probe = ism_vfio_pci_probe, .remove = ism_vfio_pci_remove, - .err_handler = &vfio_pci_core_err_handlers, + .driver = { .pm = pm_ptr(&ism_vfio_pm_ops) }, + .err_handler = &ism_vfio_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index de306dee1d1a..18a6d224b1fd 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -21,6 +21,9 @@ #include "cmd.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(mlx5vf, struct mlx5vf_pci_core_device, + core_device) + /* Device specification max LOAD size */ #define MAX_LOAD_SIZE (BIT_ULL(__mlx5_bit_sz(load_vhca_state_in, size)) - 1) @@ -28,10 +31,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct mlx5vf_pci_core_device, - core_device); + return pci_get_drvdata(pdev); } static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf) @@ -1415,7 +1415,7 @@ static int mlx5vf_pci_probe(struct pci_dev *pdev, if (IS_ERR(mvdev)) return PTR_ERR(mvdev); - dev_set_drvdata(&pdev->dev, &mvdev->core_device); + dev_set_drvdata(&pdev->dev, mvdev); ret = vfio_pci_core_register_device(&mvdev->core_device); if (ret) goto out_put_vdev; @@ -1443,7 +1443,7 @@ MODULE_DEVICE_TABLE(pci, mlx5vf_pci_table); static const struct pci_error_handlers mlx5vf_err_handlers = { .reset_done = mlx5vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = mlx5vf_aer_err_detected, }; static struct pci_driver mlx5vf_pci_driver = { @@ -1451,6 +1451,7 @@ static struct pci_driver mlx5vf_pci_driver = { .id_table = mlx5vf_pci_table, .probe = mlx5vf_pci_probe, .remove = mlx5vf_pci_remove, + .driver = { .pm = pm_ptr(&mlx5vf_pm_ops) }, .err_handler = &mlx5vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 963fd8ded20d..087850b92092 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -73,6 +73,9 @@ struct nvgrace_gpu_pci_core_device { int cxl_dvsec; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(nvgrace_gpu, struct nvgrace_gpu_pci_core_device, + core_device) + static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev) { struct nvgrace_gpu_pci_core_device *nvdev = @@ -1341,7 +1344,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, if (ret) goto out_put_vdev; - dev_set_drvdata(&pdev->dev, &nvdev->core_device); + dev_set_drvdata(&pdev->dev, nvdev); if (ops == &nvgrace_gpu_pci_ops) { nvdev->has_mig_hw_bug = nvgrace_gpu_has_mig_hw_bug(pdev); @@ -1372,10 +1375,10 @@ out_put_vdev: static void nvgrace_gpu_remove(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); + struct nvgrace_gpu_pci_core_device *nvdev = pci_get_drvdata(pdev); - vfio_pci_core_unregister_device(core_device); - vfio_put_device(&core_device->vdev); + vfio_pci_core_unregister_device(&nvdev->core_device); + vfio_put_device(&nvdev->core_device.vdev); } static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { @@ -1407,17 +1410,14 @@ MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table); */ static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - struct nvgrace_gpu_pci_core_device *nvdev = - container_of(core_device, struct nvgrace_gpu_pci_core_device, - core_device); + struct nvgrace_gpu_pci_core_device *nvdev = pci_get_drvdata(pdev); nvdev->reset_done = true; } static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = { .reset_done = nvgrace_gpu_vfio_pci_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = nvgrace_gpu_aer_err_detected, }; static struct pci_driver nvgrace_gpu_vfio_pci_driver = { @@ -1425,6 +1425,7 @@ static struct pci_driver nvgrace_gpu_vfio_pci_driver = { .id_table = nvgrace_gpu_vfio_pci_table, .probe = nvgrace_gpu_probe, .remove = nvgrace_gpu_remove, + .driver = { .pm = pm_ptr(&nvgrace_gpu_pm_ops) }, .err_handler = &nvgrace_gpu_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/pds/pci_drv.c b/drivers/vfio/pci/pds/pci_drv.c index 4923f1823126..cf7e33d4d17b 100644 --- a/drivers/vfio/pci/pds/pci_drv.c +++ b/drivers/vfio/pci/pds/pci_drv.c @@ -16,6 +16,9 @@ #include "pci_drv.h" #include "cmds.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(pds_vfio, struct pds_vfio_pci_device, + vfio_coredev) + #define PDS_VFIO_DRV_DESCRIPTION "AMD/Pensando VFIO Device Driver" #define PCI_VENDOR_ID_PENSANDO 0x1dd8 @@ -118,7 +121,7 @@ static int pds_vfio_pci_probe(struct pci_dev *pdev, if (IS_ERR(pds_vfio)) return PTR_ERR(pds_vfio); - dev_set_drvdata(&pdev->dev, &pds_vfio->vfio_coredev); + dev_set_drvdata(&pdev->dev, pds_vfio); err = vfio_pci_core_register_device(&pds_vfio->vfio_coredev); if (err) @@ -173,7 +176,7 @@ static void pds_vfio_pci_aer_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers pds_vfio_pci_err_handlers = { .reset_done = pds_vfio_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = pds_vfio_aer_err_detected, }; static struct pci_driver pds_vfio_pci_driver = { @@ -181,6 +184,7 @@ static struct pci_driver pds_vfio_pci_driver = { .id_table = pds_vfio_pci_table, .probe = pds_vfio_pci_probe, .remove = pds_vfio_pci_remove, + .driver = { .pm = pm_ptr(&pds_vfio_pm_ops) }, .err_handler = &pds_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c index be103c74e969..6d2b2dc61a28 100644 --- a/drivers/vfio/pci/pds/vfio_dev.c +++ b/drivers/vfio/pci/pds/vfio_dev.c @@ -20,10 +20,7 @@ struct device *pds_vfio_to_dev(struct pds_vfio_pci_device *pds_vfio) struct pds_vfio_pci_device *pds_vfio_pci_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct pds_vfio_pci_device, - vfio_coredev); + return pci_get_drvdata(pdev); } void pds_vfio_reset(struct pds_vfio_pci_device *pds_vfio, diff --git a/drivers/vfio/pci/qat/main.c b/drivers/vfio/pci/qat/main.c index 60ff907b6a67..69448fa57df4 100644 --- a/drivers/vfio/pci/qat/main.c +++ b/drivers/vfio/pci/qat/main.c @@ -55,6 +55,9 @@ struct qat_vf_core_device { struct qat_vf_migration_file *saving_migf; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(qat_vf, struct qat_vf_core_device, + core_device) + static int qat_vf_pci_open_device(struct vfio_device *core_vdev) { struct qat_vf_core_device *qat_vdev = @@ -623,9 +626,7 @@ static const struct vfio_device_ops qat_vf_pci_ops = { static struct qat_vf_core_device *qat_vf_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = pci_get_drvdata(pdev); - - return container_of(core_device, struct qat_vf_core_device, core_device); + return pci_get_drvdata(pdev); } static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev) @@ -651,7 +652,7 @@ qat_vf_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (IS_ERR(qat_vdev)) return PTR_ERR(qat_vdev); - pci_set_drvdata(pdev, &qat_vdev->core_device); + pci_set_drvdata(pdev, qat_vdev); ret = vfio_pci_core_register_device(&qat_vdev->core_device); if (ret) goto out_put_device; @@ -686,7 +687,7 @@ MODULE_DEVICE_TABLE(pci, qat_vf_vfio_pci_table); static const struct pci_error_handlers qat_vf_err_handlers = { .reset_done = qat_vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = qat_vf_aer_err_detected, }; static struct pci_driver qat_vf_vfio_pci_driver = { @@ -694,6 +695,7 @@ static struct pci_driver qat_vf_vfio_pci_driver = { .id_table = qat_vf_vfio_pci_table, .probe = qat_vf_vfio_pci_probe, .remove = qat_vf_vfio_pci_remove, + .driver = { .pm = pm_ptr(&qat_vf_pm_ops) }, .err_handler = &qat_vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 830369ff878d..e7c012febe91 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -223,13 +223,44 @@ static const struct pci_device_id vfio_pci_table[] = { MODULE_DEVICE_TABLE(pci, vfio_pci_table); +static int __maybe_unused vfio_pci_pm_suspend(struct device *dev) +{ + struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); + + return vfio_pci_core_runtime_suspend(vdev); +} + +static int __maybe_unused vfio_pci_pm_resume(struct device *dev) +{ + struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); + + return vfio_pci_core_runtime_resume(vdev); +} + +static const struct dev_pm_ops vfio_pci_pm_ops = { + SET_RUNTIME_PM_OPS(vfio_pci_pm_suspend, vfio_pci_pm_resume, NULL) +}; + +static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state) +{ + struct vfio_pci_core_device *vdev = pci_get_drvdata(pdev); + + return vfio_pci_core_aer_err_detected(vdev); +} + +static const struct pci_error_handlers vfio_pci_err_handlers = { + .error_detected = vfio_pci_aer_err_detected, +}; + static struct pci_driver vfio_pci_driver = { .name = "vfio-pci", .id_table = vfio_pci_table, .probe = vfio_pci_probe, .remove = vfio_pci_remove, .sriov_configure = vfio_pci_sriov_configure, - .err_handler = &vfio_pci_core_err_handlers, + .err_handler = &vfio_pci_err_handlers, + .driver = { .pm = pm_ptr(&vfio_pci_pm_ops) }, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index d101c31d0a01..fcbb2da860e2 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -489,10 +489,8 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags, } #ifdef CONFIG_PM -static int vfio_pci_core_runtime_suspend(struct device *dev) +int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); - down_write(&vdev->memory_lock); /* * The user can move the device into D3hot state before invoking @@ -517,11 +515,10 @@ static int vfio_pci_core_runtime_suspend(struct device *dev) return 0; } +EXPORT_SYMBOL_GPL(vfio_pci_core_runtime_suspend); -static int vfio_pci_core_runtime_resume(struct device *dev) +int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); - /* * Resume with a pm_wake_eventfd_ctx signals the eventfd and exit * low power mode. @@ -538,6 +535,7 @@ static int vfio_pci_core_runtime_resume(struct device *dev) return 0; } +EXPORT_SYMBOL_GPL(vfio_pci_core_runtime_resume); #endif /* CONFIG_PM */ /* @@ -577,18 +575,6 @@ static void vfio_pci_core_map_bars(struct vfio_pci_core_device *vdev) } } -/* - * The pci-driver core runtime PM routines always save the device state - * before going into suspended state. If the device is going into low power - * state with only with runtime PM ops, then no explicit handling is needed - * for the devices which have NoSoftRst-. - */ -static const struct dev_pm_ops vfio_pci_core_pm_ops = { - SET_RUNTIME_PM_OPS(vfio_pci_core_runtime_suspend, - vfio_pci_core_runtime_resume, - NULL) -}; - int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) { struct pci_dev *pdev = vdev->pdev; @@ -2222,10 +2208,6 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) struct device *dev = &pdev->dev; int ret; - /* Drivers must set the vfio_pci_core_device to their drvdata */ - if (WARN_ON(vdev != dev_get_drvdata(dev))) - return -EINVAL; - /* Drivers must set a name. Required for sequestering SR-IOV VFs */ if (WARN_ON(!vdev->vdev.ops->name)) return -EINVAL; @@ -2293,7 +2275,6 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) */ vfio_pci_set_power_state(vdev, PCI_D0); - dev->driver->pm = &vfio_pci_core_pm_ops; pm_runtime_allow(dev); if (!vdev->disable_idle_d3) pm_runtime_put(dev); @@ -2334,10 +2315,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) } EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device); -pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, - pci_channel_state_t state) +pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); struct vfio_pci_eventfd *eventfd; rcu_read_lock(); @@ -2420,11 +2399,6 @@ out_unlock: } EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure); -const struct pci_error_handlers vfio_pci_core_err_handlers = { - .error_detected = vfio_pci_core_aer_err_detected, -}; -EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); - static bool vfio_dev_in_groups(struct vfio_device *vdev, struct vfio_pci_group_info *groups) { diff --git a/drivers/vfio/pci/virtio/main.c b/drivers/vfio/pci/virtio/main.c index d2e5cbca13c8..123ff5193812 100644 --- a/drivers/vfio/pci/virtio/main.c +++ b/drivers/vfio/pci/virtio/main.c @@ -18,6 +18,9 @@ #include "common.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(virtiovf, struct virtiovf_pci_core_device, + core_device) + static int virtiovf_pci_open_device(struct vfio_device *core_vdev) { struct virtiovf_pci_core_device *virtvdev = container_of(core_vdev, @@ -174,7 +177,7 @@ static int virtiovf_pci_probe(struct pci_dev *pdev, if (sup_lm) virtiovf_set_migratable(virtvdev); - dev_set_drvdata(&pdev->dev, &virtvdev->core_device); + dev_set_drvdata(&pdev->dev, virtvdev); ret = vfio_pci_core_register_device(&virtvdev->core_device); if (ret) goto out; @@ -211,7 +214,7 @@ static void virtiovf_pci_aer_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers virtiovf_err_handlers = { .reset_done = virtiovf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = virtiovf_aer_err_detected, }; static struct pci_driver virtiovf_pci_driver = { @@ -219,6 +222,7 @@ static struct pci_driver virtiovf_pci_driver = { .id_table = virtiovf_pci_table, .probe = virtiovf_pci_probe, .remove = virtiovf_pci_remove, + .driver = { .pm = pm_ptr(&virtiovf_pm_ops) }, .err_handler = &virtiovf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c index cbff5af385ef..fa6a29d8dfd2 100644 --- a/drivers/vfio/pci/xe/main.c +++ b/drivers/vfio/pci/xe/main.c @@ -38,6 +38,9 @@ struct xe_vfio_pci_core_device { struct xe_vfio_pci_migration_file *migf; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(xe_vfio, struct xe_vfio_pci_core_device, + core_device) + #define xe_vdev_to_dev(xe_vdev) (&(xe_vdev)->core_device.pdev->dev) static void xe_vfio_pci_disable_file(struct xe_vfio_pci_migration_file *migf) @@ -140,7 +143,7 @@ static void xe_vfio_pci_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers xe_vfio_pci_err_handlers = { .reset_prepare = xe_vfio_pci_reset_prepare, .reset_done = xe_vfio_pci_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = xe_vfio_aer_err_detected, }; static int xe_vfio_pci_open_device(struct vfio_device *core_vdev) @@ -550,7 +553,7 @@ static int xe_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i if (IS_ERR(xe_vdev)) return PTR_ERR(xe_vdev); - dev_set_drvdata(&pdev->dev, &xe_vdev->core_device); + dev_set_drvdata(&pdev->dev, xe_vdev); ret = vfio_pci_core_register_device(&xe_vdev->core_device); if (ret) { @@ -586,6 +589,7 @@ static struct pci_driver xe_vfio_pci_driver = { .id_table = xe_vfio_pci_table, .probe = xe_vfio_pci_probe, .remove = xe_vfio_pci_remove, + .driver = { .pm = pm_ptr(&xe_vfio_pm_ops) }, .err_handler = &xe_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 9a1674c152aa..2f3fa2787737 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -8,6 +8,7 @@ * Author: Tom Lyon, pugs@cisco.com */ +#include #include #include #include @@ -16,6 +17,7 @@ #include #include #include +#include #ifndef VFIO_PCI_CORE_H #define VFIO_PCI_CORE_H @@ -168,7 +170,6 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev); void vfio_pci_core_release_dev(struct vfio_device *core_vdev); int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev); void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev); -extern const struct pci_error_handlers vfio_pci_core_err_handlers; int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev, int nr_virtfn); long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, @@ -193,8 +194,6 @@ int vfio_pci_core_match_token_uuid(struct vfio_device *core_vdev, int vfio_pci_core_enable(struct vfio_pci_core_device *vdev); void vfio_pci_core_disable(struct vfio_pci_core_device *vdev); void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev); -pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, - pci_channel_state_t state); ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, void __iomem *io, char __user *buf, loff_t off, size_t count, size_t x_start, @@ -260,4 +259,40 @@ vfio_pci_core_get_iomap(struct vfio_pci_core_device *vdev, unsigned int bar) int vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment, struct phys_vec *phys); +int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev); +int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev); +pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev); + +/* + * Per-driver PM/AER trampolines. driver_data points to the driver's private + * structure, which embeds a struct vfio_pci_core_device. + * + * @name: prefix for _pm_ops and _aer_err_detected + * @drv_type: type of the driver's private structure, including struct + * @member: member containing the struct vfio_pci_core_device + */ +#define VFIO_PCI_CORE_DEFINE_CALLBACKS(name, drv_type, member) \ +static int __maybe_unused name##_pm_suspend(struct device *dev) \ +{ \ + drv_type *drvdata = dev_get_drvdata(dev); \ + \ + return vfio_pci_core_runtime_suspend(&drvdata->member); \ +} \ +static int __maybe_unused name##_pm_resume(struct device *dev) \ +{ \ + drv_type *drvdata = dev_get_drvdata(dev); \ + \ + return vfio_pci_core_runtime_resume(&drvdata->member); \ +} \ +static const struct dev_pm_ops name##_pm_ops = { \ + SET_RUNTIME_PM_OPS(name##_pm_suspend, name##_pm_resume, NULL) \ +}; \ +static pci_ers_result_t name##_aer_err_detected(struct pci_dev *pdev, \ + pci_channel_state_t state) \ +{ \ + drv_type *drvdata = pci_get_drvdata(pdev); \ + \ + return vfio_pci_core_aer_err_detected(&drvdata->member); \ +} + #endif /* VFIO_PCI_CORE_H */ -- cgit v1.2.3 From e1d2d60b1f50296828d3f7ff26c45d1a90f142ef Mon Sep 17 00:00:00 2001 From: Longfang Liu Date: Tue, 13 Oct 2026 15:19:28 +0800 Subject: hisi_acc_vfio_pci: fix NULL dereference in reset_prepare on PF passthrough When a PF is bound to the driver via driver_override and passed through to a VM, its pf_qm stays NULL. The PCI error handler reset_prepare() runs during open_device through pci_try_reset_function(), before the mig_ops gate, and dereferences the NULL pf_qm for the timeout log, crashing the kernel. Move the mig_ops check to the entry of reset_prepare() and aer_reset_done() so non-migration devices skip the QM_RESETTING coordination. The dev local resolved &qm->pdev->dev before the gate, so open code it at the dev_err() call where qm is known valid. Also clear set_reset_flag together with QM_RESETTING in aer_reset_done(); the flag was never cleared before, so a later timed-out reset could release a foreign lock. Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live migration") Fixes: a22099ed7936f ("hisi_acc_vfio_pci: fix VF reset timeout issue") Signed-off-by: Longfang Liu Link: https://lore.kernel.org/r/20261013071929.3988687-2-liulongfang@huawei.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index a5ff79b968d9..2bdd625b0bce 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -1154,14 +1154,16 @@ static void hisi_acc_vf_pci_reset_prepare(struct pci_dev *pdev) { struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_drvdata(pdev); struct hisi_qm *qm = hisi_acc_vdev->pf_qm; - struct device *dev = &qm->pdev->dev; u32 delay = 0; + if (!hisi_acc_vdev->core_device.vdev.mig_ops) + return; + /* All reset requests need to be queued for processing */ while (test_and_set_bit(QM_RESETTING, &qm->misc_ctl)) { msleep(1); if (++delay > QM_RESET_WAIT_TIMEOUT) { - dev_err(dev, "reset prepare failed\n"); + dev_err(&qm->pdev->dev, "reset prepare failed\n"); return; } } @@ -1174,12 +1176,14 @@ static void hisi_acc_vf_pci_aer_reset_done(struct pci_dev *pdev) struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_drvdata(pdev); struct hisi_qm *qm = hisi_acc_vdev->pf_qm; - if (hisi_acc_vdev->set_reset_flag) - clear_bit(QM_RESETTING, &qm->misc_ctl); - if (!hisi_acc_vdev->core_device.vdev.mig_ops) return; + if (hisi_acc_vdev->set_reset_flag) { + clear_bit(QM_RESETTING, &qm->misc_ctl); + hisi_acc_vdev->set_reset_flag = false; + } + mutex_lock(&hisi_acc_vdev->state_mutex); hisi_acc_vf_reset(hisi_acc_vdev); mutex_unlock(&hisi_acc_vdev->state_mutex); -- cgit v1.2.3 From a80597900b8a0f1a9e87360164695f6a0a10556c Mon Sep 17 00:00:00 2001 From: Longfang Liu Date: Tue, 13 Oct 2026 15:19:29 +0800 Subject: hisi_acc_vfio_pci: reject live migration on 64KB page with QM_HW_V3 hardware On QM_HW_V3 (VF_CTRL mode) hardware with 64KB pages, the functional region and migration registers share one BAR2 physical page with no isolation, which leaks migration registers to the guest and causes guest kernel calltrace under the KVM emulated device scheme. The previous fix rejected the device in hisi_acc_vf_qm_init() during open_device, returning -EINVAL and breaking passthrough entirely. Move the drv_mode check into a helper called from probe() so that a VF_CTRL device on a 64KB page is bound to the generic ops, keeping passthrough working while disabling live migration. Fixes: b0eed085903e ("hisi_acc_vfio_pci: Add support for VFIO live migration") Signed-off-by: Longfang Liu Link: https://lore.kernel.org/r/20261013071929.3988687-3-liulongfang@huawei.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 32 ++++++++++++++++---------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index 2bdd625b0bce..c82c59246a3f 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -1195,13 +1195,6 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev) struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; struct pci_dev *vf_dev = vdev->pdev; - u32 val; - - val = readl(pf_qm->io_base + QM_MIG_REGION_SEL); - if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN)) - hisi_acc_vdev->drv_mode = HW_ACC_MIG_PF_CTRL; - else - hisi_acc_vdev->drv_mode = HW_ACC_MIG_VF_CTRL; if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_PF_CTRL) { /* @@ -1665,21 +1658,34 @@ static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device *hisi_acc_vd hisi_acc_vdev->debug_migf = NULL; } +static enum hw_drv_mode hisi_acc_vf_get_drv_mode(struct hisi_qm *pf_qm) +{ + u32 val; + + val = readl(pf_qm->io_base + QM_MIG_REGION_SEL); + if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN)) + return HW_ACC_MIG_PF_CTRL; + + return HW_ACC_MIG_VF_CTRL; +} + static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct hisi_acc_vf_core_device *hisi_acc_vdev; const struct vfio_device_ops *ops = &hisi_acc_vfio_pci_ops; + enum hw_drv_mode drv_mode = HW_ACC_MIG_VF_CTRL; + resource_size_t func_len; struct hisi_qm *pf_qm; - int vf_id; int ret; pf_qm = hisi_acc_get_pf_qm(pdev); if (pf_qm && pf_qm->ver >= QM_HW_V3) { - vf_id = pci_iov_vf_id(pdev); - if (vf_id >= 0) - ops = &hisi_acc_vfio_pci_migrn_ops; + func_len = pci_resource_len(pdev, VFIO_PCI_BAR2_REGION_INDEX) >> 1; + drv_mode = hisi_acc_vf_get_drv_mode(pf_qm); + if (drv_mode == HW_ACC_MIG_VF_CTRL && func_len < PAGE_SIZE) + pci_warn(pdev, "migration not supported on 64KB pages with QM_HW_V3\n"); else - pci_warn(pdev, "migration support failed, continue with generic interface\n"); + ops = &hisi_acc_vfio_pci_migrn_ops; } hisi_acc_vdev = vfio_alloc_device(hisi_acc_vf_core_device, @@ -1687,6 +1693,8 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device if (IS_ERR(hisi_acc_vdev)) return PTR_ERR(hisi_acc_vdev); + hisi_acc_vdev->drv_mode = drv_mode; + dev_set_drvdata(&pdev->dev, hisi_acc_vdev); ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); if (ret) -- cgit v1.2.3 From 60519dd563d91369bfae59f37895459a542eff66 Mon Sep 17 00:00:00 2001 From: Satyanarayana K V P Date: Wed, 14 Oct 2026 18:48:50 +0530 Subject: vfio/xe: Add CRI and NVL device IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Crescent Island (CRI), Nova Lake-S (NVL-S) and Nova Lake-P (NVL-P) support SR-IOV in the Xe driver. Add their PCI IDs to the xe-vfio-pci device table, so that the generated vfio_pci module aliases allow userspace to select this variant driver for their VFs. Also reordered the table entries. Signed-off-by: Satyanarayana K V P Cc: Michał Winiarski Cc: Jason Gunthorpe Cc: Yishai Hadas Cc: Shameer Kolothum Cc: Kevin Tian Cc: Alex Williamson Cc: kvm@vger.kernel.org Cc: intel-xe@lists.freedesktop.org Cc: linux-kernel@vger.kernel.org Reviewed-by: Michał Winiarski Link: https://lore.kernel.org/r/20261014131849.157593-2-satyanarayana.k.v.p@intel.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/xe/main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c index fa6a29d8dfd2..7dab3e024aa8 100644 --- a/drivers/vfio/pci/xe/main.c +++ b/drivers/vfio/pci/xe/main.c @@ -577,9 +577,12 @@ static void xe_vfio_pci_remove(struct pci_dev *pdev) } static const struct pci_device_id xe_vfio_pci_table[] = { + INTEL_BMG_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_CRI_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_NVLS_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_NVLP_IDS(INTEL_PCI_VFIO_DEVICE), INTEL_PTL_IDS(INTEL_PCI_VFIO_DEVICE), INTEL_WCL_IDS(INTEL_PCI_VFIO_DEVICE), - INTEL_BMG_IDS(INTEL_PCI_VFIO_DEVICE), {} }; MODULE_DEVICE_TABLE(pci, xe_vfio_pci_table); -- cgit v1.2.3 From 9c48b994b66d7b585698f9aaee8ff559a7755c0a Mon Sep 17 00:00:00 2001 From: Andrea Parri Date: Thu, 15 Oct 2026 10:19:34 +0200 Subject: vfio/nvgrace-gpu: Use a core-only open_device in fallback mode When the ACPI memory properties are missing or the C2C link is down, the device is exposed through nvgrace_gpu_pci_core_ops. That table opens through nvgrace_gpu_open_device() but closes through vfio_pci_core_close_device(), so open and close are not symmetric. nvgrace_gpu_open_device() registers nvdev->usemem for poison handling unconditionally. In fallback mode usemem.memlength is zero, and register_pfn_address_space() inserts a node covering [0, ULONG_MAX]. Nothing removes it: a second open fails with -EBUSY, and once the device is unbound the node outlives the nvdev that contains it. The next open after a rebind reports: BUG: KASAN: slab-use-after-free in interval_tree_iter_first+0x209/0x310 Read of size 8 at addr ffff88800894aa30 by task nvgrace_repro/203 Call Trace: dump_stack_lvl+0x5d/0x80 print_report+0x153/0x4c6 kasan_report+0xe4/0x1a0 interval_tree_iter_first+0x209/0x310 register_pfn_address_space+0x97/0x100 nvgrace_gpu_open_device+0x21c/0x730 vfio_df_open+0x213/0x520 vfio_df_ioctl_bind_iommufd+0x3e8/0x6e0 vfio_device_fops_unl_ioctl+0x782/0x1570 __x64_sys_ioctl+0x13c/0x1c0 do_syscall_64+0x106/0x630 entry_SYSCALL_64_after_hwframe+0x77/0x7f Allocated by task 188: ... _vfio_alloc_device+0x3a/0x500 nvgrace_gpu_probe+0xcb/0x660 local_pci_probe+0xcf/0x170 ... Freed by task 188: ... kfree+0x20d/0x600 device_release+0xcf/0x250 kobject_put+0x152/0x2f0 pci_device_remove+0xb0/0x1c0 device_release_driver_internal+0x38d/0x540 unbind_store+0xde/0x100 ... Give the fallback table its own open_device that only calls vfio_pci_core_enable() and vfio_pci_core_finish_enable(), the counterpart of vfio_pci_core_close_device(). Fallback mode then never touches the nvgrace memory regions, as Alex suggested. This should be safe because the fallback table uses the vfio-pci core read, write and mmap handlers, so nothing there uses the state nvgrace_gpu_open_device() sets up. The one visible difference is that a fallback open no longer fails when BAR0 cannot be mapped, which matches vfio-pci. Fixes: e5f19b619fa0 ("vfio/nvgrace-gpu: register device memory for poison handling") Cc: stable@vger.kernel.org Suggested-by: Alex Williamson Assisted-by: LLM Signed-off-by: Andrea Parri Reviewed-by: Ankit Agrawal Link: https://lore.kernel.org/r/20261015081937.405320-2-parri.andrea@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/nvgrace-gpu/main.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 087850b92092..846a1e354c26 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1086,6 +1086,21 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { .detach_ioas = vfio_iommufd_physical_detach_ioas, }; +static int nvgrace_gpu_core_open_device(struct vfio_device *core_vdev) +{ + struct vfio_pci_core_device *vdev = + container_of(core_vdev, struct vfio_pci_core_device, vdev); + int ret; + + ret = vfio_pci_core_enable(vdev); + if (ret) + return ret; + + vfio_pci_core_finish_enable(vdev); + + return 0; +} + static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = { .get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys, }; @@ -1094,7 +1109,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = { .name = "nvgrace-gpu-vfio-pci-core", .init = vfio_pci_core_init_dev, .release = vfio_pci_core_release_dev, - .open_device = nvgrace_gpu_open_device, + .open_device = nvgrace_gpu_core_open_device, .close_device = vfio_pci_core_close_device, .ioctl = vfio_pci_core_ioctl, .get_region_info_caps = vfio_pci_ioctl_get_region_info, -- cgit v1.2.3 From 076d7e989cb69f6ded7d60f2a080eb909930061b Mon Sep 17 00:00:00 2001 From: Andrea Parri Date: Thu, 15 Oct 2026 10:19:35 +0200 Subject: vfio/nvgrace-gpu: Drop the usemem length check in open_device nvgrace_gpu_open_device() sets up the fake BAR registers and remap_lock only when usemem.memlength is nonzero. The check exists for fallback mode, which used to share this open path. Fallback mode now has its own open_device, so only devices with ACPI memory properties reach this one. For those devices usemem.memlength is never zero: an empty region from nvgrace_gpu_fetch_memory_property() selects fallback mode, and probe fails when nvgrace_gpu_init_nvdev_struct() rounds usemem down to nothing. Do the setup unconditionally. Assisted-by: LLM Signed-off-by: Andrea Parri Reviewed-by: Ankit Agrawal Link: https://lore.kernel.org/r/20261015081937.405320-3-parri.andrea@gmail.com Signed-off-by: Alex Williamson --- drivers/vfio/pci/nvgrace-gpu/main.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 846a1e354c26..5ddec340b73b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -188,10 +188,8 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev) if (ret) return ret; - if (nvdev->usemem.memlength) { - nvgrace_gpu_init_fake_bar_emu_regs(core_vdev); - mutex_init(&nvdev->remap_lock); - } + nvgrace_gpu_init_fake_bar_emu_regs(core_vdev); + mutex_init(&nvdev->remap_lock); /* * GPU readiness is checked by reading the BAR0 registers. -- cgit v1.2.3