diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-23 14:22:52 -1000 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-23 14:22:52 -1000 |
| commit | 26344c0dd32af8790d8474d233619db825441138 (patch) | |
| tree | e0965df49f27418a3adee33643915220d042a618 | |
| parent | 9f2516f45b7cfca237f4d8e1c91c1f307e295dc5 (diff) | |
| parent | 076d7e989cb69f6ded7d60f2a080eb909930061b (diff) | |
Merge tag 'vfio-v7.4-rc1' of https://github.com/awilliam/linux-vfio
Pull VFIO updates from Alex Williamson:
- Fix missing selftest <libgen.h> include for basename() to support
non-glibc builds (Hisam Mehboob)
- Add vfio selftest documentation covering the libvfio helper library
and guidance for developers to aid in adding and understanding the
framework (David Matlack)
- Fix vfio cdev open path to prevent a second open from clobbering the
vf_token or clearing the KVM reference. Add a selftest to avoid
future recurrences (Alex Williamson)
- Extend vfio selftest mix_and_match timeouts, in particular to account
for systems where generated faults can induce heavy SMM load,
delaying internal timeouts. Also extend timeout for execution via
runner to account for various tests exceeding default 45s timeout
(Alex Williamson)
- Add tracking for non-reserved mappings into a vfio_dma object in
order to optimize the tear-down paths where per-page tests can be
entirely skipped. This resolves a soft lockup for large memory GPUs
(Samuel Crossley)
- Add vfio selftest driver for mlx5, providing another target for
exercising DMA and interrupts against a physical device (Jason
Gunthorpe)
- Hardening against read-only region promotion, preserving vm_pgoff for
unmap_mapping_range(), and reject non-shared MMIO mmaps as necessary
through vfio-platform, vfio-fsl-mc, and vfio-cdx. These prevent
future issues, though none are currently reachable (Abdifatah Suruur)
- Fix 8-byte ioeventfd support, removing a static error test that was
erroneously left in place when generalizing 8-byte io accesses (Li
RongQing)
- Fix missed RCU drain on module exit (Jiale Yao)
- Fix off-by-one boundary check in migration support within the mlx5
vfio-pci variant driver (Seongjun Hong)
- Remove drvdata usage convention across vfio-pci-core, allowing
variant drivers to freely create their own conventions and enabling a
future Rust-based variant driver. A DRM topic branch is also merged
in support of this effort, modifying the vga arbiter API to more
cleanly remove the drvdata dependency (Simon Song)
- Fix hisi_acc vfio-pci variant driver to avoid NULL pointer accesses
when bound to unsupported devices and also disabling migration
support when running on 64K host page size due to BAR mapping
constraints (Longfang Liu)
- Add device IDs for Crescent Island and Nova Lake devices to the Xe
vfio-pci variant driver (Satyanarayana K V P)
- Fix fallback mode in nvgrace-gpu variant driver to use symmetric open
and close handlers to avoid reported use-after-free issue. Remove the
resulting unnecessary test in the now single-purpose open function
(Andrea Parri)
* tag 'vfio-v7.4-rc1' of https://github.com/awilliam/linux-vfio: (35 commits)
vfio/nvgrace-gpu: Drop the usemem length check in open_device
vfio/nvgrace-gpu: Use a core-only open_device in fallback mode
vfio/xe: Add CRI and NVL device IDs
hisi_acc_vfio_pci: reject live migration on 64KB page with QM_HW_V3 hardware
hisi_acc_vfio_pci: fix NULL dereference in reset_prepare on PF passthrough
vfio/pci: Remove the core dependency on driver data
vfio/mlx5: Fix boundary check in set_report_output()
vfio/pci: Drain eventfd RCU callbacks on module exit
PCI/VGA: pass a data pointer to be used in the decode callback (v2)
vfio/pci: Restore 8-byte ioeventfd support
vfio/cdx: reject non-shared MMIO mmaps
vfio/cdx: keep logical vm_pgoff in MMIO region mmap
vfio/fsl-mc: keep logical vm_pgoff in MMIO region mmap
vfio/platform: keep logical vm_pgoff in MMIO region mmap
vfio/cdx: prevent read-only region mappings from becoming writable
vfio/fsl-mc: prevent read-only region mappings from becoming writable
vfio/platform: prevent read-only region mappings from becoming writable
vfio: selftests: Verify a failed second open preserves the vf_token
vfio: selftests: mlx5 driver - add send_msi support
vfio: selftests: Add mlx5 driver - data path and memcpy ops
...
61 files changed, 3080 insertions, 420 deletions
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 26b7638a327d..aaf5d76ed074 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -47,6 +47,7 @@ of interest to most developers working on device drivers. vfio-mediated-device vfio vfio-pci-device-specific-driver-acceptance + vfio-selftests Bus-level documentation ======================= diff --git a/Documentation/driver-api/vfio-selftests.rst b/Documentation/driver-api/vfio-selftests.rst new file mode 100644 index 000000000000..9d39a22519d6 --- /dev/null +++ b/Documentation/driver-api/vfio-selftests.rst @@ -0,0 +1,252 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +VFIO Selftests +============== + +VFIO selftests are built on top of the kernel's selftests framework and are +located in ``tools/testing/selftests/vfio``. + +VFIO selftests enable kernel developers to write and run tests that take the +form of userspace programs that interact with VFIO and IOMMUFD uAPIs. VFIO +selftests can be used to write functional tests for new features, regression +tests for bugs, and performance tests for optimizations. + +These tests are designed to interact with real PCI devices, i.e. they do not +rely on mocking out or faking any behavior in the kernel. This allows the tests +to exercise not only VFIO but also IOMMUFD, the IOMMU driver, interrupt +remapping, IRQ handling, etc. + +Running Tests +============= + +Running VFIO selftests requires a device bound to the ``vfio-pci`` driver. +There are scripts in ``tools/testing/selftests/vfio/scripts/`` to help with +setting one up. + +Picking a Device +---------------- + +Some VFIO selftests require a device with a supported driver implementation in +the selftests library (see `Driver Framework`_), while other tests (such as +basic PCI configuration space or reset tests) can use any arbitrary PCI device. + +This section will be expanded in the future to guide developers on how to find +a device with a supported driver. In the meantime, the selftests library +includes driver support for the following devices: + +- Intel IGB Gigabit Ethernet controllers +- Intel IOAT (I/O Acceleration Technology) DMA engines +- Intel DSA (Data Streaming Accelerator) +- NVIDIA Falcon engines (found in many NVIDIA GPUs, e.g., GTX 1080 or RTX 2080) + +Note that using Intel DSA devices requires setting the module parameter +``vfio_pci.disable_denylist=Y`` to allow ``vfio-pci`` to bind to DSA devices. + +Setting up Devices +------------------ + +.. code-block:: sh + + tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0 + +The script handles unbinding the device from its current driver and binding it +to ``vfio-pci``. Metadata about this device is stored in +``/tmp/vfio-selftests-devices`` so that the device can be cleaned up later +(bound back to its original driver). + +Execution +--------- + +VFIO selftests expect the PCI device BDF string as their final command-line +argument. e.g. + +.. code-block:: sh + + tools/testing/selftests/vfio/vfio_pci_device_test 0000:01:00.0 + +Alternatively, the PCI device BDF can be passed via the ``VFIO_SELFTESTS_BDF`` +environment variable, which can be useful when running all tests: + +.. code-block:: sh + + export VFIO_SELFTESTS_BDF=0000:01:00.0 + make -C tools/testing/selftests TARGETS=vfio install + tools/testing/selftests/kselftest_install/run_kselftest.sh -c vfio + +Cleanup +------- + +.. code-block:: sh + + tools/testing/selftests/vfio/scripts/cleanup.sh + +This script handles unbinding all previously setup devices from ``vfio-pci`` +and binding them back to their original driver. Specific devices can be cleaned +up by passing their BDF string to the cleanup script. + +Writing Tests +============= + +VFIO selftests leverage the ``kselftest_harness.h`` macros to structure setup +and teardown scenarios. A basic test involves: + +1. Declaring a test FIXTURE containing the device and IOMMU state. +2. Generating test variants that span the different IOMMU modes utilizing + ``FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES()``. +3. Initializing the device and binding the default driver ops. + +For a complete example of writing a VFIO selftest, refer to +``tools/testing/selftests/vfio/vfio_pci_device_test.c``. + +VFIO Selftests Library (libvfio) +================================ + +The selftests directory contains a helper library compiled from the ``lib/`` +subdirectory. It provides structures and helper functions to handle standard +operations like opening VFIO devices, creating VFIO containers, and interacting +with VFIO and IOMMUFD. + +All VFIO selftests have libvfio linked in by default. Tests can access the +library by including ``<libvfio.h>``. + +Core Objects +------------ + +The VFIO selftests framework revolves around two distinct core objects: +``struct iommu`` and ``struct vfio_pci_device``. It is important to understand +the distinction and relationship between them: + +- ``struct iommu`` represents an IOMMU domain and the API used to program it + (e.g. legacy VFIO Type1 vs IOMMUFD). It is responsible for managing the I/O + address space, handling DMA mappings, and translating between host virtual + addresses (HVA) and IO virtual addresses (IOVA). + +- ``struct vfio_pci_device`` represents a single PCI device. It encapsulates + the VFIO device file descriptor, handles access to the PCI configuration + space, manages memory-mapped BARs, and drives device interrupts. + +A typical test initializes an IOMMU instance first, then passes it when +initializing one or more PCI devices, linking the devices to the shared IOMMU +domain. + +``struct iommu`` +^^^^^^^^^^^^^^^^ + +The ``struct iommu`` abstracts away the complexity of managing VFIO containers, +IOMMU groups, and IOMMUFD contexts. A test allocates its IOMMU handle using +``iommu_init()`` passing one of the supported IOMMU modes: + +.. code-block:: c + + struct iommu *iommu = iommu_init(MODE_IOMMUFD); + ... + iommu_cleanup(iommu); + +The selftest framework is designed to work uniformly across all standard kernel +IOMMU APIs. The defined modes (in ``iommu.h``) include: + +- ``MODE_VFIO_TYPE1_IOMMU`` +- ``MODE_VFIO_TYPE1V2_IOMMU`` +- ``MODE_IOMMUFD_COMPAT_TYPE1`` +- ``MODE_IOMMUFD_COMPAT_TYPE1V2`` +- ``MODE_IOMMUFD`` + +VFIO selftests can replicate their test cases across all possible IOMMU modes +using FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES() to ensure kernel IOMMU APIs remain +compatible. + +``struct vfio_pci_device`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A PCI device is represented as a ``struct vfio_pci_device``. The device +representation is initialized using ``vfio_pci_device_init()``, which takes a +target BDF string, opens the device in VFIO, and binds it to an existing +``struct iommu`` instance: + +.. code-block:: c + + struct vfio_pci_device *device = vfio_pci_device_init(device_bdf, iommu); + ... + vfio_pci_device_cleanup(device); + +``struct iova_allocator`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To facilitate DMA mappings without hardcoding IOVAs (IO virtual addresses) that +might conflict with reserved platform address ranges, the library provides a +``struct iova_allocator``. + +The IOVA allocator queries the underlying IOMMU (via ``struct iommu``) for its +valid target IOVA ranges. Tests can safely carve out unique contiguous chunks +of IOVA space for device DMA by repeatedly calling: + +.. code-block:: c + + struct iova_allocator *allocator = iova_allocator_init(iommu); + iova_t addr = iova_allocator_alloc(allocator, size); + +Driver Framework +---------------- + +The primary goal of VFIO selftests is to verify the correctness of the kernel +code handling VFIO devices, including VFIO core, IOMMUFD, IOMMU drivers, page +pinning, DMA mapping/unmapping, interrupt remapping, and IRQ delivery. + +Because different physical devices have vastly different programming interfaces +for triggering work, the selftests library implements a standard driver +framework. This framework abstracts diverse hardware endpoints (such as Intel +IGB, IOAT, or DSA) behind a uniform interface, allowing tests to provoke a +broad range of devices to generate DMA operations and interrupts in a +standardized manner. + +By exercising DMA and interrupts with real endpoints, tests can thoroughly +stress and validate the underlying kernel paths, such as IOMMU page table +management, dirty page tracking, Device TLB and IOTLB invalidations, and +interrupt delivery under authentic hardware conditions. + +All endpoint driver implementations populate a common set of operations defined +in ``struct vfio_pci_driver_ops``: + +- ``probe()``: Validate whether the driver supports the current PCI device + (e.g. matching vendor/device IDs). +- ``init()``: Set up specific control registers or ring-buffer data structures. +- ``remove()``: Clean up structures before destruction. +- ``memcpy_start()``: Initiate DMA copies via the device. +- ``memcpy_wait()``: Poll and wait until previously initiated copies complete + safely. +- ``send_msi()``: Provoke the device to signal an MSI interrupt. + +Tests are required to set up a dedicated DMA region for the driver to use +within ``struct vfio_pci_driver`` to serve as the driver's main memory +allocation region (e.g. used for in-memory descriptor rings and hardware +buffers). Tests must provision and map this DMA region before completing driver +initialization. + +For a complete example of interacting with the driver framework from a test +perspective, refer to ``tools/testing/selftests/vfio/vfio_pci_driver_test.c``. + +When contributing a new driver implementation to the selftests library, the +primary acceptance criteria is that ``vfio_pci_driver_test`` passes using the +new driver and target device. + +Integration with KVM Selftests +------------------------------ + +libvfio is not strictly limited to VFIO selftests, it is also leveraged within +KVM selftests (``tools/testing/selftests/kvm/``). KVM selftests build their own +copy of libvfio and link against it, enabling libvfio to be used in KVM +selftests. + +This integration enables testing the interaction between KVM and VFIO and +IOMMUFD. For example, testing the end-to-end delivery of device interrupts into +running vCPUs. + +Contribution Guidelines +======================= + +- Prefer the shortlog prefix ``vfio: selftests: ...`` for commits that modify + files in ``tools/testing/selftests/vfio/``. +- Follow the Coding Style, Comments, Changelogs, and Function References + guidelines from the + :doc:`KVM x86 maintainers handbook </process/maintainer-kvm-x86>`. diff --git a/MAINTAINERS b/MAINTAINERS index 2790a256b406..cdd7434fdb6f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -28974,6 +28974,7 @@ VFIO SELFTESTS M: David Matlack <dmatlack@google.com> L: kvm@vger.kernel.org S: Maintained +F: Documentation/driver-api/vfio-selftests.rst F: tools/testing/selftests/vfio/ VFIO VIRTIO PCI DRIVER diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 9269e780feb7..ef9ff30c0a7b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -1429,16 +1429,15 @@ bool amdgpu_device_should_use_aspm(struct amdgpu_device *adev) /** * amdgpu_device_vga_set_decode - enable/disable vga decode * - * @pdev: PCI device pointer + * @data: driver private data (adev) * @state: enable/disable vga decode * * Enable/disable vga decode (all asics). * Returns VGA resource flags. */ -static unsigned int amdgpu_device_vga_set_decode(struct pci_dev *pdev, - bool state) +static unsigned int amdgpu_device_vga_set_decode(void *data, bool state) { - struct amdgpu_device *adev = drm_to_adev(pci_get_drvdata(pdev)); + struct amdgpu_device *adev = data; amdgpu_asic_set_vga_state(adev, state); if (state) @@ -4199,7 +4198,7 @@ fence_driver_init: * ignore it */ if ((adev->pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) - vga_client_register(adev->pdev, amdgpu_device_vga_set_decode); + vga_client_register(adev->pdev, amdgpu_device_vga_set_decode, adev); px = amdgpu_device_supports_px(adev); diff --git a/drivers/gpu/drm/i915/display/intel_vga.c b/drivers/gpu/drm/i915/display/intel_vga.c index 4966fd7eea92..4c292fae7b92 100644 --- a/drivers/gpu/drm/i915/display/intel_vga.c +++ b/drivers/gpu/drm/i915/display/intel_vga.c @@ -283,9 +283,10 @@ reset_vgacntr: intel_de_posting_read(display, vga_reg); } -static unsigned int intel_vga_set_decode(struct pci_dev *pdev, bool enable_decode) +static unsigned int intel_vga_set_decode(void *data, bool enable_decode) { - struct intel_display *display = to_intel_display(pdev); + struct intel_display *display = data; + struct pci_dev *pdev = to_pci_dev(display->drm->dev); unsigned int decodes = VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; drm_dbg_kms(display->drm, "%s VGA decode due to VGA arbitration\n", @@ -331,7 +332,7 @@ void intel_vga_register(struct intel_display *display) * then we do not take part in VGA arbitration and the * vga_client_register() fails with -ENODEV. */ - ret = vga_client_register(pdev, intel_vga_set_decode); + ret = vga_client_register(pdev, intel_vga_set_decode, display); drm_WARN_ON(display->drm, ret && ret != -ENODEV); } diff --git a/drivers/gpu/drm/loongson/lsdc_drv.c b/drivers/gpu/drm/loongson/lsdc_drv.c index 3fd828d23271..5a13df31eb90 100644 --- a/drivers/gpu/drm/loongson/lsdc_drv.c +++ b/drivers/gpu/drm/loongson/lsdc_drv.c @@ -254,7 +254,7 @@ lsdc_create_device(struct pci_dev *pdev, /* For multiple GPU driver instance co-exixt in the system */ -static unsigned int lsdc_vga_set_decode(struct pci_dev *pdev, bool state) +static unsigned int lsdc_vga_set_decode(void *data, bool state) { return VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; } @@ -291,7 +291,7 @@ static int lsdc_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) pci_set_drvdata(pdev, ddev); - vga_client_register(pdev, lsdc_vga_set_decode); + vga_client_register(pdev, lsdc_vga_set_decode, NULL); drmm_kms_helper_poll_init(ddev); diff --git a/drivers/gpu/drm/nouveau/nouveau_vga.c b/drivers/gpu/drm/nouveau/nouveau_vga.c index a6c375a24154..17464c846ffd 100644 --- a/drivers/gpu/drm/nouveau/nouveau_vga.c +++ b/drivers/gpu/drm/nouveau/nouveau_vga.c @@ -9,9 +9,9 @@ #include "nouveau_vga.h" static unsigned int -nouveau_vga_set_decode(struct pci_dev *pdev, bool state) +nouveau_vga_set_decode(void *data, bool state) { - struct nouveau_drm *drm = pci_get_drvdata(pdev); + struct nouveau_drm *drm = data; struct nvif_object *device = &drm->client.device.object; if (drm->client.device.info.family == NV_DEVICE_INFO_V0_CURIE && @@ -95,7 +95,7 @@ nouveau_vga_init(struct nouveau_drm *drm) return; pdev = to_pci_dev(dev->dev); - vga_client_register(pdev, nouveau_vga_set_decode); + vga_client_register(pdev, nouveau_vga_set_decode, drm); /* don't register Thunderbolt eGPU with vga_switcheroo */ if (pci_is_thunderbolt_attached(pdev)) diff --git a/drivers/gpu/drm/radeon/radeon_device.c b/drivers/gpu/drm/radeon/radeon_device.c index 705c012fcf9e..d21fdecd163f 100644 --- a/drivers/gpu/drm/radeon/radeon_device.c +++ b/drivers/gpu/drm/radeon/radeon_device.c @@ -1070,16 +1070,15 @@ void radeon_combios_fini(struct radeon_device *rdev) /** * radeon_vga_set_decode - enable/disable vga decode * - * @pdev: PCI device + * @data: driver private data (rdev) * @state: enable/disable vga decode * * Enable/disable vga decode (all asics). * Returns VGA resource flags. */ -static unsigned int radeon_vga_set_decode(struct pci_dev *pdev, bool state) +static unsigned int radeon_vga_set_decode(void *data, bool state) { - struct drm_device *dev = pci_get_drvdata(pdev); - struct radeon_device *rdev = dev->dev_private; + struct radeon_device *rdev = data; radeon_vga_set_state(rdev, state); if (state) return VGA_RSRC_LEGACY_IO | VGA_RSRC_LEGACY_MEM | @@ -1424,7 +1423,7 @@ int radeon_device_init(struct radeon_device *rdev, /* if we have > 1 VGA cards, then disable the radeon VGA resources */ /* this will fail for cards that aren't VGA class devices, just * ignore it */ - vga_client_register(rdev->pdev, radeon_vga_set_decode); + vga_client_register(rdev->pdev, radeon_vga_set_decode, rdev); if (rdev->flags & RADEON_IS_PX) runtime = true; diff --git a/drivers/pci/vgaarb.c b/drivers/pci/vgaarb.c index 3de05aee7859..3e58b2c05c85 100644 --- a/drivers/pci/vgaarb.c +++ b/drivers/pci/vgaarb.c @@ -51,7 +51,8 @@ struct vga_device { unsigned int mem_norm_cnt; /* normal MEM count */ bool bridge_has_one_vga; bool is_firmware_default; /* device selected by firmware */ - unsigned int (*set_decode)(struct pci_dev *pdev, bool decode); + unsigned int (*set_decode)(void *data, bool decode); + void *set_decode_data; }; static LIST_HEAD(vga_list); @@ -935,6 +936,7 @@ EXPORT_SYMBOL(vga_set_legacy_decoding); * vga_client_register - register or unregister a VGA arbitration client * @pdev: PCI device of the VGA client * @set_decode: VGA decode change callback + * @data: private data to pass to callback * * Clients have two callback mechanisms they can use. * @@ -956,15 +958,18 @@ EXPORT_SYMBOL(vga_set_legacy_decoding); * Returns: 0 on success, -ENODEV on failure */ int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool decode)) + unsigned int (*set_decode)(void *data, bool decode), + void *data) { unsigned long flags; struct vga_device *vgadev; spin_lock_irqsave(&vga_lock, flags); vgadev = vgadev_find(pdev); - if (vgadev) + if (vgadev) { vgadev->set_decode = set_decode; + vgadev->set_decode_data = data; + } spin_unlock_irqrestore(&vga_lock, flags); if (!vgadev) return -ENODEV; @@ -1464,7 +1469,7 @@ static void vga_arbiter_notify_clients(void) spin_lock_irqsave(&vga_lock, flags); list_for_each_entry(vgadev, &vga_list, list) { if (vgadev->set_decode) { - new_decodes = vgadev->set_decode(vgadev->pdev, + new_decodes = vgadev->set_decode(vgadev->set_decode_data, new_state); vga_update_device_decodes(vgadev, new_decodes); } diff --git a/drivers/vfio/cdx/main.c b/drivers/vfio/cdx/main.c index b31ed4be7bdc..28dc8c825600 100644 --- a/drivers/vfio/cdx/main.c +++ b/drivers/vfio/cdx/main.c @@ -254,10 +254,10 @@ static int vfio_cdx_mmap_mmio(struct vfio_cdx_region region, if (base + size > region.size) return -EINVAL; - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; vma->vm_page_prot = pgprot_device(vma->vm_page_prot); - return io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return io_remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, size, vma->vm_page_prot); } @@ -271,6 +271,9 @@ static int vfio_cdx_mmap(struct vfio_device *core_vdev, index = vma->vm_pgoff >> (VFIO_CDX_OFFSET_SHIFT - PAGE_SHIFT); + if (!(vma->vm_flags & VM_SHARED)) + return -EINVAL; + if (index >= cdx_dev->res_count) return -EINVAL; @@ -281,9 +284,13 @@ static int vfio_cdx_mmap(struct vfio_device *core_vdev, (vma->vm_flags & VM_READ)) return -EPERM; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) && - (vma->vm_flags & VM_WRITE)) - return -EPERM; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EPERM; + + vm_flags_clear(vma, VM_MAYWRITE); + } return vfio_cdx_mmap_mmio(vdev->regions[index], vma); } diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c index c1c26d7a9aec..92627295182e 100644 --- a/drivers/vfio/device_cdev.c +++ b/drivers/vfio/device_cdev.c @@ -130,6 +130,18 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, goto out_unlock; } + /* + * The cdev path allows only a single open. Reject a second open here, + * before the VF token and device->kvm updates below would clobber the + * current opener's state on a bind that cannot complete. Return -EBUSY + * rather than -EINVAL since a delayed release of the prior opener can + * make this transient. + */ + if (device->open_count) { + ret = -EBUSY; + goto out_unlock; + } + ret = vfio_df_check_token(device, &bind); if (ret) goto out_unlock; diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index b4c3958201b2..15b05b8c0c90 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -369,9 +369,8 @@ static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region, if (!region_cacheable) vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; - - return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, size, vma->vm_page_prot); } @@ -403,9 +402,13 @@ static int vfio_fsl_mc_mmap(struct vfio_device *core_vdev, && (vma->vm_flags & VM_READ)) return -EINVAL; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) - && (vma->vm_flags & VM_WRITE)) - return -EINVAL; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EINVAL; + + vm_flags_clear(vma, VM_MAYWRITE); + } vma->vm_private_data = mc_dev; diff --git a/drivers/vfio/mdev/mdev_core.c b/drivers/vfio/mdev/mdev_core.c index 3b4aa0632fcc..34337292a9b1 100644 --- a/drivers/vfio/mdev/mdev_core.c +++ b/drivers/vfio/mdev/mdev_core.c @@ -50,7 +50,7 @@ static int mdev_device_remove_cb(struct device *dev, void *data) * @types: Array of supported mdev types * @nr_types: Number of entries in @types * - * Registers the @parent stucture as a parent for mdev types and thus mdev + * Registers the @parent structure as a parent for mdev types and thus mdev * devices. The caller needs to hold a reference on @dev that must not be * released until after the call to mdev_unregister_parent(). * diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index 86362ec424a5..c82c59246a3f 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -16,6 +16,9 @@ #include "hisi_acc_vfio_pci.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(hisi_acc_vf, struct hisi_acc_vf_core_device, + core_device) + /* Return 0 on VM acc device ready, -ETIMEDOUT hardware timeout */ static int qm_wait_dev_not_ready(struct hisi_qm *qm) { @@ -588,10 +591,7 @@ static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, static struct hisi_acc_vf_core_device *hisi_acc_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct hisi_acc_vf_core_device, - core_device); + return pci_get_drvdata(pdev); } /* Check the PF's RAS state and Function INT state */ @@ -1154,14 +1154,16 @@ static void hisi_acc_vf_pci_reset_prepare(struct pci_dev *pdev) { struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_drvdata(pdev); struct hisi_qm *qm = hisi_acc_vdev->pf_qm; - struct device *dev = &qm->pdev->dev; u32 delay = 0; + if (!hisi_acc_vdev->core_device.vdev.mig_ops) + return; + /* All reset requests need to be queued for processing */ while (test_and_set_bit(QM_RESETTING, &qm->misc_ctl)) { msleep(1); if (++delay > QM_RESET_WAIT_TIMEOUT) { - dev_err(dev, "reset prepare failed\n"); + dev_err(&qm->pdev->dev, "reset prepare failed\n"); return; } } @@ -1174,12 +1176,14 @@ static void hisi_acc_vf_pci_aer_reset_done(struct pci_dev *pdev) struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_drvdata(pdev); struct hisi_qm *qm = hisi_acc_vdev->pf_qm; - if (hisi_acc_vdev->set_reset_flag) - clear_bit(QM_RESETTING, &qm->misc_ctl); - if (!hisi_acc_vdev->core_device.vdev.mig_ops) return; + if (hisi_acc_vdev->set_reset_flag) { + clear_bit(QM_RESETTING, &qm->misc_ctl); + hisi_acc_vdev->set_reset_flag = false; + } + mutex_lock(&hisi_acc_vdev->state_mutex); hisi_acc_vf_reset(hisi_acc_vdev); mutex_unlock(&hisi_acc_vdev->state_mutex); @@ -1191,13 +1195,6 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev) struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; struct pci_dev *vf_dev = vdev->pdev; - u32 val; - - val = readl(pf_qm->io_base + QM_MIG_REGION_SEL); - if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN)) - hisi_acc_vdev->drv_mode = HW_ACC_MIG_PF_CTRL; - else - hisi_acc_vdev->drv_mode = HW_ACC_MIG_VF_CTRL; if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_PF_CTRL) { /* @@ -1416,9 +1413,8 @@ static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device *vde static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; u64 value; int ret; @@ -1445,9 +1441,8 @@ static int hisi_acc_vf_debug_cmd(struct seq_file *seq, void *data) static int hisi_acc_vf_dev_read(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); + struct vfio_device *vdev = &hisi_acc_vdev->core_device.vdev; size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); struct acc_vf_data *vf_data; int ret; @@ -1492,9 +1487,7 @@ mutex_release: static int hisi_acc_vf_migf_read(struct seq_file *seq, void *data) { struct device *vf_dev = seq->private; - struct vfio_pci_core_device *core_device = dev_get_drvdata(vf_dev); - struct vfio_device *vdev = &core_device->vdev; - struct hisi_acc_vf_core_device *hisi_acc_vdev = hisi_acc_get_vf_dev(vdev); + struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(vf_dev); size_t vf_data_sz = offsetofend(struct acc_vf_data, padding); struct hisi_acc_vf_migration_file *debug_migf = hisi_acc_vdev->debug_migf; @@ -1665,21 +1658,34 @@ static void hisi_acc_vf_debugfs_exit(struct hisi_acc_vf_core_device *hisi_acc_vd hisi_acc_vdev->debug_migf = NULL; } +static enum hw_drv_mode hisi_acc_vf_get_drv_mode(struct hisi_qm *pf_qm) +{ + u32 val; + + val = readl(pf_qm->io_base + QM_MIG_REGION_SEL); + if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN)) + return HW_ACC_MIG_PF_CTRL; + + return HW_ACC_MIG_VF_CTRL; +} + static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct hisi_acc_vf_core_device *hisi_acc_vdev; const struct vfio_device_ops *ops = &hisi_acc_vfio_pci_ops; + enum hw_drv_mode drv_mode = HW_ACC_MIG_VF_CTRL; + resource_size_t func_len; struct hisi_qm *pf_qm; - int vf_id; int ret; pf_qm = hisi_acc_get_pf_qm(pdev); if (pf_qm && pf_qm->ver >= QM_HW_V3) { - vf_id = pci_iov_vf_id(pdev); - if (vf_id >= 0) - ops = &hisi_acc_vfio_pci_migrn_ops; + func_len = pci_resource_len(pdev, VFIO_PCI_BAR2_REGION_INDEX) >> 1; + drv_mode = hisi_acc_vf_get_drv_mode(pf_qm); + if (drv_mode == HW_ACC_MIG_VF_CTRL && func_len < PAGE_SIZE) + pci_warn(pdev, "migration not supported on 64KB pages with QM_HW_V3\n"); else - pci_warn(pdev, "migration support failed, continue with generic interface\n"); + ops = &hisi_acc_vfio_pci_migrn_ops; } hisi_acc_vdev = vfio_alloc_device(hisi_acc_vf_core_device, @@ -1687,7 +1693,9 @@ static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device if (IS_ERR(hisi_acc_vdev)) return PTR_ERR(hisi_acc_vdev); - dev_set_drvdata(&pdev->dev, &hisi_acc_vdev->core_device); + hisi_acc_vdev->drv_mode = drv_mode; + + dev_set_drvdata(&pdev->dev, hisi_acc_vdev); ret = vfio_pci_core_register_device(&hisi_acc_vdev->core_device); if (ret) goto out_put_vdev; @@ -1721,7 +1729,7 @@ MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table); static const struct pci_error_handlers hisi_acc_vf_err_handlers = { .reset_prepare = hisi_acc_vf_pci_reset_prepare, .reset_done = hisi_acc_vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = hisi_acc_vf_aer_err_detected, }; static struct pci_driver hisi_acc_vfio_pci_driver = { @@ -1729,6 +1737,7 @@ static struct pci_driver hisi_acc_vfio_pci_driver = { .id_table = hisi_acc_vfio_pci_table, .probe = hisi_acc_vfio_pci_probe, .remove = hisi_acc_vfio_pci_remove, + .driver = { .pm = pm_ptr(&hisi_acc_vf_pm_ops) }, .err_handler = &hisi_acc_vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/ism/main.c b/drivers/vfio/pci/ism/main.c index f83e09b915ed..cce3bdf1f070 100644 --- a/drivers/vfio/pci/ism/main.c +++ b/drivers/vfio/pci/ism/main.c @@ -47,6 +47,9 @@ struct ism_vfio_pci_core_device { struct kmem_cache *store_block_cache; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(ism_vfio, struct ism_vfio_pci_core_device, + core_device) + static int ism_vfio_pci_open_device(struct vfio_device *core_vdev) { struct ism_vfio_pci_core_device *ivpcd; @@ -363,7 +366,7 @@ static int ism_vfio_pci_probe(struct pci_dev *pdev, if (IS_ERR(ivpcd)) return PTR_ERR(ivpcd); - dev_set_drvdata(&pdev->dev, &ivpcd->core_device); + dev_set_drvdata(&pdev->dev, ivpcd); ret = vfio_pci_core_register_device(&ivpcd->core_device); if (ret) @@ -374,12 +377,7 @@ static int ism_vfio_pci_probe(struct pci_dev *pdev, static void ism_vfio_pci_remove(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device; - struct ism_vfio_pci_core_device *ivpcd; - - core_device = dev_get_drvdata(&pdev->dev); - ivpcd = container_of(core_device, struct ism_vfio_pci_core_device, - core_device); + struct ism_vfio_pci_core_device *ivpcd = pci_get_drvdata(pdev); vfio_pci_core_unregister_device(&ivpcd->core_device); vfio_put_device(&ivpcd->core_device.vdev); @@ -392,12 +390,17 @@ static const struct pci_device_id ism_device_table[] = { }; MODULE_DEVICE_TABLE(pci, ism_device_table); +static const struct pci_error_handlers ism_vfio_err_handlers = { + .error_detected = ism_vfio_aer_err_detected, +}; + static struct pci_driver ism_vfio_pci_driver = { .name = KBUILD_MODNAME, .id_table = ism_device_table, .probe = ism_vfio_pci_probe, .remove = ism_vfio_pci_remove, - .err_handler = &vfio_pci_core_err_handlers, + .driver = { .pm = pm_ptr(&ism_vfio_pm_ops) }, + .err_handler = &ism_vfio_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 5fe0621b5fbd..04180035a7fe 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -1644,7 +1644,7 @@ set_report_output(u32 size, int index, struct mlx5_vhca_qp *qp, int i; buf_offset = index * qp->max_msg_size; - if (WARN_ON(buf_offset + size >= qp->recv_buf.npages * PAGE_SIZE || + if (WARN_ON(buf_offset + size > qp->recv_buf.npages * PAGE_SIZE || (nent > qp->max_msg_size / entry_size))) return; diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index de306dee1d1a..18a6d224b1fd 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -21,6 +21,9 @@ #include "cmd.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(mlx5vf, struct mlx5vf_pci_core_device, + core_device) + /* Device specification max LOAD size */ #define MAX_LOAD_SIZE (BIT_ULL(__mlx5_bit_sz(load_vhca_state_in, size)) - 1) @@ -28,10 +31,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct mlx5vf_pci_core_device, - core_device); + return pci_get_drvdata(pdev); } static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf) @@ -1415,7 +1415,7 @@ static int mlx5vf_pci_probe(struct pci_dev *pdev, if (IS_ERR(mvdev)) return PTR_ERR(mvdev); - dev_set_drvdata(&pdev->dev, &mvdev->core_device); + dev_set_drvdata(&pdev->dev, mvdev); ret = vfio_pci_core_register_device(&mvdev->core_device); if (ret) goto out_put_vdev; @@ -1443,7 +1443,7 @@ MODULE_DEVICE_TABLE(pci, mlx5vf_pci_table); static const struct pci_error_handlers mlx5vf_err_handlers = { .reset_done = mlx5vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = mlx5vf_aer_err_detected, }; static struct pci_driver mlx5vf_pci_driver = { @@ -1451,6 +1451,7 @@ static struct pci_driver mlx5vf_pci_driver = { .id_table = mlx5vf_pci_table, .probe = mlx5vf_pci_probe, .remove = mlx5vf_pci_remove, + .driver = { .pm = pm_ptr(&mlx5vf_pm_ops) }, .err_handler = &mlx5vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 963fd8ded20d..5ddec340b73b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -73,6 +73,9 @@ struct nvgrace_gpu_pci_core_device { int cxl_dvsec; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(nvgrace_gpu, struct nvgrace_gpu_pci_core_device, + core_device) + static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev) { struct nvgrace_gpu_pci_core_device *nvdev = @@ -185,10 +188,8 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev) if (ret) return ret; - if (nvdev->usemem.memlength) { - nvgrace_gpu_init_fake_bar_emu_regs(core_vdev); - mutex_init(&nvdev->remap_lock); - } + nvgrace_gpu_init_fake_bar_emu_regs(core_vdev); + mutex_init(&nvdev->remap_lock); /* * GPU readiness is checked by reading the BAR0 registers. @@ -1083,6 +1084,21 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { .detach_ioas = vfio_iommufd_physical_detach_ioas, }; +static int nvgrace_gpu_core_open_device(struct vfio_device *core_vdev) +{ + struct vfio_pci_core_device *vdev = + container_of(core_vdev, struct vfio_pci_core_device, vdev); + int ret; + + ret = vfio_pci_core_enable(vdev); + if (ret) + return ret; + + vfio_pci_core_finish_enable(vdev); + + return 0; +} + static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = { .get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys, }; @@ -1091,7 +1107,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = { .name = "nvgrace-gpu-vfio-pci-core", .init = vfio_pci_core_init_dev, .release = vfio_pci_core_release_dev, - .open_device = nvgrace_gpu_open_device, + .open_device = nvgrace_gpu_core_open_device, .close_device = vfio_pci_core_close_device, .ioctl = vfio_pci_core_ioctl, .get_region_info_caps = vfio_pci_ioctl_get_region_info, @@ -1341,7 +1357,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, if (ret) goto out_put_vdev; - dev_set_drvdata(&pdev->dev, &nvdev->core_device); + dev_set_drvdata(&pdev->dev, nvdev); if (ops == &nvgrace_gpu_pci_ops) { nvdev->has_mig_hw_bug = nvgrace_gpu_has_mig_hw_bug(pdev); @@ -1372,10 +1388,10 @@ out_put_vdev: static void nvgrace_gpu_remove(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); + struct nvgrace_gpu_pci_core_device *nvdev = pci_get_drvdata(pdev); - vfio_pci_core_unregister_device(core_device); - vfio_put_device(&core_device->vdev); + vfio_pci_core_unregister_device(&nvdev->core_device); + vfio_put_device(&nvdev->core_device.vdev); } static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { @@ -1407,17 +1423,14 @@ MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table); */ static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - struct nvgrace_gpu_pci_core_device *nvdev = - container_of(core_device, struct nvgrace_gpu_pci_core_device, - core_device); + struct nvgrace_gpu_pci_core_device *nvdev = pci_get_drvdata(pdev); nvdev->reset_done = true; } static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = { .reset_done = nvgrace_gpu_vfio_pci_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = nvgrace_gpu_aer_err_detected, }; static struct pci_driver nvgrace_gpu_vfio_pci_driver = { @@ -1425,6 +1438,7 @@ static struct pci_driver nvgrace_gpu_vfio_pci_driver = { .id_table = nvgrace_gpu_vfio_pci_table, .probe = nvgrace_gpu_probe, .remove = nvgrace_gpu_remove, + .driver = { .pm = pm_ptr(&nvgrace_gpu_pm_ops) }, .err_handler = &nvgrace_gpu_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/pds/pci_drv.c b/drivers/vfio/pci/pds/pci_drv.c index 4923f1823126..cf7e33d4d17b 100644 --- a/drivers/vfio/pci/pds/pci_drv.c +++ b/drivers/vfio/pci/pds/pci_drv.c @@ -16,6 +16,9 @@ #include "pci_drv.h" #include "cmds.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(pds_vfio, struct pds_vfio_pci_device, + vfio_coredev) + #define PDS_VFIO_DRV_DESCRIPTION "AMD/Pensando VFIO Device Driver" #define PCI_VENDOR_ID_PENSANDO 0x1dd8 @@ -118,7 +121,7 @@ static int pds_vfio_pci_probe(struct pci_dev *pdev, if (IS_ERR(pds_vfio)) return PTR_ERR(pds_vfio); - dev_set_drvdata(&pdev->dev, &pds_vfio->vfio_coredev); + dev_set_drvdata(&pdev->dev, pds_vfio); err = vfio_pci_core_register_device(&pds_vfio->vfio_coredev); if (err) @@ -173,7 +176,7 @@ static void pds_vfio_pci_aer_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers pds_vfio_pci_err_handlers = { .reset_done = pds_vfio_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = pds_vfio_aer_err_detected, }; static struct pci_driver pds_vfio_pci_driver = { @@ -181,6 +184,7 @@ static struct pci_driver pds_vfio_pci_driver = { .id_table = pds_vfio_pci_table, .probe = pds_vfio_pci_probe, .remove = pds_vfio_pci_remove, + .driver = { .pm = pm_ptr(&pds_vfio_pm_ops) }, .err_handler = &pds_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c index be103c74e969..6d2b2dc61a28 100644 --- a/drivers/vfio/pci/pds/vfio_dev.c +++ b/drivers/vfio/pci/pds/vfio_dev.c @@ -20,10 +20,7 @@ struct device *pds_vfio_to_dev(struct pds_vfio_pci_device *pds_vfio) struct pds_vfio_pci_device *pds_vfio_pci_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); - - return container_of(core_device, struct pds_vfio_pci_device, - vfio_coredev); + return pci_get_drvdata(pdev); } void pds_vfio_reset(struct pds_vfio_pci_device *pds_vfio, diff --git a/drivers/vfio/pci/qat/main.c b/drivers/vfio/pci/qat/main.c index 60ff907b6a67..69448fa57df4 100644 --- a/drivers/vfio/pci/qat/main.c +++ b/drivers/vfio/pci/qat/main.c @@ -55,6 +55,9 @@ struct qat_vf_core_device { struct qat_vf_migration_file *saving_migf; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(qat_vf, struct qat_vf_core_device, + core_device) + static int qat_vf_pci_open_device(struct vfio_device *core_vdev) { struct qat_vf_core_device *qat_vdev = @@ -623,9 +626,7 @@ static const struct vfio_device_ops qat_vf_pci_ops = { static struct qat_vf_core_device *qat_vf_drvdata(struct pci_dev *pdev) { - struct vfio_pci_core_device *core_device = pci_get_drvdata(pdev); - - return container_of(core_device, struct qat_vf_core_device, core_device); + return pci_get_drvdata(pdev); } static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev) @@ -651,7 +652,7 @@ qat_vf_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (IS_ERR(qat_vdev)) return PTR_ERR(qat_vdev); - pci_set_drvdata(pdev, &qat_vdev->core_device); + pci_set_drvdata(pdev, qat_vdev); ret = vfio_pci_core_register_device(&qat_vdev->core_device); if (ret) goto out_put_device; @@ -686,7 +687,7 @@ MODULE_DEVICE_TABLE(pci, qat_vf_vfio_pci_table); static const struct pci_error_handlers qat_vf_err_handlers = { .reset_done = qat_vf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = qat_vf_aer_err_detected, }; static struct pci_driver qat_vf_vfio_pci_driver = { @@ -694,6 +695,7 @@ static struct pci_driver qat_vf_vfio_pci_driver = { .id_table = qat_vf_vfio_pci_table, .probe = qat_vf_vfio_pci_probe, .remove = qat_vf_vfio_pci_remove, + .driver = { .pm = pm_ptr(&qat_vf_pm_ops) }, .err_handler = &qat_vf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 830369ff878d..e7c012febe91 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -223,13 +223,44 @@ static const struct pci_device_id vfio_pci_table[] = { MODULE_DEVICE_TABLE(pci, vfio_pci_table); +static int __maybe_unused vfio_pci_pm_suspend(struct device *dev) +{ + struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); + + return vfio_pci_core_runtime_suspend(vdev); +} + +static int __maybe_unused vfio_pci_pm_resume(struct device *dev) +{ + struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); + + return vfio_pci_core_runtime_resume(vdev); +} + +static const struct dev_pm_ops vfio_pci_pm_ops = { + SET_RUNTIME_PM_OPS(vfio_pci_pm_suspend, vfio_pci_pm_resume, NULL) +}; + +static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state) +{ + struct vfio_pci_core_device *vdev = pci_get_drvdata(pdev); + + return vfio_pci_core_aer_err_detected(vdev); +} + +static const struct pci_error_handlers vfio_pci_err_handlers = { + .error_detected = vfio_pci_aer_err_detected, +}; + static struct pci_driver vfio_pci_driver = { .name = "vfio-pci", .id_table = vfio_pci_table, .probe = vfio_pci_probe, .remove = vfio_pci_remove, .sriov_configure = vfio_pci_sriov_configure, - .err_handler = &vfio_pci_core_err_handlers, + .err_handler = &vfio_pci_err_handlers, + .driver = { .pm = pm_ptr(&vfio_pci_pm_ops) }, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 6757054e9d87..fcbb2da860e2 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -162,9 +162,10 @@ static inline void vfio_pci_core_debugfs_init(struct vfio_pci_core_device *vdev) * has no way to get to it and routing can be disabled externally at the * bridge. */ -static unsigned int vfio_pci_set_decode(struct pci_dev *pdev, bool single_vga) +static unsigned int vfio_pci_set_decode(void *data, bool single_vga) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); + struct vfio_pci_core_device *vdev = data; + struct pci_dev *pdev = vdev->pdev; struct pci_dev *tmp = NULL; unsigned char max_busnr; unsigned int decodes; @@ -488,10 +489,8 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags, } #ifdef CONFIG_PM -static int vfio_pci_core_runtime_suspend(struct device *dev) +int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); - down_write(&vdev->memory_lock); /* * The user can move the device into D3hot state before invoking @@ -516,11 +515,10 @@ static int vfio_pci_core_runtime_suspend(struct device *dev) return 0; } +EXPORT_SYMBOL_GPL(vfio_pci_core_runtime_suspend); -static int vfio_pci_core_runtime_resume(struct device *dev) +int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(dev); - /* * Resume with a pm_wake_eventfd_ctx signals the eventfd and exit * low power mode. @@ -537,6 +535,7 @@ static int vfio_pci_core_runtime_resume(struct device *dev) return 0; } +EXPORT_SYMBOL_GPL(vfio_pci_core_runtime_resume); #endif /* CONFIG_PM */ /* @@ -576,18 +575,6 @@ static void vfio_pci_core_map_bars(struct vfio_pci_core_device *vdev) } } -/* - * The pci-driver core runtime PM routines always save the device state - * before going into suspended state. If the device is going into low power - * state with only with runtime PM ops, then no explicit handling is needed - * for the devices which have NoSoftRst-. - */ -static const struct dev_pm_ops vfio_pci_core_pm_ops = { - SET_RUNTIME_PM_OPS(vfio_pci_core_runtime_suspend, - vfio_pci_core_runtime_resume, - NULL) -}; - int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) { struct pci_dev *pdev = vdev->pdev; @@ -2159,10 +2146,10 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev) if (ret) return ret; - ret = vga_client_register(pdev, vfio_pci_set_decode); + ret = vga_client_register(pdev, vfio_pci_set_decode, vdev); if (ret) return ret; - vga_set_legacy_decoding(pdev, vfio_pci_set_decode(pdev, false)); + vga_set_legacy_decoding(pdev, vfio_pci_set_decode(vdev, false)); return 0; } @@ -2221,10 +2208,6 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) struct device *dev = &pdev->dev; int ret; - /* Drivers must set the vfio_pci_core_device to their drvdata */ - if (WARN_ON(vdev != dev_get_drvdata(dev))) - return -EINVAL; - /* Drivers must set a name. Required for sequestering SR-IOV VFs */ if (WARN_ON(!vdev->vdev.ops->name)) return -EINVAL; @@ -2292,7 +2275,6 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) */ vfio_pci_set_power_state(vdev, PCI_D0); - dev->driver->pm = &vfio_pci_core_pm_ops; pm_runtime_allow(dev); if (!vdev->disable_idle_d3) pm_runtime_put(dev); @@ -2333,10 +2315,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) } EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device); -pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, - pci_channel_state_t state) +pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); struct vfio_pci_eventfd *eventfd; rcu_read_lock(); @@ -2419,11 +2399,6 @@ out_unlock: } EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure); -const struct pci_error_handlers vfio_pci_core_err_handlers = { - .error_detected = vfio_pci_core_aer_err_detected, -}; -EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); - static bool vfio_dev_in_groups(struct vfio_device *vdev, struct vfio_pci_group_info *groups) { @@ -2675,6 +2650,7 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set) static void vfio_pci_core_cleanup(void) { vfio_pci_uninit_perm_bits(); + rcu_barrier(); } static int __init vfio_pci_core_init(void) diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c index 7f14dd46de17..df8848087681 100644 --- a/drivers/vfio/pci/vfio_pci_rdwr.c +++ b/drivers/vfio/pci/vfio_pci_rdwr.c @@ -423,9 +423,6 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset, pos >= vdev->msix_offset + vdev->msix_size)) return -EINVAL; - if (count == 8) - return -EINVAL; - io = vfio_pci_core_get_iomap(vdev, bar); if (IS_ERR(io)) return PTR_ERR(io); diff --git a/drivers/vfio/pci/virtio/main.c b/drivers/vfio/pci/virtio/main.c index d2e5cbca13c8..123ff5193812 100644 --- a/drivers/vfio/pci/virtio/main.c +++ b/drivers/vfio/pci/virtio/main.c @@ -18,6 +18,9 @@ #include "common.h" +VFIO_PCI_CORE_DEFINE_CALLBACKS(virtiovf, struct virtiovf_pci_core_device, + core_device) + static int virtiovf_pci_open_device(struct vfio_device *core_vdev) { struct virtiovf_pci_core_device *virtvdev = container_of(core_vdev, @@ -174,7 +177,7 @@ static int virtiovf_pci_probe(struct pci_dev *pdev, if (sup_lm) virtiovf_set_migratable(virtvdev); - dev_set_drvdata(&pdev->dev, &virtvdev->core_device); + dev_set_drvdata(&pdev->dev, virtvdev); ret = vfio_pci_core_register_device(&virtvdev->core_device); if (ret) goto out; @@ -211,7 +214,7 @@ static void virtiovf_pci_aer_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers virtiovf_err_handlers = { .reset_done = virtiovf_pci_aer_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = virtiovf_aer_err_detected, }; static struct pci_driver virtiovf_pci_driver = { @@ -219,6 +222,7 @@ static struct pci_driver virtiovf_pci_driver = { .id_table = virtiovf_pci_table, .probe = virtiovf_pci_probe, .remove = virtiovf_pci_remove, + .driver = { .pm = pm_ptr(&virtiovf_pm_ops) }, .err_handler = &virtiovf_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c index cbff5af385ef..7dab3e024aa8 100644 --- a/drivers/vfio/pci/xe/main.c +++ b/drivers/vfio/pci/xe/main.c @@ -38,6 +38,9 @@ struct xe_vfio_pci_core_device { struct xe_vfio_pci_migration_file *migf; }; +VFIO_PCI_CORE_DEFINE_CALLBACKS(xe_vfio, struct xe_vfio_pci_core_device, + core_device) + #define xe_vdev_to_dev(xe_vdev) (&(xe_vdev)->core_device.pdev->dev) static void xe_vfio_pci_disable_file(struct xe_vfio_pci_migration_file *migf) @@ -140,7 +143,7 @@ static void xe_vfio_pci_reset_done(struct pci_dev *pdev) static const struct pci_error_handlers xe_vfio_pci_err_handlers = { .reset_prepare = xe_vfio_pci_reset_prepare, .reset_done = xe_vfio_pci_reset_done, - .error_detected = vfio_pci_core_aer_err_detected, + .error_detected = xe_vfio_aer_err_detected, }; static int xe_vfio_pci_open_device(struct vfio_device *core_vdev) @@ -550,7 +553,7 @@ static int xe_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i if (IS_ERR(xe_vdev)) return PTR_ERR(xe_vdev); - dev_set_drvdata(&pdev->dev, &xe_vdev->core_device); + dev_set_drvdata(&pdev->dev, xe_vdev); ret = vfio_pci_core_register_device(&xe_vdev->core_device); if (ret) { @@ -574,9 +577,12 @@ static void xe_vfio_pci_remove(struct pci_dev *pdev) } static const struct pci_device_id xe_vfio_pci_table[] = { + INTEL_BMG_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_CRI_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_NVLS_IDS(INTEL_PCI_VFIO_DEVICE), + INTEL_NVLP_IDS(INTEL_PCI_VFIO_DEVICE), INTEL_PTL_IDS(INTEL_PCI_VFIO_DEVICE), INTEL_WCL_IDS(INTEL_PCI_VFIO_DEVICE), - INTEL_BMG_IDS(INTEL_PCI_VFIO_DEVICE), {} }; MODULE_DEVICE_TABLE(pci, xe_vfio_pci_table); @@ -586,6 +592,7 @@ static struct pci_driver xe_vfio_pci_driver = { .id_table = xe_vfio_pci_table, .probe = xe_vfio_pci_probe, .remove = xe_vfio_pci_remove, + .driver = { .pm = pm_ptr(&xe_vfio_pm_ops) }, .err_handler = &xe_vfio_pci_err_handlers, .driver_managed_dma = true, }; diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c index c72db5a99ebd..e8cfa76ad7f8 100644 --- a/drivers/vfio/platform/vfio_platform_common.c +++ b/drivers/vfio/platform/vfio_platform_common.c @@ -559,9 +559,8 @@ static int vfio_platform_mmap_mmio(struct vfio_platform_region region, return -EINVAL; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; - - return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + return remap_pfn_range(vma, vma->vm_start, + (region.addr >> PAGE_SHIFT) + pgoff, req_len, vma->vm_page_prot); } @@ -591,9 +590,13 @@ int vfio_platform_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma && (vma->vm_flags & VM_READ)) return -EINVAL; - if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) - && (vma->vm_flags & VM_WRITE)) - return -EINVAL; + /* Prevent read-only region mappings from being upgraded with mprotect() */ + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)) { + if (vma->vm_flags & VM_WRITE) + return -EINVAL; + + vm_flags_clear(vma, VM_MAYWRITE); + } vma->vm_private_data = vdev; diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index c8151ba54de3..f7addfbe1aab 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -94,6 +94,7 @@ struct vfio_dma { bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; bool has_rsvd; /* has 1 or more rsvd pfns */ + bool has_non_rsvd; /* has 1 or more !rsvd pfns */ struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; @@ -791,6 +792,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, out: dma->has_rsvd |= rsvd; + dma->has_non_rsvd |= !rsvd; ret = vfio_lock_acct(dma, lock_acct, false); unpin_out: @@ -821,11 +823,13 @@ static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, long unlocked = 0, locked = vpfn_pages(dma, iova, npage); if (dma->has_rsvd) { - unsigned long i; + if (dma->has_non_rsvd) { + unsigned long i; - for (i = 0; i < npage; i++) - if (put_pfn(pfn++, dma->prot)) - unlocked++; + for (i = 0; i < npage; i++) + if (put_pfn(pfn++, dma->prot)) + unlocked++; + } } else { put_valid_unreserved_pfns(pfn, npage, dma->prot); unlocked = npage; diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index ea8e6cc13c45..4e5d738ccf4b 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -574,13 +574,6 @@ int vfio_df_open(struct vfio_device_file *df) lockdep_assert_held(&device->dev_set->lock); - /* - * Only the group path allows the device to be opened multiple - * times. The device cdev path doesn't have a secure way for it. - */ - if (device->open_count != 0 && !df->group) - return -EINVAL; - device->open_count++; if (device->open_count == 1) { ret = vfio_df_device_first_open(df); diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h index 9d47cdc727ad..a1c14479e462 100644 --- a/include/linux/mlx5/cq.h +++ b/include/linux/mlx5/cq.h @@ -81,16 +81,6 @@ enum { enum { MLX5_CQE_OWNER_MASK = 1, - MLX5_CQE_REQ = 0, - MLX5_CQE_RESP_WR_IMM = 1, - MLX5_CQE_RESP_SEND = 2, - MLX5_CQE_RESP_SEND_IMM = 3, - MLX5_CQE_RESP_SEND_INV = 4, - MLX5_CQE_RESIZE_CQ = 5, - MLX5_CQE_SIG_ERR = 12, - MLX5_CQE_REQ_ERR = 13, - MLX5_CQE_RESP_ERR = 14, - MLX5_CQE_INVALID = 15, }; enum { diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 8cb321a9fb3d..37727b5ab78f 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -36,6 +36,7 @@ #include <linux/types.h> #include <rdma/ib_verbs.h> #include <linux/mlx5/mlx5_ifc.h> +#include <linux/mlx5/mlx5_ifc_macros.h> #include <linux/bitfield.h> #if defined(__LITTLE_ENDIAN) @@ -46,122 +47,6 @@ #error Host endianness not defined #endif -/* helper macros */ -#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0) -#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld) -#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld)) -#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16) -#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32) -#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64) -#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf)) -#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f)) -#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) -#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld)) -#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) -#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld)) -#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) - -#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) -#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) -#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) -#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) -#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8) -#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32) -#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8) -#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld))) - -/* insert a value to a struct */ -#define MLX5_SET(typ, p, fld, v) do { \ - u32 _v = v; \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ - *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ - cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ - (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \ - << __mlx5_dw_bit_off(typ, fld))); \ -} while (0) - -#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \ - MLX5_SET(typ, p, fld[idx], v); \ -} while (0) - -#define MLX5_SET_TO_ONES(typ, p, fld) do { \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ - *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ - cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ - (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \ - << __mlx5_dw_bit_off(typ, fld))); \ -} while (0) - -#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\ -__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \ -__mlx5_mask(typ, fld)) - -#define MLX5_GET_PR(typ, p, fld) ({ \ - u32 ___t = MLX5_GET(typ, p, fld); \ - pr_debug(#fld " = 0x%x\n", ___t); \ - ___t; \ -}) - -#define __MLX5_SET64(typ, p, fld, v) do { \ - BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \ - *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \ -} while (0) - -#define MLX5_SET64(typ, p, fld, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ - __MLX5_SET64(typ, p, fld, v); \ -} while (0) - -#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \ - BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ - __MLX5_SET64(typ, p, fld[idx], v); \ -} while (0) - -#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld))) - -#define MLX5_GET64_PR(typ, p, fld) ({ \ - u64 ___t = MLX5_GET64(typ, p, fld); \ - pr_debug(#fld " = 0x%llx\n", ___t); \ - ___t; \ -}) - -#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\ -__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \ -__mlx5_mask16(typ, fld)) - -#define MLX5_SET16(typ, p, fld, v) do { \ - u16 _v = v; \ - BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \ - *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \ - cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \ - (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \ - << __mlx5_16_bit_off(typ, fld))); \ -} while (0) - -/* Big endian getters */ -#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\ - __mlx5_64_off(typ, fld))) - -#define MLX5_GET_BE(type_t, typ, p, fld) ({ \ - type_t tmp; \ - switch (sizeof(tmp)) { \ - case sizeof(u8): \ - tmp = (__force type_t)MLX5_GET(typ, p, fld); \ - break; \ - case sizeof(u16): \ - tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \ - break; \ - case sizeof(u32): \ - tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \ - break; \ - case sizeof(u64): \ - tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \ - break; \ - } \ - tmp; \ - }) - enum mlx5_inline_modes { MLX5_INLINE_MODE_NONE, MLX5_INLINE_MODE_L2, @@ -172,7 +57,7 @@ enum mlx5_inline_modes { enum { MLX5_MAX_COMMANDS = 32, MLX5_CMD_DATA_BLOCK_SIZE = 512, - MLX5_PCI_CMD_XPORT = 7, + MLX5_PCI_CMD_XPORT = MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT, MLX5_MKEY_BSF_OCTO_SIZE = 4, MLX5_MAX_PSVS = 4, }; @@ -308,63 +193,6 @@ enum { MLX5_EVENT_QUEUE_TYPE_DCT = 6, }; -/* mlx5 components can subscribe to any one of these events via - * mlx5_eq_notifier_register API. - */ -enum mlx5_event { - /* Special value to subscribe to any event */ - MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0, - /* HW events enum start: comp events are not subscribable */ - MLX5_EVENT_TYPE_COMP = 0x0, - /* HW Async events enum start: subscribable events */ - MLX5_EVENT_TYPE_PATH_MIG = 0x01, - MLX5_EVENT_TYPE_COMM_EST = 0x02, - MLX5_EVENT_TYPE_SQ_DRAINED = 0x03, - MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13, - MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14, - - MLX5_EVENT_TYPE_CQ_ERROR = 0x04, - MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05, - MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07, - MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10, - MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11, - MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12, - MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27, - - MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08, - MLX5_EVENT_TYPE_PORT_CHANGE = 0x09, - MLX5_EVENT_TYPE_GPIO_EVENT = 0x15, - MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16, - MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17, - MLX5_EVENT_TYPE_XRQ_ERROR = 0x18, - MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19, - MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22, - MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24, - MLX5_EVENT_TYPE_PPS_EVENT = 0x25, - - MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a, - MLX5_EVENT_TYPE_STALL_EVENT = 0x1b, - - MLX5_EVENT_TYPE_CMD = 0x0a, - MLX5_EVENT_TYPE_PAGE_REQUEST = 0xb, - - MLX5_EVENT_TYPE_PAGE_FAULT = 0xc, - MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0xd, - - MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0xe, - MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0xf, - - MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c, - MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d, - - MLX5_EVENT_TYPE_FPGA_ERROR = 0x20, - MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21, - - MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26, - - MLX5_EVENT_TYPE_MAX = 0x100, -}; - enum mlx5_driver_event { MLX5_DRIVER_EVENT_TYPE_TRAP = 0, MLX5_DRIVER_EVENT_UPLINK_NETDEV, @@ -420,22 +248,6 @@ enum { }; enum { - MLX5_OPCODE_NOP = 0x00, - MLX5_OPCODE_SEND_INVAL = 0x01, - MLX5_OPCODE_RDMA_WRITE = 0x08, - MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, - MLX5_OPCODE_SEND = 0x0a, - MLX5_OPCODE_SEND_IMM = 0x0b, - MLX5_OPCODE_LSO = 0x0e, - MLX5_OPCODE_RDMA_READ = 0x10, - MLX5_OPCODE_ATOMIC_CS = 0x11, - MLX5_OPCODE_ATOMIC_FA = 0x12, - MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, - MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, - MLX5_OPCODE_BIND_MW = 0x18, - MLX5_OPCODE_CONFIG_CMD = 0x1f, - MLX5_OPCODE_ENHANCED_MPSW = 0x29, - MLX5_RECV_OPCODE_RDMA_WRITE_IMM = 0x00, MLX5_RECV_OPCODE_SEND = 0x01, MLX5_RECV_OPCODE_SEND_IMM = 0x02, @@ -443,19 +255,6 @@ enum { MLX5_CQE_OPCODE_ERROR = 0x1e, MLX5_CQE_OPCODE_RESIZE = 0x16, - - MLX5_OPCODE_SET_PSV = 0x20, - MLX5_OPCODE_GET_PSV = 0x21, - MLX5_OPCODE_CHECK_PSV = 0x22, - MLX5_OPCODE_DUMP = 0x23, - MLX5_OPCODE_RGET_PSV = 0x26, - MLX5_OPCODE_RCHECK_PSV = 0x27, - - MLX5_OPCODE_UMR = 0x25, - - MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c, - - MLX5_OPCODE_ACCESS_ASO = 0x2d, }; enum { @@ -1224,12 +1023,6 @@ enum mlx5_flex_parser_protos { /* MLX5 DEV CAPs */ -/* TODO: EAT.ME */ -enum mlx5_cap_mode { - HCA_CAP_OPMOD_GET_MAX = 0, - HCA_CAP_OPMOD_GET_CUR = 1, -}; - /* Any new cap addition must update mlx5_hca_caps_alloc() to allocate * capability memory. */ @@ -1508,26 +1301,6 @@ enum mlx5_qcam_feature_groups { MLX5_GET(psp_cap, (mdev)->caps.hca[MLX5_CAP_PSP]->cur, cap) enum { - MLX5_CMD_STAT_OK = 0x0, - MLX5_CMD_STAT_INT_ERR = 0x1, - MLX5_CMD_STAT_BAD_OP_ERR = 0x2, - MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3, - MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4, - MLX5_CMD_STAT_BAD_RES_ERR = 0x5, - MLX5_CMD_STAT_RES_BUSY = 0x6, - MLX5_CMD_STAT_NOT_READY = 0x7, - MLX5_CMD_STAT_LIM_ERR = 0x8, - MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9, - MLX5_CMD_STAT_IX_ERR = 0xa, - MLX5_CMD_STAT_NO_RES_ERR = 0xf, - MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50, - MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51, - MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10, - MLX5_CMD_STAT_BAD_PKT_ERR = 0x30, - MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40, -}; - -enum { MLX5_IEEE_802_3_COUNTERS_GROUP = 0x0, MLX5_RFC_2863_COUNTERS_GROUP = 0x1, MLX5_RFC_2819_COUNTERS_GROUP = 0x2, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index a5fb747f6f55..697fc9387b73 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -5986,6 +5986,108 @@ struct mlx5_ifc_cqe_error_syndrome_bits { u8 syndrome[0x8]; }; +struct mlx5_ifc_wqe_ctrl_seg_bits { + u8 opmod[0x8]; + u8 wqe_index[0x10]; + u8 opcode[0x8]; + + u8 qp_or_sq[0x18]; + u8 reserved_at_38[0x2]; + u8 ds[0x6]; + + u8 signature[0x8]; + u8 reserved_at_48[0x10]; + u8 fm[0x3]; + u8 reserved_at_5b[0x1]; + u8 ce[0x2]; + u8 se[0x1]; + u8 reserved_at_5f[0x1]; + + u8 imm[0x20]; +}; + +/* Values for wqe_ctrl_seg_bits.opcode */ +enum { + MLX5_OPCODE_NOP = 0x00, + MLX5_OPCODE_SEND_INVAL = 0x01, + MLX5_OPCODE_RDMA_WRITE = 0x08, + MLX5_OPCODE_RDMA_WRITE_IMM = 0x09, + MLX5_OPCODE_SEND = 0x0a, + MLX5_OPCODE_SEND_IMM = 0x0b, + MLX5_OPCODE_LSO = 0x0e, + MLX5_OPCODE_RDMA_READ = 0x10, + MLX5_OPCODE_ATOMIC_CS = 0x11, + MLX5_OPCODE_ATOMIC_FA = 0x12, + MLX5_OPCODE_ATOMIC_MASKED_CS = 0x14, + MLX5_OPCODE_ATOMIC_MASKED_FA = 0x15, + MLX5_OPCODE_BIND_MW = 0x18, + MLX5_OPCODE_CONFIG_CMD = 0x1f, + MLX5_OPCODE_SET_PSV = 0x20, + MLX5_OPCODE_GET_PSV = 0x21, + MLX5_OPCODE_CHECK_PSV = 0x22, + MLX5_OPCODE_DUMP = 0x23, + MLX5_OPCODE_UMR = 0x25, + MLX5_OPCODE_RGET_PSV = 0x26, + MLX5_OPCODE_RCHECK_PSV = 0x27, + MLX5_OPCODE_ENHANCED_MPSW = 0x29, + MLX5_OPCODE_FLOW_TBL_ACCESS = 0x2c, + MLX5_OPCODE_ACCESS_ASO = 0x2d, +}; + +/* Values for wqe_ctrl_seg_bits.ce */ +enum { + MLX5_WQE_CE_CQE_ALWAYS = 2, +}; + +struct mlx5_ifc_wqe_raddr_seg_bits { + u8 raddr[0x40]; + + u8 rkey[0x20]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_wqe_data_seg_bits { + u8 reserved_at_0[0x1]; + u8 byte_count[0x1f]; + + u8 lkey[0x20]; + + u8 addr[0x40]; +}; + +struct mlx5_ifc_cqe64_bits { + u8 reserved_at_0[0x1a0]; + + union { + u8 reserved_at_1a0[0x20]; + struct mlx5_ifc_cqe_error_syndrome_bits error_syndrome; + }; + + u8 send_wqe_opcode[0x8]; + u8 qpn_or_dctn_or_flow_tag[0x18]; + + u8 wqe_counter[0x10]; + u8 signature[0x8]; + u8 opcode[0x4]; + u8 cqe_format[0x2]; + u8 se[0x1]; + u8 owner[0x1]; +}; + +/* Values for cqe64_bits.opcode */ +enum { + MLX5_CQE_REQ = 0, + MLX5_CQE_RESP_WR_IMM = 1, + MLX5_CQE_RESP_SEND = 2, + MLX5_CQE_RESP_SEND_IMM = 3, + MLX5_CQE_RESP_SEND_INV = 4, + MLX5_CQE_RESIZE_CQ = 5, + MLX5_CQE_SIG_ERR = 12, + MLX5_CQE_REQ_ERR = 13, + MLX5_CQE_RESP_ERR = 14, + MLX5_CQE_INVALID = 15, +}; + struct mlx5_ifc_qp_context_extension_bits { u8 reserved_at_0[0x60]; @@ -6482,6 +6584,12 @@ struct mlx5_ifc_query_hca_cap_in_bits { u8 reserved_at_60[0x20]; }; +/* Values for query_hca_cap_in_bits.op_mod */ +enum mlx5_cap_mode { + HCA_CAP_OPMOD_GET_MAX = 0, + HCA_CAP_OPMOD_GET_CUR = 1, +}; + struct mlx5_ifc_other_hca_cap_bits { u8 roce[0x1]; u8 reserved_at_1[0x27f]; @@ -11280,6 +11388,55 @@ struct mlx5_ifc_eqe_bits { u8 owner[0x1]; }; +/* Values for eqe_bits.event_type */ +enum mlx5_event { + /* Special value to subscribe to any event */ + MLX5_EVENT_TYPE_NOTIFY_ANY = 0x0, + /* HW events enum start: comp events are not subscribable */ + MLX5_EVENT_TYPE_COMP = 0x0, + /* HW Async events enum start: subscribable events */ + MLX5_EVENT_TYPE_PATH_MIG = 0x01, + MLX5_EVENT_TYPE_COMM_EST = 0x02, + MLX5_EVENT_TYPE_SQ_DRAINED = 0x03, + MLX5_EVENT_TYPE_SRQ_LAST_WQE = 0x13, + MLX5_EVENT_TYPE_SRQ_RQ_LIMIT = 0x14, + + MLX5_EVENT_TYPE_CQ_ERROR = 0x04, + MLX5_EVENT_TYPE_WQ_CATAS_ERROR = 0x05, + MLX5_EVENT_TYPE_PATH_MIG_FAILED = 0x07, + MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR = 0x10, + MLX5_EVENT_TYPE_WQ_ACCESS_ERROR = 0x11, + MLX5_EVENT_TYPE_SRQ_CATAS_ERROR = 0x12, + MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27, + + MLX5_EVENT_TYPE_INTERNAL_ERROR = 0x08, + MLX5_EVENT_TYPE_PORT_CHANGE = 0x09, + MLX5_EVENT_TYPE_CMD = 0x0a, + MLX5_EVENT_TYPE_PAGE_REQUEST = 0x0b, + MLX5_EVENT_TYPE_PAGE_FAULT = 0x0c, + MLX5_EVENT_TYPE_NIC_VPORT_CHANGE = 0x0d, + MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED = 0x0e, + MLX5_EVENT_TYPE_VHCA_STATE_CHANGE = 0x0f, + MLX5_EVENT_TYPE_GPIO_EVENT = 0x15, + MLX5_EVENT_TYPE_PORT_MODULE_EVENT = 0x16, + MLX5_EVENT_TYPE_TEMP_WARN_EVENT = 0x17, + MLX5_EVENT_TYPE_XRQ_ERROR = 0x18, + MLX5_EVENT_TYPE_REMOTE_CONFIG = 0x19, + MLX5_EVENT_TYPE_DB_BF_CONGESTION = 0x1a, + MLX5_EVENT_TYPE_STALL_EVENT = 0x1b, + MLX5_EVENT_TYPE_DCT_DRAINED = 0x1c, + MLX5_EVENT_TYPE_DCT_KEY_VIOLATION = 0x1d, + MLX5_EVENT_TYPE_FPGA_ERROR = 0x20, + MLX5_EVENT_TYPE_FPGA_QP_ERROR = 0x21, + MLX5_EVENT_TYPE_GENERAL_EVENT = 0x22, + MLX5_EVENT_TYPE_MONITOR_COUNTER = 0x24, + MLX5_EVENT_TYPE_PPS_EVENT = 0x25, + MLX5_EVENT_TYPE_DEVICE_TRACER = 0x26, + + MLX5_EVENT_TYPE_MAX = 0x100, +}; + +/* Values for cmd_queue_entry_bits.type */ enum { MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT = 0x7, }; @@ -11322,6 +11479,27 @@ struct mlx5_ifc_cmd_out_bits { u8 command_output[0x20]; }; +/* Values for cmd_out_bits.status */ +enum { + MLX5_CMD_STAT_OK = 0x0, + MLX5_CMD_STAT_INT_ERR = 0x1, + MLX5_CMD_STAT_BAD_OP_ERR = 0x2, + MLX5_CMD_STAT_BAD_PARAM_ERR = 0x3, + MLX5_CMD_STAT_BAD_SYS_STATE_ERR = 0x4, + MLX5_CMD_STAT_BAD_RES_ERR = 0x5, + MLX5_CMD_STAT_RES_BUSY = 0x6, + MLX5_CMD_STAT_NOT_READY = 0x7, + MLX5_CMD_STAT_LIM_ERR = 0x8, + MLX5_CMD_STAT_BAD_RES_STATE_ERR = 0x9, + MLX5_CMD_STAT_IX_ERR = 0xa, + MLX5_CMD_STAT_NO_RES_ERR = 0xf, + MLX5_CMD_STAT_BAD_QP_STATE_ERR = 0x10, + MLX5_CMD_STAT_BAD_PKT_ERR = 0x30, + MLX5_CMD_STAT_BAD_SIZE_OUTS_CQES_ERR = 0x40, + MLX5_CMD_STAT_BAD_INP_LEN_ERR = 0x50, + MLX5_CMD_STAT_BAD_OUTP_LEN_ERR = 0x51, +}; + struct mlx5_ifc_cmd_in_bits { u8 opcode[0x10]; u8 reserved_at_10[0x10]; diff --git a/include/linux/mlx5/mlx5_ifc_macros.h b/include/linux/mlx5/mlx5_ifc_macros.h new file mode 100644 index 000000000000..1b9e39a50949 --- /dev/null +++ b/include/linux/mlx5/mlx5_ifc_macros.h @@ -0,0 +1,185 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * Copyright (c) 2013-2026, Mellanox Technologies. All rights reserved. + * + * Accessor macros for mlx5 IFC structures. + * + * Extracted from device.h so that code which cannot include device.h + * (e.g. selftests) can still use the MLX5_SET/GET family directly. + */ + +#ifndef MLX5_IFC_MACROS_H +#define MLX5_IFC_MACROS_H + +/* Internal helpers -- 32-bit */ +#define __mlx5_nullp(typ) ((struct mlx5_ifc_##typ##_bits *)0) +#define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld) +#define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld)) +#define __mlx5_16_off(typ, fld) (__mlx5_bit_off(typ, fld) / 16) +#define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32) +#define __mlx5_64_off(typ, fld) (__mlx5_bit_off(typ, fld) / 64) +#define __mlx5_16_bit_off(typ, fld) (16 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0xf)) +#define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f)) +#define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) +#define __mlx5_dw_mask(typ, fld) (__mlx5_mask(typ, fld) << __mlx5_dw_bit_off(typ, fld)) +#define __mlx5_mask16(typ, fld) ((u16)((1ull << __mlx5_bit_sz(typ, fld)) - 1)) +#define __mlx5_16_mask(typ, fld) (__mlx5_mask16(typ, fld) << __mlx5_16_bit_off(typ, fld)) +#define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) + +/* Size and address macros */ +#define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) +#define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) +#define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) +#define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) +#define MLX5_UN_SZ_BYTES(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 8) +#define MLX5_UN_SZ_DW(typ) (sizeof(union mlx5_ifc_##typ##_bits) / 32) +#define MLX5_BYTE_OFF(typ, fld) (__mlx5_bit_off(typ, fld) / 8) +#define MLX5_ADDR_OF(typ, p, fld) ((void *)((u8 *)(p) + MLX5_BYTE_OFF(typ, fld))) + +/* insert a value to a struct */ +#define MLX5_SET(typ, p, fld, v) do { \ + u32 _v = v; \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ + cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ + (~__mlx5_dw_mask(typ, fld))) | (((_v) & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld))); \ +} while (0) + +#define MLX5_ARRAY_SET(typ, p, fld, idx, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 32); \ + MLX5_SET(typ, p, fld[idx], v); \ +} while (0) + +#define MLX5_SET_TO_ONES(typ, p, fld) do { \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + *((__be32 *)(p) + __mlx5_dw_off(typ, fld)) = \ + cpu_to_be32((be32_to_cpu(*((__be32 *)(p) + __mlx5_dw_off(typ, fld))) & \ + (~__mlx5_dw_mask(typ, fld))) | ((__mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld))); \ +} while (0) + +#define MLX5_GET(typ, p, fld) ((be32_to_cpu(*((__be32 *)(p) +\ +__mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \ +__mlx5_mask(typ, fld)) + +#define MLX5_GET_PR(typ, p, fld) ({ \ + u32 ___t = MLX5_GET(typ, p, fld); \ + pr_debug(#fld " = 0x%x\n", ___t); \ + ___t; \ +}) + +/* 64-bit field accessors */ +#define __MLX5_SET64(typ, p, fld, v) do { \ + BUILD_BUG_ON(__mlx5_bit_sz(typ, fld) != 64); \ + *((__be64 *)(p) + __mlx5_64_off(typ, fld)) = cpu_to_be64(v); \ +} while (0) + +#define MLX5_SET64(typ, p, fld, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + __MLX5_SET64(typ, p, fld, v); \ +} while (0) + +#define MLX5_ARRAY_SET64(typ, p, fld, idx, v) do { \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + __MLX5_SET64(typ, p, fld[idx], v); \ +} while (0) + +#define MLX5_ARRAY_GET64(typ, p, fld, idx) \ + ({ \ + BUILD_BUG_ON(__mlx5_bit_off(typ, fld) % 64); \ + be64_to_cpu( \ + *((__be64 *)(p) + __mlx5_64_off(typ, fld) + (idx))); \ + }) + +#define MLX5_GET64(typ, p, fld) be64_to_cpu(*((__be64 *)(p) + __mlx5_64_off(typ, fld))) + +#define MLX5_GET64_PR(typ, p, fld) ({ \ + u64 ___t = MLX5_GET64(typ, p, fld); \ + pr_debug(#fld " = 0x%llx\n", ___t); \ + ___t; \ +}) + +/* 16-bit field accessors */ +#define MLX5_GET16(typ, p, fld) ((be16_to_cpu(*((__be16 *)(p) +\ +__mlx5_16_off(typ, fld))) >> __mlx5_16_bit_off(typ, fld)) & \ +__mlx5_mask16(typ, fld)) + +#define MLX5_SET16(typ, p, fld, v) do { \ + u16 _v = v; \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 16); \ + *((__be16 *)(p) + __mlx5_16_off(typ, fld)) = \ + cpu_to_be16((be16_to_cpu(*((__be16 *)(p) + __mlx5_16_off(typ, fld))) & \ + (~__mlx5_16_mask(typ, fld))) | (((_v) & __mlx5_mask16(typ, fld)) \ + << __mlx5_16_bit_off(typ, fld))); \ +} while (0) + +/* Big endian getters */ +#define MLX5_GET64_BE(typ, p, fld) (*((__be64 *)(p) +\ + __mlx5_64_off(typ, fld))) + +#define MLX5_GET_BE(type_t, typ, p, fld) ({ \ + type_t tmp; \ + switch (sizeof(tmp)) { \ + case sizeof(u8): \ + tmp = (__force type_t)MLX5_GET(typ, p, fld); \ + break; \ + case sizeof(u16): \ + tmp = (__force type_t)cpu_to_be16(MLX5_GET(typ, p, fld)); \ + break; \ + case sizeof(u32): \ + tmp = (__force type_t)cpu_to_be32(MLX5_GET(typ, p, fld)); \ + break; \ + case sizeof(u64): \ + tmp = (__force type_t)MLX5_GET64_BE(typ, p, fld); \ + break; \ + } \ + tmp; \ + }) + +/* + * Use READ_ONCE/WRITE_ONCE for a single field that hardware may read/write + * unpredictably, mostly owner bits. All other bits in the DW must be stable. + * Usually a dma_wmb() will be required before a write and a dma_rmb() after a + * read. + */ +#define MLX5_GET_ONCE(typ, p, fld) \ + ((be32_to_cpu(READ_ONCE(*((__be32 *)(p) + __mlx5_dw_off(typ, fld)))) >> \ + __mlx5_dw_bit_off(typ, fld)) & \ + __mlx5_mask(typ, fld)) + +#define MLX5_SET_ONCE(typ, p, fld, v) \ + do { \ + u32 _v = v; \ + __be32 *_dw = (__be32 *)(p) + __mlx5_dw_off(typ, fld); \ + BUILD_BUG_ON(__mlx5_st_sz_bits(typ) % 32); \ + WRITE_ONCE(*_dw, \ + cpu_to_be32((be32_to_cpu(READ_ONCE(*_dw)) & \ + (~__mlx5_dw_mask(typ, fld))) | \ + (((_v) & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld)))); \ + } while (0) + +/* Access MMIO registers, usually the init segment, using IFC structs. */ +#define MLX5_GET_MMIO(typ, p, fld) \ + ((ioread32be(((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld))) >> \ + __mlx5_dw_bit_off(typ, fld)) & \ + __mlx5_mask(typ, fld)) + +/* The set is not relaxed so there is an integrated dma_wmb(). */ +#define MLX5_SET_MMIO(typ, p, fld, v) \ + do { \ + u32 _v = v; \ + void __iomem *_dw = \ + ((__be32 __iomem *)(p) + __mlx5_dw_off(typ, fld)); \ + if (__mlx5_bit_sz(typ, fld) == 32) \ + iowrite32be(_v, _dw); \ + else \ + iowrite32be((ioread32be(_dw) & \ + (~__mlx5_dw_mask(typ, fld))) | \ + ((_v & __mlx5_mask(typ, fld)) \ + << __mlx5_dw_bit_off(typ, fld)), \ + _dw); \ + } while (0) + +#endif /* MLX5_IFC_MACROS_H */ diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 9a1674c152aa..2f3fa2787737 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -8,6 +8,7 @@ * Author: Tom Lyon, pugs@cisco.com */ +#include <linux/device.h> #include <linux/mutex.h> #include <linux/pci.h> #include <linux/vfio.h> @@ -16,6 +17,7 @@ #include <linux/types.h> #include <linux/uuid.h> #include <linux/notifier.h> +#include <linux/pm.h> #ifndef VFIO_PCI_CORE_H #define VFIO_PCI_CORE_H @@ -168,7 +170,6 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev); void vfio_pci_core_release_dev(struct vfio_device *core_vdev); int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev); void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev); -extern const struct pci_error_handlers vfio_pci_core_err_handlers; int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev, int nr_virtfn); long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, @@ -193,8 +194,6 @@ int vfio_pci_core_match_token_uuid(struct vfio_device *core_vdev, int vfio_pci_core_enable(struct vfio_pci_core_device *vdev); void vfio_pci_core_disable(struct vfio_pci_core_device *vdev); void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev); -pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, - pci_channel_state_t state); ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, void __iomem *io, char __user *buf, loff_t off, size_t count, size_t x_start, @@ -260,4 +259,40 @@ vfio_pci_core_get_iomap(struct vfio_pci_core_device *vdev, unsigned int bar) int vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment, struct phys_vec *phys); +int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev); +int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev); +pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev); + +/* + * Per-driver PM/AER trampolines. driver_data points to the driver's private + * structure, which embeds a struct vfio_pci_core_device. + * + * @name: prefix for <name>_pm_ops and <name>_aer_err_detected + * @drv_type: type of the driver's private structure, including struct + * @member: member containing the struct vfio_pci_core_device + */ +#define VFIO_PCI_CORE_DEFINE_CALLBACKS(name, drv_type, member) \ +static int __maybe_unused name##_pm_suspend(struct device *dev) \ +{ \ + drv_type *drvdata = dev_get_drvdata(dev); \ + \ + return vfio_pci_core_runtime_suspend(&drvdata->member); \ +} \ +static int __maybe_unused name##_pm_resume(struct device *dev) \ +{ \ + drv_type *drvdata = dev_get_drvdata(dev); \ + \ + return vfio_pci_core_runtime_resume(&drvdata->member); \ +} \ +static const struct dev_pm_ops name##_pm_ops = { \ + SET_RUNTIME_PM_OPS(name##_pm_suspend, name##_pm_resume, NULL) \ +}; \ +static pci_ers_result_t name##_aer_err_detected(struct pci_dev *pdev, \ + pci_channel_state_t state) \ +{ \ + drv_type *drvdata = pci_get_drvdata(pdev); \ + \ + return vfio_pci_core_aer_err_detected(&drvdata->member); \ +} + #endif /* VFIO_PCI_CORE_H */ diff --git a/include/linux/vgaarb.h b/include/linux/vgaarb.h index 97129a1bbb7d..71a364669eaf 100644 --- a/include/linux/vgaarb.h +++ b/include/linux/vgaarb.h @@ -33,7 +33,8 @@ struct pci_dev *vga_default_device(void); void vga_set_default_device(struct pci_dev *pdev); int vga_remove_vgacon(struct pci_dev *pdev); int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool state)); + unsigned int (*set_decode)(void *data, bool state), + void *data); #else /* CONFIG_VGA_ARB */ static inline void vga_set_legacy_decoding(struct pci_dev *pdev, unsigned int decodes) @@ -59,7 +60,8 @@ static inline int vga_remove_vgacon(struct pci_dev *pdev) return 0; } static inline int vga_client_register(struct pci_dev *pdev, - unsigned int (*set_decode)(struct pci_dev *pdev, bool state)) + unsigned int (*set_decode)(void *data, bool state), + void *data) { return 0; } @@ -97,7 +99,7 @@ static inline int vga_get_uninterruptible(struct pci_dev *pdev, static inline void vga_client_unregister(struct pci_dev *pdev) { - vga_client_register(pdev, NULL); + vga_client_register(pdev, NULL, NULL); } #endif /* LINUX_VGA_H */ diff --git a/tools/arch/arm64/include/asm/barrier.h b/tools/arch/arm64/include/asm/barrier.h index 3b9b41331c4f..abdc64fc3c70 100644 --- a/tools/arch/arm64/include/asm/barrier.h +++ b/tools/arch/arm64/include/asm/barrier.h @@ -24,6 +24,10 @@ #define smp_wmb() asm volatile("dmb ishst" ::: "memory") #define smp_rmb() asm volatile("dmb ishld" ::: "memory") +/* DMA barriers use outer-shareable (osh) for device visibility */ +#define dma_rmb() asm volatile("dmb oshld" ::: "memory") +#define dma_wmb() asm volatile("dmb oshst" ::: "memory") + #define smp_store_release(p, v) \ do { \ union { typeof(*p) __val; char __c[1]; } __u = \ diff --git a/tools/arch/x86/include/asm/barrier.h b/tools/arch/x86/include/asm/barrier.h index 0adf295dd5b6..0b51431fa530 100644 --- a/tools/arch/x86/include/asm/barrier.h +++ b/tools/arch/x86/include/asm/barrier.h @@ -43,4 +43,9 @@ do { \ ___p1; \ }) #endif /* defined(__x86_64__) */ + +/* x86 is DMA-coherent so DMA barriers are just compiler barriers */ +#define dma_rmb() barrier() +#define dma_wmb() barrier() + #endif /* _TOOLS_LINUX_ASM_X86_BARRIER_H */ diff --git a/tools/include/asm-generic/io.h b/tools/include/asm-generic/io.h index e5a0b07ad452..0d89decdafb8 100644 --- a/tools/include/asm-generic/io.h +++ b/tools/include/asm-generic/io.h @@ -479,4 +479,32 @@ static inline void writesq(volatile void __iomem *addr, const void *buffer, } #endif +/* + * ioread/iowrite for big-endian MMIO registers. + */ + +#ifndef ioread32be +#define ioread32be ioread32be +static inline u32 ioread32be(const volatile void __iomem *addr) +{ + return bswap_32(readl(addr)); +} +#endif + +#ifndef iowrite32be +#define iowrite32be iowrite32be +static inline void iowrite32be(u32 value, volatile void __iomem *addr) +{ + writel(bswap_32(value), addr); +} +#endif + +#ifndef iowrite64be +#define iowrite64be iowrite64be +static inline void iowrite64be(u64 value, volatile void __iomem *addr) +{ + writeq(bswap_64(value), addr); +} +#endif + #endif /* _TOOLS_ASM_GENERIC_IO_H */ diff --git a/tools/include/asm/barrier.h b/tools/include/asm/barrier.h index 0c21678ac5e6..e7e0c7de5a2f 100644 --- a/tools/include/asm/barrier.h +++ b/tools/include/asm/barrier.h @@ -47,6 +47,14 @@ # define smp_mb() mb() #endif +#ifndef dma_rmb +# define dma_rmb() rmb() +#endif + +#ifndef dma_wmb +# define dma_wmb() wmb() +#endif + #ifndef smp_store_release # define smp_store_release(p, v) \ do { \ diff --git a/tools/include/linux/stddef.h b/tools/include/linux/stddef.h new file mode 100644 index 000000000000..99182ea4a141 --- /dev/null +++ b/tools/include/linux/stddef.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TOOLS_LINUX_STDDEF_H +#define _TOOLS_LINUX_STDDEF_H + +#include_next <linux/stddef.h> + +#define DECLARE_FLEX_ARRAY(TYPE, NAME) \ + __DECLARE_FLEX_ARRAY(TYPE, NAME) + +#endif /* _TOOLS_LINUX_STDDEF_H */ diff --git a/tools/testing/selftests/kvm/include/arm64/processor.h b/tools/testing/selftests/kvm/include/arm64/processor.h index b8a902ba8573..5878a6abdf20 100644 --- a/tools/testing/selftests/kvm/include/arm64/processor.h +++ b/tools/testing/selftests/kvm/include/arm64/processor.h @@ -12,6 +12,7 @@ #include <linux/stringify.h> #include <linux/types.h> +#include <asm/barrier.h> #include <asm/brk-imm.h> #include <asm/esr.h> #include <asm/sysreg.h> @@ -191,11 +192,8 @@ static inline void cpu_relax(void) #define dsb(opt) asm volatile("dsb " #opt : : : "memory") #define dmb(opt) asm volatile("dmb " #opt : : : "memory") -#define dma_wmb() dmb(oshst) #define __iowmb() dma_wmb() -#define dma_rmb() dmb(oshld) - #define __iormb(v) \ ({ \ unsigned long tmp; \ diff --git a/tools/testing/selftests/kvm/irq_test.c b/tools/testing/selftests/kvm/irq_test.c index 168c52734fef..791e576c334c 100644 --- a/tools/testing/selftests/kvm/irq_test.c +++ b/tools/testing/selftests/kvm/irq_test.c @@ -6,7 +6,6 @@ #include "proc_util.h" #include <libvfio.h> -#include <linux/sizes.h> #include <stdio.h> #include <stdlib.h> #include <unistd.h> @@ -101,7 +100,6 @@ static int vfio_setup_msi(struct vfio_pci_device *device) allocator = iova_allocator_init(device->iommu); region = &device->driver.region; - region->size = SZ_2M; region->iova = iova_allocator_alloc(allocator, region->size); region->vaddr = kvm_mmap(region->size, prot, flags, -1); TEST_ASSERT(region->vaddr != MAP_FAILED, "mmap() failed\n"); diff --git a/tools/testing/selftests/vfio/.gitignore b/tools/testing/selftests/vfio/.gitignore index 7fadc19d3bca..fc14bc18bcd8 100644 --- a/tools/testing/selftests/vfio/.gitignore +++ b/tools/testing/selftests/vfio/.gitignore @@ -8,3 +8,4 @@ !*.mk !.gitignore !Makefile +!settings diff --git a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c index 19d9630b24c2..40b8541b588e 100644 --- a/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c +++ b/tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c @@ -418,6 +418,7 @@ static void dsa_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops dsa_ops = { .name = "dsa", + .region_size = sizeof(struct dsa_state), .probe = dsa_probe, .init = dsa_init, .remove = dsa_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c index fd9e05d77ea4..2fafcc9f51c0 100644 --- a/tools/testing/selftests/vfio/lib/drivers/igb/igb.c +++ b/tools/testing/selftests/vfio/lib/drivers/igb/igb.c @@ -576,6 +576,7 @@ static void igb_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops igb_ops = { .name = "igb", + .region_size = sizeof(struct igb), .probe = igb_probe, .init = igb_init, .remove = igb_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c index a871b935542b..c9b28365c5eb 100644 --- a/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c +++ b/tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c @@ -226,6 +226,7 @@ static void ioat_send_msi(struct vfio_pci_device *device) const struct vfio_pci_driver_ops ioat_ops = { .name = "ioat", + .region_size = sizeof(struct ioat_state), .probe = ioat_probe, .init = ioat_init, .remove = ioat_remove, diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c new file mode 100644 index 000000000000..a65b52e183bc --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c @@ -0,0 +1,1928 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * mlx5 VFIO selftest driver + * + * Programs mlx5 ConnectX VFs and PFs through the bare-metal command interface + * and RDMA WRITE self-loopback to perform DMA. Implements vfio_pci_driver_ops + * (probe/init/remove) and plugs into the VFIO selftest framework. + */ +#include <sched.h> +#include <stdbool.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> + +#include <linux/errno.h> +#include <linux/io.h> +#include <linux/log2.h> +#include <linux/pci_regs.h> +#include <linux/pci_ids.h> + +#include <libvfio.h> + +#include "mlx5_hw.h" + +/* + * Driver state — overlaid on device->driver.region.vaddr. + * + * Contains both software-only state and HW-visible DMA buffers. HW buffers need + * strict IOVA alignment. + */ +struct mlx5st_device { + /* Back pointer */ + struct vfio_pci_device *device; + + /* BAR0 */ + struct mlx5st_initial_seg __iomem *init_seg; + void __iomem *bar0; + + /* Command interface */ + struct mlx5st_cmd_queue_entry *cmd_lay; + struct mlx5st_cmd_queue_entry *pages_cmd_lay; + u8 cmd_log_stride; + unsigned int pages_slot; + u8 cmd_token; + bool cmd_sig_enabled; + + /* PD */ + u32 pdn; + + /* Global PA-mode MKEY */ + u32 global_lkey; + u32 global_rkey; + u32 mkey_index; + + /* CQ */ + u32 cqn; + u32 cq_ci; + u32 cq_arm_sn; + + /* UAR */ + u32 uar_page; + void __iomem *uar_base; + unsigned int uar_bf_offset; + + /* EQ (cmd/pages events — polled, not interrupt-driven) */ + u32 eqn; + u32 eq_cons_index; + bool have_eq; + + /* MSI EQ (CQ completion events — fires MSI-X) */ + u32 msi_eqn; + u32 msi_eq_cons_index; + bool have_msi_eq; + + /* Async pages slot state */ + bool pages_slot_in_use; + bool pages_slot_is_reclaim; + unsigned int pages_reclaim_npages; + unsigned int pages_pending_give; + unsigned int pages_pending_reclaim; + u16 pages_pending_func_id; + bool pages_func_id_seen; + + /* QP */ + u32 qpn; + u32 sq_pi; + u32 sq_ci; + + /* FW pages bitmap */ + u64 fw_pages_bitmap[MAX_FW_PAGES / 64]; + u32 fw_pages_given; + u16 fw_func_id; + + /* Capabilities */ + bool fl_supported; + u8 log_max_msg; + + /* Buffers used by send_msi() to trigger an interrupt */ + u64 send_msi_src; + u64 send_msi_dst; + + /* + * HW-visible DMA buffers below — device reads/writes via DMA. + */ + struct mlx5st_cmd_queue_entry cmd_queue + [MLX5_HW_PAGE_SIZE / sizeof(struct mlx5st_cmd_queue_entry)] + __aligned(MLX5_HW_PAGE_SIZE); + struct mlx5st_send_wqe sq_buf[SQ_WQE_CNT]; + struct mlx5st_dbrec cq_dbrec; + struct mlx5st_dbrec qp_dbrec; + struct mlx5st_cqe64 cq_buf[CQ_CQE_CNT]; + + /* Slot 0 mailboxes (regular commands) */ + struct mlx5st_mbox_entry cmd_in_mbox[CMD_MBOX_NENT]; + struct mlx5st_mbox_entry cmd_out_mbox[CMD_MBOX_NENT]; + + /* Pages slot mailboxes (async MANAGE_PAGES) */ + struct mlx5st_mbox_entry pages_in_mbox[CMD_MBOX_NENT]; + struct mlx5st_mbox_entry pages_out_mbox[CMD_MBOX_NENT]; + + /* EQ does not support page_offset */ + struct mlx5st_eqe eq_buf[EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE); + + /* MSI EQ buffer — CQ completions generate EQEs here -> MSI-X */ + struct mlx5st_eqe msi_eq_buf[MSI_EQ_NENT] __aligned(MLX5_HW_PAGE_SIZE); + + u8 fw_pages[MAX_FW_PAGES][MLX5_HW_PAGE_SIZE] + __aligned(MLX5_HW_PAGE_SIZE); +}; + +/* Check against HW limits on IOVA alignment */ +static_assert(offsetof(struct mlx5st_device, cmd_in_mbox) % + CMD_MBOX_STRIDE == 0, + "cmd_in_mbox must be stride-aligned"); +static_assert(offsetof(struct mlx5st_device, pages_in_mbox) % + CMD_MBOX_STRIDE == 0, + "pages_in_mbox must be stride-aligned"); +static_assert(offsetof(struct mlx5st_device, cq_buf) % 64 == 0, + "cq_buf must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, sq_buf) % 64 == 0, + "sq_buf must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, cq_dbrec) % 64 == 0, + "cq_dbrec must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, qp_dbrec) % 64 == 0, + "qp_dbrec must be 64-byte aligned"); +static_assert(offsetof(struct mlx5st_device, eq_buf) % + MLX5_HW_PAGE_SIZE == 0, + "eq_buf must be page-aligned"); +static_assert(offsetof(struct mlx5st_device, msi_eq_buf) % + MLX5_HW_PAGE_SIZE == 0, + "msi_eq_buf must be page-aligned"); +static_assert(offsetof(struct mlx5st_device, fw_pages) % + MLX5_HW_PAGE_SIZE == 0, + "fw_pages must be page-aligned"); + +static struct mlx5st_device *to_mlx5st(struct vfio_pci_device *device) +{ + return device->driver.region.vaddr; +} + +/* + * Fill a PAS (Physical Address Segment) for a buffer in the driver region. + * Sets pas[0] to the page-aligned IOVA and returns the page_offset (the + * buffer's byte offset within that page, in units of 64 bytes). + */ +static unsigned int mlx5st_fill_pas(struct vfio_pci_device *device, void *buf, + __be64 *pas) +{ + u64 iova = to_iova(device, buf); + + pas[0] = cpu_to_be64(iova & ~(u64)(MLX5_HW_PAGE_SIZE - 1)); + return (iova & (MLX5_HW_PAGE_SIZE - 1)) / 64; +} + +/* + * Probe — match mlx5 devices by PCI vendor/device ID. + */ + +static int mlx5st_probe(struct vfio_pci_device *device) +{ + static const u16 mlx5st_pci_ids[] = { + 0x1011, /* Connect-IB */ + 0x1012, /* Connect-IB VF */ + 0x1013, /* ConnectX-4 */ + 0x1014, /* ConnectX-4 VF */ + 0x1015, /* ConnectX-4LX */ + 0x1016, /* ConnectX-4LX VF */ + 0x1017, /* ConnectX-5 */ + 0x1018, /* ConnectX-5 VF */ + 0x1019, /* ConnectX-5 Ex */ + 0x101a, /* ConnectX-5 Ex VF */ + 0x101b, /* ConnectX-6 */ + 0x101c, /* ConnectX-6 VF */ + 0x101d, /* ConnectX-6 Dx */ + 0x101e, /* ConnectX-6 Dx VF */ + 0x101f, /* ConnectX-6 LX */ + 0x1021, /* ConnectX-7 */ + 0x1023, /* ConnectX-8 */ + 0x1025, /* ConnectX-9 */ + 0x1027, /* ConnectX-10 */ + 0x2101, /* ConnectX-10 NVLink-C2C */ + 0xa2d2, /* BlueField integrated ConnectX-5 */ + 0xa2d3, /* BlueField integrated ConnectX-5 VF */ + 0xa2d6, /* BlueField-2 integrated ConnectX-6 Dx */ + 0xa2dc, /* BlueField-3 integrated ConnectX-7 */ + 0xa2df, /* BlueField-4 integrated ConnectX-8 */ + }; + unsigned int i; + u16 did; + + if (vfio_pci_config_readw(device, PCI_VENDOR_ID) != + PCI_VENDOR_ID_MELLANOX) + return -ENODEV; + + did = vfio_pci_config_readw(device, PCI_DEVICE_ID); + for (i = 0; i < ARRAY_SIZE(mlx5st_pci_ids); i++) { + if (mlx5st_pci_ids[i] == did) + return 0; + } + + return -ENODEV; +} + +/* + * Command interface + */ + +static u8 xor8_buf(const void *buf, size_t offset, size_t len) +{ + const u8 *p = buf; + u8 sum = 0; + size_t i; + + for (i = offset; i < offset + len; i++) + sum ^= p[i]; + return sum; +} + +#define CMD_IF_BOX_CTRL_OFF MLX5_BYTE_OFF(cmd_if_box, reserved_at_1000) +#define CMD_IF_BOX_CTRL_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, ctrl_signature) +#define CMD_IF_BOX_SIG_OFF MLX5_BYTE_OFF(cmd_if_box, signature) + +static void mlx5st_cmd_calc_block_sig(struct mlx5st_cmd_if_box *blk) +{ + MLX5_SET(cmd_if_box, blk, ctrl_signature, + ~xor8_buf(blk, CMD_IF_BOX_CTRL_OFF, + CMD_IF_BOX_CTRL_SIG_OFF - CMD_IF_BOX_CTRL_OFF)); + MLX5_SET(cmd_if_box, blk, signature, + ~xor8_buf(blk, 0, CMD_IF_BOX_SIG_OFF)); +} + +static int mlx5st_cmd_verify_block_sig(struct mlx5st_cmd_if_box *blk) +{ + if (xor8_buf(blk, CMD_IF_BOX_CTRL_OFF, + CMD_IF_BOX_SIG_OFF - CMD_IF_BOX_CTRL_OFF) != 0xff) + return -1; + if (xor8_buf(blk, 0, sizeof(struct mlx5st_cmd_if_box)) != 0xff) + return -1; + return 0; +} + +static unsigned int mlx5st_cmd_setup_mbox_chain(struct vfio_pci_device *device, + struct mlx5st_mbox_entry *mbox, + unsigned int nblocks, u8 token) +{ + unsigned int i; + + for (i = 0; i < nblocks; i++) { + struct mlx5st_cmd_if_box *blk = &mbox[i].block; + u64 next_iova; + + memset(blk, 0, sizeof(struct mlx5st_cmd_if_box)); + MLX5_SET(cmd_if_box, blk, block_number, i); + MLX5_SET(cmd_if_box, blk, token, token); + if (i < nblocks - 1) { + next_iova = to_iova(device, &mbox[i + 1]); + MLX5_SET(cmd_if_box, blk, next_pointer_63_32, + next_iova >> 32); + MLX5_SET(cmd_if_box, blk, next_pointer_31_10, + (u32)next_iova >> 10); + } + } + return nblocks; +} + +static void mlx5st_cmd_copy_to_mbox(struct mlx5st_mbox_entry *mbox, + const void *data, unsigned int len) +{ + const u8 *src = data; + unsigned int i = 0; + + while (len > 0) { + unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ? + len : + MLX5_CMD_DATA_BLOCK_SIZE; + + memcpy(MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data), + src, chunk); + src += chunk; + len -= chunk; + i++; + } +} + +static void mlx5st_cmd_copy_from_mbox(void *data, + const struct mlx5st_mbox_entry *mbox, + unsigned int len) +{ + unsigned int i = 0; + u8 *dst = data; + + while (len > 0) { + unsigned int chunk = len < MLX5_CMD_DATA_BLOCK_SIZE ? + len : + MLX5_CMD_DATA_BLOCK_SIZE; + + memcpy(dst, + MLX5_ADDR_OF(cmd_if_box, &mbox[i].block, mailbox_data), + chunk); + dst += chunk; + len -= chunk; + i++; + } +} + +/* Forward declaration — cmd_exec polls events during command wait */ +static void mlx5st_process_events(struct mlx5st_device *dev); + +static const char *mlx5st_cmd_name(u16 opcode) +{ + switch (opcode) { + case MLX5_CMD_OP_QUERY_HCA_CAP: return "QUERY_HCA_CAP"; + case MLX5_CMD_OP_INIT_HCA: return "INIT_HCA"; + case MLX5_CMD_OP_TEARDOWN_HCA: return "TEARDOWN_HCA"; + case MLX5_CMD_OP_ENABLE_HCA: return "ENABLE_HCA"; + case MLX5_CMD_OP_DISABLE_HCA: return "DISABLE_HCA"; + case MLX5_CMD_OP_QUERY_PAGES: return "QUERY_PAGES"; + case MLX5_CMD_OP_MANAGE_PAGES: return "MANAGE_PAGES"; + case MLX5_CMD_OP_SET_HCA_CAP: return "SET_HCA_CAP"; + case MLX5_CMD_OP_SET_ISSI: return "SET_ISSI"; + case MLX5_CMD_OP_CREATE_MKEY: return "CREATE_MKEY"; + case MLX5_CMD_OP_DESTROY_MKEY: return "DESTROY_MKEY"; + case MLX5_CMD_OP_CREATE_EQ: return "CREATE_EQ"; + case MLX5_CMD_OP_DESTROY_EQ: return "DESTROY_EQ"; + case MLX5_CMD_OP_CREATE_CQ: return "CREATE_CQ"; + case MLX5_CMD_OP_DESTROY_CQ: return "DESTROY_CQ"; + case MLX5_CMD_OP_CREATE_QP: return "CREATE_QP"; + case MLX5_CMD_OP_DESTROY_QP: return "DESTROY_QP"; + case MLX5_CMD_OP_RST2INIT_QP: return "RST2INIT_QP"; + case MLX5_CMD_OP_INIT2RTR_QP: return "INIT2RTR_QP"; + case MLX5_CMD_OP_RTR2RTS_QP: return "RTR2RTS_QP"; + case MLX5_CMD_OP_ALLOC_PD: return "ALLOC_PD"; + case MLX5_CMD_OP_DEALLOC_PD: return "DEALLOC_PD"; + case MLX5_CMD_OP_ALLOC_UAR: return "ALLOC_UAR"; + case MLX5_CMD_OP_DEALLOC_UAR: return "DEALLOC_UAR"; + default: return "UNKNOWN"; + } +} + +/* + * Post a command on a given slot: fill the cmd_queue_entry, set up mailbox + * chains, compute signatures, hand ownership to FW, and ring the doorbell. + */ +static void mlx5st_cmd_post(struct mlx5st_device *dev, + struct mlx5st_cmd_queue_entry *cmd, + struct mlx5st_mbox_entry *in_mbox, + struct mlx5st_mbox_entry *out_mbox, + void *in, unsigned int ilen, unsigned int olen, + u32 doorbell) +{ + struct vfio_pci_device *device = dev->device; + unsigned int in_remain, out_remain, in_nblk, out_nblk; + unsigned int i; + void *cin, *cout; + u8 token; + + /* Rotating non-zero token ties cmd entry to its mailbox blocks */ + token = ++dev->cmd_token; + if (!token) + token = ++dev->cmd_token; + + in_remain = ilen > MLX5_CMD_INLINE_SZ ? ilen - MLX5_CMD_INLINE_SZ : 0; + out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0; + in_nblk = (in_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + + /* Set up mailbox chains */ + if (in_nblk > 0) { + mlx5st_cmd_setup_mbox_chain(device, in_mbox, in_nblk, token); + mlx5st_cmd_copy_to_mbox(in_mbox, + (u8 *)in + MLX5_CMD_INLINE_SZ, + in_remain); + } + if (out_nblk > 0) + mlx5st_cmd_setup_mbox_chain(device, out_mbox, out_nblk, token); + + /* Copy inline input */ + cin = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_input_inline_data); + memset(cin, 0, MLX5_CMD_INLINE_SZ); + memcpy(cin, in, ilen < MLX5_CMD_INLINE_SZ ? ilen : MLX5_CMD_INLINE_SZ); + MLX5_SET(cmd_queue_entry, cmd, input_length, ilen); + MLX5_SET(cmd_queue_entry, cmd, token, token); + + /* Zero inline output */ + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + memset(cout, 0, MLX5_CMD_INLINE_SZ); + MLX5_SET(cmd_queue_entry, cmd, output_length, olen); + + /* + * Compute signatures: mailbox blocks first, then cmd_queue_entry. + * The sig must cover the final state including ownership=0x1, but + * we must not set ownership until after the sig is in place - + * XOR in the 0x1 without storing it to memory. + */ + for (i = 0; i < in_nblk; i++) + mlx5st_cmd_calc_block_sig(&in_mbox[i].block); + for (i = 0; i < out_nblk; i++) + mlx5st_cmd_calc_block_sig(&out_mbox[i].block); + MLX5_SET(cmd_queue_entry, cmd, signature, 0); + MLX5_SET(cmd_queue_entry, cmd, signature, + ~(xor8_buf(cmd, 0, sizeof(struct mlx5st_cmd_queue_entry)) ^ + 0x1)); + + /* Ensure all cmd data (including sig) is visible, then hand to FW */ + dma_wmb(); + MLX5_SET_ONCE(cmd_queue_entry, cmd, ownership, 1); + + /* Ring doorbell */ + MLX5_SET_MMIO(initial_seg, dev->init_seg, command_doorbell_vector, + doorbell); +} + +static void mlx5st_cmd_exec(struct mlx5st_device *dev, void *in, + unsigned int ilen, void *out, unsigned int olen) +{ + struct mlx5st_cmd_queue_entry *cmd = dev->cmd_lay; + unsigned int out_remain, out_nblk; + struct timespec start, now; + unsigned int elapsed; + unsigned int i; + void *cout; + + mlx5st_cmd_post(dev, cmd, dev->cmd_in_mbox, dev->cmd_out_mbox, in, + ilen, olen, 1); + + out_remain = olen > MLX5_CMD_INLINE_SZ ? olen - MLX5_CMD_INLINE_SZ : 0; + out_nblk = (out_remain + MLX5_CMD_DATA_BLOCK_SIZE - 1) / + MLX5_CMD_DATA_BLOCK_SIZE; + + /* Poll for completion — also process EQ events for PF page requests */ + clock_gettime(CLOCK_MONOTONIC, &start); + for (;;) { + if (!MLX5_GET_ONCE(cmd_queue_entry, cmd, ownership)) + break; + if (dev->have_eq) + mlx5st_process_events(dev); + sched_yield(); + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > MLX5_CMD_TIMEOUT_MS) + VFIO_FAIL("cmd timeout after %d ms", elapsed); + } + /* Ensure output data reads happen after ownership is seen clear */ + dma_rmb(); + + /* Verify output signatures when FW has checksums enabled */ + if (dev->cmd_sig_enabled) { + if (xor8_buf(cmd, 0, + sizeof(struct mlx5st_cmd_queue_entry)) != 0xff) + VFIO_FAIL("cmd output signature mismatch"); + for (i = 0; i < out_nblk; i++) { + if (mlx5st_cmd_verify_block_sig( + &dev->cmd_out_mbox[i].block)) + VFIO_FAIL("cmd output mailbox block %d signature mismatch", + i); + } + } + + /* Copy output: inline first */ + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + memcpy(out, cout, olen < MLX5_CMD_INLINE_SZ ? olen : MLX5_CMD_INLINE_SZ); + + /* Copy remaining from output mailbox chain */ + if (out_remain > 0) + mlx5st_cmd_copy_from_mbox((u8 *)out + MLX5_CMD_INLINE_SZ, + dev->cmd_out_mbox, out_remain); + + /* Check command status */ + if (MLX5_GET(enable_hca_out, out, status) != MLX5_CMD_STAT_OK) + VFIO_FAIL("%s: status=0x%x syndrome=0x%x", + mlx5st_cmd_name(MLX5_GET(enable_hca_in, in, opcode)), + MLX5_GET(enable_hca_out, out, status), + MLX5_GET(enable_hca_out, out, syndrome)); +} + +static struct mlx5st_cmd_queue_entry * +mlx5st_cmd_slot_init(struct mlx5st_device *dev, unsigned int slot, + struct mlx5st_mbox_entry *in_mbox, + struct mlx5st_mbox_entry *out_mbox) +{ + struct vfio_pci_device *device = dev->device; + struct mlx5st_cmd_queue_entry *cmd = + &dev->cmd_queue[(slot << dev->cmd_log_stride) / + sizeof(struct mlx5st_cmd_queue_entry)]; + u64 iova; + + MLX5_SET(cmd_queue_entry, cmd, type, + MLX5_CMD_QUEUE_ENTRY_TYPE_PCIE_CMD_IF_TRANSPORT); + iova = to_iova(device, in_mbox); + MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_63_32, + iova >> 32); + MLX5_SET(cmd_queue_entry, cmd, input_mailbox_pointer_31_9, iova >> 9); + iova = to_iova(device, out_mbox); + MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_63_32, + iova >> 32); + MLX5_SET(cmd_queue_entry, cmd, output_mailbox_pointer_31_9, + iova >> 9); + return cmd; +} + +static void mlx5st_cmd_init(struct mlx5st_device *dev) +{ + struct mlx5st_initial_seg __iomem *seg = dev->init_seg; + struct vfio_pci_device *device = dev->device; + u16 cmdif_rev; + u8 log_sz; + u64 iova; + + cmdif_rev = MLX5_GET_MMIO(initial_seg, seg, cmd_interface_rev); + VFIO_ASSERT_EQ(cmdif_rev, 5); + + /* Read command queue geometry from BAR */ + log_sz = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_size); + dev->cmd_log_stride = MLX5_GET_MMIO(initial_seg, seg, log_cmdq_stride); + dev->pages_slot = (1 << log_sz) - 1; + + VFIO_ASSERT_LE((unsigned int)(1 << log_sz), 32u); + VFIO_ASSERT_GE((unsigned int)(1 << dev->cmd_log_stride), + (unsigned int)sizeof(struct mlx5st_cmd_queue_entry)); + VFIO_ASSERT_LE((unsigned int)((dev->pages_slot + 1) << + dev->cmd_log_stride), + (unsigned int)sizeof(dev->cmd_queue)); + + /* Set up slot 0 — regular commands */ + dev->cmd_lay = mlx5st_cmd_slot_init(dev, 0, dev->cmd_in_mbox, + dev->cmd_out_mbox); + + /* Set up pages slot — async MANAGE_PAGES */ + dev->pages_cmd_lay = mlx5st_cmd_slot_init(dev, dev->pages_slot, + dev->pages_in_mbox, + dev->pages_out_mbox); + + /* Write command queue page address to BAR0 */ + iova = to_iova(device, dev->cmd_queue); + MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_63_32, iova >> 32); + MLX5_SET_MMIO(initial_seg, seg, cmdq_phy_addr_31_12, iova >> 12); + + dev_dbg(device, + "Command interface initialized (cmdif_rev=5, log_sz=%u, log_stride=%u, pages_slot=%u)\n", + log_sz, dev->cmd_log_stride, dev->pages_slot); +} + +/* + * FW pages: bitmap allocator + MANAGE_PAGES + */ + +static void mlx5st_fw_pages_alloc(struct mlx5st_device *dev, + unsigned int npages, u64 *iovas) +{ + struct vfio_pci_device *device = dev->device; + unsigned int found = 0; + unsigned int w, b; + u64 word; + + for (w = 0; w < MAX_FW_PAGES / 64 && found < npages; w++) { + word = dev->fw_pages_bitmap[w]; + + for (b = 0; b < 64 && found < npages; b++) { + if (!(word & (1ULL << b))) { + unsigned int idx = w * 64 + b; + + dev->fw_pages_bitmap[w] |= (1ULL << b); + iovas[found++] = to_iova(device, + dev->fw_pages[idx]); + } + } + } + VFIO_ASSERT_EQ(found, npages); + dev->fw_pages_given += npages; +} + +static void mlx5st_fw_pages_free(struct mlx5st_device *dev, + unsigned int npages, const u64 *iovas) +{ + struct vfio_pci_device *device = dev->device; + unsigned int i, idx; + u64 off; + + for (i = 0; i < npages; i++) { + off = iovas[i] - to_iova(device, dev->fw_pages); + idx = off / MLX5_HW_PAGE_SIZE; + + VFIO_ASSERT_TRUE(idx < MAX_FW_PAGES); + dev->fw_pages_bitmap[idx / 64] &= ~(1ULL << (idx % 64)); + } + dev->fw_pages_given -= npages; +} + +static void *mlx5st_build_manage_pages_give(u16 func_id, unsigned int npages, + const u64 *iovas, + unsigned int *out_inlen) +{ + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in) + npages * 8; + unsigned int i; + void *in; + + in = calloc(1, inlen); + VFIO_ASSERT_NOT_NULL(in); + + MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_ALLOCATION_SUCCESS); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, npages); + + for (i = 0; i < npages; i++) + MLX5_ARRAY_SET64(manage_pages_in, in, pas, i, iovas[i]); + + *out_inlen = inlen; + return in; +} + +static void mlx5st_fw_pages_give_one(struct mlx5st_device *dev, u16 func_id, + unsigned int npages, u64 *iovas) +{ + u32 out[MLX5_ST_SZ_DW(manage_pages_out)] = {}; + unsigned int inlen; + void *in; + + in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen); + mlx5st_cmd_exec(dev, in, inlen, out, sizeof(out)); + free(in); +} + +static void mlx5st_fw_pages_give(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int remaining = npages; + u64 *iovas; + + if (!npages) + return; + + iovas = calloc(npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(iovas); + + mlx5st_fw_pages_alloc(dev, npages, iovas); + + /* Batch into chunks that fit in one mailbox */ + for (unsigned int off = 0; remaining > 0;) { + unsigned int batch = remaining < MAX_FW_PAGES_PER_CMD ? + remaining : + MAX_FW_PAGES_PER_CMD; + + mlx5st_fw_pages_give_one(dev, func_id, batch, iovas + off); + off += batch; + remaining -= batch; + } + + dev_dbg(dev->device, "MANAGE_PAGES GIVE: %d pages to func_id=%u\n", + npages, func_id); + free(iovas); +} + +static void mlx5st_fw_pages_satisfy(struct mlx5st_device *dev, int boot) +{ + u32 qo[MLX5_ST_SZ_DW(query_pages_out)] = {}; + u32 qi[MLX5_ST_SZ_DW(query_pages_in)] = {}; + u16 func_id; + int npages; + + MLX5_SET(query_pages_in, qi, opcode, MLX5_CMD_OP_QUERY_PAGES); + MLX5_SET(query_pages_in, qi, op_mod, boot ? 0x01 : 0x02); + mlx5st_cmd_exec(dev, qi, sizeof(qi), qo, sizeof(qo)); + + npages = MLX5_GET(query_pages_out, qo, num_pages); + func_id = MLX5_GET(query_pages_out, qo, function_id); + dev_dbg(dev->device, "QUERY_PAGES (%s): %d pages (func_id=%u)\n", + boot ? "boot" : "init", npages, func_id); + + if (npages > 0) { + dev->fw_func_id = func_id; + mlx5st_fw_pages_give(dev, func_id, npages); + } +} + +/* + * Async MANAGE_PAGES on the pages command slot. + * + * On PFs, firmware sends PAGE_REQUEST events via the EQ during command + * execution. We must respond with MANAGE_PAGES on a second command slot + * before the first (regular) command can complete. + */ + +static void mlx5st_pages_slot_post(struct mlx5st_device *dev, void *in, + unsigned int ilen, unsigned int olen) +{ + mlx5st_cmd_post(dev, dev->pages_cmd_lay, dev->pages_in_mbox, + dev->pages_out_mbox, in, ilen, olen, + 1 << dev->pages_slot); +} + +static void mlx5st_pages_slot_give(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int inlen; + u64 *iovas; + void *in; + + iovas = calloc(npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(iovas); + + mlx5st_fw_pages_alloc(dev, npages, iovas); + + in = mlx5st_build_manage_pages_give(func_id, npages, iovas, &inlen); + free(iovas); + + mlx5st_pages_slot_post(dev, in, inlen, + MLX5_ST_SZ_BYTES(manage_pages_out)); + dev->pages_slot_in_use = true; + dev->pages_slot_is_reclaim = false; + free(in); + + dev_dbg(dev->device, + "PAGE_REQUEST: %d pages given async to func_id=%u\n", + npages, func_id); +} + +static void mlx5st_pages_slot_reclaim(struct mlx5st_device *dev, u16 func_id, + unsigned int npages) +{ + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in); + unsigned int outlen = + MLX5_ST_SZ_BYTES(manage_pages_out) + npages * 8; + void *in; + + in = calloc(1, inlen); + VFIO_ASSERT_NOT_NULL(in); + + MLX5_SET(manage_pages_in, in, opcode, MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, npages); + + mlx5st_pages_slot_post(dev, in, inlen, outlen); + dev->pages_slot_in_use = true; + dev->pages_slot_is_reclaim = true; + dev->pages_reclaim_npages = npages; + free(in); + + dev_dbg(dev->device, + "PAGE_REQUEST: reclaim %d pages async from func_id=%u\n", + npages, func_id); +} + +static void mlx5st_pages_slot_kick(struct mlx5st_device *dev) +{ + unsigned int batch; + + if (dev->pages_slot_in_use) + return; + + if (dev->pages_pending_give) { + batch = dev->pages_pending_give < MAX_FW_PAGES_PER_CMD ? + dev->pages_pending_give : + MAX_FW_PAGES_PER_CMD; + dev->pages_pending_give -= batch; + mlx5st_pages_slot_give(dev, dev->pages_pending_func_id, batch); + } else if (dev->pages_pending_reclaim) { + batch = dev->pages_pending_reclaim < MAX_FW_PAGES_PER_CMD ? + dev->pages_pending_reclaim : + MAX_FW_PAGES_PER_CMD; + dev->pages_pending_reclaim -= batch; + mlx5st_pages_slot_reclaim(dev, dev->pages_pending_func_id, + batch); + } +} + +static void mlx5st_fw_pages_give_async(struct mlx5st_device *dev, + u16 func_id, unsigned int npages) +{ + if (!npages) + return; + + dev->pages_pending_give += npages; + dev->pages_pending_func_id = func_id; + mlx5st_pages_slot_kick(dev); +} + +static void mlx5st_fw_pages_reclaim_async(struct mlx5st_device *dev, + u16 func_id, unsigned int npages) +{ + dev->pages_pending_reclaim += npages; + dev->pages_pending_func_id = func_id; + mlx5st_pages_slot_kick(dev); +} + +static void mlx5st_pages_slot_complete(struct mlx5st_device *dev) +{ + struct mlx5st_cmd_queue_entry *cmd = dev->pages_cmd_lay; + void *cout; + + dma_rmb(); + + cout = MLX5_ADDR_OF(cmd_queue_entry, cmd, command_output_inline_data); + if (MLX5_GET(enable_hca_out, cout, status) != MLX5_CMD_STAT_OK) + VFIO_FAIL("async MANAGE_PAGES failed: status=0x%x syndrome=0x%x", + MLX5_GET(enable_hca_out, cout, status), + MLX5_GET(enable_hca_out, cout, syndrome)); + + if (dev->pages_slot_is_reclaim) { + unsigned int outlen = MLX5_ST_SZ_BYTES(manage_pages_out) + + dev->pages_reclaim_npages * 8; + unsigned int num_claimed; + unsigned int i; + void *out; + u64 *iovas; + + out = calloc(1, outlen); + iovas = calloc(dev->pages_reclaim_npages, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(out); + VFIO_ASSERT_NOT_NULL(iovas); + + /* Copy inline output */ + memcpy(out, cout, MLX5_CMD_INLINE_SZ); + if (outlen > MLX5_CMD_INLINE_SZ) + mlx5st_cmd_copy_from_mbox( + (u8 *)out + MLX5_CMD_INLINE_SZ, + dev->pages_out_mbox, + outlen - MLX5_CMD_INLINE_SZ); + + num_claimed = + MLX5_GET(manage_pages_out, out, output_num_entries); + for (i = 0; i < num_claimed; i++) + iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas, + i); + + mlx5st_fw_pages_free(dev, num_claimed, iovas); + dev_dbg(dev->device, "PAGE_REQUEST: reclaimed %d pages\n", + num_claimed); + + free(iovas); + free(out); + } + + dev->pages_slot_in_use = false; + mlx5st_pages_slot_kick(dev); +} + +/* + * UAR alloc/dealloc + */ + +static void mlx5st_alloc_uar(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(alloc_uar_out)] = {}; + u32 in[MLX5_ST_SZ_DW(alloc_uar_in)] = {}; + + MLX5_SET(alloc_uar_in, in, opcode, MLX5_CMD_OP_ALLOC_UAR); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->uar_page = MLX5_GET(alloc_uar_out, out, uar); + dev->uar_base = (u8 __iomem*)dev->bar0 + dev->uar_page * MLX5_HW_PAGE_SIZE; + dev->uar_bf_offset = MLX5_BF_OFFSET; + + dev_dbg(dev->device, + "Allocated UAR page_id=%u, doorbell offset=0x%x\n", + dev->uar_page, + dev->uar_page * MLX5_HW_PAGE_SIZE + MLX5_BF_OFFSET); +} + +static void mlx5st_dealloc_uar(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(dealloc_uar_out)] = {}; + u32 in[MLX5_ST_SZ_DW(dealloc_uar_in)] = {}; + + MLX5_SET(dealloc_uar_in, in, opcode, MLX5_CMD_OP_DEALLOC_UAR); + MLX5_SET(dealloc_uar_in, in, uar, dev->uar_page); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * EQ infrastructure + */ + +static struct mlx5st_eqe *mlx5st_eq_get_eqe(struct mlx5st_device *dev, u32 cc) +{ + u32 ci = dev->eq_cons_index + cc; + struct mlx5st_eqe *eqe = &dev->eq_buf[ci % EQ_NENT]; + u8 owner = MLX5_GET_ONCE(eqe, eqe, owner); + u8 expected = !!(ci & EQ_NENT); + + if (owner != expected) + return NULL; + dma_rmb(); + return eqe; +} + +static void mlx5st_eq_update_ci(struct mlx5st_device *dev, u32 cc, bool arm) +{ + u32 val; + + dev->eq_cons_index += cc; + val = (dev->eq_cons_index & 0xffffff) | (dev->eqn << 24); + iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET + + (arm ? 0 : 8)); +} + +static void mlx5st_create_eq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {}; + struct mlx5_ifc_eqc_bits *eqc; + unsigned int i; + __be64 *pas; + + /* Initialize EQE owner bits */ + for (i = 0; i < EQ_NENT; i++) { + struct mlx5st_eqe *eqe = &dev->eq_buf[i]; + + MLX5_SET_ONCE(eqe, eqe, owner, 1); + } + + MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ); + + /* Subscribe to CMD completions and PAGE_REQUEST events */ + MLX5_ARRAY_SET64(create_eq_in, in, event_bitmask, 0, + (1ULL << MLX5_EVENT_TYPE_CMD) | + (1ULL << MLX5_EVENT_TYPE_PAGE_REQUEST)); + + eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry); + MLX5_SET(eqc, eqc, log_eq_size, LOG_EQ_SIZE); + MLX5_SET(eqc, eqc, uar_page, dev->uar_page); + pas = MLX5_ADDR_OF(create_eq_in, in, pas); + VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->eq_buf, pas), 0u); + MLX5_SET(eqc, eqc, log_page_size, 0); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->eqn = MLX5_GET(create_eq_out, out, eq_number); + dev->eq_cons_index = 0; + mlx5st_eq_update_ci(dev, 0, 0); + dev->have_eq = true; + + dev_dbg(device, "Created EQ: eqn=%u, %d entries (CMD+PAGE_REQUEST)\n", + dev->eqn, EQ_NENT); +} + +static void mlx5st_destroy_eq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {}; + + MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ); + MLX5_SET(destroy_eq_in, in, eq_number, dev->eqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * Drain all pending EQ events. Dispatches PAGE_REQUEST to the async pages + * slot and CMD completions to the pages slot completion handler. + */ +static void mlx5st_process_events(struct mlx5st_device *dev) +{ + struct mlx5st_eqe *eqe; + u32 cc = 0; + + while ((eqe = mlx5st_eq_get_eqe(dev, cc))) { + u8 type = MLX5_GET(eqe, eqe, event_type); + + switch (type) { + case MLX5_EVENT_TYPE_PAGE_REQUEST: { + void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data); + u16 func_id = MLX5_GET(pages_req_event, evdata, + function_id); + s32 npages = (s32)MLX5_GET(pages_req_event, evdata, + num_pages); + + /* + * The selftest doesn't use more than one func_id so a + * simple counter approach is possible. + */ + if (dev->pages_func_id_seen) + VFIO_ASSERT_EQ(func_id, + dev->pages_pending_func_id); + dev->pages_func_id_seen = true; + + if (npages > 0) + mlx5st_fw_pages_give_async(dev, func_id, + npages); + else if (npages < 0) + mlx5st_fw_pages_reclaim_async(dev, func_id, + -npages); + break; + } + case MLX5_EVENT_TYPE_CMD: { + void *evdata = MLX5_ADDR_OF(eqe, eqe, event_data); + u32 vector = MLX5_GET(cmd_inter_comp_event, evdata, + command_completion_vector); + + if (vector & (1U << dev->pages_slot)) + mlx5st_pages_slot_complete(dev); + break; + } + default: + break; + } + cc++; + } + + if (cc) + mlx5st_eq_update_ci(dev, cc, 0); +} + +/* + * MSI EQ — dedicated EQ for CQ completion events that fires MSI-X. + * Separate from the cmd/pages EQ so that only CQ completions (from + * send_msi or memcpy) trigger the interrupt vector. + */ + +static void mlx5st_msi_eq_drain(struct mlx5st_device *dev) +{ + u32 cc = 0; + u32 val; + + while (cc < MSI_EQ_NENT) { + u32 ci = dev->msi_eq_cons_index + cc; + struct mlx5st_eqe *eqe = + &dev->msi_eq_buf[ci % MSI_EQ_NENT]; + + if (MLX5_GET_ONCE(eqe, eqe, owner) != !!(ci & MSI_EQ_NENT)) + break; + cc++; + } + + /* Update consumer index and re-arm for next interrupt */ + dev->msi_eq_cons_index += cc; + val = (dev->msi_eq_cons_index & 0xffffff) | (dev->msi_eqn << 24); + iowrite32be(val, (u8 __iomem *)dev->uar_base + MLX5_EQ_DOORBELL_OFFSET); +} + +static void mlx5st_create_msi_eq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_eq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {}; + struct mlx5_ifc_eqc_bits *eqc; + unsigned int i; + __be64 *pas; + + /* Initialize EQE owner bits */ + for (i = 0; i < MSI_EQ_NENT; i++) { + struct mlx5st_eqe *eqe = &dev->msi_eq_buf[i]; + + MLX5_SET_ONCE(eqe, eqe, owner, 1); + } + + MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ); + + /* + * No event_bitmask — completion events are routed to this EQ via + * the CQ's c_eqn field, not through CREATE_EQ subscription. + */ + eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry); + MLX5_SET(eqc, eqc, log_eq_size, LOG_MSI_EQ_SIZE); + MLX5_SET(eqc, eqc, uar_page, dev->uar_page); + MLX5_SET(eqc, eqc, intr, MSI_VECTOR); + pas = MLX5_ADDR_OF(create_eq_in, in, pas); + VFIO_ASSERT_EQ(mlx5st_fill_pas(device, dev->msi_eq_buf, pas), 0u); + MLX5_SET(eqc, eqc, log_page_size, 0); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->msi_eqn = MLX5_GET(create_eq_out, out, eq_number); + dev->msi_eq_cons_index = 0; + dev->have_msi_eq = true; + mlx5st_msi_eq_drain(dev); + + dev_dbg(device, + "Created MSI EQ: eqn=%u, %d entries (COMP), vector=%d\n", + dev->msi_eqn, MSI_EQ_NENT, MSI_VECTOR); +} + +static void mlx5st_destroy_msi_eq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_eq_in)] = {}; + + MLX5_SET(destroy_eq_in, in, opcode, MLX5_CMD_OP_DESTROY_EQ); + MLX5_SET(destroy_eq_in, in, eq_number, dev->msi_eqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * HCA init / teardown + */ + +#define FW_INIT_TIMEOUT_MS 120000 +#define FW_INIT_WAIT_MS 200 + +static void mlx5st_wait_fw_init(struct mlx5st_device *dev) +{ + struct timespec start, now; + unsigned int elapsed; + + clock_gettime(CLOCK_MONOTONIC, &start); + while (MLX5_GET_MMIO(initial_seg, dev->init_seg, initializing)) { + usleep(FW_INIT_WAIT_MS * 1000); + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > FW_INIT_TIMEOUT_MS) + VFIO_FAIL("FW init timeout after %d ms", elapsed); + } +} + +static void mlx5st_set_issi(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(set_issi_out)] = {}; + u32 in[MLX5_ST_SZ_DW(set_issi_in)] = {}; + + MLX5_SET(set_issi_in, in, opcode, MLX5_CMD_OP_SET_ISSI); + MLX5_SET(set_issi_in, in, current_issi, 1); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "SET_ISSI: OK (issi=1)\n"); +} + +static void mlx5st_set_hca_caps(struct mlx5st_device *dev) +{ + u32 qout[MLX5_ST_SZ_DW(query_hca_cap_out)] = {}; + u32 qin[MLX5_ST_SZ_DW(query_hca_cap_in)] = {}; + u32 sout[MLX5_ST_SZ_DW(set_hca_cap_out)] = {}; + u32 sin[MLX5_ST_SZ_DW(set_hca_cap_in)] = {}; + struct mlx5_ifc_cmd_hca_cap_bits *set_hca_cap; + u32 max_checksum; + + /* Query max caps to learn cmdif_checksum support */ + MLX5_SET(query_hca_cap_in, qin, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, qin, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1)); + mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout)); + + max_checksum = MLX5_GET( + cmd_hca_cap, + MLX5_ADDR_OF(query_hca_cap_out, qout, capability), + cmdif_checksum); + + /* Query current caps as base for SET */ + memset(qout, 0, sizeof(qout)); + MLX5_SET(query_hca_cap_in, qin, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) | + HCA_CAP_OPMOD_GET_CUR); + mlx5st_cmd_exec(dev, qin, sizeof(qin), qout, sizeof(qout)); + + set_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, sin, capability); + memcpy(set_hca_cap, + MLX5_ADDR_OF(query_hca_cap_out, qout, capability), + MLX5_ST_SZ_BYTES(cmd_hca_cap)); + + MLX5_SET(cmd_hca_cap, set_hca_cap, cmdif_checksum, max_checksum); + MLX5_SET(cmd_hca_cap, set_hca_cap, log_uar_page_sz, 0); + + MLX5_SET(set_hca_cap_in, sin, opcode, MLX5_CMD_OP_SET_HCA_CAP); + MLX5_SET(set_hca_cap_in, sin, op_mod, + MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1); + + mlx5st_cmd_exec(dev, sin, sizeof(sin), sout, sizeof(sout)); + + dev->cmd_sig_enabled = max_checksum == 0x3; + dev_dbg(dev->device, "SET_HCA_CAP: OK (cmdif_checksum=%u)\n", + max_checksum); +} + +static void mlx5st_hca_init(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(enable_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(enable_hca_in)] = {}; + + mlx5st_wait_fw_init(dev); + dev_dbg(dev->device, "Firmware ready\n"); + + MLX5_SET(enable_hca_in, in, opcode, MLX5_CMD_OP_ENABLE_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "ENABLE_HCA: OK\n"); + + mlx5st_set_issi(dev); + mlx5st_fw_pages_satisfy(dev, 1); + + mlx5st_set_hca_caps(dev); + mlx5st_fw_pages_satisfy(dev, 0); + + memset(in, 0, sizeof(in)); + memset(out, 0, sizeof(out)); + MLX5_SET(init_hca_in, in, opcode, MLX5_CMD_OP_INIT_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "INIT_HCA: OK\n"); + + /* + * Create EQ immediately after INIT_HCA so PAGE_REQUEST events + * are captured during all subsequent commands. + */ + mlx5st_alloc_uar(dev); + mlx5st_create_eq(dev); +} + +static void mlx5st_disable_hca(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(disable_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(disable_hca_in)] = {}; + + MLX5_SET(disable_hca_in, in, opcode, MLX5_CMD_OP_DISABLE_HCA); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +static void mlx5st_fw_pages_reclaim(struct mlx5st_device *dev, u16 func_id) +{ + unsigned int npages = dev->fw_pages_given; + unsigned int total_claimed = 0; + + while (npages > 0) { + unsigned int batch = npages < MAX_FW_PAGES_PER_CMD ? + npages : + MAX_FW_PAGES_PER_CMD; + unsigned int outlen = + MLX5_ST_SZ_BYTES(manage_pages_out) + batch * 8; + unsigned int inlen = MLX5_ST_SZ_BYTES(manage_pages_in); + unsigned int num_claimed; + unsigned int i; + void *in, *out; + u64 *iovas; + + in = calloc(1, inlen); + out = calloc(1, outlen); + iovas = calloc(batch, sizeof(u64)); + VFIO_ASSERT_NOT_NULL(in); + VFIO_ASSERT_NOT_NULL(out); + VFIO_ASSERT_NOT_NULL(iovas); + + MLX5_SET(manage_pages_in, in, opcode, + MLX5_CMD_OP_MANAGE_PAGES); + MLX5_SET(manage_pages_in, in, op_mod, + MLX5_MANAGE_PAGES_IN_OP_MOD_HCA_RETURN_PAGES); + MLX5_SET(manage_pages_in, in, function_id, func_id); + MLX5_SET(manage_pages_in, in, input_num_entries, batch); + + mlx5st_cmd_exec(dev, in, inlen, out, outlen); + + num_claimed = + MLX5_GET(manage_pages_out, out, output_num_entries); + for (i = 0; i < num_claimed; i++) + iovas[i] = MLX5_ARRAY_GET64(manage_pages_out, out, pas, + i); + + mlx5st_fw_pages_free(dev, num_claimed, iovas); + total_claimed += num_claimed; + npages -= num_claimed; + + free(iovas); + free(in); + free(out); + + if (!num_claimed && !dev->fw_pages_given) + break; + if (!num_claimed) + VFIO_FAIL("MANAGE_PAGES RECLAIM: FW returned 0 but %d pages still given", + dev->fw_pages_given); + } + + dev_dbg(dev->device, + "MANAGE_PAGES RECLAIM: %d pages (%d still given)\n", + total_claimed, dev->fw_pages_given); +} + +static void mlx5st_hca_teardown(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(teardown_hca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(teardown_hca_in)] = {}; + + /* Drain async pages slot, then stop EQ processing */ + while (dev->pages_slot_in_use) { + if (!MLX5_GET_ONCE(cmd_queue_entry, dev->pages_cmd_lay, + ownership)) + mlx5st_pages_slot_complete(dev); + else + sched_yield(); + } + dev->have_eq = false; + + if (dev->eqn) { + mlx5st_destroy_eq(dev); + dev->eqn = 0; + } + if (dev->uar_page) { + mlx5st_dealloc_uar(dev); + dev->uar_page = 0; + } + + dev_dbg(dev->device, " hca_teardown: TEARDOWN_HCA\n"); + MLX5_SET(teardown_hca_in, in, opcode, MLX5_CMD_OP_TEARDOWN_HCA); + MLX5_SET(teardown_hca_in, in, profile, + MLX5_TEARDOWN_HCA_IN_PROFILE_GRACEFUL_CLOSE); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + if (dev->fw_pages_given > 0) { + dev_dbg(dev->device, " hca_teardown: reclaim %d pages\n", + dev->fw_pages_given); + mlx5st_fw_pages_reclaim(dev, dev->fw_func_id); + } + + dev_dbg(dev->device, " hca_teardown: DISABLE_HCA\n"); + mlx5st_disable_hca(dev); +} + +/* + * Query capabilities + */ +static void mlx5st_query_fl_caps(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {}; + u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {}; + bool fl_roce_en, fl_roce_dis; + + /* Query RoCE capabilities */ + MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, in, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_ROCE << 1) | HCA_CAP_OPMOD_GET_CUR); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + fl_roce_en = MLX5_GET(query_hca_cap_out, out, + capability.roce_cap.fl_rc_qp_when_roce_enabled); + fl_roce_dis = MLX5_GET(query_hca_cap_out, out, + capability.roce_cap.fl_rc_qp_when_roce_disabled); + + /* Also check general caps */ + memset(in, 0, sizeof(in)); + memset(out, 0, sizeof(out)); + MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, in, op_mod, + (MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1) | + HCA_CAP_OPMOD_GET_CUR); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + fl_roce_dis |= + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.fl_rc_qp_when_roce_disabled); + + dev->fl_supported = fl_roce_en || fl_roce_dis; + dev->log_max_msg = MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.log_max_msg); + dev_dbg(dev->device, + "HCA capabilities: fl_roce_enabled=%d fl_roce_disabled=%d log_max_msg=%u\n", + fl_roce_en, fl_roce_dis, dev->log_max_msg); + + VFIO_ASSERT_TRUE(dev->fl_supported, + "Force-loopback not supported on this device"); +} + +/* + * Resource allocation + */ + +static void mlx5st_alloc_pd(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(alloc_pd_out)] = {}; + u32 in[MLX5_ST_SZ_DW(alloc_pd_in)] = {}; + + MLX5_SET(alloc_pd_in, in, opcode, MLX5_CMD_OP_ALLOC_PD); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->pdn = MLX5_GET(alloc_pd_out, out, pd); + dev_dbg(dev->device, "Allocated PD pdn=%u\n", dev->pdn); +} + +static void mlx5st_dealloc_pd(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(dealloc_pd_out)] = {}; + u32 in[MLX5_ST_SZ_DW(dealloc_pd_in)] = {}; + + MLX5_SET(dealloc_pd_in, in, opcode, MLX5_CMD_OP_DEALLOC_PD); + MLX5_SET(dealloc_pd_in, in, pd, dev->pdn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +static void mlx5st_create_mkey(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(create_mkey_out)] = {}; + u32 in[MLX5_ST_SZ_DW(create_mkey_in)] = {}; + struct mlx5_ifc_mkc_bits *mkc; + + MLX5_SET(create_mkey_in, in, opcode, MLX5_CMD_OP_CREATE_MKEY); + + mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); + MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_PA); + MLX5_SET(mkc, mkc, length64, 1); + MLX5_SET(mkc, mkc, pd, dev->pdn); + MLX5_SET(mkc, mkc, qpn, 0xffffff); + MLX5_SET(mkc, mkc, lr, 1); + MLX5_SET(mkc, mkc, lw, 1); + MLX5_SET(mkc, mkc, rw, 1); + MLX5_SET(mkc, mkc, rr, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->mkey_index = MLX5_GET(create_mkey_out, out, mkey_index); + dev->global_lkey = mlx5st_idx_to_mkey(dev->mkey_index); + dev->global_rkey = dev->global_lkey; + + dev_dbg(dev->device, "Created global PA-mode MKEY: lkey=0x%x\n", + dev->global_lkey); +} + +static void mlx5st_destroy_mkey(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_mkey_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_mkey_in)] = {}; + + MLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY); + MLX5_SET(destroy_mkey_in, in, mkey_index, dev->mkey_index); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * CQ create/destroy + */ + +static void mlx5st_create_cq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_cq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_cq_out)] = {}; + struct mlx5_ifc_cqc_bits *cqc; + unsigned int i; + __be64 *pas; + + /* Initialize CQEs before CREATE_CQ: opcode=0xF, owner=1 */ + for (i = 0; i < CQ_CQE_CNT; i++) { + struct mlx5st_cqe64 *cqe = &dev->cq_buf[i]; + + MLX5_SET(cqe64, cqe, opcode, 0xF); + MLX5_SET_ONCE(cqe64, cqe, owner, 1); + } + + MLX5_SET(create_cq_in, in, opcode, MLX5_CMD_OP_CREATE_CQ); + + cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context); + MLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE); + MLX5_SET(cqc, cqc, uar_page, dev->uar_page); + MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->msi_eqn); + MLX5_SET(cqc, cqc, cqe_sz, 0); + pas = MLX5_ADDR_OF(create_cq_in, in, pas); + MLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev->cq_buf, pas)); + MLX5_SET(cqc, cqc, log_page_size, 0); + MLX5_SET64(cqc, cqc, dbr_addr, to_iova(device, &dev->cq_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->cqn = MLX5_GET(create_cq_out, out, cqn); + dev->cq_ci = 0; + dev_dbg(device, "Created CQ: cqn=%u, %d entries\n", dev->cqn, + CQ_CQE_CNT); +} + +static void mlx5st_destroy_cq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_cq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {}; + + MLX5_SET(destroy_cq_in, in, opcode, MLX5_CMD_OP_DESTROY_CQ); + MLX5_SET(destroy_cq_in, in, cqn, dev->cqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * Arm CQ for event generation. The CQ event delivery state machine is + * single-shot: after generating one EQE the CQ enters "Fired" state and + * won't generate another until re-armed via ARM_NEXT. Both the CQ doorbell + * record and the UAR CQ doorbell register must be written. + */ +static void mlx5st_arm_cq(struct mlx5st_device *dev) +{ + u32 sn = dev->cq_arm_sn & 3; + u32 ci = dev->cq_ci & 0xffffff; + u64 doorbell; + + /* Update CQ doorbell record arm word */ + WRITE_ONCE(dev->cq_dbrec.send_counter, + cpu_to_be32(sn << 28 | ci)); + + /* Ring CQ doorbell register, iowrite has an internal dma_wmb() */ + doorbell = ((u64)(sn << 28 | ci) << 32) | dev->cqn; + iowrite64be(doorbell, + (u8 __iomem *)dev->uar_base + MLX5_CQ_DOORBELL_OFFSET); + + dev->cq_arm_sn++; +} + +/* + * QP create/destroy + */ + +static void mlx5st_create_qp(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_qp_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_qp_out)] = {}; + struct mlx5_ifc_qpc_bits *qpc; + __be64 *pas; + + MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP); + + qpc = MLX5_ADDR_OF(create_qp_in, in, qpc); + MLX5_SET(qpc, qpc, st, MLX5_QPC_ST_RC); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, pd, dev->pdn); + MLX5_SET(qpc, qpc, uar_page, dev->uar_page); + MLX5_SET(qpc, qpc, cqn_snd, dev->cqn); + MLX5_SET(qpc, qpc, cqn_rcv, dev->cqn); + MLX5_SET(qpc, qpc, log_sq_size, LOG_SQ_SIZE); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, rq_type, 0x3); + MLX5_SET(qpc, qpc, ts_format, 1); + pas = MLX5_ADDR_OF(create_qp_in, in, pas); + MLX5_SET(qpc, qpc, page_offset, + mlx5st_fill_pas(device, dev->sq_buf, pas)); + MLX5_SET(qpc, qpc, log_page_size, 0); + MLX5_SET64(qpc, qpc, dbr_addr, to_iova(device, &dev->qp_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->qpn = MLX5_GET(create_qp_out, out, qpn); + dev->sq_pi = 0; + dev_dbg(device, "Created QP: qpn=%u, RC, sq=%d wqes\n", dev->qpn, + SQ_WQE_CNT); +} + +static void mlx5st_destroy_qp(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_qp_in)] = {}; + + MLX5_SET(destroy_qp_in, in, opcode, MLX5_CMD_OP_DESTROY_QP); + MLX5_SET(destroy_qp_in, in, qpn, dev->qpn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * QP state transitions + */ + +static void mlx5st_qp_rst2init(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rst2init_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc); + + MLX5_SET(rst2init_qp_in, in, opcode, MLX5_CMD_OP_RST2INIT_QP); + MLX5_SET(rst2init_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, rre, 1); + MLX5_SET(qpc, qpc, rwe, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RST->INIT\n"); +} + +static void mlx5st_qp_init2rtr(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(init2rtr_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc); + + MLX5_SET(init2rtr_qp_in, in, opcode, MLX5_CMD_OP_INIT2RTR_QP); + MLX5_SET(init2rtr_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, mtu, 3); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, remote_qpn, dev->qpn); + MLX5_SET(qpc, qpc, min_rnr_nak, 12); + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, primary_address_path.fl, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP INIT->RTR (fl=1)\n"); +} + +static void mlx5st_qp_rtr2rts(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rtr2rts_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc); + + MLX5_SET(rtr2rts_qp_in, in, opcode, MLX5_CMD_OP_RTR2RTS_QP); + MLX5_SET(rtr2rts_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, log_ack_req_freq, 0); + MLX5_SET(qpc, qpc, retry_count, 7); + MLX5_SET(qpc, qpc, rnr_retry, 7); + MLX5_SET(qpc, qpc, primary_address_path.ack_timeout, 14); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RTR->RTS\n"); +} + +/* + * Post RDMA Write WQE + */ +static void mlx5st_post_rdma_write(struct mlx5st_device *dev, u64 src_addr, + u32 src_lkey, u64 dst_addr, u32 dst_rkey, + u32 length, bool signaled) +{ + struct mlx5st_send_wqe *wqe; + unsigned int idx; + + idx = dev->sq_pi % SQ_WQE_CNT; + wqe = &dev->sq_buf[idx]; + + memset(wqe, 0, sizeof(*wqe)); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, opcode, MLX5_OPCODE_RDMA_WRITE); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, wqe_index, dev->sq_pi); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, qp_or_sq, dev->qpn); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ds, MLX5_RDMA_WRITE_DS); + if (signaled) + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ce, MLX5_WQE_CE_CQE_ALWAYS); + + MLX5_SET64(wqe_raddr_seg, &wqe->raddr, raddr, dst_addr); + MLX5_SET(wqe_raddr_seg, &wqe->raddr, rkey, dst_rkey); + + MLX5_SET(wqe_data_seg, &wqe->data, byte_count, length); + MLX5_SET(wqe_data_seg, &wqe->data, lkey, src_lkey); + MLX5_SET64(wqe_data_seg, &wqe->data, addr, src_addr); + + dev->sq_pi++; + + /* Ensure WQE is visible to device before doorbell record */ + dma_wmb(); + + WRITE_ONCE(dev->qp_dbrec.send_counter, + cpu_to_be32(dev->sq_pi & 0xffff)); + + /* + * Ring doorbell: write first 8 bytes of ctrl to UAR BF register, + * iowrite has an internal dma_wmb() so the doorbell record will be + * visible. + */ + iowrite64be(be64_to_cpu(*(__be64 *)wqe), + (u8 __iomem *)dev->uar_base + dev->uar_bf_offset); + dev->uar_bf_offset ^= MLX5_BF_SIZE; +} + +/* + * Poll CQ + */ +static int mlx5st_poll_cq_batch(struct mlx5st_device *dev, + unsigned int max_cqe) +{ + unsigned int polled = 0; + + while (polled < max_cqe) { + unsigned int idx = dev->cq_ci % CQ_CQE_CNT; + struct mlx5st_cqe64 *cqe = &dev->cq_buf[idx]; + u8 owner, opcode; + + owner = MLX5_GET_ONCE(cqe64, cqe, owner); + if (owner != ((dev->cq_ci >> LOG_CQ_SIZE) & 1)) + break; + + dma_rmb(); + + opcode = MLX5_GET(cqe64, cqe, opcode); + + dev->cq_ci++; + WRITE_ONCE(dev->cq_dbrec.recv_counter, + cpu_to_be32(dev->cq_ci & 0xffffff)); + + if (opcode == MLX5_CQE_REQ) { + dev->sq_ci = + (u16)(MLX5_GET(cqe64, cqe, wqe_counter) + 1); + polled++; + continue; + } + if (opcode == MLX5_CQE_REQ_ERR || + opcode == MLX5_CQE_RESP_ERR) { + dev_dbg(dev->device, + "CQE error: opcode=0x%x syndrome=0x%x vendor=0x%x\n", + opcode, + MLX5_GET(cqe64, cqe, error_syndrome.syndrome), + MLX5_GET(cqe64, cqe, + error_syndrome.vendor_error_syndrome)); + return -1; + } + dev_err(dev->device, "CQE unexpected opcode=0x%x\n", opcode); + return -1; + } + + return polled; +} + +static int mlx5st_poll_cq(struct mlx5st_device *dev, unsigned int timeout_ms) +{ + struct timespec start, now; + unsigned int elapsed; + int ret; + + clock_gettime(CLOCK_MONOTONIC, &start); + for (;;) { + ret = mlx5st_poll_cq_batch(dev, 1); + if (ret < 0) + return -1; + if (ret > 0) + return 0; + + if (dev->have_eq) + mlx5st_process_events(dev); + + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > timeout_ms) { + dev_err(dev->device, "CQ poll timeout after %u ms\n", + timeout_ms); + return -1; + } + } +} + +/* + * Data path setup/teardown helpers + */ + +static void mlx5st_setup_datapath(struct mlx5st_device *dev) +{ + mlx5st_create_cq(dev); + mlx5st_create_qp(dev); + mlx5st_qp_rst2init(dev); + mlx5st_qp_init2rtr(dev); + mlx5st_qp_rtr2rts(dev); +} + +static void mlx5st_teardown_datapath(struct mlx5st_device *dev) +{ + if (dev->qpn) { + mlx5st_destroy_qp(dev); + dev->qpn = 0; + } + if (dev->cqn) { + mlx5st_destroy_cq(dev); + dev->cqn = 0; + } + dev->sq_pi = 0; + dev->sq_ci = 0; + dev->cq_arm_sn = 0; + memset(&dev->qp_dbrec, 0, sizeof(dev->qp_dbrec)); + memset(&dev->cq_dbrec, 0, sizeof(dev->cq_dbrec)); +} + +/* + * memcpy callbacks + */ + +#define MLX5ST_MEMCPY_TIMEOUT_MS 60000 + +static void mlx5st_memcpy_start(struct vfio_pci_device *device, + iova_t src, iova_t dst, u64 size, u64 count) +{ + struct mlx5st_device *dev = to_mlx5st(device); + u64 i; + + for (i = 0; i < count; i++) { + bool signaled = (i == count - 1); + + mlx5st_post_rdma_write(dev, src, dev->global_lkey, dst, + dev->global_rkey, size, signaled); + } +} + +static int mlx5st_memcpy_wait(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + int ret; + + ret = mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS); + if (ret) { + /* + * CQE error puts the QP in error state. Rebuild the data path + * so subsequent operations can succeed. + */ + mlx5st_teardown_datapath(dev); + mlx5st_setup_datapath(dev); + } + return ret; +} + +/* + * send_msi callback — trigger CQE -> EQE -> MSI-X via a small RDMA Write. + * + * Both the CQ and MSI EQ use single-shot arming: the CQ must be armed so the + * CQE generates an EQE, and the MSI EQ must be armed so the EQE fires MSI-X. + */ +static void mlx5st_send_msi(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + + /* Drain accumulated MSI EQ events and re-arm for next interrupt */ + mlx5st_msi_eq_drain(dev); + + /* Arm CQ so the next CQE generates an EQE on the MSI EQ */ + mlx5st_arm_cq(dev); + + /* Post a signaled RDMA Write to trigger CQE -> EQE -> MSI-X */ + mlx5st_post_rdma_write(dev, + to_iova(device, &dev->send_msi_src), + dev->global_lkey, + to_iova(device, &dev->send_msi_dst), + dev->global_rkey, + sizeof(dev->send_msi_src), true); + + /* Consume the CQE to avoid stale completions */ + VFIO_ASSERT_EQ(mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS), 0); +} + +/* + * Driver ops callbacks + */ + +static void mlx5st_init(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + iova_t iova_align = + device->driver.region.iova % __alignof__(struct mlx5st_device); + + VFIO_ASSERT_GE(device->driver.region.size, sizeof(*dev)); + VFIO_ASSERT_EQ(iova_align, 0); + memset(dev, 0, sizeof(*dev)); + + dev->device = device; + dev->bar0 = device->bars[0].vaddr; + dev->init_seg = dev->bar0; + + vfio_pci_config_writew(device, PCI_COMMAND, + vfio_pci_config_readw(device, PCI_COMMAND) | + PCI_COMMAND_MASTER); + + mlx5st_wait_fw_init(dev); + + mlx5st_cmd_init(dev); + mlx5st_hca_init(dev); + mlx5st_query_fl_caps(dev); + mlx5st_alloc_pd(dev); + mlx5st_create_mkey(dev); + + /* MSI EQ must be created before CQ so CQ can reference its eqn */ + mlx5st_create_msi_eq(dev); + mlx5st_setup_datapath(dev); + + vfio_pci_msix_enable(device, MSI_VECTOR, 1); + device->driver.msi = MSI_VECTOR; + + device->driver.max_memcpy_size = 1ULL << dev->log_max_msg; + device->driver.max_memcpy_count = SQ_WQE_CNT - 1; + + dev_dbg(device, "mlx5 driver initialized\n"); +} + +static void mlx5st_remove(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + + vfio_pci_msix_disable(device); + mlx5st_teardown_datapath(dev); + + if (dev->have_msi_eq) { + mlx5st_destroy_msi_eq(dev); + dev->have_msi_eq = false; + } + + dev_dbg(device, "teardown: destroy_mkey\n"); + if (dev->mkey_index) { + mlx5st_destroy_mkey(dev); + dev->mkey_index = 0; + } + + dev_dbg(device, "teardown: dealloc_pd\n"); + if (dev->pdn) { + mlx5st_dealloc_pd(dev); + dev->pdn = 0; + } + + dev_dbg(device, "teardown: hca_teardown\n"); + mlx5st_hca_teardown(dev); + + vfio_pci_config_writew(device, PCI_COMMAND, + vfio_pci_config_readw(device, PCI_COMMAND) & + ~(u16)PCI_COMMAND_MASTER); + + dev_dbg(device, "Teardown complete\n"); +} + +struct vfio_pci_driver_ops mlx5st_ops = { + .name = "mlx5", + .region_size = sizeof(struct mlx5st_device), + .probe = mlx5st_probe, + .init = mlx5st_init, + .remove = mlx5st_remove, + .memcpy_start = mlx5st_memcpy_start, + .memcpy_wait = mlx5st_memcpy_wait, + .send_msi = mlx5st_send_msi, +}; diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h new file mode 100644 index 000000000000..2c451e411ec1 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h @@ -0,0 +1,114 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * mlx5 VFIO selftest driver - HW definitions + * + * Typed wrappers, constants, and helpers for programming mlx5 hardware + * via the VFIO selftest framework. Most HW constants and all MLX5_SET/GET + * macros come from the kernel headers (mlx5_ifc.h, mlx5_ifc_macros.h). + */ +#ifndef SELFTESTS_VFIO_MLX5_HW_H +#define SELFTESTS_VFIO_MLX5_HW_H + +#include <linux/io.h> +#include <linux/build_bug.h> +#include <vdso/bits.h> + +#include "mlx5_ifc.h" +#include "mlx5_ifc_macros.h" + +/* + * Typed HW object wrappers for driver region arrays. + * + * The IFC _bits structs have sizeof == num_bits (not bytes), so they cannot + * be used as array elements. These wrappers provide byte-sized types. + */ +#define MLX5ST_MAKE_DATA32(name) \ + struct mlx5st_##name { \ + u32 data[MLX5_ST_SZ_DW(name)]; \ + } +#define MLX5ST_MAKE_DATA64(name) \ + struct mlx5st_##name { \ + u64 data[MLX5_ST_SZ_QW(name)]; \ + } + +MLX5ST_MAKE_DATA32(initial_seg); +MLX5ST_MAKE_DATA64(cmd_queue_entry); +MLX5ST_MAKE_DATA64(cmd_if_box); +MLX5ST_MAKE_DATA64(wqe_ctrl_seg); +MLX5ST_MAKE_DATA64(wqe_raddr_seg); +MLX5ST_MAKE_DATA64(wqe_data_seg); +MLX5ST_MAKE_DATA64(cqe64) __aligned(64); +MLX5ST_MAKE_DATA64(eqe); + +/* + * Mailbox blocks: 512 data + 64 header = 576 bytes, but the + * next_pointer field stores bits [31:10], requiring 1024-byte alignment. + */ +#define CMD_MBOX_SIZE (2 * MLX5_HW_PAGE_SIZE) +#define CMD_MBOX_STRIDE 1024 +#define CMD_MBOX_NENT (CMD_MBOX_SIZE / CMD_MBOX_STRIDE) +/* Stride-aligned mailbox entry — block + padding to 1024 bytes */ +struct mlx5st_mbox_entry { + struct mlx5st_cmd_if_box block; +} __aligned(CMD_MBOX_STRIDE); + +#define MLX5_CMD_INLINE_SZ \ + MLX5_FLD_SZ_BYTES(cmd_queue_entry, command_input_inline_data) + +/* Command interface mailbox block (512 data + 64 header) */ +#define MLX5_CMD_DATA_BLOCK_SIZE MLX5_FLD_SZ_BYTES(cmd_if_box, mailbox_data) + +/* RDMA Write WQE — one basic block: ctrl + raddr + data + padding */ +struct mlx5st_send_wqe { + struct mlx5st_wqe_ctrl_seg ctrl; + struct mlx5st_wqe_raddr_seg raddr; + struct mlx5st_wqe_data_seg data; +} __aligned(64); +static_assert(sizeof(struct mlx5st_send_wqe) == 64, + "send WQE segments must fit in one BB"); + +/* DS = number of 16-byte segments in the WQE (ctrl + raddr + data) */ +#define MLX5_RDMA_WRITE_DS 3 + +/* Doorbell record — two __be32 in a 64-byte aligned pair */ +struct mlx5st_dbrec { + __be32 recv_counter; + __be32 send_counter; +} __aligned(64); + +/* UAR BlueFlame buffer offsets within a UAR page */ +#define MLX5_BF_OFFSET 0x800 +#define MLX5_BF_SIZE 0x100 + +/* CQ doorbell offset within UAR page */ +#define MLX5_CQ_DOORBELL_OFFSET 0x20 + +/* EQ doorbell offset within UAR page */ +#define MLX5_EQ_DOORBELL_OFFSET 0x40 + +#define MLX5_HW_PAGE_SIZE 4096 + +/* + * Test parameters + */ +#define SQ_WQE_CNT 16 +#define LOG_SQ_SIZE 4 +#define CQ_CQE_CNT 16 +#define LOG_CQ_SIZE 4 +#define EQ_NENT 64 +#define LOG_EQ_SIZE 6 +#define MSI_EQ_NENT 16 +#define LOG_MSI_EQ_SIZE 4 +#define MSI_VECTOR 0 + +#define MAX_FW_PAGES 8192 +#define MAX_FW_PAGES_PER_CMD 512 + +#define MLX5_CMD_TIMEOUT_MS 5000 + +static inline u32 mlx5st_idx_to_mkey(u32 mkey_idx) +{ + return mkey_idx << 8; +} + +#endif /* SELFTESTS_VFIO_MLX5_HW_H */ diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h new file mode 120000 index 000000000000..7dcbb79e1af0 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc.h
\ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h new file mode 120000 index 000000000000..865d99e2aeec --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc_fpga.h
\ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h new file mode 120000 index 000000000000..97408c247f06 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h @@ -0,0 +1 @@ +../../../../../../../include/linux/mlx5/mlx5_ifc_macros.h
\ No newline at end of file diff --git a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c index c08aa81c44f4..6958877ad463 100644 --- a/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c +++ b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c @@ -775,6 +775,7 @@ static int nv_falcon_memcpy_wait(struct vfio_pci_device *device) const struct vfio_pci_driver_ops nv_falcon_ops = { .name = "nv_falcon", + .region_size = sizeof(struct gpu_device), .probe = nv_falcon_probe, .init = nv_falcon_init, .remove = nv_falcon_remove, diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h index e19bd94b8dd2..007adc322c1e 100644 --- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h @@ -40,6 +40,17 @@ struct vfio_pci_device { struct vfio_pci_device *vfio_pci_device_alloc(const char *bdf, struct iommu *iommu); void vfio_pci_device_free(struct vfio_pci_device *device); + +#ifdef DEBUG +#define dev_dbg dev_info +#else +#define dev_dbg(_dev, _fmt, ...) \ + do { \ + if (0) \ + dev_info(_dev, _fmt, ##__VA_ARGS__); \ + } while (0) +#endif + struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu); void vfio_pci_device_cleanup(struct vfio_pci_device *device); diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h index e5ada209b1d1..547369c5cff9 100644 --- a/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h +++ b/tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h @@ -9,6 +9,12 @@ struct vfio_pci_device; struct vfio_pci_driver_ops { const char *name; + /* + * Size of the driver's state structure overlaid on + * device->driver.region.vaddr + */ + u64 region_size; + /** * @probe() - Check if the driver supports the given device. * diff --git a/tools/testing/selftests/vfio/lib/iova_allocator.c b/tools/testing/selftests/vfio/lib/iova_allocator.c index 4a660f636f49..a8e5815a1b16 100644 --- a/tools/testing/selftests/vfio/lib/iova_allocator.c +++ b/tools/testing/selftests/vfio/lib/iova_allocator.c @@ -13,8 +13,10 @@ #include <linux/iommufd.h> #include <linux/limits.h> +#include <linux/log2.h> #include <linux/mman.h> #include <linux/overflow.h> +#include <linux/sizes.h> #include <linux/types.h> #include <linux/vfio.h> @@ -50,7 +52,10 @@ void iova_allocator_cleanup(struct iova_allocator *allocator) iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size) { VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n"); - VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n"); + + VFIO_ASSERT_LE(size, rounddown_pow_of_two(SIZE_MAX), + "Invalid size arg, too large (%zu)\n", size); + size = roundup_pow_of_two(size); for (;;) { struct iommu_iova_range *range; diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/selftests/vfio/lib/libvfio.mk index bcfa74ae040e..eff95828719e 100644 --- a/tools/testing/selftests/vfio/lib/libvfio.mk +++ b/tools/testing/selftests/vfio/lib/libvfio.mk @@ -15,6 +15,7 @@ LIBVFIO_C += drivers/ioat/ioat.c LIBVFIO_C += drivers/dsa/dsa.c endif +LIBVFIO_C += drivers/mlx5/mlx5.c LIBVFIO_C += drivers/nv_falcon/nv_falcon.c LIBVFIO_C += drivers/igb/igb.c diff --git a/tools/testing/selftests/vfio/lib/sysfs.c b/tools/testing/selftests/vfio/lib/sysfs.c index 98a46a2543cd..d88e348a1e0d 100644 --- a/tools/testing/selftests/vfio/lib/sysfs.c +++ b/tools/testing/selftests/vfio/lib/sysfs.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include <fcntl.h> +#include <libgen.h> #include <unistd.h> #include <stdlib.h> #include <string.h> diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c index 5e65434d2318..b31abfe2c7ff 100644 --- a/tools/testing/selftests/vfio/lib/vfio_pci_driver.c +++ b/tools/testing/selftests/vfio/lib/vfio_pci_driver.c @@ -6,6 +6,7 @@ extern struct vfio_pci_driver_ops dsa_ops; extern struct vfio_pci_driver_ops ioat_ops; #endif +extern struct vfio_pci_driver_ops mlx5st_ops; extern struct vfio_pci_driver_ops nv_falcon_ops; extern struct vfio_pci_driver_ops igb_ops; @@ -14,6 +15,7 @@ static struct vfio_pci_driver_ops *driver_ops[] = { &dsa_ops, &ioat_ops, #endif + &mlx5st_ops, &nv_falcon_ops, &igb_ops, }; @@ -32,6 +34,10 @@ void vfio_pci_driver_probe(struct vfio_pci_device *device) continue; device->driver.ops = ops; + + VFIO_ASSERT_NE(ops->region_size, 0); + device->driver.region.size = + round_up(ops->region_size, getpagesize()); } } diff --git a/tools/testing/selftests/vfio/settings b/tools/testing/selftests/vfio/settings new file mode 100644 index 000000000000..c9abf1c19f19 --- /dev/null +++ b/tools/testing/selftests/vfio/settings @@ -0,0 +1,5 @@ +# Extend kselftest default 45s timeout based on worst case test runtime, +# currently vfio:vfio_pci_driver_test running on physical NIC, battling +# SMIs due to invoked firmware-first error handling from bad DMAs and +# device wedging. Observed worst case ~450s, extend to 600s. +timeout=600 diff --git a/tools/testing/selftests/vfio/vfio_pci_driver_test.c b/tools/testing/selftests/vfio/vfio_pci_driver_test.c index 761bf117d624..9538a3a436a6 100644 --- a/tools/testing/selftests/vfio/vfio_pci_driver_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_driver_test.c @@ -87,7 +87,8 @@ FIXTURE_SETUP(vfio_pci_driver_test) driver = &self->device->driver; region_setup(self->iommu, self->iova_allocator, &self->memcpy_region, SZ_1G); - region_setup(self->iommu, self->iova_allocator, &driver->region, SZ_2M); + region_setup(self->iommu, self->iova_allocator, &driver->region, + driver->region.size); /* Any IOVA that doesn't overlap memcpy_region and driver->region. */ self->unmapped_iova = iova_allocator_alloc(self->iova_allocator, SZ_1G); @@ -190,7 +191,7 @@ TEST_F(vfio_pci_driver_test, send_msi) ASSERT_EQ(1, value); } -TEST_F(vfio_pci_driver_test, mix_and_match) +TEST_F_TIMEOUT(vfio_pci_driver_test, mix_and_match, 90) { u64 value; int i; diff --git a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c index 19d657d00b75..f3a018b8699d 100644 --- a/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c @@ -157,6 +157,41 @@ TEST_F(vfio_pci_sriov_uapi_test, override_token) ASSERT_COND_VF_CREATION(ret); } +TEST(failed_second_open_does_not_clobber_token) +{ + struct vfio_pci_device *pf = NULL, *pf_second_fd = NULL, *vf = NULL; + struct iommu *iommu; + int ret; + + iommu = iommu_init(MODE_IOMMUFD); + + /* Create and bind PF using UUID_1 */ + ret = device_init(pf_bdf, iommu, UUID_1, &pf); + ASSERT_EQ(ret, 0); + + /* + * Attempt to open the same PF again and bind it with a *different* + * token (UUID_2). This must fail with -EBUSY because the cdev path + * only supports a single open per device. Enforce it with EXPECT_EQ() + * so the clobber assertion below still runs if the errno differs. + */ + ret = device_init(pf_bdf, iommu, UUID_2, &pf_second_fd); + EXPECT_EQ(ret, -EBUSY); + + /* + * Attempt to initialize a VF using the original PF token (UUID_1). + * If the failed open above clobbered the PF's token (i.e. updated it to + * UUID_2), this VF initialization will fail. + */ + ret = device_init(vf_bdf, iommu, UUID_1, &vf); + ASSERT_EQ(ret, 0); + + device_cleanup(vf); + device_cleanup(pf_second_fd); + device_cleanup(pf); + iommu_cleanup(iommu); +} + static void vf_teardown(void) { /* |