diff options
| author | Paolo Bonzini <pbonzini@redhat.com> | 2026-10-14 18:26:28 +0200 |
|---|---|---|
| committer | Paolo Bonzini <pbonzini@redhat.com> | 2026-10-14 18:26:28 +0200 |
| commit | 439d8f9cb01cb04f9b31910efd0ae8dbdfa4ad97 (patch) | |
| tree | ea4bdb8cc608ba5ba4b90bfdabff8d521060ec93 | |
| parent | e17f9dca0a6d29f72662ae0a501a5b236eea06a6 (diff) | |
| parent | dd0bc00e3e9201fd49b62ff9cc07890c89003afe (diff) | |
Merge tag 'kvmarm-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 updates for 7.4
* New features
- pKVM state isolation: years after pKVM started being enabled
upstream, we finally get the basics in the form of state
isolation. The vcpu state is now off-limit for the host for
protected VMs.
- pKVM also grows a hypervisor-private memory allocator, allowing it
to deal with sub-page allocation, much like everywhere else.
- pKVM also lands a trace_printk() equivalent, which hopefully will
make debugging a bit easier.
- First shot at annotating portions of the KVM/arm64 code to be
extracted and built on s390. The real fun is yet to come.
- Allow pre-faulting of stage-2 canonical mappings from userspace,
joining x86 and s390 in implementing this feature.
- Add a reverse mapping infrastructure for shadow stage-2,
significantly improving performance when MMU notifiers kick in,
and avoiding full unmap of all shadow tables for this VM.
- Next step of GICv5 enablement, adding support for the IRS block,
and providing both SPI and LPI support. This is the first point
where KVM/arm64 can support an SMP GICv5 guest.
* Improvements
- As a preliminary to eventually enabling CCA, add a bunch of
infrastructure to KVM/arm64, splitting some of the basic
operations into individual backends (normal VM, normal pKVM VM,
protected pKVM VM). This allows CCA to later slot in without
polluting the rest of KVM with its firmware-driven nonsense.
- Make ID registers immutable from the point where a vcpu is able
to run, even if it fails to run. This prevents KVM from sampling
untrusted values.
- Restore some type checking to the EL2 object. KVM/arm64 originally
had some strong type checking between the kernel and the EL2
object, but lost it along the way when the private hypercall
interface was (somewhat pointlessly) converted to SMCCC. Type
checking is back, syscall style, and with added sparse goodies.
- Expand the FFA support to FFA_NOTIFICATION_*, and add some
sanitisation of the protocol that pKVM relays to the "trusted"
runtime.
* Fixes
- Make GICv3 RD assignment more robust to userspace nonsense
- Convert the remaining MMU fault handler backends to
kvm_s2_fault_vma_info and co, fixing a number of bugs in the
process.
- Fix the SError/SEA selftests to be both more robust and less
braindead.
- Fix a smattering of documentation bugs, some of them as a result
of our AI friends spitting out credible assertions based on
non-existing symbols.
- The usual collection of low impact bug fixes (SMCCC, timers, PMU).
122 files changed, 14777 insertions, 1312 deletions
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index e0430cc750c9..ce01afa2a9a4 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -918,7 +918,8 @@ The irq_type field has the following values: - KVM_ARM_IRQ_TYPE_SPI: in-kernel GICv2/GICv3: SPI, irq_id between 32 and 1019 (incl.) (the vcpu_index field is ignored) - in-kernel GICv5: SPI, irq_id between 0 and 65535 (incl.) + in-kernel GICv5: SPI, irq_id between 0 and the configured + number of SPIs minus one (1023 maximum) - KVM_ARM_IRQ_TYPE_PPI: in-kernel GICv2/GICv3: PPI, irq_id between 16 and 31 (incl.) in-kernel GICv5: PPI, irq_id between 0 and 127 (incl.) @@ -1322,6 +1323,9 @@ guests, across different userspace implementations. Nevertheless, userspace can still emulate all Arm exceptions by manipulating individual registers using the KVM_SET_ONE_REG API. +For a protected VM, setting ext_dabt_pending returns -EPERM; see +Documentation/virt/kvm/arm/pkvm.rst. Injecting an SError is unaffected. + See KVM_GET_VCPU_EVENTS for the data structure. Calling this ioctl on a vCPU that hasn't been initialized will return @@ -1545,7 +1549,9 @@ Possible values are: KVM_MP_STATE_RUNNABLE the vcpu is currently running [x86,arm64,riscv,loongarch] KVM_MP_STATE_UNINITIALIZED the vcpu is an application processor (AP) - which has not yet received an INIT signal [x86] + which has not yet received an INIT signal + [x86], or a protected VM's vcpu that the + guest holds powered off [arm64] KVM_MP_STATE_INIT_RECEIVED the vcpu has received an INIT signal, and is now ready for a SIPI [x86] KVM_MP_STATE_HALTED the vcpu has executed a HLT instruction and @@ -1624,6 +1630,12 @@ For arm64/riscv: The only states that are valid are KVM_MP_STATE_STOPPED and KVM_MP_STATE_RUNNABLE which reflect if the vcpu should be paused or not. +On arm64, once a protected VM's vcpu has run, it reads as +KVM_MP_STATE_UNINITIALIZED while the guest holds it powered off, with +CPU_OFF or by not yet bringing it online with CPU_ON, and +KVM_MP_STATE_RUNNABLE and KVM_MP_STATE_SUSPENDED return -EPERM on it: only +the guest can power it on. See Documentation/virt/kvm/arm/pkvm.rst. + On LoongArch, only the KVM_MP_STATE_RUNNABLE state is used to reflect whether the vcpu is runnable. @@ -2324,7 +2336,8 @@ Errors: ENOENT no such register EINVAL invalid register ID, or no such register or used with VMs in protected virtualization mode on s390 - EPERM (arm64) register access not allowed before vcpu finalization + EPERM (arm64) register access not allowed before vcpu + finalization, or after a protected VM's vcpu has run EBUSY (riscv) changing register value not allowed after the vcpu has run at least once ====== ============================================================ @@ -2949,7 +2962,8 @@ Errors include: ENOENT no such register EINVAL invalid register ID, or no such register or used with VMs in protected virtualization mode on s390 - EPERM (arm64) register access not allowed before vcpu finalization + EPERM (arm64) register access not allowed before vcpu + finalization, or after a protected VM's vcpu has run ======== ============================================================ (These error codes are indicative only: do not rely on a specific error @@ -3484,6 +3498,7 @@ Errors: ====== ================================================================= EINVAL the target is unknown, or the combination of features is invalid. ENOENT a features bit specified is unknown. + EPERM the vcpu belongs to a protected VM and has already run. ====== ================================================================= This tells KVM what type of CPU to present to the guest, and what @@ -3512,6 +3527,10 @@ after the vcpu has been run. This will reset the vcpu to its initial state. All calls to this function after the initial call must use the same target and same set of feature flags, otherwise EINVAL will be returned. +For a protected VM this ioctl returns EPERM once the vcpu has run, and the +features it accepts are restricted: KVM_ARM_VCPU_PSCI_0_2 is required. See +Documentation/virt/kvm/arm/pkvm.rst. + Possible features: - KVM_ARM_VCPU_POWER_OFF: Starts the CPU in a power-off state. @@ -3600,20 +3619,14 @@ Possible features: :Parameters: struct kvm_vcpu_init (out) :Returns: 0 on success; -1 on error -Errors: - - ====== ========================================== - ENODEV no preferred target available for the host - ====== ========================================== - This queries KVM for preferred CPU target type which can be emulated by KVM on underlying host. -The ioctl returns struct kvm_vcpu_init instance containing information -about preferred CPU target type and recommended features for it. The -kvm_vcpu_init->features bitmap returned will have feature bits set if -the preferred target recommends setting these features, but this is -not mandatory. +The ioctl returns a struct kvm_vcpu_init instance containing the +preferred CPU target type. The kvm_vcpu_init->features bitmap is +returned empty: userspace selects the vCPU features itself, and their +availability is reported by the capabilities listed under +KVM_ARM_VCPU_INIT. The information returned by this ioctl can be used to prepare an instance of struct kvm_vcpu_init for KVM_ARM_VCPU_INIT ioctl which will result in @@ -3676,7 +3689,7 @@ type KVM_X86_REG_TYPE_MSR, but are NOT enumerated via KVM_GET_REG_LIST. :Capability: KVM_CAP_ARM_SET_DEVICE_ADDR :Architectures: arm64 :Type: vm ioctl -:Parameters: struct kvm_arm_device_address (in) +:Parameters: struct kvm_arm_device_addr (in) :Returns: 0 on success, -1 on error Errors: @@ -3790,6 +3803,9 @@ can be determined by querying the KVM_CAP_GUEST_DEBUG_HW_BPS and KVM_CAP_GUEST_DEBUG_HW_WPS capabilities which return a positive number indicating the number of supported registers. +On arm64, this ioctl returns -EPERM for a protected VM: debugging a +protected guest is not supported. See Documentation/virt/kvm/arm/pkvm.rst. + For ppc, the KVM_CAP_PPC_GUEST_DEBUG_SSTEP capability indicates whether the single-step debug event (KVM_GUESTDBG_SINGLESTEP) is supported. @@ -6494,7 +6510,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. --------------------------- :Capability: KVM_CAP_PRE_FAULT_MEMORY -:Architectures: none +:Architectures: x86, s390, arm64 :Type: vcpu ioctl :Parameters: struct kvm_pre_fault_memory (in/out) :Returns: 0 if at least one page is processed, < 0 on error @@ -6502,12 +6518,15 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details. Errors: ========== =============================================================== + EAGAIN A race occurred before progress was made, but a retry may succeed. EINVAL The specified `gpa` and `size` were invalid (e.g. not page aligned, causes an overflow, or size is zero), or the VM is UCONTROL (s390). ENOENT The specified `gpa` is outside defined memslots. + ENOEXEC The vCPU has not been initialised (arm64). EINTR An unmasked signal is pending and no page was processed. EFAULT The parameter address was invalid. + EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the hypervisor, and/or for the current vCPU state/mode. EIO unexpected error conditions (also causes a WARN) @@ -6527,7 +6546,17 @@ Errors: KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory for the current vCPU state. KVM maps memory as if the vCPU generated a stage-2 read page fault, e.g. faults in memory as needed, but doesn't break -CoW. On x86, KVM does not mark any newly created stage-2 PTE as Accessed. +CoW. On arm64, KVM marks newly created stage-2 PTEs as Accessed, as it +does for any stage-2 fault, but leaves the Accessed state of existing PTEs +unchanged. On x86, KVM does not mark any newly created stage-2 PTE as +Accessed, and for s390 it is not applicable. + +On arm64, a GPA is interpreted as an IPA, and never interpreted as the IPA +of a nested guest. Pre-faulting only populates canonical stage-2 page +tables. + +The feature is not supported on arm64 if the protected KVM (pKVM) feature +is enabled. In the case of confidential VM types where there is an initial set up of private guest memory before the guest is 'finalized'/measured, this ioctl @@ -6542,7 +6571,7 @@ When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue the ioctl again with the same `struct kvm_map_memory` argument. -Shadow page tables cannot support this ioctl because they +On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address. Calling this ioctl when the guest is using shadow page tables (for example because it is running a nested guest with nested page tables) diff --git a/Documentation/virt/kvm/arm/fw-pseudo-registers.rst b/Documentation/virt/kvm/arm/fw-pseudo-registers.rst index d78b53b05dfc..c07471e258ea 100644 --- a/Documentation/virt/kvm/arm/fw-pseudo-registers.rst +++ b/Documentation/virt/kvm/arm/fw-pseudo-registers.rst @@ -33,6 +33,8 @@ The following registers are defined: highest PSCI version implemented by KVM and compatible with v0.2) - Allows any PSCI version implemented by KVM and compatible with v0.2 to be set with SET_ONE_REG + - Returns -EINVAL for a version below v1.1 on a protected VM (see + Documentation/virt/kvm/arm/pkvm.rst) - Affects the whole VM (even if the register view is per-vcpu) * KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1: diff --git a/Documentation/virt/kvm/arm/pkvm.rst b/Documentation/virt/kvm/arm/pkvm.rst index 514992a79a83..b75bb8ad6f26 100644 --- a/Documentation/virt/kvm/arm/pkvm.rst +++ b/Documentation/virt/kvm/arm/pkvm.rst @@ -15,7 +15,9 @@ Overview Booting a host kernel with '``kvm-arm.mode=protected``' enables "Protected KVM" (pKVM). During boot, pKVM installs a stage-2 identity map page-table for the host and uses it to isolate the hypervisor -running at EL2 from the rest of the host running at EL1/0. +running at EL2 from the rest of the host running at EL1/0. pKVM supports +GICv3 only: GICv2 is not supported in protected mode, and neither are +GICv5-based guests. pKVM permits creation of protected virtual machines (pVMs) by passing the ``KVM_VM_TYPE_ARM_PROTECTED`` machine type identifier to the @@ -31,7 +33,7 @@ See hypercalls.rst for more details. Isolation mechanisms ==================== -pKVM relies on a number of mechanisms to isolate PVMs from the host: +pKVM relies on a number of mechanisms to isolate pVMs from the host: CPU memory isolation -------------------- @@ -50,8 +52,9 @@ stage-2 identity map as part of the donation procedure. This gives rise to some user-visible differences when compared to non-protected VMs, largely due to the lack of MMU notifiers: -* Memslots cannot be moved or deleted once the pVM has started running. -* Read-only memslots and dirty logging are not supported. +* Memslots cannot be moved or deleted once the pVM has started running + (``-EPERM``). +* Read-only memslots and dirty logging are not supported (``-EPERM``). * With the exception of swap, file-backed pages cannot be mapped into a pVM. * Donated pages are accounted against ``RLIMIT_MLOCK`` and so the VMM @@ -67,12 +70,19 @@ largely due to the lack of MMU notifiers: then it will either return ``-EFAULT`` or forcefully reclaim the memory pages. Reclaimed memory is zeroed by the hypervisor and a subsequent attempt to access it in the pVM will return ``-EFAULT`` - from the ``VCPU_RUN`` ioctl(). + from the ``KVM_RUN`` ioctl(). CPU state isolation ------------------- -Status: **Unimplemented.** +Status: CPU register state of protected vCPUs is managed entirely at EL2. + +pKVM performs the complete context switch for protected vCPUs within the +hypervisor. The hypervisor sets a protected vCPU's initial state, not the +host, and only what each exit needs is synchronised back. + +The user-visible consequences are described under `API behaviour for +protected VMs`_. DMA isolation using an IOMMU ---------------------------- @@ -89,13 +99,153 @@ The FF-A proxy ensures that the host cannot share pVM or hypervisor memory with Trustzone as part of a "confused deputy" attack. The PSCI proxy ensures that CPUs always have the stage-2 identity map -installed when they are executing in the host. +installed when they are executing in the host. This proxy is distinct +from the PSCI handling provided to protected guests, which is described +under `API behaviour for protected VMs`_. Protected VM firmware (pvmfw) ----------------------------- Status: **Unimplemented.** +API behaviour for protected VMs +=============================== + +Protected vCPU state is owned by EL2 (see `CPU state isolation`_). The VMM +configures a vCPU before its first ``KVM_RUN``; afterwards the state is +private to the guest and the ioctls that access it return ``-EPERM``. The +vCPU ioctls' errors follow one rule: ``-EPERM`` means the host asked for +state that the guest owns, ``-EINVAL`` means the request is not valid for a +protected VM, and a feature pKVM doesn't support for protected VMs is +rejected the way KVM rejects one that isn't implemented (``-ENXIO`` from +``KVM_ARM_VCPU_PVTIME_CTRL``, for example). The ioctls themselves are +described in Documentation/virt/kvm/api.rst. + +Boot +---- + +A protected VM boots from a single primary vCPU. Before the first +``KVM_RUN``, the VMM prepares the boot state: + +* Set ``KVM_MP_STATE_RUNNABLE`` on the primary vCPU and + ``KVM_MP_STATE_STOPPED`` on every other vCPU. EL2 allows only one + RUNNABLE primary per protected VM. A second RUNNABLE vCPU fails at its + first ``KVM_RUN``. +* Set the primary vCPU's boot state with ``KVM_SET_ONE_REG``: the kernel + entry address in ``PC`` and the DTB pointer in ``x0``. + +``PC`` and ``x0`` are the only registers EL2 takes from the host. Other +pre-run writes are accepted, but the guest starts from the state the +hypervisor resets it to. + +Secondary vCPUs are started by the guest itself through PSCI ``CPU_ON`` +(see `Power state`_), which supplies their entry point and context ID. +The VMM cannot choose where they boot. + +vCPU state +---------- + +* ``KVM_GET_ONE_REG`` and ``KVM_SET_ONE_REG`` return ``-EPERM`` once the + vCPU has run. Before that, they access the host-side copy from which + EL2 builds the guest's boot state (see `Boot`_). +* ``KVM_ARM_VCPU_INIT`` accepts only the vCPU features that a protected + guest supports and returns ``-EINVAL`` otherwise. + ``KVM_ARM_VCPU_PSCI_0_2`` is required, as EL2 implements PSCI 1.1 for + the guest (see `Power state`_). ``KVM_ARM_VCPU_EL1_32BIT`` is not + supported: protected guests run in AArch64 only and see no AArch32 + support in ``ID_AA64PFR0_EL1``. Once the vCPU has run, + ``KVM_ARM_VCPU_INIT`` returns ``-EPERM``, as re-initialising it would + reset the host-side copy alone. +* ``KVM_SET_VCPU_EVENTS`` returns ``-EPERM`` for external-abort injection + (``ext_dabt_pending``), and ``KVM_CAP_ARM_INJECT_EXT_DABT`` is not + advertised to a protected VM. SError injection is unaffected. +* ``KVM_SET_GUEST_DEBUG`` returns ``-EPERM``, and ``KVM_CAP_SET_GUEST_DEBUG`` + is not advertised to a protected VM (see `Debug`_). + +Power state +----------- + +EL2 implements PSCI 1.1 for a protected guest. The calls that move a +vCPU's power state, ``CPU_ON`` and ``CPU_OFF``, are handled at EL2, and +the host cannot change the outcome: for ``CPU_ON`` the host only +schedules the target, which EL2 has already reset to the entry point +the guest chose, and for ``CPU_OFF`` it only stops scheduling it. +``AFFINITY_INFO`` is handled by the host, which returns ``OFF`` once +it has stopped scheduling every vCPU the query covers, as for a +non-protected VM. A vCPU becomes a valid ``CPU_ON`` target at its first +``KVM_RUN``, and before that EL2 returns ``INVALID_PARAMETERS``. The +platform calls, ``CPU_SUSPEND``, ``SYSTEM_OFF``, ``SYSTEM_RESET`` and +``SYSTEM_RESET2``, are forwarded to the host and behave as for a +non-protected VM, with the ``SYSTEM_*`` calls exiting to the VMM as +``KVM_EXIT_SYSTEM_EVENT``; the ``SYSTEM_RESET2`` reset type and cookie +are in the guest's registers, which ``KVM_GET_ONE_REG`` rejects once +the vCPU has run. Any other function returns ``NOT_SUPPORTED``, and +``PSCI_FEATURES`` reports the same set. Because the host handles those +forwarded calls, ``KVM_SET_ONE_REG`` on ``KVM_REG_ARM_PSCI_VERSION`` +returns ``-EINVAL`` for a version below 1.1. + +Once a vCPU has run, its power state follows the guest's PSCI calls, not +the VMM's. A vCPU that the guest has powered off with ``CPU_OFF``, or has +not yet brought online with ``CPU_ON``, reads as +``KVM_MP_STATE_UNINITIALIZED``, and only an in-guest ``CPU_ON`` can power +it on: ``KVM_SET_MP_STATE`` returns ``-EPERM`` for ``KVM_MP_STATE_RUNNABLE`` +and ``KVM_MP_STATE_SUSPENDED`` on it, accepts ``KVM_MP_STATE_STOPPED`` and +leaves it as it is, and never takes ``KVM_MP_STATE_UNINITIALIZED`` from +the VMM. ``KVM_MP_STATE_STOPPED`` still stops a vCPU EL2 has powered on, +and ``KVM_MP_STATE_RUNNABLE`` resumes it, so the VMM can pause it. + +Other interface differences +--------------------------- + +* ``KVM_CHECK_EXTENSION`` reports only the arm64 capabilities that pKVM + supports for protected guests, and ``KVM_ENABLE_CAP`` accepts only + those. Query them on the VM file descriptor: the system file descriptor + has no VM to filter against. The capabilities the generic KVM code + answers itself (``KVM_CAP_SYNC_MMU`` or ``KVM_CAP_DEVICE_CTRL``, for + example) are reported as for any VM, and don't mean the feature works + for a protected one. The filter does not cover every interface either: + device-fd configuration (for example the VGIC after + ``KVM_CREATE_DEVICE``) and vCPU attributes are unfiltered, so a feature + the filter excludes is rejected by the attribute itself. + ``KVM_ARM_VCPU_PVTIME_CTRL`` returns ``-ENXIO``, for example, as it does + where steal time isn't implemented. +* The vGIC of a protected VM remains host-managed: device creation, + configuration and interrupt injection all work as they do for a + non-protected VM. +* ``KVM_ARM_SET_COUNTER_OFFSET`` and ``KVM_ARM_GET_REG_WRITABLE_MASKS`` + return ``-EINVAL``: their capabilities are not advertised to a protected + VM, whose counter offset and ID registers are set by EL2. A protected + guest sees the physical timebase. +* A protected guest's first access to each page of memory exits to the + host, since the hypervisor cannot tell memory from a device before the + page is mapped. For a store, the host sees the value of the register + the syndrome names, clamped to the access width, and nothing else from + the register file. For a load, the register the syndrome names takes + the value the host returns, as for any emulated access. +* A protected guest that uses a feature it was not given, or executes an + ``SMC``, takes an undefined instruction exception from the hypervisor; + the host is not involved. +* Apart from PSCI, the hypervisor handles a protected guest's SMCCC + calls itself and does not involve the host. A function it does not + implement returns ``NOT_SUPPORTED``, and the host's SMCCC filter does + not apply: the VM device attribute ioctls (``KVM_ARM_VM_SMCCC_CTRL``) + return ``-EINVAL`` for a protected VM. +* The hypervisor can decode a trapped guest access only from the CPU's + instruction syndrome, which is provided only for a load or store of a + single general-purpose register. An access without one (for example a + load/store pair or a SIMD/FP access) cannot be decoded. For a + non-protected VM it can exit to the VMM as ``KVM_EXIT_ARM_NISV``. For a + protected VM it cannot be emulated by the VMM, so the guest takes a + synchronous external abort instead. + +Debug +----- + +Hardware-assisted debugging is not available to protected guests. +``MDSCR_EL1``, ``OSLAR_EL1``, ``OSLSR_EL1``, ``OSDLR_EL1``, and breakpoint +and watchpoint 0 are RAZ/WI; any other debug register access takes an +undefined instruction exception. + Resources ========= diff --git a/Documentation/virt/kvm/devices/arm-vgic-v5.rst b/Documentation/virt/kvm/devices/arm-vgic-v5.rst index 70b9162755c7..8a691c8938b6 100644 --- a/Documentation/virt/kvm/devices/arm-vgic-v5.rst +++ b/Documentation/virt/kvm/devices/arm-vgic-v5.rst @@ -12,12 +12,39 @@ Only one VGIC instance may be instantiated through this API. The created VGIC will act as the VM interrupt controller, requiring emulated user-space devices to inject interrupts to the VGIC instead of directly to CPUs. -Creating a guest GICv5 device requires a GICv5 host. The current VGICv5 device -only supports PPI interrupts. These can either be injected from emulated -in-kernel devices (such as the Arch Timer, or PMU), or via the KVM_IRQ_LINE -ioctl. +Creating a guest GICv5 device requires a GICv5 host. The VGICv5 device supports +PPI, SPI, and LPI interrupts. The PPI and SPI interrupts can either be injected +from emulated in-kernel devices (such as the Arch Timer, or PMU), or via the +KVM_IRQ_LINE ioctl. LPIs are not externally injected, but are handled in +hardware via the LPI IST. Their pending state is driven directly by the guest. Groups: + KVM_DEV_ARM_VGIC_GRP_ADDR + Attributes: + + KVM_VGIC_V5_ADDR_TYPE_IRS (rw, 64-bit) + Base address in the guest physical address space of the GICv5 IRS + (Interrupt Routing Service) register mappings. Only valid for + KVM_DEV_TYPE_ARM_VGIC_V5. This address needs to be 64K aligned and the + region covers 128 KByte - the IRS has a CONFIG_FRAME and a SETLPI_FRAME, + each of which is 64 KBytes in size. + + Setting the address of the IRS in GPA space is mandatory before VGIC + resources are mapped, as the IRS is responsible for handling SPIs and + LPIs. Failure to set the IRS address before the first vCPU run results in + an error. + + KVM_DEV_ARM_VGIC_GRP_NR_IRQS + Attributes: + + A value describing the number of SPIs for this GIC instance. This is + GICv5-specific: unlike GICv2/v3, the value does not include SGIs or PPIs. + The value ranges from 32 to KVM's VGICv5 maximum of 1024 SPIs, in + increments of 32. If userspace does not set this attribute, KVM uses 32 + SPIs by default. + + kvm_device_attr.addr points to a __u32 value. + KVM_DEV_ARM_VGIC_GRP_CTRL Attributes: @@ -48,3 +75,239 @@ Groups: -EFAULT Invalid guest ram access -EBUSY One or more VCPUS are running ======= ======================================================== + + KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS + Attributes: + + The attr field of kvm_device_attr encodes two values:: + + bits: | 63 .... 32 | 31 .... 16 | 15 .... 0 | + values: | mpidr | RES | instr | + + The mpidr field encodes the CPU ID based on the affinity information in the + architecture defined MPIDR, and the field is encoded as follows:: + + | 63 .... 56 | 55 .... 48 | 47 .... 40 | 39 .... 32 | + | Aff3 | Aff2 | Aff1 | Aff0 | + + The instr field encodes the system register to access based on the fields + defined in the A64 instruction set encoding for system register access + (RES means the bits are reserved for future use and should be zero):: + + | 15 ... 14 | 13 ... 11 | 10 ... 7 | 6 ... 3 | 2 ... 0 | + | Op 0 | Op1 | CRn | CRm | Op2 | + + All system regs accessed through this API are (rw, 64-bit) and + kvm_device_attr.addr points to a __u64 value. + + KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS accesses the CPU interface registers for the + CPU specified by the mpidr field. + + The available registers are: + + ======================= =================================================== + ICC_ICSR_EL1 + ICC_PPI_ENABLER0_EL1 + ICC_PPI_ENABLER1_EL1 + ICC_PPI_SACTIVER0_EL1 ICC_PPI_CACTIVER0_EL1 is not supported. Writes to + ICC_PPI_SACTIVER0_EL1 are treated as RAW writes of + the underlying state. + ICC_PPI_SACTIVER1_EL1 ICC_PPI_CACTIVER1_EL1 is not supported. Writes to + ICC_PPI_SACTIVER1_EL1 are treated as RAW writes of + the underlying state. + ICC_PPI_SPENDR0_EL1 ICC_PPI_CPENDR0_EL1 is not supported. Writes to + ICC_PPI_SPENDR0_EL1 are treated as RAW writes of + the underlying state. + ICC_PPI_SPENDR1_EL1 ICC_PPI_CPENDR1_EL1 is not supported. Writes to + ICC_PPI_SPENDR1_EL1 are treated as RAW writes of + the underlying state. + ICC_PPI_PRIORITYR0_EL1 + ICC_PPI_PRIORITYR1_EL1 + ICC_PPI_PRIORITYR2_EL1 + ICC_PPI_PRIORITYR3_EL1 + ICC_PPI_PRIORITYR4_EL1 + ICC_PPI_PRIORITYR5_EL1 + ICC_PPI_PRIORITYR6_EL1 + ICC_PPI_PRIORITYR7_EL1 + ICC_PPI_PRIORITYR8_EL1 + ICC_PPI_PRIORITYR9_EL1 + ICC_PPI_PRIORITYR10_EL1 + ICC_PPI_PRIORITYR11_EL1 + ICC_PPI_PRIORITYR12_EL1 + ICC_PPI_PRIORITYR13_EL1 + ICC_PPI_PRIORITYR14_EL1 + ICC_PPI_PRIORITYR15_EL1 + ICC_APR_EL1 + ICC_CR0_EL1 + ICC_PCR_EL1 + ======================= =================================================== + + Errors: + + ======= ============================================================= + -ENXIO Getting or setting this register is not supported + -EBUSY VCPU is running, or write attempted after a VCPU has run + -EINVAL Invalid mpidr or register value supplied + ======= ============================================================= + + + KVM_DEV_ARM_VGIC_GRP_IRS_REGS + Attributes: + The attr field of kvm_device_attr encodes the offset of the IRS register, + relative to the IRS CONFIG_FRAME base address. This is the address that + was provided via KVM_VGIC_V5_ADDR_TYPE_IRS when creating VGICv5 in the + first place. + + kvm_device_attr.addr points to a __u64 value whatever the width + of the addressed register (32/64 bits). 64 bit registers can only + be accessed with full length. + + Writes to read-only registers are ignored by the kernel except for + IRS_IDR0 - IRS_IDR2 and IRS_IDR5 - IRS_IDR7, which are sanity checked to + ensure that they match a sane config. + + IRS_IDR3 and IRS_IDR4 are RAZ/WI, as nested virtualization is not + supported. + + For registers without dedicated userspace accessors, getting or setting a + register uses the same emulated MMIO handlers as guest reads/writes. + Dedicated userspace accessors may instead save or restore migration state + without triggering guest-visible side effects. For example, restoring + IRS_IST_BASER only restores the emulated register state; any host LPI IST + allocation based on the restored IRS_IST_CFGR and IRS_IST_BASER state + happens when KVM_DEV_ARM_VGIC_GRP_IST is restored. + + Once an LPI IST has been allocated, IRS_IST_CFGR is immutable. Userspace + may write back its current value, but setting a different value is + rejected with -EINVAL. + + Once an LPI IST has been allocated, IRS_IST_BASER is immutable. + Userspace may write back its current value, but setting a different value + is rejected with -EINVAL. + + Errors: + + ======= ================================================================= + -ENXIO Offset does not correspond to any supported register + -EFAULT Invalid user pointer for attr->addr + -EINVAL Offset is not 32-bit aligned for 32-bit MMIO registers, not + 64-bit aligned for 64-bit registers, or the supplied register + value is not compatible with the configured VGICv5 IRS state + -EBUSY VGIC is not initialized, one or more VCPUs are running, or a + write is attempted after a VCPU has run + ======= ================================================================= + + KVM_DEV_ARM_VGIC_GRP_IST + Attributes: + This interface is used to either save the state of the IRS's Interrupt + State Tables (ISTs), or to restore them. A get operation saves IST state, + and a set operation restores IST state. kvm_device_attr.attr is reserved + and must be zero. + + The VGIC must be initialized before using this interface. Restore must be + performed before the VM has run. For restore, userspace must have already + restored the IRS state needed to describe any guest LPI IST. + + Saving first asks the IRS to save and quiesce the VM so that interrupt + state has been written back to the ISTs. KVM checks that the VM remains + quiesced while copying out the SPI and LPI IST state. + + kvm_device_attr.addr points to a struct kvm_vgic_v5_ist:: + + struct kvm_vgic_v5_ist { + __u64 spi_ist_addr; + __u64 spi_ist_size; + __u64 lpi_ist_addr; + __u64 lpi_ist_size; + }; + + The SPI and LPI IST buffers contain one little-endian 32-bit IST entry per + interrupt, in interrupt number order. Only the architected 32-bit ISTE + state is exposed to userspace; host IST layout and metadata are private to + KVM. + + The SPI IST buffer is required and its size must be: + + nr_spis * sizeof(__u32) + + where nr_spis is the value returned by KVM_DEV_ARM_VGIC_GRP_NR_IRQS for + the VGICv5 device. For VGICv5 this value is the number of SPIs, not the + total number of interrupts. + + The LPI IST buffer is required if the guest's IRS_IST_CFGR and + IRS_IST_BASER state describes a valid LPI IST, and its size must be: + + BIT(lpi_id_bits) * sizeof(__u32) + + where lpi_id_bits is the LPI ID width configured in IRS_IST_CFGR. If no + LPI IST is configured, lpi_ist_addr and lpi_ist_size must both be zero. + + Errors: + + =========== ============================================================ + -EBUSY One or more VCPUs are running, the VGIC is not initialized, + restore was requested after the VM has run, an LPI IST + already exists, or the save operation completed but the VM + did not remain quiesced + -EINVAL attr->addr is NULL, the userspace IST descriptor is missing + a required buffer, supplies a non-zero LPI buffer when no + LPI IST is configured, has a buffer size that does not match + the configured VGICv5 IST state, or an internal VM table + operation rejected the VM state + -EFAULT Invalid user pointer for attr->addr, spi_ist_addr, or + lpi_ist_addr + -ENXIO Required per-VM VGICv5/IST backing state is missing or + inconsistent + -ENOMEM Restoring IST state failed while allocating the host LPI IST + or tracking pending interrupts + -ETIMEDOUT An IRS save/VM operation timed out + =========== ============================================================ + +IRS Save Sequence: +------------------ + +The following operations are required when saving the virtual GICv5 IRS: + +a) Save the ISTs by issuing KVM_GET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IST. +b) Save the IRS MMIO register state by issuing KVM_GET_DEVICE_ATTR on + KVM_DEV_ARM_VGIC_GRP_IRS_REGS. + +These two steps may be performed in either order. Saving the ISTs writes all +IST state to userspace-provided buffers and does not update guest memory. + +IRS Restore Sequence: +--------------------- + +The following ordering must be followed when restoring the virtual GICv5 and +IRS: + +a) Create vCPUs. +b) Provide the IRS base address by issuing KVM_SET_DEVICE_ATTR on + KVM_DEV_ARM_VGIC_GRP_ADDR +c) Restore the number of SPIs by issuing KVM_SET_DEVICE_ATTR on + KVM_DEV_ARM_VGIC_GRP_NR_IRQS. +d) Initialise the GIC - this sets up the default state and creates the SPI + IST - by issuing KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_CTRL with + KVM_DEV_ARM_VGIC_CTRL_INIT +e) Restore the IRS MMIO register state by issuing KVM_SET_DEVICE_ATTR on + KVM_DEV_ARM_VGIC_GRP_IRS_REGS. The IRS registers may be restored in any + order, but any IRS_IDR* state and the IRS_IST_CFGR/IRS_IST_BASER state + describing an LPI IST must be restored before restoring the ISTs. KVM uses + the restored IRS_IST_CFGR and IRS_IST_BASER state to allocate the LPI IST + during the following step. +f) Restore the ISTs by issuing KVM_SET_DEVICE_ATTR on + KVM_DEV_ARM_VGIC_GRP_IST. + +The number of SPIs must be restored before VGIC initialization because +initialization allocates the SPI state and fixes the SPI range exposed by the +IRS ID registers. + +The IRS's MMIO registers must be restored prior to restoring the ISTs as these +are used to convey the number of LPIs the guest has configured to KVM. + +The various ``*_STATUSR`` registers are observational state in the current KVM +implementation. Userspace may save them for validation or debugging purposes, +but they are not required as restore input and do not need to be replayed during +restore. + +Then vCPUs can be started. diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst index deb5c51bc00c..7e23584fd74a 100644 --- a/Documentation/virt/kvm/devices/vcpu.rst +++ b/Documentation/virt/kvm/devices/vcpu.rst @@ -135,7 +135,7 @@ after selecting a hardware PMU. Request that the VCPU uses the specified hardware PMU when creating guest events for the purpose of PMU emulation. The PMU identifier can be read from the "type" file for the desired PMU instance under /sys/devices (or, equivalent, -/sys/bus/even_source). This attribute is particularly useful on heterogeneous +/sys/bus/event_source). This attribute is particularly useful on heterogeneous systems where there are at least two CPU PMUs on the system. The PMU that is set for one VCPU will be used by all the other VCPUs. It isn't possible to set a PMU if a PMU event filter is already present. @@ -231,7 +231,8 @@ Returns: Specifies the base address of the stolen time structure for this VCPU. The base address must be 64 byte aligned and exist within a valid guest memory region. See Documentation/virt/kvm/arm/pvtime.rst for more information -including the layout of the stolen time structure. +including the layout of the stolen time structure. Stolen time is not +supported for protected VMs (see Documentation/virt/kvm/arm/pkvm.rst). 4. GROUP: KVM_VCPU_TSC_CTRL =========================== diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6b005c8fef70..a2ed76cfdc0a 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -45,6 +45,9 @@ KBUILD_CFLAGS += $(CC_FLAGS_NO_FPU) \ KBUILD_CFLAGS += $(call cc-disable-warning, psabi) KBUILD_AFLAGS += $(compat_vdso) +# Enable all code shared to s390 +KBUILD_CPPFLAGS += -DARM64_S390_COMMON=1 + ifeq ($(call rustc-min-version, 108500),y) KBUILD_RUSTFLAGS += --target=aarch64-unknown-none-softfloat else diff --git a/arch/arm64/include/asm/brk-imm.h b/arch/arm64/include/asm/brk-imm.h index beb42c62b6ac..dd2d153dc0d8 100644 --- a/arch/arm64/include/asm/brk-imm.h +++ b/arch/arm64/include/asm/brk-imm.h @@ -2,6 +2,7 @@ /* * Copyright (C) 2012 ARM Ltd. */ +/* Whole file is shared with s390 */ #ifndef __ASM_BRK_IMM_H #define __ASM_BRK_IMM_H diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index f816f5d77f1a..a584f88dbaf9 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -3,11 +3,12 @@ * Copyright (C) 2013 - ARM Ltd * Author: Marc Zyngier <marc.zyngier@arm.com> */ +/* Whole file is shared with s390 */ #ifndef __ASM_ESR_H #define __ASM_ESR_H -#include <asm/memory.h> +#include <asm/brk-imm.h> #include <asm/sysreg.h> #define ESR_ELx_EC_UNKNOWN UL(0x00) @@ -435,7 +436,36 @@ #define ESR_ELx_IT_GCSPOPX 7 #ifndef __ASSEMBLER__ -#include <asm/types.h> +#include <linux/types.h> + +static __always_inline bool esr_trap_is_iabt(unsigned long esr) +{ + return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; +} + +/* Always check for S1PTW *before* using this. */ +static __always_inline bool esr_dabt_is_write(unsigned long esr) +{ + return esr & ESR_ELx_WNR; +} + +static __always_inline bool esr_dabt_is_cm(unsigned long esr) +{ + return esr & ESR_ELx_CM; +} + +static __always_inline bool esr_abt_is_sea(unsigned long esr) +{ + switch (esr & ESR_ELx_FSC) { + case ESR_ELx_FSC_EXTABT: + case ESR_ELx_FSC_SEA_TTW(-1) ... ESR_ELx_FSC_SEA_TTW(3): + case ESR_ELx_FSC_SECC: + case ESR_ELx_FSC_SECC_TTW(-1) ... ESR_ELx_FSC_SECC_TTW(3): + return true; + default: + return false; + } +} static inline unsigned long esr_brk_comment(unsigned long esr) { diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 4bfbd827c5aa..678f0daed445 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -3,14 +3,16 @@ * Copyright (C) 2012,2013 - ARM Ltd * Author: Marc Zyngier <marc.zyngier@arm.com> */ +/* Whole file is shared with s390 */ #ifndef __ARM64_KVM_ARM_H__ #define __ARM64_KVM_ARM_H__ +#include <linux/const.h> +#include <linux/bits.h> +#include <linux/types.h> #include <asm/esr.h> -#include <asm/memory.h> #include <asm/sysreg.h> -#include <asm/types.h> /* * Because I'm terribly lazy and that repainting the whole of the KVM diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index e5b92ac09e69..7d799cd05dcd 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -62,6 +62,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_enable_ssbs, __KVM_HOST_SMCCC_FUNC___vgic_v3_init_lrs, __KVM_HOST_SMCCC_FUNC___vgic_v3_get_gic_config, + __KVM_HOST_SMCCC_FUNC___pkvm_hyp_alloc_selftest, MARKER(__KVM_HOST_SMCCC_FUNC_MIN_PKVM), @@ -87,14 +88,20 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___tracing_write_event, __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs, __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs, + __KVM_HOST_SMCCC_FUNC___vgic_v5_make_resident, + __KVM_HOST_SMCCC_FUNC___vgic_v5_make_non_resident, + __KVM_HOST_SMCCC_FUNC___vgic_v5_vdpend, __KVM_HOST_SMCCC_FUNC___vgic_v5_save_apr, __KVM_HOST_SMCCC_FUNC___vgic_v5_restore_vmcr_apr, + __KVM_HOST_SMCCC_FUNC___pkvm_hyp_topup, + __KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaim, + __KVM_HOST_SMCCC_FUNC___pkvm_hyp_reclaimable, + __KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp, + __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp, MARKER(__KVM_HOST_SMCCC_FUNC_PKVM_ONLY), /* Hypercalls that are available only when pKVM has finalised. */ - __KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp, - __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_hyp, __KVM_HOST_SMCCC_FUNC___pkvm_host_donate_guest, __KVM_HOST_SMCCC_FUNC___pkvm_host_share_guest, __KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_guest, diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index a3c1928bdf74..85289a4b8d52 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -44,6 +44,7 @@ enum exception_type { bool kvm_condition_valid32(const struct kvm_vcpu *vcpu); void kvm_skip_instr32(struct kvm_vcpu *vcpu); +#if ARM64_S390_COMMON == 1 void kvm_inject_undefined(struct kvm_vcpu *vcpu); void kvm_inject_sync(struct kvm_vcpu *vcpu, u64 esr); int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr); @@ -75,6 +76,8 @@ static inline int kvm_inject_serror(struct kvm_vcpu *vcpu) void kvm_vcpu_wfi(struct kvm_vcpu *vcpu); +#endif /* ARM64_S390_COMMON == 1 */ + void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu); int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2); int kvm_inject_nested_irq(struct kvm_vcpu *vcpu); @@ -133,12 +136,37 @@ static inline void vcpu_set_vsesr(struct kvm_vcpu *vcpu, u64 vsesr) static __always_inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu_gp_regs(vcpu)->pc; + return (unsigned long *)&vcpu->arch.ctxt.regs.pc; } static __always_inline unsigned long *vcpu_cpsr(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu_gp_regs(vcpu)->pstate; + return (unsigned long *)&vcpu->arch.ctxt.regs.pstate; +} + +static __always_inline unsigned long *vcpu_sp_el0(struct kvm_vcpu *vcpu) +{ + return (unsigned long *)&vcpu->arch.ctxt.regs.sp; +} + +static __always_inline u64 *vcpu_sp_el1(struct kvm_vcpu *vcpu) +{ + return __ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1); +} + +static __always_inline __u128 *vcpu_vreg(struct kvm_vcpu *vcpu, int n) +{ + return &vcpu->arch.ctxt.fp_regs.vregs[n]; +} + +static __always_inline __u32 *vcpu_fpsr(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.ctxt.fp_regs.fpsr; +} + +static __always_inline __u32 *vcpu_fpcr(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.ctxt.fp_regs.fpcr; } static __always_inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu) @@ -159,6 +187,7 @@ static inline void vcpu_set_thumb(struct kvm_vcpu *vcpu) *vcpu_cpsr(vcpu) |= PSR_AA32_T_BIT; } +#if ARM64_S390_COMMON == 1 /* * vcpu_get_reg and vcpu_set_reg should always be passed a register number * coming from a read of ESR_EL2. Otherwise, it may give the wrong result on @@ -167,16 +196,18 @@ static inline void vcpu_set_thumb(struct kvm_vcpu *vcpu) static __always_inline unsigned long vcpu_get_reg(const struct kvm_vcpu *vcpu, u8 reg_num) { - return (reg_num == 31) ? 0 : vcpu_gp_regs(vcpu)->regs[reg_num]; + return (reg_num == 31) ? 0 : vcpu_gp_regs(vcpu)[reg_num]; } static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, unsigned long val) { if (reg_num != 31) - vcpu_gp_regs(vcpu)->regs[reg_num] = val; + vcpu_gp_regs(vcpu)[reg_num] = val; } +#endif /* ARM64_S390_COMMON == 1 */ + static inline bool vcpu_is_el2_ctxt(const struct kvm_cpu_context *ctxt) { switch (ctxt->regs.pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) { @@ -379,6 +410,12 @@ static inline u64 kvm_vcpu_get_disr(const struct kvm_vcpu *vcpu) return vcpu->arch.fault.disr_el1; } +#if ARM64_S390_COMMON == 1 +static __always_inline bool esr_abt_is_s1ptw(unsigned long esr) +{ + return esr & ESR_ELx_S1PTW; +} + static inline u32 kvm_vcpu_hvc_get_imm(const struct kvm_vcpu *vcpu) { return kvm_vcpu_get_esr(vcpu) & ESR_ELx_xVC_IMM_MASK; @@ -411,18 +448,13 @@ static __always_inline int kvm_vcpu_dabt_get_rd(const struct kvm_vcpu *vcpu) static __always_inline bool kvm_vcpu_abt_iss1tw(const struct kvm_vcpu *vcpu) { - return !!(kvm_vcpu_get_esr(vcpu) & ESR_ELx_S1PTW); + return esr_abt_is_s1ptw(kvm_vcpu_get_esr(vcpu)); } /* Always check for S1PTW *before* using this. */ static __always_inline bool kvm_vcpu_dabt_iswrite(const struct kvm_vcpu *vcpu) { - return kvm_vcpu_get_esr(vcpu) & ESR_ELx_WNR; -} - -static inline bool kvm_vcpu_dabt_is_cm(const struct kvm_vcpu *vcpu) -{ - return !!(kvm_vcpu_get_esr(vcpu) & ESR_ELx_CM); + return esr_dabt_is_write(kvm_vcpu_get_esr(vcpu)); } static __always_inline unsigned int kvm_vcpu_dabt_get_as(const struct kvm_vcpu *vcpu) @@ -443,17 +475,19 @@ static __always_inline u8 kvm_vcpu_trap_get_class(const struct kvm_vcpu *vcpu) static inline bool kvm_vcpu_trap_is_iabt(const struct kvm_vcpu *vcpu) { - return kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_IABT_LOW; + return esr_trap_is_iabt(kvm_vcpu_get_esr(vcpu)); } -static inline bool kvm_vcpu_trap_is_exec_fault(const struct kvm_vcpu *vcpu) +static __always_inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) { - return kvm_vcpu_trap_is_iabt(vcpu) && !kvm_vcpu_abt_iss1tw(vcpu); + return kvm_vcpu_get_esr(vcpu) & ESR_ELx_FSC; } -static __always_inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) +#endif /* ARM64_S390_COMMON == 1 */ + +static __always_inline bool esr_abt_is_exec_fault(unsigned long esr) { - return kvm_vcpu_get_esr(vcpu) & ESR_ELx_FSC; + return esr_trap_is_iabt(esr) && !esr_abt_is_s1ptw(esr); } static inline @@ -468,26 +502,9 @@ bool kvm_vcpu_trap_is_translation_fault(const struct kvm_vcpu *vcpu) return esr_fsc_is_translation_fault(kvm_vcpu_get_esr(vcpu)); } -static inline -u64 kvm_vcpu_trap_get_perm_fault_granule(const struct kvm_vcpu *vcpu) -{ - unsigned long esr = kvm_vcpu_get_esr(vcpu); - - BUG_ON(!esr_fsc_is_permission_fault(esr)); - return BIT(ARM64_HW_PGTABLE_LEVEL_SHIFT(esr & ESR_ELx_FSC_LEVEL)); -} - static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu) { - switch (kvm_vcpu_trap_get_fault(vcpu)) { - case ESR_ELx_FSC_EXTABT: - case ESR_ELx_FSC_SEA_TTW(-1) ... ESR_ELx_FSC_SEA_TTW(3): - case ESR_ELx_FSC_SECC: - case ESR_ELx_FSC_SECC_TTW(-1) ... ESR_ELx_FSC_SECC_TTW(3): - return true; - default: - return false; - } + return esr_abt_is_sea(kvm_vcpu_get_esr(vcpu)); } static __always_inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu) @@ -496,9 +513,10 @@ static __always_inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu) return ESR_ELx_SYS64_ISS_RT(esr); } -static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) +#if ARM64_S390_COMMON == 1 +static inline bool esr_abt_is_write_fault(unsigned long esr) { - if (kvm_vcpu_abt_iss1tw(vcpu)) { + if (esr_abt_is_s1ptw(esr)) { /* * Only a permission fault on a S1PTW should be * considered as a write. Otherwise, page tables baked @@ -511,15 +529,22 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) * first), then a permission fault to allow the flags * to be set. */ - return kvm_vcpu_trap_is_permission_fault(vcpu); + return esr_fsc_is_permission_fault(esr); } - if (kvm_vcpu_trap_is_iabt(vcpu)) + if (esr_trap_is_iabt(esr)) return false; - return kvm_vcpu_dabt_iswrite(vcpu); + return esr_dabt_is_write(esr); } +static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) +{ + return esr_abt_is_write_fault(kvm_vcpu_get_esr(vcpu)); +} + +#endif /* ARM64_S390_COMMON == 1 */ + static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu) { return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; @@ -561,6 +586,7 @@ static inline bool kvm_vcpu_is_be(struct kvm_vcpu *vcpu) return vcpu_read_sys_reg(vcpu, r) & bit; } +#if ARM64_S390_COMMON == 1 static inline unsigned long vcpu_data_guest_to_host(struct kvm_vcpu *vcpu, unsigned long data, unsigned int len) @@ -636,6 +662,8 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu) vcpu_set_flag((v), e); \ } while (0) +#endif /* ARM64_S390_COMMON == 1 */ + /* * Returns a 'sanitised' view of CPTR_EL2, translating from nVHE to the VHE * format if E2H isn't set. @@ -737,6 +765,12 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu) } } +static inline void kvm_reset_fpsimd(struct kvm_vcpu *vcpu) +{ + memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs)); +} + +#if ARM64_S390_COMMON == 1 /* Reset a vcpu's core registers. */ static inline void kvm_reset_vcpu_core(struct kvm_vcpu *vcpu) { @@ -750,15 +784,19 @@ static inline void kvm_reset_vcpu_core(struct kvm_vcpu *vcpu) pstate = VCPU_RESET_PSTATE_EL1; /* Reset core registers */ - memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu))); - memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs)); + memset(vcpu_gp_regs(vcpu), 0, sizeof(vcpu_gp_regs(vcpu))); + *vcpu_pc(vcpu) = 0; + *vcpu_sp_el0(vcpu) = 0; + kvm_reset_fpsimd(vcpu); vcpu->arch.ctxt.spsr_abt = 0; vcpu->arch.ctxt.spsr_und = 0; vcpu->arch.ctxt.spsr_irq = 0; vcpu->arch.ctxt.spsr_fiq = 0; - vcpu_gp_regs(vcpu)->pstate = pstate; + *vcpu_cpsr(vcpu) = pstate; } +#endif /* ARM64_S390_COMMON == 1 */ + /* PSCI reset handling for a vcpu. */ static inline void kvm_reset_vcpu_psci(struct kvm_vcpu *vcpu, struct vcpu_reset_state *reset_state) diff --git a/arch/arm64/include/asm/kvm_hcall.h b/arch/arm64/include/asm/kvm_hcall.h new file mode 100644 index 000000000000..5f77a8aa2963 --- /dev/null +++ b/arch/arm64/include/asm/kvm_hcall.h @@ -0,0 +1,258 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * The host<->hyp hypercall interface. + * + * Copyright (C) 2026 Google LLC + * Author: Fuad Tabba <fuad.tabba@linux.dev> + */ + +#ifndef __ARM64_KVM_HCALL_H__ +#define __ARM64_KVM_HCALL_H__ + +#include <linux/args.h> +#include <linux/arm-smccc.h> +#include <linux/bug.h> +#include <linux/errno.h> +#include <linux/types.h> + +#include <asm/barrier.h> +#include <asm/kvm_asm.h> +#include <asm/spectre.h> +#include <asm/virt.h> + +typedef u16 pkvm_handle_t; + +enum pkvm_topup_id { + PKVM_TOPUP_HYP_ALLOC, + NR_PKVM_TOPUP_HYP_IDS, + PKVM_TOPUP_HYP_ALLOC_SELFTEST, +}; + +struct kvm; +struct kvm_s2_mmu; +struct kvm_vcpu; +struct vgic_v3_cpu_if; +struct vgic_v5_cpu_if; + +/* + * A host VA carried by a hypercall argument. At EL2 such a pointer must not + * be dereferenced until it is translated with kern_hyp_va_host(); sparse + * flags any use that skips the translation. The tag describes the EL2 view + * only: the host dereferences its own VAs freely. + */ +#if defined(__KVM_NVHE_HYPERVISOR__) && defined(__CHECKER__) +#define __kern __attribute__((noderef, address_space(__kern))) +#else +#define __kern +#endif + +/* + * Hypercall signatures are declared as (type, name) argument pairs. + * __KVM_HCALL_MAP() applies a macro to each pair, in the mold of __MAP() + * in <linux/syscalls.h>. The ladder is indexed by list entries, two per + * argument; __KVM_HCALL_MAP_N() takes that count explicitly. + */ +#define __KVM_HCALL_MAP2(m, t, a, ...) m(t, a) +#define __KVM_HCALL_MAP4(m, t, a, ...) m(t, a), __KVM_HCALL_MAP2(m, __VA_ARGS__) +#define __KVM_HCALL_MAP6(m, t, a, ...) m(t, a), __KVM_HCALL_MAP4(m, __VA_ARGS__) +#define __KVM_HCALL_MAP8(m, t, a, ...) m(t, a), __KVM_HCALL_MAP6(m, __VA_ARGS__) +#define __KVM_HCALL_MAP10(m, t, a, ...) m(t, a), __KVM_HCALL_MAP8(m, __VA_ARGS__) +#define __KVM_HCALL_MAP12(m, t, a, ...) m(t, a), __KVM_HCALL_MAP10(m, __VA_ARGS__) +#define __KVM_HCALL_MAP_N(n, m, ...) CONCATENATE(__KVM_HCALL_MAP, n)(m, __VA_ARGS__) +#define __KVM_HCALL_MAP(m, ...) __KVM_HCALL_MAP_N(COUNT_ARGS(__VA_ARGS__), m, __VA_ARGS__) + +#define __KVM_HCALL_DECL(t, a) t a +#define __KVM_HCALL_LONG(t, a) unsigned long a +#define __KVM_HCALL_CAST(t, a) (__force t) a +#define __KVM_HCALL_ARGS(t, a) a + +#ifndef __KVM_NVHE_HYPERVISOR__ +#define __kvm_call_hyp_nvhe(f, ...) \ + ({ \ + struct arm_smccc_res res; \ + \ + arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), \ + ##__VA_ARGS__, &res); \ + if (WARN_ON(res.a0 != SMCCC_RET_SUCCESS)) \ + res.a1 = -EOPNOTSUPP; \ + \ + res.a1; \ + }) + +/* + * Generate a typed stub for each declared hypercall. kvm_call_hyp_nvhe() + * resolves to the stub, so a call with the wrong argument count or types + * fails to compile instead of being silently truncated to an SMCCC function + * number and a pile of registers. The stub inlines to the same SMCCC call + * the untyped macro used to make. + */ +#define DECLARE_KVM_HOST_HCALL(ret, name, ...) \ + static __always_inline \ + ret nvhe_hvc_##name(__KVM_HCALL_MAP(__KVM_HCALL_DECL, __VA_ARGS__)) \ + { \ + return (ret)__kvm_call_hyp_nvhe(name, \ + __KVM_HCALL_MAP(__KVM_HCALL_ARGS, __VA_ARGS__));\ + } + +#define DECLARE_KVM_HOST_HCALL0(ret, name) \ + static __always_inline ret nvhe_hvc_##name(void) \ + { \ + return (ret)__kvm_call_hyp_nvhe(name); \ + } + +#define kvm_call_hyp_nvhe(f, ...) nvhe_hvc_##f(__VA_ARGS__) + +/* + * The isb() below is there to guarantee the same behaviour on VHE as on !VHE, + * where the eret to EL1 acts as a context synchronization event. + */ +#define kvm_call_hyp(f, ...) \ + do { \ + if (has_vhe()) { \ + f(__VA_ARGS__); \ + isb(); \ + } else { \ + kvm_call_hyp_nvhe(f, ##__VA_ARGS__); \ + } \ + } while (0) + +#define kvm_call_hyp_ret(f, ...) \ + ({ \ + typeof(f(__VA_ARGS__)) ret; \ + \ + if (has_vhe()) { \ + ret = f(__VA_ARGS__); \ + } else { \ + ret = kvm_call_hyp_nvhe(f, ##__VA_ARGS__); \ + } \ + \ + ret; \ + }) +#else /* __KVM_NVHE_HYPERVISOR__ */ +#define kvm_call_hyp(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_ret(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) + +/* + * At EL2 each declaration emits the canonical signature of the hypercall, + * which DEFINE_KVM_HOST_HCALL() in hyp-main.c checks the handler + * definition against. + */ +#define DECLARE_KVM_HOST_HCALL(ret, name, ...) \ + typedef ret kvm_host_hcall_sig_##name(__KVM_HCALL_MAP(__KVM_HCALL_DECL, __VA_ARGS__)); +#define DECLARE_KVM_HOST_HCALL0(ret, name) \ + typedef ret kvm_host_hcall_sig_##name(void); +#endif /* __KVM_NVHE_HYPERVISOR__ */ + +/* Hypercalls that are unavailable once pKVM has finalised. */ +DECLARE_KVM_HOST_HCALL(int, __pkvm_init, + phys_addr_t, phys, unsigned long, size, + unsigned long *, per_cpu_base, u32, hyp_va_bits) +DECLARE_KVM_HOST_HCALL(ulong, __pkvm_create_private_mapping, + phys_addr_t, phys, size_t, size, u64, prot) +DECLARE_KVM_HOST_HCALL(int, __pkvm_cpu_set_vector, + enum arm64_hyp_spectre_vector, slot) +DECLARE_KVM_HOST_HCALL0(void, __kvm_enable_ssbs) +DECLARE_KVM_HOST_HCALL0(void, __vgic_v3_init_lrs) +DECLARE_KVM_HOST_HCALL0(u64, __vgic_v3_get_gic_config) +DECLARE_KVM_HOST_HCALL0(int, __pkvm_hyp_alloc_selftest) + +DECLARE_KVM_HOST_HCALL0(int, __pkvm_prot_finalize) + +/* Hypercalls that are always available and common to [nh]VHE/pKVM. */ +DECLARE_KVM_HOST_HCALL(void, __kvm_adjust_pc, + struct kvm_vcpu __kern *, vcpu) +DECLARE_KVM_HOST_HCALL(int, __kvm_vcpu_run, + struct kvm_vcpu __kern *, vcpu) +DECLARE_KVM_HOST_HCALL0(void, __kvm_flush_vm_context) +DECLARE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_ipa, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, ipa, int, level) +DECLARE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_ipa_nsh, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, ipa, int, level) +DECLARE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid, + struct kvm_s2_mmu __kern *, mmu) +DECLARE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_range, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, start, unsigned long, pages) +DECLARE_KVM_HOST_HCALL(void, __kvm_flush_cpu_context, + struct kvm_s2_mmu __kern *, mmu) +DECLARE_KVM_HOST_HCALL(void, __kvm_timer_set_cntvoff, + u64, cntvoff) +DECLARE_KVM_HOST_HCALL(int, __tracing_load, + void __kern *, desc_hva, size_t, desc_size) +DECLARE_KVM_HOST_HCALL0(void, __tracing_unload) +DECLARE_KVM_HOST_HCALL(int, __tracing_enable, + bool, enable) +DECLARE_KVM_HOST_HCALL(int, __tracing_swap_reader, + unsigned int, cpu) +DECLARE_KVM_HOST_HCALL(void, __tracing_update_clock, + u32, mult, u32, shift, u64, epoch_ns, u64, epoch_cyc) +DECLARE_KVM_HOST_HCALL(int, __tracing_reset, + unsigned int, cpu) +DECLARE_KVM_HOST_HCALL(int, __tracing_enable_event, + unsigned short, id, bool, enable) +DECLARE_KVM_HOST_HCALL(void, __tracing_write_event, + u64, id) +DECLARE_KVM_HOST_HCALL(void, __vgic_v5_make_resident, + struct vgic_v5_cpu_if __kern *, cpu_if); +DECLARE_KVM_HOST_HCALL(void, __vgic_v5_make_non_resident, + struct vgic_v5_cpu_if __kern *, cpu_if); +DECLARE_KVM_HOST_HCALL(void, __vgic_v3_save_aprs, + struct vgic_v3_cpu_if __kern *, cpu_if) +DECLARE_KVM_HOST_HCALL(void, __vgic_v3_restore_vmcr_aprs, + struct vgic_v3_cpu_if __kern *, cpu_if) +DECLARE_KVM_HOST_HCALL(void, __vgic_v5_save_apr, + struct vgic_v5_cpu_if __kern *, cpu_if) +DECLARE_KVM_HOST_HCALL(void, __vgic_v5_restore_vmcr_apr, + struct vgic_v5_cpu_if __kern *, cpu_if) +DECLARE_KVM_HOST_HCALL(void, __vgic_v5_vdpend, + u32, intid, bool, pending, u16, vm); +DECLARE_KVM_HOST_HCALL(int, __pkvm_hyp_topup, + enum pkvm_topup_id, id, phys_addr_t, head, unsigned long, nr_pages) +DECLARE_KVM_HOST_HCALL(int, __pkvm_hyp_reclaim, + enum pkvm_topup_id, id, unsigned long, target) +DECLARE_KVM_HOST_HCALL(ulong, __pkvm_hyp_reclaimable, + enum pkvm_topup_id, id) + +/* Hypercalls that are available only when pKVM has finalised. */ +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_share_hyp, + u64, pfn) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_unshare_hyp, + u64, pfn) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_donate_guest, + u64, pfn, u64, gfn) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_share_guest, + u64, pfn, u64, gfn, u64, nr_pages, u64, prot) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_unshare_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_relax_perms_guest, + u64, gfn, u64, prot) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_wrprotect_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_test_clear_young_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages, bool, mkold) +DECLARE_KVM_HOST_HCALL(int, __pkvm_host_mkyoung_guest, + u64, gfn) +DECLARE_KVM_HOST_HCALL0(int, __pkvm_reserve_vm) +DECLARE_KVM_HOST_HCALL(void, __pkvm_unreserve_vm, + pkvm_handle_t, handle) +DECLARE_KVM_HOST_HCALL(int, __pkvm_init_vm, + struct kvm __kern *, host_kvm, void __kern *, pgd_hva) +DECLARE_KVM_HOST_HCALL(int, __pkvm_init_vcpu, + pkvm_handle_t, handle, struct kvm_vcpu __kern *, host_vcpu) +DECLARE_KVM_HOST_HCALL0(int, __pkvm_vcpu_in_poison_fault) +DECLARE_KVM_HOST_HCALL(int, __pkvm_force_reclaim_guest_page, + phys_addr_t, phys) +DECLARE_KVM_HOST_HCALL(int, __pkvm_reclaim_dying_guest_page, + pkvm_handle_t, handle, u64, gfn) +DECLARE_KVM_HOST_HCALL(int, __pkvm_start_teardown_vm, + pkvm_handle_t, handle) +DECLARE_KVM_HOST_HCALL(int, __pkvm_finalize_teardown_vm, + pkvm_handle_t, handle) +DECLARE_KVM_HOST_HCALL(void, __pkvm_vcpu_load, + pkvm_handle_t, handle, unsigned int, vcpu_idx, u64, hcr_el2) +DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_put) +DECLARE_KVM_HOST_HCALL0(void, __pkvm_vcpu_sync_state) +DECLARE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid, + pkvm_handle_t, handle) + +#endif /* __ARM64_KVM_HCALL_H__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e58f482d5160..5c9843338c51 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -14,6 +14,7 @@ #include <linux/arm-smccc.h> #include <linux/bitmap.h> #include <linux/types.h> +#include <linux/interval_tree.h> #include <linux/jump_label.h> #include <linux/kvm_types.h> #include <linux/maple_tree.h> @@ -27,6 +28,7 @@ #include <asm/fpsimd.h> #include <asm/kvm.h> #include <asm/kvm_asm.h> +#include <asm/kvm_hcall.h> #include <asm/vncr_mapping.h> #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -42,6 +44,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS +#if ARM64_S390_COMMON == 1 #define KVM_VCPU_MAX_FEATURES 10 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) @@ -58,6 +61,51 @@ #define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(9) #define KVM_REQ_MAP_L1_VNCR_EL2 KVM_ARCH_REQ(10) #define KVM_REQ_VGIC_PROCESS_UPDATE KVM_ARCH_REQ(11) +#define KVM_REQ_RELOAD_GICv5 KVM_ARCH_REQ(12) + +/* Bit indices for kvm_arch::flags */ +enum { + /* + * If we encounter a data abort without valid instruction syndrome + * information, report this to user space. User space can (and + * should) opt in to this feature if KVM_CAP_ARM_NISV_TO_USER is + * supported. + */ + KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER, + /* Memory Tagging Extension enabled for the guest */ + KVM_ARCH_FLAG_MTE_ENABLED, + /* At least one vCPU has ran in the VM */ + KVM_ARCH_FLAG_HAS_RAN_ONCE, + /* The vCPU feature set for the VM is configured */ + KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, + /* PSCI SYSTEM_SUSPEND enabled for the guest */ + KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, + /* VM counter offset */ + KVM_ARCH_FLAG_VM_COUNTER_OFFSET, + /* Timer PPIs made immutable */ + KVM_ARCH_FLAG_TIMER_PPIS_IMMUTABLE, + /* Initial ID reg values loaded */ + KVM_ARCH_FLAG_ID_REGS_INITIALIZED, + /* Fine-Grained UNDEF initialised */ + KVM_ARCH_FLAG_FGU_INITIALIZED, + /* SVE exposed to guest */ + KVM_ARCH_FLAG_GUEST_HAS_SVE, + /* MIDR_EL1, REVIDR_EL1, and AIDR_EL1 are writable from userspace */ + KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, + /* Unhandled SEAs are taken to userspace */ + KVM_ARCH_FLAG_EXIT_SEA, + /* No further ID register changes possible */ + KVM_ARCH_FLAG_ID_REGS_FINAL, +}; + +struct vcpu_reset_state { + unsigned long pc; + unsigned long r0; + bool be; + bool reset; +}; + +#endif /* ARM64_S390_COMMON == 1 */ #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) @@ -94,9 +142,22 @@ struct kvm_hyp_memcache { struct pkvm_mapping *mapping; /* only used from EL1 */ #define HYP_MEMCACHE_ACCOUNT_STAGE2 BIT(1) +#define HYP_MEMCACHE_ACCOUNT_KMEMCG BIT(2) unsigned long flags; }; +static inline void init_hyp_memcache(struct kvm_hyp_memcache *mc) +{ + memset(mc, 0, sizeof(*mc)); + mc->mapping = ZERO_SIZE_PTR; /* Prevent allocation, solely useful for stage2 memcache */ +} + +static inline void init_hyp_stage2_memcache(struct kvm_hyp_memcache *mc) +{ + memset(mc, 0, sizeof(*mc)); + mc->flags = HYP_MEMCACHE_ACCOUNT_STAGE2 | HYP_MEMCACHE_ACCOUNT_KMEMCG; +} + static inline void push_hyp_memcache(struct kvm_hyp_memcache *mc, phys_addr_t *p, phys_addr_t (*to_pa)(void *virt)) @@ -153,6 +214,36 @@ struct kvm_vmid { atomic64_t id; }; +/* + * Record of a guest stage-2 mapping, storing canonical and nested IPA + * ranges. Both ranges have the same size. The lower bits of nested.start + * store the index of the nested mmu this mapping belongs to. + */ +struct kvm_guest_s2_mapping { + struct interval_tree_node canonical; + struct interval_tree_node nested; +}; + +struct kvm_vcpu_ops { + void (*vcpu_load)(struct kvm_vcpu *vcpu); + void (*vcpu_put)(struct kvm_vcpu *vcpu); +}; + +struct kvm_gfn_range; +struct kvm_s2_fault_desc; + +struct kvm_vm_s2_ops { + bool (*vm_age_gfn)(struct kvm *kvm, struct kvm_gfn_range *range); + bool (*vm_test_age_gfn)(struct kvm *kvm, struct kvm_gfn_range *range); + int (*vm_flush_remote_tlbs)(struct kvm *kvm); + int (*vm_flush_remote_tlbs_range)(struct kvm *kvm, gfn_t gfn, + u64 nr_pages); + void (*vm_stage2_unmap_range)(struct kvm_s2_mmu *mmu, + phys_addr_t start, u64 size, + bool may_block); + int (*vm_mem_abort)(const struct kvm_s2_fault_desc *s2fd); +}; + struct kvm_s2_mmu { struct kvm_vmid vmid; @@ -214,22 +305,24 @@ struct kvm_s2_mmu { u64 tlb_vttbr; u64 tlb_vtcr; + /* Guest s2 mapping records indexed in this MMU's IPA space. */ + struct rb_root_cached guest_s2_mappings; + /* * true when this represents a nested context where virtual * HCR_EL2.VM == 1 */ bool nested_stage2_enabled; -#ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS - struct dentry *shadow_pt_debugfs_dentry; -#endif - /* * true when this MMU needs to be unmapped before being used for a new * purpose. */ bool pending_unmap; + /* Index in the S2 MMU array, only valid for a shadow S2 */ + u16 s2_mmu_idx; + /* * 0: Nobody is currently using this, check vttbr for validity * >0: Somebody is actively using this. @@ -254,13 +347,9 @@ struct kvm_smccc_features { unsigned long vendor_hyp_bmap_2; /* Function numbers 64-127 */ }; -typedef u16 pkvm_handle_t; - struct kvm_protected_vm { pkvm_handle_t handle; - struct kvm_hyp_memcache teardown_mc; struct kvm_hyp_memcache stage2_teardown_mc; - bool is_protected; bool is_created; /* @@ -309,8 +398,24 @@ enum fgt_group_id { __NR_FGT_GROUP_IDS__ }; +enum kvm_arm_vm_flavor { + VM_NVHE, + VM_VHE, + /* VMs running on a hyp that doesn't trust */ + MARKER(__VM_DISTRUSTING_HYP), + VM_PKVM, /* Normal guests on pKVM */ + MARKER(__VM_PROTECTED), + VM_PROTECTED_PKVM, /* Protected VM */ + VM_FLAVOR_MAX +}; + struct kvm_arch { struct kvm_s2_mmu mmu; + const struct kvm_vm_s2_ops *vm_s2_ops; + + enum kvm_arm_vm_flavor vm_flavor; + /* Mandated version of PSCI */ + u32 psci_version; /* * Fine-Grained UNDEF, mimicking the FGT layout defined by the @@ -329,47 +434,21 @@ struct kvm_arch { size_t nested_mmus_size; int nested_mmus_next; + /* + * Serializes guest s2 tracking trees access when the mmu_lock + * is only held for read. + */ + spinlock_t guest_s2_tracking_lock; + /* Interrupt controller */ struct vgic_dist vgic; /* Timers */ struct arch_timer_vm_data timer_data; - /* Mandated version of PSCI */ - u32 psci_version; - /* Protects VM-scoped configuration data */ struct mutex config_lock; - /* - * If we encounter a data abort without valid instruction syndrome - * information, report this to user space. User space can (and - * should) opt in to this feature if KVM_CAP_ARM_NISV_TO_USER is - * supported. - */ -#define KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER 0 - /* Memory Tagging Extension enabled for the guest */ -#define KVM_ARCH_FLAG_MTE_ENABLED 1 - /* At least one vCPU has ran in the VM */ -#define KVM_ARCH_FLAG_HAS_RAN_ONCE 2 - /* The vCPU feature set for the VM is configured */ -#define KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED 3 - /* PSCI SYSTEM_SUSPEND enabled for the guest */ -#define KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED 4 - /* VM counter offset */ -#define KVM_ARCH_FLAG_VM_COUNTER_OFFSET 5 - /* Timer PPIs made immutable */ -#define KVM_ARCH_FLAG_TIMER_PPIS_IMMUTABLE 6 - /* Initial ID reg values loaded */ -#define KVM_ARCH_FLAG_ID_REGS_INITIALIZED 7 - /* Fine-Grained UNDEF initialised */ -#define KVM_ARCH_FLAG_FGU_INITIALIZED 8 - /* SVE exposed to guest */ -#define KVM_ARCH_FLAG_GUEST_HAS_SVE 9 - /* MIDR_EL1, REVIDR_EL1, and AIDR_EL1 are writable from userspace */ -#define KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS 10 - /* Unhandled SEAs are taken to userspace */ -#define KVM_ARCH_FLAG_EXIT_SEA 11 unsigned long flags; /* VM-wide vCPU feature set */ @@ -838,17 +917,11 @@ extern s64 kvm_nvhe_sym(hyp_physvirt_offset); extern u64 kvm_nvhe_sym(hyp_cpu_logical_map)[NR_CPUS]; #define hyp_cpu_logical_map CHOOSE_NVHE_SYM(hyp_cpu_logical_map) -struct vcpu_reset_state { - unsigned long pc; - unsigned long r0; - bool be; - bool reset; -}; - struct vncr_tlb; struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; + const struct kvm_vcpu_ops *vcpu_ops; /* * Guest floating point state @@ -929,8 +1002,8 @@ struct kvm_vcpu_arch { /* Cache some mmu pages needed inside spinlock regions */ struct kvm_mmu_memory_cache mmu_page_cache; - /* Pages to top-up the pKVM/EL2 guest pool */ - struct kvm_hyp_memcache pkvm_memcache; + /* Pages to top-up the pKVM/EL2 guest stage-2 pool */ + struct kvm_hyp_memcache stage2_mc; /* Virtual SError ESR to restore when HCR_EL2.VSE is set */ u64 vsesr_el2; @@ -954,6 +1027,7 @@ struct kvm_vcpu_arch { pid_t pid; }; +#if ARM64_S390_COMMON == 1 /* * Each 'flag' is composed of a comma-separated triplet: * @@ -1036,11 +1110,28 @@ struct kvm_vcpu_arch { set; \ }) +#define __vcpu_copy_flag(vt, vs, flagset, f, m) \ + do { \ + typeof(vs->arch.flagset) tmp, val; \ + \ + __build_check_flag(vs, flagset, f, m); \ + \ + val = READ_ONCE(vs->arch.flagset); \ + val &= (m); \ + __vcpu_flags_preempt_disable(); \ + tmp = READ_ONCE(vt->arch.flagset); \ + tmp &= ~(m); \ + tmp |= val; \ + WRITE_ONCE(vt->arch.flagset, tmp); \ + __vcpu_flags_preempt_enable(); \ + } while (0) + #define vcpu_get_flag(v, ...) __vcpu_get_flag((v), __VA_ARGS__) #define vcpu_set_flag(v, ...) __vcpu_set_flag((v), __VA_ARGS__) #define vcpu_clear_flag(v, ...) __vcpu_clear_flag((v), __VA_ARGS__) #define vcpu_test_and_clear_flag(v, ...) \ __vcpu_test_and_clear_flag((v), __VA_ARGS__) +#define vcpu_copy_flag(vt, vs, ...) __vcpu_copy_flag((vt), (vs), __VA_ARGS__) /* KVM_ARM_VCPU_INIT completed */ #define VCPU_INITIALIZED __vcpu_single_flag(cflags, BIT(0)) @@ -1049,6 +1140,11 @@ struct kvm_vcpu_arch { /* pKVM VCPU setup completed */ #define VCPU_PKVM_FINALIZED __vcpu_single_flag(cflags, BIT(2)) +int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature); +bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); + +#define kvm_arm_vcpu_sve_finalized(vcpu) vcpu_get_flag(vcpu, VCPU_SVE_FINALIZED) + /* Exception pending */ #define PENDING_EXCEPTION __vcpu_single_flag(iflags, BIT(0)) /* @@ -1058,6 +1154,8 @@ struct kvm_vcpu_arch { #define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1)) /* Target EL/MODE (not a single flag, but let's abuse the macro) */ #define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1)) +/* Cover both PENDING_EXCEPTION and EXCEPT_MASK for global operations */ +#define PC_UPDATE_REQ __vcpu_single_flag(iflags, GENMASK(3, 0)) /* Host-set: the hyp flushes the non-protected vCPU state in on entry */ #define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4)) @@ -1107,6 +1205,8 @@ struct kvm_vcpu_arch { /* KVM is currently emulating an L2 to L1 exception */ #define IN_NESTED_EXCEPTION __vcpu_single_flag(sflags, BIT(9)) +#endif /* ARM64_S390_COMMON == 1 */ + #define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) #define vcpu_sve_zcr_elx(vcpu) \ @@ -1152,6 +1252,9 @@ struct kvm_vcpu_arch { #define vcpu_has_ptrauth(vcpu) false #endif +#define kvm_id_regs_final(kvm) \ + test_bit(KVM_ARCH_FLAG_ID_REGS_FINAL, &(kvm)->arch.flags) + #define vcpu_on_unsupported_cpu(vcpu) \ vcpu_get_flag(vcpu, ON_UNSUPPORTED_CPU) @@ -1161,7 +1264,7 @@ struct kvm_vcpu_arch { #define vcpu_clear_on_unsupported_cpu(vcpu) \ vcpu_clear_flag(vcpu, ON_UNSUPPORTED_CPU) -#define vcpu_gp_regs(v) (&(v)->arch.ctxt.regs) +#define vcpu_gp_regs(v) ((v)->arch.ctxt.regs.regs) /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the @@ -1242,14 +1345,6 @@ struct kvm_vcpu_stat { u64 exits; }; -unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu); -int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices); -int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg); -int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg); - -unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu); -int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices); - int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu, struct kvm_vcpu_events *events); @@ -1261,51 +1356,6 @@ void kvm_arm_resume_guest(struct kvm *kvm); #define vcpu_has_run_once(vcpu) (!!READ_ONCE((vcpu)->pid)) -#ifndef __KVM_NVHE_HYPERVISOR__ -#define kvm_call_hyp_nvhe(f, ...) \ - ({ \ - struct arm_smccc_res res; \ - \ - arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), \ - ##__VA_ARGS__, &res); \ - if (WARN_ON(res.a0 != SMCCC_RET_SUCCESS)) \ - res.a1 = -EOPNOTSUPP; \ - \ - res.a1; \ - }) - -/* - * The isb() below is there to guarantee the same behaviour on VHE as on !VHE, - * where the eret to EL1 acts as a context synchronization event. - */ -#define kvm_call_hyp(f, ...) \ - do { \ - if (has_vhe()) { \ - f(__VA_ARGS__); \ - isb(); \ - } else { \ - kvm_call_hyp_nvhe(f, ##__VA_ARGS__); \ - } \ - } while(0) - -#define kvm_call_hyp_ret(f, ...) \ - ({ \ - typeof(f(__VA_ARGS__)) ret; \ - \ - if (has_vhe()) { \ - ret = f(__VA_ARGS__); \ - } else { \ - ret = kvm_call_hyp_nvhe(f, ##__VA_ARGS__); \ - } \ - \ - ret; \ - }) -#else /* __KVM_NVHE_HYPERVISOR__ */ -#define kvm_call_hyp(f, ...) f(__VA_ARGS__) -#define kvm_call_hyp_ret(f, ...) f(__VA_ARGS__) -#define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) -#endif /* __KVM_NVHE_HYPERVISOR__ */ - int handle_exit(struct kvm_vcpu *vcpu, int exception_index); void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index); @@ -1328,6 +1378,15 @@ int __init populate_nv_trap_config(void); void kvm_calculate_traps(struct kvm_vcpu *vcpu); +#if ARM64_S390_COMMON == 1 +unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu); +int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices); +int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg); +int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg); + +unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu); +int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices); + /* MMIO helpers */ void kvm_mmio_write_buf(void *buf, unsigned int len, unsigned long data); unsigned long kvm_mmio_read_buf(const void *buf, unsigned int len); @@ -1335,6 +1394,8 @@ unsigned long kvm_mmio_read_buf(const void *buf, unsigned int len); int kvm_handle_mmio_return(struct kvm_vcpu *vcpu); int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa); +#endif /* ARM64_S390_COMMON == 1 */ + /* * Returns true if a Performance Monitoring Interrupt (PMI), a.k.a. perf event, * arrived in guest context. For arm64, any event that arrives while a vCPU is @@ -1349,7 +1410,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu); gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu); void kvm_update_stolen_time(struct kvm_vcpu *vcpu); -bool kvm_arm_pvtime_supported(void); +bool kvm_arm_pvtime_supported(struct kvm *kvm); int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, @@ -1507,14 +1568,30 @@ struct kvm *kvm_arch_alloc_vm(void); #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE -#define kvm_vm_is_protected(kvm) (is_protected_kvm_enabled() && (kvm)->arch.pkvm.is_protected) +#ifdef __KVM_NVHE_HYPERVISOR__ +#define kvm_vm_is_protected(kvm) \ + (is_protected_kvm_enabled() && ((kvm)->arch.vm_flavor == VM_PROTECTED_PKVM)) +/* + * Accessing vcpu->kvm from nVHE hyp stub is tricky, as we need to convert the + * pointer to the hyp VA. With pKVM, the nVHE code runs with the hyp_vcpu, + * which is populated correctly and is gated on is_protected_kvm_enabled(). + */ +#define vcpu_is_protected(vcpu) \ + ({ \ + struct kvm *__kvm = READ_ONCE((vcpu)->kvm); \ + \ + (__kvm && kvm_vm_is_protected(__kvm)); \ + }) + +#else + +#define kvm_vm_is_protected(kvm) ((kvm)->arch.vm_flavor >= __VM_PROTECTED) #define vcpu_is_protected(vcpu) kvm_vm_is_protected((vcpu)->kvm) -int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature); -bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); +#define kvm_vm_hyp_is_distrusting(kvm) ((kvm)->arch.vm_flavor >= __VM_DISTRUSTING_HYP) -#define kvm_arm_vcpu_sve_finalized(vcpu) vcpu_get_flag(vcpu, VCPU_SVE_FINALIZED) +#endif /* __KVM_NVHE_HYPERVISOR__ */ #define kvm_has_mte(kvm) \ (system_supports_mte() && \ @@ -1532,11 +1609,15 @@ static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature) return test_bit(feature, ka->vcpu_features); } +#if ARM64_S390_COMMON == 1 + #define kvm_vcpu_has_feature(k, f) __vcpu_has_feature(&(k)->arch, (f)) #define vcpu_has_feature(v, f) __vcpu_has_feature(&(v)->kvm->arch, (f)) #define kvm_vcpu_initialized(v) vcpu_get_flag(v, VCPU_INITIALIZED) +#endif /* ARM64_S390_COMMON == 1 */ + int kvm_trng_call(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM extern phys_addr_t hyp_mem_base; @@ -1548,6 +1629,7 @@ static inline void kvm_hyp_reserve(void) { } void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu); bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); +bool kvm_pkvm_vcpu_is_powered_off(struct kvm_vcpu *vcpu); static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg) { diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 4974492744cc..b0e42be3d9ee 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -10,6 +10,7 @@ #include <linux/compiler.h> #include <linux/kvm_host.h> #include <asm/alternative.h> +#include <asm/spectre.h> #include <asm/sysreg.h> DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); @@ -88,6 +89,8 @@ void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if); int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu); /* GICv5 */ +void __vgic_v5_make_resident(struct vgic_v5_cpu_if *cpu_if); +void __vgic_v5_make_non_resident(struct vgic_v5_cpu_if *cpu_if); void __vgic_v5_save_apr(struct vgic_v5_cpu_if *cpu_if); void __vgic_v5_restore_vmcr_apr(struct vgic_v5_cpu_if *cpu_if); /* No hypercalls for the following */ @@ -95,6 +98,7 @@ void __vgic_v5_save_ppi_state(struct vgic_v5_cpu_if *cpu_if); void __vgic_v5_restore_ppi_state(struct vgic_v5_cpu_if *cpu_if); void __vgic_v5_save_state(struct vgic_v5_cpu_if *cpu_if); void __vgic_v5_restore_state(struct vgic_v5_cpu_if *cpu_if); +void __vgic_v5_vdpend(u32 intid, bool pending, u16 vm); #ifdef __KVM_NVHE_HYPERVISOR__ void __timer_enable_traps(struct kvm_vcpu *vcpu); @@ -150,6 +154,9 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val); extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); +extern enum mitigation_state kvm_nvhe_sym(spectre_v2_state); +extern enum mitigation_state kvm_nvhe_sym(spectre_v4_state); +extern enum mitigation_state kvm_nvhe_sym(spectre_bhb_state); extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); extern unsigned long kvm_nvhe_sym(hyp_nr_cpus); extern unsigned int kvm_nvhe_sym(hyp_gicv3_nr_lr); diff --git a/arch/arm64/include/asm/kvm_hypevents.h b/arch/arm64/include/asm/kvm_hypevents.h index 5f6e6789d121..3396deb1a1f8 100644 --- a/arch/arm64/include/asm/kvm_hypevents.h +++ b/arch/arm64/include/asm/kvm_hypevents.h @@ -58,4 +58,18 @@ HYP_EVENT(selftest, ), RE_PRINTK("id=%llu", __entry->id) ); + +/* + * trace_hyp_printk() has too many specificities to be declared with HYP_EVENT(). + * However, we can use a REMOTE_EVENT macro to automatically do the declaration + * for the kernel side. + */ +REMOTE_EVENT_CUSTOM_PRINTK(hyp_printk, + 0, /* id will be overwritten during hyp event init */ + RE_STRUCT( + re_field(u16, fmt_id) + re_field(u64, args[]) + ), + __hyp_trace_printk(evt, seq) +); #endif diff --git a/arch/arm64/include/asm/kvm_hyptrace.h b/arch/arm64/include/asm/kvm_hyptrace.h index de133b735f72..16e8d60abba9 100644 --- a/arch/arm64/include/asm/kvm_hyptrace.h +++ b/arch/arm64/include/asm/kvm_hyptrace.h @@ -23,4 +23,13 @@ extern struct remote_event __hyp_events_end[]; extern struct hyp_event_id __hyp_event_ids_start[]; extern struct hyp_event_id __hyp_event_ids_end[]; +#define HYP_STRING_FMT_MAX_SIZE 127 + +struct hyp_string_fmt { + const char fmt[HYP_STRING_FMT_MAX_SIZE]; + u8 nr_args; +}; + +extern const struct hyp_string_fmt __hyp_string_fmts_start[]; +extern const struct hyp_string_fmt __hyp_string_fmts_end[]; #endif diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 6eae7e7e2a68..32392abf2e21 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -7,6 +7,8 @@ #ifndef __ARM64_KVM_MMU_H__ #define __ARM64_KVM_MMU_H__ +#include <linux/compiler.h> + #include <asm/page.h> #include <asm/memory.h> #include <asm/mmu.h> @@ -126,6 +128,9 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v) * replace the instructions with `nop`s. */ #ifndef __KVM_VHE_HYPERVISOR__ + if (is_protected_kvm_enabled() && !is_ttbr1_addr(v)) + return v; + asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n" /* mask with va_mask */ "ror %0, %0, #1\n" /* rotate to the first tag bit */ "add %0, %0, #0\n" /* insert the low 12 bits of the tag */ @@ -140,6 +145,14 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v) #define kern_hyp_va(v) ((typeof(v))(__kern_hyp_va((unsigned long)(v)))) +/* + * Translate a __kern-tagged host VA, dropping the tag: the only sanctioned + * unwrap. Translation only, no ownership or bounds validation; the result + * carries the pointee type stripped of the tag and of any cv-qualifiers. + */ +#define kern_hyp_va_host(v) \ + ((TYPEOF_UNQUAL(*(v)) *)__kern_hyp_va((unsigned long)(__force void *)(v))) + extern u32 __hyp_va_bits; /* @@ -392,12 +405,10 @@ static inline bool kvm_supports_cacheable_pfnmap(void) #ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS void kvm_s2_ptdump_create_debugfs(struct kvm *kvm); -void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu); -void kvm_nested_s2_ptdump_remove_debugfs(struct kvm_s2_mmu *mmu); +void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu, int idx); #else static inline void kvm_s2_ptdump_create_debugfs(struct kvm *kvm) {} -static inline void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu) {} -static inline void kvm_nested_s2_ptdump_remove_debugfs(struct kvm_s2_mmu *mmu) {} +static inline void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu, int idx) {} #endif /* CONFIG_PTDUMP_STAGE2_DEBUGFS */ #endif /* __ASSEMBLER__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 586026e85903..0f899addafa8 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -78,6 +78,11 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid, const union tlbi_info *info, void (*)(struct kvm_s2_mmu *, const union tlbi_info *)); +extern void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa, + gpa_t nested_ipa, size_t map_size, + struct kvm_guest_s2_mapping *mapping); +extern void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, + gpa_t nipa, size_t size); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); @@ -166,6 +171,8 @@ extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans); extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); extern void kvm_nested_s2_wp(struct kvm *kvm); +extern void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa, + size_t unmap_size, bool may_block); extern void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block); extern void kvm_nested_s2_flush(struct kvm *kvm); diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 41a8687938eb..1c9c57cd6ae4 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -7,10 +7,18 @@ #ifndef __ARM64_KVM_PGTABLE_H__ #define __ARM64_KVM_PGTABLE_H__ +#include <linux/bitfield.h> #include <linux/bits.h> -#include <linux/kvm_host.h> +#include <linux/kvm_types.h> +#include <linux/rbtree_types.h> +#include <linux/rcupdate.h> #include <linux/types.h> +#include <asm/memory.h> +#include <asm/page.h> + +struct kvm_s2_mmu; + #define KVM_PGTABLE_FIRST_LEVEL -1 #define KVM_PGTABLE_LAST_LEVEL 3 @@ -532,7 +540,7 @@ u64 kvm_pgtable_hyp_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size); * kvm_get_vtcr() - Helper to construct VTCR_EL2 * @mmfr0: Sanitized value of SYS_ID_AA64MMFR0_EL1 register. * @mmfr1: Sanitized value of SYS_ID_AA64MMFR1_EL1 register. - * @phys_shfit: Value to set in VTCR_EL2.T0SZ. + * @phys_shift: IPA size, in bits. * * The VTCR value is common across all the physical CPUs on the system. * We use system wide sanitised values to fill in different fields, @@ -859,6 +867,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, * @addr: Input address for the start of the walk. * @ptep: Pointer to storage for the retrieved PTE. * @level: Pointer to storage for the level of the retrieved PTE. + * @flags: Flags to control the page-table walk + * (see struct kvm_pgtable_visit_ctx). * * The offset of @addr within a page is ignored. * @@ -869,7 +879,8 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, * Return: 0 on success, negative error code on failure. */ int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr, - kvm_pte_t *ptep, s8 *level); + kvm_pte_t *ptep, s8 *level, + enum kvm_pgtable_walk_flags flags); /** * kvm_pgtable_stage2_pte_prot() - Retrieve the protection attributes of a diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h index cad60569f061..e7562770e7cb 100644 --- a/arch/arm64/include/asm/kvm_pkvm.h +++ b/arch/arm64/include/asm/kvm_pkvm.h @@ -7,9 +7,9 @@ #define __ARM64_KVM_PKVM_H__ #include <linux/arm_ffa.h> +#include <linux/kvm_host.h> #include <linux/memblock.h> #include <linux/scatterlist.h> -#include <asm/kvm_host.h> #include <asm/kvm_pgtable.h> /* Maximum number of VMs that can co-exist under pKVM. */ @@ -17,7 +17,8 @@ #define HYP_MEMBLOCK_REGIONS 128 -int pkvm_init_host_vm(struct kvm *kvm, unsigned long type); +void pkvm_selftests(void); +int pkvm_init_host_vm(struct kvm *kvm); int pkvm_create_hyp_vm(struct kvm *kvm); bool pkvm_hyp_vm_is_created(struct kvm *kvm); void pkvm_destroy_hyp_vm(struct kvm *kvm); @@ -42,18 +43,51 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext) case KVM_CAP_ARM_VM_IPA_SIZE: case KVM_CAP_ARM_PTRAUTH_ADDRESS: case KVM_CAP_ARM_PTRAUTH_GENERIC: + case KVM_CAP_ONE_REG: + case KVM_CAP_MP_STATE: + case KVM_CAP_VCPU_EVENTS: + case KVM_CAP_VCPU_ATTRIBUTES: + case KVM_CAP_IMMEDIATE_EXIT: + case KVM_CAP_IOEVENTFD: + case KVM_CAP_IRQFD_RESAMPLE: + case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2: + case KVM_CAP_ARM_INJECT_SERROR_ESR: return true; case KVM_CAP_ARM_MTE: - return false; case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE: case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES: + case KVM_CAP_PRE_FAULT_MEMORY: return false; default: - return !kvm || !kvm_vm_is_protected(kvm); + return !kvm || (kvm->arch.vm_flavor == VM_PKVM); } } /* + * The vCPU features a protected VM may use: checked by the host at + * KVM_ARM_VCPU_INIT, applied by EL2 when the hyp VM is created. + */ +static inline void kvm_pkvm_vcpu_allowed_features(struct kvm *kvm, + unsigned long *allowed) +{ + bitmap_zero(allowed, KVM_VCPU_MAX_FEATURES); + + set_bit(KVM_ARM_VCPU_PSCI_0_2, allowed); + + if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PMU_V3)) + set_bit(KVM_ARM_VCPU_PMU_V3, allowed); + + if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_ADDRESS)) + set_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, allowed); + + if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC)) + set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed); + + if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE)) + set_bit(KVM_ARM_VCPU_SVE, allowed); +} + +/* * Check whether the KVM VM IOCTL is allowed in pKVM. * * Certain features are allowed only for non-protected VMs in pKVM, which is why @@ -204,6 +238,116 @@ struct pkvm_mapping { u64 __subtree_last; /* Internal member for interval tree */ }; +enum pkvm_hyp_req_type { + PKVM_HYP_NO_REQ = 0, + PKVM_HYP_REQ_HYP_ALLOC, + PKVM_HYP_REQ_HYP_ALLOC_SELFTEST, + __PKVM_HYP_REQ_TYPE_MAX, +}; + +#define PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX \ + (sizeof(struct arm_smccc_res) - offsetof(struct arm_smccc_res, a2) - 1) + +struct pkvm_hyp_req { + u8 type; + union { + struct { + u32 nr_pages; + } mem; + struct { + /* Helper for SMCCC encoding/decoding */ + u8 args[PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX]; + } args; + }; +}; + +static inline size_t pkvm_hyp_req_arg_size(u8 type) +{ + struct pkvm_hyp_req *req; + + switch (type) { + case PKVM_HYP_NO_REQ: + return 0; + case PKVM_HYP_REQ_HYP_ALLOC: + case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST: + return sizeof(req->mem); + default: + WARN_ON(1); + } + + return 0; +} + +/* Encode the pending pkvm_hyp_req type into the SMCCC args */ +static inline void +pkvm_hyp_req_to_smccc(struct kvm_cpu_context *host_ctxt, struct pkvm_hyp_req *req) +{ + u8 *dst, type = req->type; + size_t size; + + if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX) { + host_ctxt->regs.regs[2] = 0; + return; + } + + size = pkvm_hyp_req_arg_size(type); + if (WARN_ON(size > PKVM_HYP_REQ_SMCCC_ARG_SIZE_MAX)) + return; + + dst = (u8 *)&host_ctxt->regs.regs[2]; + *dst = type; + + memcpy(dst + 1, &req->args, size); +} + +/* Return true if a pkvm_hyp_req has been decoded from the SMCCC args */ +static inline bool smccc_to_pkvm_hyp_req(struct pkvm_hyp_req *req, struct arm_smccc_res *res) +{ + u8 *src = (u8 *)res + offsetof(struct arm_smccc_res, a2); + u8 type = *src; + + if (type == PKVM_HYP_NO_REQ || type >= __PKVM_HYP_REQ_TYPE_MAX) + return false; + + req->type = type; + memcpy(&req->args, src + 1, pkvm_hyp_req_arg_size(type)); + + return true; +} + +int __pkvm_handle_smccc_req(struct arm_smccc_res *res); + +/** + * pkvm_call_hyp_req() - Issue an HVC that can return hypervisor requests + * @f: Hypervisor function symbol to call. + * @...: Arguments to pass to the hypercall. + * + * Re-issue an HVC and process any pending hypervisor request until completion + * or error. + * + * Only use this helper for HVCs whose hypervisor handlers format their return + * registers with pkvm_hyp_req_to_smccc(). + * + * Return: Result of the hypercall or a negative error if the hyp request + * handling failed. + */ +#define pkvm_call_hyp_req(f, ...) \ +({ \ + struct arm_smccc_res __res; \ + int __ret; \ + do { \ + __ret = -1; \ + arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(f), ##__VA_ARGS__, &__res); \ + if (WARN_ON(__res.a0 != SMCCC_RET_SUCCESS)) \ + break; \ + __ret = __res.a1; \ + if (!__ret) \ + break; \ + __ret = __pkvm_handle_smccc_req(&__res); \ + } while (!__ret); \ + __ret; \ +}) + int pkvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, struct kvm_pgtable_mm_ops *mm_ops); void pkvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt, diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h index f7dc5fb9427d..4d88ed5cc425 100644 --- a/arch/arm64/include/asm/ptrace.h +++ b/arch/arm64/include/asm/ptrace.h @@ -26,6 +26,8 @@ #define GIC_PRIO_PSR_I_SET GICV3_PRIO_PSR_I_SET +#if ARM64_S390_COMMON == 1 + /* Additional SPSR bits not exposed in the UABI */ #define PSR_MODE_THREAD_BIT (1 << 0) #define PSR_IL_BIT (1 << 20) @@ -66,6 +68,8 @@ #define PSR_AA32_IT_MASK 0x0600fc00 /* If-Then execution state mask */ #define PSR_AA32_GE_MASK 0x000f0000 +#endif /* ARM64_S390_COMMON == 1 */ + #ifdef CONFIG_CPU_BIG_ENDIAN #define PSR_AA32_ENDSTATE PSR_AA32_E_BIT #else @@ -102,26 +106,48 @@ #define COMPAT_USER_SZ 296 /* Architecturally defined mapping between AArch32 and AArch64 registers */ +enum { + __compat_fp = 11, + __compat_sp = 13, + __compat_lr = 14, + __compat_sp_hyp = 15, + __compat_lr_irq = 16, + __compat_sp_irq = 17, + __compat_lr_svc = 18, + __compat_sp_svc = 19, + __compat_lr_abt = 20, + __compat_sp_abt = 21, + __compat_lr_und = 22, + __compat_sp_und = 23, + __compat_r8_fiq = 24, + __compat_r9_fiq = 25, + __compat_r10_fiq = 26, + __compat_r11_fiq = 27, + __compat_r12_fiq = 28, + __compat_sp_fiq = 29, + __compat_lr_fiq = 30, +}; + #define compat_usr(x) regs[(x)] -#define compat_fp regs[11] -#define compat_sp regs[13] -#define compat_lr regs[14] -#define compat_sp_hyp regs[15] -#define compat_lr_irq regs[16] -#define compat_sp_irq regs[17] -#define compat_lr_svc regs[18] -#define compat_sp_svc regs[19] -#define compat_lr_abt regs[20] -#define compat_sp_abt regs[21] -#define compat_lr_und regs[22] -#define compat_sp_und regs[23] -#define compat_r8_fiq regs[24] -#define compat_r9_fiq regs[25] -#define compat_r10_fiq regs[26] -#define compat_r11_fiq regs[27] -#define compat_r12_fiq regs[28] -#define compat_sp_fiq regs[29] -#define compat_lr_fiq regs[30] +#define compat_fp regs[__compat_fp] +#define compat_sp regs[__compat_sp] +#define compat_lr regs[__compat_lr] +#define compat_sp_hyp regs[__compat_sp_hyp] +#define compat_lr_irq regs[__compat_lr_irq] +#define compat_sp_irq regs[__compat_sp_irq] +#define compat_lr_svc regs[__compat_lr_svc] +#define compat_sp_svc regs[__compat_sp_svc] +#define compat_lr_abt regs[__compat_lr_abt] +#define compat_sp_abt regs[__compat_sp_abt] +#define compat_lr_und regs[__compat_lr_und] +#define compat_sp_und regs[__compat_sp_und] +#define compat_r8_fiq regs[__compat_r8_fiq] +#define compat_r9_fiq regs[__compat_r9_fiq] +#define compat_r10_fiq regs[__compat_r10_fiq] +#define compat_r11_fiq regs[__compat_r11_fiq] +#define compat_r12_fiq regs[__compat_r12_fiq] +#define compat_sp_fiq regs[__compat_sp_fiq] +#define compat_lr_fiq regs[__compat_lr_fiq] static inline unsigned long compat_psr_to_pstate(const unsigned long psr) { diff --git a/arch/arm64/include/asm/stacktrace/nvhe.h b/arch/arm64/include/asm/stacktrace/nvhe.h index 171f9edef49f..a631a577cbe5 100644 --- a/arch/arm64/include/asm/stacktrace/nvhe.h +++ b/arch/arm64/include/asm/stacktrace/nvhe.h @@ -37,6 +37,14 @@ static inline void kvm_nvhe_unwind_init(struct unwind_state *state, state->pc = pc; } +DECLARE_KVM_NVHE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); +DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info); + +#ifdef CONFIG_PKVM_STACKTRACE +DECLARE_KVM_NVHE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], + pkvm_stacktrace); +#endif + #ifndef __KVM_NVHE_HYPERVISOR__ /* * Conventional (non-protected) nVHE HYP stack unwinder @@ -45,8 +53,6 @@ static inline void kvm_nvhe_unwind_init(struct unwind_state *state, * (by the host in EL1). */ -DECLARE_KVM_NVHE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); -DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info); DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base); void kvm_nvhe_dump_backtrace(unsigned long hyp_offset); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 7aa08d59d494..b4f5dedff5ae 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -16,6 +16,8 @@ #include <asm/gpr-num.h> +#if ARM64_S390_COMMON == 1 + /* * ARMv8 ARM reserves the following encoding for system registers: * (Ref: ARMv8 ARM, Section: "System instruction class encoding overview", @@ -50,6 +52,8 @@ #define sys_reg_CRm(id) (((id) >> CRm_shift) & CRm_mask) #define sys_reg_Op2(id) (((id) >> Op2_shift) & Op2_mask) +#endif /* ARM64_S390_COMMON == 1 */ + #ifndef CONFIG_BROKEN_GAS_INST #ifdef __ASSEMBLER__ @@ -123,6 +127,8 @@ #define GSB_SYS_BARRIER_INSN __SYS_BARRIER_INSN(1, 0, 12, 0, 0, 31) #define GSB_ACK_BARRIER_INSN __SYS_BARRIER_INSN(1, 0, 12, 0, 1, 31) +#if ARM64_S390_COMMON == 1 + /* Data cache zero operations */ #define SYS_DC_ISW sys_insn(1, 0, 7, 6, 2) #define SYS_DC_IGSW sys_insn(1, 0, 7, 6, 4) @@ -295,7 +301,6 @@ #define SYS_APGAKEYLO_EL1 sys_reg(3, 0, 2, 3, 0) #define SYS_APGAKEYHI_EL1 sys_reg(3, 0, 2, 3, 1) -#define SYS_SPSR_EL1 sys_reg(3, 0, 4, 0, 0) #define SYS_ELR_EL1 sys_reg(3, 0, 4, 0, 1) #define SYS_ICC_PMR_EL1 sys_reg(3, 0, 4, 6, 0) @@ -518,7 +523,6 @@ #define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) #define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6) -#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0) #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) #define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) #define SYS_SPSR_irq sys_reg(3, 4, 4, 3, 0) @@ -604,7 +608,6 @@ #define SYS_BRBCR_EL12 sys_reg(2, 5, 9, 0, 0) #define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0) #define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1) -#define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0) #define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) #define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0) #define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1) @@ -835,6 +838,8 @@ #define SCTLR_ELx_A (BIT(1)) #define SCTLR_ELx_M (BIT(0)) +#endif /* ARM64_S390_COMMON == 1 */ + #ifdef CONFIG_CPU_BIG_ENDIAN #define ENDIAN_SET_EL2 SCTLR_ELx_EE #else @@ -869,6 +874,7 @@ SCTLR_EL1_LSMAOE | SCTLR_EL1_nTLSMD | SCTLR_EL1_EIS | \ SCTLR_EL1_TSCXT | SCTLR_EL1_EOS) +#if ARM64_S390_COMMON == 1 /* MAIR_ELx memory attributes (used by Linux) */ #define MAIR_ATTR_DEVICE_nGnRnE UL(0x00) #define MAIR_ATTR_DEVICE_nGnRE UL(0x04) @@ -1040,7 +1046,7 @@ #define GCS_CAP(x) ((((unsigned long)x) & GCS_CAP_ADDR_MASK) | \ GCS_CAP_VALID_TOKEN) /* - * Definitions for GICv5 instructions + * Definitions for GICv5 instructions for the Current Domain */ #define GICV5_OP_GIC_CDAFF sys_insn(1, 0, 12, 1, 3) #define GICV5_OP_GIC_CDDI sys_insn(1, 0, 12, 2, 0) @@ -1105,6 +1111,19 @@ #define GICV5_GICR_CDNMIA_TYPE_MASK GENMASK_ULL(31, 29) #define GICV5_GICR_CDNMIA_ID_MASK GENMASK_ULL(23, 0) +/* + * Definitions for GICv5 instructions for the Virtual Domain + */ +#define GICV5_OP_GIC_VDPEND sys_insn(1, 4, 12, 1, 4) + +/* Shift and mask definitions for GIC VDPEND */ +#define GICV5_GIC_VDPEND_PENDING_MASK BIT_ULL(63) +#define GICV5_GIC_VDPEND_VM_MASK GENMASK_ULL(47, 32) +#define GICV5_GIC_VDPEND_TYPE_MASK GENMASK_ULL(31, 29) +#define GICV5_GIC_VDPEND_ID_MASK GENMASK_ULL(23, 0) + +#endif /* ARM64_S390_COMMON == 1 */ + #define gicr_insn(insn) read_sysreg_s(GICV5_OP_GICR_##insn) #define gic_insn(v, insn) write_sysreg_s(v, GICV5_OP_GIC_##insn) @@ -1254,6 +1273,8 @@ par; \ }) +#if ARM64_S390_COMMON == 1 + #define SYS_FIELD_VALUE(reg, field, val) reg##_##field##_##val #define SYS_FIELD_GET(reg, field, val) \ @@ -1266,6 +1287,8 @@ FIELD_PREP(reg##_##field##_MASK, \ SYS_FIELD_VALUE(reg, field, val)) -#endif +#endif /* ARM64_S390_COMMON == 1 */ + +#endif /* __ASSEMBLER__ */ #endif /* __ASM_SYSREG_H */ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 019e5e3d892e..65ee09bd1463 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -97,6 +97,11 @@ struct kvm_regs { #define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K) #define KVM_VGIC_V3_ITS_SIZE (2 * SZ_64K) +/* Supported VGICv5 address types */ +#define KVM_VGIC_V5_ADDR_TYPE_IRS 6 + +#define KVM_VGIC_V5_IRS_SIZE (2 * SZ_64K) + #define KVM_ARM_VCPU_POWER_OFF 0 /* CPU is started in OFF state */ #define KVM_ARM_VCPU_EL1_32BIT 1 /* CPU running a 32bit VM */ #define KVM_ARM_VCPU_PSCI_0_2 2 /* CPU uses PSCI v0.2 */ @@ -418,6 +423,16 @@ enum { #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 #define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 +#define KVM_DEV_ARM_VGIC_GRP_IRS_REGS 10 +#define KVM_DEV_ARM_VGIC_GRP_IST 11 + +struct kvm_vgic_v5_ist { + __u64 spi_ist_addr; + __u64 spi_ist_size; + __u64 lpi_ist_addr; + __u64 lpi_ist_size; +}; + #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 14beb7b9d304..ef9875a95f95 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -143,6 +143,7 @@ KVM_NVHE_ALIAS(__hyp_rodata_end); #ifdef CONFIG_NVHE_EL2_TRACING KVM_NVHE_ALIAS(__hyp_event_ids_start); KVM_NVHE_ALIAS(__hyp_event_ids_end); +KVM_NVHE_ALIAS(__hyp_string_fmts_start); #endif /* pKVM static key */ diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index be92d73c25b2..96243c291e39 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -34,6 +34,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) + $(call cmd,force_checksrc) obj-y := idreg-override.pi.o \ map_kernel.pi.o map_range.pi.o \ diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index af1d72020976..1869b44ae190 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -15,7 +15,7 @@ #ifdef CONFIG_NVHE_EL2_TRACING #define HYPERVISOR_EVENT_IDS \ - . = ALIGN(PAGE_SIZE); \ + . = ALIGN(SZ_8); \ __hyp_event_ids_start = .; \ *(HYP_SECTION_NAME(.event_ids)) \ __hyp_event_ids_end = .; @@ -214,6 +214,14 @@ SECTIONS HYPERVISOR_RODATA_SECTIONS +#ifdef CONFIG_NVHE_EL2_TRACING + .rodata.hyp_string_fmts : { + __hyp_string_fmts_start = .; + *(_hyp_string_fmts) + __hyp_string_fmts_end = .; + } +#endif + .got : { *(.got) } /* * Make sure that the .got.plt is either completely empty or it diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 8475f32ad9e6..48d531e302d3 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -33,6 +33,7 @@ menuconfig KVM select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS select KVM_GUEST_MEMFD + select KVM_GENERIC_PRE_FAULT_MEMORY help Support hosting virtualized guest machines. diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 59612d2f277c..7aaeeb84e788 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -24,7 +24,8 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ - vgic/vgic-v5.o + vgic/vgic-v5.o vgic/vgic-v5-tables.o vgic/vgic-irs-v5.o \ + vgic-sys-reg-v5.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) += pauth.o diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index 6ac3321f4c57..b2c5ba499aa5 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -528,17 +528,11 @@ static void timer_save_state(struct arch_timer_context *ctx) goto out; switch (index) { - u64 cval; - case TIMER_VTIMER: case TIMER_HVTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTV_CTL)); - cval = read_sysreg_el0(SYS_CNTV_CVAL); - - if (has_broken_cntvoff()) - cval -= timer_get_offset(ctx); - - timer_set_cval(ctx, cval); + if (!has_broken_cntvoff() || !timer_get_offset(ctx)) + timer_set_cval(ctx, read_sysreg_el0(SYS_CNTV_CVAL)); /* Disable the timer */ write_sysreg_el0(0, SYS_CNTV_CTL); @@ -564,11 +558,12 @@ static void timer_save_state(struct arch_timer_context *ctx) case TIMER_PTIMER: case TIMER_HPTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL)); - cval = read_sysreg_el0(SYS_CNTP_CVAL); - - cval -= timer_get_offset(ctx); - - timer_set_cval(ctx, cval); + /* + * With an offset, memory already holds the guest's CVAL (the + * trap handler or __deactivate_traps() wrote it). + */ + if (!timer_get_offset(ctx)) + timer_set_cval(ctx, read_sysreg_el0(SYS_CNTP_CVAL)); /* Disable the timer */ write_sysreg_el0(0, SYS_CNTP_CTL); @@ -647,7 +642,8 @@ static void timer_restore_state(struct arch_timer_context *ctx) offset = timer_get_offset(ctx); if (has_broken_cntvoff()) { set_cntvoff(0); - cval += offset; + if (offset) + cval = timer_apply_offset(cval, offset, kvm_phys_timer_read()); } else { set_cntvoff(offset); } @@ -660,7 +656,8 @@ static void timer_restore_state(struct arch_timer_context *ctx) cval = timer_get_cval(ctx); offset = timer_get_offset(ctx); set_cntpoff(offset); - cval += offset; + if (offset) + cval = timer_apply_offset(cval, offset, kvm_phys_timer_read()); write_sysreg_el0(cval, SYS_CNTP_CVAL); isb(); write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTP_CTL); @@ -1110,8 +1107,7 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) timer_context_init(vcpu, i); /* Synchronize offsets across timers of a VM if not already provided */ - if (!vcpu_is_protected(vcpu) && - !test_bit(KVM_ARCH_FLAG_VM_COUNTER_OFFSET, &vcpu->kvm->arch.flags)) { + if (!test_bit(KVM_ARCH_FLAG_VM_COUNTER_OFFSET, &vcpu->kvm->arch.flags)) { timer_set_offset(vcpu_vtimer(vcpu), kvm_phys_timer_read()); timer_set_offset(vcpu_ptimer(vcpu), 0); } @@ -1133,6 +1129,15 @@ void kvm_timer_init_vm(struct kvm *kvm) */ for (int i = 0; i < NR_KVM_TIMERS; i++) kvm->arch.timer_data.ppi[i] = get_vgic_ppi(kvm, default_ppi[i]); + + /* + * Protected VMs don't allow the userspace to set counter offsets, + * either set via counter register writes or the dedicated ioctls. + * Pretend the offset has already been set and rely on the default + * offset being 0. + */ + if (kvm_vm_is_protected(kvm)) + set_bit(KVM_ARCH_FLAG_VM_COUNTER_OFFSET, &kvm->arch.flags); } void kvm_timer_cpu_up(void) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index eaf583b77193..d593ba928093 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -93,6 +93,7 @@ static const struct kvm_ioctl_cap_map vm_ioctl_caps[] = { { KVM_ARM_PREFERRED_TARGET, KVM_CAP_ARM_BASIC }, }; +static void kvm_init_vcpu_ops(struct kvm_vcpu *vcpu); /* * Set *ext to the capability. * Return 0 if found, or -EINVAL if no IOCTL matches. @@ -214,6 +215,26 @@ static int kvm_arm_default_max_vcpus(void) return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; } +static int kvm_init_vm_flavor(struct kvm *kvm, unsigned long type) +{ + bool protected = type & KVM_VM_TYPE_ARM_PROTECTED; + + if (is_protected_kvm_enabled()) { + if (protected) + kvm->arch.vm_flavor = VM_PROTECTED_PKVM; + else + kvm->arch.vm_flavor = VM_PKVM; + } else if (protected) { + return -EINVAL; + } else if (has_vhe()) { + kvm->arch.vm_flavor = VM_VHE; + } else { + kvm->arch.vm_flavor = VM_NVHE; + } + + return 0; +} + /** * kvm_arch_init_vm - initializes a VM data structure * @kvm: pointer to the KVM struct @@ -236,6 +257,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) mutex_unlock(&kvm->lock); #endif + ret = kvm_init_vm_flavor(kvm, type); + if (ret) + return ret; + ret = kvm_share_hyp(kvm, kvm + 1); if (ret) return ret; @@ -259,12 +284,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) * If any failures occur after this is successful, make sure to * call __pkvm_unreserve_vm to unreserve the VM in hyp. */ - ret = pkvm_init_host_vm(kvm, type); + ret = pkvm_init_host_vm(kvm); if (ret) goto err_uninit_mmu; - } else if (type & KVM_VM_TYPE_ARM_PROTECTED) { - ret = -EINVAL; - goto err_uninit_mmu; } kvm_vgic_early_init(kvm); @@ -410,6 +432,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_COUNTER_OFFSET: case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS: case KVM_CAP_ARM_SEA_TO_USER: + case KVM_CAP_PRE_FAULT_MEMORY: r = 1; break; case KVM_CAP_SET_GUEST_DEBUG2: @@ -451,7 +474,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = system_supports_mte(); break; case KVM_CAP_STEAL_TIME: - r = kvm_arm_pvtime_supported(); + r = kvm_arm_pvtime_supported(kvm); break; case KVM_CAP_ARM_EL1_32BIT: r = cpus_have_final_cap(ARM64_HAS_32BIT_EL1); @@ -552,6 +575,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) mutex_unlock(&vcpu->mutex); #endif + kvm_init_vcpu_ops(vcpu); + /* Force users to call KVM_ARM_VCPU_INIT */ vcpu_clear_flag(vcpu, VCPU_INITIALIZED); @@ -595,7 +620,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) if (!is_protected_kvm_enabled()) kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache); else - free_hyp_memcache(&vcpu->arch.pkvm_memcache); + free_hyp_memcache(&vcpu->arch.stage2_mc); kvm_timer_vcpu_terminate(vcpu); kvm_pmu_vcpu_destroy(vcpu); kvm_vgic_vcpu_destroy(vcpu); @@ -614,33 +639,34 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) static void vcpu_set_pauth_traps(struct kvm_vcpu *vcpu) { - if (vcpu_has_ptrauth(vcpu) && !is_protected_kvm_enabled()) { - /* - * Either we're running an L2 guest, and the API/APK bits come - * from L1's HCR_EL2, or API/APK are both set. - */ - if (unlikely(is_nested_ctxt(vcpu))) { - u64 val; + if (!vcpu_has_ptrauth(vcpu)) + return; - val = __vcpu_sys_reg(vcpu, HCR_EL2); - val &= (HCR_API | HCR_APK); - vcpu->arch.hcr_el2 &= ~(HCR_API | HCR_APK); - vcpu->arch.hcr_el2 |= val; - } else { - vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK); - } + /* + * Either we're running an L2 guest, and the API/APK bits come + * from L1's HCR_EL2, or API/APK are both set. + */ + if (unlikely(is_nested_ctxt(vcpu))) { + u64 val; - /* - * Save the host keys if there is any chance for the guest - * to use pauth, as the entry code will reload the guest - * keys in that case. - */ - if (vcpu->arch.hcr_el2 & (HCR_API | HCR_APK)) { - struct kvm_cpu_context *ctxt; + val = __vcpu_sys_reg(vcpu, HCR_EL2); + val &= (HCR_API | HCR_APK); + vcpu->arch.hcr_el2 &= ~(HCR_API | HCR_APK); + vcpu->arch.hcr_el2 |= val; + } else { + vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK); + } - ctxt = this_cpu_ptr_hyp_sym(kvm_hyp_ctxt); - ptrauth_save_keys(ctxt); - } + /* + * Save the host keys if there is any chance for the guest + * to use pauth, as the entry code will reload the guest + * keys in that case. + */ + if (vcpu->arch.hcr_el2 & (HCR_API | HCR_APK)) { + struct kvm_cpu_context *ctxt; + + ctxt = this_cpu_ptr_hyp_sym(kvm_hyp_ctxt); + ptrauth_save_keys(ctxt); } } @@ -666,14 +692,11 @@ static bool kvm_vcpu_should_clear_twe(struct kvm_vcpu *vcpu) return single_task_running(); } -void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +static void vcpu_prepare_mmu(struct kvm_vcpu *vcpu) { struct kvm_s2_mmu *mmu; int *last_ran; - if (is_protected_kvm_enabled()) - goto nommu; - if (vcpu_has_nv(vcpu)) kvm_vcpu_load_hw_mmu(vcpu); @@ -703,44 +726,91 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) kvm_call_hyp(__kvm_flush_cpu_context, mmu); *last_ran = vcpu->vcpu_idx; } +} -nommu: - vcpu->cpu = cpu; +static void vcpu_put_mmu(struct kvm_vcpu *vcpu) +{ + kvm_arm_vmid_clear_active(); +} + +static void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu) +{ + if (kvm_vcpu_should_clear_twe(vcpu)) + vcpu->arch.hcr_el2 &= ~HCR_TWE; + else + vcpu->arch.hcr_el2 |= HCR_TWE; + if (kvm_vcpu_should_clear_twi(vcpu)) + vcpu->arch.hcr_el2 &= ~HCR_TWI; + else + vcpu->arch.hcr_el2 |= HCR_TWI; +} + +static void vcpu_load_pvtime(struct kvm_vcpu *vcpu) +{ + if (kvm_arm_is_pvtime_enabled(&vcpu->arch)) + kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu); +} + +static void vhe_vcpu_load(struct kvm_vcpu *vcpu) +{ + vcpu_prepare_mmu(vcpu); /* - * The timer must be loaded before the vgic to correctly set up physical - * interrupt deactivation in nested state (e.g. timer interrupt). + * For VHE, the timer must be loaded before the vgic to correctly + * set up physical interrupt deactivation in nested state (e.g. timer + * interrupt). */ kvm_timer_vcpu_load(vcpu); kvm_vgic_load(vcpu); kvm_vcpu_load_debug(vcpu); kvm_vcpu_load_fgt(vcpu); - if (has_vhe()) - kvm_vcpu_load_vhe(vcpu); + kvm_vcpu_load_vhe(vcpu); kvm_arch_vcpu_load_fp(vcpu); kvm_vcpu_pmu_restore_guest(vcpu); - if (kvm_arm_is_pvtime_enabled(&vcpu->arch)) - kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu); - if (kvm_vcpu_should_clear_twe(vcpu)) - vcpu->arch.hcr_el2 &= ~HCR_TWE; - else - vcpu->arch.hcr_el2 |= HCR_TWE; + vcpu_load_pvtime(vcpu); + vcpu_set_wfx_traps(vcpu); + vcpu_set_pauth_traps(vcpu); +} - if (kvm_vcpu_should_clear_twi(vcpu)) - vcpu->arch.hcr_el2 &= ~HCR_TWI; - else - vcpu->arch.hcr_el2 |= HCR_TWI; +static void nvhe_vcpu_load(struct kvm_vcpu *vcpu) +{ + vcpu_prepare_mmu(vcpu); + kvm_timer_vcpu_load(vcpu); + kvm_vgic_load(vcpu); + kvm_vcpu_load_debug(vcpu); + kvm_vcpu_load_fgt(vcpu); + kvm_arch_vcpu_load_fp(vcpu); + kvm_vcpu_pmu_restore_guest(vcpu); + vcpu_load_pvtime(vcpu); + vcpu_set_wfx_traps(vcpu); vcpu_set_pauth_traps(vcpu); +} - if (is_protected_kvm_enabled()) { - kvm_call_hyp_nvhe(__pkvm_vcpu_load, - vcpu->kvm->arch.pkvm.handle, - vcpu->vcpu_idx, vcpu->arch.hcr_el2); - kvm_call_hyp(__vgic_v3_restore_vmcr_aprs, - &vcpu->arch.vgic_cpu.vgic_v3); - } +static void pkvm_vcpu_load(struct kvm_vcpu *vcpu) +{ + kvm_timer_vcpu_load(vcpu); + kvm_vgic_load(vcpu); + kvm_vcpu_load_debug(vcpu); + kvm_vcpu_load_fgt(vcpu); + kvm_arch_vcpu_load_fp(vcpu); + kvm_vcpu_pmu_restore_guest(vcpu); + + vcpu_load_pvtime(vcpu); + vcpu_set_wfx_traps(vcpu); + + kvm_call_hyp_nvhe(__pkvm_vcpu_load, + vcpu->kvm->arch.pkvm.handle, + vcpu->vcpu_idx, vcpu->arch.hcr_el2); + kvm_call_hyp_nvhe(__vgic_v3_restore_vmcr_aprs, + &vcpu->arch.vgic_cpu.vgic_v3); +} + +void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +{ + vcpu->cpu = cpu; + vcpu->arch.vcpu_ops->vcpu_load(vcpu); if (!cpumask_test_cpu(cpu, vcpu->kvm->arch.supported_cpus)) vcpu_set_on_unsupported_cpu(vcpu); @@ -748,49 +818,76 @@ nommu: vcpu->arch.pid = pid_nr(vcpu->pid); } -void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) +static void vhe_vcpu_put(struct kvm_vcpu *vcpu) { - if (is_protected_kvm_enabled()) { - kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3); - kvm_call_hyp_nvhe(__pkvm_vcpu_put); - - /* __pkvm_vcpu_put implies a sync of the state */ - if (!kvm_vm_is_protected(vcpu->kvm)) - vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); - } - kvm_vcpu_put_debug(vcpu); kvm_arch_vcpu_put_fp(vcpu); - if (has_vhe()) - kvm_vcpu_put_vhe(vcpu); + kvm_vcpu_put_vhe(vcpu); kvm_timer_vcpu_put(vcpu); kvm_vgic_put(vcpu); kvm_vcpu_pmu_restore_host(vcpu); + if (vcpu_has_nv(vcpu)) kvm_vcpu_put_hw_mmu(vcpu); - kvm_arm_vmid_clear_active(); + vcpu_put_mmu(vcpu); +} + +static void nvhe_vcpu_put(struct kvm_vcpu *vcpu) +{ + kvm_vcpu_put_debug(vcpu); + kvm_arch_vcpu_put_fp(vcpu); + kvm_timer_vcpu_put(vcpu); + kvm_vgic_put(vcpu); + kvm_vcpu_pmu_restore_host(vcpu); + vcpu_put_mmu(vcpu); +} + +static void pkvm_vcpu_put(struct kvm_vcpu *vcpu) +{ + kvm_call_hyp_nvhe(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3); + kvm_call_hyp_nvhe(__pkvm_vcpu_put); + + /* __pkvm_vcpu_put implies a sync of the state */ + if (vcpu->kvm->arch.vm_flavor == VM_PKVM) + vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); + + kvm_vcpu_put_debug(vcpu); + kvm_arch_vcpu_put_fp(vcpu); + kvm_timer_vcpu_put(vcpu); + kvm_vgic_put(vcpu); + kvm_vcpu_pmu_restore_host(vcpu); +} + +void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) +{ + vcpu->arch.vcpu_ops->vcpu_put(vcpu); vcpu_clear_on_unsupported_cpu(vcpu); vcpu->cpu = -1; } -static void __kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu) +static void __kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu, u32 mp_state) { - WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED); + WRITE_ONCE(vcpu->arch.mp_state.mp_state, mp_state); kvm_make_request(KVM_REQ_SLEEP, vcpu); kvm_vcpu_kick(vcpu); } +/* The guest's own CPU_OFF: EL2 has already powered the vCPU off. */ void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu) { + u32 mp_state = vcpu_is_protected(vcpu) ? KVM_MP_STATE_UNINITIALIZED : KVM_MP_STATE_STOPPED; + spin_lock(&vcpu->arch.mp_state_lock); - __kvm_arm_vcpu_power_off(vcpu); + __kvm_arm_vcpu_power_off(vcpu, mp_state); spin_unlock(&vcpu->arch.mp_state_lock); } bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu) { - return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_STOPPED; + u32 mp_state = READ_ONCE(vcpu->arch.mp_state.mp_state); + + return mp_state == KVM_MP_STATE_STOPPED || mp_state == KVM_MP_STATE_UNINITIALIZED; } static void kvm_arm_vcpu_suspend(struct kvm_vcpu *vcpu) @@ -805,6 +902,12 @@ static bool kvm_arm_vcpu_suspended(struct kvm_vcpu *vcpu) return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_SUSPENDED; } +/* Only the guest's CPU_ON may start a vCPU EL2 holds powered off. */ +bool kvm_pkvm_vcpu_is_powered_off(struct kvm_vcpu *vcpu) +{ + return READ_ONCE(vcpu->arch.mp_state.mp_state) == KVM_MP_STATE_UNINITIALIZED; +} + int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu, struct kvm_mp_state *mp_state) { @@ -822,12 +925,24 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, switch (mp_state->mp_state) { case KVM_MP_STATE_RUNNABLE: + if (kvm_pkvm_vcpu_is_powered_off(vcpu)) { + ret = -EPERM; + break; + } + WRITE_ONCE(vcpu->arch.mp_state, *mp_state); break; case KVM_MP_STATE_STOPPED: - __kvm_arm_vcpu_power_off(vcpu); + /* Keep EL2's off record: STOPPED over it would let RUNNABLE through. */ + if (!kvm_pkvm_vcpu_is_powered_off(vcpu)) + __kvm_arm_vcpu_power_off(vcpu, KVM_MP_STATE_STOPPED); break; case KVM_MP_STATE_SUSPENDED: + if (kvm_pkvm_vcpu_is_powered_off(vcpu)) { + ret = -EPERM; + break; + } + kvm_arm_vcpu_suspend(vcpu); break; default: @@ -953,7 +1068,7 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) return ret; } - ret = kvm_finalize_sys_regs(vcpu); + ret = kvm_vcpu_finalize_sys_regs(vcpu); if (ret) return ret; @@ -983,13 +1098,13 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) return ret; } - ret = vgic_v5_finalize_ppi_state(kvm); + ret = vgic_v5_finalize_ppi_state(vcpu); if (ret) return ret; if (is_protected_kvm_enabled()) { /* Start with the vcpu in a dirty state */ - if (!kvm_vm_is_protected(vcpu->kvm)) + if (vcpu->kvm->arch.vm_flavor == VM_PKVM) vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); ret = pkvm_create_hyp_vm(kvm); if (ret) @@ -1163,6 +1278,14 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) preempt_enable(); } + if (kvm_check_request(KVM_REQ_RELOAD_GICv5, vcpu)) { + /* The IRS enable bit was changed */ + preempt_disable(); + vgic_v5_put(vcpu); + vgic_v5_load(vcpu); + preempt_enable(); + } + if (kvm_check_request(KVM_REQ_RELOAD_PMU, vcpu)) kvm_vcpu_reload_pmu(vcpu); @@ -1267,7 +1390,7 @@ static int noinstr kvm_arm_vcpu_enter_exit(struct kvm_vcpu *vcpu) * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code * @vcpu: The VCPU pointer * - * This function is called through the VCPU_RUN ioctl called from user space. It + * This function is called through the KVM_RUN ioctl called from user space. It * will execute VM code in a loop until the time slice for the process is used * or some emulation is needed from user space in which case the function will * return with return value 0 and with the kvm_run structure filled in with the @@ -1581,10 +1704,13 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level, return -EINVAL; } -static unsigned long system_supported_vcpu_features(void) +static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu) { unsigned long features = KVM_VCPU_VALID_FEATURES; + if (vcpu->kvm->arch.vm_flavor == VM_PROTECTED_PKVM) + kvm_pkvm_vcpu_allowed_features(vcpu->kvm, &features); + if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1)) clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features); @@ -1607,6 +1733,7 @@ static unsigned long system_supported_vcpu_features(void) return features; } +#if ARM64_S390_COMMON == 1 static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu, const struct kvm_vcpu_init *init) { @@ -1621,7 +1748,12 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu, return -ENOENT; } - if (features & ~system_supported_vcpu_features()) + if (features & ~system_supported_vcpu_features(vcpu)) + return -EINVAL; + + /* EL2 implements PSCI 1.1; the host must not dispatch as 0.1. */ + if (vcpu_is_protected(vcpu) && + !test_bit(KVM_ARM_VCPU_PSCI_0_2, &features)) return -EINVAL; /* @@ -1660,6 +1792,8 @@ static bool kvm_vcpu_init_changed(struct kvm_vcpu *vcpu, KVM_VCPU_MAX_FEATURES); } +#endif /* ARM64_S390_COMMON == 1 */ + static int kvm_setup_vcpu(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -1779,7 +1913,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, spin_lock(&vcpu->arch.mp_state_lock); if (power_off) - __kvm_arm_vcpu_power_off(vcpu); + __kvm_arm_vcpu_power_off(vcpu, KVM_MP_STATE_STOPPED); else WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE); @@ -1856,6 +1990,24 @@ static int kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu, return __kvm_arm_vcpu_set_events(vcpu, events); } +/* + * Once a protected vCPU has run, the host copy holds the boot state the VMM + * wrote plus what the exit handlers copy out, and EL2 has read mp_state, + * which it does only at hyp vCPU creation. + */ +static long pkvm_filter_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned int ioctl) +{ + switch (ioctl) { + case KVM_ARM_VCPU_INIT: + case KVM_SET_ONE_REG: + case KVM_GET_ONE_REG: + if (vcpu_is_protected(vcpu) && vcpu_has_run_once(vcpu)) + return -EPERM; + } + + return 0; +} + long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -1864,6 +2016,10 @@ long kvm_arch_vcpu_ioctl(struct file *filp, struct kvm_device_attr attr; long r; + r = pkvm_filter_vcpu_ioctl(vcpu, ioctl); + if (r) + return r; + switch (ioctl) { case KVM_ARM_VCPU_INIT: { struct kvm_vcpu_init init; @@ -2109,6 +2265,37 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) } } +static const struct kvm_vcpu_ops vhe_vcpu_ops = { + .vcpu_load = vhe_vcpu_load, + .vcpu_put = vhe_vcpu_put, +}; + +static const struct kvm_vcpu_ops nvhe_vcpu_ops = { + .vcpu_load = nvhe_vcpu_load, + .vcpu_put = nvhe_vcpu_put, +}; + +static const struct kvm_vcpu_ops pkvm_vcpu_ops = { + .vcpu_load = pkvm_vcpu_load, + .vcpu_put = pkvm_vcpu_put, +}; + +#define KVM_VCPU_OPS(flavor, ops) \ + [(flavor)] = &(ops) + +static const struct kvm_vcpu_ops *arm64_vcpu_ops[] = { + KVM_VCPU_OPS(VM_NVHE, nvhe_vcpu_ops), + KVM_VCPU_OPS(VM_VHE, vhe_vcpu_ops), + KVM_VCPU_OPS(VM_PKVM, pkvm_vcpu_ops), + KVM_VCPU_OPS(VM_PROTECTED_PKVM, pkvm_vcpu_ops), +}; + +static void kvm_init_vcpu_ops(struct kvm_vcpu *vcpu) +{ + BUILD_BUG_ON(ARRAY_SIZE(arm64_vcpu_ops) != VM_FLAVOR_MAX); + vcpu->arch.vcpu_ops = arm64_vcpu_ops[vcpu->kvm->arch.vm_flavor]; +} + static unsigned long nvhe_percpu_size(void) { return (unsigned long)CHOOSE_NVHE_SYM(__per_cpu_end) - @@ -2626,6 +2813,10 @@ static void kvm_hyp_init_symbols(void) kvm_nvhe_sym(__icache_flags) = __icache_flags; kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits; + kvm_nvhe_sym(spectre_v2_state) = arm64_get_spectre_v2_state(); + kvm_nvhe_sym(spectre_v4_state) = arm64_get_spectre_v4_state(); + kvm_nvhe_sym(spectre_bhb_state) = arm64_get_spectre_bhb_state(); + /* Propagate the FGT state to the nVHE side */ kvm_nvhe_sym(hfgrtr_masks) = hfgrtr_masks; kvm_nvhe_sym(hfgwtr_masks) = hfgwtr_masks; @@ -2918,6 +3109,8 @@ static int __init init_hyp_mode(void) kvm_err("Failed to init hyp memory protection\n"); goto out_err; } + + pkvm_selftests(); } return 0; diff --git a/arch/arm64/include/asm/kvm_define_hypevents.h b/arch/arm64/kvm/define_hypevents.h index 77d6790252a6..77d6790252a6 100644 --- a/arch/arm64/include/asm/kvm_define_hypevents.h +++ b/arch/arm64/kvm/define_hypevents.h diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index b01d6622b872..1079136d5cd7 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -62,6 +62,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { sizeof(kvm_vcpu_stats_desc), }; +#if ARM64_S390_COMMON == 1 static bool core_reg_offset_is_vreg(u64 off) { return off >= KVM_REG_ARM_CORE_REG(fp_regs.vregs) && @@ -134,19 +135,19 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) KVM_REG_ARM_CORE_REG(regs.regs[30]): off -= KVM_REG_ARM_CORE_REG(regs.regs[0]); off /= 2; - return &vcpu->arch.ctxt.regs.regs[off]; + return &vcpu_gp_regs(vcpu)[off]; case KVM_REG_ARM_CORE_REG(regs.sp): - return &vcpu->arch.ctxt.regs.sp; + return vcpu_sp_el0(vcpu); case KVM_REG_ARM_CORE_REG(regs.pc): - return &vcpu->arch.ctxt.regs.pc; + return vcpu_pc(vcpu); case KVM_REG_ARM_CORE_REG(regs.pstate): - return &vcpu->arch.ctxt.regs.pstate; + return vcpu_cpsr(vcpu); case KVM_REG_ARM_CORE_REG(sp_el1): - return __ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1); + return vcpu_sp_el1(vcpu); case KVM_REG_ARM_CORE_REG(elr_el1): return __ctxt_sys_reg(&vcpu->arch.ctxt, ELR_EL1); @@ -170,13 +171,13 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]): off -= KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]); off /= 4; - return &vcpu->arch.ctxt.fp_regs.vregs[off]; + return vcpu_vreg(vcpu, off); case KVM_REG_ARM_CORE_REG(fp_regs.fpsr): - return &vcpu->arch.ctxt.fp_regs.fpsr; + return vcpu_fpsr(vcpu); case KVM_REG_ARM_CORE_REG(fp_regs.fpcr): - return &vcpu->arch.ctxt.fp_regs.fpcr; + return vcpu_fpcr(vcpu); default: return NULL; @@ -306,6 +307,8 @@ out: return err; } +#endif /* ARM64_S390_COMMON == 1 */ + #define vq_word(vq) (((vq) - SVE_VQ_MIN) / 64) #define vq_mask(vq) ((u64)1 << ((vq) - SVE_VQ_MIN) % 64) #define vq_present(vqs, vq) (!!((vqs)[vq_word(vq)] & vq_mask(vq))) @@ -543,6 +546,7 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) return -EINVAL; } +#if ARM64_S390_COMMON == 1 static int copy_core_reg_indices(const struct kvm_vcpu *vcpu, u64 __user *uindices) { @@ -591,6 +595,8 @@ static unsigned long num_core_regs(const struct kvm_vcpu *vcpu) return copy_core_reg_indices(vcpu, NULL); } +#endif /* ARM64_S390_COMMON == 1 */ + static unsigned long num_sve_regs(const struct kvm_vcpu *vcpu) { const unsigned int slices = vcpu_sve_slices(vcpu); @@ -787,6 +793,13 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu, int ret = 0; /* + * EL2 injects an external abort only to complete a forwarded abort. + * SError injection is forwarded. + */ + if (vcpu_is_protected(vcpu) && ext_dabt_pending) + return -EPERM; + + /* * Immediately commit the pending SEA to the vCPU's architectural * state which is necessary since we do not return a pending SEA * to userspace via KVM_GET_VCPU_EVENTS. @@ -883,6 +896,10 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, { trace_kvm_set_guest_debug(vcpu, dbg->control); + /* A protected guest's debug state is not exposed to the host. */ + if (vcpu_is_protected(vcpu)) + return -EPERM; + if (dbg->control & ~KVM_GUESTDBG_VALID_MASK) return -EINVAL; diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 308100ed25de..9709905f21b5 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -46,7 +46,7 @@ alternative_else_nop_endif // Ensure that __guest_enter() always provides a context // synchronization event so that callers don't need ISBs for anything - // that would usually be synchonized by the ERET. + // that would usually be synchronized by the ERET. isb mov x0, #ARM_EXCEPTION_IRQ ret diff --git a/arch/arm64/kvm/hyp/exception.c b/arch/arm64/kvm/hyp/exception.c index 754e2dc1df54..7521bddf73e4 100644 --- a/arch/arm64/kvm/hyp/exception.c +++ b/arch/arm64/kvm/hyp/exception.c @@ -70,6 +70,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, enum exception_type type) { unsigned long sctlr, vbar, old, new, mode; + struct kvm *kvm; u64 exc_offset; mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); @@ -109,8 +110,10 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= (old & PSR_C_BIT); new |= (old & PSR_V_BIT); - if (kvm_has_mte(kern_hyp_va(vcpu->kvm))) + kvm = vcpu_get_kvm(vcpu); + if (kvm && kvm_has_mte(kvm)) new |= PSR_TCO_BIT; + vcpu_put_kvm(vcpu, kvm); new |= (old & PSR_DIT_BIT); @@ -276,12 +279,12 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset) switch(mode) { case PSR_AA32_MODE_ABT: __vcpu_write_spsr_abt(vcpu, host_spsr_to_spsr32(spsr)); - vcpu_gp_regs(vcpu)->compat_lr_abt = return_address; + vcpu_gp_regs(vcpu)[__compat_lr_abt] = return_address; break; case PSR_AA32_MODE_UND: __vcpu_write_spsr_und(vcpu, host_spsr_to_spsr32(spsr)); - vcpu_gp_regs(vcpu)->compat_lr_und = return_address; + vcpu_gp_regs(vcpu)[__compat_lr_und] = return_address; break; } @@ -350,12 +353,19 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) */ void __kvm_adjust_pc(struct kvm_vcpu *vcpu) { - if (vcpu_get_flag(vcpu, PENDING_EXCEPTION)) { - kvm_inject_exception(vcpu); - vcpu_clear_flag(vcpu, PENDING_EXCEPTION); - vcpu_clear_flag(vcpu, EXCEPT_MASK); - } else if (vcpu_get_flag(vcpu, INCREMENT_PC)) { - kvm_skip_instr(vcpu); - vcpu_clear_flag(vcpu, INCREMENT_PC); + struct kvm_vcpu *target = kvm_adjust_pc_get(vcpu); + + if (!target) + return; + + if (vcpu_get_flag(target, PENDING_EXCEPTION)) { + kvm_inject_exception(target); + vcpu_clear_flag(target, PENDING_EXCEPTION); + vcpu_clear_flag(target, EXCEPT_MASK); + } else if (vcpu_get_flag(target, INCREMENT_PC)) { + kvm_skip_instr(target); + vcpu_clear_flag(target, INCREMENT_PC); } + + kvm_adjust_pc_put(vcpu, target); } diff --git a/arch/arm64/kvm/hyp/hyp-constants.c b/arch/arm64/kvm/hyp/hyp-constants.c index b257a3b4bfc5..b2caae21f271 100644 --- a/arch/arm64/kvm/hyp/hyp-constants.c +++ b/arch/arm64/kvm/hyp/hyp-constants.c @@ -7,7 +7,5 @@ int main(void) { DEFINE(STRUCT_HYP_PAGE_SIZE, sizeof(struct hyp_page)); - DEFINE(PKVM_HYP_VM_SIZE, sizeof(struct pkvm_hyp_vm)); - DEFINE(PKVM_HYP_VCPU_SIZE, sizeof(struct pkvm_hyp_vcpu)); return 0; } diff --git a/arch/arm64/kvm/hyp/include/hyp/adjust_pc.h b/arch/arm64/kvm/hyp/include/hyp/adjust_pc.h index 4fdfeabefeb4..d2b9a98bf703 100644 --- a/arch/arm64/kvm/hyp/include/hyp/adjust_pc.h +++ b/arch/arm64/kvm/hyp/include/hyp/adjust_pc.h @@ -13,6 +13,34 @@ #include <asm/kvm_emulate.h> #include <asm/kvm_host.h> +#ifdef __KVM_NVHE_HYPERVISOR__ +#include <nvhe/pkvm.h> + +/* Under pKVM a host vCPU's ->kvm is host-writable. */ +static inline struct kvm *vcpu_get_kvm(struct kvm_vcpu *vcpu) +{ + if (is_protected_kvm_enabled()) + return pkvm_vcpu_get_kvm(vcpu); + + return kern_hyp_va(vcpu->kvm); +} + +static inline void vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm) +{ + if (is_protected_kvm_enabled()) + pkvm_vcpu_put_kvm(vcpu, kvm); +} +#else +static inline struct kvm *vcpu_get_kvm(struct kvm_vcpu *vcpu) +{ + return vcpu->kvm; +} + +static inline void vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm) +{ +} +#endif + static inline void kvm_skip_instr(struct kvm_vcpu *vcpu) { if (vcpu_mode_is_32bit(vcpu)) { @@ -33,11 +61,11 @@ static inline void kvm_skip_instr(struct kvm_vcpu *vcpu) static inline void __kvm_skip_instr(struct kvm_vcpu *vcpu) { *vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR); - vcpu_gp_regs(vcpu)->pstate = read_sysreg_el2(SYS_SPSR); + *vcpu_cpsr(vcpu) = read_sysreg_el2(SYS_SPSR); kvm_skip_instr(vcpu); - write_sysreg_el2(vcpu_gp_regs(vcpu)->pstate, SYS_SPSR); + write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR); write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR); } @@ -50,4 +78,24 @@ static inline void kvm_skip_host_instr(void) write_sysreg_el2(read_sysreg_el2(SYS_ELR) + 4, SYS_ELR); } +/* + * Under pKVM, the vCPU __kvm_adjust_pc() adjusts for @vcpu (NULL leaves the + * request on @vcpu for its next entry), and the copy of the consumed + * PC_UPDATE_REQ back to @vcpu. + */ +#ifdef __KVM_NVHE_HYPERVISOR__ +struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu); +void kvm_adjust_pc_put(struct kvm_vcpu *vcpu, struct kvm_vcpu *target); +#else +static inline struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu) +{ + return vcpu; +} + +static inline void kvm_adjust_pc_put(struct kvm_vcpu *vcpu, + struct kvm_vcpu *target) +{ +} +#endif + #endif diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 1ce7130e2549..a2a054fec2e2 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -470,7 +470,7 @@ static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code) u64 spsr; *vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR); - arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2); + arm64_mops_reset_regs(&vcpu->arch.ctxt.regs, vcpu->arch.fault.esr_el2); write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR); /* @@ -902,7 +902,7 @@ static inline void synchronize_vcpu_pstate(struct kvm_vcpu *vcpu) /* * Check for the conditions of Cortex-A510's #2077057. When these occur * SPSR_EL2 can't be trusted, but isn't needed either as it is - * unchanged from the value in vcpu_gp_regs(vcpu)->pstate. + * unchanged from the value in vcpu_cpsr(vcpu). * Are we single-stepping the guest, and took a PAC exception from the * active-not-pending state? */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/alloc.h b/arch/arm64/kvm/hyp/include/nvhe/alloc.h new file mode 100644 index 000000000000..bca0168e0046 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/alloc.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_NVHE_ALLOC__ +#define __KVM_NVHE_ALLOC__ +#include <linux/types.h> + +#include <asm/kvm_host.h> + +void *hyp_alloc(size_t size); +int hyp_alloc_errno(void); +u32 hyp_alloc_topup_needed(void); +void hyp_free(void *addr); + +int hyp_alloc_init(size_t size); +int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc); +unsigned long hyp_alloc_reclaimable(void); +void hyp_alloc_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target); + +#ifdef CONFIG_NVHE_EL2_DEBUG +int hyp_allocator_selftest(void); +u32 hyp_alloc_selftest_topup_needed(void); +int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc); +void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target); +#else +static inline int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc) { return 0; }; +static inline void +hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target) { }; +#endif +#endif diff --git a/arch/arm64/kvm/hyp/include/nvhe/define_events.h b/arch/arm64/kvm/hyp/include/nvhe/define_events.h index 776d4c6cb702..370e8c2d39fe 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/define_events.h +++ b/arch/arm64/kvm/hyp/include/nvhe/define_events.h @@ -10,5 +10,3 @@ #define HYP_EVENT_MULTI_READ #include <asm/kvm_hypevents.h> #undef HYP_EVENT_MULTI_READ - -#undef HYP_EVENT diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index 31cae95ddb71..72b7927a1fa5 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -25,11 +25,14 @@ int hyp_back_vmemmap(phys_addr_t back); int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot); +void pkvm_remove_mappings(void *from, void *to); int __pkvm_create_private_mapping(phys_addr_t phys, size_t size, enum kvm_pgtable_prot prot, unsigned long *haddr); int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr); int pkvm_check_host_ownership(void); int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr); +int pkvm_map_private_va_range(void *haddr, phys_addr_t phys, size_t size); +phys_addr_t pkvm_private_va_range_pa(void *va); #endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h index 5c050f21066a..4373e9b63e87 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h @@ -7,6 +7,7 @@ #ifndef __ARM64_KVM_NVHE_PKVM_H__ #define __ARM64_KVM_NVHE_PKVM_H__ +#include <asm/kvm_hcall.h> #include <asm/kvm_pkvm.h> #include <nvhe/gfp.h> @@ -35,6 +36,15 @@ struct pkvm_hyp_vcpu { * per-cpu pointer tracking us. Otherwise, NULL if not loaded. */ struct pkvm_hyp_vcpu **loaded_hyp_vcpu; + + /* The previous exit's ARM_EXCEPTION_* code. */ + u32 exit_code; + + /* + * PSCI_0_2_AFFINITY_LEVEL_{OFF, ON_PENDING, ON}. A non-protected + * vCPU is always ON. + */ + int power_state; }; /* @@ -50,8 +60,15 @@ struct pkvm_hyp_vm { struct kvm_pgtable pgt; struct kvm_pgtable_mm_ops mm_ops; struct hyp_pool pool; + unsigned short refcount; hyp_spinlock_t lock; + /* + * The vCPU initialised RUNNABLE: claimed under vm_table_lock, + * released only if its own init fails. + */ + struct pkvm_hyp_vcpu *primary_vcpu; + /* Array of the hyp vCPU structures for this VM. */ struct pkvm_hyp_vcpu *vcpus[]; }; @@ -74,14 +91,24 @@ static inline bool pkvm_hyp_vm_is_protected(struct pkvm_hyp_vm *hyp_vm) return kvm_vm_is_protected(&hyp_vm->kvm); } +static inline void pkvm_hyp_vm_ref_inc(struct pkvm_hyp_vm *hyp_vm) +{ + BUG_ON(hyp_vm->refcount == USHRT_MAX); + hyp_vm->refcount++; +} + +static inline void pkvm_hyp_vm_ref_dec(struct pkvm_hyp_vm *hyp_vm) +{ + BUG_ON(!hyp_vm->refcount); + hyp_vm->refcount--; +} + void pkvm_hyp_vm_table_init(void *tbl); int __pkvm_reserve_vm(void); void __pkvm_unreserve_vm(pkvm_handle_t handle); -int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva, - unsigned long pgd_hva); -int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu, - unsigned long vcpu_hva); +int __pkvm_init_vm(struct kvm *host_kvm, void __kern *pgd_hva); +int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu); int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn); int __pkvm_start_teardown_vm(pkvm_handle_t handle); @@ -92,6 +119,8 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle, unsigned int vcpu_idx); void pkvm_put_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu); struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu(void); +struct kvm *pkvm_vcpu_get_kvm(struct kvm_vcpu *vcpu); +void pkvm_vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm); struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle); struct pkvm_hyp_vm *get_np_pkvm_hyp_vm(pkvm_handle_t handle); @@ -101,6 +130,9 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code); bool kvm_handle_pvm_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code); bool kvm_handle_pvm_restricted(struct kvm_vcpu *vcpu, u64 *exit_code); void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu); +void kvm_reset_pvm_sys_regs(struct kvm_vcpu *vcpu); int kvm_check_pvm_sysreg_table(void); +int pkvm_reset_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu); +struct pkvm_hyp_vcpu *pkvm_mpidr_to_hyp_vcpu(struct pkvm_hyp_vm *vm, unsigned long mpidr); #endif /* __ARM64_KVM_NVHE_PKVM_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h index 7c7ea8c55405..260e14d6d200 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -17,6 +17,8 @@ #include <asm/lse.h> #include <asm/rwonce.h> +#include <linux/cleanup.h> + typedef union hyp_spinlock { u32 __val; struct { @@ -122,4 +124,6 @@ static inline void hyp_assert_lock_held(hyp_spinlock_t *lock) static inline void hyp_assert_lock_held(hyp_spinlock_t *lock) { } #endif +DEFINE_LOCK_GUARD_1(hyp_spinlock, hyp_spinlock_t, hyp_spin_lock(_T->lock), + hyp_spin_unlock(_T->lock)) #endif /* __ARM64_KVM_NVHE_SPINLOCK_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/trace.h b/arch/arm64/kvm/hyp/include/nvhe/trace.h index 8813ff250f8e..6e3225d8d7d7 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/trace.h +++ b/arch/arm64/kvm/hyp/include/nvhe/trace.h @@ -4,6 +4,7 @@ #include <linux/trace_remote_event.h> +#include <asm/kvm_hcall.h> #include <asm/kvm_hyptrace.h> static inline pid_t __tracing_get_vcpu_pid(struct kvm_cpu_context *host_ctxt) @@ -46,7 +47,71 @@ static inline pid_t __tracing_get_vcpu_pid(struct kvm_cpu_context *host_ctxt) void *tracing_reserve_entry(unsigned long length); void tracing_commit_entry(void); -int __tracing_load(unsigned long desc_va, size_t desc_size); +/* + * The trace_hyp_printk boilerplate is too fiddly to be declared with + * HYP_EVENT(): + * + * The string format is stored into a kernel-accessible ELF section. The + * hypervisor only writes the format ID. + * + * The function has a variadic prototype. We have no easy way to know each + * argument width so they must all cast to u64. + */ +#define REMOTE_EVENT_CUSTOM_PRINTK(...) + +#define __TO_U64_0() +#define __TO_U64_1(x) , (u64)(x) +#define __TO_U64_2(x, ...) , (u64)(x) __TO_U64_1(__VA_ARGS__) +#define __TO_U64_3(x, ...) , (u64)(x) __TO_U64_2(__VA_ARGS__) +#define __TO_U64_4(x, ...) , (u64)(x) __TO_U64_3(__VA_ARGS__) +#define __TO_U64_5(x, ...) , (u64)(x) __TO_U64_4(__VA_ARGS__) +#define __TO_U64_6(x, ...) , (u64)(x) __TO_U64_5(__VA_ARGS__) +#define __TO_U64_7(x, ...) , (u64)(x) __TO_U64_6(__VA_ARGS__) +#define __TO_U64_8(x, ...) , (u64)(x) __TO_U64_7(__VA_ARGS__) + +#define __TO_U64_X(N, ...) CONCATENATE(__TO_U64_, N)(__VA_ARGS__) +#define __TO_U64(...) __TO_U64_X(COUNT_ARGS(__VA_ARGS__), ##__VA_ARGS__) + +REMOTE_EVENT_FORMAT(hyp_printk, HE_STRUCT(he_field(u16, fmt_id) he_field(u64, args[]))); +extern struct hyp_event_id hyp_event_id_hyp_printk; + +static __always_inline void __trace_hyp_printk(const struct hyp_string_fmt *fmt, int nr_args, ...) +{ + struct remote_event_format_hyp_printk *entry; + va_list va; + int i; + + if (!atomic_read(&hyp_event_id_hyp_printk.enabled)) + return; + + entry = tracing_reserve_entry(struct_size(entry, args, nr_args)); + if (!entry) + return; + + entry->hdr.id = hyp_event_id_hyp_printk.id; + entry->fmt_id = fmt - __hyp_string_fmts_start; + + va_start(va, nr_args); + for (i = 0; i < nr_args; i++) + entry->args[i] = va_arg(va, u64); + va_end(va); + + tracing_commit_entry(); +} + +#define trace_hyp_printk(__fmt, __args...) \ +do { \ + static const struct hyp_string_fmt __used __section("_hyp_string_fmts") \ + __hyp_str_fmt = { \ + .fmt = __fmt, \ + .nr_args = COUNT_ARGS(__args), \ + }; \ + BUILD_BUG_ON(sizeof(__fmt) > HYP_STRING_FMT_MAX_SIZE); \ + /* __TO_U64 prepends a comma if there are arguments */ \ + __trace_hyp_printk(&__hyp_str_fmt, COUNT_ARGS(__args) __TO_U64(__args)); \ +} while (0) + +int __tracing_load(void __kern *desc_va, size_t desc_size); void __tracing_unload(void); int __tracing_enable(bool enable); int __tracing_swap_reader(unsigned int cpu); @@ -58,8 +123,10 @@ static inline void *tracing_reserve_entry(unsigned long length) { return NULL; } static inline void tracing_commit_entry(void) { } #define HYP_EVENT(__name, __proto, __struct, __assign, __printk) \ static inline void trace_##__name(__proto) {} +#define REMOTE_EVENT_CUSTOM_PRINTK(...) +#define trace_hyp_printk(fmt, args...) do { } while (0) -static inline int __tracing_load(unsigned long desc_va, size_t desc_size) { return -ENODEV; } +static inline int __tracing_load(void __kern *desc_va, size_t desc_size) { return -ENODEV; } static inline void __tracing_unload(void) { } static inline int __tracing_enable(bool enable) { return -ENODEV; } static inline int __tracing_swap_reader(unsigned int cpu) { return -ENODEV; } diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index f57450ebcb49..ee0abb2e6af5 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -23,7 +23,7 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) CFLAGS_switch.nvhe.o += -Wno-override-init hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ + hyp-main.o hyp-smp.o psci-relay.o alloc.o early_alloc.o page_alloc.o \ cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o @@ -49,6 +49,7 @@ targets += $(hyp-obj) kvm_nvhe.tmp.o kvm_nvhe.rel.o hyp.lds hyp-reloc.S hyp-relo # avoids file name clashes for files shared with VHE. $(obj)/%.nvhe.o: $(src)/%.c FORCE $(call if_changed_rule,cc_o_c) + $(call cmd,force_checksrc) $(obj)/%.nvhe.o: $(src)/%.S FORCE $(call if_changed_rule,as_o_S) diff --git a/arch/arm64/kvm/hyp/nvhe/alloc.c b/arch/arm64/kvm/hyp/nvhe/alloc.c new file mode 100644 index 000000000000..e1e6dd21cda1 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/alloc.c @@ -0,0 +1,1212 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026 Google LLC + * Author: Vincent Donnefort <vdonnefort@google.com> + * + * This heap allocator manages a reserved VA space range, dynamically mapping + * and unmapping physical pages on-demand to minimise the pKVM hypervisor + * footprint. As memory is reclaimed and relinquished to the host, unmapped + * holes are introduced within the VA space. To prevent orphans mapped regions, + * neighboring unused chunks cannot be merged if they are separated by an + * unmapped region. + * + */ + +#include <nvhe/alloc.h> +#include <nvhe/mem_protect.h> +#include <nvhe/mm.h> +#include <nvhe/spinlock.h> + +#include <linux/build_bug.h> +#include <linux/hash.h> + +/* + * A minimum of 16 bytes allows us to comply with the alignment requirement for + * FPSIMD registers that are part of struct pkvm_hyp_vcpu. + * + * It also must be a power of two. + */ +#define MIN_ALLOC_SIZE 16UL + +/* + * struct chunk_hdr - Chunk header + * @next: offset from this chunk header to the next one. + * @prev: offset from this chunk header to the previous one. + * @__unmapped: Internal field containing the offset to the unmapped page + * boundary, multiplexed with the allocation state flag. + * @hash: Hash computed over the chunk header. + * @data: Chunk payload. + */ +struct chunk_hdr { + u32 next; + u32 prev; +#define USED_BIT_MASK 1U + u32 __unmapped; + u32 hash; + char data[]; +} __aligned(MIN_ALLOC_SIZE); + +/* + * struct hyp_allocator - Heap allocator + * @start: Start in the allocator's reserved virtual address range. + * @end: End in the allocator's reserved virtual address range. + * @last_used: Pointer to the end of the last used chunk. This is + * necessary for the last chunk in the list as the + * allocated size of a chunk is derived from the next one. + * @first_unmapped: Pointer to the first unmapped page in the + * allocator's range. This is only necessary and + * updated when no chunk is in the list. + * @head: Head of the chunk list. + * @tail: Tail of the chunk list. + * @mc: Memcache containing pre-allocated pages for mapping. + * @lock: Spinlock protecting the allocator state. + * @errno: Per-CPU error code for allocation failures. + * @topup_needed: Per-CPU page counter needed to top-up the memcache. + */ +struct hyp_allocator { + void *start; + void *end; + void *last_used; + void *first_unmapped; + struct chunk_hdr *head; + struct chunk_hdr *tail; + struct kvm_hyp_memcache mc; + hyp_spinlock_t lock; + int __percpu *errno; + u32 __percpu *topup_needed; +}; + +static u32 chunk_hash_compute(const struct chunk_hdr *chunk) +{ + u32 hash = 0; + + BUILD_BUG_ON(sizeof(*chunk) != 16); + + hash ^= hash_64(*(const u64 *)chunk, 32); + hash ^= hash_32(chunk->__unmapped, 32); + return hash; +} + +static void chunk_set_hash(struct chunk_hdr *chunk) +{ + if (chunk) + chunk->hash = chunk_hash_compute(chunk); +} + +static void chunk_check_hash(const struct chunk_hdr *chunk) +{ + if (chunk) + WARN_ON(chunk->hash != chunk_hash_compute(chunk)); +} + +static bool chunk_is_used(const struct chunk_hdr *chunk) +{ + return !!(chunk->__unmapped & USED_BIT_MASK); +} + +static void chunk_set_used(struct chunk_hdr *chunk) +{ + chunk->__unmapped |= USED_BIT_MASK; +} + +static void chunk_set_unused(struct chunk_hdr *chunk) +{ + chunk->__unmapped &= ~USED_BIT_MASK; +} + +static void *chunk_unmapped(const struct chunk_hdr *chunk) +{ + u32 offset = chunk->__unmapped & ~USED_BIT_MASK; + + if (!offset) + return NULL; + + return (void *)chunk + offset; +} + +static void __chunk_set_unmapped(struct chunk_hdr *chunk, u32 unmapped) +{ + chunk->__unmapped = unmapped | (chunk_is_used(chunk) ? USED_BIT_MASK : 0); +} + +static void chunk_set_unmapped(struct chunk_hdr *chunk, void *unmapped) +{ + WARN_ON(!PAGE_ALIGNED(unmapped)); + + if (unmapped) { + WARN_ON((void *)chunk > unmapped); + __chunk_set_unmapped(chunk, unmapped - (void *)chunk); + } else { + __chunk_set_unmapped(chunk, 0); + } +} + +static void *chunk_data(const struct chunk_hdr *chunk) +{ + return (void *)&chunk->data; +} + +static struct chunk_hdr *__chunk_next(const struct chunk_hdr *chunk) +{ + if (!chunk->next) + return NULL; + + return (struct chunk_hdr *)((void *)chunk + chunk->next); +} + +static struct chunk_hdr *__chunk_prev(const struct chunk_hdr *chunk) +{ + if (!chunk->prev) + return NULL; + + return (struct chunk_hdr *)((void *)chunk - chunk->prev); +} + +static void chunk_set_next(struct chunk_hdr *chunk, struct chunk_hdr *next) +{ + if (!chunk) + return; + + if (next) { + WARN_ON(chunk > next); + chunk->next = (void *)next - (void *)chunk; + } else { + chunk->next = 0; + } +} + +static void chunk_set_prev(struct chunk_hdr *chunk, struct chunk_hdr *prev) +{ + if (!chunk) + return; + + if (prev) { + WARN_ON(chunk < prev); + chunk->prev = (void *)chunk - (void *)prev; + } else { + chunk->prev = 0; + } +} + +static struct chunk_hdr *chunk_get_next(const struct chunk_hdr *chunk) +{ + struct chunk_hdr *next = __chunk_next(chunk); + + chunk_check_hash(next); + return next; +} + +static struct chunk_hdr *chunk_get_prev(const struct chunk_hdr *chunk) +{ + struct chunk_hdr *prev = __chunk_prev(chunk); + + chunk_check_hash(prev); + return prev; +} + +static struct chunk_hdr *chunk_get(struct chunk_hdr *chunk) +{ + chunk_check_hash(chunk); + return chunk; +} + +#define chunk_hdr_size() \ + offsetof(struct chunk_hdr, data) + +#define chunk_min_size() \ + (chunk_hdr_size() + MIN_ALLOC_SIZE) + +static size_t chunk_data_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator) +{ + struct chunk_hdr *next = chunk_get_next(chunk); + void *end; + + if (next) + end = (void *)next; + else + end = allocator->end; + + return end - chunk_data(chunk); +} + +static size_t chunk_mapped_data_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator) +{ + void *unmapped = chunk_unmapped(chunk); + + if (!unmapped) + return chunk_data_size(chunk, allocator); + + return unmapped - chunk_data(chunk); +} + +static size_t chunk_used_size(const struct chunk_hdr *chunk, struct hyp_allocator *allocator) +{ + struct chunk_hdr *next = chunk_get_next(chunk); + + if (!chunk_is_used(chunk)) + return 0; + + if (next) + return chunk_mapped_data_size(chunk, allocator); + + return allocator->last_used - chunk_data(chunk); +} + +static void chunk_list_insert(struct chunk_hdr *chunk, struct chunk_hdr *prev) +{ + struct chunk_hdr *next = NULL; + + WARN_ON(!chunk); + + if (prev) { + next = chunk_get_next(prev); + chunk_set_next(prev, chunk); + chunk_set_hash(prev); + } + + if (next) { + chunk_set_prev(next, chunk); + chunk_set_hash(next); + } + + chunk_set_next(chunk, next); + chunk_set_prev(chunk, prev); +} + +static void chunk_list_del(struct chunk_hdr *chunk) +{ + struct chunk_hdr *prev, *next; + + WARN_ON(!chunk); + + prev = chunk_get_prev(chunk); + next = chunk_get_next(chunk); + + if (prev) { + chunk_set_next(prev, next); + chunk_set_hash(prev); + } + + if (next) { + chunk_set_prev(next, prev); + chunk_set_hash(next); + } +} + +/* + * Return a fixup start address for chunk creation. It makes sure the chunk + * header doesn't cross any page boundary and that it leaves enough space at the + * start of page. This is intended to prevent orphan mapped regions during chunk + * memory reclaim + */ +static void *chunk_start(void *start) +{ + void *page = PTR_ALIGN(start, PAGE_SIZE); + + if (page - start < chunk_hdr_size()) + return page; + + page = PTR_ALIGN_DOWN(start, PAGE_SIZE); + if (start - page < chunk_min_size()) + return page + chunk_min_size(); + + return start; +} + +static void hyp_allocator_set_errno(struct hyp_allocator *allocator, int errno) +{ + *this_cpu_ptr(allocator->errno) = errno; +} + +static int hyp_allocator_errno(struct hyp_allocator *allocator) +{ + int *errno = this_cpu_ptr(allocator->errno); + int ret = *errno; + + *errno = 0; + + return ret; +} + +static int hyp_allocator_map(struct hyp_allocator *allocator, struct chunk_hdr *chunk, + struct chunk_hdr *next, + void *addr, void *end) +{ + void *unmapped = chunk ? chunk_unmapped(chunk) : allocator->first_unmapped; + void *unmapped_end = next ?: allocator->end; + + /* + * hyp_allocator_can_create_chunk() already validates addr/end + * belong to the chunk. + */ + WARN_ON(end <= addr); + + /* The chunk does not span an unmapped region */ + if (!unmapped) + return 0; + + while (unmapped < end) { + void *page = pop_hyp_memcache(&allocator->mc, hyp_phys_to_virt); + int ret; + + if (!page) { + end = PTR_ALIGN(end, PAGE_SIZE); + *this_cpu_ptr(allocator->topup_needed) = + (unsigned long)(end - unmapped) >> PAGE_SHIFT; + return -ENOMEM; + } + + ret = pkvm_map_private_va_range(unmapped, hyp_virt_to_phys(page), PAGE_SIZE); + if (ret) { + push_hyp_memcache(&allocator->mc, page, hyp_virt_to_phys); + return ret; + } + + unmapped += PAGE_SIZE; + + /* + * Reset the unmap field if we've reached the next chunk or the + * allocator boundary. Chunks are always installed to cover a + * mapped region followed by an unmapped one. Therefore, we + * expect next to be PAGE_ALIGNED. + */ + WARN_ON(!PAGE_ALIGNED(unmapped_end)); + if (unmapped == unmapped_end) + unmapped = NULL; + + if (chunk) { + chunk_set_unmapped(chunk, unmapped); + chunk_set_hash(chunk); + } else { + allocator->first_unmapped = unmapped; + } + + if (!unmapped) + break; + } + + return 0; +} + +static void hyp_allocator_unmap(struct hyp_allocator *allocator, struct chunk_hdr *chunk, + void *addr, void *end) +{ + void *unmap = addr; + + /* + * hyp_allocator_chunk_reclaimable() already computes valid addr/end, no + * need to check them again + */ + WARN_ON(end <= addr); + + while (unmap < end) { + phys_addr_t pa = pkvm_private_va_range_pa((void *)unmap); + void *page = hyp_phys_to_virt(pa); + + push_hyp_memcache(&allocator->mc, page, hyp_virt_to_phys); + unmap += PAGE_SIZE; + } + + pkvm_remove_mappings(addr, end); + + if (chunk) { + chunk_set_unmapped(chunk, addr); + chunk_set_hash(chunk); + } else { + allocator->first_unmapped = addr; + } +} + +static bool hyp_allocator_can_create_chunk(struct hyp_allocator *allocator, + const struct chunk_hdr *prev, + const struct chunk_hdr *next, + void *addr, void *end) +{ + void *page, *unmapped; + + if (addr < allocator->start || end > allocator->end) + return false; + + /* First chunk created must be installed at allocator->start */ + if (!prev) + return addr == allocator->start; + + /* Must not overwrite the next chunk */ + if (next && end > (void *)next) + return false; + + /* Must not overwrite the previous chunk */ + if (addr < (chunk_data(prev) + chunk_used_size(prev, allocator))) + return false; + + /* Header must not cross page boundaries */ + page = PTR_ALIGN(addr, PAGE_SIZE); + if (page != addr && (page - addr) < chunk_hdr_size()) + return false; + + /* Must leave a minimum distance from a page-start to maximise reclaim */ + page = PTR_ALIGN_DOWN(addr, PAGE_SIZE); + if (page != addr && (addr - page) < chunk_min_size()) + return false; + + unmapped = chunk_unmapped(prev); + if (!unmapped) + return true; + + /* Must never create an orphan mapped region */ + if (addr > unmapped) + return false; + + return true; +} + +/* + * Tries to create a new chunk in the allocator whose header starts at @addr and + * whose data finishes at @end. + */ +static struct chunk_hdr *hyp_allocator_create_chunk(struct hyp_allocator *allocator, + struct chunk_hdr *prev, void *addr, + void *end, bool used) +{ + struct chunk_hdr *next, *chunk = addr; + void *unmapped; + int ret; + + if (end > allocator->end) + return ERR_PTR(-E2BIG); + + next = prev ? chunk_get_next(prev) : NULL; + if (!hyp_allocator_can_create_chunk(allocator, prev, next, addr, end)) + return ERR_PTR(-EINVAL); + + ret = hyp_allocator_map(allocator, prev, next, addr, end); + if (ret) + return ERR_PTR(ret); + + memset(chunk, 0, sizeof(*chunk)); + if (used) + chunk_set_used(chunk); + else + chunk_set_unused(chunk); + + /* First chunk, first allocation */ + if (!prev) { + chunk_set_unmapped(chunk, allocator->first_unmapped); + chunk_list_insert(chunk, NULL); + chunk_set_hash(chunk); + + allocator->last_used = end; + allocator->head = allocator->tail = chunk; + return chunk; + } + + /* Last chunk in the list */ + if (!next) { + allocator->last_used = end; + allocator->tail = chunk; + } + + /* Inherit prev's unmapped region */ + unmapped = chunk_unmapped(prev); + chunk_set_unmapped(chunk, unmapped); + chunk_list_insert(chunk, prev); + chunk_set_hash(chunk); + + chunk_set_unmapped(prev, NULL); + chunk_set_hash(prev); + + return chunk; +} + +static bool hyp_allocator_can_destroy_chunk(const struct chunk_hdr *prev, + const struct chunk_hdr *next, + const struct chunk_hdr *chunk) +{ + if (chunk_is_used(chunk)) + return false; + + /* Can't merge down unless we are the last one in the list */ + if (next && chunk_is_used(prev)) + return false; + + /* Must never create an orphan mapped region */ + if (chunk_unmapped(prev)) + return false; + + return true; +} + +static int hyp_allocator_destroy_chunk(struct hyp_allocator *allocator, + struct chunk_hdr *prev, + struct chunk_hdr *chunk) +{ + struct chunk_hdr *next = chunk_get_next(chunk); + + if (!hyp_allocator_can_destroy_chunk(prev, next, chunk)) + return -EINVAL; + + /* Last chunk in the list */ + if (!next) { + allocator->last_used = chunk; + allocator->tail = prev; + } + + chunk_set_unmapped(prev, chunk_unmapped(chunk)); + chunk_set_hash(prev); + chunk_list_del(chunk); + + return 0; +} + +/* + * Return the best unused chunk for recycling, that is the smallest chunk + * fitting the allocation which needs to use the least unmapped region. + */ +static struct chunk_hdr *hyp_allocator_find_efficient_chunk(struct hyp_allocator *allocator, + size_t size) +{ + struct chunk_hdr *chunk, *best_chunk = NULL; + size_t best_data_size = SIZE_MAX; + size_t best_missing = SIZE_MAX; + + chunk = allocator->head; + while (chunk) { + size_t missing, mapped, data_size; + + if (chunk_is_used(chunk)) + goto next; + + data_size = chunk_data_size(chunk, allocator); + if (data_size < size) + goto next; + + mapped = chunk_mapped_data_size(chunk, allocator); + missing = (size > mapped) ? DIV_ROUND_UP(size - mapped, PAGE_SIZE) : 0; + if (missing > best_missing) + goto next; + + if (missing == best_missing && data_size >= best_data_size) + goto next; + + best_missing = missing; + best_data_size = data_size; + best_chunk = chunk; + +next: + chunk = chunk_get_next(chunk); + } + + return best_chunk; +} + +static struct chunk_hdr *hyp_allocator_reuse_chunk(struct hyp_allocator *allocator, + struct chunk_hdr *chunk, size_t size) +{ + struct chunk_hdr *next = chunk_get_next(chunk); + void *start, *end, *split, *split_end; + int ret; + + start = chunk_data(chunk); + end = start + size; + + /* Last chunk in the list, no need to split */ + if (!next) { + split = split_end = NULL; + allocator->last_used = chunk_data(chunk) + size; + } else { + split = chunk_start(end); + split_end = split + chunk_min_size(); + + if (!hyp_allocator_can_create_chunk(allocator, chunk, next, split, split_end)) + split = split_end = NULL; + } + + /* Batch the mapping of the reused chunk and the split */ + ret = hyp_allocator_map(allocator, chunk, next, chunk_data(chunk), split ? split_end : end); + if (ret) + return ERR_PTR(ret); + + if (split) + WARN_ON(IS_ERR_OR_NULL( + hyp_allocator_create_chunk(allocator, chunk, split, split_end, false))); + + chunk_set_used(chunk); + chunk_set_hash(chunk); + + return chunk; +} + +static struct chunk_hdr *hyp_allocator_alloc_chunk(struct hyp_allocator *allocator, size_t size) +{ + struct chunk_hdr *chunk; + void *start, *end; + + /* First allocation */ + if (!allocator->head) { + start = allocator->start; + end = start + chunk_hdr_size() + size; + return hyp_allocator_create_chunk(allocator, NULL, start, end, true); + } + + chunk = hyp_allocator_find_efficient_chunk(allocator, size); + + /* Nothing found, create a new chunk at the end in the list */ + if (!chunk) { + start = chunk_start(chunk_data(allocator->tail) + + chunk_used_size(allocator->tail, allocator)); + end = start + chunk_hdr_size() + size; + return hyp_allocator_create_chunk(allocator, allocator->tail, start, end, true); + } + + return hyp_allocator_reuse_chunk(allocator, chunk, size); +} + +static void *hyp_allocator_alloc(struct hyp_allocator *allocator, size_t size) +{ + struct chunk_hdr *chunk; + + size = max(size, MIN_ALLOC_SIZE); + + /* Ensure we do not overflow ALIGN(MIN_ALLOC_SIZE) */ + if (size > U32_MAX) { + hyp_allocator_set_errno(allocator, -E2BIG); + return NULL; + } + + size = ALIGN(size, MIN_ALLOC_SIZE); + if (size > (allocator->end - allocator->start - chunk_hdr_size())) { + hyp_allocator_set_errno(allocator, -E2BIG); + return NULL; + } + + scoped_guard(hyp_spinlock, &allocator->lock) { + chunk = hyp_allocator_alloc_chunk(allocator, size); + if (IS_ERR_OR_NULL(chunk)) { + hyp_allocator_set_errno(allocator, + IS_ERR(chunk) ? PTR_ERR(chunk) : -EINVAL); + return NULL; + } + } + + memset(chunk_data(chunk), 0, size); + return chunk_data(chunk); +} + +static void hyp_allocator_free(struct hyp_allocator *allocator, void *data) +{ + struct chunk_hdr *chunk, *next, *prev; + + if (!data) + return; + + WARN_ON(!IS_ALIGNED((unsigned long)data, MIN_ALLOC_SIZE)); + WARN_ON(data >= allocator->end || data < allocator->start + chunk_hdr_size()); + + guard(hyp_spinlock)(&allocator->lock); + + chunk = chunk_get(container_of(data, struct chunk_hdr, data)); + WARN_ON(!chunk_is_used(chunk)); + chunk_set_unused(chunk); + chunk_set_hash(chunk); + + next = chunk_get_next(chunk); + if (next) + hyp_allocator_destroy_chunk(allocator, chunk, next); + + prev = chunk_get_prev(chunk); + if (prev) + hyp_allocator_destroy_chunk(allocator, prev, chunk); +} + +static unsigned long hyp_allocator_chunk_reclaimable(struct hyp_allocator *allocator, + const struct chunk_hdr *chunk, + u64 *__addr, u64 *__end) +{ + struct chunk_hdr *next; + void *addr, *end; + + /* Last chunk in the allocator */ + if (chunk == allocator->head && chunk == allocator->tail && !chunk_is_used(chunk)) { + addr = (void *)chunk; + end = chunk_unmapped(chunk); + if (!end) + end = allocator->end; + goto end; + } + + next = chunk_get_next(chunk); + + /* Last chunk in the list we can reclaim, even if used */ + if (!next) { + addr = chunk_data(chunk) + chunk_used_size(chunk, allocator); + addr = PTR_ALIGN(addr, PAGE_SIZE); + end = chunk_unmapped(chunk); + if (!end) + end = allocator->end; + goto end; + } + + if (chunk_is_used(chunk)) + return 0; + + addr = PTR_ALIGN(chunk_data(chunk), PAGE_SIZE); + end = chunk_unmapped(chunk); + if (!end) + end = PTR_ALIGN_DOWN(next, PAGE_SIZE); + +end: + if (addr >= end) + return 0; + + if (__end) + *__end = (u64)end; + if (__addr) + *__addr = (u64)addr; + + return (end - addr) >> PAGE_SHIFT; +} + +static void hyp_allocator_reclaim_chunk(struct hyp_allocator *allocator, struct chunk_hdr *chunk, + void *addr, void *end) +{ + struct chunk_hdr *next; + + WARN_ON(end <= addr); + + /* We are about to destroy the last chunk in the allocator */ + if (addr == allocator->start) { + allocator->tail = allocator->head = chunk = NULL; + goto unmap; + } + + next = chunk_get_next(chunk); + + /* + * Split the reclaimed chunk at the next page boundary, + * this ensures no orphan mapped region is created. Splitting at the page boundary is always + * possible because chunks always leave a minimum distance to the page start. + * + * +--------------+ + * |______________| + * |______________|<- Next chunk + * |_ _ _ __ _ _ _| + * | |<- Page-aligned split + * +--------------+ + * +--------------+ + * | | + * | |<- Page reclaimed + * | | + * | | + * +--------------+ + * +--------------+ + * | | + * |______________| + * |______________|<- Chunk to split + * | | + * +--------------+ + */ + if (next && !chunk_unmapped(chunk) && next != end) + WARN_ON(IS_ERR_OR_NULL(hyp_allocator_create_chunk(allocator, chunk, end, next, + false))); +unmap: + hyp_allocator_unmap(allocator, chunk, addr, end); +} + +/* + * Return the best reclaimable chunk which is the highest chunk in the list + * with the biggest reclaimable region. + */ +static struct chunk_hdr *hyp_allocator_find_reclaimable_chunk(struct hyp_allocator *allocator, + u64 *addr, u64 *end) +{ + struct chunk_hdr *chunk, *best_chunk = NULL; + unsigned long best_reclaimable = 0; + + chunk = allocator->head; + while (chunk) { + u64 __addr, __end; + unsigned long reclaimable = hyp_allocator_chunk_reclaimable(allocator, chunk, + &__addr, &__end); + + /* Favour the top biggest chunks */ + if (reclaimable && reclaimable >= best_reclaimable) { + best_reclaimable = reclaimable; + best_chunk = chunk; + *addr = __addr; + *end = __end; + } + + chunk = chunk_get_next(chunk); + } + + return best_chunk; +} + +static void hyp_allocator_drain_memcache(struct hyp_allocator *allocator, + struct kvm_hyp_memcache *host_mc, + unsigned long target) +{ + struct kvm_hyp_memcache *mc = &allocator->mc; + + while (target && mc->nr_pages) { + void *page = pop_hyp_memcache(mc, hyp_phys_to_virt); + + memset(page, 0, PAGE_SIZE); + kvm_flush_dcache_to_poc(page, PAGE_SIZE); + push_hyp_memcache(host_mc, page, hyp_virt_to_phys); + WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(page), 1)); + + target--; + } +} + +static void hyp_allocator_reclaim(struct hyp_allocator *allocator, struct kvm_hyp_memcache *host_mc, + unsigned long target) +{ + unsigned long reclaimed = 0; + + if (!target) + return; + + guard(hyp_spinlock)(&allocator->lock); + + do { + struct chunk_hdr *chunk; + unsigned long r; + u64 addr, end; + + chunk = hyp_allocator_find_reclaimable_chunk(allocator, &addr, &end); + if (!chunk) + break; + + r = min((end - addr) >> PAGE_SHIFT, target - reclaimed); + addr = end - (r << PAGE_SHIFT); + hyp_allocator_reclaim_chunk(allocator, chunk, (void *)addr, (void *)end); + + reclaimed += r; + } while (reclaimed < target); + + hyp_allocator_drain_memcache(allocator, host_mc, target); +} + +static unsigned long hyp_allocator_reclaimable(struct hyp_allocator *allocator) +{ + unsigned long reclaimable; + struct chunk_hdr *chunk; + + guard(hyp_spinlock)(&allocator->lock); + + /* + * While allocator->mc typically contains transient pages, + * over-allocation or an aborted operation could leave pages behind. + * Account for allocator->mc so stalled topup pages can be reclaimed. + */ + reclaimable = allocator->mc.nr_pages; + + chunk = allocator->head; + while (chunk) { + reclaimable += hyp_allocator_chunk_reclaimable(allocator, chunk, NULL, NULL); + chunk = chunk_get_next(chunk); + } + + return reclaimable; +} + +static int hyp_allocator_topup(struct hyp_allocator *allocator, + struct kvm_hyp_memcache *host_mc) +{ + struct kvm_hyp_memcache *alloc_mc = &allocator->mc; + + guard(hyp_spinlock)(&allocator->lock); + return refill_memcache(alloc_mc, host_mc->nr_pages + alloc_mc->nr_pages, host_mc); +} + +static u32 hyp_allocator_topup_needed(struct hyp_allocator *allocator) +{ + u32 *topup_needed = this_cpu_ptr(allocator->topup_needed); + u32 ret = *topup_needed; + + *topup_needed = 0; + + return ret; +} + +static int hyp_allocator_init(struct hyp_allocator *allocator, size_t size) +{ + unsigned long start; + int ret; + + size = PAGE_ALIGN(size); + + /* constrained by chunk_hdr u32 types */ + if (size > U32_MAX || !size) + return -EINVAL; + + ret = pkvm_alloc_private_va_range(size, &start); + if (ret) + return ret; + + allocator->first_unmapped = allocator->start = (void *)start; + allocator->end = allocator->start + size; + hyp_spin_lock_init(&allocator->lock); + + return 0; +} + +static DEFINE_PER_CPU(int, __hyp_allocator_errno); +static DEFINE_PER_CPU(u32, __hyp_allocator_topup_needed); + +static struct hyp_allocator hyp_allocator = { + .errno = &__hyp_allocator_errno, + .topup_needed = &__hyp_allocator_topup_needed, +}; + +/** + * hyp_alloc() - Allocate memory from the heap allocator + * + * @size: Allocation size in bytes. + * + * Callers must ensure they do not hold pkvm_pgd_lock to avoid deadlocks, + * as the allocator may modify the hyp stage-1 page-table. + * + * Return: A pointer to the allocated memory on success, else NULL. + */ +void *hyp_alloc(size_t size) +{ + return hyp_allocator_alloc(&hyp_allocator, size); +} + +/** + * hyp_free() - Free memory allocated with hyp_alloc() + * + * @data: Address returned by the original hyp_alloc(). + * + * The use of any other address than one returned by hyp_alloc() will cause a + * hypervisor panic. + */ +void hyp_free(void *data) +{ + hyp_allocator_free(&hyp_allocator, data); +} + +/** + * hyp_alloc_errno() - Read the errno on allocation error + * + * Get the return code from an allocation failure. + * + * Return: -ENOMEM if the allocator needs a refill from the host, -E2BIG if + * there is no VA space left else 0. + */ +int hyp_alloc_errno(void) +{ + return hyp_allocator_errno(&hyp_allocator); +} + +static int selftest_init(void); + +int hyp_alloc_init(size_t size) +{ + int ret; + + ret = hyp_allocator_init(&hyp_allocator, size); + if (ret) + return ret; + + return selftest_init(); +} + +void hyp_alloc_reclaim(struct kvm_hyp_memcache *mc, unsigned long target) +{ + hyp_allocator_reclaim(&hyp_allocator, mc, target); +} + +unsigned long hyp_alloc_reclaimable(void) +{ + return hyp_allocator_reclaimable(&hyp_allocator); +} + +int hyp_alloc_topup(struct kvm_hyp_memcache *host_mc) +{ + return hyp_allocator_topup(&hyp_allocator, host_mc); +} + +u32 hyp_alloc_topup_needed(void) +{ + return hyp_allocator_topup_needed(&hyp_allocator); +} + +#ifdef CONFIG_NVHE_EL2_DEBUG +#define SELFTEST_MAX_PAGES 6 +#define SELFTEST_MAX_SIZE (PAGE_SIZE * SELFTEST_MAX_PAGES) + +static DEFINE_PER_CPU(int, __selftest_errno); +static DEFINE_PER_CPU(u32, __selftest_topup_needed); + +static struct hyp_allocator selftest_allocator = { + .errno = &__selftest_errno, + .topup_needed = &__selftest_topup_needed, + .lock = __HYP_SPIN_LOCK_UNLOCKED, +}; + +int hyp_alloc_selftest_topup(struct kvm_hyp_memcache *host_mc) +{ + return hyp_allocator_topup(&selftest_allocator, host_mc); +} + +void hyp_alloc_selftest_reclaim(struct kvm_hyp_memcache *host_mc, unsigned long target) +{ + hyp_allocator_reclaim(&selftest_allocator, host_mc, target); +} + +u32 hyp_alloc_selftest_topup_needed(void) +{ + return hyp_allocator_topup_needed(&selftest_allocator); +} + +static int selftest_init(void) +{ + return hyp_allocator_init(&selftest_allocator, SELFTEST_MAX_SIZE); +} + +static void *selftest_alloc(size_t size) +{ + return hyp_allocator_alloc(&selftest_allocator, size); +} + +static void selftest_free(void *addr) +{ + hyp_allocator_free(&selftest_allocator, addr); +} + +static int selftest_errno(void) +{ + return hyp_allocator_errno(&selftest_allocator); +} + +int hyp_allocator_selftest(void) +{ + struct hyp_allocator *allocator = &selftest_allocator; + static DEFINE_HYP_SPINLOCK(selftest_lock); + struct kvm_hyp_memcache host_mc = { }; + void *addr1, *addr2, *addr3, *addr4; + int ret; + + guard(hyp_spinlock)(&selftest_lock); + + if (allocator->mc.nr_pages < SELFTEST_MAX_PAGES) { + *this_cpu_ptr(allocator->topup_needed) = SELFTEST_MAX_PAGES - + allocator->mc.nr_pages; + return -ENOMEM; + } + + selftest_alloc(SELFTEST_MAX_SIZE); + if (selftest_errno() != -E2BIG) + return -EINVAL; + + selftest_alloc(SIZE_MAX); + if (selftest_errno() != -E2BIG) + return -EINVAL; + + /* Test first chunk */ + addr1 = selftest_alloc(0); + if (!addr1 || addr1 != (void *)allocator->start + chunk_hdr_size()) + return -EINVAL; + + /* Test second contiguous chunk with unaligned size */ + addr2 = selftest_alloc(MIN_ALLOC_SIZE + 1); + if (!addr2) + return -EINVAL; + addr3 = selftest_alloc(0); + if (!addr3 || + addr3 != addr2 + (2 * MIN_ALLOC_SIZE) + chunk_hdr_size()) + return -EINVAL; + + selftest_free(addr3); + + /* Test chunk recycling */ + selftest_free(addr1); + if (addr1 != selftest_alloc(0)) + return -EINVAL; + + /* Test chunk forward merging */ + addr3 = selftest_alloc(0); + selftest_free(addr2); + selftest_free(addr1); + if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2)) + return -EINVAL; + + selftest_free(addr1); + + /* Test chunk splitting */ + if (addr1 != selftest_alloc(0)) + return -EINVAL; + if (addr2 != selftest_alloc(0)) + return -EINVAL; + + /* Test chunk backward merging */ + selftest_free(addr1); + selftest_free(addr2); + if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 2)) + return -EINVAL; + + selftest_free(addr1); + + /* Test chunk 3-way merging */ + addr1 = selftest_alloc(0); + addr2 = selftest_alloc(0); + addr4 = selftest_alloc(0); + selftest_free(addr1); + selftest_free(addr3); + selftest_free(addr2); + if (addr1 != selftest_alloc(MIN_ALLOC_SIZE * 3)) + return -EINVAL; + + selftest_free(addr4); + selftest_free(addr1); + + /* Test reclaiming */ + if (addr1 != selftest_alloc(0)) + return -EINVAL; + if (addr2 != selftest_alloc(PAGE_SIZE * 2)) + return -EINVAL; + addr3 = selftest_alloc(0); + addr4 = selftest_alloc(PAGE_SIZE); + + /* Test reclaiming the last chunk of the list */ + selftest_free(addr4); + hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES); + if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 3) + return -EINVAL; + + /* Test punching a hole in the middle of a free chunk ... */ + selftest_free(addr2); + hyp_allocator_reclaim(allocator, &host_mc, SELFTEST_MAX_PAGES); + if (host_mc.nr_pages != SELFTEST_MAX_PAGES - 2) + return -EINVAL; + + if (selftest_alloc(PAGE_SIZE)) + return -EINVAL; + if (selftest_errno() != -ENOMEM) + return -EINVAL; + + /* ... and to refill this hole */ + ret = hyp_allocator_topup(allocator, &host_mc); + if (ret) + return ret; + /* Chunk at addr2 was made smaller by the reclaim */ + if (addr2 != selftest_alloc(PAGE_SIZE)) + return -EINVAL; + + /* Test reclaiming the entire allocator from the host */ + selftest_free(addr3); + selftest_free(addr2); + selftest_free(addr1); + if (addr1 != selftest_alloc(SELFTEST_MAX_PAGES * PAGE_SIZE - chunk_hdr_size())) + return -EINVAL; + selftest_free(addr1); + + return 0; +} +#else +static int selftest_init(void) { return 0; } +#endif diff --git a/arch/arm64/kvm/hyp/nvhe/events.c b/arch/arm64/kvm/hyp/nvhe/events.c index add9383aadb5..eba14365e98b 100644 --- a/arch/arm64/kvm/hyp/nvhe/events.c +++ b/arch/arm64/kvm/hyp/nvhe/events.c @@ -7,8 +7,16 @@ #include <nvhe/mm.h> #include <nvhe/trace.h> +#include <asm/kvm_hypevents.h> + #include <nvhe/define_events.h> +/* + * The hyp_printk event is not declared with HYP_EVENT in kvm_hypevents.h, + * so we manually add the boilerplate here. + */ +HYP_EVENT(hyp_printk, 0, 0, 0, 0); + int __tracing_enable_event(unsigned short id, bool enable) { struct hyp_event_id *event_id = &__hyp_event_ids_start[id]; diff --git a/arch/arm64/kvm/hyp/nvhe/ffa.c b/arch/arm64/kvm/hyp/nvhe/ffa.c index a327c2bbb6b6..5ad74ab265b5 100644 --- a/arch/arm64/kvm/hyp/nvhe/ffa.c +++ b/arch/arm64/kvm/hyp/nvhe/ffa.c @@ -42,6 +42,9 @@ */ #define HOST_FFA_ID 0 +#define FFA_NOTIF_RECEIVER_ENDP_MASK GENMASK(15, 0) +#define FFA_NOTIF_SENDER_ENDP_MASK GENMASK(31, 16) + /* * A buffer to hold the maximum descriptor size we can see from the host, * which is required when the SPMD returns a fragmented FFA_MEM_RETRIEVE_RESP @@ -71,6 +74,21 @@ static u32 hyp_ffa_version; static bool has_version_negotiated; static hyp_spinlock_t version_lock; +static bool ffa_check_unused_args_sbz(struct kvm_cpu_context *ctxt, int first_reg) +{ + DECLARE_REG(u32, func_id, ctxt, 0); + int reg, end_reg = 7; + + if (hyp_ffa_version >= FFA_VERSION_1_2 && ARM_SMCCC_IS_64(func_id)) + end_reg = 17; + for (reg = first_reg; reg <= end_reg; reg++) { + if (cpu_reg(ctxt, reg)) + return true; + } + + return false; +} + static void ffa_to_smccc_error(struct arm_smccc_1_2_regs *res, u64 ffa_errno) { *res = (struct arm_smccc_1_2_regs) { @@ -239,6 +257,11 @@ static void do_ffa_rxtx_map(struct arm_smccc_1_2_regs *res, int ret = 0; void *rx_virt, *tx_virt; + if (ffa_check_unused_args_sbz(ctxt, 4)) { + ret = FFA_RET_INVALID_PARAMETERS; + goto out; + } + if (npages != (KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) / FFA_PAGE_SIZE) { ret = FFA_RET_INVALID_PARAMETERS; goto out; @@ -315,6 +338,11 @@ static void do_ffa_rxtx_unmap(struct arm_smccc_1_2_regs *res, DECLARE_REG(u32, id, ctxt, 1); int ret = 0; + if (ffa_check_unused_args_sbz(ctxt, 2)) { + ret = FFA_RET_INVALID_PARAMETERS; + goto out; + } + if (id != HOST_FFA_ID) { ret = FFA_RET_INVALID_PARAMETERS; goto out; @@ -421,6 +449,11 @@ static void do_ffa_mem_frag_tx(struct arm_smccc_1_2_regs *res, int ret = FFA_RET_INVALID_PARAMETERS; u32 nr_ranges; + if (ffa_check_unused_args_sbz(ctxt, 5)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + if (fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) goto out; @@ -483,6 +516,11 @@ static void __do_ffa_mem_xfer(const u64 func_id, int ret = 0; size_t mem_region_len = FFA_MEM_REGION_SZ(hyp_ffa_version); + if (ffa_check_unused_args_sbz(ctxt, 5)) { + ret = FFA_RET_INVALID_PARAMETERS; + goto out; + } + if (addr_mbz || npages_mbz || fraglen > len || fraglen > KVM_FFA_MBOX_NR_PAGES * PAGE_SIZE) { ret = FFA_RET_INVALID_PARAMETERS; @@ -586,6 +624,11 @@ static void do_ffa_mem_reclaim(struct arm_smccc_1_2_regs *res, int ret = 0; u64 handle; + if (ffa_check_unused_args_sbz(ctxt, 4)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + handle = PACK_HANDLE(handle_lo, handle_hi); hyp_spin_lock(&host_buffers.lock); @@ -691,14 +734,6 @@ static bool ffa_call_supported(u64 func_id) case FFA_RXTX_MAP: case FFA_MEM_DONATE: case FFA_MEM_RETRIEVE_REQ: - /* Optional notification interfaces added in FF-A 1.1 */ - case FFA_NOTIFICATION_BITMAP_CREATE: - case FFA_NOTIFICATION_BITMAP_DESTROY: - case FFA_NOTIFICATION_BIND: - case FFA_NOTIFICATION_UNBIND: - case FFA_NOTIFICATION_SET: - case FFA_NOTIFICATION_GET: - case FFA_NOTIFICATION_INFO_GET: /* Optional interfaces added in FF-A 1.2 */ case FFA_MSG_SEND_DIRECT_REQ2: /* Optional per 7.5.1 */ case FFA_MSG_SEND_DIRECT_RESP2: /* Optional per 7.5.1 */ @@ -785,6 +820,11 @@ static void do_ffa_version(struct arm_smccc_1_2_regs *res, { DECLARE_REG(u32, ffa_req_version, ctxt, 1); + if (ffa_check_unused_args_sbz(ctxt, 2)) { + res->a0 = FFA_RET_NOT_SUPPORTED; + return; + } + if (FFA_MAJOR_VERSION(ffa_req_version) != 1) { res->a0 = FFA_RET_NOT_SUPPORTED; return; @@ -834,6 +874,11 @@ static void do_ffa_part_get(struct arm_smccc_1_2_regs *res, DECLARE_REG(u32, flags, ctxt, 5); u32 count, partition_sz, copy_sz; + if (ffa_check_unused_args_sbz(ctxt, 6)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + hyp_spin_lock(&host_buffers.lock); if (!host_buffers.rx) { ffa_to_smccc_res(res, FFA_RET_BUSY); @@ -878,6 +923,140 @@ out_unlock: hyp_spin_unlock(&host_buffers.lock); } +static void do_ffa_notif_bitmap(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + DECLARE_REG(u32, func_id, ctxt, 0); + DECLARE_REG(u32, vmid, ctxt, 1); + struct arm_smccc_1_2_regs *args; + + if (ffa_check_unused_args_sbz(ctxt, func_id == FFA_NOTIFICATION_BITMAP_CREATE ? 3 : 2)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (vmid != HOST_FFA_ID) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + +static void do_ffa_notif_bind(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + DECLARE_REG(u32, endp_id, ctxt, 1); + DECLARE_REG(u32, flags, ctxt, 2); + struct arm_smccc_1_2_regs *args; + + if (ffa_check_unused_args_sbz(ctxt, 5)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (FIELD_GET(FFA_NOTIF_RECEIVER_ENDP_MASK, endp_id) != HOST_FFA_ID) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (flags > 1) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + +static void do_ffa_notif_unbind(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + DECLARE_REG(u32, endp_id, ctxt, 1); + DECLARE_REG(u32, reserved, ctxt, 2); + struct arm_smccc_1_2_regs *args; + + if (ffa_check_unused_args_sbz(ctxt, 5) || reserved) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (FIELD_GET(FFA_NOTIF_RECEIVER_ENDP_MASK, endp_id) != HOST_FFA_ID) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + +static void do_ffa_notif_set(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + DECLARE_REG(u32, endp_id, ctxt, 1); + DECLARE_REG(u32, flags, ctxt, 2); + struct arm_smccc_1_2_regs *args; + + if (FIELD_GET(FFA_NOTIF_SENDER_ENDP_MASK, endp_id) != HOST_FFA_ID) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (ffa_check_unused_args_sbz(ctxt, 5)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (flags & GENMASK(15, 2)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + +static void do_ffa_notif_get(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + DECLARE_REG(u32, endp_id, ctxt, 1); + DECLARE_REG(u32, flags, ctxt, 2); + struct arm_smccc_1_2_regs *args; + + if (FIELD_GET(FFA_NOTIF_RECEIVER_ENDP_MASK, endp_id) != HOST_FFA_ID) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (ffa_check_unused_args_sbz(ctxt, 3)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + if (flags & GENMASK(31, 4)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + +static void do_ffa_notif_info_get(struct arm_smccc_1_2_regs *res, + struct kvm_cpu_context *ctxt) +{ + struct arm_smccc_1_2_regs *args; + + if (ffa_check_unused_args_sbz(ctxt, 1)) { + ffa_to_smccc_res(res, FFA_RET_INVALID_PARAMETERS); + return; + } + + args = (void *)&ctxt->regs.regs[0]; + hyp_smccc_1_2_smc(args, res); +} + bool kvm_host_ffa_handler(struct kvm_cpu_context *host_ctxt, u32 func_id) { struct arm_smccc_1_2_regs res = {0}; @@ -906,6 +1085,11 @@ bool kvm_host_ffa_handler(struct kvm_cpu_context *host_ctxt, u32 func_id) switch (func_id) { case FFA_FEATURES: + if (ffa_check_unused_args_sbz(host_ctxt, 3)) { + ffa_to_smccc_res(&res, FFA_RET_INVALID_PARAMETERS); + goto out_handled; + } + if (!do_ffa_features(&res, host_ctxt)) return false; goto out_handled; @@ -936,6 +1120,26 @@ bool kvm_host_ffa_handler(struct kvm_cpu_context *host_ctxt, u32 func_id) case FFA_PARTITION_INFO_GET: do_ffa_part_get(&res, host_ctxt); goto out_handled; + case FFA_NOTIFICATION_BITMAP_CREATE: + case FFA_NOTIFICATION_BITMAP_DESTROY: + do_ffa_notif_bitmap(&res, host_ctxt); + goto out_handled; + case FFA_NOTIFICATION_BIND: + do_ffa_notif_bind(&res, host_ctxt); + goto out_handled; + case FFA_NOTIFICATION_UNBIND: + do_ffa_notif_unbind(&res, host_ctxt); + goto out_handled; + case FFA_NOTIFICATION_SET: + do_ffa_notif_set(&res, host_ctxt); + goto out_handled; + case FFA_NOTIFICATION_GET: + do_ffa_notif_get(&res, host_ctxt); + goto out_handled; + case FFA_NOTIFICATION_INFO_GET: + case FFA_FN64_NOTIFICATION_INFO_GET: + do_ffa_notif_info_get(&res, host_ctxt); + goto out_handled; } if (ffa_call_supported(func_id)) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index ac64a036b0a9..137a8144d09e 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -4,10 +4,13 @@ * Author: Andrew Scull <ascull@google.com> */ +#include <kvm/arm_hypercalls.h> + #include <hyp/adjust_pc.h> #include <hyp/switch.h> #include <linux/irqchip/arm-gic-v3.h> +#include <uapi/linux/psci.h> #include <asm/pgtable-types.h> #include <asm/kvm_asm.h> @@ -17,6 +20,7 @@ #include <asm/kvm_hypevents.h> #include <asm/kvm_mmu.h> +#include <nvhe/alloc.h> #include <nvhe/ffa.h> #include <nvhe/mem_protect.h> #include <nvhe/mm.h> @@ -26,11 +30,449 @@ DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); +/* + * Define a hypercall handler: handle_<name> unmarshals the arguments from + * the host context and hands them, correctly typed, to the body that + * follows the macro. The parameter list is type-checked against the + * signature declared in <asm/kvm_hcall.h>, so the handler cannot drift + * from what the typed caller stubs marshal in. Modelled on the syscall + * wrappers. + */ +/* Truncate the fixed list of argument registers to the declared signature. */ +#define KVM_HOST_HCALL_REGS(...) \ + __KVM_HCALL_MAP_N(COUNT_ARGS(__VA_ARGS__), __KVM_HCALL_ARGS \ + ,, cpu_reg(host_ctxt, 1),, cpu_reg(host_ctxt, 2) \ + ,, cpu_reg(host_ctxt, 3),, cpu_reg(host_ctxt, 4) \ + ,, cpu_reg(host_ctxt, 5),, cpu_reg(host_ctxt, 6)) + +#define set_cpu_reg_ulong(ctxt, r, v) { cpu_reg(ctxt, r) = v; } +#define set_cpu_reg_u64(ctxt, r, v) { cpu_reg(ctxt, r) = v; } +#define set_cpu_reg_int(ctxt, r, v) { cpu_reg(ctxt, r) = v; } +#define set_cpu_reg_void(ctxt, r, v) { v; } +#define set_cpu_reg(ctxt, r, t, v) set_cpu_reg_##t(ctxt, r, v) + +#define DEFINE_KVM_HOST_HCALL(ret, name, ...) \ + static kvm_host_hcall_sig_##name __do_##name; \ + static __always_inline \ + ret __se_##name(__KVM_HCALL_MAP(__KVM_HCALL_LONG, __VA_ARGS__)) \ + { \ + return __do_##name(__KVM_HCALL_MAP(__KVM_HCALL_CAST, __VA_ARGS__)); \ + } \ + static void handle_##name(struct kvm_cpu_context *host_ctxt) \ + { \ + set_cpu_reg(host_ctxt, 1, ret, __se_##name(KVM_HOST_HCALL_REGS(__VA_ARGS__))); \ + } \ + static ret __do_##name(__KVM_HCALL_MAP(__KVM_HCALL_DECL, __VA_ARGS__)) + +#define DEFINE_KVM_HOST_HCALL0(ret, name) \ + static kvm_host_hcall_sig_##name __do_##name; \ + static void handle_##name(struct kvm_cpu_context *host_ctxt) \ + { \ + set_cpu_reg(host_ctxt, 1, ret, __do_##name()); \ + } \ + static ret __do_##name(void) + +/* + * Encode a hypervisor request in the host SMCCC return registers for known + * error numbers. + * + * Must be paired with pkvm_call_hyp_req() on the host side. + */ +static int errno_to_smccc(int ret) +{ + struct pkvm_hyp_req req = { .type = PKVM_HYP_NO_REQ }; + + switch (ret) { + case -ENOMEM: { + u32 nr_pages = hyp_alloc_topup_needed(); + + if (nr_pages) { + req.type = PKVM_HYP_REQ_HYP_ALLOC; + req.mem.nr_pages = nr_pages; + } + break; + } + } + + pkvm_hyp_req_to_smccc(host_data_ptr(host_ctxt), &req); + + return ret; +} + /* Number of implemented GICv3 LRs. Used by flush_hyp_vcpu(). */ unsigned int hyp_gicv3_nr_lr; void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt); +typedef void (*hyp_entry_exit_handler_fn)(struct pkvm_hyp_vcpu *); + +static bool pvm_sys64_is_write(u64 esr) +{ + return (esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_WRITE; +} + +static void handle_pvm_entry_wfx(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + /* Exceptions have priority; the host injects none on WFx. */ + if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) + return; + + if (vcpu_get_flag(host_vcpu, INCREMENT_PC)) { + vcpu_clear_flag(vcpu, PC_UPDATE_REQ); + kvm_incr_pc(vcpu); + } +} + +static void handle_pvm_entry_sys64(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + bool pc_update; + + /* Exceptions have priority over anything else */ + if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) { + /* A host-requested exception on SYS64 is always an UNDEF. */ + u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL; + + __vcpu_assign_sys_reg(vcpu, ESR_EL1, esr); + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + return; + } + + /* Handle PC increment on a host-emulated access */ + pc_update = vcpu_get_flag(host_vcpu, INCREMENT_PC); + if (pc_update) { + vcpu_clear_flag(vcpu, PC_UPDATE_REQ); + kvm_incr_pc(vcpu); + } + + /* If the host emulated a read access, update the register */ + if (pc_update && !pvm_sys64_is_write(kvm_vcpu_get_esr(vcpu))) { + /* r0 as transfer register between the guest and the host. */ + u64 rt_val = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]); + int rt = kvm_vcpu_sys_get_rt(vcpu); + + vcpu_set_reg(vcpu, rt, rt_val); + } +} + +static void handle_pvm_entry_iabt(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + unsigned long cpsr = *vcpu_cpsr(vcpu); + u32 esr = ESR_ELx_IL; + + if (!vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) + return; + + /* The host's only IABT injection: an external abort. */ + if ((cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t) + esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT); + else + esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT); + + esr |= ESR_ELx_FSC_EXTABT; + + __vcpu_assign_sys_reg(vcpu, ESR_EL1, esr); + __vcpu_assign_sys_reg(vcpu, FAR_EL1, kvm_vcpu_get_hfar(vcpu)); + + /* Injected by __kvm_adjust_pc() on entry. */ + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); +} + +/* + * Clamp MMIO data to the access width, so a write does not leak the + * register's upper bits and a read takes no bits beyond the load. The + * host applies endianness. + */ +static inline u64 kvm_mmio_clamp_data(struct kvm_vcpu *vcpu, u64 val) +{ + unsigned int len = kvm_vcpu_dabt_get_as(vcpu); + + return val & GENMASK_U64(len * 8 - 1, 0); +} + +/* + * Complete an MMIO load: sign-extend from EL2's own syndrome, as the + * architecture does. + */ +static inline u64 kvm_mmio_read_data(struct kvm_vcpu *vcpu, u64 val) +{ + val = kvm_mmio_clamp_data(vcpu, val); + + if (kvm_vcpu_dabt_issext(vcpu)) + val = sign_extend64(val, kvm_vcpu_dabt_get_as(vcpu) * 8 - 1); + + if (!kvm_vcpu_dabt_issf(vcpu)) + val &= GENMASK_U64(31, 0); + + return val; +} + +static void handle_pvm_entry_dabt(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + bool pc_update; + + /* Exceptions have priority over anything else */ + if (vcpu_get_flag(host_vcpu, PENDING_EXCEPTION)) { + unsigned long cpsr = *vcpu_cpsr(vcpu); + u32 esr = ESR_ELx_IL; + + if ((cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t) + esr |= (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT); + else + esr |= (ESR_ELx_EC_DABT_CUR << ESR_ELx_EC_SHIFT); + + esr |= ESR_ELx_FSC_EXTABT; + + __vcpu_assign_sys_reg(vcpu, ESR_EL1, esr); + __vcpu_assign_sys_reg(vcpu, FAR_EL1, kvm_vcpu_get_hfar(vcpu)); + + /* Injected by __kvm_adjust_pc() on entry. */ + kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); + + /* Cancel any in-flight MMIO */ + vcpu->mmio_needed = false; + return; + } + + /* Handle PC increment on MMIO, or on a CMO the host skipped */ + pc_update = vcpu_get_flag(host_vcpu, INCREMENT_PC) && + (vcpu->mmio_needed || esr_dabt_is_cm(kvm_vcpu_get_esr(vcpu))); + if (pc_update) { + vcpu_clear_flag(vcpu, PC_UPDATE_REQ); + kvm_incr_pc(vcpu); + } + + /* If the host emulated an MMIO read, update the register */ + if (pc_update && vcpu->mmio_needed && !kvm_vcpu_dabt_iswrite(vcpu)) { + /* r0 as transfer register between the guest and the host. */ + u64 rd_val = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]); + int rd = kvm_vcpu_dabt_get_rd(vcpu); + + rd_val = kvm_mmio_read_data(vcpu, rd_val); + vcpu_set_reg(vcpu, rd, rd_val); + } + + vcpu->mmio_needed = false; +} + +static void handle_pvm_entry_hvc64(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + u64 ret = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]); + u32 psci_fn = smccc_get_function(vcpu); + + switch (psci_fn) { + case PSCI_0_2_FN_CPU_ON: + case PSCI_0_2_FN64_CPU_ON: + /* + * Roll back a CPU_ON the host failed, unless the target + * already reached ON: it is running, and the guest sees + * SUCCESS. + */ + if (ret != PSCI_RET_SUCCESS) { + unsigned long cpu_id = smccc_get_arg1(vcpu); + struct pkvm_hyp_vcpu *target_vcpu; + struct pkvm_hyp_vm *hyp_vm; + int prev; + + hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu); + target_vcpu = pkvm_mpidr_to_hyp_vcpu(hyp_vm, cpu_id); + + /* + * pvm_psci_vcpu_on() resolved this MPIDR and vcpus[] + * entries are never removed, so the lookup cannot miss. + * The release orders this vCPU's reset_state writes + * before OFF, for the next CPU_ON. + */ + prev = cmpxchg_release(&target_vcpu->power_state, + PSCI_0_2_AFFINITY_LEVEL_ON_PENDING, + PSCI_0_2_AFFINITY_LEVEL_OFF); + switch (prev) { + case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING: + /* + * Leave reset_state.reset set: a clear races a + * fresh CPU_ON's publish. The stale pc/r0/be are + * the guest's own. ALREADY_ON is PSCI's retry + * signal for a CPU_ON that raced the CPU_OFF. + */ + if (ret != PSCI_RET_ALREADY_ON) + ret = PSCI_RET_INTERNAL_FAILURE; + break; + case PSCI_0_2_AFFINITY_LEVEL_ON: + case PSCI_0_2_AFFINITY_LEVEL_OFF: + /* Target already ran (and may have stopped). */ + ret = PSCI_RET_SUCCESS; + break; + default: + ret = PSCI_RET_INTERNAL_FAILURE; + break; + } + } + + break; + default: + break; + } + + vcpu_set_reg(vcpu, 0, ret); +} + +/* The host's view of a syndrome: the guest register index is withheld. */ +static u64 pvm_host_esr(u64 esr) +{ + switch (ESR_ELx_EC(esr)) { + case ESR_ELx_EC_WFx: + return esr & ~ESR_ELx_WFx_ISS_RN; + case ESR_ELx_EC_SYS64: + return esr & ~ESR_ELx_SYS64_ISS_RT_MASK; + case ESR_ELx_EC_DABT_LOW: + return esr & ~ESR_ELx_SRT_MASK; + default: + return esr; + } +} + +/* + * The host's view of PSTATE: the mode, with SErrors masked so that the host + * pends an SError through HCR_EL2.VSE rather than emulating the entry. + */ +static unsigned long pvm_host_pstate(unsigned long pstate) +{ + return (pstate & PSR_MODE_MASK) | PSR_A_BIT; +} + +static void handle_pvm_exit_wfx(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + *vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu)); +} + +static void handle_pvm_exit_sys64(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + u64 esr = kvm_vcpu_get_esr(vcpu); + + /* The mode is required for the host to emulate some sysregs */ + *vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu)); + + /* r0 as transfer register between the guest and the host. */ + if (pvm_sys64_is_write(esr)) { + int rt = kvm_vcpu_sys_get_rt(vcpu); + u64 rt_val = vcpu_get_reg(vcpu, rt); + + vcpu_set_reg(host_vcpu, 0, rt_val); + } +} + +static void handle_pvm_exit_iabt(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2; +} + +static void handle_pvm_exit_dabt(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + u64 sctlr; + + /* + * EL2 has no memslot view: a decodable data abort is prepared as MMIO + * for the host to resolve. On unbacked memory the host injects an SEA + * for one with ISV clear (LDP/STP, atomics), which EL2 does not + * decode, and skips a cache maintenance operation, as for any guest. + */ + vcpu->mmio_needed = kvm_vcpu_dabt_isvalid(vcpu); + + /* r0 as transfer register between the guest and the host. */ + if (vcpu->mmio_needed && kvm_vcpu_dabt_iswrite(vcpu)) { + int rt = kvm_vcpu_dabt_get_rd(vcpu); + u64 rt_val = vcpu_get_reg(vcpu, rt); + + rt_val = kvm_mmio_clamp_data(vcpu, rt_val); + vcpu_set_reg(host_vcpu, 0, rt_val); + } + + *vcpu_cpsr(host_vcpu) = pvm_host_pstate(*vcpu_cpsr(vcpu)); + host_vcpu->arch.fault.far_el2 = vcpu->arch.fault.far_el2 & GENMASK(11, 0); + host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2; + sctlr = __vcpu_sys_reg(vcpu, SCTLR_EL1) & (SCTLR_ELx_EE | SCTLR_EL1_E0E); + __vcpu_assign_sys_reg(host_vcpu, SCTLR_EL1, sctlr); +} + +static void handle_pvm_exit_hvc64(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + int n, i; + + switch (smccc_get_function(vcpu)) { + /* + * CPU_ON: the host uses only the target MPIDR (x1). EL2 resets the + * target from its own copy of the entry point and context id. + */ + case PSCI_0_2_FN_CPU_ON: + case PSCI_0_2_FN64_CPU_ON: + n = 2; + break; + + case PSCI_0_2_FN_CPU_OFF: + case PSCI_0_2_FN_SYSTEM_OFF: + case PSCI_0_2_FN_SYSTEM_RESET: + case PSCI_0_2_FN_CPU_SUSPEND: + case PSCI_0_2_FN64_CPU_SUSPEND: + n = 1; + break; + + case PSCI_0_2_FN_AFFINITY_INFO: + case PSCI_0_2_FN64_AFFINITY_INFO: + case PSCI_1_1_FN_SYSTEM_RESET2: + case PSCI_1_1_FN64_SYSTEM_RESET2: + n = 3; + break; + + /* Unreachable: kvm_handle_pvm_hvc64() forwards only the calls above. */ + default: + hyp_panic(); + } + + /* Pass the HVC function id (r0) and its arguments. */ + for (i = 0; i < n; i++) + vcpu_set_reg(host_vcpu, i, vcpu_get_reg(vcpu, i)); +} + +static const hyp_entry_exit_handler_fn entry_hyp_pvm_handlers[] = { + [0 ... ESR_ELx_EC_MAX] = NULL, + [ESR_ELx_EC_WFx] = handle_pvm_entry_wfx, + [ESR_ELx_EC_SYS64] = handle_pvm_entry_sys64, + [ESR_ELx_EC_IABT_LOW] = handle_pvm_entry_iabt, + [ESR_ELx_EC_DABT_LOW] = handle_pvm_entry_dabt, + [ESR_ELx_EC_HVC64] = handle_pvm_entry_hvc64, +}; + +static const hyp_entry_exit_handler_fn exit_hyp_pvm_handlers[] = { + [0 ... ESR_ELx_EC_MAX] = NULL, + [ESR_ELx_EC_WFx] = handle_pvm_exit_wfx, + [ESR_ELx_EC_SYS64] = handle_pvm_exit_sys64, + [ESR_ELx_EC_IABT_LOW] = handle_pvm_exit_iabt, + [ESR_ELx_EC_DABT_LOW] = handle_pvm_exit_dabt, + [ESR_ELx_EC_HVC64] = handle_pvm_exit_hvc64, +}; + static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) { __vcpu_assign_sys_reg(vcpu, ZCR_EL1, read_sysreg_el1(SYS_ZCR)); @@ -143,6 +585,35 @@ static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu) host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i]; } +static void flush_hyp_timer_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return; + + /* A hyp vcpu has no offset, and sees vtime == ptime. */ + write_sysreg(0, cntvoff_el2); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CVAL_EL0), SYS_CNTV_CVAL); + isb(); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CTL_EL0), SYS_CNTV_CTL); +} + +static void sync_hyp_timer_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return; + + /* + * Preserve the vtimer state so that it is always correct, + * even if the host tries to make a mess. + */ + __vcpu_assign_sys_reg(vcpu, CNTV_CVAL_EL0, read_sysreg_el0(SYS_CNTV_CVAL)); + __vcpu_assign_sys_reg(vcpu, CNTV_CTL_EL0, read_sysreg_el0(SYS_CNTV_CTL)); +} + static void __copy_vcpu_state(const struct kvm_vcpu *from_vcpu, struct kvm_vcpu *to_vcpu) { @@ -189,6 +660,9 @@ static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return; + hyp_vcpu->vcpu.arch.debug_owner = host_vcpu->arch.debug_owner; if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) { @@ -207,6 +681,9 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return; + if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) host_vcpu->arch.vcpu_debug_state = hyp_vcpu->vcpu.arch.vcpu_debug_state; else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu)) @@ -217,6 +694,8 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; u64 host_hcr_mask = PKVM_HCR_EL2_HOST_PVM; + hyp_entry_exit_handler_fn ec_handler; + u8 esr_ec; fpsimd_sve_flush(); flush_debug_state(hyp_vcpu); @@ -229,42 +708,75 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY)) flush_hyp_vcpu_state(hyp_vcpu); + + hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; + hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; host_hcr_mask = PKVM_HCR_EL2_HOST_NPVM; - } else { - hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; } /* __hyp_running_vcpu must be NULL in a guest context. */ hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL; - hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; /* - * HCR_EL2.VSE is host-owned (a pending virtual SError to inject), not a - * trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE - * for the vSError to be delivered. sync_hyp_vcpu() reflects it back. + * A host-injected vSError is masked by the guest's own PSTATE.A, so it + * applies to protected guests too. */ hyp_vcpu->vcpu.arch.hcr_el2 &= ~host_hcr_mask; hyp_vcpu->vcpu.arch.hcr_el2 |= READ_ONCE(host_vcpu->arch.hcr_el2) & host_hcr_mask; - - hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; - - hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2; + hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2; flush_hyp_vgic_state(hyp_vcpu); + flush_hyp_timer_state(hyp_vcpu); hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid; + + switch (ARM_EXCEPTION_CODE(hyp_vcpu->exit_code)) { + case ARM_EXCEPTION_IRQ: + case ARM_EXCEPTION_EL1_SERROR: + case ARM_EXCEPTION_IL: + break; + case ARM_EXCEPTION_TRAP: + /* Nothing was marshalled for this trap, see sync_hyp_vcpu(). */ + if (ARM_SERROR_PENDING(hyp_vcpu->exit_code)) + break; + + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + esr_ec = ESR_ELx_EC(kvm_vcpu_get_esr(&hyp_vcpu->vcpu)); + ec_handler = entry_hyp_pvm_handlers[esr_ec]; + if (ec_handler) + ec_handler(hyp_vcpu); + } + break; + default: + BUG(); + } + + hyp_vcpu->exit_code = 0; } -static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) +static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, u32 exit_reason) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + hyp_entry_exit_handler_fn ec_handler; + u8 esr_ec; fpsimd_sve_sync(&hyp_vcpu->vcpu); sync_debug_state(hyp_vcpu); if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { - host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt; + /* + * Protected: the host sees ESR_EL2 as EL2 took it, register + * index withheld; the fault addresses stay withheld unless the + * EC handler below adds them. + */ + host_vcpu->arch.fault = (struct kvm_vcpu_fault_info) { + .esr_el2 = pvm_host_esr(hyp_vcpu->vcpu.arch.fault.esr_el2), + .disr_el1 = hyp_vcpu->vcpu.arch.fault.disr_el1, + }; } else { + /* Non-protected: the host gets the full fault. */ + host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; + host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; /* * PC feeds trace_kvm_exit(), PSTATE.SS the host software-step * machine, and both run before the next on-demand ctxt sync. @@ -273,18 +785,45 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate; } - host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; + switch (ARM_EXCEPTION_CODE(exit_reason)) { + case ARM_EXCEPTION_IRQ: + break; + case ARM_EXCEPTION_TRAP: + /* SError pending: not handled at EL2, the guest replays it. */ + if (ARM_SERROR_PENDING(exit_reason)) + break; - host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; + /* Per-EC marshalling is for protected guests only. */ + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + esr_ec = ESR_ELx_EC(kvm_vcpu_get_esr(&hyp_vcpu->vcpu)); + ec_handler = exit_hyp_pvm_handlers[esr_ec]; + if (ec_handler) + ec_handler(hyp_vcpu); + } + break; + case ARM_EXCEPTION_EL1_SERROR: + case ARM_EXCEPTION_IL: + break; + default: + BUG(); + } + + /* Cleared by hardware once the guest takes the vSError. */ + host_vcpu->arch.hcr_el2 &= ~HCR_VSE; + host_vcpu->arch.hcr_el2 |= hyp_vcpu->vcpu.arch.hcr_el2 & HCR_VSE; sync_hyp_vgic_state(hyp_vcpu); + sync_hyp_timer_state(hyp_vcpu); + + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + vcpu_clear_flag(host_vcpu, PC_UPDATE_REQ); + + hyp_vcpu->exit_code = exit_reason; } -static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __pkvm_vcpu_load, + pkvm_handle_t, handle, unsigned int, vcpu_idx, u64, hcr_el2) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(unsigned int, vcpu_idx, host_ctxt, 2); - DECLARE_REG(u64, hcr_el2, host_ctxt, 3); struct pkvm_hyp_vcpu *hyp_vcpu; hyp_vcpu = pkvm_load_hyp_vcpu(handle, vcpu_idx); @@ -292,16 +831,24 @@ static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt) return; if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + u64 dfr0 = read_sysreg(id_aa64dfr0_el1); + /* Propagate WFx trapping flags */ hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWE | HCR_TWI); hyp_vcpu->vcpu.arch.hcr_el2 |= hcr_el2 & (HCR_TWE | HCR_TWI); + + /* HPMN == 0 is reserved without FEAT_HPMN0. */ + if (pmuv3_implemented(SYS_FIELD_GET(ID_AA64DFR0_EL1, PMUVer, dfr0))) + u64p_replace_bits(&hyp_vcpu->vcpu.arch.mdcr_el2, + FIELD_GET(ARMV8_PMU_PMCR_N, read_sysreg(pmcr_el0)), + MDCR_EL2_HPMN); } else { memcpy(&hyp_vcpu->vcpu.arch.fgt, hyp_vcpu->host_vcpu->arch.fgt, sizeof(hyp_vcpu->vcpu.arch.fgt)); } } -static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __pkvm_vcpu_put) { struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); @@ -317,7 +864,7 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt) } } -static void handle___pkvm_vcpu_sync_state(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __pkvm_vcpu_sync_state) { struct pkvm_hyp_vcpu *hyp_vcpu; @@ -328,10 +875,9 @@ static void handle___pkvm_vcpu_sync_state(struct kvm_cpu_context *host_ctxt) sync_hyp_vcpu_state(hyp_vcpu); } -static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg, +static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *host_vcpu, struct pkvm_hyp_vcpu **hyp_vcpup) { - struct kvm_vcpu *host_vcpu = kern_hyp_va(arg); struct pkvm_hyp_vcpu *hyp_vcpu = NULL; if (unlikely(is_protected_kvm_enabled())) { @@ -347,34 +893,26 @@ static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg, return host_vcpu; } -#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup) \ - ({ \ - DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr); \ - __get_host_hyp_vcpus(__vcpu, hyp_vcpup); \ - }) +static struct kvm_vcpu * +__get_host_hyp_vcpus_from_vgic_v3_cpu_if(struct vgic_v3_cpu_if __kern *cpu_if, + struct pkvm_hyp_vcpu **hyp_vcpup) +{ + struct vgic_v3_cpu_if *host_cpu_if = kern_hyp_va_host(cpu_if); + struct kvm_vcpu *host_vcpu = container_of(host_cpu_if, struct kvm_vcpu, + arch.vgic_cpu.vgic_v3); -#define get_host_hyp_vcpus_from_vgic_v3_cpu_if(ctxt, regnr, hyp_vcpup) \ - ({ \ - DECLARE_REG(struct vgic_v3_cpu_if *, cif, ctxt, regnr);\ - struct kvm_vcpu *__vcpu = container_of(cif, \ - struct kvm_vcpu, \ - arch.vgic_cpu.vgic_v3); \ - \ - __get_host_hyp_vcpus(__vcpu, hyp_vcpup); \ - }) + return __get_host_hyp_vcpus(host_vcpu, hyp_vcpup); +} -static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run, struct kvm_vcpu __kern *, vcpu) { struct pkvm_hyp_vcpu *hyp_vcpu; struct kvm_vcpu *host_vcpu; - int ret; + int ret = ARM_EXCEPTION_IL; - host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu); - - if (!host_vcpu) { - ret = -EINVAL; - goto out; - } + host_vcpu = __get_host_hyp_vcpus(kern_hyp_va_host(vcpu), &hyp_vcpu); + if (!host_vcpu) + return ret; if (unlikely(hyp_vcpu)) { /* @@ -383,213 +921,245 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) * loading a vcpu. Therefore, if SME features enabled the host * is misbehaving. */ - if (unlikely(system_supports_sme() && read_sysreg_s(SYS_SVCR))) { - ret = -EINVAL; - goto out; + if (unlikely(system_supports_sme() && read_sysreg_s(SYS_SVCR))) + return ret; + + /* + * ON has a single writer, pkvm_reset_vcpu() on this CPU, so + * READ_ONCE suffices. ON_PENDING takes the reset; -ECANCELED + * is a rollback that raced it. + */ + switch (READ_ONCE(hyp_vcpu->power_state)) { + case PSCI_0_2_AFFINITY_LEVEL_ON: + break; + case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING: + if (pkvm_reset_vcpu(hyp_vcpu)) + return ret; + break; + default: + return ret; } flush_hyp_vcpu(hyp_vcpu); ret = __kvm_vcpu_run(&hyp_vcpu->vcpu); - sync_hyp_vcpu(hyp_vcpu); + sync_hyp_vcpu(hyp_vcpu, ret); } else { /* The host is fully trusted, run its vCPU directly. */ fpsimd_lazy_switch_to_guest(host_vcpu); ret = __kvm_vcpu_run(host_vcpu); fpsimd_lazy_switch_to_host(host_vcpu); } -out: - cpu_reg(host_ctxt, 1) = ret; + + return ret; } static int pkvm_refill_memcache(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; - return refill_memcache(&hyp_vcpu->vcpu.arch.pkvm_memcache, - host_vcpu->arch.pkvm_memcache.nr_pages, - &host_vcpu->arch.pkvm_memcache); + return refill_memcache(&hyp_vcpu->vcpu.arch.stage2_mc, + host_vcpu->arch.stage2_mc.nr_pages, + &host_vcpu->arch.stage2_mc); } -static void handle___pkvm_host_donate_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_donate_guest, + u64, pfn, u64, gfn) { - DECLARE_REG(u64, pfn, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); struct pkvm_hyp_vcpu *hyp_vcpu; - int ret = -EINVAL; + int ret; hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); if (!hyp_vcpu || !pkvm_hyp_vcpu_is_protected(hyp_vcpu)) - goto out; + return -EINVAL; ret = pkvm_refill_memcache(hyp_vcpu); if (ret) - goto out; + return ret; - ret = __pkvm_host_donate_guest(pfn, gfn, hyp_vcpu); -out: - cpu_reg(host_ctxt, 1) = ret; + return __pkvm_host_donate_guest(pfn, gfn, hyp_vcpu); } -static void handle___pkvm_host_share_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_share_guest, + u64, pfn, u64, gfn, u64, nr_pages, u64, prot) { - DECLARE_REG(u64, pfn, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); - DECLARE_REG(u64, nr_pages, host_ctxt, 3); - DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 4); struct pkvm_hyp_vcpu *hyp_vcpu; - int ret = -EINVAL; + int ret; hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); if (!hyp_vcpu || pkvm_hyp_vcpu_is_protected(hyp_vcpu)) - goto out; + return -EINVAL; ret = pkvm_refill_memcache(hyp_vcpu); if (ret) - goto out; + return ret; - ret = __pkvm_host_share_guest(pfn, gfn, nr_pages, hyp_vcpu, prot); -out: - cpu_reg(host_ctxt, 1) = ret; + return __pkvm_host_share_guest(pfn, gfn, nr_pages, hyp_vcpu, prot); } -static void handle___pkvm_host_unshare_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_unshare_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); - DECLARE_REG(u64, nr_pages, host_ctxt, 3); struct pkvm_hyp_vm *hyp_vm; - int ret = -EINVAL; + int ret; hyp_vm = get_np_pkvm_hyp_vm(handle); if (!hyp_vm) - goto out; + return -EINVAL; ret = __pkvm_host_unshare_guest(gfn, nr_pages, hyp_vm); put_pkvm_hyp_vm(hyp_vm); -out: - cpu_reg(host_ctxt, 1) = ret; + + return ret; } -static void handle___pkvm_host_relax_perms_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_relax_perms_guest, + u64, gfn, u64, prot) { - DECLARE_REG(u64, gfn, host_ctxt, 1); - DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 2); struct pkvm_hyp_vcpu *hyp_vcpu; - int ret = -EINVAL; hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); if (!hyp_vcpu || pkvm_hyp_vcpu_is_protected(hyp_vcpu)) - goto out; + return -EINVAL; - ret = __pkvm_host_relax_perms_guest(gfn, hyp_vcpu, prot); -out: - cpu_reg(host_ctxt, 1) = ret; + return __pkvm_host_relax_perms_guest(gfn, hyp_vcpu, prot); } -static void handle___pkvm_host_wrprotect_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_wrprotect_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); - DECLARE_REG(u64, nr_pages, host_ctxt, 3); struct pkvm_hyp_vm *hyp_vm; - int ret = -EINVAL; + int ret; hyp_vm = get_np_pkvm_hyp_vm(handle); if (!hyp_vm) - goto out; + return -EINVAL; ret = __pkvm_host_wrprotect_guest(gfn, nr_pages, hyp_vm); put_pkvm_hyp_vm(hyp_vm); -out: - cpu_reg(host_ctxt, 1) = ret; + + return ret; } -static void handle___pkvm_host_test_clear_young_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_test_clear_young_guest, + pkvm_handle_t, handle, u64, gfn, u64, nr_pages, bool, mkold) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); - DECLARE_REG(u64, nr_pages, host_ctxt, 3); - DECLARE_REG(bool, mkold, host_ctxt, 4); struct pkvm_hyp_vm *hyp_vm; - int ret = -EINVAL; + int ret; hyp_vm = get_np_pkvm_hyp_vm(handle); if (!hyp_vm) - goto out; + return -EINVAL; ret = __pkvm_host_test_clear_young_guest(gfn, nr_pages, mkold, hyp_vm); put_pkvm_hyp_vm(hyp_vm); -out: - cpu_reg(host_ctxt, 1) = ret; + + return ret; } -static void handle___pkvm_host_mkyoung_guest(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_mkyoung_guest, + u64, gfn) { - DECLARE_REG(u64, gfn, host_ctxt, 1); struct pkvm_hyp_vcpu *hyp_vcpu; - int ret = -EINVAL; hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); if (!hyp_vcpu || pkvm_hyp_vcpu_is_protected(hyp_vcpu)) - goto out; + return -EINVAL; - ret = __pkvm_host_mkyoung_guest(gfn, hyp_vcpu); -out: - cpu_reg(host_ctxt, 1) = ret; + return __pkvm_host_mkyoung_guest(gfn, hyp_vcpu); } -static void handle___kvm_adjust_pc(struct kvm_cpu_context *host_ctxt) +/* + * PKVM_HOST_STATE_DIRTY names the authoritative copy, the host's when set. + * A loaded protected vCPU takes the request at its next entry instead. + */ +struct kvm_vcpu *kvm_adjust_pc_get(struct kvm_vcpu *vcpu) { - DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1); + struct pkvm_hyp_vcpu *hyp_vcpu; + + if (!is_protected_kvm_enabled()) + return vcpu; + + hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); + if (!hyp_vcpu || vcpu == &hyp_vcpu->vcpu) + return vcpu; - __kvm_adjust_pc(kern_hyp_va(vcpu)); + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return NULL; + + if (vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY)) + return vcpu; + + vcpu_copy_flag(&hyp_vcpu->vcpu, vcpu, PC_UPDATE_REQ); + return &hyp_vcpu->vcpu; } -static void handle___kvm_flush_vm_context(struct kvm_cpu_context *host_ctxt) +/* Reflect the consumed request back, otherwise it stays pending. */ +void kvm_adjust_pc_put(struct kvm_vcpu *vcpu, struct kvm_vcpu *target) { - __kvm_flush_vm_context(); + if (target != vcpu) + vcpu_copy_flag(vcpu, target, PC_UPDATE_REQ); } -static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __kvm_adjust_pc, + struct kvm_vcpu __kern *, vcpu) { - DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); - DECLARE_REG(phys_addr_t, ipa, host_ctxt, 2); - DECLARE_REG(int, level, host_ctxt, 3); + struct pkvm_hyp_vcpu *hyp_vcpu; + struct kvm_vcpu *host_vcpu; - __kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level); + host_vcpu = __get_host_hyp_vcpus(kern_hyp_va_host(vcpu), &hyp_vcpu); + if (host_vcpu) { + __kvm_adjust_pc(host_vcpu); + return; + } + + /* + * With no hyp vCPU loaded for it, the host vCPU may be unpinned, + * and so unmapped at EL2: its first run pins it. A pin fails only + * for memory the host isn't sharing, a bad pointer, so the request + * is dropped. + */ + host_vcpu = kern_hyp_va_host(vcpu); + if (hyp_pin_shared_mem(host_vcpu, host_vcpu + 1)) + return; + + __kvm_adjust_pc(host_vcpu); + hyp_unpin_shared_mem(host_vcpu, host_vcpu + 1); } -static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __kvm_flush_vm_context) { - DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); - DECLARE_REG(phys_addr_t, ipa, host_ctxt, 2); - DECLARE_REG(int, level, host_ctxt, 3); - - __kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level); + __kvm_flush_vm_context(); } -static void -handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_ipa, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, ipa, int, level) { - DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); - DECLARE_REG(phys_addr_t, start, host_ctxt, 2); - DECLARE_REG(unsigned long, pages, host_ctxt, 3); + __kvm_tlb_flush_vmid_ipa(kern_hyp_va_host(mmu), ipa, level); +} - __kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages); +DEFINE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_ipa_nsh, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, ipa, int, level) +{ + __kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va_host(mmu), ipa, level); } -static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid_range, + struct kvm_s2_mmu __kern *, mmu, phys_addr_t, start, unsigned long, pages) { - DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + __kvm_tlb_flush_vmid_range(kern_hyp_va_host(mmu), start, pages); +} - __kvm_tlb_flush_vmid(kern_hyp_va(mmu)); +DEFINE_KVM_HOST_HCALL(void, __kvm_tlb_flush_vmid, + struct kvm_s2_mmu __kern *, mmu) +{ + __kvm_tlb_flush_vmid(kern_hyp_va_host(mmu)); } -static void handle___pkvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __pkvm_tlb_flush_vmid, + pkvm_handle_t, handle) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); struct pkvm_hyp_vm *hyp_vm = get_np_pkvm_hyp_vm(handle); if (!hyp_vm) @@ -599,19 +1169,19 @@ static void handle___pkvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) put_pkvm_hyp_vm(hyp_vm); } -static void handle___kvm_flush_cpu_context(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __kvm_flush_cpu_context, + struct kvm_s2_mmu __kern *, mmu) { - DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); - - __kvm_flush_cpu_context(kern_hyp_va(mmu)); + __kvm_flush_cpu_context(kern_hyp_va_host(mmu)); } -static void handle___kvm_timer_set_cntvoff(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __kvm_timer_set_cntvoff, + u64, cntvoff) { - __kvm_timer_set_cntvoff(cpu_reg(host_ctxt, 1)); + __kvm_timer_set_cntvoff(cntvoff); } -static void handle___kvm_enable_ssbs(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __kvm_enable_ssbs) { u64 tmp; @@ -620,23 +1190,23 @@ static void handle___kvm_enable_ssbs(struct kvm_cpu_context *host_ctxt) write_sysreg_el2(tmp, SYS_SCTLR); } -static void handle___vgic_v3_get_gic_config(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(u64, __vgic_v3_get_gic_config) { - cpu_reg(host_ctxt, 1) = __vgic_v3_get_gic_config(); + return __vgic_v3_get_gic_config(); } -static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __vgic_v3_init_lrs) { __vgic_v3_init_lrs(); } -static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __vgic_v3_save_aprs, + struct vgic_v3_cpu_if __kern *, cpu_if) { struct pkvm_hyp_vcpu *hyp_vcpu; struct kvm_vcpu *host_vcpu; - host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1, - &hyp_vcpu); + host_vcpu = __get_host_hyp_vcpus_from_vgic_v3_cpu_if(cpu_if, &hyp_vcpu); if (!host_vcpu) return; @@ -658,13 +1228,13 @@ static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt) } } -static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __vgic_v3_restore_vmcr_aprs, + struct vgic_v3_cpu_if __kern *, cpu_if) { struct pkvm_hyp_vcpu *hyp_vcpu; struct kvm_vcpu *host_vcpu; - host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1, - &hyp_vcpu); + host_vcpu = __get_host_hyp_vcpus_from_vgic_v3_cpu_if(cpu_if, &hyp_vcpu); if (!host_vcpu) return; @@ -691,48 +1261,39 @@ static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt } } -static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_init, + phys_addr_t, phys, unsigned long, size, + unsigned long *, per_cpu_base, u32, hyp_va_bits) { - DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); - DECLARE_REG(unsigned long, size, host_ctxt, 2); - DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 3); - DECLARE_REG(u32, hyp_va_bits, host_ctxt, 4); - /* * __pkvm_init() will return only if an error occurred, otherwise it * will tail-call in __pkvm_init_finalise() which will have to deal * with the host context directly. */ - cpu_reg(host_ctxt, 1) = __pkvm_init(phys, size, per_cpu_base, hyp_va_bits); + return __pkvm_init(phys, size, per_cpu_base, hyp_va_bits); } -static void handle___pkvm_cpu_set_vector(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_cpu_set_vector, + enum arm64_hyp_spectre_vector, slot) { - DECLARE_REG(enum arm64_hyp_spectre_vector, slot, host_ctxt, 1); - - cpu_reg(host_ctxt, 1) = pkvm_cpu_set_vector(slot); + return pkvm_cpu_set_vector(slot); } -static void handle___pkvm_host_share_hyp(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_share_hyp, + u64, pfn) { - DECLARE_REG(u64, pfn, host_ctxt, 1); - - cpu_reg(host_ctxt, 1) = __pkvm_host_share_hyp(pfn); + return __pkvm_host_share_hyp(pfn); } -static void handle___pkvm_host_unshare_hyp(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_host_unshare_hyp, + u64, pfn) { - DECLARE_REG(u64, pfn, host_ctxt, 1); - - cpu_reg(host_ctxt, 1) = __pkvm_host_unshare_hyp(pfn); + return __pkvm_host_unshare_hyp(pfn); } -static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(ulong, __pkvm_create_private_mapping, + phys_addr_t, phys, size_t, size, u64, prot) { - DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); - DECLARE_REG(size_t, size, host_ctxt, 2); - DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); - /* * __pkvm_create_private_mapping() populates a pointer with the * hypervisor start address of the allocation. @@ -743,161 +1304,237 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct * Instead pass the allocation address as the return value (or return * ERR_PTR() on failure). */ - unsigned long haddr; + ulong haddr; int err = __pkvm_create_private_mapping(phys, size, prot, &haddr); if (err) - haddr = (unsigned long)ERR_PTR(err); + haddr = (ulong)ERR_PTR(err); - cpu_reg(host_ctxt, 1) = haddr; + return haddr; } -static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(int, __pkvm_prot_finalize) { - cpu_reg(host_ctxt, 1) = __pkvm_prot_finalize(); + return __pkvm_prot_finalize(); } -static void handle___pkvm_reserve_vm(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(int, __pkvm_reserve_vm) { - cpu_reg(host_ctxt, 1) = __pkvm_reserve_vm(); + return __pkvm_reserve_vm(); } -static void handle___pkvm_unreserve_vm(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __pkvm_unreserve_vm, + pkvm_handle_t, handle) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - __pkvm_unreserve_vm(handle); } -static void handle___pkvm_init_vm(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_init_vm, + struct kvm __kern *, host_kvm, void __kern *, pgd_hva) { - DECLARE_REG(struct kvm *, host_kvm, host_ctxt, 1); - DECLARE_REG(unsigned long, vm_hva, host_ctxt, 2); - DECLARE_REG(unsigned long, pgd_hva, host_ctxt, 3); - - host_kvm = kern_hyp_va(host_kvm); - cpu_reg(host_ctxt, 1) = __pkvm_init_vm(host_kvm, vm_hva, pgd_hva); + return errno_to_smccc(__pkvm_init_vm(kern_hyp_va_host(host_kvm), pgd_hva)); } -static void handle___pkvm_init_vcpu(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_init_vcpu, + pkvm_handle_t, handle, struct kvm_vcpu __kern *, host_vcpu) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 2); - DECLARE_REG(unsigned long, vcpu_hva, host_ctxt, 3); - - host_vcpu = kern_hyp_va(host_vcpu); - cpu_reg(host_ctxt, 1) = __pkvm_init_vcpu(handle, host_vcpu, vcpu_hva); + return errno_to_smccc(__pkvm_init_vcpu(handle, kern_hyp_va_host(host_vcpu))); } -static void handle___pkvm_vcpu_in_poison_fault(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(int, __pkvm_vcpu_in_poison_fault) { - int ret; struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); - ret = hyp_vcpu ? __pkvm_vcpu_in_poison_fault(hyp_vcpu) : -EINVAL; - cpu_reg(host_ctxt, 1) = ret; + return hyp_vcpu ? __pkvm_vcpu_in_poison_fault(hyp_vcpu) : -EINVAL; } -static void handle___pkvm_force_reclaim_guest_page(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_force_reclaim_guest_page, + phys_addr_t, phys) { - DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + return __pkvm_host_force_reclaim_page_guest(phys); +} - cpu_reg(host_ctxt, 1) = __pkvm_host_force_reclaim_page_guest(phys); +DEFINE_KVM_HOST_HCALL(int, __pkvm_reclaim_dying_guest_page, + pkvm_handle_t, handle, u64, gfn) +{ + return __pkvm_reclaim_dying_guest_page(handle, gfn); } -static void handle___pkvm_reclaim_dying_guest_page(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_start_teardown_vm, + pkvm_handle_t, handle) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); - DECLARE_REG(u64, gfn, host_ctxt, 2); + return __pkvm_start_teardown_vm(handle); +} - cpu_reg(host_ctxt, 1) = __pkvm_reclaim_dying_guest_page(handle, gfn); +DEFINE_KVM_HOST_HCALL(int, __pkvm_finalize_teardown_vm, + pkvm_handle_t, handle) +{ + return __pkvm_finalize_teardown_vm(handle); } -static void handle___pkvm_start_teardown_vm(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(int, __pkvm_hyp_alloc_selftest) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); + struct pkvm_hyp_req req = { .type = PKVM_HYP_NO_REQ }; + int ret = -EPERM; + +#ifdef CONFIG_NVHE_EL2_DEBUG + ret = hyp_allocator_selftest(); + if (ret == -ENOMEM) { + req.type = PKVM_HYP_REQ_HYP_ALLOC_SELFTEST; + req.mem.nr_pages = hyp_alloc_selftest_topup_needed(); + } +#endif + pkvm_hyp_req_to_smccc(host_data_ptr(host_ctxt), &req); - cpu_reg(host_ctxt, 1) = __pkvm_start_teardown_vm(handle); + return ret; } -static void handle___pkvm_finalize_teardown_vm(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_hyp_topup, + enum pkvm_topup_id, id, phys_addr_t, head, unsigned long, nr_pages) { - DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1); + struct kvm_cpu_context *host_ctxt = host_data_ptr(host_ctxt); + struct kvm_hyp_memcache host_mc = { + .head = head, + .nr_pages = nr_pages, + }; + int ret; + + switch (id) { + case PKVM_TOPUP_HYP_ALLOC: + ret = hyp_alloc_topup(&host_mc); + break; + case PKVM_TOPUP_HYP_ALLOC_SELFTEST: + ret = hyp_alloc_selftest_topup(&host_mc); + break; + default: + ret = -EINVAL; + } - cpu_reg(host_ctxt, 1) = __pkvm_finalize_teardown_vm(handle); + cpu_reg(host_ctxt, 2) = host_mc.head; + cpu_reg(host_ctxt, 3) = host_mc.nr_pages; + + return ret; } -static void handle___tracing_load(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __pkvm_hyp_reclaim, + enum pkvm_topup_id, id, unsigned long, target) { - DECLARE_REG(unsigned long, desc_hva, host_ctxt, 1); - DECLARE_REG(size_t, desc_size, host_ctxt, 2); + struct kvm_cpu_context *host_ctxt = host_data_ptr(host_ctxt); + struct kvm_hyp_memcache host_mc = {}; + int ret = 0; - cpu_reg(host_ctxt, 1) = __tracing_load(desc_hva, desc_size); + switch (id) { + case PKVM_TOPUP_HYP_ALLOC: + hyp_alloc_reclaim(&host_mc, target); + break; + case PKVM_TOPUP_HYP_ALLOC_SELFTEST: + hyp_alloc_selftest_reclaim(&host_mc, target); + break; + default: + ret = -EINVAL; + } + + cpu_reg(host_ctxt, 2) = host_mc.head; + cpu_reg(host_ctxt, 3) = host_mc.nr_pages; + + return ret; } -static void handle___tracing_unload(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(ulong, __pkvm_hyp_reclaimable, + enum pkvm_topup_id, id) { - __tracing_unload(); + switch (id) { + case PKVM_TOPUP_HYP_ALLOC: + return hyp_alloc_reclaimable(); + default: + return 0; + } } -static void handle___tracing_enable(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __tracing_load, + void __kern *, desc_hva, size_t, desc_size) { - DECLARE_REG(bool, enable, host_ctxt, 1); - - cpu_reg(host_ctxt, 1) = __tracing_enable(enable); + return errno_to_smccc(__tracing_load(desc_hva, desc_size)); } -static void handle___tracing_swap_reader(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL0(void, __tracing_unload) { - DECLARE_REG(unsigned int, cpu, host_ctxt, 1); + __tracing_unload(); +} - cpu_reg(host_ctxt, 1) = __tracing_swap_reader(cpu); +DEFINE_KVM_HOST_HCALL(int, __tracing_enable, + bool, enable) +{ + return __tracing_enable(enable); } -static void handle___tracing_update_clock(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __tracing_swap_reader, + unsigned int, cpu) { - DECLARE_REG(u32, mult, host_ctxt, 1); - DECLARE_REG(u32, shift, host_ctxt, 2); - DECLARE_REG(u64, epoch_ns, host_ctxt, 3); - DECLARE_REG(u64, epoch_cyc, host_ctxt, 4); + return __tracing_swap_reader(cpu); +} +DEFINE_KVM_HOST_HCALL(void, __tracing_update_clock, + u32, mult, u32, shift, u64, epoch_ns, u64, epoch_cyc) +{ __tracing_update_clock(mult, shift, epoch_ns, epoch_cyc); } -static void handle___tracing_reset(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(int, __tracing_reset, + unsigned int, cpu) { - DECLARE_REG(unsigned int, cpu, host_ctxt, 1); + return __tracing_reset(cpu); +} - cpu_reg(host_ctxt, 1) = __tracing_reset(cpu); +DEFINE_KVM_HOST_HCALL(int, __tracing_enable_event, + unsigned short, id, bool, enable) +{ + return __tracing_enable_event(id, enable); } -static void handle___tracing_enable_event(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __tracing_write_event, + u64, id) { - DECLARE_REG(unsigned short, id, host_ctxt, 1); - DECLARE_REG(bool, enable, host_ctxt, 2); + trace_selftest(id); +} - cpu_reg(host_ctxt, 1) = __tracing_enable_event(id, enable); +DEFINE_KVM_HOST_HCALL(void, __vgic_v5_make_resident, + struct vgic_v5_cpu_if __kern *, cpu_if) +{ + if (unlikely(is_protected_kvm_enabled())) + return; + + __vgic_v5_make_resident(kern_hyp_va_host(cpu_if)); } -static void handle___tracing_write_event(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __vgic_v5_make_non_resident, + struct vgic_v5_cpu_if __kern *, cpu_if) { - DECLARE_REG(u64, id, host_ctxt, 1); + if (unlikely(is_protected_kvm_enabled())) + return; - trace_selftest(id); + __vgic_v5_make_non_resident(kern_hyp_va_host(cpu_if)); } -static void handle___vgic_v5_save_apr(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __vgic_v5_save_apr, + struct vgic_v5_cpu_if __kern *, cpu_if) { - DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1); + __vgic_v5_save_apr(kern_hyp_va_host(cpu_if)); +} - __vgic_v5_save_apr(kern_hyp_va(cpu_if)); +DEFINE_KVM_HOST_HCALL(void, __vgic_v5_restore_vmcr_apr, + struct vgic_v5_cpu_if __kern *, cpu_if) +{ + __vgic_v5_restore_vmcr_apr(kern_hyp_va_host(cpu_if)); } -static void handle___vgic_v5_restore_vmcr_apr(struct kvm_cpu_context *host_ctxt) +DEFINE_KVM_HOST_HCALL(void, __vgic_v5_vdpend, + u32, intid, bool, pending, u16, vm) { - DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1); + if (unlikely(is_protected_kvm_enabled())) + return; - __vgic_v5_restore_vmcr_apr(kern_hyp_va(cpu_if)); + __vgic_v5_vdpend(intid, pending, vm); } typedef void (*hcall_t)(struct kvm_cpu_context *); @@ -912,6 +1549,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_enable_ssbs), HANDLE_FUNC(__vgic_v3_init_lrs), HANDLE_FUNC(__vgic_v3_get_gic_config), + HANDLE_FUNC(__pkvm_hyp_alloc_selftest), HANDLE_FUNC(__pkvm_prot_finalize), HANDLE_FUNC(__kvm_adjust_pc), @@ -933,11 +1571,18 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__tracing_write_event), HANDLE_FUNC(__vgic_v3_save_aprs), HANDLE_FUNC(__vgic_v3_restore_vmcr_aprs), + HANDLE_FUNC(__vgic_v5_make_resident), + HANDLE_FUNC(__vgic_v5_make_non_resident), + HANDLE_FUNC(__vgic_v5_vdpend), HANDLE_FUNC(__vgic_v5_save_apr), HANDLE_FUNC(__vgic_v5_restore_vmcr_apr), + HANDLE_FUNC(__pkvm_hyp_topup), + HANDLE_FUNC(__pkvm_hyp_reclaim), + HANDLE_FUNC(__pkvm_hyp_reclaimable), HANDLE_FUNC(__pkvm_host_share_hyp), HANDLE_FUNC(__pkvm_host_unshare_hyp), + HANDLE_FUNC(__pkvm_host_donate_guest), HANDLE_FUNC(__pkvm_host_share_guest), HANDLE_FUNC(__pkvm_host_unshare_guest), diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index 7a02837203d1..89222425359a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -17,7 +17,6 @@ SECTIONS { HYP_SECTION(.data..ro_after_init) HYP_SECTION(.rodata) #ifdef CONFIG_NVHE_EL2_TRACING - . = ALIGN(PAGE_SIZE); BEGIN_HYP_SECTION(.event_ids) *(SORT(.hyp.event_ids.*)) END_HYP_SECTION diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index a6a47c1e058b..c3035aa5a470 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -540,7 +540,7 @@ static int host_stage2_adjust_range(u64 addr, struct kvm_mem_range *range) int ret; hyp_assert_lock_held(&host_mmu.lock); - ret = kvm_pgtable_get_leaf(&host_mmu.pgt, addr, &pte, &level); + ret = kvm_pgtable_get_leaf(&host_mmu.pgt, addr, &pte, &level, 0); if (ret) return ret; @@ -930,7 +930,7 @@ static int get_valid_guest_pte(struct pkvm_hyp_vm *vm, u64 ipa, kvm_pte_t *ptep, s8 level; int ret; - ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level); + ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0); if (ret) return ret; if (guest_pte_is_poisoned(pte)) @@ -979,7 +979,7 @@ int __pkvm_vcpu_in_poison_fault(struct pkvm_hyp_vcpu *hyp_vcpu) ipa |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(&hyp_vcpu->vcpu)); guest_lock_component(vm); - ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level); + ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0); if (ret) goto unlock; @@ -1043,7 +1043,7 @@ int __pkvm_guest_share_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn) ret = 0; WARN_ON(kvm_pgtable_stage2_map(&vm->pgt, ipa, PAGE_SIZE, phys, pkvm_mkstate(KVM_PGTABLE_PROT_RWX, PKVM_PAGE_SHARED_OWNED), - &vcpu->vcpu.arch.pkvm_memcache, 0)); + &vcpu->vcpu.arch.stage2_mc, 0)); WARN_ON(__host_set_page_state_range(phys, PAGE_SIZE, PKVM_PAGE_SHARED_BORROWED)); unlock: guest_unlock_component(vm); @@ -1078,7 +1078,7 @@ int __pkvm_guest_unshare_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn) PKVM_ID_GUEST, meta)); WARN_ON(kvm_pgtable_stage2_map(&vm->pgt, ipa, PAGE_SIZE, phys, pkvm_mkstate(KVM_PGTABLE_PROT_RWX, PKVM_PAGE_OWNED), - &vcpu->vcpu.arch.pkvm_memcache, 0)); + &vcpu->vcpu.arch.stage2_mc, 0)); unlock: guest_unlock_component(vm); host_unlock_component(); @@ -1335,7 +1335,7 @@ static int host_stage2_get_guest_info(phys_addr_t phys, struct pkvm_hyp_vm **vm, return -EPERM; } - ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, &level); + ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, &level, 0); if (ret) return ret; @@ -1442,7 +1442,7 @@ static int __guest_check_pgtable_memcache(struct pkvm_hyp_vcpu *vcpu) { struct pkvm_hyp_vm *vm = pkvm_hyp_vcpu_to_hyp_vm(vcpu); - if (vcpu->vcpu.arch.pkvm_memcache.nr_pages < kvm_mmu_cache_min_pages(vm->pgt.mmu)) + if (vcpu->vcpu.arch.stage2_mc.nr_pages < kvm_mmu_cache_min_pages(vm->pgt.mmu)) return -ENOMEM; return 0; @@ -1476,7 +1476,7 @@ int __pkvm_host_donate_guest(u64 pfn, u64 gfn, struct pkvm_hyp_vcpu *vcpu) PKVM_ID_GUEST, meta)); WARN_ON(kvm_pgtable_stage2_map(&vm->pgt, ipa, PAGE_SIZE, phys, pkvm_mkstate(KVM_PGTABLE_PROT_RWX, PKVM_PAGE_OWNED), - &vcpu->vcpu.arch.pkvm_memcache, 0)); + &vcpu->vcpu.arch.stage2_mc, 0)); unlock: guest_unlock_component(vm); @@ -1547,7 +1547,7 @@ int __pkvm_host_share_guest(u64 pfn, u64 gfn, u64 nr_pages, struct pkvm_hyp_vcpu WARN_ON(kvm_pgtable_stage2_map(&vm->pgt, ipa, size, phys, pkvm_mkstate(prot, PKVM_PAGE_SHARED_BORROWED), - &vcpu->vcpu.arch.pkvm_memcache, 0)); + &vcpu->vcpu.arch.stage2_mc, 0)); unlock: guest_unlock_component(vm); @@ -1564,7 +1564,7 @@ static int __check_host_shared_guest(struct pkvm_hyp_vm *vm, u64 *__phys, u64 ip s8 level; int ret; - ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level); + ret = kvm_pgtable_get_leaf(&vm->pgt, ipa, &pte, &level, 0); if (ret) return ret; if (!kvm_pte_valid(pte)) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index 29ab5ee9d57f..b95d98cfe57d 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -91,6 +91,59 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr) return ret; } +/** + * pkvm_map_private_va_range() - Map a physical range into the private VA range + * @haddr: The virtual address in the private range. + * @phys: The physical address to map. + * @size: The size of the range to map. + * + * The hypervisor VA @haddr must have been first allocated with + * pkvm_alloc_private_va_range(). The created mapping can later be removed + * with pkvm_remove_mappings(). + * + * Return: 0 on success, negative error code on failure. + */ +int pkvm_map_private_va_range(void *haddr, phys_addr_t phys, size_t size) +{ + unsigned long addr = (unsigned long)haddr; + int ret; + + if (!PAGE_ALIGNED(addr | phys | size)) + return -EINVAL; + + guard(hyp_spinlock)(&pkvm_pgd_lock); + + if (addr < __io_map_base || addr >= __io_map_next) + return -EINVAL; + + if (size > __io_map_next - addr) + return -EINVAL; + + ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, size, phys, PAGE_HYP); + if (ret) + kvm_pgtable_hyp_unmap(&pkvm_pgtable, addr, size); + + return ret; +} + +/** + * pkvm_private_va_range_pa - Translate a private VA to a physical address. + * @va: The private virtual address to translate. + * + * Return: The physical address corresponding to @va. + */ +phys_addr_t pkvm_private_va_range_pa(void *va) +{ + kvm_pte_t pte; + s8 level; + + guard(hyp_spinlock)(&pkvm_pgd_lock); + WARN_ON(kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)va, &pte, &level, 0)); + WARN_ON(!kvm_pte_valid(pte)); + + return kvm_pte_to_phys(pte) + ((u64)va & (kvm_granule_size(level) - 1)); +} + int __pkvm_create_private_mapping(phys_addr_t phys, size_t size, enum kvm_pgtable_prot prot, unsigned long *haddr) @@ -147,6 +200,30 @@ int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) return ret; } +/** + * pkvm_remove_mappings - Remove mappings from the hypervisor page-table + * @from: The starting virtual address of the range to remove + * @to: The ending virtual address of the range to remove + * + * The range [@from, @to) can be a subset of the range previously mapped by + * pkvm_create_mappings(), as the latter enforces PTE-level mappings. However, + * it cannot be a subset of a range mapped by pkvm_map_private_va_range() as + * that function allows block mappings which the hypervisor page-table does not + * split. + */ +void pkvm_remove_mappings(void *from, void *to) +{ + u64 size; + + WARN_ON(from > to); + to = PTR_ALIGN(to, PAGE_SIZE); + from = PTR_ALIGN_DOWN(from, PAGE_SIZE); + size = (u64)to - (u64)from; + + guard(hyp_spinlock)(&pkvm_pgd_lock); + WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, (u64)from, size) != size); +} + int hyp_back_vmemmap(phys_addr_t back) { unsigned long i, start, size, end = 0; @@ -486,7 +563,7 @@ static int check_page_ownership(phys_addr_t phys) return -EPERM; } - ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL); + ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL, 0); if (ret) return ret; diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 6290c4b62659..5168321713d9 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -5,17 +5,25 @@ */ #include <kvm/arm_hypercalls.h> +#include <kvm/arm_psci.h> #include <linux/kvm_host.h> #include <linux/mm.h> #include <asm/kvm_emulate.h> +#include <asm/spectre.h> +#include <nvhe/alloc.h> #include <nvhe/mem_protect.h> #include <nvhe/memory.h> #include <nvhe/pkvm.h> #include <nvhe/trap_handler.h> +/* The host's Spectre mitigation state, for SMCCC_ARCH_FEATURES. */ +enum mitigation_state spectre_v2_state; +enum mitigation_state spectre_v4_state; +enum mitigation_state spectre_bhb_state; + /* Used by icache_is_aliasing(). */ unsigned long __icache_flags; @@ -109,7 +117,7 @@ static void pvm_init_traps_mdcr(struct kvm_vcpu *vcpu) if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, PMUVer, IMP)) { val |= MDCR_EL2_TPM | MDCR_EL2_TPMCR; - val &= ~(MDCR_EL2_HPME | MDCR_EL2_MTPME | MDCR_EL2_HPMN_MASK); + val &= ~(MDCR_EL2_HPME | MDCR_EL2_MTPME); } if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, DebugVer, IMP)) @@ -285,7 +293,7 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle, } hyp_vcpu->loaded_hyp_vcpu = this_cpu_ptr(&loaded_hyp_vcpu); - hyp_page_ref_inc(hyp_virt_to_page(hyp_vm)); + pkvm_hyp_vm_ref_inc(hyp_vm); unlock: hyp_spin_unlock(&vm_table_lock); @@ -301,7 +309,7 @@ void pkvm_put_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_spin_lock(&vm_table_lock); hyp_vcpu->loaded_hyp_vcpu = NULL; __this_cpu_write(loaded_hyp_vcpu, NULL); - hyp_page_ref_dec(hyp_virt_to_page(hyp_vm)); + pkvm_hyp_vm_ref_dec(hyp_vm); hyp_spin_unlock(&vm_table_lock); } @@ -311,6 +319,43 @@ struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu(void) } +static struct pkvm_hyp_vm *pkvm_get_loaded_hyp_vm(struct kvm_vcpu *vcpu) +{ + struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); + + if (hyp_vcpu && + (vcpu == &hyp_vcpu->vcpu || vcpu == hyp_vcpu->host_vcpu)) + return pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu); + + return NULL; +} + +/* + * A loaded vCPU's VM is the hyp VM. An unloaded host vCPU's is mapped at + * EL2 only while pinned, so it's read once and pinned; the pin fails for + * memory the host isn't sharing. + */ +struct kvm *pkvm_vcpu_get_kvm(struct kvm_vcpu *vcpu) +{ + struct pkvm_hyp_vm *hyp_vm = pkvm_get_loaded_hyp_vm(vcpu); + struct kvm *kvm; + + if (hyp_vm) + return &hyp_vm->kvm; + + kvm = kern_hyp_va(READ_ONCE(vcpu->kvm)); + if (hyp_pin_shared_mem(kvm, kvm + 1)) + return NULL; + + return kvm; +} + +void pkvm_vcpu_put_kvm(struct kvm_vcpu *vcpu, struct kvm *kvm) +{ + if (kvm && !pkvm_get_loaded_hyp_vm(vcpu)) + hyp_unpin_shared_mem(kvm, kvm + 1); +} + struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle) { struct pkvm_hyp_vm *hyp_vm; @@ -318,7 +363,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle) hyp_spin_lock(&vm_table_lock); hyp_vm = get_vm_by_handle(handle); if (hyp_vm) - hyp_page_ref_inc(hyp_virt_to_page(hyp_vm)); + pkvm_hyp_vm_ref_inc(hyp_vm); hyp_spin_unlock(&vm_table_lock); return hyp_vm; @@ -327,7 +372,7 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle) void put_pkvm_hyp_vm(struct pkvm_hyp_vm *hyp_vm) { hyp_spin_lock(&vm_table_lock); - hyp_page_ref_dec(hyp_virt_to_page(hyp_vm)); + pkvm_hyp_vm_ref_dec(hyp_vm); hyp_spin_unlock(&vm_table_lock); } @@ -373,21 +418,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_MTE)) kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_MTE_ENABLED); - bitmap_zero(allowed_features, KVM_VCPU_MAX_FEATURES); - - set_bit(KVM_ARM_VCPU_PSCI_0_2, allowed_features); - - if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PMU_V3)) - set_bit(KVM_ARM_VCPU_PMU_V3, allowed_features); - - if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_ADDRESS)) - set_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, allowed_features); - - if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_PTRAUTH_GENERIC)) - set_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, allowed_features); - - if (kvm_pkvm_ext_allowed(kvm, KVM_CAP_ARM_SVE)) - set_bit(KVM_ARM_VCPU_SVE, allowed_features); + kvm_pkvm_vcpu_allowed_features(kvm, allowed_features); bitmap_and(kvm->arch.vcpu_features, host_kvm->arch.vcpu_features, allowed_features, KVM_VCPU_MAX_FEATURES); @@ -396,6 +427,40 @@ out: kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SVE)); } +static int pkvm_vcpu_init_psci(struct pkvm_hyp_vcpu *hyp_vcpu, u32 mp_state) +{ + struct vcpu_reset_state *reset_state = &hyp_vcpu->vcpu.arch.reset_state; + struct pkvm_hyp_vm *hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu); + struct kvm_vcpu *host_vcpu; + + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + /* The host manages a non-protected vCPU: always ON at EL2. */ + hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_ON; + return 0; + } + + if (mp_state != KVM_MP_STATE_RUNNABLE && mp_state != KVM_MP_STATE_STOPPED) + return -EINVAL; + + if (mp_state == KVM_MP_STATE_STOPPED) { + reset_state->reset = false; + hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_OFF; + return 0; + } + + hyp_assert_lock_held(&vm_table_lock); + if (hyp_vm->primary_vcpu) + return -EINVAL; + hyp_vm->primary_vcpu = hyp_vcpu; + + host_vcpu = hyp_vcpu->host_vcpu; + reset_state->pc = READ_ONCE(*vcpu_pc(host_vcpu)); + reset_state->r0 = READ_ONCE(vcpu_gp_regs(host_vcpu)[0]); + reset_state->reset = true; + hyp_vcpu->power_state = PSCI_0_2_AFFINITY_LEVEL_ON_PENDING; + return 0; +} + static void unpin_host_vcpu(struct kvm_vcpu *host_vcpu) { if (host_vcpu) @@ -440,7 +505,11 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm, hyp_vm->host_kvm = host_kvm; hyp_vm->kvm.created_vcpus = nr_vcpus; - hyp_vm->kvm.arch.pkvm.is_protected = READ_ONCE(host_kvm->arch.pkvm.is_protected); + if (READ_ONCE(host_kvm->arch.vm_flavor) == VM_PROTECTED_PKVM) + hyp_vm->kvm.arch.vm_flavor = VM_PROTECTED_PKVM; + else + hyp_vm->kvm.arch.vm_flavor = VM_PKVM; + hyp_vm->kvm.arch.flags = 0; pkvm_init_features_from_host(hyp_vm, host_kvm); @@ -518,10 +587,12 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *host_vcpu) { int ret = 0; + u32 mp_state; if (hyp_pin_shared_mem(host_vcpu, host_vcpu + 1)) return -EBUSY; + mp_state = READ_ONCE(host_vcpu->arch.mp_state.mp_state); hyp_vcpu->host_vcpu = host_vcpu; hyp_vcpu->vcpu.kvm = &hyp_vm->kvm; @@ -530,7 +601,6 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, hyp_vcpu->vcpu.arch.hw_mmu = &hyp_vm->kvm.arch.mmu; hyp_vcpu->vcpu.arch.cflags = READ_ONCE(host_vcpu->arch.cflags); - hyp_vcpu->vcpu.arch.mp_state.mp_state = KVM_MP_STATE_STOPPED; if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { /* @@ -555,9 +625,17 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, goto done; ret = pkvm_vcpu_init_sve(hyp_vcpu, host_vcpu); -done: if (ret) + goto done; + + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + kvm_reset_pvm_sys_regs(&hyp_vcpu->vcpu); + ret = pkvm_vcpu_init_psci(hyp_vcpu, mp_state); +done: + if (ret) { unpin_host_vcpu(host_vcpu); + unpin_host_sve_state(hyp_vcpu); + } return ret; } @@ -661,9 +739,9 @@ static size_t pkvm_get_hyp_vm_size(unsigned int nr_vcpus) size_mul(sizeof(struct pkvm_hyp_vcpu *), nr_vcpus)); } -static void *map_donated_memory_noclear(unsigned long host_va, size_t size) +static void *map_donated_memory_noclear(void __kern *host_va, size_t size) { - void *va = (void *)kern_hyp_va(host_va); + void *va = kern_hyp_va_host(host_va); if (!PAGE_ALIGNED(va)) return NULL; @@ -675,16 +753,6 @@ static void *map_donated_memory_noclear(unsigned long host_va, size_t size) return va; } -static void *map_donated_memory(unsigned long host_va, size_t size) -{ - void *va = map_donated_memory_noclear(host_va, size); - - if (va) - memset(va, 0, size); - - return va; -} - static void __unmap_donated_memory(void *va, size_t size) { kvm_flush_dcache_to_poc(va, size); @@ -787,7 +855,7 @@ struct pkvm_hyp_vcpu *init_selftest_vm(void *virt) continue; p[i].refcount = 1; if (seeded < min_pages) { - push_hyp_memcache(&selftest_vcpu.vcpu.arch.pkvm_memcache, + push_hyp_memcache(&selftest_vcpu.vcpu.arch.stage2_mc, hyp_page_to_virt(&p[i]), hyp_virt_to_phys); seeded++; } else { @@ -814,16 +882,13 @@ void teardown_selftest_vm(void) * Unmap the donated memory from the host at stage 2. * * host_kvm: A pointer to the host's struct kvm. - * vm_hva: The host va of the area being donated for the VM state. - * Must be page aligned. * pgd_hva: The host va of the area being donated for the stage-2 PGD for * the VM. Must be page aligned. Its size is implied by the VM's * VTCR. * * Return 0 success, negative error code on failure. */ -int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva, - unsigned long pgd_hva) +int __pkvm_init_vm(struct kvm *host_kvm, void __kern *pgd_hva) { struct pkvm_hyp_vm *hyp_vm = NULL; size_t vm_size, pgd_size; @@ -851,15 +916,17 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva, vm_size = pkvm_get_hyp_vm_size(nr_vcpus); pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.mmu.vtcr); - ret = -ENOMEM; + hyp_vm = hyp_alloc(vm_size); + if (!hyp_vm) { + ret = hyp_alloc_errno(); + goto err_unpin_kvm; + } - hyp_vm = map_donated_memory(vm_hva, vm_size); - if (!hyp_vm) - goto err_remove_mappings; + ret = -ENOMEM; pgd = map_donated_memory_noclear(pgd_hva, pgd_size); if (!pgd) - goto err_remove_mappings; + goto err_free_hyp_vm; init_pkvm_hyp_vm(host_kvm, hyp_vm, nr_vcpus, handle); @@ -877,8 +944,9 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva, err_destroy_stage2: kvm_guest_destroy_stage2(hyp_vm); err_remove_mappings: - unmap_donated_memory(hyp_vm, vm_size); unmap_donated_memory(pgd, pgd_size); +err_free_hyp_vm: + hyp_free(hyp_vm); err_unpin_kvm: hyp_unpin_shared_mem(host_kvm, host_kvm + 1); return ret; @@ -913,16 +981,15 @@ static int register_hyp_vcpu(struct pkvm_hyp_vm *hyp_vm, return 0; } -int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu, - unsigned long vcpu_hva) +int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu) { struct pkvm_hyp_vcpu *hyp_vcpu; struct pkvm_hyp_vm *hyp_vm; int ret; - hyp_vcpu = map_donated_memory(vcpu_hva, sizeof(*hyp_vcpu)); + hyp_vcpu = hyp_alloc(sizeof(*hyp_vcpu)); if (!hyp_vcpu) - return -ENOMEM; + return hyp_alloc_errno(); hyp_spin_lock(&vm_table_lock); @@ -934,31 +1001,28 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu, ret = init_pkvm_hyp_vcpu(hyp_vcpu, hyp_vm, host_vcpu); if (ret) - goto unlock; + goto unclaim; ret = register_hyp_vcpu(hyp_vm, hyp_vcpu); if (ret) { unpin_host_vcpu(host_vcpu); unpin_host_sve_state(hyp_vcpu); + goto unclaim; } + goto unlock; +unclaim: + /* + * Under vm_table_lock, so no other claim can have landed: undo + * this one. + */ + if (hyp_vm->primary_vcpu == hyp_vcpu) + hyp_vm->primary_vcpu = NULL; unlock: hyp_spin_unlock(&vm_table_lock); - if (ret) - unmap_donated_memory(hyp_vcpu, sizeof(*hyp_vcpu)); - return ret; -} - -static void -teardown_donated_memory(struct kvm_hyp_memcache *mc, void *addr, size_t size) -{ - size = PAGE_ALIGN(size); - memset(addr, 0, size); - - for (void *start = addr; start < addr + size; start += PAGE_SIZE) - push_hyp_memcache(mc, start, hyp_virt_to_phys); + hyp_free(hyp_vcpu); - unmap_donated_memory_noclear(addr, size); + return ret; } int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn) @@ -983,7 +1047,7 @@ static struct pkvm_hyp_vm *get_pkvm_unref_hyp_vm_locked(pkvm_handle_t handle) hyp_assert_lock_held(&vm_table_lock); hyp_vm = get_vm_by_handle(handle); - if (!hyp_vm || hyp_page_count(hyp_vm)) + if (!hyp_vm || hyp_vm->refcount) return NULL; return hyp_vm; @@ -1010,11 +1074,10 @@ unlock: int __pkvm_finalize_teardown_vm(pkvm_handle_t handle) { - struct kvm_hyp_memcache *mc, *stage2_mc; + struct kvm_hyp_memcache *stage2_mc; struct pkvm_hyp_vm *hyp_vm; struct kvm *host_kvm; unsigned int idx; - size_t vm_size; int err; hyp_spin_lock(&vm_table_lock); @@ -1032,12 +1095,11 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle) hyp_spin_unlock(&vm_table_lock); /* Reclaim guest pages (including page-table pages) */ - mc = &host_kvm->arch.pkvm.teardown_mc; stage2_mc = &host_kvm->arch.pkvm.stage2_teardown_mc; reclaim_pgtable_pages(hyp_vm, stage2_mc); unpin_host_vcpus(hyp_vm->vcpus, hyp_vm->kvm.created_vcpus); - /* Push the metadata pages to the teardown memcache */ + /* Push the stage-2 pages to the teardown memcache */ for (idx = 0; idx < hyp_vm->kvm.created_vcpus; ++idx) { struct pkvm_hyp_vcpu *hyp_vcpu = hyp_vm->vcpus[idx]; struct kvm_hyp_memcache *vcpu_mc; @@ -1045,7 +1107,7 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle) if (!hyp_vcpu) continue; - vcpu_mc = &hyp_vcpu->vcpu.arch.pkvm_memcache; + vcpu_mc = &hyp_vcpu->vcpu.arch.stage2_mc; while (vcpu_mc->nr_pages) { void *addr = pop_hyp_memcache(vcpu_mc, hyp_phys_to_virt); @@ -1054,11 +1116,10 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle) unmap_donated_memory_noclear(addr, PAGE_SIZE); } - teardown_donated_memory(mc, hyp_vcpu, sizeof(*hyp_vcpu)); + hyp_free(hyp_vcpu); } - vm_size = pkvm_get_hyp_vm_size(hyp_vm->kvm.created_vcpus); - teardown_donated_memory(mc, hyp_vm, vm_size); + hyp_free(hyp_vm); hyp_unpin_shared_mem(host_kvm, host_kvm + 1); return 0; @@ -1133,6 +1194,238 @@ static void pkvm_memunshare_call(u64 *ret, struct kvm_vcpu *vcpu) } /* + * Reset the vCPU to its power-on state and commit ON_PENDING -> ON, on the + * target's own CPU. Returns -ECANCELED, with no side effects, if a rollback + * raced the reset. + * + * The requestor's PSCI CPU_ON, in pvm_psci_vcpu_on(): + * + * cmpxchg_relaxed(power_state): OFF -> ON_PENDING + * <ctrl> + * write_reset_state(); + * smp_store_release(reset_state->reset, true); + * + * The target coming online here, running, then its PSCI CPU_OFF in + * pvm_psci_vcpu_off(): + * + * smp_load_acquire(reset_state->reset) == true; + * cmpxchg_relaxed(power_state): ON_PENDING -> ON; + * read_reset_state(); + * + * <run guest> + * + * WRITE_ONCE(reset_state->reset, false); + * smp_store_release(power_state, OFF); + */ +int pkvm_reset_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct vcpu_reset_state *reset_state = &hyp_vcpu->vcpu.arch.reset_state; + + /* + * Pairs with smp_store_release(&reset_state->reset, true) in + * pvm_psci_vcpu_on(). The acquire must precede the cmpxchg: reversed, a + * winning cmpxchg with a false acquire would leave power_state == ON + * with the reset skipped. + */ + if (!smp_load_acquire(&reset_state->reset)) + return -ECANCELED; + + if (cmpxchg_relaxed(&hyp_vcpu->power_state, PSCI_0_2_AFFINITY_LEVEL_ON_PENDING, + PSCI_0_2_AFFINITY_LEVEL_ON) != PSCI_0_2_AFFINITY_LEVEL_ON_PENDING) + return -ECANCELED; + + kvm_reset_vcpu_core(&hyp_vcpu->vcpu); + kvm_reset_pvm_sys_regs(&hyp_vcpu->vcpu); + + /* Must be done after resetting sys registers. */ + kvm_reset_vcpu_psci(&hyp_vcpu->vcpu, reset_state); + + hyp_vcpu->exit_code = 0; + return 0; +} + +struct pkvm_hyp_vcpu *pkvm_mpidr_to_hyp_vcpu(struct pkvm_hyp_vm *hyp_vm, unsigned long mpidr) +{ + struct pkvm_hyp_vcpu *hyp_vcpu; + int i; + + mpidr &= MPIDR_HWID_BITMASK; + + for (i = 0; i < hyp_vm->kvm.created_vcpus; i++) { + /* Pairs with smp_store_release() in register_hyp_vcpu(). */ + hyp_vcpu = smp_load_acquire(&hyp_vm->vcpus[i]); + + if (!hyp_vcpu) + continue; + + if (mpidr == kvm_vcpu_get_mpidr_aff(&hyp_vcpu->vcpu)) + return hyp_vcpu; + } + + return NULL; +} + +/* + * Returns true when handled at EL2, false when the host must wake the target + * vCPU. + */ +static bool pvm_psci_vcpu_on(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct pkvm_hyp_vm *hyp_vm = pkvm_hyp_vcpu_to_hyp_vm(hyp_vcpu); + struct vcpu_reset_state *reset_state; + struct pkvm_hyp_vcpu *target; + unsigned long cpu_id, ret; + int power_state; + + cpu_id = smccc_get_arg1(&hyp_vcpu->vcpu); + if (!kvm_psci_valid_affinity(&hyp_vcpu->vcpu, cpu_id)) { + ret = PSCI_RET_INVALID_PARAMS; + goto error; + } + + target = pkvm_mpidr_to_hyp_vcpu(hyp_vm, cpu_id); + if (!target) { + ret = PSCI_RET_INVALID_PARAMS; + goto error; + } + + /* + * vCPUs race to power on the same target. The writes below depend on + * the cmpxchg reading OFF, and that control dependency with the release + * of OFF in pvm_psci_vcpu_off() orders them after the target's accesses + * to reset_state, so relaxed suffices. + */ + power_state = cmpxchg_relaxed(&target->power_state, + PSCI_0_2_AFFINITY_LEVEL_OFF, + PSCI_0_2_AFFINITY_LEVEL_ON_PENDING); + switch (power_state) { + case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING: + ret = PSCI_RET_ON_PENDING; + goto error; + case PSCI_0_2_AFFINITY_LEVEL_ON: + ret = PSCI_RET_ALREADY_ON; + goto error; + case PSCI_0_2_AFFINITY_LEVEL_OFF: + break; + default: + ret = PSCI_RET_INTERNAL_FAILURE; + goto error; + } + + reset_state = &target->vcpu.arch.reset_state; + WRITE_ONCE(reset_state->pc, smccc_get_arg2(&hyp_vcpu->vcpu)); + WRITE_ONCE(reset_state->r0, smccc_get_arg3(&hyp_vcpu->vcpu)); + WRITE_ONCE(reset_state->be, kvm_vcpu_is_be(&hyp_vcpu->vcpu)); + /* + * Publish reset_state.{pc, r0, be} to the target vCPU. Pairs with + * smp_load_acquire(&reset_state->reset) in pkvm_reset_vcpu(). + */ + smp_store_release(&reset_state->reset, true); + + /* The host requests KVM_REQ_VCPU_RESET and wakes the target. */ + return false; + +error: + smccc_set_retval(&hyp_vcpu->vcpu, ret, 0, 0, 0); + return true; +} + +/* + * Returns true when handled at EL2, false when the host must stop scheduling + * the vCPU. + */ +static bool pvm_psci_vcpu_off(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + /* No other writer runs while this vCPU is ON and executing. */ + WARN_ON(READ_ONCE(hyp_vcpu->power_state) != PSCI_0_2_AFFINITY_LEVEL_ON); + + WRITE_ONCE(hyp_vcpu->vcpu.arch.reset_state.reset, false); + + /* + * Orders pkvm_reset_vcpu()'s accesses to reset_state, and the clear + * above, before OFF. Pairs with the control dependency in + * pvm_psci_vcpu_on(). + */ + smp_store_release(&hyp_vcpu->power_state, PSCI_0_2_AFFINITY_LEVEL_OFF); + + /* Return to the host so that it can finish powering off the vcpu. */ + return false; +} + +static bool pvm_psci_version(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + /* Nothing to be handled by the host. Go back to the guest. */ + smccc_set_retval(&hyp_vcpu->vcpu, KVM_ARM_PSCI_1_1, 0, 0, 0); + return true; +} + +static bool pvm_psci_features(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + u32 feature = smccc_get_arg1(vcpu); + unsigned long val; + + switch (feature) { + case PSCI_0_2_FN_PSCI_VERSION: + case PSCI_0_2_FN_CPU_SUSPEND: + case PSCI_0_2_FN64_CPU_SUSPEND: + case PSCI_0_2_FN_CPU_OFF: + case PSCI_0_2_FN_CPU_ON: + case PSCI_0_2_FN64_CPU_ON: + case PSCI_0_2_FN_AFFINITY_INFO: + case PSCI_0_2_FN64_AFFINITY_INFO: + case PSCI_0_2_FN_SYSTEM_OFF: + case PSCI_0_2_FN_SYSTEM_RESET: + case PSCI_1_0_FN_PSCI_FEATURES: + case PSCI_1_1_FN_SYSTEM_RESET2: + case PSCI_1_1_FN64_SYSTEM_RESET2: + case ARM_SMCCC_VERSION_FUNC_ID: + val = PSCI_RET_SUCCESS; + break; + default: + val = PSCI_RET_NOT_SUPPORTED; + break; + } + + /* Nothing to be handled by the host. Go back to the guest. */ + smccc_set_retval(vcpu, val, 0, 0, 0); + return true; +} + +static bool pkvm_handle_psci(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + u32 psci_fn = smccc_get_function(vcpu); + + switch (psci_fn) { + case PSCI_0_2_FN_CPU_ON: + kvm_psci_narrow_to_32bit(vcpu); + fallthrough; + case PSCI_0_2_FN64_CPU_ON: + return pvm_psci_vcpu_on(hyp_vcpu); + case PSCI_0_2_FN_CPU_OFF: + return pvm_psci_vcpu_off(hyp_vcpu); + case PSCI_0_2_FN_PSCI_VERSION: + return pvm_psci_version(hyp_vcpu); + case PSCI_1_0_FN_PSCI_FEATURES: + return pvm_psci_features(hyp_vcpu); + case PSCI_0_2_FN_AFFINITY_INFO: + case PSCI_0_2_FN64_AFFINITY_INFO: + case PSCI_0_2_FN_SYSTEM_RESET: + case PSCI_0_2_FN_CPU_SUSPEND: + case PSCI_0_2_FN64_CPU_SUSPEND: + case PSCI_0_2_FN_SYSTEM_OFF: + case PSCI_1_1_FN_SYSTEM_RESET2: + case PSCI_1_1_FN64_SYSTEM_RESET2: + return false; /* Handled by the host. */ + default: + /* Unknown PSCI calls are handled here, not forwarded. */ + smccc_set_retval(vcpu, PSCI_RET_NOT_SUPPORTED, 0, 0, 0); + return true; + } +} + +/* * Handler for protected VM HVC calls. * * Returns true if the hypervisor has handled the exit (and control @@ -1141,10 +1434,73 @@ static void pkvm_memunshare_call(u64 *ret, struct kvm_vcpu *vcpu) */ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) { + struct pkvm_hyp_vcpu *hyp_vcpu = container_of(vcpu, struct pkvm_hyp_vcpu, vcpu); u64 val[4] = { SMCCC_RET_INVALID_PARAMETER }; bool handled = true; + u32 feature; + uuid_t uuid; switch (smccc_get_function(vcpu)) { + case ARM_SMCCC_VERSION_FUNC_ID: + /* Nothing to be handled by the host. Go back to the guest. */ + val[0] = ARM_SMCCC_VERSION_1_1; + break; + case ARM_SMCCC_ARCH_FEATURES_FUNC_ID: + /* The workaround calls themselves are handled on hyp entry. */ + val[0] = SMCCC_RET_NOT_SUPPORTED; + feature = smccc_get_arg1(vcpu); + switch (feature) { + case ARM_SMCCC_VERSION_FUNC_ID: + case ARM_SMCCC_ARCH_FEATURES_FUNC_ID: + val[0] = SMCCC_RET_SUCCESS; + break; + case ARM_SMCCC_ARCH_WORKAROUND_1: + switch (spectre_v2_state) { + case SPECTRE_VULNERABLE: + break; + case SPECTRE_MITIGATED: + val[0] = SMCCC_RET_SUCCESS; + break; + case SPECTRE_UNAFFECTED: + val[0] = SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED; + break; + } + break; + case ARM_SMCCC_ARCH_WORKAROUND_2: + switch (spectre_v4_state) { + case SPECTRE_VULNERABLE: + break; + case SPECTRE_MITIGATED: + /* With SSBS advertised the guest's default is safe. */ + if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, SSBS, IMP)) + break; + fallthrough; + case SPECTRE_UNAFFECTED: + val[0] = SMCCC_RET_NOT_REQUIRED; + break; + } + break; + case ARM_SMCCC_ARCH_WORKAROUND_3: + switch (spectre_bhb_state) { + case SPECTRE_VULNERABLE: + break; + case SPECTRE_MITIGATED: + val[0] = SMCCC_RET_SUCCESS; + break; + case SPECTRE_UNAFFECTED: + val[0] = SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED; + break; + } + break; + } + break; + case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID: + uuid = ARM_SMCCC_VENDOR_HYP_UID_KVM; + val[0] = smccc_uuid_to_reg(&uuid, 0); + val[1] = smccc_uuid_to_reg(&uuid, 1); + val[2] = smccc_uuid_to_reg(&uuid, 2); + val[3] = smccc_uuid_to_reg(&uuid, 3); + break; case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID: val[0] = BIT(ARM_SMCCC_KVM_FUNC_FEATURES); val[0] |= BIT(ARM_SMCCC_KVM_FUNC_HYP_MEMINFO); @@ -1177,8 +1533,7 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) pkvm_memunshare_call(val, vcpu); break; default: - /* Punt everything else back to the host, for now. */ - handled = false; + return pkvm_handle_psci(hyp_vcpu); } if (handled) diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 45ac5f2ba4f7..7bbc26b28faa 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -10,6 +10,7 @@ #include <asm/kvm_pgtable.h> #include <asm/kvm_pkvm.h> +#include <nvhe/alloc.h> #include <nvhe/early_alloc.h> #include <nvhe/ffa.h> #include <nvhe/gfp.h> @@ -377,6 +378,11 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas if (ret) return ret; + /* Arbitrary size, unmapped at boot and only consume VA-space */ + ret = hyp_alloc_init(SZ_128M); + if (ret) + return ret; + update_nvhe_init_params(); /* Jump in the idmap page to switch to the new page-tables */ diff --git a/arch/arm64/kvm/hyp/nvhe/stacktrace.c b/arch/arm64/kvm/hyp/nvhe/stacktrace.c index 7c832d60d22b..11fadbebbf1d 100644 --- a/arch/arm64/kvm/hyp/nvhe/stacktrace.c +++ b/arch/arm64/kvm/hyp/nvhe/stacktrace.c @@ -8,6 +8,7 @@ #include <asm/kvm_hyp.h> #include <asm/memory.h> #include <asm/percpu.h> +#include <asm/stacktrace/nvhe.h> DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack) __aligned(16); @@ -35,8 +36,6 @@ static void hyp_prepare_backtrace(unsigned long fp, unsigned long pc) } #ifdef CONFIG_PKVM_STACKTRACE -#include <asm/stacktrace/nvhe.h> - DEFINE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], pkvm_stacktrace); static struct stack_info stackinfo_get_overflow(void) diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 7318e3e6a5f3..91e1edf40bcd 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -203,15 +203,36 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops, }; +/* WFI/WFE exit to the host, which emulates them. WFxT is not advertised. */ +static bool kvm_handle_pvm_wfx(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFxT) + return kvm_handle_pvm_restricted(vcpu, exit_code); + + return false; +} + +/* Lazy FP/SIMD switch, or the UNDEF that would otherwise be left to the host. */ +static bool kvm_handle_pvm_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + if (kvm_hyp_handle_fpsimd(vcpu, exit_code)) + return true; + + return kvm_handle_pvm_restricted(vcpu, exit_code); +} + +/* + * A class not listed takes an UNDEF at EL2: the host has no way to + * inject one into a protected vCPU. + */ static const exit_handler_fn pvm_exit_handlers[] = { - [0 ... ESR_ELx_EC_MAX] = NULL, + [0 ... ESR_ELx_EC_MAX] = kvm_handle_pvm_restricted, + [ESR_ELx_EC_WFx] = kvm_handle_pvm_wfx, [ESR_ELx_EC_HVC64] = kvm_handle_pvm_hvc64, [ESR_ELx_EC_SYS64] = kvm_handle_pvm_sys64, - [ESR_ELx_EC_SVE] = kvm_handle_pvm_restricted, - [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_pvm_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, - [ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low, [ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops, }; diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c index 8758c6801776..32e82653ebf7 100644 --- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c +++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c @@ -121,6 +121,9 @@ static const struct pvm_ftr_bits pvmid_aa64mmfr0[] = { MAX_FEAT(ID_AA64MMFR0_EL1, SNSMEM, IMP), MAX_FEAT(ID_AA64MMFR0_EL1, BIGENDEL0, IMP), MAX_FEAT(ID_AA64MMFR0_EL1, EXS, IMP), + MAX_FEAT(ID_AA64MMFR0_EL1, TGRAN4, IMP), + MAX_FEAT(ID_AA64MMFR0_EL1, TGRAN16, IMP), + MAX_FEAT(ID_AA64MMFR0_EL1, TGRAN64, IMP), FEAT_END }; @@ -525,6 +528,66 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = { /* Performance Monitoring Registers are restricted. */ }; +struct sys_reg_desc_reset { + int reg; + void (*reset)(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *rd); + u64 value; +}; + +static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *r) +{ + __vcpu_assign_sys_reg(vcpu, r->reg, kvm_calculate_mpidr(vcpu)); +} + +static void reset_value(struct kvm_vcpu *vcpu, const struct sys_reg_desc_reset *r) +{ + __vcpu_assign_sys_reg(vcpu, r->reg, r->value); +} + +#define RESET_VAL(REG, VAL) { REG, reset_value, VAL } + +#define RESET_ZERO(REG) RESET_VAL(REG, 0) + +#define RESET_UNKNOWN(REG) RESET_VAL(REG, 0x1de7ec7edbadc0deULL) + +#define RESET_FUNC(REG, FN) { REG, FN, 0 } + +static const struct sys_reg_desc_reset pvm_sys_reg_reset_vals[] = { + RESET_FUNC(MPIDR_EL1, reset_mpidr), + RESET_UNKNOWN(TPIDR_EL0), + RESET_UNKNOWN(TPIDRRO_EL0), + RESET_UNKNOWN(TPIDR_EL1), + RESET_ZERO(CNTKCTL_EL1), + RESET_UNKNOWN(PAR_EL1), + RESET_ZERO(DISR_EL1), + RESET_ZERO(CPACR_EL1), + RESET_VAL(CONTEXTIDR_EL1, 0x00000000dbadc0deULL), + RESET_VAL(SCTLR_EL1, 0x00C50078ULL), + RESET_ZERO(TCR_EL1), + RESET_UNKNOWN(AFSR0_EL1), + RESET_UNKNOWN(AFSR1_EL1), + RESET_UNKNOWN(ESR_EL1), + RESET_UNKNOWN(MAIR_EL1), + RESET_ZERO(AMAIR_EL1), + RESET_ZERO(MDSCR_EL1), + RESET_ZERO(CNTV_CTL_EL0), + RESET_UNKNOWN(TTBR0_EL1), + RESET_UNKNOWN(TTBR1_EL1), + RESET_UNKNOWN(FAR_EL1), + RESET_VAL(VBAR_EL1, 0x1de7ec7edbadc000ULL), +}; + +void kvm_reset_pvm_sys_regs(struct kvm_vcpu *vcpu) +{ + unsigned long i; + + for (i = 0; i < ARRAY_SIZE(pvm_sys_reg_reset_vals); i++) { + const struct sys_reg_desc_reset *r = &pvm_sys_reg_reset_vals[i]; + + r->reset(vcpu, r); + } +} + /* * Initializes feature registers for protected vms. */ diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c index 65be6c9fb379..0d7e4db0c3e1 100644 --- a/arch/arm64/kvm/hyp/nvhe/trace.c +++ b/arch/arm64/kvm/hyp/nvhe/trace.c @@ -4,6 +4,7 @@ * Author: Vincent Donnefort <vdonnefort@google.com> */ +#include <nvhe/alloc.h> #include <nvhe/clock.h> #include <nvhe/mem_protect.h> #include <nvhe/mm.h> @@ -62,18 +63,34 @@ static void __release_host_mem(void *start, u64 size) WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(start), size >> PAGE_SHIFT)); } -static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_buffer, - struct hyp_trace_desc *desc) +static int hyp_trace_buffer_alloc_bpages(struct hyp_trace_buffer *trace_buffer, + struct hyp_trace_desc *desc) { - void *start = (void *)kern_hyp_va(desc->bpages_backing_start); - size_t size = desc->bpages_backing_size; + void *start; + size_t size; int ret; - ret = __admit_host_mem(start, size); - if (ret) - return ret; + if (is_protected_kvm_enabled()) { + struct ring_buffer_desc *rb_desc; + int cpu; + + size = 0; + for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc) + size += rb_desc->nr_page_va * sizeof(struct simple_buffer_page); - memset(start, 0, size); + start = hyp_alloc(size); + if (!start) + return hyp_alloc_errno(); + } else { + start = (void *)kern_hyp_va(desc->bpages_backing_start); + size = desc->bpages_backing_size; + + ret = __admit_host_mem(start, size); + if (ret) + return ret; + + memset(start, 0, size); + } trace_buffer->bpages_backing_start = start; trace_buffer->bpages_backing_size = size; @@ -81,7 +98,7 @@ static int hyp_trace_buffer_load_bpage_backing(struct hyp_trace_buffer *trace_bu return 0; } -static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace_buffer) +static void hyp_trace_buffer_free_bpages(struct hyp_trace_buffer *trace_buffer) { void *start = trace_buffer->bpages_backing_start; size_t size = trace_buffer->bpages_backing_size; @@ -89,11 +106,14 @@ static void hyp_trace_buffer_unload_bpage_backing(struct hyp_trace_buffer *trace if (!size) return; - memset(start, 0, size); - - __release_host_mem(start, size); + if (is_protected_kvm_enabled()) { + hyp_free(start); + } else { + memset(start, 0, size); + __release_host_mem(start, size); + } - trace_buffer->bpages_backing_start = 0; + trace_buffer->bpages_backing_start = NULL; trace_buffer->bpages_backing_size = 0; } @@ -128,7 +148,7 @@ static void hyp_trace_buffer_unload(struct hyp_trace_buffer *trace_buffer) simple_ring_buffer_unload_mm(per_cpu_ptr(trace_buffer->simple_rbs, cpu), __unpin_shared_page); - hyp_trace_buffer_unload_bpage_backing(trace_buffer); + hyp_trace_buffer_free_bpages(trace_buffer); } static int hyp_trace_buffer_load(struct hyp_trace_buffer *trace_buffer, @@ -143,7 +163,7 @@ static int hyp_trace_buffer_load(struct hyp_trace_buffer *trace_buffer, if (hyp_trace_buffer_loaded(trace_buffer)) return -EINVAL; - ret = hyp_trace_buffer_load_bpage_backing(trace_buffer, desc); + ret = hyp_trace_buffer_alloc_bpages(trace_buffer, desc); if (ret) return ret; @@ -168,47 +188,43 @@ static bool hyp_trace_desc_is_valid(struct hyp_trace_desc *desc, size_t desc_siz { struct ring_buffer_desc *rb_desc; unsigned int cpu; - size_t nr_bpages; void *desc_end; if (!is_protected_kvm_enabled()) return true; /* - * Both desc_size and bpages_backing_size are untrusted host-provided - * values. We rely on __pkvm_host_donate_hyp() to enforce their validity. + * desc_size is an untrusted host-provided value. We rely on + * __pkvm_host_donate_hyp() to enforce its validity. */ desc_end = (void *)desc + desc_size; - nr_bpages = desc->bpages_backing_size / sizeof(struct simple_buffer_page); + + if (desc->trace_buffer_desc.nr_cpus != hyp_nr_cpus) + return false; for_each_ring_buffer_desc(rb_desc, cpu, &desc->trace_buffer_desc) { /* Can we read nr_page_va? */ if ((void *)rb_desc + struct_size(rb_desc, page_va, 0) > desc_end) return false; - /* Overflow desc? */ - if ((void *)rb_desc + struct_size(rb_desc, page_va, rb_desc->nr_page_va) > desc_end) - return false; - - /* Overflow bpages backing memory? */ - if (nr_bpages < rb_desc->nr_page_va) + /* simple_ring_buffer_init_mm() expects at least 3 pages */ + if (rb_desc->nr_page_va < 3) return false; - if (cpu >= hyp_nr_cpus) + /* Overflow desc? */ + if ((void *)rb_desc + struct_size(rb_desc, page_va, rb_desc->nr_page_va) > desc_end) return false; if (cpu != rb_desc->cpu) return false; - - nr_bpages -= rb_desc->nr_page_va; } return true; } -int __tracing_load(unsigned long desc_hva, size_t desc_size) +int __tracing_load(void __kern *desc_hva, size_t desc_size) { - struct hyp_trace_desc *desc = (struct hyp_trace_desc *)kern_hyp_va(desc_hva); + struct hyp_trace_desc *desc = kern_hyp_va_host(desc_hva); int ret; ret = __admit_host_mem(desc, desc_size); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b74dd5ce1efd..9c83837b184d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -8,6 +8,7 @@ */ #include <linux/bitfield.h> +#include <linux/kvm_host.h> #include <asm/kvm_pgtable.h> #include <asm/stage2_pgtable.h> @@ -298,12 +299,13 @@ static int leaf_walker(const struct kvm_pgtable_visit_ctx *ctx, } int kvm_pgtable_get_leaf(struct kvm_pgtable *pgt, u64 addr, - kvm_pte_t *ptep, s8 *level) + kvm_pte_t *ptep, s8 *level, + enum kvm_pgtable_walk_flags flags) { struct leaf_walk_data data; struct kvm_pgtable_walker walker = { .cb = leaf_walker, - .flags = KVM_PGTABLE_WALK_LEAF, + .flags = flags | KVM_PGTABLE_WALK_LEAF, .arg = &data, }; int ret; diff --git a/arch/arm64/kvm/hyp/vgic-v5-sr.c b/arch/arm64/kvm/hyp/vgic-v5-sr.c index 6d69dfe89a96..94746317ca79 100644 --- a/arch/arm64/kvm/hyp/vgic-v5-sr.c +++ b/arch/arm64/kvm/hyp/vgic-v5-sr.c @@ -7,6 +7,41 @@ #include <asm/kvm_hyp.h> +void __vgic_v5_make_resident(struct vgic_v5_cpu_if *cpu_if) +{ + write_sysreg_s(cpu_if->vgic_contextr, SYS_ICH_CONTEXTR_EL2); + isb(); + + /* Catch any faults */ + cpu_if->vgic_contextr = read_sysreg_s(SYS_ICH_CONTEXTR_EL2); + if (!!FIELD_GET(ICH_CONTEXTR_EL2_F, cpu_if->vgic_contextr)) + return; + + cpu_if->gicv5_vpe.resident = true; +} + +void __vgic_v5_make_non_resident(struct vgic_v5_cpu_if *cpu_if) +{ + /* + * Clear the db_fired state to ensure that we're ready for the next + * doorbell when it is requested. If a doorbell firing caused us to + * enter the guest, then we've already consumed that state at this + * point, so this is safe to clear. Use WRITE_ONCE() to ensure we're not + * racing with the doorbell firing and setting the state true again. + */ + WRITE_ONCE(cpu_if->gicv5_vpe.db_fired, false); + + /* + * Make as non-resident before actually making non-resident. Avoids race + * with doorbell arriving. + */ + cpu_if->gicv5_vpe.resident = false; + dsb(st); + + write_sysreg_s(cpu_if->vgic_contextr, SYS_ICH_CONTEXTR_EL2); + isb(); +} + void __vgic_v5_save_apr(struct vgic_v5_cpu_if *cpu_if) { cpu_if->vgic_apr = read_sysreg_s(SYS_ICH_APR_EL2); @@ -114,3 +149,13 @@ void __vgic_v5_restore_state(struct vgic_v5_cpu_if *cpu_if) { write_sysreg_s(cpu_if->vgic_icsr, SYS_ICC_ICSR_EL1); } + +void __vgic_v5_vdpend(u32 intid, bool pending, u16 vm) +{ + u64 value; + + value = intid & (GICV5_GIC_VDPEND_ID_MASK | GICV5_GIC_VDPEND_TYPE_MASK); + value |= FIELD_PREP(GICV5_GIC_VDPEND_PENDING_MASK, pending); + value |= FIELD_PREP(GICV5_GIC_VDPEND_VM_MASK, vm); + gic_insn(value, VDPEND); +} diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 7875911c0506..4c965f9b1e1d 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -175,7 +175,8 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) offset = read_sysreg_s(SYS_CNTPOFF_EL2); if (map.direct_ptimer && offset) { - write_sysreg_el0(val + offset, SYS_CNTP_CVAL); + val = timer_apply_offset(val, offset, arch_timer_read_cntpct_el0()); + write_sysreg_el0(val, SYS_CNTP_CVAL); isb(); } } @@ -296,10 +297,10 @@ static bool kvm_hyp_handle_timer(struct kvm_vcpu *vcpu, u64 *exit_code) break; case SYS_CNTP_CVAL_EL0: if (vcpu_el2_e2h_is_set(vcpu)) { - val = read_sysreg_el0(SYS_CNTP_CVAL); - - if (!has_cntpoff()) - val -= timer_get_offset(vcpu_hptimer(vcpu)); + if (!has_cntpoff() && timer_get_offset(vcpu_hptimer(vcpu))) + val = __vcpu_sys_reg(vcpu, CNTHP_CVAL_EL2); + else + val = read_sysreg_el0(SYS_CNTP_CVAL); } else { val = __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0); } @@ -321,10 +322,14 @@ static bool kvm_hyp_handle_timer(struct kvm_vcpu *vcpu, u64 *exit_code) val = __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); break; case SYS_CNTV_CVAL_EL0: - if (vcpu_el2_e2h_is_set(vcpu)) - val = read_sysreg_el0(SYS_CNTV_CVAL); - else + if (vcpu_el2_e2h_is_set(vcpu)) { + if (has_broken_cntvoff() && timer_get_offset(vcpu_hvtimer(vcpu))) + val = __vcpu_sys_reg(vcpu, CNTHV_CVAL_EL2); + else + val = read_sysreg_el0(SYS_CNTV_CVAL); + } else { val = __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); + } break; case SYS_CNTVCT_EL0: case SYS_CNTVCTSS_EL0: diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c index 9644c424819b..b9404496f977 100644 --- a/arch/arm64/kvm/hyp_trace.c +++ b/arch/arm64/kvm/hyp_trace.c @@ -13,6 +13,7 @@ #include <asm/kvm_host.h> #include <asm/kvm_hyptrace.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_pkvm.h> #include "hyp_trace.h" @@ -59,14 +60,8 @@ static void __hyp_clock_work(struct work_struct *work) if (hyp_clock->mult) { u64 err, cur = delta_cycles; - if (WARN_ON_ONCE(cur >= hyp_clock->cyc_overflow64)) { - __uint128_t tmp = (__uint128_t)cur * hyp_clock->mult; - - cur = tmp >> hyp_clock->shift; - } else { - cur *= hyp_clock->mult; - cur >>= hyp_clock->shift; - } + WARN_ON_ONCE(cur >= hyp_clock->cyc_overflow64); + cur = mul_u64_u32_shr(cur, hyp_clock->mult, hyp_clock->shift); cur += hyp_clock->boot; err = abs_diff(cur, boot); @@ -84,7 +79,7 @@ static void __hyp_clock_work(struct work_struct *work) err / NSEC_PER_USEC); } - rate = div64_u64(delta_cycles * NSEC_PER_SEC, delta_boot); + rate = mul_u64_u64_div_u64(delta_cycles, NSEC_PER_SEC, delta_boot); clocks_calc_mult_shift(&hyp_clock->mult, &hyp_clock->shift, rate, NSEC_PER_SEC, CLOCK_MAX_CONVERSION_S); @@ -151,11 +146,18 @@ static void __unshare_page(unsigned long va) static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_buffer, size_t size) { - int nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1; + unsigned long nr_bpages; size_t backing_size; void *start; int ret; + if (is_protected_kvm_enabled()) { + trace_buffer->desc->bpages_backing_start = 0; + trace_buffer->desc->bpages_backing_size = 0; + return 0; + } + + nr_bpages = __calc_nr_pages_ring_buffer_desc(size); backing_size = PAGE_ALIGN(sizeof(struct simple_buffer_page) * nr_bpages * num_possible_cpus()); @@ -177,6 +179,9 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_ static void hyp_trace_buffer_free_bpages_backing(struct hyp_trace_buffer *trace_buffer) { + if (!trace_buffer->desc->bpages_backing_start) + return; + free_pages_exact((void *)trace_buffer->desc->bpages_backing_start, trace_buffer->desc->bpages_backing_size); } @@ -269,7 +274,7 @@ static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv) if (ret) goto err_free_buffer; - ret = kvm_call_hyp_nvhe(__tracing_load, (unsigned long)desc, desc_size); + ret = pkvm_call_hyp_req(__tracing_load, desc, desc_size); if (ret) goto err_unload_pages; @@ -402,7 +407,10 @@ static struct trace_remote_callbacks trace_remote_callbacks = { static const char *__hyp_enter_exit_reason_str(u8 reason); -#include <asm/kvm_define_hypevents.h> +struct remote_event_format_hyp_printk; +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq); + +#include "define_hypevents.h" static const char *__hyp_enter_exit_reason_str(u8 reason) { @@ -421,6 +429,61 @@ static const char *__hyp_enter_exit_reason_str(u8 reason) return strs[min(reason, HYP_REASON_UNKNOWN)]; } +static void __hyp_trace_printk(struct remote_event_format_hyp_printk *entry, struct trace_seq *seq) +{ + const struct hyp_string_fmt *hyp_str_fmt; + const char *fmt; + + trace_seq_putc(seq, ' '); + + if (entry->fmt_id >= (__hyp_string_fmts_end - __hyp_string_fmts_start)) { + trace_seq_printf(seq, "Unknown hyp_string_fmt ID %u\n", entry->fmt_id); + return; + } + + hyp_str_fmt = &__hyp_string_fmts_start[entry->fmt_id]; + fmt = hyp_str_fmt->fmt; + + switch (hyp_str_fmt->nr_args) { + case 0: + trace_seq_puts(seq, fmt); + break; + case 1: + trace_seq_printf(seq, fmt, entry->args[0]); + break; + case 2: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1]); + break; + case 3: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2]); + break; + case 4: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2], + entry->args[3]); + break; + case 5: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2], + entry->args[3], entry->args[4]); + break; + case 6: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2], + entry->args[3], entry->args[4], entry->args[5]); + break; + case 7: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], entry->args[2], + entry->args[3], entry->args[4], entry->args[5], entry->args[6]); + break; + default: + trace_seq_printf(seq, fmt, entry->args[0], entry->args[1], + entry->args[2], entry->args[3], entry->args[4], entry->args[5], + entry->args[6], entry->args[7]); + break; + } + + if (trace_seq_used(seq) && seq->buffer[trace_seq_used(seq) - 1] != '\n') + trace_seq_putc(seq, '\n'); +} + static void __init hyp_trace_init_events(void) { struct hyp_event_id *hyp_event_id = __hyp_event_ids_start; @@ -430,6 +493,9 @@ static void __init hyp_trace_init_events(void) /* Events on both sides hypervisor are sorted */ for (; event < __hyp_events_end; event++, hyp_event_id++, id++) event->id = hyp_event_id->id = id; + + WARN(__hyp_string_fmts_end - __hyp_string_fmts_start > U16_MAX + 1, + "Too many trace_hyp_printk() callsites\n"); } int __init kvm_hyp_trace_init(void) diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c index dfa25bb6f25d..f4b932b7bf12 100644 --- a/arch/arm64/kvm/hypercalls.c +++ b/arch/arm64/kvm/hypercalls.c @@ -294,6 +294,10 @@ int kvm_smccc_call_handler(struct kvm_vcpu *vcpu) case ARM_SMCCC_ARCH_FEATURES_FUNC_ID: feature = smccc_get_arg1(vcpu); switch (feature) { + case ARM_SMCCC_VERSION_FUNC_ID: + case ARM_SMCCC_ARCH_FEATURES_FUNC_ID: + val[0] = SMCCC_RET_SUCCESS; + break; case ARM_SMCCC_ARCH_WORKAROUND_1: switch (arm64_get_spectre_v2_state()) { case SPECTRE_VULNERABLE: @@ -581,6 +585,13 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) wants_02 = vcpu_has_feature(vcpu, KVM_ARM_VCPU_PSCI_0_2); + /* + * EL2 advertises PSCI 1.1; the host handles the forwarded calls + * by this version. + */ + if (vcpu_is_protected(vcpu) && val < KVM_ARM_PSCI_1_1) + return -EINVAL; + switch (val) { case KVM_ARM_PSCI_0_1: if (wants_02) diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c index d6c4fc16f879..96bf926ca520 100644 --- a/arch/arm64/kvm/inject_fault.c +++ b/arch/arm64/kvm/inject_fault.c @@ -371,15 +371,15 @@ int kvm_inject_serror_esr(struct kvm_vcpu *vcpu, u64 esr) } /* - * Emulate the exception entry if SErrors are unmasked. This is useful if - * the vCPU is in a nested context w/ vSErrors enabled then we've already - * delegated he hardware vSError context (i.e. HCR_EL2.VSE, VSESR_EL2, - * VDISR_EL2) to the guest hypervisor. + * With NV, emulate the exception entry if SErrors are unmasked: in a + * nested context with vSErrors enabled, the hardware vSError context + * (i.e. HCR_EL2.VSE, VSESR_EL2, VDISR_EL2) is delegated to the guest + * hypervisor. Otherwise, HCR_EL2.VSE delivers it once it is unmasked. * * As we're emulating the SError injection we need to explicitly populate * ESR_ELx.EC because hardware will not do it on our behalf. */ - if (!serror_is_masked(vcpu)) { + if (vcpu_has_nv(vcpu) && !serror_is_masked(vcpu)) { pend_serror_exception(vcpu); esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_SERROR) | ESR_ELx_IL; vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu)); diff --git a/arch/arm64/kvm/mmio.c b/arch/arm64/kvm/mmio.c index d1c3a352d5a2..9cd1d3d8f707 100644 --- a/arch/arm64/kvm/mmio.c +++ b/arch/arm64/kvm/mmio.c @@ -6,10 +6,12 @@ #include <linux/kvm_host.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_mmu.h> #include <trace/events/kvm.h> #include "trace.h" +#if ARM64_S390_COMMON == 1 void kvm_mmio_write_buf(void *buf, unsigned int len, unsigned long data) { void *datap = NULL; @@ -258,3 +260,4 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) run->exit_reason = KVM_EXIT_MMIO; return 0; } +#endif /* ARM64_S390_COMMON == 1 */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index bf6d6526639f..605a9b25f62b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -5,8 +5,10 @@ */ #include <linux/acpi.h> +#include <linux/cleanup.h> #include <linux/mman.h> #include <linux/kvm_host.h> +#include <linux/interval_tree.h> #include <linux/io.h> #include <linux/hugetlb.h> #include <linux/sched/signal.h> @@ -37,6 +39,8 @@ static unsigned long __ro_after_init io_map_base; #define KVM_PGT_FN(fn) (!is_protected_kvm_enabled() ? fn : p ## fn) +static int kvm_vm_init_vm_s2_ops(struct kvm *kvm); + static phys_addr_t __stage2_range_addr_end(phys_addr_t addr, phys_addr_t end, phys_addr_t size) { @@ -175,6 +179,18 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY); } +static int pkvm_flush_remote_tlbs(struct kvm *kvm) +{ + kvm_call_hyp_nvhe(__pkvm_tlb_flush_vmid, kvm->arch.pkvm.handle); + return 0; +} + +static int kvm_vm_flush_remote_tlbs(struct kvm *kvm) +{ + kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + return 0; +} + /** * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8 * @kvm: pointer to kvm structure. @@ -183,26 +199,31 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { - if (is_protected_kvm_enabled()) - kvm_call_hyp_nvhe(__pkvm_tlb_flush_vmid, kvm->arch.pkvm.handle); - else - kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); - return 0; + return kvm->arch.vm_s2_ops->vm_flush_remote_tlbs(kvm); } -int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, - gfn_t gfn, u64 nr_pages) +static int pkvm_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 nr_pages) +{ + return pkvm_flush_remote_tlbs(kvm); +} + +static int kvm_vm_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 nr_pages) { u64 size = nr_pages << PAGE_SHIFT; u64 addr = gfn << PAGE_SHIFT; - if (is_protected_kvm_enabled()) - kvm_call_hyp_nvhe(__pkvm_tlb_flush_vmid, kvm->arch.pkvm.handle); - else - kvm_tlb_flush_vmid_range(&kvm->arch.mmu, addr, size); + kvm_tlb_flush_vmid_range(&kvm->arch.mmu, addr, size); return 0; } +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 nr_pages) +{ + return kvm->arch.vm_s2_ops->vm_flush_remote_tlbs_range(kvm, gfn, nr_pages); +} + static void *stage2_memcache_zalloc_page(void *arg) { struct kvm_mmu_memory_cache *mc = arg; @@ -298,6 +319,37 @@ static void invalidate_icache_guest_page(void *va, size_t size) __invalidate_icache_guest_page(va, size); } +static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt, + u64 addr, u64 size); + +static void kvm_vm_stage2_unmap_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, + u64 size, bool may_block) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); + int (*fn)(struct kvm_pgtable *, u64, u64); + + if (kvm_is_nested_s2_mmu(kvm, mmu)) + fn = kvm_pgtable_stage2_unmap_tracked; + else + fn = kvm_pgtable_stage2_unmap; + + WARN_ON(stage2_apply_range(mmu, start, start + size, fn, may_block)); +} + +static void pkvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, + u64 size, bool may_block) +{ + WARN_ON(stage2_apply_range(mmu, start, start + size, + pkvm_pgtable_stage2_unmap, may_block)); +} + +static void no_stage2_unmap_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, u64 size, bool may_block) +{ +} + /* * Unmapping vs dcache management: * @@ -322,37 +374,39 @@ static void invalidate_icache_guest_page(void *va, size_t size) * we then fully enforce cacheability of RAM, no matter what the guest * does. */ + +static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt, u64 addr, u64 size) +{ + int ret; + + ret = kvm_pgtable_stage2_unmap(pgt, addr, size); + if (ret) + return ret; + + kvm_remove_guest_s2_mappings(pgt->mmu, addr, size); + return 0; +} + /** - * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range + * kvm_stage2_unmap_range -- Clear stage2 page table entries to unmap a range * @mmu: The KVM stage-2 MMU pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap * @may_block: Whether or not we are permitted to block * * Clear a range of stage-2 mappings, lowering the various ref-counts. Must - * be called while holding mmu_lock (unless for freeing the stage2 pgd before - * destroying the VM), otherwise another faulting VCPU may come in and mess - * with things behind our backs. + * be called while holding mmu_lock otherwise another faulting VCPU may + * come in and mess with things behind our backs. */ -static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size, - bool may_block) +void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start, + u64 size, bool may_block) { struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); - phys_addr_t end = start + size; lockdep_assert_held_write(&kvm->mmu_lock); WARN_ON(size & ~PAGE_MASK); - WARN_ON(stage2_apply_range(mmu, start, end, KVM_PGT_FN(kvm_pgtable_stage2_unmap), - may_block)); -} -void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start, - u64 size, bool may_block) -{ - if (kvm_vm_is_protected(kvm_s2_mmu_to_kvm(mmu))) - return; - - __unmap_stage2_range(mmu, start, size, may_block); + kvm->arch.vm_s2_ops->vm_stage2_unmap_range(mmu, start, size, may_block); } void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) @@ -875,7 +929,7 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr) * IPI-ing threads). */ local_irq_save(flags); - ret = kvm_pgtable_get_leaf(&pgt, addr, &pte, &level); + ret = kvm_pgtable_get_leaf(&pgt, addr, &pte, &level, 0); local_irq_restore(flags); if (ret) @@ -992,6 +1046,12 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t int cpu, err; struct kvm_pgtable *pgt; + /* Initialize the VM ops for the VM instance for the first time */ + if (mmu == &kvm->arch.mmu) { + err = kvm_vm_init_vm_s2_ops(kvm); + if (err) + return err; + } /* * If we already have our page tables in place, and that the * MMU context is the canonical one, we have a bug somewhere, @@ -1042,6 +1102,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t mmu->pgd_phys = __pa(pgt->pgd); + mmu->guest_s2_mappings = RB_ROOT_CACHED; + if (kvm_is_nested_s2_mmu(kvm, mmu)) kvm_init_nested_s2_mmu(mmu); @@ -1131,10 +1193,25 @@ void stage2_unmap_vm(struct kvm *kvm) srcu_read_unlock(&kvm->srcu, idx); } +static void guest_s2_tracking_destroy(struct rb_root_cached *tree) +{ + struct kvm_guest_s2_mapping *mapping; + struct interval_tree_node *node; + + while ((node = interval_tree_iter_first(tree, 0, ULONG_MAX))) { + interval_tree_remove(node, tree); + mapping = container_of(node, struct kvm_guest_s2_mapping, + canonical); + kfree(mapping); + cond_resched(); + } +} + void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt = NULL; + struct rb_root_cached mappings_tree; write_lock(&kvm->mmu_lock); pgt = mmu->pgt; @@ -1147,12 +1224,18 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) if (kvm_is_nested_s2_mmu(kvm, mmu)) kvm_init_nested_s2_mmu(mmu); + mappings_tree = mmu->guest_s2_mappings; + mmu->guest_s2_mappings = RB_ROOT_CACHED; + write_unlock(&kvm->mmu_lock); if (pgt) { kvm_stage2_destroy(pgt); kfree(pgt); } + + if (!kvm_is_nested_s2_mmu(kvm, mmu)) + guest_s2_tracking_destroy(&mappings_tree); } static void hyp_mc_free_fn(void *addr, void *mc) @@ -1169,8 +1252,10 @@ static void *hyp_mc_alloc_fn(void *mc) { struct kvm_hyp_memcache *memcache = mc; void *addr; + gfp_t gfp; - addr = (void *)__get_free_page(GFP_KERNEL_ACCOUNT); + gfp = memcache->flags & HYP_MEMCACHE_ACCOUNT_KMEMCG ? GFP_KERNEL_ACCOUNT : GFP_KERNEL; + addr = (void *)__get_free_page(gfp); if (addr && memcache->flags & HYP_MEMCACHE_ACCOUNT_STAGE2) kvm_account_pgtable_pages(addr, 1); @@ -1223,7 +1308,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, KVM_PGTABLE_PROT_R | (writable ? KVM_PGTABLE_PROT_W : 0); - if (is_protected_kvm_enabled()) + if (kvm_vm_hyp_is_distrusting(kvm)) return -EPERM; size += offset_in_page(guest_ipa); @@ -1362,21 +1447,53 @@ static void kvm_send_hwpoison_signal(unsigned long address, short lsb) send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, lsb, current); } -static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, - unsigned long hva, +struct kvm_s2_fault_desc { + struct kvm_vcpu *vcpu; + phys_addr_t fault_ipa; + struct kvm_s2_trans *nested; + struct kvm_memory_slot *memslot; + unsigned long hva; + unsigned long esr; + struct kvm_s2_mmu *mmu; +}; + +struct kvm_s2_fault_vma_info { + unsigned long mmu_seq; + long vma_pagesize; + vm_flags_t vm_flags; + unsigned long max_map_size; + struct page *page; + kvm_pfn_t pfn; + gfn_t gfn; + bool device; + bool mte_allowed; + bool is_vma_cacheable; + bool map_writable; + bool map_non_cacheable; +}; + +struct kvm_s2_fault_result { + unsigned long mapping_size; +}; + +static bool fault_supports_stage2_huge_mapping(const struct kvm_s2_fault_desc *s2fd, unsigned long map_size) { - gpa_t gpa_start; + struct kvm_memory_slot *memslot = s2fd->memslot; + unsigned long hva = s2fd->hva; hva_t uaddr_start, uaddr_end; + gpa_t gpa_start; size_t size; /* The memslot and the VMA are guaranteed to be aligned to PAGE_SIZE */ if (map_size == PAGE_SIZE) return true; - /* pKVM only supports PMD_SIZE huge-mappings */ - if (is_protected_kvm_enabled() && map_size != PMD_SIZE) - return false; + /* pKVM only supports PMD_SIZE huge-mappings for non-protected VMs */ + if (is_protected_kvm_enabled()) { + if (vcpu_is_protected(s2fd->vcpu) || map_size != PMD_SIZE) + return false; + } size = memslot->npages * PAGE_SIZE; @@ -1436,9 +1553,9 @@ static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, * Returns the size of the mapping. */ static long -transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, - unsigned long hva, kvm_pfn_t *pfnp, gfn_t *gfnp) +transparent_hugepage_adjust(const struct kvm_s2_fault_desc *s2fd, kvm_pfn_t *pfnp, gfn_t *gfnp) { + struct kvm *kvm = s2fd->vcpu->kvm; kvm_pfn_t pfn = *pfnp; gfn_t gfn = *gfnp; @@ -1447,8 +1564,8 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, * sure that the HVA and IPA are sufficiently aligned and that the * block map is contained within the memslot. */ - if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE)) { - int sz = get_user_mapping_size(kvm, hva); + if (fault_supports_stage2_huge_mapping(s2fd, PMD_SIZE)) { + int sz = get_user_mapping_size(kvm, s2fd->hva); if (sz < 0) return sz; @@ -1539,12 +1656,12 @@ static void *get_mmu_memcache(struct kvm_vcpu *vcpu) if (!is_protected_kvm_enabled()) return &vcpu->arch.mmu_page_cache; else - return &vcpu->arch.pkvm_memcache; + return &vcpu->arch.stage2_mc; } -static int topup_mmu_memcache(struct kvm_vcpu *vcpu, void *memcache) +static int topup_mmu_memcache(struct kvm_s2_mmu *mmu, void *memcache) { - int min_pages = kvm_mmu_cache_min_pages(vcpu->arch.hw_mmu); + int min_pages = kvm_mmu_cache_min_pages(mmu); if (!is_protected_kvm_enabled()) return kvm_mmu_topup_memory_cache(memcache, min_pages); @@ -1585,55 +1702,85 @@ static enum kvm_pgtable_prot adjust_nested_exec_perms(struct kvm *kvm, return prot; } -struct kvm_s2_fault_desc { - struct kvm_vcpu *vcpu; - phys_addr_t fault_ipa; - struct kvm_s2_trans *nested; - struct kvm_memory_slot *memslot; - unsigned long hva; -}; +static bool kvm_s2_fault_is_perm(const struct kvm_s2_fault_desc *s2fd) +{ + return esr_fsc_is_permission_fault(s2fd->esr); +} + +static bool kvm_s2_fault_is_exec(const struct kvm_s2_fault_desc *s2fd) +{ + return esr_abt_is_exec_fault(s2fd->esr); +} + +static bool kvm_s2_fault_is_write(const struct kvm_s2_fault_desc *s2fd) +{ + return esr_abt_is_write_fault(s2fd->esr); +} -static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) +static u64 kvm_s2_perm_fault_granule(const struct kvm_s2_fault_desc *s2fd) +{ + u64 level; + + if (!kvm_s2_fault_is_perm(s2fd)) + return 0; + level = s2fd->esr & ESR_ELx_FSC_LEVEL; + return BIT(ARM64_HW_PGTABLE_LEVEL_SHIFT(level)); +} + +static int kvm_s2_fault_get_vma_info(const struct kvm_s2_fault_desc *s2fd, + struct kvm_s2_fault_vma_info *s2vi); + +static gfn_t get_canonical_gfn(const struct kvm_s2_fault_desc *s2fd, + const struct kvm_s2_fault_vma_info *s2vi); + +static int gmem_abort(const struct kvm_s2_fault_desc *s2fd, + struct kvm_s2_fault_result *result) { bool write_fault, exec_fault; - bool perm_fault = kvm_vcpu_trap_is_permission_fault(s2fd->vcpu); + bool perm_fault = kvm_s2_fault_is_perm(s2fd); enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED; enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; - struct kvm_pgtable *pgt = s2fd->vcpu->arch.hw_mmu->pgt; - unsigned long mmu_seq; - struct page *page; + struct kvm_pgtable *pgt = s2fd->mmu->pgt; + struct kvm_guest_s2_mapping *mapping = NULL; + struct kvm_s2_fault_vma_info s2vi = {}; struct kvm *kvm = s2fd->vcpu->kvm; void *memcache = NULL; - kvm_pfn_t pfn; - gfn_t gfn; + gfn_t canonical_gfn; int ret; if (!perm_fault) { memcache = get_mmu_memcache(s2fd->vcpu); - ret = topup_mmu_memcache(s2fd->vcpu, memcache); + ret = topup_mmu_memcache(s2fd->mmu, memcache); if (ret) return ret; + if (kvm_is_nested_s2_mmu(kvm, pgt->mmu)) { + mapping = kmalloc_obj(struct kvm_guest_s2_mapping, GFP_KERNEL_ACCOUNT); + if (!mapping) + return -ENOMEM; + } } - if (s2fd->nested) - gfn = kvm_s2_trans_output(s2fd->nested) >> PAGE_SHIFT; - else - gfn = s2fd->fault_ipa >> PAGE_SHIFT; + s2vi.vma_pagesize = PAGE_SIZE; + s2vi.gfn = ALIGN_DOWN(s2fd->fault_ipa, s2vi.vma_pagesize) >> PAGE_SHIFT; + canonical_gfn = get_canonical_gfn(s2fd, &s2vi); - write_fault = kvm_is_write_fault(s2fd->vcpu); - exec_fault = kvm_vcpu_trap_is_exec_fault(s2fd->vcpu); + write_fault = kvm_s2_fault_is_write(s2fd); + exec_fault = kvm_s2_fault_is_exec(s2fd); VM_WARN_ON_ONCE(write_fault && exec_fault); - mmu_seq = kvm->mmu_invalidate_seq; + s2vi.mmu_seq = kvm->mmu_invalidate_seq; /* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */ smp_rmb(); - ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, gfn, &pfn, &page, NULL); + ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, canonical_gfn, &s2vi.pfn, &s2vi.page, NULL); if (ret) { - kvm_prepare_memory_fault_exit(s2fd->vcpu, s2fd->fault_ipa, PAGE_SIZE, - write_fault, exec_fault, false); - return ret; + /* If result is non-NULL this is a synthetic fault. */ + if (!result) + kvm_prepare_memory_fault_exit(s2fd->vcpu, gfn_to_gpa(canonical_gfn), + s2vi.vma_pagesize, write_fault, exec_fault, + false); + kfree(mapping); } if (!(s2fd->memslot->flags & KVM_MEM_READONLY)) @@ -1649,7 +1796,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) prot = adjust_nested_exec_perms(kvm, s2fd->nested, prot); kvm_fault_lock(kvm); - if (mmu_invalidate_retry(kvm, mmu_seq)) { + if (mmu_invalidate_retry(kvm, s2vi.mmu_seq)) { ret = -EAGAIN; goto out_unlock; } @@ -1660,61 +1807,68 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd) * PTE, which will be preserved. */ prot &= ~KVM_NV_GUEST_MAP_SZ; - ret = KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, s2fd->fault_ipa, + ret = KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, gfn_to_gpa(s2vi.gfn), prot, flags); } else { - ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, s2fd->fault_ipa, PAGE_SIZE, - __pfn_to_phys(pfn), prot, + ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, gfn_to_gpa(s2vi.gfn), + s2vi.vma_pagesize, + __pfn_to_phys(s2vi.pfn), prot, memcache, flags); + /* + * -EAGAIN from kvm_pgtable_stage2_map() can install mappings. + * We don't know which subrange is installed, track the whole + * thing. + */ + if ((ret == 0 || ret == -EAGAIN) && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) { + kvm_record_guest_s2_mapping(pgt->mmu, canonical_gfn << PAGE_SHIFT, + s2fd->fault_ipa, s2vi.vma_pagesize, mapping); + mapping = NULL; + } } out_unlock: - kvm_release_faultin_page(kvm, page, !!ret, prot & KVM_PGTABLE_PROT_W); + kvm_release_faultin_page(kvm, s2vi.page, !!ret, prot & KVM_PGTABLE_PROT_W); kvm_fault_unlock(kvm); + kfree(mapping); if ((prot & KVM_PGTABLE_PROT_W) && !ret) - mark_page_dirty_in_slot(kvm, s2fd->memslot, gfn); + mark_page_dirty_in_slot(kvm, s2fd->memslot, canonical_gfn); - return ret != -EAGAIN ? ret : 0; -} + if (ret == -EAGAIN) + return result ? ret : 0; -struct kvm_s2_fault_vma_info { - unsigned long mmu_seq; - long vma_pagesize; - vm_flags_t vm_flags; - unsigned long max_map_size; - struct page *page; - kvm_pfn_t pfn; - gfn_t gfn; - bool device; - bool mte_allowed; - bool is_vma_cacheable; - bool map_writable; - bool map_non_cacheable; -}; + if (result && !ret) + result->mapping_size = s2vi.vma_pagesize; -static int pkvm_mem_abort(const struct kvm_s2_fault_desc *s2fd) + return ret; +} + +static int protected_pkvm_mem_abort(const struct kvm_s2_fault_desc *s2fd) { unsigned int flags = FOLL_HWPOISON | FOLL_LONGTERM | FOLL_WRITE; + struct kvm_s2_fault_vma_info s2vi = {}; struct kvm_vcpu *vcpu = s2fd->vcpu; - struct kvm_pgtable *pgt = vcpu->arch.hw_mmu->pgt; + struct kvm_pgtable *pgt = s2fd->mmu->pgt; struct mm_struct *mm = current->mm; struct kvm *kvm = vcpu->kvm; void *hyp_memcache; - struct page *page; int ret; hyp_memcache = get_mmu_memcache(vcpu); - ret = topup_mmu_memcache(vcpu, hyp_memcache); + ret = topup_mmu_memcache(s2fd->mmu, hyp_memcache); if (ret) return -ENOMEM; + ret = kvm_s2_fault_get_vma_info(s2fd, &s2vi); + if (ret) + return ret; + ret = account_locked_vm(mm, 1, true); if (ret) return ret; mmap_read_lock(mm); - ret = pin_user_pages(s2fd->hva, 1, flags, &page); + ret = pin_user_pages(s2fd->hva, 1, flags, &s2vi.page); mmap_read_unlock(mm); if (ret == -EHWPOISON) { @@ -1724,7 +1878,7 @@ static int pkvm_mem_abort(const struct kvm_s2_fault_desc *s2fd) } else if (ret != 1) { ret = -EFAULT; goto dec_account; - } else if (!folio_test_swapbacked(page_folio(page))) { + } else if (!folio_test_swapbacked(page_folio(s2vi.page))) { /* * We really can't deal with page-cache pages returned by GUP * because (a) we may trigger writeback of a page for which we @@ -1744,8 +1898,8 @@ static int pkvm_mem_abort(const struct kvm_s2_fault_desc *s2fd) } write_lock(&kvm->mmu_lock); - ret = pkvm_pgtable_stage2_map(pgt, s2fd->fault_ipa, PAGE_SIZE, - page_to_phys(page), KVM_PGTABLE_PROT_RWX, + ret = pkvm_pgtable_stage2_map(pgt, gfn_to_gpa(s2vi.gfn), PAGE_SIZE, + page_to_phys(s2vi.page), KVM_PGTABLE_PROT_RWX, hyp_memcache, 0); write_unlock(&kvm->mmu_lock); if (ret) { @@ -1756,7 +1910,7 @@ static int pkvm_mem_abort(const struct kvm_s2_fault_desc *s2fd) return 0; unpin: - unpin_user_pages(&page, 1); + unpin_user_page(s2vi.page); dec_account: account_locked_vm(mm, 1, false); return ret; @@ -1779,7 +1933,7 @@ static short kvm_s2_resolve_vma_size(const struct kvm_s2_fault_desc *s2fd, switch (vma_shift) { #ifndef __PAGETABLE_PMD_FOLDED case PUD_SHIFT: - if (fault_supports_stage2_huge_mapping(s2fd->memslot, s2fd->hva, PUD_SIZE)) + if (fault_supports_stage2_huge_mapping(s2fd, PUD_SIZE)) break; fallthrough; #endif @@ -1787,7 +1941,7 @@ static short kvm_s2_resolve_vma_size(const struct kvm_s2_fault_desc *s2fd, vma_shift = PMD_SHIFT; fallthrough; case PMD_SHIFT: - if (fault_supports_stage2_huge_mapping(s2fd->memslot, s2fd->hva, PMD_SIZE)) + if (fault_supports_stage2_huge_mapping(s2fd, PMD_SIZE)) break; fallthrough; case CONT_PTE_SHIFT: @@ -1828,11 +1982,6 @@ static short kvm_s2_resolve_vma_size(const struct kvm_s2_fault_desc *s2fd, return vma_shift; } -static bool kvm_s2_fault_is_perm(const struct kvm_s2_fault_desc *s2fd) -{ - return kvm_vcpu_trap_is_permission_fault(s2fd->vcpu); -} - static int kvm_s2_fault_get_vma_info(const struct kvm_s2_fault_desc *s2fd, struct kvm_s2_fault_vma_info *s2vi) { @@ -1898,13 +2047,11 @@ static int kvm_s2_fault_pin_pfn(const struct kvm_s2_fault_desc *s2fd, return ret; s2vi->pfn = __kvm_faultin_pfn(s2fd->memslot, get_canonical_gfn(s2fd, s2vi), - kvm_is_write_fault(s2fd->vcpu) ? FOLL_WRITE : 0, + kvm_s2_fault_is_write(s2fd) ? FOLL_WRITE : 0, &s2vi->map_writable, &s2vi->page); if (unlikely(is_error_noslot_pfn(s2vi->pfn))) { - if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) { - kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi->vma_pagesize)); - return 0; - } + if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) + return -EHWPOISON; return -EFAULT; } @@ -1946,7 +2093,7 @@ static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd, { struct kvm *kvm = s2fd->vcpu->kvm; - if (kvm_vcpu_trap_is_exec_fault(s2fd->vcpu) && s2vi->map_non_cacheable) + if (kvm_s2_fault_is_exec(s2fd) && s2vi->map_non_cacheable) return -ENOEXEC; /* @@ -1955,7 +2102,7 @@ static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd, * and trigger the exception here. Since the memslot is valid, inject * the fault back to the guest. */ - if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(s2fd->vcpu))) { + if (esr_fsc_is_excl_atomic_fault(s2fd->esr)) { kvm_inject_dabt_excl_atomic(s2fd->vcpu, kvm_vcpu_get_hfar(s2fd->vcpu)); return 1; } @@ -1964,13 +2111,13 @@ static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd, if (s2vi->map_writable && (s2vi->device || !memslot_is_logging(s2fd->memslot) || - kvm_is_write_fault(s2fd->vcpu))) + kvm_s2_fault_is_write(s2fd))) *prot |= KVM_PGTABLE_PROT_W; if (s2fd->nested) *prot = adjust_nested_fault_perms(s2fd->nested, *prot); - if (kvm_vcpu_trap_is_exec_fault(s2fd->vcpu)) + if (kvm_s2_fault_is_exec(s2fd)) *prot |= KVM_PGTABLE_PROT_X; if (s2vi->map_non_cacheable) @@ -1994,11 +2141,14 @@ static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd, static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd, const struct kvm_s2_fault_vma_info *s2vi, enum kvm_pgtable_prot prot, - void *memcache) + void *memcache, + struct kvm_s2_fault_result *result) { enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED; + struct kvm_guest_s2_mapping *mapping = NULL; bool writable = prot & KVM_PGTABLE_PROT_W; struct kvm *kvm = s2fd->vcpu->kvm; + phys_addr_t canonical_ipa; struct kvm_pgtable *pgt; long perm_fault_granule; long mapping_size; @@ -2006,17 +2156,26 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd, gfn_t gfn; int ret; + if (kvm_is_nested_s2_mmu(kvm, s2fd->mmu)) { + mapping = kmalloc_obj(struct kvm_guest_s2_mapping, + GFP_KERNEL_ACCOUNT); + if (!mapping) { + kvm_release_page_unused(s2vi->page); + return -ENOMEM; + } + } + kvm_fault_lock(kvm); - pgt = s2fd->vcpu->arch.hw_mmu->pgt; + pgt = s2fd->mmu->pgt; ret = -EAGAIN; if (mmu_invalidate_retry(kvm, s2vi->mmu_seq)) goto out_unlock; - perm_fault_granule = (kvm_s2_fault_is_perm(s2fd) ? - kvm_vcpu_trap_get_perm_fault_granule(s2fd->vcpu) : 0); + perm_fault_granule = kvm_s2_perm_fault_granule(s2fd); mapping_size = s2vi->vma_pagesize; pfn = s2vi->pfn; gfn = s2vi->gfn; + canonical_ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi)); /* * If we are not forced to use page mapping, check if we are @@ -2027,14 +2186,13 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd, if (perm_fault_granule > PAGE_SIZE) { mapping_size = perm_fault_granule; } else { - mapping_size = transparent_hugepage_adjust(kvm, s2fd->memslot, - s2fd->hva, &pfn, - &gfn); + mapping_size = transparent_hugepage_adjust(s2fd, &pfn, &gfn); if (mapping_size < 0) { ret = mapping_size; goto out_unlock; } } + canonical_ipa = ALIGN_DOWN(canonical_ipa, mapping_size); } if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm)) @@ -2057,31 +2215,45 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd, ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, gfn_to_gpa(gfn), mapping_size, __pfn_to_phys(pfn), prot, memcache, flags); + /* + * -EAGAIN from kvm_pgtable_stage2_map() can install mappings. + * We don't know which subrange is installed, track the whole + * thing. + */ + if ((ret == 0 || ret == -EAGAIN) && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) { + kvm_record_guest_s2_mapping(pgt->mmu, canonical_ipa, + gfn_to_gpa(gfn), mapping_size, mapping); + mapping = NULL; + } } out_unlock: kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable); kvm_fault_unlock(kvm); + kfree(mapping); /* * Mark the page dirty only if the fault is handled successfully, * making sure we adjust the canonical IPA if the mapping size has * been updated (via a THP upgrade, for example). */ - if (writable && !ret) { - phys_addr_t ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi)); - ipa &= ~(mapping_size - 1); - mark_page_dirty_in_slot(kvm, s2fd->memslot, gpa_to_gfn(ipa)); - } + if (writable && !ret) + mark_page_dirty_in_slot(kvm, s2fd->memslot, + gpa_to_gfn(canonical_ipa)); - if (ret != -EAGAIN) - return ret; - return 0; + if (ret == -EAGAIN) + return result ? ret : 0; + + if (result && !ret) + result->mapping_size = mapping_size; + + return ret; } -static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd) +static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd, + struct kvm_s2_fault_result *result) { - bool perm_fault = kvm_vcpu_trap_is_permission_fault(s2fd->vcpu); + bool perm_fault = kvm_s2_fault_is_perm(s2fd); struct kvm_s2_fault_vma_info s2vi = {}; enum kvm_pgtable_prot prot; void *memcache; @@ -2099,12 +2271,19 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd) memcache = get_mmu_memcache(s2fd->vcpu); if (!perm_fault || memslot_is_logging(s2fd->memslot) || is_protected_kvm_enabled()) { - ret = topup_mmu_memcache(s2fd->vcpu, memcache); + ret = topup_mmu_memcache(s2fd->mmu, memcache); if (ret) return ret; } ret = kvm_s2_fault_pin_pfn(s2fd, &s2vi); + if (ret == -EHWPOISON) { + /* If result is specified, let the caller handle this. */ + if (result) + return -EHWPOISON; + kvm_send_hwpoison_signal(s2fd->hva, __ffs(s2vi.vma_pagesize)); + return 0; + } if (ret != 1) return ret; @@ -2114,7 +2293,19 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd) return ret; } - return kvm_s2_fault_map(s2fd, &s2vi, prot, memcache); + return kvm_s2_fault_map(s2fd, &s2vi, prot, memcache, result); +} + +static int kvm_vm_mem_abort(const struct kvm_s2_fault_desc *s2fd) +{ + VM_WARN_ON_ONCE(kvm_s2_fault_is_perm(s2fd) && + !esr_abt_is_write_fault(s2fd->esr) && + !kvm_s2_fault_is_exec(s2fd)); + + if (kvm_slot_has_gmem(s2fd->memslot)) + return gmem_abort(s2fd, NULL); + else + return user_mem_abort(s2fd, NULL); } /* Resolve the access fault by making the page young again. */ @@ -2224,7 +2415,9 @@ int kvm_handle_guest_sea(struct kvm_vcpu *vcpu) int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) { struct kvm_s2_trans nested_trans, *nested = NULL; - unsigned long esr; + unsigned long esr = kvm_vcpu_get_esr(vcpu); + struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu; + struct kvm *kvm = vcpu->kvm; phys_addr_t fault_ipa; /* The address we faulted on */ phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */ struct kvm_memory_slot *memslot; @@ -2233,20 +2426,18 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) gfn_t gfn; int ret, idx; - if (kvm_vcpu_abt_issea(vcpu)) + if (esr_abt_is_sea(esr)) return kvm_handle_guest_sea(vcpu); - esr = kvm_vcpu_get_esr(vcpu); - /* * The fault IPA should be reliable at this point as we're not dealing * with an SEA. */ ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); - if (KVM_BUG_ON(ipa == INVALID_GPA, vcpu->kvm)) + if (KVM_BUG_ON(ipa == INVALID_GPA, kvm)) return -EFAULT; - is_iabt = kvm_vcpu_trap_is_iabt(vcpu); + is_iabt = esr_trap_is_iabt(esr); if (esr_fsc_is_translation_fault(esr)) { /* Beyond sanitised PARange (which is the IPA limit) */ @@ -2256,14 +2447,14 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) } /* Falls between the IPA range and the PARange? */ - if (fault_ipa >= BIT_ULL(VTCR_EL2_IPA(vcpu->arch.hw_mmu->vtcr))) { + if (fault_ipa >= BIT_ULL(VTCR_EL2_IPA(mmu->vtcr))) { fault_ipa |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(vcpu)); return kvm_inject_sea(vcpu, is_iabt, fault_ipa); } } - trace_kvm_guest_fault(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu), + trace_kvm_guest_fault(*vcpu_pc(vcpu), esr, kvm_vcpu_get_hfar(vcpu), fault_ipa); /* Check the stage-2 fault is trans. fault or write fault */ @@ -2271,14 +2462,14 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) !esr_fsc_is_permission_fault(esr) && !esr_fsc_is_access_flag_fault(esr) && !esr_fsc_is_excl_atomic_fault(esr)) { - kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n", - kvm_vcpu_trap_get_class(vcpu), - (unsigned long)kvm_vcpu_trap_get_fault(vcpu), - (unsigned long)kvm_vcpu_get_esr(vcpu)); + kvm_err("Unsupported FSC: EC=%#lx xFSC=%#lx ESR_EL2=%#lx\n", + ESR_ELx_EC(esr), + (unsigned long)(esr & ESR_ELx_FSC), + (unsigned long)esr); return -EFAULT; } - idx = srcu_read_lock(&vcpu->kvm->srcu); + idx = srcu_read_lock(&kvm->srcu); /* * We may have faulted on a shadow stage 2 page table if we are @@ -2293,8 +2484,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) * nothing to walk and we treat it as a 1:1 before going through the * canonical translation. */ - if (kvm_is_nested_s2_mmu(vcpu->kvm,vcpu->arch.hw_mmu) && - vcpu->arch.hw_mmu->nested_stage2_enabled) { + if (kvm_is_nested_s2_mmu(kvm, mmu) && mmu->nested_stage2_enabled) { u32 esr; ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans); @@ -2321,9 +2511,9 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) } gfn = ipa >> PAGE_SHIFT; - memslot = gfn_to_memslot(vcpu->kvm, gfn); + memslot = gfn_to_memslot(kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); - write_fault = kvm_is_write_fault(vcpu); + write_fault = esr_abt_is_write_fault(esr); if (kvm_is_error_hva(hva) || (write_fault && !writable)) { /* * The guest has put either its instructions or its page-tables @@ -2336,7 +2526,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) goto out; } - if (kvm_vcpu_abt_iss1tw(vcpu)) { + if (esr_abt_is_s1ptw(esr)) { ret = kvm_inject_sea_dabt(vcpu, kvm_vcpu_get_hfar(vcpu)); goto out_unlock; } @@ -2351,7 +2541,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) * So let's assume that the guest is just being * cautious, and skip the instruction. */ - if (kvm_is_error_hva(hva) && kvm_vcpu_dabt_is_cm(vcpu)) { + if (kvm_is_error_hva(hva) && esr_dabt_is_cm(esr)) { kvm_incr_pc(vcpu); ret = 1; goto out_unlock; @@ -2369,7 +2559,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) } /* Userspace should not be able to register out-of-bounds IPAs */ - VM_BUG_ON(ipa >= kvm_phys_size(vcpu->arch.hw_mmu)); + VM_BUG_ON(ipa >= kvm_phys_size(mmu)); if (esr_fsc_is_access_flag_fault(esr)) { handle_access_fault(vcpu, fault_ipa); @@ -2383,72 +2573,98 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) .nested = nested, .memslot = memslot, .hva = hva, + .esr = esr, + .mmu = mmu, }; - if (kvm_vm_is_protected(vcpu->kvm)) { - ret = pkvm_mem_abort(&s2fd); - } else { - VM_WARN_ON_ONCE(kvm_vcpu_trap_is_permission_fault(vcpu) && - !write_fault && - !kvm_vcpu_trap_is_exec_fault(vcpu)); - - if (kvm_slot_has_gmem(memslot)) - ret = gmem_abort(&s2fd); - else - ret = user_mem_abort(&s2fd); - } - + ret = kvm->arch.vm_s2_ops->vm_mem_abort(&s2fd); if (ret == 0) ret = 1; out: if (ret == -ENOEXEC) ret = kvm_inject_sea_iabt(vcpu, kvm_vcpu_get_hfar(vcpu)); out_unlock: - srcu_read_unlock(&vcpu->kvm->srcu, idx); + srcu_read_unlock(&kvm->srcu, idx); return ret; } bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { - if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm)) + gpa_t gpa = range->start << PAGE_SHIFT; + size_t size = (range->end - range->start) << PAGE_SHIFT; + bool may_block = range->may_block; + + if (!kvm->arch.mmu.pgt) return false; - __unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT, - (range->end - range->start) << PAGE_SHIFT, - range->may_block); + kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, may_block); + kvm_nested_unmap_cipa_range(kvm, gpa, size, may_block); - kvm_nested_s2_unmap(kvm, range->may_block); return false; } -bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +static bool kvm_vm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; - if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm)) - return false; - - return KVM_PGT_FN(kvm_pgtable_stage2_test_clear_young)(kvm->arch.mmu.pgt, + return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, size, true); +} + +static bool pkvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + u64 size = (range->end - range->start) << PAGE_SHIFT; + + return pkvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, true); +} + +static bool no_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + /* The hypervisor doesn't support aging */ + return false; +} + +bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + if (!kvm->arch.mmu.pgt) + return false; + + return kvm->arch.vm_s2_ops->vm_age_gfn(kvm, range); /* * TODO: Handle nested_mmu structures here using the reverse mapping in * a later version of patch series. */ } -bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +static bool kvm_vm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; - if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm)) - return false; - - return KVM_PGT_FN(kvm_pgtable_stage2_test_clear_young)(kvm->arch.mmu.pgt, + return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, size, false); } +static bool pkvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + u64 size = (range->end - range->start) << PAGE_SHIFT; + + return pkvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, false); +} + +bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + if (!kvm->arch.mmu.pgt) + return false; + + return kvm->arch.vm_s2_ops->vm_test_age_gfn(kvm, range); +} + phys_addr_t kvm_mmu_get_httbr(void) { return __pa(hyp_pgtable->pgd); @@ -2699,7 +2915,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, write_lock(&kvm->mmu_lock); kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, true); - kvm_nested_s2_unmap(kvm, true); + kvm_nested_unmap_cipa_range(kvm, gpa, size, true); write_unlock(&kvm->mmu_lock); } @@ -2770,3 +2986,193 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled) trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled); } + +/* + * Try to walk to the specified GPA in canonical mmu - if unmapped returns 0, if + * mapped returns the granule size, otherwise returns an error. + */ +static long kvm_walk_s2(struct kvm_pgtable *pgt, + gpa_t gpa, s8 *level) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu); + kvm_pte_t pte; + long ret; + + guard(read_lock)(&kvm->mmu_lock); + + ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, level, + KVM_PGTABLE_WALK_SHARED); + if (ret) + return ret; + /* Unpopulated, must fault. */ + if (!kvm_pte_valid(pte)) + return 0; + return kvm_granule_size(*level); +} + +/* Synthesised data abort at specified page table level. */ +#define PRE_FAULT_ESR(level) \ + ((ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) | \ + ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(level)) + +/* Retrieve either a read-only or a read/write hva. */ +static hva_t gfn_to_hva_memslot_read(struct kvm_memory_slot *slot, gfn_t gfn) +{ + return gfn_to_hva_memslot_prot(slot, gfn, /*writable=*/NULL); +} + +static long __pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot, s8 level) +{ + const bool is_gmem = kvm_slot_has_gmem(memslot); + const gfn_t gfn = gpa_to_gfn(gpa); + const hva_t hva = is_gmem ? 0 : gfn_to_hva_memslot_read(memslot, gfn); + const struct kvm_s2_fault_desc s2fd = { + .vcpu = vcpu, + .fault_ipa = gpa, + .nested = NULL, + .memslot = memslot, + .hva = hva, + .esr = PRE_FAULT_ESR(level), + .mmu = mmu, + }; + struct kvm_s2_fault_result result = {}; + long ret; + + if (kvm_is_error_hva(hva)) + return -EFAULT; + + if (is_gmem) + ret = gmem_abort(&s2fd, &result); + else + ret = user_mem_abort(&s2fd, &result); + if (IS_ERR_VALUE(ret)) + return ret; + return result.mapping_size; +} + +static long pre_fault_s2(struct kvm_s2_mmu *mmu, struct kvm_vcpu *vcpu, + gpa_t gpa, struct kvm_memory_slot *memslot) +{ + s8 level; + long ret; + + /* Try a walk first. */ + ret = kvm_walk_s2(mmu->pgt, gpa, &level); + if (ret) + return ret; + /* OK, have to fault page in. */ + return __pre_fault_s2(mmu, vcpu, gpa, memslot, level); +} + +static unsigned long +pre_fault_bytes_consumed(gpa_t gpa, unsigned long granule_size, + unsigned long bytes_remaining) +{ + /* Granules are always a power-of-2. */ + const unsigned long granule_bytes_remaining = + granule_size - (gpa % granule_size); + + return min(granule_bytes_remaining, bytes_remaining); +} + +/* If you lose the race this many times, time to give up. */ +#define MAX_PRE_FAULT_RETRIES 3 + +int kvm_arch_pre_fault_allowed(struct kvm_vcpu *vcpu) +{ + if (is_protected_kvm_enabled()) + return -EOPNOTSUPP; + if (!kvm_vcpu_initialized(vcpu)) + return -ENOEXEC; + + return 0; +} + +/** + * kvm_arch_vcpu_pre_fault_memory - pre-fault stage-2 page tables for the + * specified GPA. + * @vcpu: The VCPU pointer + * @range: {gpa, size, flags} tuple + * + * The mapping performed is always best-effort - faulting in is necessarily + * racey. The ranges faulted in are canonical, nested page tables are ignored. + * + * @range->gpa specifies the GPA to pre-fault, @range->size specifies how many + * bytes remain to be pre-faulted and @range->flags is reserved and must be 0. + * + * Returns: the number of bytes the pre-fault consumed, or an error. + */ +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, + struct kvm_pre_fault_memory *range) +{ + struct kvm *kvm = vcpu->kvm; + const u64 bytes_remaining = range->size; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; /* Canonical. */ + struct kvm_memory_slot *memslot; + const gpa_t gpa = range->gpa; + int num_retries = 0; + long ret; + + memslot = gfn_to_memslot(kvm, gpa_to_gfn(gpa)); + if (!memslot) + return -ENOENT; + /* SRCU must be released for progress and only userland can do that. */ + if (memslot->flags & KVM_MEMSLOT_INVALID) + return -EAGAIN; + + do { + ret = pre_fault_s2(mmu, vcpu, gpa, memslot); + } while (ret == -EAGAIN && num_retries++ < MAX_PRE_FAULT_RETRIES); + + if (IS_ERR_VALUE(ret)) + return ret; + return pre_fault_bytes_consumed(gpa, ret, bytes_remaining); +} + +static const struct kvm_vm_s2_ops protected_pkvm_vm_s2_ops = { + .vm_flush_remote_tlbs = pkvm_flush_remote_tlbs, + .vm_flush_remote_tlbs_range = pkvm_flush_remote_tlbs_range, + .vm_age_gfn = no_age_gfn, + .vm_test_age_gfn = no_age_gfn, + .vm_stage2_unmap_range = no_stage2_unmap_range, + .vm_mem_abort = protected_pkvm_mem_abort, +}; + +static const struct kvm_vm_s2_ops pkvm_vm_s2_ops = { + .vm_flush_remote_tlbs = pkvm_flush_remote_tlbs, + .vm_flush_remote_tlbs_range = pkvm_flush_remote_tlbs_range, + .vm_age_gfn = pkvm_age_gfn, + .vm_test_age_gfn = pkvm_test_age_gfn, + .vm_stage2_unmap_range = pkvm_stage2_unmap_range, + .vm_mem_abort = kvm_vm_mem_abort, +}; + +static const struct kvm_vm_s2_ops kvm_default_vm_s2_ops = { + .vm_flush_remote_tlbs = kvm_vm_flush_remote_tlbs, + .vm_flush_remote_tlbs_range = kvm_vm_flush_remote_tlbs_range, + .vm_age_gfn = kvm_vm_age_gfn, + .vm_test_age_gfn = kvm_vm_test_age_gfn, + .vm_stage2_unmap_range = kvm_vm_stage2_unmap_range, + .vm_mem_abort = kvm_vm_mem_abort, +}; + +#define KVM_VM_S2_OPS(flavor, ops) \ + [(flavor)] = &(ops) + +static const struct kvm_vm_s2_ops *arm64_vm_s2_ops[] = { + KVM_VM_S2_OPS(VM_VHE, kvm_default_vm_s2_ops), + KVM_VM_S2_OPS(VM_NVHE, kvm_default_vm_s2_ops), + KVM_VM_S2_OPS(VM_PKVM, pkvm_vm_s2_ops), + KVM_VM_S2_OPS(VM_PROTECTED_PKVM, protected_pkvm_vm_s2_ops), +}; + +static int kvm_vm_init_vm_s2_ops(struct kvm *kvm) +{ + BUILD_BUG_ON(ARRAY_SIZE(arm64_vm_s2_ops) != VM_FLAVOR_MAX); + + kvm->arch.vm_s2_ops = arm64_vm_s2_ops[kvm->arch.vm_flavor]; + if (WARN_ON(!kvm->arch.vm_s2_ops)) + return -EINVAL; + return 0; +} diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index b191365d97cc..f0f473978d98 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -5,6 +5,7 @@ */ #include <linux/bitfield.h> +#include <linux/interval_tree.h> #include <linux/kvm.h> #include <linux/kvm_host.h> @@ -44,14 +45,16 @@ struct vncr_tlb { * will invalidate them more often). */ #define S2_MMU_PER_VCPU 2 +#define S2_MMU_PER_VM (KVM_MAX_VCPUS * S2_MMU_PER_VCPU) int kvm_init_nested(struct kvm *kvm) { kvm->arch.nested_mmus = kvmalloc_objs(struct kvm_s2_mmu *, - KVM_MAX_VCPUS * S2_MMU_PER_VCPU, + S2_MMU_PER_VM, GFP_KERNEL_ACCOUNT); kvm->arch.nested_mmus_size = 0; atomic_set(&kvm->arch.vncr_tlb_count, 0); + spin_lock_init(&kvm->arch.guest_s2_tracking_lock); return kvm->arch.nested_mmus ? 0 : -ENOMEM; } @@ -124,10 +127,15 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) return ret; } + for (i = 0; i < S2_MMU_PER_VCPU; i++) + kvm_nested_s2_ptdump_create_debugfs(&tmp[i], i + kvm->arch.nested_mmus_size); + guard(write_lock)(&kvm->mmu_lock); - for (i = 0; i < S2_MMU_PER_VCPU; i++) + for (i = 0; i < S2_MMU_PER_VCPU; i++) { + tmp[i].s2_mmu_idx = i + kvm->arch.nested_mmus_size; kvm->arch.nested_mmus[i + kvm->arch.nested_mmus_size] = &tmp[i]; + } kvm->arch.nested_mmus_size += S2_MMU_PER_VCPU; } @@ -657,7 +665,7 @@ again: return 0; tmp &= ~(sz - 1); - if (kvm_pgtable_get_leaf(mmu->pgt, tmp, &pte, NULL)) + if (kvm_pgtable_get_leaf(mmu->pgt, tmp, &pte, NULL, 0)) goto again; if (!(pte & PTE_VALID)) goto again; @@ -837,10 +845,8 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) kvm->arch.nested_mmus_next = (i + 1) % kvm->arch.nested_mmus_size; /* Make sure we don't forget to do the laundry */ - if (kvm_s2_mmu_valid(s2_mmu)) { - kvm_nested_s2_ptdump_remove_debugfs(s2_mmu); + if (kvm_s2_mmu_valid(s2_mmu)) s2_mmu->pending_unmap = true; - } /* * The virtual VMID (modulo CnP) will be used as a key when matching @@ -854,8 +860,6 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) s2_mmu->tlb_vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); s2_mmu->nested_stage2_enabled = vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_VM; - kvm_nested_s2_ptdump_create_debugfs(s2_mmu); - out: atomic_inc(&s2_mmu->refcnt); @@ -871,6 +875,87 @@ out: return s2_mmu; } +#define S2_MMU_IDX_MASK GENMASK_ULL(11, 0) + +static void tag_s2_mapping_mmu(struct kvm_guest_s2_mapping *mapping, + struct kvm_s2_mmu *mmu) +{ + BUILD_BUG_ON(S2_MMU_PER_VM > SZ_4K); + mapping->nested.start &= ~S2_MMU_IDX_MASK; + mapping->nested.start |= mmu->s2_mmu_idx; +} + +static struct kvm_s2_mmu *s2_mapping_to_mmu(struct kvm *kvm, + struct kvm_guest_s2_mapping *mapping) +{ + return kvm->arch.nested_mmus[mapping->nested.start & S2_MMU_IDX_MASK]; +} + +static unsigned long s2_mapping_to_nested_start(struct kvm_guest_s2_mapping *mapping) +{ + return mapping->nested.start & ~S2_MMU_IDX_MASK; +} + +void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa, + gpa_t nested_ipa, size_t map_size, + struct kvm_guest_s2_mapping *mapping) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); + + lockdep_assert_held_read(&kvm->mmu_lock); + + canonical_ipa = ALIGN_DOWN(canonical_ipa, map_size); + nested_ipa = ALIGN_DOWN(nested_ipa, map_size); + + mapping->canonical.start = canonical_ipa; + mapping->canonical.last = canonical_ipa + map_size - 1; + + mapping->nested.start = nested_ipa; + mapping->nested.last = nested_ipa + map_size - 1; + + tag_s2_mapping_mmu(mapping, mmu); + + guard(spinlock)(&kvm->arch.guest_s2_tracking_lock); + interval_tree_insert(&mapping->nested, &mmu->guest_s2_mappings); + interval_tree_insert(&mapping->canonical, &kvm->arch.mmu.guest_s2_mappings); +} + +void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa, + size_t size) +{ + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); + struct interval_tree_node *node, *next; + struct kvm_guest_s2_mapping *mapping; + gpa_t nipa_end = nipa + size - 1; + + /* + * See kvm_nested_unmap_cipa_range() for why guest_s2_tracking_lock + * isn't taken here. + */ + lockdep_assert_held_write(&kvm->mmu_lock); + + node = interval_tree_iter_first(&mmu->guest_s2_mappings, nipa, nipa_end); + while (node) { + unsigned long nested_start; + + next = interval_tree_iter_next(node, nipa, nipa_end); + mapping = container_of(node, struct kvm_guest_s2_mapping, + nested); + /* + * Tracking must be conservative on removal, only remove + * mappings that are within the unmap range. + */ + nested_start = s2_mapping_to_nested_start(mapping); + if (nipa <= nested_start && nipa_end >= mapping->nested.last) { + interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings); + interval_tree_remove(&mapping->canonical, + &kvm->arch.mmu.guest_s2_mappings); + kfree(mapping); + } + node = next; + } +} + void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) { /* CnP being set denotes an invalid entry */ @@ -1293,6 +1378,56 @@ void kvm_nested_s2_wp(struct kvm *kvm) kvm_invalidate_vncr_ipa_all(kvm); } +void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa, size_t unmap_size, + bool may_block) +{ + gpa_t cipa_end = cipa + unmap_size - 1; + struct kvm_guest_s2_mapping *mapping; + struct interval_tree_node *node; + size_t mapping_size; + + /* + * Guest s2 tracking interval trees are only accessed while holding the + * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the + * mmu_lock is held for write. This saves us from having to manually + * lock/unlock guest_s2_tracking_lock below around + * cond_resched_rwlock_write(). + */ + lockdep_assert_held_write(&kvm->mmu_lock); + + if (!kvm->arch.nested_mmus_size) + return; + + while ((node = interval_tree_iter_first(&kvm->arch.mmu.guest_s2_mappings, + cipa, cipa_end))) { + unsigned long nested_start; + struct kvm_s2_mmu *mmu; + + mapping = container_of(node, struct kvm_guest_s2_mapping, + canonical); + nested_start = s2_mapping_to_nested_start(mapping); + mmu = s2_mapping_to_mmu(kvm, mapping); + + /* We could race against MMU teardown, which frees mmu->pgt. */ + if (mmu->pgt) { + mapping_size = mapping->nested.last - nested_start + 1; + + if (WARN_ON_ONCE(kvm_pgtable_stage2_unmap(mmu->pgt, nested_start, + mapping_size))) + return; + + interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings); + } + interval_tree_remove(node, &kvm->arch.mmu.guest_s2_mappings); + kfree(mapping); + + if (may_block) + cond_resched_rwlock_write(&kvm->mmu_lock); + } + + kvm_invalidate_vncr_ipa(kvm, cipa, cipa + unmap_size); +} + void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block) { int i; diff --git a/arch/arm64/kvm/pauth.c b/arch/arm64/kvm/pauth.c index d5eb3ae876be..bb6c42e78058 100644 --- a/arch/arm64/kvm/pauth.c +++ b/arch/arm64/kvm/pauth.c @@ -5,7 +5,7 @@ * * Primitive PAuth emulation for ERETAA/ERETAB. * - * This code assumes that is is run from EL2, and that it is part of + * This code assumes that it is run from EL2, and that it is part of * the emulation of ERETAx for a guest hypervisor. That's a lot of * baked-in assumptions and shortcuts. * diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 8e4c6e4bec12..380dfb1db0be 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -16,6 +16,9 @@ #include "hyp_constants.h" +#define CREATE_TRACE_POINTS +#include "trace_pkvm.h" + DEFINE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory); @@ -85,6 +88,77 @@ void __init kvm_hyp_reserve(void) hyp_mem_base); } +static int pkvm_hyp_topup(enum pkvm_topup_id id, unsigned long nr_pages) +{ + struct kvm_hyp_memcache mc; + struct arm_smccc_res res; + int ret; + + init_hyp_memcache(&mc); + ret = topup_hyp_memcache(&mc, nr_pages); + if (ret) + goto err; + + arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__pkvm_hyp_topup), id, mc.head, + mc.nr_pages, &res); + if (WARN_ON_ONCE(res.a0 != SMCCC_RET_SUCCESS)) { + ret = -EINVAL; + goto err; + } + + ret = res.a1; + mc.head = res.a2; + mc.nr_pages = res.a3; + +err: + free_hyp_memcache(&mc); + return ret; +} + +static unsigned long __pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target) +{ + struct kvm_hyp_memcache mc; + struct arm_smccc_res res; + unsigned long reclaimed; + + arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__pkvm_hyp_reclaim), id, target, &res); + if (WARN_ON_ONCE(res.a0 != SMCCC_RET_SUCCESS) || WARN_ON_ONCE(res.a1)) + return 0; + + init_hyp_memcache(&mc); + mc.head = res.a2; + mc.nr_pages = reclaimed = res.a3; + free_hyp_memcache(&mc); + + return reclaimed; +} + +static unsigned long pkvm_hyp_reclaim(enum pkvm_topup_id id, unsigned long target) +{ + unsigned long reclaimed = 0; + + while (reclaimed < target) { + /* Arbitrary limit to avoid blocking in EL2 for too long */ + unsigned long r = __pkvm_hyp_reclaim(id, min(target - reclaimed, 16)); + + if (!r) + break; + + reclaimed += r; + if (reclaimed >= target) + break; + + cond_resched(); + } + + return reclaimed; +} + +static unsigned long pkvm_hyp_reclaimable(enum pkvm_topup_id id) +{ + return kvm_call_hyp_nvhe(__pkvm_hyp_reclaimable, id); +} + static void __pkvm_destroy_hyp_vm(struct kvm *kvm) { if (pkvm_hyp_vm_is_created(kvm)) { @@ -100,30 +174,34 @@ static void __pkvm_destroy_hyp_vm(struct kvm *kvm) kvm->arch.pkvm.handle = 0; kvm->arch.pkvm.is_created = false; - free_hyp_memcache(&kvm->arch.pkvm.teardown_mc); free_hyp_memcache(&kvm->arch.pkvm.stage2_teardown_mc); } static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu) { - size_t hyp_vcpu_sz = PAGE_ALIGN(PKVM_HYP_VCPU_SIZE); pkvm_handle_t handle = vcpu->kvm->arch.pkvm.handle; - void *hyp_vcpu; int ret; - vcpu->arch.pkvm_memcache.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2; + init_hyp_stage2_memcache(&vcpu->arch.stage2_mc); - hyp_vcpu = alloc_pages_exact(hyp_vcpu_sz, GFP_KERNEL_ACCOUNT); - if (!hyp_vcpu) - return -ENOMEM; + ret = pkvm_call_hyp_req(__pkvm_init_vcpu, handle, vcpu); + if (ret) + return ret; - ret = kvm_call_hyp_nvhe(__pkvm_init_vcpu, handle, vcpu, hyp_vcpu); - if (!ret) - vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED); - else - free_pages_exact(hyp_vcpu, hyp_vcpu_sz); + /* + * Mirror EL2's seeding of power_state from mp_state. The hyp vCPU is + * published, so take mp_state_lock against kvm_psci_vcpu_on(). + */ + if (vcpu_is_protected(vcpu)) { + spin_lock(&vcpu->arch.mp_state_lock); + if (kvm_arm_vcpu_stopped(vcpu)) + WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_UNINITIALIZED); + spin_unlock(&vcpu->arch.mp_state_lock); + } - return ret; + vcpu_set_flag(vcpu, VCPU_PKVM_FINALIZED); + + return 0; } /* @@ -138,8 +216,8 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu) */ static int __pkvm_create_hyp_vm(struct kvm *kvm) { - size_t pgd_sz, hyp_vm_sz; - void *pgd, *hyp_vm; + size_t pgd_sz; + void *pgd; int ret; if (kvm->created_vcpus < 1) @@ -156,28 +234,15 @@ static int __pkvm_create_hyp_vm(struct kvm *kvm) if (!pgd) return -ENOMEM; - /* Allocate memory to donate to hyp for vm and vcpu pointers. */ - hyp_vm_sz = PAGE_ALIGN(size_add(PKVM_HYP_VM_SIZE, - size_mul(sizeof(void *), - kvm->created_vcpus))); - hyp_vm = alloc_pages_exact(hyp_vm_sz, GFP_KERNEL_ACCOUNT); - if (!hyp_vm) { - ret = -ENOMEM; - goto free_pgd; - } - - /* Donate the VM memory to hyp and let hyp initialize it. */ - ret = kvm_call_hyp_nvhe(__pkvm_init_vm, kvm, hyp_vm, pgd); + ret = pkvm_call_hyp_req(__pkvm_init_vm, kvm, pgd); if (ret) - goto free_vm; + goto free_pgd; kvm->arch.pkvm.is_created = true; - kvm->arch.pkvm.stage2_teardown_mc.flags |= HYP_MEMCACHE_ACCOUNT_STAGE2; + init_hyp_stage2_memcache(&kvm->arch.pkvm.stage2_teardown_mc); kvm_account_pgtable_pages(pgd, pgd_sz / PAGE_SIZE); return 0; -free_vm: - free_pages_exact(hyp_vm, hyp_vm_sz); free_pgd: free_pages_exact(pgd, pgd_sz); return ret; @@ -229,10 +294,9 @@ void pkvm_destroy_hyp_vm(struct kvm *kvm) mutex_unlock(&kvm->arch.config_lock); } -int pkvm_init_host_vm(struct kvm *kvm, unsigned long type) +int pkvm_init_host_vm(struct kvm *kvm) { int ret; - bool protected = type & KVM_VM_TYPE_ARM_PROTECTED; /* Reserve the VM in hyp and obtain a hyp handle for the VM. */ ret = kvm_call_hyp_nvhe(__pkvm_reserve_vm); @@ -240,8 +304,7 @@ int pkvm_init_host_vm(struct kvm *kvm, unsigned long type) return ret; kvm->arch.pkvm.handle = ret; - kvm->arch.pkvm.is_protected = protected; - if (protected) { + if (kvm_vm_is_protected(kvm)) { pr_warn_once("kvm: protected VMs are experimental and for development only, tainting kernel\n"); add_taint(TAINT_USER, LOCKDEP_STILL_OK); } @@ -270,8 +333,55 @@ static int __init pkvm_drop_host_privileges(void) return ret; } +void __init pkvm_selftests(void) +{ +#ifdef CONFIG_NVHE_EL2_DEBUG + int ret = pkvm_call_hyp_req(__pkvm_hyp_alloc_selftest); + unsigned long reclaimed; + + reclaimed = pkvm_hyp_reclaim(PKVM_TOPUP_HYP_ALLOC_SELFTEST, ULONG_MAX); + + /* On failure, not all the pages may be reclaimable */ + if (!ret) + WARN_ON(reclaimed != 6 /* SELFTEST_MAX_PAGES */); + else + kvm_err("pKVM hyp allocator selftest failed (%d)\n", ret); +#endif +} + +static unsigned long pkvm_shrinker_count(struct shrinker *shrink, struct shrink_control *sc) +{ + unsigned long reclaimable = 0; + int id; + + for (id = 0; id < NR_PKVM_TOPUP_HYP_IDS; id++) + reclaimable += pkvm_hyp_reclaimable(id); + + return reclaimable ?: SHRINK_EMPTY; +} + +static unsigned long pkvm_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + unsigned long reclaimed = 0; + int id; + + sc->nr_scanned = 0; + + for (id = 0; id < NR_PKVM_TOPUP_HYP_IDS; id++) { + unsigned long r = pkvm_hyp_reclaim(id, sc->nr_to_scan - sc->nr_scanned); + + reclaimed += r; + sc->nr_scanned += r; + if (sc->nr_scanned >= sc->nr_to_scan) + break; + } + + return reclaimed ?: SHRINK_STOP; +} + static int __init finalize_pkvm(void) { + struct shrinker *pkvm_shrinker; int ret; if (!is_protected_kvm_enabled() || !is_kvm_arm_initialised()) @@ -287,10 +397,21 @@ static int __init finalize_pkvm(void) kmemleak_free_part_phys(hyp_mem_base, hyp_mem_size); ret = pkvm_drop_host_privileges(); - if (ret) + if (ret) { pr_err("Failed to finalize Hyp protection: %d\n", ret); + return ret; + } - return ret; + pkvm_shrinker = shrinker_alloc(0, "pkvm"); + if (pkvm_shrinker) { + pkvm_shrinker->count_objects = pkvm_shrinker_count; + pkvm_shrinker->scan_objects = pkvm_shrinker_scan; + shrinker_register(pkvm_shrinker); + } else { + kvm_err("Failed to register shrinker for pKVM\n"); + } + + return 0; } device_initcall_sync(finalize_pkvm); @@ -414,6 +535,9 @@ int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 end = addr + size; int ret; + if (WARN_ON_ONCE(!PAGE_ALIGNED(addr | size))) + return -EINVAL; + lockdep_assert_held_write(&kvm->mmu_lock); mapping = pkvm_mapping_iter_first(&pgt->pkvm_mappings, addr, end - 1); @@ -594,3 +718,31 @@ bool pkvm_force_reclaim_guest_page(phys_addr_t phys) return !ret || ret == -EAGAIN; } + +static int pkvm_handle_hyp_req(struct pkvm_hyp_req *req) +{ + int ret = -EINVAL; + + switch (req->type) { + case PKVM_HYP_REQ_HYP_ALLOC: + ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC, req->mem.nr_pages); + break; + case PKVM_HYP_REQ_HYP_ALLOC_SELFTEST: + ret = pkvm_hyp_topup(PKVM_TOPUP_HYP_ALLOC_SELFTEST, req->mem.nr_pages); + break; + } + + trace_kvm_handle_pkvm_hyp_req(req, ret); + + return ret; +} + +int __pkvm_handle_smccc_req(struct arm_smccc_res *res) +{ + struct pkvm_hyp_req req; + + if (smccc_to_pkvm_hyp_req(&req, res)) + return pkvm_handle_hyp_req(&req); + + return res->a1; +} diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c index 5b1af7e2176f..e2b28dbc0451 100644 --- a/arch/arm64/kvm/pmu-emul.c +++ b/arch/arm64/kvm/pmu-emul.c @@ -50,8 +50,11 @@ static u32 __kvm_pmu_event_mask(unsigned int pmuver) case ID_AA64DFR0_EL1_PMUVer_V3P5: case ID_AA64DFR0_EL1_PMUVer_V3P7: return GENMASK(15, 0); - default: /* Shouldn't be here, just for sanity */ + default: + /* Shouldn't be here, just for sanity */ WARN_ONCE(1, "Unknown PMU version %d\n", pmuver); + fallthrough; + case ID_AA64DFR0_EL1_PMUVer_NI: return 0; } } diff --git a/arch/arm64/kvm/psci.c b/arch/arm64/kvm/psci.c index e3db84400d1f..b01f8d44033f 100644 --- a/arch/arm64/kvm/psci.c +++ b/arch/arm64/kvm/psci.c @@ -63,7 +63,12 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) return PSCI_RET_INVALID_PARAMS; spin_lock(&vcpu->arch.mp_state_lock); - if (!kvm_arm_vcpu_stopped(vcpu)) { + /* + * A protected vCPU is off once the host has acted on EL2's CPU_OFF. + * STOPPED before that is the VMM's pause. + */ + if (!kvm_arm_vcpu_stopped(vcpu) || + (vcpu_is_protected(vcpu) && !kvm_pkvm_vcpu_is_powered_off(vcpu))) { if (kvm_psci_version(source_vcpu) != KVM_ARM_PSCI_0_1) ret = PSCI_RET_ALREADY_ON; else @@ -161,7 +166,9 @@ static void kvm_prepare_system_event(struct kvm_vcpu *vcpu, u32 type, u64 flags) */ kvm_for_each_vcpu(i, tmp, vcpu->kvm) { spin_lock(&tmp->arch.mp_state_lock); - WRITE_ONCE(tmp->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED); + /* Keep EL2's off record for the guest's next CPU_ON. */ + if (!kvm_pkvm_vcpu_is_powered_off(tmp)) + WRITE_ONCE(tmp->arch.mp_state.mp_state, KVM_MP_STATE_STOPPED); spin_unlock(&tmp->arch.mp_state_lock); } kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_SLEEP); diff --git a/arch/arm64/kvm/ptdump.c b/arch/arm64/kvm/ptdump.c index 69899797dbad..9f0aebacea5b 100644 --- a/arch/arm64/kvm/ptdump.c +++ b/arch/arm64/kvm/ptdump.c @@ -17,7 +17,7 @@ #define MARKERS_LEN 2 #define KVM_PGTABLE_MAX_LEVELS (KVM_PGTABLE_LAST_LEVEL + 1) -#define S2FNAMESZ sizeof("0x0123456789abcdef-0x0123456789abcdef-s2-disabled") +#define S2FNAMESZ sizeof("nested_mmu0000") struct kvm_ptdump_guest_state { struct kvm_s2_mmu *mmu; @@ -115,15 +115,28 @@ static int kvm_ptdump_build_levels(struct ptdump_pg_level *level, u32 start_lvl) static struct kvm_ptdump_guest_state *kvm_ptdump_parser_create(struct kvm_s2_mmu *mmu) { + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); struct kvm_ptdump_guest_state *st; - struct kvm_pgtable *pgtable = mmu->pgt; + s8 start_level; int ret; + /* + * We only need the pgt start level to initialize the ptdump, get it + * while holding the mmu_lock. It's fine if the pgt gets freed + * afterwards, we'll check again when doing the actual dump. + */ + scoped_guard(read_lock, &kvm->mmu_lock) { + if (mmu->pgt) + start_level = mmu->pgt->start_level; + else + return ERR_PTR(-ENOENT); + } + st = kzalloc_obj(struct kvm_ptdump_guest_state, GFP_KERNEL_ACCOUNT); if (!st) return ERR_PTR(-ENOMEM); - ret = kvm_ptdump_build_levels(&st->level[0], pgtable->start_level); + ret = kvm_ptdump_build_levels(&st->level[0], start_level); if (ret) { kfree(st); return ERR_PTR(ret); @@ -149,6 +162,9 @@ static int kvm_ptdump_guest_show(struct seq_file *m, void *unused) }; guard(write_lock)(&kvm->mmu_lock); + if (!mmu->pgt) + return 0; + st->parser_state = (struct ptdump_pg_state) { .marker = &st->ipa_marker[0], .end_address = BIT(mmu->pgt->ia_bits), @@ -157,6 +173,15 @@ static int kvm_ptdump_guest_show(struct seq_file *m, void *unused) .seq = m, }; + if (kvm_is_nested_s2_mmu(kvm, mmu)) { + if (kvm_s2_mmu_valid(mmu)) + seq_printf(m, "VTCR: 0x%016llx VTTBR: 0x%016llx s2: %s\n", + mmu->tlb_vtcr, mmu->tlb_vttbr, + mmu->nested_stage2_enabled ? "enabled" : "disabled"); + else + return 0; + } + ret = kvm_pgtable_walk(mmu->pgt, 0, BIT(mmu->pgt->ia_bits), &walker); if (ret) return ret; @@ -211,17 +236,27 @@ static const struct file_operations kvm_ptdump_guest_fops = { static int kvm_pgtable_range_show(struct seq_file *m, void *unused) { - struct kvm_pgtable *pgtable = m->private; + struct kvm_s2_mmu *mmu = m->private; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); + + guard(read_lock)(&kvm->mmu_lock); + + if (mmu->pgt) + seq_printf(m, "%2u\n", mmu->pgt->ia_bits); - seq_printf(m, "%2u\n", pgtable->ia_bits); return 0; } static int kvm_pgtable_levels_show(struct seq_file *m, void *unused) { - struct kvm_pgtable *pgtable = m->private; + struct kvm_s2_mmu *mmu = m->private; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); + + guard(read_lock)(&kvm->mmu_lock); + + if (mmu->pgt) + seq_printf(m, "%1d\n", KVM_PGTABLE_MAX_LEVELS - mmu->pgt->start_level); - seq_printf(m, "%1d\n", KVM_PGTABLE_MAX_LEVELS - pgtable->start_level); return 0; } @@ -230,15 +265,12 @@ static int kvm_pgtable_debugfs_open(struct inode *m, struct file *file, { struct kvm_s2_mmu *mmu = m->i_private; struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); - struct kvm_pgtable *pgtable; int ret; if (!kvm_get_kvm_safe(kvm)) return -ENOENT; - pgtable = mmu->pgt; - - ret = single_open(file, show, pgtable); + ret = single_open(file, show, mmu); if (ret < 0) kvm_put_kvm(kvm); return ret; @@ -276,30 +308,23 @@ static const struct file_operations kvm_pgtable_levels_fops = { .release = kvm_pgtable_debugfs_close, }; -void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu) +void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu, int idx) { - struct dentry *dent; char file_name[S2FNAMESZ]; - snprintf(file_name, sizeof(file_name), "0x%016llx-0x%016llx-s2-%sabled", - mmu->tlb_vttbr, - mmu->tlb_vtcr, - mmu->nested_stage2_enabled ? "en" : "dis"); - - dent = debugfs_create_file(file_name, 0400, - mmu->arch->debugfs_nv_dentry, mmu, - &kvm_ptdump_guest_fops); + snprintf(file_name, sizeof(file_name), "nested_mmu%d", idx); - mmu->shadow_pt_debugfs_dentry = dent; -} - -void kvm_nested_s2_ptdump_remove_debugfs(struct kvm_s2_mmu *mmu) -{ - debugfs_remove(mmu->shadow_pt_debugfs_dentry); + debugfs_create_file(file_name, 0400, + mmu->arch->debugfs_nv_dentry, mmu, + &kvm_ptdump_guest_fops); } void kvm_s2_ptdump_create_debugfs(struct kvm *kvm) { + /* pKVM handles the page-tables in the hypervisor */ + if (is_protected_kvm_enabled()) + return; + debugfs_create_file("stage2_page_tables", 0400, kvm->debugfs_dentry, &kvm->arch.mmu, &kvm_ptdump_guest_fops); debugfs_create_file("ipa_range", 0400, kvm->debugfs_dentry, diff --git a/arch/arm64/kvm/pvtime.c b/arch/arm64/kvm/pvtime.c index 4ceabaa4c30b..053fe831fa54 100644 --- a/arch/arm64/kvm/pvtime.c +++ b/arch/arm64/kvm/pvtime.c @@ -67,9 +67,9 @@ gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu) return base; } -bool kvm_arm_pvtime_supported(void) +bool kvm_arm_pvtime_supported(struct kvm *kvm) { - return !!sched_info_on(); + return !!sched_info_on() && (!kvm || !kvm_vm_is_protected(kvm)); } int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, @@ -81,7 +81,7 @@ int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, int ret = 0; int idx; - if (!kvm_arm_pvtime_supported() || + if (!kvm_arm_pvtime_supported(kvm) || attr->attr != KVM_ARM_VCPU_PVTIME_IPA) return -ENXIO; @@ -110,7 +110,7 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, u64 __user *user = (u64 __user *)attr->addr; u64 ipa; - if (!kvm_arm_pvtime_supported() || + if (!kvm_arm_pvtime_supported(vcpu->kvm) || attr->attr != KVM_ARM_VCPU_PVTIME_IPA) return -ENXIO; @@ -126,7 +126,7 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu, { switch (attr->attr) { case KVM_ARM_VCPU_PVTIME_IPA: - if (kvm_arm_pvtime_supported()) + if (kvm_arm_pvtime_supported(vcpu->kvm)) return 0; } return -ENXIO; diff --git a/arch/arm64/kvm/stacktrace.c b/arch/arm64/kvm/stacktrace.c index 9724c320126b..69377195e18b 100644 --- a/arch/arm64/kvm/stacktrace.c +++ b/arch/arm64/kvm/stacktrace.c @@ -198,9 +198,6 @@ static void hyp_dump_backtrace(unsigned long hyp_offset) } #ifdef CONFIG_PKVM_STACKTRACE -DECLARE_KVM_NVHE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], - pkvm_stacktrace); - /* * pkvm_dump_backtrace - Dump the protected nVHE HYP backtrace. * diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index a2f4e769a428..e9cf6cc34034 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1029,7 +1029,7 @@ static unsigned int hidden_visibility(const struct kvm_vcpu *vcpu, static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { - if (kvm_vcpu_has_pmu(vcpu)) + if (kvm_has_feat(vcpu->kvm, ID_AA64DFR0_EL1, PMUVer, IMP)) return 0; return REG_HIDDEN; @@ -1059,10 +1059,6 @@ static u64 reset_pmevcntr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) static u64 reset_pmevtyper(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { - /* This thing will UNDEF, who cares about the reset value? */ - if (!kvm_vcpu_has_pmu(vcpu)) - return 0; - reset_unknown(vcpu, r); __vcpu_rmw_sys_reg(vcpu, r->reg, &=, kvm_pmu_evtyper_mask(vcpu->kvm)); @@ -2512,9 +2508,10 @@ static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, /* * Once the VM has started the ID registers are immutable. Reject any - * write that does not match the final register value. + * write that does not match the final register value once we have + * got far enough into first running the VM to use the values. */ - if (kvm_vm_has_ran_once(vcpu->kvm)) { + if (kvm_id_regs_final(vcpu->kvm)) { if (val != read_id_reg(vcpu, rd)) ret = -EBUSY; else @@ -2548,7 +2545,7 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val) lockdep_assert_held(&kvm->arch.config_lock); - if (KVM_BUG_ON(kvm_vm_has_ran_once(kvm) || !p, kvm)) + if (KVM_BUG_ON(kvm_id_regs_final(kvm) || !p, kvm)) return; *p = val; @@ -3244,10 +3241,10 @@ static int set_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, return -EINVAL; /* - * Once the VM has started the ID registers are immutable. Reject the - * write if userspace tries to change it. + * Once we have been far enough into starting the VM the ID registers + * are immutable. Reject the write if userspace tries to change it. */ - if (kvm_vm_has_ran_once(kvm)) + if (kvm_id_regs_final(kvm)) return -EBUSY; /* @@ -5862,26 +5859,17 @@ out: } /* - * Perform last adjustments to the ID registers that are implied by the + * Do system register finalization that is shared by the whole guest. This + * includes last adjustments to the ID registers that are implied by the * configuration outside of the ID regs themselves, as well as any * initialisation that directly depend on these ID registers (such as * RES0/RES1 behaviours). This is not the place to configure traps though. - * - * Because this can be called once per CPU, changes must be idempotent. */ -int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu) +static int kvm_vm_finalize_sys_regs(struct kvm *kvm) { - struct kvm *kvm = vcpu->kvm; - - guard(mutex)(&kvm->arch.config_lock); - - if (vcpu_has_nv(vcpu)) { - int ret = kvm_init_nv_sysregs(vcpu); - if (ret) - return ret; - } + lockdep_assert_held(&kvm->arch.config_lock); - if (kvm_vm_has_ran_once(kvm)) + if (kvm_id_regs_final(kvm)) return 0; /* @@ -5929,12 +5917,37 @@ int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu) kvm_vgic_finalize_idregs(kvm); } + set_bit(KVM_ARCH_FLAG_ID_REGS_FINAL, &kvm->arch.flags); + + return 0; +} + +/* + * Because this can be called once per CPU, changes must be idempotent. + */ +int kvm_vcpu_finalize_sys_regs(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + int ret; + + guard(mutex)(&kvm->arch.config_lock); + + ret = kvm_vm_finalize_sys_regs(kvm); + if (ret) + return ret; + + if (vcpu_has_nv(vcpu)) { + ret = kvm_init_nv_sysregs(vcpu); + if (ret) + return ret; + } + return 0; } int __init kvm_sys_reg_table_init(void) { - const struct sys_reg_desc *gicv3_regs; + const struct sys_reg_desc *gicv3_regs, *gicv5_regs; bool valid = true; unsigned int i, sz; int ret = 0; @@ -5947,8 +5960,12 @@ int __init kvm_sys_reg_table_init(void) valid &= check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs), false); valid &= check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs), false); + /* The GICv3 system registers... */ gicv3_regs = vgic_v3_get_sysreg_table(&sz); valid &= check_sysreg_table(gicv3_regs, sz, false); + /* ...and the GICv5 system registers. */ + gicv5_regs = vgic_v5_get_sysreg_table(&sz); + valid &= check_sysreg_table(gicv5_regs, sz, false); if (!valid) return -EINVAL; diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index bd56a45abbf9..a3cccad2766f 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -254,7 +254,7 @@ int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index); -int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu); +int kvm_vcpu_finalize_sys_regs(struct kvm_vcpu *vcpu); #define AA32(_x) .aarch32_map = AA32_##_x #define Op0(_x) .Op0 = _x diff --git a/arch/arm64/kvm/trace_pkvm.h b/arch/arm64/kvm/trace_pkvm.h new file mode 100644 index 000000000000..493225e3282d --- /dev/null +++ b/arch/arm64/kvm/trace_pkvm.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_PKVM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_PKVM_ARM64_KVM_H + +#include <linux/tracepoint.h> +#include <asm/kvm_pkvm.h> + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm + +TRACE_DEFINE_ENUM(PKVM_HYP_NO_REQ); +TRACE_DEFINE_ENUM(PKVM_HYP_REQ_HYP_ALLOC); + +#define PKVM_HYP_REQ_TYPES \ + { PKVM_HYP_NO_REQ, "NO_REQ" }, \ + { PKVM_HYP_REQ_HYP_ALLOC, "HYP_ALLOC" } + +TRACE_EVENT(kvm_handle_pkvm_hyp_req, + TP_PROTO(struct pkvm_hyp_req *req, int ret), + TP_ARGS(req, ret), + + TP_STRUCT__entry( + __field(u8, type) + __field(int, ret) + ), + + TP_fast_assign( + __entry->type = req->type; + __entry->ret = ret; + ), + + TP_printk("type: %s ret: %d", + __print_symbolic(__entry->type, PKVM_HYP_REQ_TYPES), + __entry->ret) +); + +#endif /* _TRACE_PKVM_ARM64_KVM_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE trace_pkvm + +/* This part must be outside protection */ +#include <trace/define_trace.h> diff --git a/arch/arm64/kvm/vgic-sys-reg-v5.c b/arch/arm64/kvm/vgic-sys-reg-v5.c new file mode 100644 index 000000000000..6b255eacbf60 --- /dev/null +++ b/arch/arm64/kvm/vgic-sys-reg-v5.c @@ -0,0 +1,519 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025, 2026 Arm Ltd. + */ + +/* + * VGICv5 system registers handling functions for AArch64 mode + */ + +#include <linux/irqchip/arm-gic-v5.h> + +#include <linux/kvm.h> +#include <linux/kvm_host.h> +#include <linux/wordpart.h> + +#include <asm/kvm_emulate.h> + +#include "vgic/vgic.h" +#include "sys_regs.h" + +#define ICC_PPI_PRIORITYR_PRIORITY_MASK REPEAT_BYTE(0x1f) + +static int set_gic_apr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + /* The upper 32 bits are RES0 */ + cpu_if->vgic_apr = val & ~ICC_APR_EL1_RES0; + + return 0; +} + +static int get_gic_apr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 *val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + *val = cpu_if->vgic_apr; + + return 0; +} + +static int set_gic_cr0(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + /* + * We only support setting the ICC_CR0_EL1.En bit, which is actually + * stored in the VMCR. + */ + FIELD_MODIFY(FEAT_GCIE_ICH_VMCR_EL2_EN, &cpu_if->vgic_vmcr, + FIELD_GET(ICC_CR0_EL1_EN, val)); + + return 0; +} + +static int get_gic_cr0(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 *val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + /* + * PID only applies if EL3 is present. Same applies to IPPT. Hence, + * those fields are always presented as 0. + * + * We always present the link as connected and idle: + * (LINK = 1, LINK_IDLE = 1). + */ + *val = FIELD_PREP(ICC_CR0_EL1_EN, + FIELD_GET(FEAT_GCIE_ICH_VMCR_EL2_EN, cpu_if->vgic_vmcr)); + *val |= ICC_CR0_EL1_LINK_MASK; + *val |= ICC_CR0_EL1_LINK_IDLE_MASK; + + return 0; +} + +static int set_gic_pcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + /* Set the VPMR field in the VMCR */ + FIELD_MODIFY(FEAT_GCIE_ICH_VMCR_EL2_VPMR, &cpu_if->vgic_vmcr, + FIELD_GET(ICC_PCR_EL1_PRIORITY, val)); + + return 0; +} + +static int get_gic_pcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 *val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + *val = FIELD_PREP(ICC_PCR_EL1_PRIORITY, + FIELD_GET(FEAT_GCIE_ICH_VMCR_EL2_VPMR, cpu_if->vgic_vmcr)); + + return 0; +} + +static int set_gic_icsr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + cpu_if->vgic_icsr = val & ~ICC_ICSR_EL1_RES0; + + return 0; +} + +static int get_gic_icsr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 *val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + + *val = cpu_if->vgic_icsr; + + return 0; +} + +/* + * Helper macro to iterate over a range of PPIs and execute some code (to either + * extract or set the vgic_irq state). This is used when `get`-ing the PPI + * ENABLER, ACTIVER, PENDR and when setting the PRIORITYR state. + * + * vcpu: Pointer to struct kvm_vcpu (to which these PPIs belong) + * r: The register index. 0 or 1 for all except PRIORITYR (which is 0-15) + * nr: The number of PPIs iterated over. 64 for all but PRIORITYR (which is 8) + * code: The code snippet to execute for each vgic_irq + */ +#define for_ppi_state(vcpu, r, nr, code) \ + do { \ + struct kvm_vcpu *__vcpu = (vcpu); \ + int __r = (r); \ + int __nr = (nr); \ + \ + for (int i = 0; i < __nr; i++) { \ + u32 id = vgic_v5_make_ppi(__r * __nr + i); \ + struct vgic_irq *irq; \ + \ + irq = vgic_get_vcpu_irq(__vcpu, id); \ + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { \ + code; \ + } \ + vgic_put_irq(__vcpu->kvm, irq); \ + } \ + } while (0) + +static int set_gic_ppi_enabler(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + int i, start, end, reg = r->Op2 % 2; + + /* + * If we're only handling architected PPIs and the guest writes to the + * enable for the non-architected PPIs, we just return as there's + * nothing to do at all. We don't even allocate the storage for them in + * this case. + */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) + return 0; + + /* + * Merge the raw guest write into our bitmap at an offset of either 0 or + * 64. + * + * Note that there is *NO* masking applied - the enable state is written + * unfiltered. The assumption is that userspace uses this interface to + * set initial state before the guest runs, and then the exposed PPI + * mask is applied later, when vgic_v5_finalize_ppi_state() runs on + * first entry to each vCPU. If userspace chooses to set the enabler + * state later, it is fully capable of breaking the illusion we provided + * to the guest by exposing register state (and PPIs) to the guest that + * were not initially exposed. Good luck! + */ + bitmap_write(cpu_if->vgic_ppi_enabler, val, 64 * reg, 64); + + /* + * Sync the change in enable states to the vgic_irqs for the written + * register slice. + */ + start = reg * 64; + end = min(start + 64, VGIC_V5_NR_PRIVATE_IRQS); + for (i = start; i < end; i++) { + u32 intid = vgic_v5_make_ppi(i); + struct vgic_irq *irq; + + irq = vgic_get_vcpu_irq(vcpu, intid); + + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) + irq->enabled = test_bit(i, cpu_if->vgic_ppi_enabler); + + vgic_put_irq(vcpu->kvm, irq); + } + + return 0; +} + +static int get_gic_ppi_enabler(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 *val) +{ + unsigned long enabler = 0; + int reg = r->Op2 % 2; + + /* If we only support architected PPIs, return 0 */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) { + *val = 0; + return 0; + } + + /* Iterate over each struct vgic_irq to build the ENABLER value. */ + for_ppi_state(vcpu, reg, 64, __assign_bit(i % 64, &enabler, irq->enabled)); + + *val = enabler; + + return 0; +} + +static int set_gic_ppi_activer(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + int i, start, end, reg = r->Op2 % 2; + + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) + return 0; + + /* + * Store the raw guest write. The exposed PPI mask is applied later, + * when vgic_v5_finalize_ppi_state() runs on first entry to each + * vCPU. See comment on set_gic_ppi_enabler() for details. + */ + bitmap_write(cpu_if->vgic_ppi_activer, val, 64 * reg, 64); + + start = reg * 64; + end = min(start + 64, VGIC_V5_NR_PRIVATE_IRQS); + for (i = start; i < end; i++) { + u32 intid = vgic_v5_make_ppi(i); + struct vgic_irq *irq; + + irq = vgic_get_vcpu_irq(vcpu, intid); + + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) + irq->active = test_bit(i, cpu_if->vgic_ppi_activer); + + vgic_put_irq(vcpu->kvm, irq); + } + + return 0; +} + +static int get_gic_ppi_activer(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 *val) +{ + unsigned long activer = 0; + int reg = r->Op2 % 2; + + /* If we only support architected PPIs, return 0 */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) { + *val = 0; + return 0; + } + + /* Iterate over each struct vgic_irq to build the ACTIVER value. */ + for_ppi_state(vcpu, reg, 64, __assign_bit(i % 64, &activer, irq->active)); + + *val = activer; + + return 0; +} + +static int set_gic_ppi_pendr(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 val) +{ + int i, start, end, reg = r->Op2 % 2; + + /* If we only support architected PPIs, return */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) + return 0; + + /* + * Update each struct vgic_irq with the pending state, treating Level + * and Edge interrupts differently. The exposed PPI mask is applied + * later, when vgic_v5_finalize_ppi_state() runs on first entry to each + * vCPU. See comment on set_gic_ppi_enabler() for details. + */ + start = reg * 64; + end = min(start + 64, VGIC_V5_NR_PRIVATE_IRQS); + for (i = start; i < end; i++) { + u32 intid = vgic_v5_make_ppi(i); + struct vgic_irq *irq; + + irq = vgic_get_vcpu_irq(vcpu, intid); + + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { + bool level = !!(val & BIT_ULL(i - start)); + + if (irq->config == VGIC_CONFIG_LEVEL) + irq->line_level = level; + else + irq->pending_latch = level; + } + + vgic_put_irq(vcpu->kvm, irq); + } + + /* + * The pending state is generated from the vgic_irqs on each guest + * entry. Therefore, we don't store the raw value written anywhere in + * the case of userspace PPI_PENDRx_EL1 writes. + */ + + return 0; +} + +static int get_gic_ppi_pendr(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 *val) +{ + unsigned long pendr = 0; + int reg = r->Op2 % 2; + + /* If we only support architected PPIs, return 0 */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg == 1) { + *val = 0; + return 0; + } + + /* Iterate over each struct vgic_irq to build the PENDR value. */ + for_ppi_state(vcpu, reg, 64, { + if (irq_is_pending(irq)) + __assign_bit(i % 64, &pendr, 1); + }); + + *val = pendr; + + return 0; +} + +static int set_gic_ppi_priorityr(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + int reg = ((r->CRm & 0x1) << 3) + r->Op2; + + /* If we only support architected PPIs, return */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg > 7) + return 0; + + val &= ICC_PPI_PRIORITYR_PRIORITY_MASK; + + /* + * Although priorities are not regularly synced back to the vgic_irq + * state, they are explicitly synced back here. This is to ensure that + * any pending PPIs are evaluated correctly when first running the guest + * after setting the state. + */ + for_ppi_state(vcpu, reg, 8, + irq->priority = (u8)(val >> (8 * i)); + ); + + /* + * Update the state that will be written to the ICH_PPI_PRIORITYRx_EL2 + * on next guest entry. + */ + cpu_if->vgic_ppi_priorityr[reg] = val; + + return 0; +} + +static int get_gic_ppi_priorityr(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r, u64 *val) +{ + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + int reg = ((r->CRm & 0x1) << 3) + r->Op2; + + /* If we only support architected PPIs, return 0 */ + if (VGIC_V5_NR_PRIVATE_IRQS == 64 && reg > 7) { + *val = 0; + return 0; + } + + /* + * The priorities are only synced back to the vgic_irq state when the + * vcpu is entering WFI (KVM only needs to know the priorities when + * evaluating if there are pending PPI interrupts for a vcpu). The raw + * register ICH_PPI_PRIORITYRx_EL1 state is simply saved and restored + * blindly. This state is just returned as it contains the most recent + * priorities written by the guest. + */ + *val = cpu_if->vgic_ppi_priorityr[reg]; + + return 0; +} + +/* + * The following registers are NOT supported: + * + * - ICC_HAPR_EL1 + * The value of this is directly generated by the GICv5 hardware based on + * the ICC_APR_EL1 when the guest is running. + * - ICC_IAFFIDR_EL1 + * The IAFFID for a GICv5 VPE is the same as the VPE ID, which is the index + * into the in-memory VPE Table. This is not configurable, and instead we + * rely on userspace recreating the VPEs in the same order prior to + * restoring guest state. + * - ICC_PPI_CACTIVER<n>_EL1 + * Only raw state writes are supported via the S(et) variant. + * - ICC_PPI_CPENDR<n>_EL1 + * Only raw state writes are supported via the S(et) variant. + */ +static const struct sys_reg_desc gic_v5_icc_reg_descs[] = { + { SYS_DESC(SYS_ICC_ICSR_EL1), + .set_user = set_gic_icsr, .get_user = get_gic_icsr, }, + { SYS_DESC(SYS_ICC_PPI_ENABLER0_EL1), + .set_user = set_gic_ppi_enabler, .get_user = get_gic_ppi_enabler, }, + { SYS_DESC(SYS_ICC_PPI_ENABLER1_EL1), + .set_user = set_gic_ppi_enabler, .get_user = get_gic_ppi_enabler, }, + /* + * Only ICC_SACTIVER<n>_EL1 is exposed to the guest. This is treated as + * a *RAW* write of register state for writes. + */ + { SYS_DESC(SYS_ICC_PPI_SACTIVER0_EL1), + .set_user = set_gic_ppi_activer, .get_user = get_gic_ppi_activer, }, + { SYS_DESC(SYS_ICC_PPI_SACTIVER1_EL1), + .set_user = set_gic_ppi_activer, .get_user = get_gic_ppi_activer, }, + /* + * Only ICC_SPENDR<n>_EL1 is exposed to the guest. This is treated as + * a *RAW* write of register state for writes. + */ + { SYS_DESC(SYS_ICC_PPI_SPENDR0_EL1), + .set_user = set_gic_ppi_pendr, .get_user = get_gic_ppi_pendr, }, + { SYS_DESC(SYS_ICC_PPI_SPENDR1_EL1), + .set_user = set_gic_ppi_pendr, .get_user = get_gic_ppi_pendr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR0_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR1_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR2_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR3_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR4_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR5_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR6_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR7_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR8_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR9_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR10_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR11_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR12_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR13_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR14_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_PPI_PRIORITYR15_EL1), + .set_user = set_gic_ppi_priorityr, .get_user = get_gic_ppi_priorityr, }, + { SYS_DESC(SYS_ICC_APR_EL1), + .set_user = set_gic_apr, .get_user = get_gic_apr, }, + { SYS_DESC(SYS_ICC_CR0_EL1), + .set_user = set_gic_cr0, .get_user = get_gic_cr0, }, + { SYS_DESC(SYS_ICC_PCR_EL1), + .set_user = set_gic_pcr, .get_user = get_gic_pcr, }, +}; + +const struct sys_reg_desc *vgic_v5_get_sysreg_table(unsigned int *sz) +{ + *sz = ARRAY_SIZE(gic_v5_icc_reg_descs); + return gic_v5_icc_reg_descs; +} + +static u64 attr_to_id(u64 attr) +{ + return ARM64_SYS_REG(FIELD_GET(KVM_REG_ARM_VGIC_SYSREG_OP0_MASK, attr), + FIELD_GET(KVM_REG_ARM_VGIC_SYSREG_OP1_MASK, attr), + FIELD_GET(KVM_REG_ARM_VGIC_SYSREG_CRN_MASK, attr), + FIELD_GET(KVM_REG_ARM_VGIC_SYSREG_CRM_MASK, attr), + FIELD_GET(KVM_REG_ARM_VGIC_SYSREG_OP2_MASK, attr)); +} + +int vgic_v5_has_cpu_sysregs_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) +{ + const struct sys_reg_desc *r; + + r = get_reg_by_id(attr_to_id(attr->attr), gic_v5_icc_reg_descs, + ARRAY_SIZE(gic_v5_icc_reg_descs)); + + if (r && !sysreg_hidden(vcpu, r)) + return 0; + + return -ENXIO; +} + +int vgic_v5_cpu_sysregs_uaccess(struct kvm_vcpu *vcpu, + struct kvm_device_attr *attr, + bool is_write) +{ + struct kvm_one_reg reg = { + .id = attr_to_id(attr->attr), + .addr = attr->addr, + }; + + if (is_write) + return kvm_sys_reg_set_user(vcpu, ®, gic_v5_icc_reg_descs, + ARRAY_SIZE(gic_v5_icc_reg_descs)); + else + return kvm_sys_reg_get_user(vcpu, ®, gic_v5_icc_reg_descs, + ARRAY_SIZE(gic_v5_icc_reg_descs)); +} diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index 4012df6002ea..b5ea1e39c686 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -84,6 +84,8 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) !kvm_vgic_global_state.can_emulate_gicv2) return -ENODEV; + if (kvm_vm_hyp_is_distrusting(kvm) && type != KVM_DEV_TYPE_ARM_VGIC_V3) + return -ENODEV; /* * Ensure mutual exclusion with vCPU creation and any vCPU ioctls by: * @@ -123,19 +125,34 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) goto out_unlock; } - kvm_for_each_vcpu(i, vcpu, kvm) { - if (vcpu_has_run_once(vcpu)) - goto out_unlock; - } + if (kvm_id_regs_final(kvm)) + goto out_unlock; ret = 0; - if (type == KVM_DEV_TYPE_ARM_VGIC_V2) + switch (type) { + case KVM_DEV_TYPE_ARM_VGIC_V2: kvm->max_vcpus = VGIC_V2_MAX_CPUS; - else if (type == KVM_DEV_TYPE_ARM_VGIC_V3) + break; + case KVM_DEV_TYPE_ARM_VGIC_V3: kvm->max_vcpus = VGIC_V3_MAX_CPUS; - else if (type == KVM_DEV_TYPE_ARM_VGIC_V5) - kvm->max_vcpus = min(VGIC_V5_MAX_CPUS, - kvm_vgic_global_state.max_gic_vcpus); + break; + case KVM_DEV_TYPE_ARM_VGIC_V5: + kvm->max_vcpus = kvm_vgic_global_state.max_gicv5_vcpus; + break; + } + + /* + * KVM_CREATE_VCPU enforces the model-specific limit if the VGIC has + * already been created. Apply the same limit to any existing vCPUs so + * that the result does not depend on the order in which userspace + * creates them. + */ + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vcpu->vcpu_id >= kvm->max_vcpus) { + ret = -E2BIG; + goto out_unlock; + } + } if (atomic_read(&kvm->online_vcpus) > kvm->max_vcpus) { ret = -E2BIG; @@ -154,6 +171,8 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) case KVM_DEV_TYPE_ARM_VGIC_V3: INIT_LIST_HEAD(&kvm->arch.vgic.rd_regions); break; + case KVM_DEV_TYPE_ARM_VGIC_V5: + kvm->arch.vgic.gicv5_vm.vm_id = VGIC_V5_VM_ID_INVAL; } /* @@ -168,29 +187,49 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) break; } - if (ret) { - kvm_for_each_vcpu(i, vcpu, kvm) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; - kfree(vgic_cpu->private_irqs); - vgic_cpu->private_irqs = NULL; - } - - kvm->arch.vgic.vgic_model = 0; - kvm->arch.vgic.in_kernel = false; - goto out_unlock; - } + if (ret) + goto out_free_private_irqs; if (type == KVM_DEV_TYPE_ARM_VGIC_V3) kvm->arch.vgic.nassgicap = system_supports_direct_sgis(); - /* - * We now know that we have a GICv5. The Arch Timer PPI interrupts may - * have been initialised at this stage, but will have done so assuming - * that we have an older GIC, meaning that the IntIDs won't be - * correct. We init them again, and this time they will be correct. - */ - if (type == KVM_DEV_TYPE_ARM_VGIC_V5) + if (type == KVM_DEV_TYPE_ARM_VGIC_V5) { + /* Allocate a vIRS for GICv5 systems */ + kvm->arch.vgic.vgic_v5_irs_data = kzalloc_obj(struct vgic_v5_irs, + GFP_KERNEL_ACCOUNT); + if (!kvm->arch.vgic.vgic_v5_irs_data) { + ret = -ENOMEM; + goto out_free_private_irqs; + } + + /* + * Initialization happens later, for now just explicitly + * disable the device and undef its base address. + */ + kvm->arch.vgic.vgic_v5_irs_data->vgic_v5_irs_base = VGIC_ADDR_UNDEF; + + /* + * We now know that we have a GICv5. The Arch Timer PPI + * interrupts may have been initialised at this stage, but will + * have done so assuming that we have an older GIC, meaning that + * the IntIDs won't be correct. We init them again, and this + * time they will be correct. + */ kvm_timer_init_vm(kvm); + } + + goto out_unlock; + +out_free_private_irqs: + kvm_for_each_vcpu(i, vcpu, kvm) { + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + kfree(vgic_cpu->private_irqs); + vgic_cpu->private_irqs = NULL; + } + + kvm->arch.vgic.in_kernel = false; + kvm->arch.vgic.vgic_model = 0; out_unlock: mutex_unlock(&kvm->arch.config_lock); @@ -397,15 +436,27 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) if (ret) return ret; - /* - * If we are creating a VCPU with a GICv3 we must also register the - * KVM io device for the redistributor that belongs to this VCPU. - */ - if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) { + switch (dist->vgic_model) { + case KVM_DEV_TYPE_ARM_VGIC_V3: + /* + * If we are creating a VCPU with a GICv3 we must also register + * the KVM io device for the redistributor that belongs to this + * VCPU. + */ mutex_lock(&vcpu->kvm->slots_lock); ret = vgic_register_redist_iodev(vcpu); mutex_unlock(&vcpu->kvm->slots_lock); + break; + case KVM_DEV_TYPE_ARM_VGIC_V5: + /* + * Ensure that it is possible to represent the + * userspace-allocated vcpu_id in the hardware-limited (or + * KVM-capped) VPE table used by GICv5. + */ + if (vcpu->vcpu_id >= kvm_vgic_global_state.max_gicv5_vcpus) + return -EINVAL; } + return ret; } @@ -465,6 +516,9 @@ int vgic_init(struct kvm *kvm) return ret; } } else { + if (!dist->nr_spis) + dist->nr_spis = VGIC_V5_DEFAULT_NR_SPIS; + ret = vgic_v5_init(kvm); if (ret) return ret; @@ -474,8 +528,12 @@ int vgic_init(struct kvm *kvm) kvm_vgic_vcpu_reset(vcpu); ret = kvm_vgic_setup_default_irq_routing(kvm); - if (ret) + if (ret) { + if (vgic_is_v5(kvm)) + vgic_v5_teardown(kvm); + return ret; + } vgic_debug_init(kvm); dist->initialized = true; @@ -496,16 +554,24 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm) dist->nr_spis = 0; dist->vgic_dist_base = VGIC_ADDR_UNDEF; - if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) { + switch (dist->vgic_model) { + case KVM_DEV_TYPE_ARM_VGIC_V2: + dist->vgic_cpu_base = VGIC_ADDR_UNDEF; + break; + case KVM_DEV_TYPE_ARM_VGIC_V3: list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) vgic_v3_free_redist_region(kvm, rdreg); INIT_LIST_HEAD(&dist->rd_regions); - } else { - dist->vgic_cpu_base = VGIC_ADDR_UNDEF; - } - if (vgic_supports_direct_irqs(kvm)) - vgic_v4_teardown(kvm); + if (vgic_supports_direct_irqs(kvm)) + vgic_v4_teardown(kvm); + break; + case KVM_DEV_TYPE_ARM_VGIC_V5: + vgic_v5_teardown(kvm); + kfree(dist->vgic_v5_irs_data); + dist->vgic_v5_irs_data = NULL; + break; + } xa_destroy(&dist->lpi_xa); } @@ -624,9 +690,8 @@ int vgic_lazy_init(struct kvm *kvm) int kvm_vgic_map_resources(struct kvm *kvm) { struct vgic_dist *dist = &kvm->arch.vgic; - bool needs_dist = true; enum vgic_type type; - gpa_t dist_base; + gpa_t dist_base, irs_base; int ret = 0; if (likely(smp_load_acquire(&dist->ready))) @@ -649,13 +714,12 @@ int kvm_vgic_map_resources(struct kvm *kvm) } else { ret = vgic_v5_map_resources(kvm); type = VGIC_V5; - needs_dist = false; } if (ret) goto out; - if (needs_dist) { + if (type != VGIC_V5) { dist_base = dist->vgic_dist_base; mutex_unlock(&kvm->arch.config_lock); @@ -665,7 +729,19 @@ int kvm_vgic_map_resources(struct kvm *kvm) goto out_slots; } } else { + irs_base = dist->vgic_v5_irs_data->vgic_v5_irs_base; mutex_unlock(&kvm->arch.config_lock); + + if (IS_VGIC_ADDR_UNDEF(irs_base)) { + ret = -ENXIO; + goto out_slots; + } + + ret = vgic_v5_register_irs_iodev(kvm, irs_base); + if (ret) { + kvm_err("Unable to register VGIC IRS MMIO regions\n"); + goto out_slots; + } } smp_store_release(&dist->ready, true); diff --git a/arch/arm64/kvm/vgic/vgic-irqfd.c b/arch/arm64/kvm/vgic/vgic-irqfd.c index 19a1094536e6..ee1a1b214f03 100644 --- a/arch/arm64/kvm/vgic/vgic-irqfd.c +++ b/arch/arm64/kvm/vgic/vgic-irqfd.c @@ -19,9 +19,14 @@ static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm, int irq_source_id, int level, bool line_status) { - unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS; + unsigned int spi_id; int ret; + if (kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V5) + spi_id = vgic_v5_make_spi(e->irqchip.pin); + else + spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS; + if (!vgic_valid_spi(kvm, spi_id)) return -EINVAL; @@ -45,6 +50,13 @@ int kvm_set_routing_entry(struct kvm *kvm, struct kvm_kernel_irq_routing_entry *e, const struct kvm_irq_routing_entry *ue) { + /* + * The common routing table reserves enough entries for all GICv5 SPI + * pins. KVM limits GICv5 to 1024 SPI pins, while GICv2 and GICv3 + * only support SPIs 32..1019, leaving 988 pins. + */ + unsigned int max_irqchip_pins = vgic_is_v5(kvm) ? + VGIC_V5_MAX_NR_SPIS : VGIC_MAX_SPI - VGIC_NR_PRIVATE_IRQS + 1; int r = -EINVAL; switch (ue->type) { @@ -52,8 +64,8 @@ int kvm_set_routing_entry(struct kvm *kvm, e->set = vgic_irqfd_set_irq; e->irqchip.irqchip = ue->u.irqchip.irqchip; e->irqchip.pin = ue->u.irqchip.pin; - if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) || - (e->irqchip.irqchip >= KVM_NR_IRQCHIPS)) + if (e->irqchip.pin >= max_irqchip_pins || + e->irqchip.irqchip >= KVM_NR_IRQCHIPS) goto out; break; case KVM_IRQ_ROUTING_MSI: diff --git a/arch/arm64/kvm/vgic/vgic-irs-v5.c b/arch/arm64/kvm/vgic/vgic-irs-v5.c new file mode 100644 index 000000000000..1874c3c289a6 --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-irs-v5.c @@ -0,0 +1,1217 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 ARM Limited, All Rights Reserved. + */ +#include <linux/bitops.h> +#include <linux/bsearch.h> +#include <linux/interrupt.h> +#include <linux/irq.h> +#include <linux/kvm.h> +#include <linux/kvm_host.h> +#include <kvm/iodev.h> +#include <kvm/arm_arch_timer.h> +#include <kvm/arm_vgic.h> + +#include "vgic.h" +#include "vgic-mmio.h" +#include "vgic-v5-tables.h" + +#define irs_caps kvm_vgic_global_state.vgic_v5_irs_caps + +static struct vgic_dist *vgic_v5_get_vgic(struct kvm_vcpu *vcpu) +{ + return &vcpu->kvm->arch.vgic; +} + +static struct vgic_v5_irs *vgic_v5_get_irs(struct kvm_vcpu *vcpu) +{ + return vcpu->kvm->arch.vgic.vgic_v5_irs_data; +} + +static unsigned long vgic_v5_mmio_read_irs_misc(struct kvm_vcpu *vcpu, + gpa_t addr, unsigned int len) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + const size_t offset = addr & (SZ_64K - 1); + struct kvm_vcpu *target_vcpu; + u8 vpe_id_bits; + u64 value = 0; + + switch (offset) { + case GICV5_IRS_IDR0: + value = FIELD_PREP(GICV5_IRS_IDR0_INT_DOM, irs->idr0.domain); + value |= FIELD_PREP(GICV5_IRS_IDR0_PA_RANGE, irs->idr0.pa_range); + if (irs->idr0.virt) + value |= GICV5_IRS_IDR0_VIRT; + if (irs->idr0.setlpi) + value |= GICV5_IRS_IDR0_SETLPI; + if (irs->idr0.mec) + value |= GICV5_IRS_IDR0_MEC; + if (irs->idr0.mpam) + value |= GICV5_IRS_IDR0_MPAM; + if (irs->idr0.swe) + value |= GICV5_IRS_IDR0_SWE; + value |= FIELD_PREP(GICV5_IRS_IDR0_IRSID, irs->idr0.irs_id); + break; + case GICV5_IRS_IDR1: + value = FIELD_PREP(GICV5_IRS_IDR1_PE_CNT, + atomic_read(&vcpu->kvm->online_vcpus)); + /* + * IRS_IDR1 encodes IAFFID_BITS as N - 1. + */ + vpe_id_bits = vgic_v5_vmte_vpe_id_bits(vcpu); + value |= FIELD_PREP(GICV5_IRS_IDR1_IAFFID_BITS, vpe_id_bits - 1); + value |= FIELD_PREP(GICV5_IRS_IDR1_PRIORITY_BITS, irs->idr1.priority_bits); + break; + case GICV5_IRS_IDR2: + value = FIELD_PREP(GICV5_IRS_IDR2_ISTMD_SZ, irs->idr2.istmd_sz); + if (irs->idr2.istmd) + value |= GICV5_IRS_IDR2_ISTMD; + value |= FIELD_PREP(GICV5_IRS_IDR2_IST_L2SZ, irs->idr2.ist_l2sz); + if (irs->idr2.ist_levels) + value |= GICV5_IRS_IDR2_IST_LEVELS; + value |= FIELD_PREP(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, irs->idr2.min_lpi_id_bits); + value |= GICV5_IRS_IDR2_LPI; + value |= FIELD_PREP(GICV5_IRS_IDR2_ID_BITS, irs->idr2.id_bits); + break; + case GICV5_IRS_IDR5: + value = FIELD_PREP(GICV5_IRS_IDR5_SPI_RANGE, irs->idr5.spi_range); + break; + case GICV5_IRS_IDR6: + value = FIELD_PREP(GICV5_IRS_IDR6_SPI_IRS_RANGE, irs->idr6.spi_irs_range); + break; + case GICV5_IRS_IDR7: + value = FIELD_PREP(GICV5_IRS_IDR7_SPI_BASE, irs->idr7.spi_base); + break; + case GICV5_IRS_IIDR: + /* Revision, Variant, ProductID are implementation defined */ + value = FIELD_PREP(GICV5_IRS_IIDR_PRODUCT_ID, PRODUCT_ID_KVM); + value |= FIELD_PREP(GICV5_IRS_IIDR_VARIANT, 0); + value |= FIELD_PREP(GICV5_IRS_IIDR_REVISION, 0); + value |= FIELD_PREP(GICV5_IRS_IIDR_IMPLEMENTER, IMPLEMENTER_ARM); + break; + case GICV5_IRS_AIDR: + value = FIELD_PREP(GICV5_IRS_AIDR_COMPONENT, + GICV5_AIDR_COMPONENT_IRS); + value |= FIELD_PREP(GICV5_IRS_AIDR_ARCHMAJORREV, + GICV5_AIDR_ARCH_MAJ_REV_V5); + value |= FIELD_PREP(GICV5_IRS_AIDR_ARCHMINORREV, + GICV5_AIDR_ARCH_MIN_REV_V0); + break; + case GICV5_IRS_CR0: + /* + * The IRS is ALWAYS idle as we handle things instantaneously + * from a guest's viewpoint. + */ + value = GICV5_IRS_CR0_IDLE; + if (READ_ONCE(vcpu->kvm->arch.vgic.enabled)) + value |= GICV5_IRS_CR0_IRSEN; + break; + case GICV5_IRS_CR1: + if (irs->cr1.vped_wa) + value |= GICV5_IRS_CR1_VPED_WA; + if (irs->cr1.vped_ra) + value |= GICV5_IRS_CR1_VPED_RA; + if (irs->cr1.vmd_wa) + value |= GICV5_IRS_CR1_VMD_WA; + if (irs->cr1.vmd_ra) + value |= GICV5_IRS_CR1_VMD_RA; + if (irs->cr1.vpet_ra) + value |= GICV5_IRS_CR1_VPET_RA; + if (irs->cr1.vmt_ra) + value |= GICV5_IRS_CR1_VMT_RA; + if (irs->cr1.ist_wa) + value |= GICV5_IRS_CR1_IST_WA; + if (irs->cr1.ist_ra) + value |= GICV5_IRS_CR1_IST_RA; + value |= FIELD_PREP(GICV5_IRS_CR1_IC, irs->cr1.ic); + value |= FIELD_PREP(GICV5_IRS_CR1_OC, irs->cr1.oc); + value |= FIELD_PREP(GICV5_IRS_CR1_SH, irs->cr1.sh); + break; + case GICV5_IRS_SYNC_STATUSR: + value = GICV5_IRS_SYNC_STATUSR_IDLE; + break; + case GICV5_IRS_PE_SELR: + value = FIELD_PREP(GICV5_IRS_PE_SELR_IAFFID, + READ_ONCE(irs->pe_selr.iaffid)); + break; + case GICV5_IRS_PE_STATUSR: + /* We assume that the PE is Online if present. Always IDLE too */ + value = GICV5_IRS_PE_STATUSR_IDLE; + + /* Set ONLINE and V if IAFFID selects a present PE */ + if (kvm_get_vcpu_by_id(vcpu->kvm, + READ_ONCE(irs->pe_selr.iaffid))) { + value |= GICV5_IRS_PE_STATUSR_ONLINE; + value |= GICV5_IRS_PE_STATUSR_V; + } + break; + case GICV5_IRS_PE_CR0: { + u32 iaffid = READ_ONCE(irs->pe_selr.iaffid); + + /* + * Make sure that we are doing something reasonable first. + * Remember, the IAFFID is the same as the VPE_ID + */ + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, iaffid); + if (!target_vcpu) { + kvm_debug("Guest programmed invalid IAFFID (0x%x) into the IRS_PE_SELR\n", + iaffid); + break; + } + + value = GICV5_IRS_PE_CR0_DPS; + break; + } + default: + return 0; + } + + return value; +} + +static void vgic_v5_mmio_write_irs_misc(struct kvm_vcpu *vcpu, gpa_t addr, + unsigned int len, unsigned long val) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + struct vgic_dist *vgic = vgic_v5_get_vgic(vcpu); + const size_t offset = addr & (SZ_64K - 1); + + switch (offset) { + case GICV5_IRS_CR0: + scoped_guard(mutex, &vcpu->kvm->arch.config_lock) { + bool was_enabled = vgic->enabled; + + WRITE_ONCE(vgic->enabled, !!(val & GICV5_IRS_CR0_IRSEN)); + + /* Reload other vCPUs on an edge */ + if (was_enabled != !!(val & GICV5_IRS_CR0_IRSEN)) + kvm_make_all_cpus_request(vcpu->kvm, + KVM_REQ_RELOAD_GICv5); + } + return; + case GICV5_IRS_CR1: + irs->cr1.sh = FIELD_GET(GICV5_IRS_CR1_SH, val); + irs->cr1.oc = FIELD_GET(GICV5_IRS_CR1_OC, val); + irs->cr1.ic = FIELD_GET(GICV5_IRS_CR1_IC, val); + irs->cr1.ist_ra = !!(val & GICV5_IRS_CR1_IST_RA); + irs->cr1.ist_wa = !!(val & GICV5_IRS_CR1_IST_WA); + irs->cr1.vmt_ra = !!(val & GICV5_IRS_CR1_VMT_RA); + irs->cr1.vpet_ra = !!(val & GICV5_IRS_CR1_VPET_RA); + irs->cr1.vmd_ra = !!(val & GICV5_IRS_CR1_VMD_RA); + irs->cr1.vmd_wa = !!(val & GICV5_IRS_CR1_VMD_WA); + irs->cr1.vped_ra = !!(val & GICV5_IRS_CR1_VPED_RA); + irs->cr1.vped_wa = !!(val & GICV5_IRS_CR1_VPED_WA); + return; + case GICV5_IRS_PE_SELR: + WRITE_ONCE(irs->pe_selr.iaffid, + FIELD_GET(GICV5_IRS_PE_SELR_IAFFID, val)); + return; + case GICV5_IRS_PE_CR0: + /* + * We actually have nothing to do here as we don't support + * 1-of-N routing. The only thing that the guest can correctly + * write here is 0x1. However, there's no way to fault if it + * writes something else. This is effectively a WI in our case, + * but we keep it here for the purposes of documenting it. + */ + return; + default: + return; + } +} + +static bool vgic_v5_is_spi_selr_valid(struct vgic_v5_irs *irs, u32 id) +{ + /* Invalid - we don't have any SPIs at all */ + if (irs->idr5.spi_range == 0) + return false; + + /* Invalid - we don't have any on this IRS */ + if (irs->idr6.spi_irs_range == 0) + return false; + + /* Invalid - ID is less than min */ + if (id < irs->idr7.spi_base) + return false; + + /* Invalid - ID is greater than max */ + if (id >= (irs->idr7.spi_base + irs->idr6.spi_irs_range)) + return false; + + return true; +} + +static unsigned long vgic_v5_mmio_read_irs_spi(struct kvm_vcpu *vcpu, + gpa_t addr, unsigned int len) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + const size_t offset = addr & (SZ_64K - 1); + struct vgic_irq *irq; + u64 value = 0; + + switch (offset) { + case GICV5_IRS_SPI_SELR: + /* Return whatever was last written */ + value = FIELD_PREP(GICV5_IRS_SPI_SELR_ID, READ_ONCE(irs->spi_selr.id)); + break; + case GICV5_IRS_SPI_STATUSR: + /* We assume that we can always claim to be idle */ + value = GICV5_IRS_SPI_STATUSR_IDLE; + if (vgic_v5_is_spi_selr_valid(irs, READ_ONCE(irs->spi_selr.id))) + value |= GICV5_IRS_SPI_STATUSR_V; + break; + case GICV5_IRS_SPI_DOMAINR: + value = FIELD_PREP(GICV5_IRS_SPI_DOMAINR_DOMAIN, + GICV5_IRS_SPI_DOMAINR_DOMAIN_NON_SECURE); + break; + case GICV5_IRS_SPI_CFGR: { + u32 id = READ_ONCE(irs->spi_selr.id); + + if (!vgic_v5_is_spi_selr_valid(irs, id)) { + /* Fault with IRS_SPI_SELR; return 0*/ + value = 0; + break; + } + + irq = vgic_get_irq(vcpu->kvm, vgic_v5_make_spi(id)); + if (!irq) { + kvm_debug_ratelimited("Guest trying to access SPI not backed by KVM\n"); + value = 0; + break; + } + + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { + if (irq->config == VGIC_CONFIG_LEVEL) + value = GICV5_IRS_SPI_CFGR_TM; + } + + vgic_put_irq(vcpu->kvm, irq); + + break; + } + default: + return 0; + } + + return value; +} + +static void vgic_v5_mmio_write_irs_spi(struct kvm_vcpu *vcpu, gpa_t addr, + unsigned int len, unsigned long val) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + const size_t offset = addr & (SZ_64K - 1); + struct vgic_irq *irq; + + switch (offset) { + case GICV5_IRS_SPI_SELR: + WRITE_ONCE(irs->spi_selr.id, + FIELD_GET(GICV5_IRS_SPI_SELR_ID, val)); + return; + case GICV5_IRS_SPI_CFGR: { + u32 id = READ_ONCE(irs->spi_selr.id); + + if (!vgic_v5_is_spi_selr_valid(irs, id)) + return; + + /* + * Find KVM's representation of the interrupt - we need to make + * sure that KVM's view agrees with the guest's, else interrupt + * injection won't work properly for level-triggered interrupts + * (we fail to handle the clearing of the pending state if KVM + * thinks that the interrupt is edge-triggered, which is the + * default.) + */ + irq = vgic_get_irq(vcpu->kvm, vgic_v5_make_spi(id)); + if (!irq) + return; + + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { + if (val & GICV5_IRS_SPI_CFGR_TM) + irq->config = VGIC_CONFIG_LEVEL; + else + irq->config = VGIC_CONFIG_EDGE; + } + + vgic_put_irq(vcpu->kvm, irq); + + return; + } + default: + return; + } +} + +static bool vgic_v5_ist_cfgr_valid(struct vgic_v5_irs *irs) +{ + unsigned int expected_istsz; + + if (irs->ist_cfgr.lpi_id_bits < irs->idr2.min_lpi_id_bits || + irs->ist_cfgr.lpi_id_bits > irs->idr2.id_bits) + return false; + + if (!irs->idr2.istmd) + expected_istsz = GICV5_IRS_IST_CFGR_ISTSZ_4; + else if (irs->ist_cfgr.lpi_id_bits >= irs->idr2.istmd_sz) + expected_istsz = GICV5_IRS_IST_CFGR_ISTSZ_16; + else + expected_istsz = GICV5_IRS_IST_CFGR_ISTSZ_8; + + if (irs->ist_cfgr.istsz != expected_istsz) + return false; + + if (irs->ist_cfgr.structure && !irs->idr2.ist_levels) + return false; + + if (!irs->ist_cfgr.structure) + return true; + + return irs->ist_cfgr.l2sz == irs->idr2.ist_l2sz; +} + +static u64 vgic_v5_read_irs_ist_baser(struct vgic_v5_irs *irs) +{ + u64 value; + + value = FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, + irs->ist_baser.addr >> GICV5_IRS_IST_BASER_ADDR_SHIFT); + if (irs->ist_baser.valid) + value |= GICV5_IRS_IST_BASER_VALID; + + return value; +} + +static bool vgic_v5_ist_cfgr_matches(const struct vgic_v5_irs *irs, + unsigned long val) +{ + u8 lpi_id_bits = FIELD_GET(GICV5_IRS_IST_CFGR_LPI_ID_BITS, val); + u8 l2sz = FIELD_GET(GICV5_IRS_IST_CFGR_L2SZ, val); + u8 istsz = FIELD_GET(GICV5_IRS_IST_CFGR_ISTSZ, val); + bool structure = !!(val & GICV5_IRS_IST_CFGR_STRUCTURE); + + return irs->ist_cfgr.lpi_id_bits == lpi_id_bits && + irs->ist_cfgr.l2sz == l2sz && + irs->ist_cfgr.istsz == istsz && + irs->ist_cfgr.structure == structure; +} + +static unsigned long vgic_v5_mmio_read_irs_ist(struct kvm_vcpu *vcpu, + gpa_t addr, unsigned int len) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + const size_t offset = addr & (SZ_64K - 1); + u64 value = 0; + + switch (offset) { + case GICV5_IRS_IST_STATUSR: + return GICV5_IRS_IST_STATUSR_IDLE; + case GICV5_IRS_IST_CFGR: + if (irs->ist_cfgr.structure) + value |= GICV5_IRS_IST_CFGR_STRUCTURE; + value |= FIELD_PREP(GICV5_IRS_IST_CFGR_ISTSZ, irs->ist_cfgr.istsz); + value |= FIELD_PREP(GICV5_IRS_IST_CFGR_L2SZ, irs->ist_cfgr.l2sz); + value |= FIELD_PREP(GICV5_IRS_IST_CFGR_LPI_ID_BITS, irs->ist_cfgr.lpi_id_bits); + break; + case GICV5_IRS_IST_BASER: + case GICV5_IRS_IST_BASER + 4: + value = vgic_v5_read_irs_ist_baser(irs); + return extract_bytes(value, offset & 7, len); + default: + return 0; + } + + return value; +} + +static bool vgic_v5_ist_baser_matches(const struct vgic_v5_irs *irs, + unsigned long val) +{ + u64 addr = FIELD_GET(GICV5_IRS_IST_BASER_ADDR_MASK, val) + << GICV5_IRS_IST_BASER_ADDR_SHIFT; + bool valid = !!(val & GICV5_IRS_IST_BASER_VALID); + + return irs->ist_baser.addr == addr && irs->ist_baser.valid == valid; +} + +static void vgic_v5_update_irs_ist_baser(struct vgic_v5_irs *irs, + unsigned long val) +{ + irs->ist_baser.valid = !!(val & GICV5_IRS_IST_BASER_VALID); + irs->ist_baser.addr = FIELD_GET(GICV5_IRS_IST_BASER_ADDR_MASK, val) + << GICV5_IRS_IST_BASER_ADDR_SHIFT; +} + +static int vgic_v5_write_irs_ist_baser(struct kvm_vcpu *vcpu, unsigned long val) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + enum gicv5_vcpu_cmd cmd = LPI_VIST_MAKE_INVALID; + bool valid = !!(val & GICV5_IRS_IST_BASER_VALID); + int rc; + + /* The address cannot be changed while the IST is valid. */ + if (irs->ist_baser.valid && valid) + return 0; + + /* Valid -> Invalid */ + if (irs->ist_baser.valid && !valid) { + /* Make the LPI IST invalid and then ... */ + rc = irq_set_vcpu_affinity(vgic_v5_vpe_db(vcpu), &cmd); + if (rc) + return rc; + + vgic_v5_update_irs_ist_baser(irs, val); + + /* + * ... reflect that in the emulated BASER before freeing the + * host IST. If the free fails, the guest-visible valid bit + * still matches the hardware state. + */ + return vgic_v5_lpi_ist_free(vcpu->kvm); + } else if (!irs->ist_baser.valid && valid) { /* Invalid -> Valid */ + if (!vgic_v5_ist_cfgr_valid(irs)) + return -EINVAL; + + rc = vgic_v5_lpi_ist_alloc(vcpu->kvm, irs->ist_cfgr.lpi_id_bits); + if (rc) + return rc; + } + + /* Now that we've handled the edges, update the valid bit and addr */ + vgic_v5_update_irs_ist_baser(irs, val); + + return 0; +} + +static void vgic_v5_mmio_write_irs_ist(struct kvm_vcpu *vcpu, gpa_t addr, + unsigned int len, unsigned long val) +{ + struct vgic_v5_irs *irs = vgic_v5_get_irs(vcpu); + const size_t offset = addr & (SZ_64K - 1); + + switch (offset) { + case GICV5_IRS_IST_CFGR: + scoped_guard(mutex, &vcpu->kvm->arch.config_lock) { + if (irs->ist_baser.valid) + return; + + irs->ist_cfgr.lpi_id_bits = FIELD_GET(GICV5_IRS_IST_CFGR_LPI_ID_BITS, val); + irs->ist_cfgr.l2sz = FIELD_GET(GICV5_IRS_IST_CFGR_L2SZ, val); + irs->ist_cfgr.istsz = FIELD_GET(GICV5_IRS_IST_CFGR_ISTSZ, val); + irs->ist_cfgr.structure = !!(val & GICV5_IRS_IST_CFGR_STRUCTURE); + } + return; + case GICV5_IRS_IST_BASER: + case GICV5_IRS_IST_BASER + 4: { + scoped_guard(mutex, &vcpu->kvm->arch.config_lock) { + u64 baser; + + baser = update_64bit_reg(vgic_v5_read_irs_ist_baser(irs), + offset & 4, len, val); + vgic_v5_write_irs_ist_baser(vcpu, baser); + } + return; + } + default: + return; + } +} + +static unsigned long vgic_v5_coresight_read(struct kvm_vcpu *vcpu, + gpa_t addr, unsigned int len) +{ + const size_t offset = addr & (SZ_64K - 1); + + switch (offset) { + case GICV5_CORESIGHT_DEVARCH: + return GICV5_CORESIGHT_DEVARCH_VAL; + case GICV5_CORESIGHT_PIDR4: + return GICV5_CORESIGHT_PIDR4_JEP106_CONT; + case GICV5_CORESIGHT_PIDR5: + return GICV5_CORESIGHT_PIDR5_RES0; + case GICV5_CORESIGHT_PIDR6: + return GICV5_CORESIGHT_PIDR6_RES0; + case GICV5_CORESIGHT_PIDR7: + return GICV5_CORESIGHT_PIDR7_RES0; + case GICV5_CORESIGHT_PIDR0: + return GICV5_CORESIGHT_PIDR0_PART_0; + case GICV5_CORESIGHT_PIDR1: + return GICV5_CORESIGHT_PIDR1_DES_0_PART_1; + case GICV5_CORESIGHT_PIDR2: + return GICV5_CORESIGHT_PIDR2_DES_1; + case GICV5_CORESIGHT_PIDR3: + return GICV5_CORESIGHT_PIDR3_REVAND_CMOD; + case GICV5_CORESIGHT_CIDR0: + return GICV5_CORESIGHT_CIDR0_VAL; + case GICV5_CORESIGHT_CIDR1: + return GICV5_CORESIGHT_CIDR1_VAL; + case GICV5_CORESIGHT_CIDR2: + return GICV5_CORESIGHT_CIDR2_VAL; + case GICV5_CORESIGHT_CIDR3: + return GICV5_CORESIGHT_CIDR3_VAL; + default: + return 0; + } +} + +static unsigned long vgic_v5_mmio_uaccess_read_irs_status(struct kvm_vcpu *vcpu, + gpa_t addr, + unsigned int len) +{ + const size_t offset = addr & (SZ_64K - 1); + + switch (offset) { + case GICV5_IRS_SYNC_STATUSR: + return GICV5_IRS_SYNC_STATUSR_IDLE; + case GICV5_IRS_SPI_STATUSR: + return GICV5_IRS_SPI_STATUSR_IDLE; + case GICV5_IRS_PE_STATUSR: + return GICV5_IRS_PE_STATUSR_IDLE; + case GICV5_IRS_IST_STATUSR: + return GICV5_IRS_IST_STATUSR_IDLE; + default: + return 0; + } +} + +static int vgic_v5_mmio_uaccess_write_irs(struct kvm_vcpu *vcpu, gpa_t addr, + unsigned int len, unsigned long val) +{ + struct vgic_dist *vgic = &vcpu->kvm->arch.vgic; + struct vgic_v5_irs *irs_data = vgic->vgic_v5_irs_data; + size_t offset = addr & (SZ_64K - 1); + int ret; + + /* + * The following registers are ONLY settable via uaccesses. The guest + * cannot write them! + */ + + switch (offset) { + case GICV5_IRS_IDR0: + if (FIELD_GET(GICV5_IRS_IDR0_INT_DOM, val) != + GICV5_IRS_IDR0_INT_DOM_NON_SECURE) + return -EINVAL; + + if ((val & GICV5_IRS_IDR0_VIRT) || + (val & GICV5_IRS_IDR0_ONE_N) || + (val & GICV5_IRS_IDR0_VIRT_ONE_N) || + (val & GICV5_IRS_IDR0_SETLPI) || + (val & GICV5_IRS_IDR0_MEC) || + (val & GICV5_IRS_IDR0_MPAM) || + (val & GICV5_IRS_IDR0_SWE)) + return -EINVAL; + + irs_data->idr0.domain = FIELD_GET(GICV5_IRS_IDR0_INT_DOM, val); + irs_data->idr0.pa_range = FIELD_GET(GICV5_IRS_IDR0_PA_RANGE, val); + irs_data->idr0.virt = !!(val & GICV5_IRS_IDR0_VIRT); + irs_data->idr0.setlpi = !!(val & GICV5_IRS_IDR0_SETLPI); + irs_data->idr0.mec = !!(val & GICV5_IRS_IDR0_MEC); + irs_data->idr0.mpam = !!(val & GICV5_IRS_IDR0_MPAM); + irs_data->idr0.swe = !!(val & GICV5_IRS_IDR0_SWE); + irs_data->idr0.irs_id = FIELD_GET(GICV5_IRS_IDR0_IRSID, val); + break; + case GICV5_IRS_IDR1: { + unsigned int iaffid_bits, priority_bits; + u8 vpe_id_bits; + + /* Ignore writes to PE_CNT as this is populated from num vcpus */ + iaffid_bits = FIELD_GET(GICV5_IRS_IDR1_IAFFID_BITS, val); + priority_bits = FIELD_GET(GICV5_IRS_IDR1_PRIORITY_BITS, val); + + /* IAFFID_BITS is derived from the VMTE and encoded as N - 1. */ + vpe_id_bits = vgic_v5_vmte_vpe_id_bits(vcpu); + if (iaffid_bits != vpe_id_bits - 1) + return -EINVAL; + + if (priority_bits > gicv5_global_data.irs_pri_bits - 1) + return -EINVAL; + + irs_data->idr1.priority_bits = priority_bits; + break; + } + case GICV5_IRS_IDR2: + /* We always support LPIs */ + if (!(val & GICV5_IRS_IDR2_LPI)) + return -EINVAL; + + /* We only support LPIs with linear, non-metadata guest ISTs */ + if (val & GICV5_IRS_IDR2_IST_LEVELS) + return -EINVAL; + + if ((val & GICV5_IRS_IDR2_ISTMD) || + FIELD_GET(GICV5_IRS_IDR2_ISTMD_SZ, val)) + return -EINVAL; + + /* We can't present more bits than we have support for in HW */ + if (FIELD_GET(GICV5_IRS_IDR2_ID_BITS, val) > + vgic_v5_irs_ist_id_bits(&irs_caps)) + return -EINVAL; + + /* Min LPI ID bits must be greater than or equal to the HW */ + if (FIELD_GET(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, val) < + vgic_v5_irs_min_lpi_id_bits(&irs_caps)) + return -EINVAL; + + if (FIELD_GET(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, val) > + FIELD_GET(GICV5_IRS_IDR2_ID_BITS, val)) + return -EINVAL; + + irs_data->idr2.istmd_sz = FIELD_GET(GICV5_IRS_IDR2_ISTMD_SZ, val); + irs_data->idr2.istmd = !!(val & GICV5_IRS_IDR2_ISTMD); + irs_data->idr2.ist_l2sz = FIELD_GET(GICV5_IRS_IDR2_IST_L2SZ, val); + irs_data->idr2.ist_levels = !!(val & GICV5_IRS_IDR2_IST_LEVELS); + irs_data->idr2.min_lpi_id_bits = FIELD_GET(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, val); + irs_data->idr2.id_bits = FIELD_GET(GICV5_IRS_IDR2_ID_BITS, val); + break; + case GICV5_IRS_IDR5: + if (FIELD_GET(GICV5_IRS_IDR5_SPI_RANGE, val) != irs_data->idr5.spi_range) + return -EINVAL; + break; + case GICV5_IRS_IDR6: + if (FIELD_GET(GICV5_IRS_IDR6_SPI_IRS_RANGE, val) != irs_data->idr6.spi_irs_range) + return -EINVAL; + break; + case GICV5_IRS_IDR7: + if (FIELD_GET(GICV5_IRS_IDR7_SPI_BASE, val) != irs_data->idr7.spi_base) + return -EINVAL; + break; + case GICV5_IRS_IST_BASER: + ret = vgic_v5_lpi_ist_exists(vcpu->kvm); + if (ret < 0) + return ret; + + if (ret && !vgic_v5_ist_baser_matches(irs_data, val)) + return -EINVAL; + + if (!irs_data->ist_baser.valid && + (val & GICV5_IRS_IST_BASER_VALID)) + irs_data->lpi_ist_restore_pending = true; + + vgic_v5_update_irs_ist_baser(irs_data, val); + break; + case GICV5_IRS_IST_CFGR: + ret = vgic_v5_lpi_ist_exists(vcpu->kvm); + if (ret < 0) + return ret; + + if (ret && !vgic_v5_ist_cfgr_matches(irs_data, val)) + return -EINVAL; + + irs_data->ist_cfgr.lpi_id_bits = FIELD_GET(GICV5_IRS_IST_CFGR_LPI_ID_BITS, val); + irs_data->ist_cfgr.l2sz = FIELD_GET(GICV5_IRS_IST_CFGR_L2SZ, val); + irs_data->ist_cfgr.istsz = FIELD_GET(GICV5_IRS_IST_CFGR_ISTSZ, val); + irs_data->ist_cfgr.structure = !!(val & GICV5_IRS_IST_CFGR_STRUCTURE); + break; + case GICV5_IRS_CR0: + vgic->enabled = !!(val & GICV5_IRS_CR0_IRSEN); + break; + case GICV5_IRS_SPI_CFGR: + break; + case GICV5_IRS_IIDR: + fallthrough; + case GICV5_IRS_AIDR: + break; + default: + return -EINVAL; + } + + return 0; +} + +static const struct vgic_register_region vgic_v5_irs_registers[] = { + /* + * This is the IRS_CONFIG_FRAME. + */ + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR0, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR1, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR2, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_IDR3, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_IDR4, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR5, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR6, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IDR7, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IIDR, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_AIDR, vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_CR0, + vgic_v5_mmio_read_irs_misc, + vgic_v5_mmio_write_irs_misc, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_CR1, vgic_v5_mmio_read_irs_misc, + vgic_v5_mmio_write_irs_misc, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SYNCR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_SYNC_STATUSR, + vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, + vgic_v5_mmio_uaccess_read_irs_status, + vgic_mmio_uaccess_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SPI_VMR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SPI_SELR, vgic_v5_mmio_read_irs_spi, + vgic_v5_mmio_write_irs_spi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SPI_DOMAINR, vgic_v5_mmio_read_irs_spi, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SPI_RESAMPLER, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_SPI_CFGR, + vgic_v5_mmio_read_irs_spi, + vgic_v5_mmio_write_irs_spi, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_SPI_STATUSR, + vgic_v5_mmio_read_irs_spi, + vgic_mmio_write_wi, + vgic_v5_mmio_uaccess_read_irs_status, + vgic_mmio_uaccess_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_PE_SELR, + vgic_v5_mmio_read_irs_misc, + vgic_v5_mmio_write_irs_misc, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_PE_STATUSR, + vgic_v5_mmio_read_irs_misc, + vgic_mmio_write_wi, + vgic_v5_mmio_uaccess_read_irs_status, + vgic_mmio_uaccess_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_PE_CR0, vgic_v5_mmio_read_irs_misc, + vgic_v5_mmio_write_irs_misc, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IST_BASER, + vgic_v5_mmio_read_irs_ist, + vgic_v5_mmio_write_irs_ist, NULL, + vgic_v5_mmio_uaccess_write_irs, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IST_CFGR, + vgic_v5_mmio_read_irs_ist, + vgic_v5_mmio_write_irs_ist, NULL, + vgic_v5_mmio_uaccess_write_irs, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH_UACCESS(GICV5_IRS_IST_STATUSR, + vgic_v5_mmio_read_irs_ist, + vgic_mmio_write_wi, + vgic_v5_mmio_uaccess_read_irs_status, + vgic_mmio_uaccess_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_MAP_L2_ISTR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + + /* + * The following registers are only for running VMs. They are not yet + * supported as we don't currently support nested, so expose them as + * read-as-zero/write-ignored. + */ + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMT_BASER, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMT_CFGR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMT_STATUSR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VPE_SELR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VPE_DBR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VPE_HPPIR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VPE_CR0, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VPE_STATUSR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VM_DBR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VM_SELR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VM_STATUSR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMAP_L2_VMTR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMAP_VMR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMAP_VISTR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMAP_L2_VISTR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_VMAP_VPER, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SAVE_VMR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SAVE_VM_STATUSR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + + /* MEC, MPAM, SWERR - all unimplemented */ + + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_MEC_IDR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_MEC_MECID_R, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_MPAM_IDR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_MPAM_PARTID_R, vgic_mmio_read_raz, + vgic_mmio_write_wi, 4, VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SWERR_STATUSR, vgic_mmio_read_raz, + vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SWERR_SYNDROMER0, + vgic_mmio_read_raz, vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_IRS_SWERR_SYNDROMER1, + vgic_mmio_read_raz, vgic_mmio_write_wi, 8, + VGIC_ACCESS_64bit | VGIC_ACCESS_32bit), + + /* CoreSight identification registers */ + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_DEVARCH, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR4, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR5, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR6, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR7, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR0, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR1, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR2, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_PIDR3, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_CIDR0, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_CIDR1, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_CIDR2, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), + REGISTER_DESC_WITH_LENGTH(GICV5_CORESIGHT_CIDR3, + vgic_v5_coresight_read, vgic_mmio_write_wi, 4, + VGIC_ACCESS_32bit), +}; + +unsigned int vgic_v5_init_irs_iodev(struct vgic_io_device *dev) +{ + dev->regions = vgic_v5_irs_registers; + dev->nr_regions = ARRAY_SIZE(vgic_v5_irs_registers); + + kvm_iodevice_init(&dev->dev, &kvm_io_gic_ops); + + /* We represent both of the IRS frames back to back, so this is 128K */ + return KVM_VGIC_V5_IRS_SIZE; +} + +int vgic_v5_register_irs_iodev(struct kvm *kvm, gpa_t irs_base_address) +{ + struct vgic_io_device *io_device = &kvm->arch.vgic.vgic_v5_irs_data->iodev; + unsigned int len; + + /* + * Design choice: Force MMIO region to be 64k aligned. Simplifies + * pulling out registers. + */ + if (!IS_ALIGNED(irs_base_address, SZ_64K)) + return -EINVAL; + + len = vgic_v5_init_irs_iodev(io_device); + + io_device->base_addr = irs_base_address; + io_device->iodev_type = IODEV_GICV5_IRS; + io_device->redist_vcpu = NULL; + + return kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, irs_base_address, len, + &io_device->dev); +} + +/** + * kvm_vgic_v5_irs_init: initialize the IRS data structures + * @kvm: kvm struct pointer + * @nr_spis: number of spis, frozen by caller + */ +int kvm_vgic_v5_irs_init(struct kvm *kvm, unsigned int nr_spis) +{ + struct vgic_dist *dist = &kvm->arch.vgic; + struct vgic_v5_irs *irs = dist->vgic_v5_irs_data; + struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0); + size_t nr_spi_bits; + u64 mmfr0; + int ret, i; + + /* + * We (KVM) allocate an Interrupt State Table (IST) for SPIs. The + * hardware mandates that lower 6 bits of the address are 0. Each ISTE + * is 4 bytes in size (or larger if metadata storage is required), so 16 + * entries would be enough for alignment. We require userspace to + * provide at least 32 SPIs, so we always meet alignment requirements. + */ + if (nr_spis) { + dist->spis = kcalloc(nr_spis, sizeof(struct vgic_irq), + GFP_KERNEL_ACCOUNT); + if (!dist->spis) + return -ENOMEM; + + /* + * In the following code we do not take the irq struct lock since + * no other action on irq structs can happen while the VGIC is + * not initialized yet. + */ + for (i = 0; i < nr_spis; i++) { + struct vgic_irq *irq = &dist->spis[i]; + + irq->intid = vgic_v5_make_spi(i); + INIT_LIST_HEAD(&irq->ap_list); + raw_spin_lock_init(&irq->irq_lock); + irq->vcpu = NULL; + irq->target_vcpu = vcpu0; + refcount_set(&irq->refcount, 0); + /* + * The guest controls the enable state, and again it is + * directly handled by the hardware. From our point of + * view it is always enabled. + */ + irq->enabled = 1; + vgic_v5_set_spi_ops(irq); + } + + nr_spi_bits = fls(roundup_pow_of_two(nr_spis)) - 1; + + ret = vgic_v5_spi_ist_alloc(kvm, nr_spi_bits); + if (ret) { + kfree(dist->spis); + dist->spis = NULL; + return ret; + } + } + + /* Set sane initial state for the IRS MMIO registers */ + + irs->idr0.domain = GICV5_IRS_IDR0_INT_DOM_NON_SECURE; + + mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + irs->idr0.pa_range = cpuid_feature_extract_unsigned_field(mmfr0, + ID_AA64MMFR0_EL1_PARANGE_SHIFT); + + irs->idr0.virt = 0; + irs->idr0.setlpi = 0; + irs->idr0.mec = 0; + irs->idr0.mpam = 0; + irs->idr0.swe = 0; + irs->idr0.irs_id = 0; + + irs->idr1.priority_bits = gicv5_global_data.irs_pri_bits - 1; + + /* + * Support 16-bits of ID space for the IRS. This should be sufficient + * for most applications, and the CPUIF is guaranteed to have at least + * 16-bits of ID space support (we actually present 16-bits there, even + * if the hardware supports more). Warn if the hardware doesn't support + * 16 bits, and use the smaller value. YMMV! + * + * As for the minimum number of ID bits, we match the hardware's + * capability. + */ + if (vgic_v5_irs_ist_id_bits(&irs_caps) < 16) + pr_warn_ratelimited("Host IRS supports fewer than 16 ID bits for ISTs (%u)\n", + vgic_v5_irs_ist_id_bits(&irs_caps)); + + irs->idr2.id_bits = min(16, vgic_v5_irs_ist_id_bits(&irs_caps)); + irs->idr2.min_lpi_id_bits = vgic_v5_irs_min_lpi_id_bits(&irs_caps); + + /* Only allow the guest to create Linear ISTs - simplifies Save/Restore */ + irs->idr2.ist_levels = 0; + irs->idr2.ist_l2sz = GICV5_IRS_IST_CFGR_L2SZ_4K; + irs->idr2.istmd = 0; + irs->idr2.istmd_sz = 0; + + /* We have a single IRS, only. All SPIs reside here! */ + irs->idr5.spi_range = nr_spis; + irs->idr6.spi_irs_range = nr_spis; + irs->idr7.spi_base = 0; + + irs->cr1.sh = 0; + irs->cr1.oc = 0; + irs->cr1.ic = 0; + irs->cr1.ist_ra = 0; + irs->cr1.ist_wa = 0; + irs->cr1.vmt_ra = 0; + irs->cr1.vpet_ra = 0; + irs->cr1.vmd_ra = 0; + irs->cr1.vmd_wa = 0; + irs->cr1.vped_ra = 0; + irs->cr1.vped_wa = 0; + + irs->spi_selr.id = -1; + + irs->pe_selr.iaffid = -1; + + irs->ist_cfgr.lpi_id_bits = 0; + irs->ist_cfgr.l2sz = 0; + irs->ist_cfgr.istsz = 0; + irs->ist_cfgr.structure = 0; + + irs->ist_baser.valid = 0; + irs->ist_baser.addr = 0; + irs->lpi_ist_restore_pending = false; + + return 0; +} + +int vgic_v5_irs_lpi_ist_id_bits(struct kvm *kvm, unsigned int *id_bits) +{ + struct vgic_v5_irs *irs = kvm->arch.vgic.vgic_v5_irs_data; + + if (!irs) + return -ENXIO; + + if (!irs->ist_baser.valid) + return 0; + + if (!vgic_v5_ist_cfgr_valid(irs)) + return -EINVAL; + + *id_bits = irs->ist_cfgr.lpi_id_bits; + + return 1; +} + +int vgic_v5_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr) +{ + const struct vgic_register_region *region; + struct vgic_reg_attr reg_attr; + struct kvm_vcpu *vcpu; + gpa_t addr, offset; + int ret, align; + + ret = vgic_v5_parse_attr(dev, attr, ®_attr); + if (ret) + return ret; + + vcpu = reg_attr.vcpu; + addr = reg_attr.addr; + + if (attr->group == KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS) + return vgic_v5_has_cpu_sysregs_attr(vcpu, attr); + + offset = attr->attr; + + region = vgic_find_mmio_region(vgic_v5_irs_registers, + ARRAY_SIZE(vgic_v5_irs_registers), + offset); + if (!region) + return -ENXIO; + + align = region->access_flags & VGIC_ACCESS_64bit ? 0x7 : 0x3; + if (offset & align) + return -EINVAL; + + return 0; +} + +/* + * Access the IRS MMIO Regs. Relevant locks have been taken by the calling code. + */ +int vgic_v5_irs_attr_regs_access(struct kvm_device *dev, + struct kvm_device_attr *attr, + u64 *reg, bool is_write) +{ + const struct vgic_register_region *region; + gpa_t addr, offset; + unsigned int len; + int align, ret = 0; + + offset = attr->attr; + + if (IS_VGIC_ADDR_UNDEF(dev->kvm->arch.vgic.vgic_v5_irs_data->vgic_v5_irs_base)) + return -ENXIO; + + region = vgic_find_mmio_region(vgic_v5_irs_registers, + ARRAY_SIZE(vgic_v5_irs_registers), + offset); + if (!region) + return -ENXIO; + + /* + * Although the spec supports upper/lower 32-bit accesses to + * 64-bit IRS registers, the userspace ABI requires 64-bit + * accesses to all 64-bit wide registers. We therefore only + * support 32-bit accesses to 32-bit-wide registers. + */ + align = region->access_flags & VGIC_ACCESS_64bit ? 0x7 : 0x3; + len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4; + + if (offset & align) + return -EINVAL; + + addr = dev->kvm->arch.vgic.vgic_v5_irs_data->vgic_v5_irs_base + offset; + + if (is_write) { + if (region->uaccess_write) + ret = region->uaccess_write(kvm_get_vcpu(dev->kvm, 0), + addr, len, *reg); + else + region->write(kvm_get_vcpu(dev->kvm, 0), addr, len, *reg); + } else { + if (region->uaccess_read) + *reg = region->uaccess_read(kvm_get_vcpu(dev->kvm, 0), + addr, len); + else + *reg = region->read(kvm_get_vcpu(dev->kvm, 0), addr, len); + } + + return ret; +} diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index c10a11373e10..e5657cb60356 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -2134,9 +2134,9 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev, u64 val; /* - * MAPC with V=0 keeps the ITEs mapped but drops their collection, - * and with it the ICID. Save a zeroed entry, which the restore path - * reads back as invalid. + * MAPC with V=0 and clearing GITS_BASER<coll>.Valid both keep the + * ITEs mapped while dropping their collection, and with it the ICID. + * Save a zeroed entry, which the restore path reads back as invalid. */ if (!ite->collection) return vgic_its_write_entry_lock(its, gpa, 0ULL, ite); diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c index 90be99443df3..1c205cb1361f 100644 --- a/arch/arm64/kvm/vgic/vgic-kvm-device.c +++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c @@ -181,6 +181,14 @@ static int kvm_vgic_addr(struct kvm *kvm, struct kvm_device_attr *attr, bool wri addr |= (u64)rdreg->count << KVM_VGIC_V3_RDIST_COUNT_SHIFT; goto out; } + case KVM_VGIC_V5_ADDR_TYPE_IRS: + r = vgic_check_type(kvm, KVM_DEV_TYPE_ARM_VGIC_V5); + if (r) + break; + addr_ptr = &vgic->vgic_v5_irs_data->vgic_v5_irs_base; + alignment = SZ_64K; + size = KVM_VGIC_V5_IRS_SIZE; + break; default: r = -ENODEV; } @@ -224,31 +232,48 @@ static int vgic_set_common_attr(struct kvm_device *dev, if (get_user(val, uaddr)) return -EFAULT; - /* - * We require: - * - at least 32 SPIs on top of the 16 SGIs and 16 PPIs - * - at most 1024 interrupts - * - a multiple of 32 interrupts - */ - if (val < (VGIC_NR_PRIVATE_IRQS + 32) || - val > VGIC_MAX_RESERVED || - (val & 31)) - return -EINVAL; + if (!vgic_is_v5(dev->kvm)) { + /* + * We require: + * - at least 32 SPIs on top of the 16 SGIs and 16 PPIs + * - at most 1024 interrupts + * - a multiple of 32 interrupts + */ + if (val < (VGIC_NR_PRIVATE_IRQS + 32) || + val > VGIC_MAX_RESERVED || (val & 31)) + return -EINVAL; + + mutex_lock(&dev->kvm->arch.config_lock); - mutex_lock(&dev->kvm->arch.config_lock); + /* + * Either userspace has already configured NR_IRQS or + * the vgic has already been initialized and vgic_init() + * supplied a default amount of SPIs. + */ + if (dev->kvm->arch.vgic.nr_spis) + ret = -EBUSY; + else + dev->kvm->arch.vgic.nr_spis = + val - VGIC_NR_PRIVATE_IRQS; - /* - * Either userspace has already configured NR_IRQS or - * the vgic has already been initialized and vgic_init() - * supplied a default amount of SPIs. - */ - if (dev->kvm->arch.vgic.nr_spis) - ret = -EBUSY; - else - dev->kvm->arch.vgic.nr_spis = - val - VGIC_NR_PRIVATE_IRQS; + mutex_unlock(&dev->kvm->arch.config_lock); + } else { + /* + * GICv5 reports a number of SPIs, not a total number of + * interrupts. Require a multiple of 32 SPIs. + */ + if (val < VGIC_V5_DEFAULT_NR_SPIS || + val > VGIC_V5_MAX_NR_SPIS || + (val & 31)) + return -EINVAL; - mutex_unlock(&dev->kvm->arch.config_lock); + mutex_lock(&dev->kvm->arch.config_lock); + if (vgic_initialized(dev->kvm) || dev->kvm->arch.vgic.nr_spis) + ret = -EBUSY; + else + dev->kvm->arch.vgic.nr_spis = val; + mutex_unlock(&dev->kvm->arch.config_lock); + } return ret; } @@ -299,9 +324,14 @@ static int vgic_get_common_attr(struct kvm_device *dev, return (r == -ENODEV) ? -ENXIO : r; case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: { u32 __user *uaddr = (u32 __user *)(long)attr->addr; - - r = put_user(dev->kvm->arch.vgic.nr_spis + - VGIC_NR_PRIVATE_IRQS, uaddr); + /* Older GICs */ + if (!vgic_is_v5(dev->kvm)) { + r = put_user(dev->kvm->arch.vgic.nr_spis + + VGIC_NR_PRIVATE_IRQS, + uaddr); + } else { + r = put_user(dev->kvm->arch.vgic.nr_spis, uaddr); + } break; } } @@ -512,7 +542,7 @@ int vgic_v3_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr, * Allow access to certain ID-like registers prior to VGIC initialization, * thereby allowing the VMM to provision the features / sizing of the VGIC. */ -static bool reg_allowed_pre_init(struct kvm_device_attr *attr) +static bool v3_reg_allowed_pre_init(struct kvm_device_attr *attr) { if (attr->group != KVM_DEV_ARM_VGIC_GRP_DIST_REGS) return false; @@ -575,7 +605,7 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev, mutex_lock(&dev->kvm->arch.config_lock); - if (!(vgic_initialized(dev->kvm) || reg_allowed_pre_init(attr))) { + if (!(vgic_initialized(dev->kvm) || v3_reg_allowed_pre_init(attr))) { ret = -EBUSY; goto out; } @@ -743,26 +773,178 @@ static int vgic_v5_get_userspace_ppis(struct kvm_device *dev, return ret; } +int vgic_v5_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr, + struct vgic_reg_attr *reg_attr) +{ + unsigned long vgic_mpidr, mpidr_reg; + + switch (attr->group) { + case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + vgic_mpidr = (attr->attr & KVM_DEV_ARM_VGIC_V3_MPIDR_MASK) >> + KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT; + + mpidr_reg = VGIC_TO_MPIDR(vgic_mpidr); + reg_attr->vcpu = kvm_mpidr_to_vcpu(dev->kvm, mpidr_reg); + break; + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + reg_attr->vcpu = kvm_get_vcpu(dev->kvm, 0); + break; + default: + return -EINVAL; + } + + if (!reg_attr->vcpu) + return -EINVAL; + + reg_attr->addr = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK; + + return 0; +} + +/* + * Some registers can potentially be read before the core GIC & IRS has been + * initialised. Right now, everything is required to be post-init. + */ +static bool v5_reg_allowed_pre_init(struct kvm_device_attr *attr) +{ + return false; +} + +static bool vgic_v5_vm_has_run_once(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vcpu_has_run_once(vcpu)) + return true; + } + + return false; +} + +/* + * vgic_v5_attr_regs_access - allows user space to access VGIC v5 state + * + * @dev: kvm device handle + * @attr: kvm device attribute + * @is_write: true if userspace is writing a register + */ +static int vgic_v5_attr_regs_access(struct kvm_device *dev, + struct kvm_device_attr *attr, + bool is_write) +{ + u64 __user *uaddr = (u64 __user *)(unsigned long)attr->addr; + struct vgic_reg_attr reg_attr; + struct kvm_vcpu *vcpu; + bool uaccess; + u64 val; + int ret; + + ret = vgic_v5_parse_attr(dev, attr, ®_attr); + if (ret) + return ret; + + vcpu = reg_attr.vcpu; + + switch (attr->group) { + case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + /* Sysregs uaccess is performed by the sysreg handling code */ + uaccess = false; + break; + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + fallthrough; + default: + uaccess = true; + } + + if (uaccess && is_write) { + if (get_user(val, uaddr)) + return -EFAULT; + } + + mutex_lock(&dev->kvm->lock); + + if (kvm_trylock_all_vcpus(dev->kvm)) { + mutex_unlock(&dev->kvm->lock); + return -EBUSY; + } + + mutex_lock(&dev->kvm->arch.config_lock); + + if (!(vgic_initialized(dev->kvm) || v5_reg_allowed_pre_init(attr))) { + ret = -EBUSY; + goto out; + } + + if (is_write && vgic_v5_vm_has_run_once(dev->kvm)) { + ret = -EBUSY; + goto out; + } + + switch (attr->group) { + case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + ret = vgic_v5_cpu_sysregs_uaccess(vcpu, attr, is_write); + break; + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + /* + * The IRS registers are a mixture of 32-bit and 64-bit + * registers. Internally, we always perform the correctly sized + * access, but the UAPI is defined in such a way that we are + * always provided a __u64 by userspace. When userspace writes, + * the upper 32-bits are ignored for 32-bit accesses, and on a + * read any 32-bit accesses are written back to user memory + * using the full 64-bits. + */ + ret = vgic_v5_irs_attr_regs_access(dev, attr, &val, is_write); + break; + default: + ret = -EINVAL; + break; + } + +out: + mutex_unlock(&dev->kvm->arch.config_lock); + kvm_unlock_all_vcpus(dev->kvm); + mutex_unlock(&dev->kvm->lock); + + if (!ret && uaccess && !is_write) + ret = put_user(val, uaddr); + + return ret; +} + static int vgic_v5_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr) { switch (attr->group) { case KVM_DEV_ARM_VGIC_GRP_ADDR: + break; + case KVM_DEV_ARM_VGIC_GRP_IST: + if (attr->attr) + return -ENXIO; + + return vgic_v5_irs_restore_ists(dev->kvm, attr); + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + fallthrough; case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + return vgic_v5_attr_regs_access(dev, attr, true); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: - return -ENXIO; + break; case KVM_DEV_ARM_VGIC_GRP_CTRL: switch (attr->attr) { case KVM_DEV_ARM_VGIC_CTRL_INIT: - return vgic_set_common_attr(dev, attr); + break; case KVM_DEV_ARM_VGIC_USERSPACE_PPIS: default: return -ENXIO; } + break; default: return -ENXIO; } + return vgic_set_common_attr(dev, attr); } static int vgic_v5_get_attr(struct kvm_device *dev, @@ -770,21 +952,33 @@ static int vgic_v5_get_attr(struct kvm_device *dev, { switch (attr->group) { case KVM_DEV_ARM_VGIC_GRP_ADDR: + break; + case KVM_DEV_ARM_VGIC_GRP_IST: + if (attr->attr) + return -ENXIO; + + return vgic_v5_irs_save_ists(dev->kvm, attr); + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + fallthrough; case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + return vgic_v5_attr_regs_access(dev, attr, false); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: - return -ENXIO; + break; case KVM_DEV_ARM_VGIC_GRP_CTRL: switch (attr->attr) { case KVM_DEV_ARM_VGIC_CTRL_INIT: - return vgic_get_common_attr(dev, attr); + break; case KVM_DEV_ARM_VGIC_USERSPACE_PPIS: return vgic_v5_get_userspace_ppis(dev, attr); default: return -ENXIO; } + break; default: return -ENXIO; } + + return vgic_get_common_attr(dev, attr); } static int vgic_v5_has_attr(struct kvm_device *dev, @@ -792,18 +986,30 @@ static int vgic_v5_has_attr(struct kvm_device *dev, { switch (attr->group) { case KVM_DEV_ARM_VGIC_GRP_ADDR: + switch (attr->attr) { + case KVM_VGIC_V5_ADDR_TYPE_IRS: + return 0; + } + return -ENXIO; + case KVM_DEV_ARM_VGIC_GRP_IRS_REGS: + fallthrough; case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: + return vgic_v5_has_attr_regs(dev, attr); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: - return -ENXIO; + return 0; case KVM_DEV_ARM_VGIC_GRP_CTRL: switch (attr->attr) { case KVM_DEV_ARM_VGIC_CTRL_INIT: return 0; case KVM_DEV_ARM_VGIC_USERSPACE_PPIS: return 0; + case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES: default: return -ENXIO; } + break; + case KVM_DEV_ARM_VGIC_GRP_IST: + return attr->attr ? -ENXIO : 0; default: return -ENXIO; } diff --git a/arch/arm64/kvm/vgic/vgic-mmio-v3.c b/arch/arm64/kvm/vgic/vgic-mmio-v3.c index 5913a20d8301..5ab789accc9d 100644 --- a/arch/arm64/kvm/vgic/vgic-mmio-v3.c +++ b/arch/arm64/kvm/vgic/vgic-mmio-v3.c @@ -818,16 +818,21 @@ int vgic_register_redist_iodev(struct kvm_vcpu *vcpu) rd_dev->nr_regions = ARRAY_SIZE(vgic_v3_rd_registers); rd_dev->redist_vcpu = vcpu; + /* Protected by slots_lock */ + rdreg->free_index++; + mutex_unlock(&kvm->arch.config_lock); ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, rd_base, 2 * SZ_64K, &rd_dev->dev); - if (ret) - return ret; + if (ret) { + guard(mutex)(&vcpu->kvm->arch.config_lock); + vgic_cpu->rdreg->free_index--; + vgic_cpu->rdreg = NULL; + vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF; + } - /* Protected by slots_lock */ - rdreg->free_index++; - return 0; + return ret; out_unlock: mutex_unlock(&kvm->arch.config_lock); @@ -841,7 +846,8 @@ void vgic_unregister_redist_iodev(struct kvm_vcpu *vcpu) kvm_io_bus_unregister_dev(vcpu->kvm, KVM_MMIO_BUS, &rd_dev->dev); } -static int vgic_register_all_redist_iodevs(struct kvm *kvm) +static int vgic_register_all_redist_iodevs(struct kvm *kvm, + struct vgic_redist_region *rdreg) { struct kvm_vcpu *vcpu; unsigned long c; @@ -861,7 +867,14 @@ static int vgic_register_all_redist_iodevs(struct kvm *kvm) for (i = 0; i < c; i++) { vcpu = kvm_get_vcpu(kvm, i); + if (vcpu->arch.vgic_cpu.rdreg != rdreg) + continue; + vgic_unregister_redist_iodev(vcpu); + + guard(mutex)(&vcpu->kvm->arch.config_lock); + vcpu->arch.vgic_cpu.rdreg = NULL; + vcpu->arch.vgic_cpu.rd_iodev.base_addr = VGIC_ADDR_UNDEF; } } @@ -970,10 +983,13 @@ void vgic_v3_free_redist_region(struct kvm *kvm, struct vgic_redist_region *rdre int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count) { + struct vgic_redist_region *rdreg; int ret; mutex_lock(&kvm->arch.config_lock); ret = vgic_v3_alloc_redist_region(kvm, index, addr, count); + if (!ret) + rdreg = vgic_v3_rdist_region_from_index(kvm, index); mutex_unlock(&kvm->arch.config_lock); if (ret) return ret; @@ -982,12 +998,9 @@ int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count) * Register iodevs for each existing VCPU. Adding more VCPUs * afterwards will register the iodevs when needed. */ - ret = vgic_register_all_redist_iodevs(kvm); + ret = vgic_register_all_redist_iodevs(kvm, rdreg); if (ret) { - struct vgic_redist_region *rdreg; - mutex_lock(&kvm->arch.config_lock); - rdreg = vgic_v3_rdist_region_from_index(kvm, index); vgic_v3_free_redist_region(kvm, rdreg); mutex_unlock(&kvm->arch.config_lock); return ret; diff --git a/arch/arm64/kvm/vgic/vgic-mmio.c b/arch/arm64/kvm/vgic/vgic-mmio.c index 74d76dec9730..fddb9da0403d 100644 --- a/arch/arm64/kvm/vgic/vgic-mmio.c +++ b/arch/arm64/kvm/vgic/vgic-mmio.c @@ -1065,6 +1065,9 @@ static int dispatch_mmio_read(struct kvm_vcpu *vcpu, struct kvm_io_device *dev, case IODEV_ITS: data = region->its_read(vcpu->kvm, iodev->its, addr, len); break; + case IODEV_GICV5_IRS: + data = region->read(vcpu, addr, len); + break; } vgic_data_host_to_mmio_bus(val, len, data); @@ -1095,6 +1098,9 @@ static int dispatch_mmio_write(struct kvm_vcpu *vcpu, struct kvm_io_device *dev, case IODEV_ITS: region->its_write(vcpu->kvm, iodev->its, addr, len, data); break; + case IODEV_GICV5_IRS: + region->write(vcpu, addr, len, data); + break; } return 0; diff --git a/arch/arm64/kvm/vgic/vgic-mmio.h b/arch/arm64/kvm/vgic/vgic-mmio.h index 50dc80220b0f..38ed730d68ac 100644 --- a/arch/arm64/kvm/vgic/vgic-mmio.h +++ b/arch/arm64/kvm/vgic/vgic-mmio.h @@ -217,6 +217,8 @@ unsigned int vgic_v2_init_cpuif_iodev(struct vgic_io_device *dev); unsigned int vgic_v3_init_dist_iodev(struct vgic_io_device *dev); +unsigned int vgic_v5_init_irs_iodev(struct vgic_io_device *dev); + u64 vgic_sanitise_outer_cacheability(u64 reg); u64 vgic_sanitise_inner_cacheability(u64 reg); u64 vgic_sanitise_shareability(u64 reg); diff --git a/arch/arm64/kvm/vgic/vgic-v5-tables.c b/arch/arm64/kvm/vgic/vgic-v5-tables.c new file mode 100644 index 000000000000..d577e485ebe9 --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-v5-tables.c @@ -0,0 +1,1872 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025, 2026 Arm Ltd. + */ + +#include <kvm/arm_vgic.h> +#include <linux/dma-mapping.h> +#include <linux/kernel.h> +#include <linux/mm.h> +#include <linux/mutex.h> +#include <linux/sizes.h> +#include <linux/slab.h> +#include <linux/uaccess.h> +#include <linux/xarray.h> +#include <asm/kvm_mmu.h> + +#include "vgic.h" +#include "vgic-v5-tables.h" + +#define irs_caps kvm_vgic_global_state.vgic_v5_irs_caps + +static struct vgic_v5_vmt *vmt_info; + +/* Serialises IRS MMIO interaction (commands) and accesses to IRS tables. */ +DEFINE_RAW_SPINLOCK(vgic_v5_irs_lock); + +/* Serialises lazy installation of shared second-level VMTs. */ +static DEFINE_MUTEX(vmt_l2_lock); + +static DEFINE_XARRAY(vm_info); + +/* Level 1 Virtual Machine Table Entry */ +#define GICV5_VMTEL1E_VALID BIT_ULL(0) +/* Note that there is no shift for the address by design */ +#define GICV5_VMTEL1E_L2_ADDR GENMASK(51, 12) + +#define GICV5_VMTEL2E_SIZE 32ULL +/* An L2 table (two-level VMT) is ALWAYS 4kB! */ +#define GICV5_VMT_L2_TABLE_SIZE 4096ULL +#define GICV5_VMT_L2_TABLE_ENTRIES (GICV5_VMT_L2_TABLE_SIZE / GICV5_VMTEL2E_SIZE) + +/* + * As the L2 VMTE is a large data structure, we are splitting it into 4 parts. + * We only mask and shift WITHIN each part for simplicity. + */ +/* First 64-bit chunk */ +#define GICV5_VMTEL2E_VALID BIT_ULL(0) +#define GICV5_VMTEL2E_VMD_ADDR_SHIFT 3ULL +#define GICV5_VMTEL2E_VMD_ADDR GENMASK_ULL(55, 3) +/* Second 64-bit chunk */ +#define GICV5_VMTEL2E_VPET_ADDR_SHIFT 3ULL +#define GICV5_VMTEL2E_VPET_ADDR GENMASK_ULL(55, 3) +#define GICV5_VMTEL2E_VPE_ID_BITS GENMASK_ULL(63, 59) +/* Third & fourth 64-bit chunks (the encodings are the same for each) */ +#define GICV5_VMTEL2E_IST_VALID BIT_ULL(0) +#define GICV5_VMTEL2E_IST_L2SZ GENMASK_ULL(2, 1) +#define GICV5_VMTEL2E_IST_ADDR_SHIFT 6ULL +#define GICV5_VMTEL2E_IST_ADDR GENMASK_ULL(55, 6) +#define GICV5_VMTEL2E_IST_ISTSZ GENMASK_ULL(57, 56) +#define GICV5_VMTEL2E_IST_STRUCTURE BIT_ULL(58) +#define GICV5_VMTEL2E_IST_ID_BITS GENMASK_ULL(63, 59) + +/* Virtual PE Table Entry */ +#define GICV5_VPE_VALID BIT_ULL(0) +/* Note that there is no shift for the address by design. */ +#define GICV5_VPED_ADDR GENMASK_ULL(55, 3) + +/* L2 Interrupt State Table Entry */ +#define GICV5_ISTL2E_PENDING BIT(0) +#define GICV5_ISTL2E_ACTIVE BIT(1) +#define GICV5_ISTL2E_HM BIT(2) +#define GICV5_ISTL2E_ENABLE BIT(3) +#define GICV5_ISTL2E_IRM BIT(4) +#define GICV5_ISTL2E_HWU GENMASK(10, 9) +#define GICV5_ISTL2E_PRIORITY GENMASK(15, 11) +#define GICV5_ISTL2E_IAFFID GENMASK(31, 16) + +#define GICV5_ISTE_SIZE(istsz) BIT((istsz) + 2) +#define GICV5_LINEAR_IST_SIZE(id_bits, istsz) \ + (BIT(id_bits) * GICV5_ISTE_SIZE(istsz)) + +/* + * The LPI and SPI configuration is stored in the 2nd and 3rd 64-bit chunks of + * the VMTE (0-based). We call this a section here in an attempt to simplify the + * code. + */ +#define GICV5_VMTEL2_LPI_SECTION 2 +#define GICV5_VMTEL2_SPI_SECTION 3 + +struct vgic_v5_ist_desc { + struct vgic_v5_vm_info *vmi; + void *base; + unsigned int id_bits; + unsigned int istsz; + unsigned int l2sz; + size_t iste_size; + bool present; +}; + +struct vgic_v5_two_level_ist_shape { + size_t l1_entries; + size_t l2_entries; +}; + +struct vgic_v5_pending_irq { + u32 irq; + struct list_head next; +}; + +static int vgic_v5_alloc_linear_ist(struct kvm *kvm, bool spi_ist, + unsigned int id_bits, + unsigned int istsz); +static int vgic_v5_alloc_l1_ist(struct kvm *kvm, unsigned int id_bits, + unsigned int istsz, unsigned int l2_split); +static int vgic_v5_alloc_l2_ists(struct kvm *kvm, unsigned int id_bits, + unsigned int istsz, unsigned int l2_split); +static int vgic_v5_alloc_two_level_lpi_ist(struct kvm *kvm, + unsigned int id_bits, + unsigned int istsz, + unsigned int l2_split); +static int vgic_v5_linear_ist_free(struct kvm *kvm, bool spi); +static int vgic_v5_two_level_ist_free(struct kvm *kvm, bool spi); +static int vgic_v5_spi_ist_free(struct kvm *kvm); + +/* + * Our IRS might be coherent or non-coherent. If coherent, we can just emit a + * DSB to ensure that we're in sync. However, when non-coherent, we need to + * manage our cached data explicitly. + * + * This helper is used to handle both coherent and non-coherent IRSes, and + * handles all combinations of cleaning and invalidating to the PoC. + */ +static void vgic_v5_clean_inval(void *va, size_t size) +{ + unsigned long base = (unsigned long)va; + + dsb(ishst); + + if (kvm_vgic_global_state.vgic_v5_irs_caps.non_coherent) + dcache_clean_inval_poc(base, base + size); +} + +static void vgic_v5_drain_pending_irqs(struct kvm *kvm, + struct vgic_v5_vm_info *vmi, + bool reinject) +{ + struct vgic_v5_pending_irq *pirq, *tmp; + + list_for_each_entry_safe(pirq, tmp, &vmi->pending_irqs, next) { + if (reinject) + kvm_call_hyp(__vgic_v5_vdpend, pirq->irq, true, + vgic_v5_vm_id(kvm)); + + list_del(&pirq->next); + kfree(pirq); + } +} + +/* + * Create a linear VM Table. Directly using the number of entries supplied as + * the size of an L2 VMTE (32 bytes) guarantees that our allocation is aligned per + * the GICv5 requirements for the IRS_VMT_BASER. + */ +static int vgic_v5_alloc_vmt_linear(unsigned int num_entries) +{ + vmt_info->linear.vmt_base = kzalloc_objs(*vmt_info->linear.vmt_base, + num_entries); + if (!vmt_info->linear.vmt_base) + return -ENOMEM; + + vgic_v5_clean_inval(vmt_info->linear.vmt_base, + num_entries * sizeof(struct vmtl2_entry)); + + return 0; +} + +/* + * Allocate the first level of a two-level VM table. The second-level VM tables + * are allocated on demand (by vgic_v5_alloc_l2_vmt()). + */ +static int vgic_v5_alloc_vmt_two_level(unsigned int num_entries) +{ + /* + * Each L2 VMT array is always 4k-sized (covering 128 VMs). This is + * mandated by the GICv5 specification (GICv5 EAC0 Specification rule + * D_LSPBK). Hence, round up the number of entries to be at least 128 + * (or the next highest power of two as we give the HW the number of VM + * ID bits). + */ + if (num_entries < GICV5_VMT_L2_TABLE_ENTRIES) + num_entries = GICV5_VMT_L2_TABLE_ENTRIES; + num_entries = roundup_pow_of_two(num_entries); + + vmt_info->l2.num_l1_ents = (num_entries / GICV5_VMT_L2_TABLE_ENTRIES); + vmt_info->l2.vmt_base = kzalloc_objs(*vmt_info->l2.vmt_base, + vmt_info->l2.num_l1_ents); + if (!vmt_info->l2.vmt_base) + return -ENOMEM; + + vmt_info->l2.l2ptrs = kzalloc_objs(*vmt_info->l2.l2ptrs, + vmt_info->l2.num_l1_ents, + GFP_KERNEL); + if (!vmt_info->l2.l2ptrs) { + kfree(vmt_info->l2.vmt_base); + return -ENOMEM; + } + + vgic_v5_clean_inval(vmt_info->l2.vmt_base, + vmt_info->l2.num_l1_ents * sizeof(vmtl1_entry)); + + return 0; +} + +/* + * Allocate a second level VMT, if required. This can be called eagerly, and + * will only perform the allocation if required. + */ +static int vgic_v5_alloc_l2_vmt(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0); + u32 vm_id = vgic_v5_vm_id(kvm); + enum gicv5_vcpu_cmd cmd = VMT_L2_MAP; + struct vmtl2_entry *l2_table; + unsigned int l1_index; + int ret; + + /* Nothing to do if we have linear tables! */ + if (!vmt_info->two_level) + return 0; + + if (vm_id == VGIC_V5_VM_ID_INVAL) + return -EINVAL; + + /* + * We have 4k-sized L2 tables - this is mandated by the spec for + * two-level VMTs (GICv5 EAC0 Specification rule D_LSPBK). This means + * that we have 128 entries per L1 VMTE. + */ + l1_index = vm_id / GICV5_VMT_L2_TABLE_ENTRIES; + + guard(mutex)(&vmt_l2_lock); + + /* Already valid? Great! */ + if (vmt_info->l2.l2ptrs[l1_index]) + return 0; + + l2_table = kzalloc_objs(*l2_table, GICV5_VMT_L2_TABLE_ENTRIES); + if (!l2_table) + return -ENOMEM; + + /* The VMT is shared between all VMs. */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(l2_table, GICV5_VMT_L2_TABLE_SIZE); + vgic_v5_clean_inval(vmt_info->l2.vmt_base + l1_index, + sizeof(vmtl1_entry)); + + WRITE_ONCE(vmt_info->l2.vmt_base[l1_index], + cpu_to_le64(virt_to_phys(l2_table))); + + vgic_v5_clean_inval(vmt_info->l2.vmt_base + l1_index, + sizeof(vmtl1_entry)); + + } + + /* + * VMAP in the L2 VMT via the IRS. We use any of the VM's CPUs as a + * conduit for interacting with the host's IRS. In the current case, + * this lets us resolve the VM ID to pass to the hardware. + */ + ret = irq_set_vcpu_affinity(vgic_v5_vpe_db(vcpu0), &cmd); + + /* We've failed to make the L2 VMT valid - things are very broken! */ + if (ret) { + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + /* Remove the pointer from L1 table */ + WRITE_ONCE(vmt_info->l2.vmt_base[l1_index], 0); + + vgic_v5_clean_inval(vmt_info->l2.vmt_base + l1_index, + sizeof(vmtl1_entry)); + } + + kfree(l2_table); + return ret; + } + + vmt_info->l2.l2ptrs[l1_index] = l2_table; + + return 0; +} + +/* + * Allocate the top-level VMT. This can either be linear or two-level. + */ +int vgic_v5_vmt_allocate(unsigned int max_vpes) +{ + int ret; + + /* Allocate the tracking structure */ + vmt_info = kzalloc_obj(*vmt_info, GFP_KERNEL); + if (!vmt_info) + return -ENOMEM; + + ida_init(&vmt_info->vm_id_ida); + vmt_info->max_vpes = max_vpes; + vmt_info->vmd_size = vgic_v5_irs_vmd_size(&irs_caps); + vmt_info->vped_size = vgic_v5_irs_vped_size(&irs_caps); + vmt_info->two_level = vgic_v5_irs_two_level_vmt_support(&irs_caps); + vmt_info->num_entries = vgic_v5_irs_max_vms(&irs_caps); + + if (vmt_info->two_level) + ret = vgic_v5_alloc_vmt_two_level(vmt_info->num_entries); + else + ret = vgic_v5_alloc_vmt_linear(vmt_info->num_entries); + + /* If anything failed, free our tracking structure before returning */ + if (ret) { + kfree(vmt_info); + vmt_info = NULL; + } + + return ret; +} + +/* + * Free the VMT and associated tracking structures. This isn't strictly expected + * to be called in general operation, but instead exists for completeness. + */ +int vgic_v5_vmt_free(void) +{ + if (!vmt_info) + return 0; + + if (!vmt_info->two_level) { + kfree(vmt_info->linear.vmt_base); + } else { + /* Free the L2 tables; kfree(NULL) is safe */ + for (int i = 0; i < vmt_info->l2.num_l1_ents; ++i) + kfree(vmt_info->l2.l2ptrs[i]); + kfree(vmt_info->l2.l2ptrs); + + /* And now free the L1 table */ + kfree(vmt_info->l2.vmt_base); + } + + ida_destroy(&vmt_info->vm_id_ida); + kfree(vmt_info); + vmt_info = NULL; + + return 0; +} + +/* + * Look up a VMT Entry by VM ID. + */ +static struct vmtl2_entry *vgic_v5_get_l2_vmte(u32 vm_id) +{ + unsigned int l1_index, l2_index; + struct vmtl2_entry *l2_table; + + if (vm_id == VGIC_V5_VM_ID_INVAL) + return ERR_PTR(-EINVAL); + + if (!vmt_info->two_level) + return &vmt_info->linear.vmt_base[vm_id]; + + l1_index = vm_id / GICV5_VMT_L2_TABLE_ENTRIES; + l2_index = vm_id % GICV5_VMT_L2_TABLE_ENTRIES; + + if (l1_index >= vmt_info->l2.num_l1_ents) + return ERR_PTR(-E2BIG); + + if (!vmt_info->l2.l2ptrs[l1_index]) + return ERR_PTR(-EINVAL); + + l2_table = vmt_info->l2.l2ptrs[l1_index]; + return &l2_table[l2_index]; +} + +/* + * Zero a VMT Entry, and flush & invalidate to the PoC, if required. + */ +static int vgic_v5_reset_vmte(struct kvm *kvm) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vmtl2_entry *vmte; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + /* + * The VMT is normal memory shared with the IRS. Invalidate before + * rewriting the entry so that cacheline-granular maintenance cannot + * later push stale data for neighbouring IRS-visible state back to + * memory. + */ + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + + /* + * Prevent the compiler from eliding the individual VMTE + * stores. Ordering and visibility to the IRS are provided by the + * surrounding cache maintenance and command protocol, not by + * WRITE_ONCE(). + * + * The same compiler-access constraint applies to READ_ONCE() users in + * this file: when inspecting IRS-visible table entries, read the field + * exactly once and prevent the compiler from reusing, merging or + * tearing the access. Coherency and freshness for non-coherent IRSes + * still come from the surrounding cache maintenance. + */ + WRITE_ONCE(vmte->val[0], cpu_to_le64(0ULL)); + WRITE_ONCE(vmte->val[1], cpu_to_le64(0ULL)); + WRITE_ONCE(vmte->val[2], cpu_to_le64(0ULL)); + WRITE_ONCE(vmte->val[3], cpu_to_le64(0ULL)); + + /* And make our write visible to the IRS (if non-coherent) */ + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + return 0; +} + +/* + * Use the IDA to allocate a new VM ID, and track it in the gicv5_vm data + * structure. If we're out of VM IDs, the IDA catches that, and we return the + * error (-ENOSPC). If we've previously allocated a VM ID, we catch that too and + * return -EBUSY. + */ +int vgic_v5_allocate_vm_id(struct kvm *kvm) +{ + int id; + + if (kvm->arch.vgic.gicv5_vm.vm_id != VGIC_V5_VM_ID_INVAL) + return -EBUSY; + + id = ida_alloc_max(&vmt_info->vm_id_ida, vmt_info->num_entries - 1u, + GFP_KERNEL); + if (id < 0) + return id; + + kvm->arch.vgic.gicv5_vm.vm_id = id; + + return 0; +} + +/* + * Release the VM ID to allow it to be reallocated in the future. + */ +void vgic_v5_release_vm_id(struct kvm *kvm) +{ + if (kvm->arch.vgic.gicv5_vm.vm_id == VGIC_V5_VM_ID_INVAL) + return; + + ida_free(&vmt_info->vm_id_ida, kvm->arch.vgic.gicv5_vm.vm_id); + kvm->arch.vgic.gicv5_vm.vm_id = VGIC_V5_VM_ID_INVAL; +} + +/* + * Initialise an entry in the VMT based on the index of the VM. + * + * Note: We don't mark the VMTE as valid as this needs to be done by + * the hardware. + */ +int vgic_v5_vmte_init(struct kvm *kvm) +{ + size_t vmd_alloc_size, vpet_alloc_size, vped_alloc_size; + void *vped_base = NULL, *vmd = NULL; + struct vgic_v5_vm_info *vmi = NULL; + u64 tmp, vmte_val0 = 0, vmte_val1; + u32 vm_id = vgic_v5_vm_id(kvm); + int ret, nr_cpus, nr_vcpus; + bool vmi_inserted = false; + struct vmtl2_entry *vmte; + vpe_entry *vpet = NULL; + struct kvm_vcpu *vcpu; + u16 max_vpe_id = 0; + unsigned long i; + + nr_vcpus = atomic_read(&kvm->online_vcpus); + if (nr_vcpus > vmt_info->max_vpes) + return -E2BIG; + + /* + * If we're using two-level VMTs, L2 is allocated on demand. For linear + * VMTs, this is a NOP. + */ + ret = vgic_v5_alloc_l2_vmt(kvm); + if (ret) + return ret; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + /* If the entry is already valid, something went wrong */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + if (le64_to_cpu(READ_ONCE(vmte->val[0])) & GICV5_VMTEL2E_VALID) + return -EINVAL; + } + + ret = vgic_v5_reset_vmte(kvm); + if (ret) + return ret; + + vmi = kzalloc_obj(*vmi); + if (!vmi) { + ret = -ENOMEM; + goto out_fail; + } + + ret = xa_insert(&vm_info, vm_id, vmi, GFP_KERNEL); + if (ret) + goto out_fail; + vmi_inserted = true; + + /* + * If we are restoring the state of a guest, we need to re-inject any + * IRQs that were pending when the state of the guest was originally + * saved. We use the pending_irqs list for this. + */ + INIT_LIST_HEAD(&vmi->pending_irqs); + + /* Allocate and assign the VM Descriptor, if required. */ + if (vmt_info->vmd_size != 0) { + vmd_alloc_size = round_up(vmt_info->vmd_size, + dma_get_cache_alignment()); + vmd = kzalloc(vmd_alloc_size, GFP_KERNEL); + if (!vmd) { + ret = -ENOMEM; + goto out_fail; + } + + /* Stash the VA so we can free it later */ + vmi->vmd_base = vmd; + + tmp = FIELD_PREP(GICV5_VMTEL2E_VMD_ADDR, + virt_to_phys(vmd) >> GICV5_VMTEL2E_VMD_ADDR_SHIFT); + vmte_val0 = tmp; + } + + /* + * Allocate and assign the VPE Table. + * + * First of all, iterate over all vcpus to find the highest VPE ID we + * require - we need to ensure that we have enough storage for all + * vcpu_id values that userspace has picked and not just the total + * number of vcpus. This gives us the number of VPEs required for the + * VM. + * + * Round up the number of VPEs to a whole power of two as we cannot + * describe non-powers-of-two in the VMTE field as it conveys the number + * of ID bits used and not the number of vPEs. IRS_IDR1.IAFFID_BITS is + * encoded as N - 1, so expose at least one VPE ID bit even for a + * single-vCPU VM to keep the views consistent. + */ + kvm_for_each_vcpu(i, vcpu, kvm) { + u16 vpe_id = vgic_v5_vpe_id(vcpu); + + if (vpe_id > max_vpe_id) + max_vpe_id = vpe_id; + } + + nr_cpus = max(2UL, roundup_pow_of_two(max_vpe_id + 1)); + vmi->vpe_id_bits = fls(nr_cpus) - 1; + + vpet_alloc_size = round_up((size_t)nr_cpus * sizeof(*vpet), + dma_get_cache_alignment()); + vpet = kzalloc(vpet_alloc_size, GFP_KERNEL); + if (!vpet) { + ret = -ENOMEM; + goto out_fail; + } + + /* Stash the VA so we can free it later */ + vmi->vpet_base = vpet; + + tmp = FIELD_PREP(GICV5_VMTEL2E_VPET_ADDR, + virt_to_phys(vpet) >> GICV5_VMTEL2E_VPET_ADDR_SHIFT); + tmp |= FIELD_PREP(GICV5_VMTEL2E_VPE_ID_BITS, vmi->vpe_id_bits); + vmte_val1 = tmp; + + /* + * Allocate a dense VPED array indexed by vcpu_idx. The VPET is indexed + * by the potentially sparse vcpu_id, but using that ID here would waste + * memory. Given that this is not userspace visible, we can cheat a bit + * and use the dense index instead. This is the ONLY place that we do + * this. + * + * Round the requested size up to a whole cacheline. kzalloc() gurantees + * natural alignment, so we ensure that the cachelines cannot be shared + * with unrelated slab objects. VPED and cacheline sizes are powers of + * two, so this also preserves the required VPED alignment when a VPED + * is larger than a cacheline. + */ + vped_alloc_size = round_up((size_t)nr_vcpus * vmt_info->vped_size, + dma_get_cache_alignment()); + vped_base = kzalloc(vped_alloc_size, GFP_KERNEL); + if (!vped_base) { + ret = -ENOMEM; + goto out_fail; + } + vmi->vped_base = vped_base; + + if (vmd) + vgic_v5_clean_inval(vmd, vmd_alloc_size); + vgic_v5_clean_inval(vpet, vpet_alloc_size); + vgic_v5_clean_inval(vped_base, vped_alloc_size); + + /* Publish the VMTE while serialising access to the shared VMT. */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + WRITE_ONCE(vmte->val[0], cpu_to_le64(vmte_val0)); + WRITE_ONCE(vmte->val[1], cpu_to_le64(vmte_val1)); + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + kvm->arch.vgic.gicv5_vm.vmte_allocated = true; + + return 0; + +out_fail: + /* kfree(NULL) is safe so we can just kfree() at leisure */ + kfree(vmd); + kfree(vpet); + kfree(vped_base); + if (vmi_inserted) + xa_erase(&vm_info, vm_id); + kfree(vmi); + + vgic_v5_reset_vmte(kvm); + + return ret; +} + +/* + * Release the VMT Entry, freeing up any allocated data structures before + * zeroing the VMTE. + * + * The VMTE must be marked as invalid before it is released. + */ +int vgic_v5_vmte_release(struct kvm *kvm) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + struct vmtl2_entry *vmte; + int ret; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + /* Reject if the VMTE has not been marked as invalid! */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + if (le64_to_cpu(READ_ONCE(vmte->val[0])) & GICV5_VMTEL2E_VALID) + return -EINVAL; + } + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + goto no_vmi; + + /* Unlikely, but possible. Avoid leaking the memory. */ + vgic_v5_drain_pending_irqs(kvm, vmi, false); + + /* If we have an LPI IST, free it */ + if (vmi->h_lpi_ist) { + ret = vgic_v5_lpi_ist_free(kvm); + if (ret) + return ret; + } + vmi->h_lpi_ist = NULL; + + /* If we have an SPI IST, free it */ + if (vmi->h_spi_ist) { + ret = vgic_v5_spi_ist_free(kvm); + if (ret) + return ret; + } + vmi->h_spi_ist = NULL; + + kfree(vmi->vped_base); + kfree(vmi->vpet_base); + kfree(vmi->vmd_base); + + xa_erase(&vm_info, vm_id); + kfree(vmi); + +no_vmi: + /* + * If we didn't get far enough into allocating a VMTE to create the VM + * info structure, then we just zero the VMTE and move on. There's + * nothing else we can realistically do here. + */ + ret = vgic_v5_reset_vmte(kvm); + if (ret) + return ret; + + kvm->arch.vgic.gicv5_vm.vmte_allocated = false; + + return 0; +} + +/* + * Provide a way for the IRS MMIO emulation to correctly populate the number of + * IAFFID bits (which correspond to our vpe_id_bits. + */ +u8 vgic_v5_vmte_vpe_id_bits(struct kvm_vcpu *vcpu) +{ + u32 vm_id = vgic_v5_vm_id(vcpu->kvm); + struct vgic_v5_vm_info *vmi; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return 0; + + return vmi->vpe_id_bits; +} + +/* Provide the preallocated VPE descriptor to the hardware via the VPE Table. */ +int vgic_v5_vmte_alloc_vpe(struct kvm_vcpu *vcpu) +{ + u32 vm_id = vgic_v5_vm_id(vcpu->kvm); + u16 vpe_id = vgic_v5_vpe_id(vcpu); + struct vgic_v5_vm_info *vmi; + vpe_entry *vpet_base; + void *vped; + + /* Make sure we're not over what the hardware supports */ + if (vpe_id >= vmt_info->max_vpes) + return -E2BIG; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + if (vpe_id >= 1 << vmi->vpe_id_bits) + return -E2BIG; + + vpet_base = vmi->vpet_base; + + /* If the VPETE for this CPU is already valid we've gone wrong */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(&vpet_base[vpe_id], sizeof(*vpet_base)); + if (le64_to_cpu(READ_ONCE(vpet_base[vpe_id])) & GICV5_VPE_VALID) + return -EBUSY; + } + + vped = (u8 *)vmi->vped_base + + (size_t)vcpu->vcpu_idx * vmt_info->vped_size; + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + phys_addr_t paddr = virt_to_phys(vped) & GICV5_VPED_ADDR; + WRITE_ONCE(vpet_base[vpe_id], cpu_to_le64(paddr)); + vgic_v5_clean_inval(vpet_base + vpe_id, sizeof(vpe_entry)); + } + + return 0; +} + +/* Clear the VPE's table entry after the VMTE has been made invalid. */ +int vgic_v5_vmte_free_vpe(struct kvm_vcpu *vcpu) +{ + u32 vm_id = vgic_v5_vm_id(vcpu->kvm); + u16 vpe_id = vgic_v5_vpe_id(vcpu); + struct vgic_v5_vm_info *vmi; + struct vmtl2_entry *vmte; + vpe_entry *vpet_base; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + if (le64_to_cpu(READ_ONCE(vmte->val[0])) & GICV5_VMTEL2E_VALID) + return -EBUSY; + } + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + if (vpe_id >= 1 << vmi->vpe_id_bits) + return -E2BIG; + + vpet_base = vmi->vpet_base; + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + WRITE_ONCE(vpet_base[vpe_id], 0ULL); + vgic_v5_clean_inval(vpet_base + vpe_id, sizeof(vpe_entry)); + } + + return 0; +} + +phys_addr_t vgic_v5_get_vmt_base(void) +{ + phys_addr_t vmt_base; + + if (!vmt_info->two_level) + vmt_base = virt_to_phys(vmt_info->linear.vmt_base); + else + vmt_base = virt_to_phys(vmt_info->l2.vmt_base); + + return vmt_base; +} + +/* + * Assign an already allocated IST to the VM by populating the fields in the + * corresponding VMTE. We re-use this code for both an SPI IST and LPI IST, even + * if the paths to reach it might be vastly different. + */ +static int vgic_v5_vmte_assign_ist(struct kvm *kvm, phys_addr_t ist_base, + bool two_level, unsigned int id_bits, + unsigned int l2sz, unsigned int istsz, + bool spi_ist) +{ + struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0); + u32 vm_id = vgic_v5_vm_id(kvm); + enum gicv5_vcpu_cmd cmd; + struct vmtl2_entry *vmte; + unsigned int section; + u64 tmp; + int ret; + + /* + * The L2 VMTE comprises four 64-bit "sections", where sections 2 & 3 + * describe the LPI and SPI ISTs, respectively. Both the LPI and SPI + * sections have the same layout, and as we are either operating on SPIs + * or LPIs we pick a section of the VMTE to modify up-front. + * + * See the GICv5 EAC0 Specification 11.2.2 for more details about the + * VMTE layout. + */ + section = spi_ist ? GICV5_VMTEL2_SPI_SECTION : GICV5_VMTEL2_LPI_SECTION; + + if (ist_base & ~GICV5_VMTEL2E_IST_ADDR) { + pr_err_ratelimited("kvm [%i]: IST misaligned: address 0x%llx, mask 0x%llx\n", + task_pid_nr(current), ist_base, + GICV5_VMTEL2E_IST_ADDR); + return -EINVAL; + } + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + tmp = FIELD_PREP(GICV5_VMTEL2E_IST_L2SZ, l2sz); + tmp |= FIELD_PREP(GICV5_VMTEL2E_IST_ADDR, + ist_base >> GICV5_VMTEL2E_IST_ADDR_SHIFT); + tmp |= FIELD_PREP(GICV5_VMTEL2E_IST_ISTSZ, istsz); + tmp |= FIELD_PREP(GICV5_VMTEL2E_IST_ID_BITS, id_bits); + if (two_level) + tmp |= GICV5_VMTEL2E_IST_STRUCTURE; + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + /* Bail if already allocated */ + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + if (le64_to_cpu(READ_ONCE(vmte->val[section])) & + GICV5_VMTEL2E_IST_VALID) + return -EINVAL; + + WRITE_ONCE(vmte->val[section], cpu_to_le64(tmp)); + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + /* Finally, mark the entry as valid */ + cmd = spi_ist ? SPI_VIST_MAKE_VALID : LPI_VIST_MAKE_VALID; + ret = irq_set_vcpu_affinity(vgic_v5_vpe_db(vcpu0), &cmd); + if (ret) { + /* + * A timeout does not tell us whether the IRS consumed the + * command, so we're in some indeterminate and potentially + * transient state. It may still make the VIST valid and access + * the backing memory, so leave both the VMTE and allocation + * intact. Killing the VM ensures that teardown is the only path + * that can reclaim them, after it has successfully made the + * complete VMTE invalid. + */ + if (ret == -ETIMEDOUT) { + kvm_vm_dead(kvm); + return ret; + } + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + WRITE_ONCE(vmte->val[section], 0ULL); + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + return ret; + } + + return 0; +} + +/* + * Allocate a Linear IST - always used for SPIs and potentially LPIs. + * + * The calculation for n has been taken from section 11.2.2 of the GICv5 EAC0 + * spec. + * + * NOTE: istsz is the FIELD used by GICv5, not the actual size (or log2() of the + * size). + */ +static int vgic_v5_alloc_linear_ist(struct kvm *kvm, bool spi_ist, + unsigned int id_bits, unsigned int istsz) +{ + const size_t n = max(5, id_bits + 1 + istsz); + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + __le64 *ist; + u32 l1sz; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + /* + * Allocate the IST. We only have one level, so we just use the L2 ISTE. + */ + l1sz = BIT(n + 1); + ist = kzalloc(l1sz, GFP_KERNEL_ACCOUNT); + if (!ist) + return -ENOMEM; + + if (spi_ist) { + vmi->h_spi_ist = ist; + } else { + vmi->h_lpi_ist_structure = false; + vmi->h_lpi_ist = ist; + } + + vgic_v5_clean_inval(ist, l1sz); + + return 0; +} + +/* + * Allocate the first level of a two-level IST - LPI, only. + * + * The calculation for n has been taken from section 11.2.2 of the GICv5 EAC0 + * spec. + * + * NOTE: istsz and l2sz are the FIELDS used by GICv5, not the actual sizes (or + * log2() of the sizes). + */ +static int vgic_v5_alloc_l1_ist(struct kvm *kvm, unsigned int id_bits, + unsigned int istsz, unsigned int l2sz) +{ + const size_t n = max(5, id_bits - ((10 - istsz) + (2 * l2sz)) + 3 - 1); + u32 vm_id = vgic_v5_vm_id(kvm); + const u32 l1_size = BIT(n + 1); + struct vgic_v5_vm_info *vmi; + __le64 *ist; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + ist = kzalloc(l1_size, GFP_KERNEL_ACCOUNT); + if (!ist) + return -ENOMEM; + + vmi->h_lpi_ist_structure = true; + vmi->h_lpi_ist = ist; + + vgic_v5_clean_inval(ist, l1_size); + + return 0; +} + +/* + * Allocate ALL of the second level ISTs for a two-level IST - LPI, only. + * + * The calculation for n has been taken from section 11.2.2 of the GICv5 EAC0 + * spec. The l2_size calculation is from section 11.2.3 of the same document. + * + * NOTE: istsz and l2sz are the FIELDS used by GICv5, not the actual sizes (or + * log2() of the sizes). + */ +static int vgic_v5_alloc_l2_ists(struct kvm *kvm, unsigned int id_bits, + unsigned int istsz, unsigned int l2sz) +{ + const size_t n = max(5, id_bits - ((10 - istsz) + (2 * l2sz)) + 3 - 1); + const int l1_entries = BIT(n + 1) / GICV5_IRS_ISTL1E_SIZE; + const size_t l2_size = BIT(11 + (2 * l2sz) + 1); + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + __le64 *l2ist; + __le64 *l1ist; + int index; + u64 val; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + l1ist = vmi->h_lpi_ist; + + /* + * Allocate the storage for the pointers to the L2 ISTs (used when + * freeing later). + */ + vmi->h_lpi_l2_ists = kzalloc_objs(*vmi->h_lpi_l2_ists, l1_entries, + GFP_KERNEL_ACCOUNT); + if (!vmi->h_lpi_l2_ists) + return -ENOMEM; + + /* Allocate the L2 IST for each L1 IST entry */ + for (index = 0; index < l1_entries; ++index) { + l2ist = kzalloc(l2_size, GFP_KERNEL_ACCOUNT); + if (!l2ist) { + while (--index >= 0) + kfree(vmi->h_lpi_l2_ists[index]); + + kfree(vmi->h_lpi_l2_ists); + vmi->h_lpi_l2_ists = NULL; + + return -ENOMEM; + } + + /* + * We are not doing on-demand allocation of the L2 ISTs, and are + * instead provisioning the whole IST up front. This means that + * we are able to mark the L2 ISTs as valid in the L1 ISTEs as + * the overall IST is not yet valid. + */ + val = (virt_to_phys(l2ist) & GICV5_ISTL1E_L2_ADDR_MASK) | + GICV5_ISTL1E_VALID; + l1ist[index] = cpu_to_le64(val); + + vmi->h_lpi_l2_ists[index] = l2ist; + + vgic_v5_clean_inval(l2ist, l2_size); + } + + /* Handle CMOs for the whole L1 IST in one go */ + vgic_v5_clean_inval(l1ist, l1_entries * sizeof(*l1ist)); + + return 0; +} + +/* Allocate a two-level IST - LPIs, only */ +static int vgic_v5_alloc_two_level_lpi_ist(struct kvm *kvm, unsigned int id_bits, + unsigned int istsz, unsigned int l2sz) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + int ret; + + /* + * Allocate the L1 IST first, then all of the L2s. Everything + * is preallocated and we do no on-demand IST allocation. This + * is to avoid needing to track if and when the guest is doing + * on-demand IST allocation. + */ + ret = vgic_v5_alloc_l1_ist(kvm, id_bits, istsz, l2sz); + if (ret) + return ret; + + ret = vgic_v5_alloc_l2_ists(kvm, id_bits, istsz, l2sz); + if (ret) { + /* Free the L1 IST again */ + vmi = xa_load(&vm_info, vm_id); + kfree(vmi->h_lpi_ist); + vmi->h_lpi_ist = NULL; + + return ret; + } + + return 0; +} + +static void vgic_v5_free_allocated_lpi_ist(struct vgic_v5_vm_info *vmi, + unsigned int id_bits, + unsigned int istsz, + unsigned int l2sz) +{ + if (!vmi->h_lpi_ist_structure) { + kfree(vmi->h_lpi_ist); + vmi->h_lpi_ist = NULL; + return; + } + + if (vmi->h_lpi_l2_ists) { + const size_t n = max(5, id_bits - ((10 - istsz) + (2 * l2sz)) + 3 - 1); + const int l1_entries = BIT(n + 1) / GICV5_IRS_ISTL1E_SIZE; + int index; + + for (index = 0; index < l1_entries; ++index) + kfree(vmi->h_lpi_l2_ists[index]); + + kfree(vmi->h_lpi_l2_ists); + vmi->h_lpi_l2_ists = NULL; + } + + kfree(vmi->h_lpi_ist); + vmi->h_lpi_ist = NULL; +} + +static void vgic_v5_free_allocated_spi_ist(struct kvm *kvm) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return; + + kfree(vmi->h_spi_ist); + vmi->h_spi_ist = NULL; +} + +/* + * Free a Linear IST. Can only happen once the VM is dead. + */ +static int vgic_v5_linear_ist_free(struct kvm *kvm, bool spi) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vmtl2_entry *vmte; + struct vgic_v5_vm_info *vmi; + int section; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + if (spi) { + section = GICV5_VMTEL2_SPI_SECTION; + vgic_v5_free_allocated_spi_ist(kvm); + } else { + section = GICV5_VMTEL2_LPI_SECTION; + vgic_v5_free_allocated_lpi_ist(vmi, 0, 0, 0); + } + + /* The VM should be dead here, so we can just zero the VMT section */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + WRITE_ONCE(vmte->val[section], cpu_to_le64(0)); + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + return 0; +} + +/* + * Free a Two-Level IST. Can only happen once the VM is dead. + */ +static int vgic_v5_two_level_ist_free(struct kvm *kvm, bool spi) +{ + unsigned int id_bits, istsz, l2sz; + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + struct vmtl2_entry *vmte; + u64 tmp; + int section; + + /* We don't create two-level SPI ISTs, so freeing is a bad idea! */ + if (spi) + return -EINVAL; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + section = GICV5_VMTEL2_LPI_SECTION; + + if (!vmi->h_lpi_ist_structure) + return -EINVAL; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + tmp = le64_to_cpu(READ_ONCE(vmte->val[section])); + } + + id_bits = FIELD_GET(GICV5_VMTEL2E_IST_ID_BITS, tmp); + istsz = FIELD_GET(GICV5_VMTEL2E_IST_ISTSZ, tmp); + l2sz = FIELD_GET(GICV5_VMTEL2E_IST_L2SZ, tmp); + + vgic_v5_free_allocated_lpi_ist(vmi, id_bits, istsz, l2sz); + + /* The VM must be dead, so we can just zero the VMT section */ + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + WRITE_ONCE(vmte->val[section], cpu_to_le64(0)); + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + } + + return 0; +} + +/* Helper to determine ISTE size based on metadata requirements */ +static unsigned int vgic_v5_ist_istsz(unsigned int id_bits) +{ + if (!vgic_v5_irs_istmd(&irs_caps)) + return GICV5_IRS_IST_CFGR_ISTSZ_4; + + if (id_bits >= vgic_v5_irs_istmd_sz(&irs_caps)) + return GICV5_IRS_IST_CFGR_ISTSZ_16; + + return GICV5_IRS_IST_CFGR_ISTSZ_8; +} + +/* + * Allocate an IST for SPIs. + * + * We don't anticipate a large number of SPIs being allocated. Therefore, we + * always allocate a Linear IST for SPIs. This will need to be revisited should + * that assumption no longer hold. + */ +int vgic_v5_spi_ist_alloc(struct kvm *kvm, unsigned int id_bits) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + phys_addr_t base_addr; + unsigned int istsz; + int ret; + + lockdep_assert_held(&kvm->arch.config_lock); + + istsz = vgic_v5_ist_istsz(id_bits); + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + if (vmi->h_spi_ist) + return -EBUSY; + + ret = vgic_v5_alloc_linear_ist(kvm, true, id_bits, istsz); + if (ret) + return ret; + base_addr = virt_to_phys(vmi->h_spi_ist); + + ret = vgic_v5_vmte_assign_ist(kvm, base_addr, false, id_bits, 0, istsz, + true); + if (ret) { + if (ret != -ETIMEDOUT) + vgic_v5_free_allocated_spi_ist(kvm); + return ret; + } + + return 0; +} + +/* + * Free the IST for SPIs. Should only happen once the VM is dead. + */ +static int vgic_v5_spi_ist_free(struct kvm *kvm) +{ + lockdep_assert_held(&kvm->arch.config_lock); + + return vgic_v5_linear_ist_free(kvm, true); +} + +int vgic_v5_lpi_ist_exists(struct kvm *kvm) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -ENXIO; + + return !!vmi->h_lpi_ist; +} + +/* + * Allocate an IST for LPIs. + * + * Unlike with SPIs, we anticipate that the guest will allocate a relatively + * large number of LPIs. Therefore, while we support doing a linear LPI IST, it + * is expected that LPI ISTs will be two-level. + */ +int vgic_v5_lpi_ist_alloc(struct kvm *kvm, unsigned int id_bits) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + unsigned int istsz, l2sz; + phys_addr_t phys_addr; + bool two_level; + int ret; + + lockdep_assert_held(&kvm->arch.config_lock); + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -EINVAL; + + if (vmi->h_lpi_ist) + return -EBUSY; + + istsz = vgic_v5_ist_istsz(id_bits); + l2sz = gicv5_irs_l2_sz(vgic_v5_irs_ist_l2sz(&irs_caps)); + + /* + * Determine if we want to create a Linear or a Two-Level IST. + * + * A two-level IST is only required when a single L2 IST cannot cover + * the requested ID space. This depends on the L2 IST size selected for + * the IRS, not PAGE_SIZE. Using PAGE_SIZE here would switch to + * two-level too early when the selected L2 IST is larger than a page, + * and the allocation sizing arithmetic would underflow. + */ + two_level = vgic_v5_irs_ist_levels(&irs_caps) && + id_bits > ((10 - istsz) + (2 * l2sz)); + + if (!two_level) + ret = vgic_v5_alloc_linear_ist(kvm, false /* LPIs, not SPIs */, + id_bits, istsz); + else + ret = vgic_v5_alloc_two_level_lpi_ist(kvm, id_bits, istsz, + l2sz); + + if (ret) + return ret; + + phys_addr = virt_to_phys(vmi->h_lpi_ist); + ret = vgic_v5_vmte_assign_ist(kvm, phys_addr, two_level, id_bits, l2sz, + istsz, false); + if (ret && ret != -ETIMEDOUT) + vgic_v5_free_allocated_lpi_ist(vmi, id_bits, istsz, l2sz); + + return ret; +} + +/* Free the LPI IST again */ +int vgic_v5_lpi_ist_free(struct kvm *kvm) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + + lockdep_assert_held(&kvm->arch.config_lock); + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -ENXIO; + + if (!vmi->h_lpi_ist_structure) + return vgic_v5_linear_ist_free(kvm, false); + else + return vgic_v5_two_level_ist_free(kvm, false); +} + +static struct vgic_v5_two_level_ist_shape +vgic_v5_two_level_ist_shape(const struct vgic_v5_ist_desc *ist) +{ + struct vgic_v5_two_level_ist_shape shape; + size_t l2bits, n; + + l2bits = (10 - ist->istsz) + (2 * ist->l2sz); + n = max(2, ist->id_bits - l2bits + 3 - 1); + + shape.l1_entries = BIT(n + 1) / GICV5_IRS_ISTL1E_SIZE; + shape.l2_entries = BIT(l2bits); + + return shape; +} + +static int vgic_v5_read_vm_ist_desc(struct kvm *kvm, unsigned int section, + struct vgic_v5_ist_desc *ist) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vmtl2_entry *vmte; + u64 vmte_ist_section; + + vmte = vgic_v5_get_l2_vmte(vm_id); + if (IS_ERR(vmte)) + return PTR_ERR(vmte); + + scoped_guard(raw_spinlock_irqsave, &vgic_v5_irs_lock) { + vgic_v5_clean_inval(vmte, sizeof(*vmte)); + vmte_ist_section = le64_to_cpu(READ_ONCE(vmte->val[section])); + } + + ist->id_bits = FIELD_GET(GICV5_VMTEL2E_IST_ID_BITS, vmte_ist_section); + ist->istsz = FIELD_GET(GICV5_VMTEL2E_IST_ISTSZ, vmte_ist_section); + ist->l2sz = FIELD_GET(GICV5_VMTEL2E_IST_L2SZ, vmte_ist_section); + ist->iste_size = GICV5_ISTE_SIZE(ist->istsz); + + return !!(vmte_ist_section & GICV5_VMTEL2E_IST_VALID); +} + +static int vgic_v5_get_spi_ist_desc(struct kvm *kvm, + struct vgic_v5_ist_desc *ist) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + int ret; + + memset(ist, 0, sizeof(*ist)); + + ist->vmi = xa_load(&vm_info, vm_id); + if (!ist->vmi) + return -ENXIO; + + ret = vgic_v5_read_vm_ist_desc(kvm, GICV5_VMTEL2_SPI_SECTION, ist); + if (ret < 0) + return ret; + + ist->base = ist->vmi->h_spi_ist; + if (!ret || !ist->base) + return -ENXIO; + + ist->present = true; + return 0; +} + +static int vgic_v5_get_lpi_ist_desc(struct kvm *kvm, + struct vgic_v5_ist_desc *ist) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + bool guest_valid, host_valid; + int ret; + + memset(ist, 0, sizeof(*ist)); + + ist->vmi = xa_load(&vm_info, vm_id); + if (WARN_ON_ONCE(!ist->vmi)) + return -ENXIO; + + ret = vgic_v5_read_vm_ist_desc(kvm, GICV5_VMTEL2_LPI_SECTION, ist); + if (ret < 0) + return ret; + + host_valid = ret; + guest_valid = kvm->arch.vgic.vgic_v5_irs_data->ist_baser.valid; + ist->base = ist->vmi->h_lpi_ist; + + /* If there is no IST to save/restore, return without error. */ + if (!guest_valid && !host_valid && !ist->base) + return 0; + + /* Mismatched combination of valid state */ + if (!guest_valid || !host_valid || !ist->base) + return -ENXIO; + + if (ist->vmi->h_lpi_ist_structure && !ist->vmi->h_lpi_l2_ists) + return -ENXIO; + + ist->present = true; + return 0; +} + +/* + * Save a linear host IST to userspace memory. + * + * Only the architected 32-bit ISTE state is stored. Metadata is skipped when + * striding through the host IST. + */ +static int vgic_v5_save_linear_ist(const struct vgic_v5_ist_desc *ist, + u32 __user *uaddr, size_t nr_entries) +{ + size_t index; + int ret; + + vgic_v5_clean_inval(ist->base, + GICV5_LINEAR_IST_SIZE(ist->id_bits, ist->istsz)); + + for (index = 0; index < nr_entries; index++) { + __le32 *h_iste_addr = ist->base + index * ist->iste_size; + u32 h_iste; + + h_iste = le32_to_cpu(READ_ONCE(*h_iste_addr)); + ret = put_user(h_iste, uaddr); + if (ret) + return ret; + + uaddr++; + } + + return 0; +} + +/* + * Save a two-level host IST to userspace memory. + * + * Only the architected 32-bit ISTE state is stored. Metadata is skipped when + * striding through the host IST. + */ +static int vgic_v5_save_two_level_ist(const struct vgic_v5_ist_desc *ist, + u32 __user *uaddr) +{ + struct vgic_v5_two_level_ist_shape shape; + size_t h_l1_index, h_l2_index; + void *h_l2_ist_base; + int ret; + + shape = vgic_v5_two_level_ist_shape(ist); + + vgic_v5_clean_inval(ist->base, + shape.l1_entries * sizeof(*ist->vmi->h_lpi_ist)); + + for (h_l1_index = 0; h_l1_index < shape.l1_entries; h_l1_index++) { + u64 l1_iste; + + /* + * Host L2 ISTs are preallocated. Any invalid L1 entry means the + * host IST state is inconsistent. + */ + l1_iste = le64_to_cpu(READ_ONCE(ist->vmi->h_lpi_ist[h_l1_index])); + if (!FIELD_GET(GICV5_ISTL1E_VALID, l1_iste)) + return -ENXIO; + + h_l2_ist_base = ist->vmi->h_lpi_l2_ists[h_l1_index]; + if (!h_l2_ist_base) + return -ENXIO; + + vgic_v5_clean_inval(h_l2_ist_base, + shape.l2_entries * ist->iste_size); + + for (h_l2_index = 0; h_l2_index < shape.l2_entries; h_l2_index++) { + void *h_iste_addr = h_l2_ist_base + h_l2_index * ist->iste_size; + u32 h_iste = le32_to_cpu(*(__le32 *)h_iste_addr); + + ret = put_user(h_iste, uaddr); + if (ret) + return ret; + + uaddr++; + } + } + + return 0; +} + +/* + * Save the SPI IST to userspace-provided memory. + */ +int vgic_v5_save_spi_ist(struct kvm *kvm, struct kvm_vgic_v5_ist *ist_attr) +{ + struct vgic_v5_ist_desc ist; + u32 __user *uaddr; + int ret; + + ret = vgic_v5_get_spi_ist_desc(kvm, &ist); + if (ret) + return ret; + + uaddr = (u32 __user *)(unsigned long)ist_attr->spi_ist_addr; + + /* The host SPI IST is always linear. */ + return vgic_v5_save_linear_ist(&ist, uaddr, + kvm->arch.vgic.nr_spis); +} + +/* + * Save the LPI IST to userspace memory. + * + * The LPI IST may be linear or two-level, so host iteration depends on the + * allocated host shape. + */ +int vgic_v5_save_lpi_ist(struct kvm *kvm, struct kvm_vgic_v5_ist *ist_attr) +{ + struct vgic_v5_ist_desc ist; + u32 __user *uaddr; + int ret; + + ret = vgic_v5_get_lpi_ist_desc(kvm, &ist); + if (ret) + return ret; + + if (!ist.present) + return 0; + + /* + * Userspace sized the buffer from the guest-visible configuration. + * Refuse to walk a host IST with a different number of entries. + */ + if (ist.id_bits != + kvm->arch.vgic.vgic_v5_irs_data->ist_cfgr.lpi_id_bits) + return -EINVAL; + + uaddr = (u32 __user *)(unsigned long)ist_attr->lpi_ist_addr; + + if (!ist.vmi->h_lpi_ist_structure) + return vgic_v5_save_linear_ist(&ist, uaddr, + BIT(ist.id_bits)); + + return vgic_v5_save_two_level_ist(&ist, uaddr); +} + +/* + * Track any SPIs and LPIs that were marked as pending at the point where the + * IST was restored. + * + * Restored pending state is cleared from the host ISTE and replayed with VDPEND + * before the VM first runs. + */ +static int vgic_v5_track_pending_irq(struct list_head *pending_irqs, u32 intid, + u32 type) +{ + struct vgic_v5_pending_irq *pirq; + + pirq = kzalloc_obj(*pirq, GFP_KERNEL); + if (!pirq) + return -ENOMEM; + + /* Encode the interrupt as a GICv5 IntID. */ + pirq->irq = FIELD_PREP(GICV5_HWIRQ_TYPE, type) | + FIELD_PREP(GICV5_HWIRQ_ID, intid); + + INIT_LIST_HEAD(&pirq->next); + list_add_tail(&pirq->next, pending_irqs); + + return 0; +} + +/* + * Process and sanitise each restored ISTE. + * + * HWU is for hardware use and must not survive migration. Pending state is + * tracked, cleared from the ISTE, and replayed before the VM first runs. + */ +static int vgic_v5_process_iste(__le32 *iste, struct list_head *pending_irqs, + u32 intid, u32 type) +{ + u32 iste_data = le32_to_cpu(READ_ONCE(*iste)); + int ret; + + /* Pending state is replayed later with VDPEND. */ + if (iste_data & GICV5_ISTL2E_PENDING) { + ret = vgic_v5_track_pending_irq(pending_irqs, intid, type); + if (ret) + return ret; + } + + iste_data &= ~GICV5_ISTL2E_PENDING; + iste_data &= ~GICV5_ISTL2E_HWU; + + WRITE_ONCE(*iste, cpu_to_le32(iste_data)); + + return 0; +} + +static void vgic_v5_restore_spi_config(struct kvm *kvm, __le32 iste, u32 spi) +{ + u32 iste_data = le32_to_cpu(iste); + bool pending = iste_data & GICV5_ISTL2E_PENDING; + struct vgic_irq *irq; + unsigned long flags; + + irq = vgic_get_irq(kvm, vgic_v5_make_spi(spi)); + if (WARN_ON_ONCE(!irq)) + return; + + raw_spin_lock_irqsave(&irq->irq_lock, flags); + + if (iste_data & GICV5_ISTL2E_HM) + irq->config = VGIC_CONFIG_LEVEL; + else + irq->config = VGIC_CONFIG_EDGE; + + if (irq->config == VGIC_CONFIG_EDGE) + irq->pending_latch = pending; + else if (pending) + irq->pending_latch = true; + else if (!irq->active) + irq->pending_latch = false; + + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + vgic_put_irq(kvm, irq); +} + +static int vgic_v5_restore_ist_entry(struct kvm *kvm, + const struct vgic_v5_ist_desc *ist, + void *h_iste_addr, __le32 h_iste, + u32 intid, u32 intid_type) +{ + __le32 raw_iste = h_iste; + int ret; + + /* + * Sanitise the IST, clearing HWU & pending fields. Pending state is + * later replayed via GIC VDPEND. + */ + ret = vgic_v5_process_iste(&h_iste, &ist->vmi->pending_irqs, + intid, intid_type); + if (ret) + return ret; + + if (intid_type == GICV5_HWIRQ_TYPE_SPI) + vgic_v5_restore_spi_config(kvm, raw_iste, intid); + + /* + * Zero the full ISTE (incl metadata), and write back the non-metadata + * region, only. + */ + memset(h_iste_addr, 0, ist->iste_size); + WRITE_ONCE(*(__le32 *)h_iste_addr, h_iste); + vgic_v5_clean_inval(h_iste_addr, ist->iste_size); + + return 0; +} + +/* + * Restore a userspace IST image to a linear host IST. + * + * The userspace IST image is a linear array of 32-bit ISTEs. + */ +static int vgic_v5_restore_linear_ist(struct kvm *kvm, + const struct vgic_v5_ist_desc *ist, + u32 __user *uaddr, size_t nr_entries, + u32 intid_type) +{ + size_t index; + int ret; + + for (index = 0; index < nr_entries; index++) { + void *h_iste_addr = ist->base + index * ist->iste_size; + u32 h_iste; + + ret = get_user(h_iste, uaddr); + if (ret) + return ret; + + ret = vgic_v5_restore_ist_entry(kvm, ist, h_iste_addr, + cpu_to_le32(h_iste), index, intid_type); + if (ret) + return ret; + + uaddr++; + } + + return 0; +} + +/* + * Restore a userspace IST image to a two-level host IST. + * + * The userspace IST image is a linear array of 32-bit ISTEs. + */ +static int vgic_v5_restore_two_level_ist(struct kvm *kvm, + const struct vgic_v5_ist_desc *ist, + u32 __user *uaddr, u32 intid_type) +{ + struct vgic_v5_two_level_ist_shape shape; + size_t h_l1_index, h_l2_index; + void *h_l2_ist_base; + int ret; + + shape = vgic_v5_two_level_ist_shape(ist); + + vgic_v5_clean_inval(ist->vmi->h_lpi_ist, + shape.l1_entries * sizeof(*ist->vmi->h_lpi_ist)); + + for (h_l1_index = 0; h_l1_index < shape.l1_entries; ++h_l1_index) { + u64 l1_iste; + + /* + * Host L2 ISTs are preallocated. Any invalid L1 entry means the + * host IST state is inconsistent. + */ + l1_iste = le64_to_cpu(READ_ONCE(ist->vmi->h_lpi_ist[h_l1_index])); + if (!FIELD_GET(GICV5_ISTL1E_VALID, l1_iste)) + return -ENXIO; + + h_l2_ist_base = ist->vmi->h_lpi_l2_ists[h_l1_index]; + if (!h_l2_ist_base) + return -ENXIO; + + for (h_l2_index = 0; h_l2_index < shape.l2_entries; h_l2_index++) { + void *h_iste_addr = h_l2_ist_base + + h_l2_index * ist->iste_size; + u32 intid = h_l1_index * shape.l2_entries + h_l2_index; + u32 h_iste; + + ret = get_user(h_iste, uaddr); + if (ret) + return ret; + + ret = vgic_v5_restore_ist_entry(kvm, ist, h_iste_addr, + cpu_to_le32(h_iste), intid, + intid_type); + if (ret) + return ret; + + uaddr++; + } + } + + return 0; +} + +/* + * Restore the SPI IST from userspace-provided buffer to the host-allocated IST. + */ +int vgic_v5_restore_spi_ist(struct kvm *kvm, struct kvm_vgic_v5_ist *ist_attr) +{ + u32 __user *uaddr; + struct vgic_v5_ist_desc ist; + int ret; + + ret = vgic_v5_get_spi_ist_desc(kvm, &ist); + if (ret) + return ret; + + uaddr = (u32 __user *)(unsigned long)ist_attr->spi_ist_addr; + + /* The host SPI IST is always linear. */ + return vgic_v5_restore_linear_ist(kvm, &ist, uaddr, + kvm->arch.vgic.nr_spis, + GICV5_HWIRQ_TYPE_SPI); +} + +/* + * Restore the LPI IST from userspace memory to the host-allocated LPI IST. + * + * The host LPI IST may be linear or two-level, so host iteration depends on the + * host IST's shape. + */ +int vgic_v5_restore_lpi_ist(struct kvm *kvm, struct kvm_vgic_v5_ist *ist_attr) +{ + u32 __user *uaddr; + struct vgic_v5_ist_desc ist; + int ret; + + ret = vgic_v5_get_lpi_ist_desc(kvm, &ist); + if (ret) + return ret; + + if (!ist.present) + return 0; + + uaddr = (u32 __user *)(unsigned long)ist_attr->lpi_ist_addr; + + if (!ist.vmi->h_lpi_ist_structure) + return vgic_v5_restore_linear_ist(kvm, &ist, uaddr, + BIT(ist.id_bits), + GICV5_HWIRQ_TYPE_LPI); + + return vgic_v5_restore_two_level_ist(kvm, &ist, uaddr, + GICV5_HWIRQ_TYPE_LPI); +} + +/* + * Process the pending IRQs removing them from the list and optionally injecting + * them. + */ +static int vgic_v5_process_pending_irqs(struct kvm *kvm, bool inject) +{ + u32 vm_id = vgic_v5_vm_id(kvm); + struct vgic_v5_vm_info *vmi; + + lockdep_assert_held(&kvm->arch.config_lock); + + vmi = xa_load(&vm_info, vm_id); + if (!vmi) + return -ENXIO; + + vgic_v5_drain_pending_irqs(kvm, vmi, inject); + + return 0; +} + +/* Replay pending state that was cleared while restoring guest IST state. */ +int vgic_v5_restore_pending_irqs(struct kvm *kvm) +{ + return vgic_v5_process_pending_irqs(kvm, true); +} + +/* Drop pending state collected by a failed IST restore. */ +void vgic_v5_discard_pending_irqs(struct kvm *kvm) +{ + vgic_v5_process_pending_irqs(kvm, false); +} diff --git a/arch/arm64/kvm/vgic/vgic-v5-tables.h b/arch/arm64/kvm/vgic/vgic-v5-tables.h new file mode 100644 index 000000000000..89096926563f --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-v5-tables.h @@ -0,0 +1,140 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025, 2026 Arm Ltd. + */ + +#ifndef __KVM_ARM_VGICV5_TABLES_H__ +#define __KVM_ARM_VGICV5_TABLES_H__ + +#include <linux/spinlock.h> +#include <linux/idr.h> +#include <linux/irqchip/arm-gic-v5.h> +#include <linux/list.h> + +/* Level 1 Virtual Machine Table Entry */ +typedef __le64 vmtl1_entry; + +/* Level 2 Virtual Machine Table Entry */ +struct vmtl2_entry { + __le64 val[4]; +}; + +/* Virtual PE Table Entry */ +typedef __le64 vpe_entry; + +struct vgic_v5_vm_info { + void *vmd_base; + vpe_entry *vpet_base; + void *vped_base; + u8 vpe_id_bits; + + /* + * Both the LPI and SPI ISTs are allocated by the hypervisor. While it + * would be possible to track and access them by iterating over the ISTs + * themselves, it makes more sense to store pointers to the ISTs. + * + * The LPI IST can either be two-level or linear. Hence, we keep track + * of the structure. If it is two-level, we retain pointers to the L1 + * IST and to each L2 IST array. If it is linear, we just store the base + * address of the IST array. + * + * The SPI IST is linear, and therefore we just store the base address + * of the SPI IST array. + */ + bool h_lpi_ist_structure; + __le64 *h_lpi_ist; + __le64 **h_lpi_l2_ists; + __le64 *h_spi_ist; + + /* Tracking of pending interrupts as part of IST restore */ + struct list_head pending_irqs; +}; + +struct vgic_v5_vmt { + union { + struct { + struct vmtl2_entry *vmt_base; + unsigned int num_ents; + } linear; + struct { + vmtl1_entry *vmt_base; + struct vmtl2_entry **l2ptrs; + unsigned int num_l1_ents; + } l2; + }; + bool two_level; + unsigned int num_entries; + unsigned int max_vpes; + size_t vmd_size; + size_t vped_size; + struct ida vm_id_ida; +}; + +static inline u32 vgic_v5_vm_id(struct kvm *kvm) +{ + return kvm->arch.vgic.gicv5_vm.vm_id; +} + +/* + * For vGICv5, we need to consolidate two views: + * - The vcpu ID assigned by userspace, which it can assign however it + * sees fit. + * - The index into the VPET, which is also presented to the guest as + * the IAFFID for the VPE in question. + * + * Ideally, we don't want to intercept the guest setting an interrupt's affinity + * - this would somewhat defeat the point of allowing the hardware to manage the + * interrupt directly. Therefore, we need to use vcpu->vcpu_id as our VPE ID, + * which ensures that both userspace and the GICv5 hardware have a consistent + * view. + * + * This means two things: + * - Our VPET might be sparse, depending on the IDs picked by userspace. + * - We need to check the vcpu ID provided by userspace, and reject anything + * we cannot handle. + * + * In order to at least ensure consistent usage in the places that we need to + * use the VPE ID, we provide this helper. + */ +static inline u16 vgic_v5_vpe_id(struct kvm_vcpu *vcpu) +{ + return vcpu->vcpu_id; +} + +static inline int vgic_v5_vpe_db(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.vgic_cpu.vgic_v5.gicv5_vpe.db; +} + +extern raw_spinlock_t vgic_v5_irs_lock; + +int vgic_v5_vmt_allocate(unsigned int max_vpes); +int vgic_v5_vmt_free(void); +phys_addr_t vgic_v5_get_vmt_base(void); + +int vgic_v5_allocate_vm_id(struct kvm *kvm); +void vgic_v5_release_vm_id(struct kvm *kvm); + +int vgic_v5_vmte_init(struct kvm *kvm); +int vgic_v5_vmte_release(struct kvm *kvm); +u8 vgic_v5_vmte_vpe_id_bits(struct kvm_vcpu *vcpu); +int vgic_v5_vmte_alloc_vpe(struct kvm_vcpu *vcpu); +int vgic_v5_vmte_free_vpe(struct kvm_vcpu *vcpu); + +int vgic_v5_spi_ist_alloc(struct kvm *kvm, unsigned int id_bits); +int vgic_v5_lpi_ist_exists(struct kvm *kvm); +int vgic_v5_lpi_ist_alloc(struct kvm *kvm, unsigned int id_bits); +int vgic_v5_lpi_ist_free(struct kvm *kvm); + +int vgic_v5_save_spi_ist(struct kvm *kvm, + struct kvm_vgic_v5_ist *ist_attr); +int vgic_v5_save_lpi_ist(struct kvm *kvm, + struct kvm_vgic_v5_ist *ist_attr); +int vgic_v5_restore_spi_ist(struct kvm *kvm, + struct kvm_vgic_v5_ist *ist_attr); +int vgic_v5_restore_lpi_ist(struct kvm *kvm, + struct kvm_vgic_v5_ist *ist_attr); +int vgic_v5_restore_pending_irqs(struct kvm *kvm); +void vgic_v5_discard_pending_irqs(struct kvm *kvm); + +#endif diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c index 4e3e4db24e77..91810fbd99af 100644 --- a/arch/arm64/kvm/vgic/vgic-v5.c +++ b/arch/arm64/kvm/vgic/vgic-v5.c @@ -7,10 +7,18 @@ #include <linux/bitops.h> #include <linux/irqchip/arm-vgic-info.h> +#include <linux/irqdomain.h> +#include <linux/kvm_host.h> +#include <linux/uaccess.h> +#include "vgic-v5-tables.h" #include "vgic.h" #define ppi_caps kvm_vgic_global_state.vgic_v5_ppi_caps +#define irs_caps kvm_vgic_global_state.vgic_v5_irs_caps + +static int vgic_v5_irs_assign_vmt(bool two_level, u8 vm_id_bits, phys_addr_t vmt_base); +static int vgic_v5_irs_clear_vmt(void); /* * Not all PPIs are guaranteed to be implemented for GICv5. Deterermine which @@ -34,6 +42,36 @@ static void vgic_v5_get_implemented_ppis(void) __assign_bit(GICV5_ARCH_PPI_PMUIRQ, ppi_caps.impl_ppi_mask, system_supports_pmuv3()); } +static u32 irs_readl_relaxed(const u32 reg_offset) +{ + return readl_relaxed(irs_caps.irs_base + reg_offset); +} + +static void irs_writel_relaxed(const u32 val, const u32 reg_offset) +{ + writel_relaxed(val, irs_caps.irs_base + reg_offset); +} + +static u64 irs_readq_relaxed(const u32 reg_offset) +{ + return readq_relaxed(irs_caps.irs_base + reg_offset); +} + +static void irs_writeq_relaxed(const u64 val, const u32 reg_offset) +{ + writeq_relaxed(val, irs_caps.irs_base + reg_offset); +} + +static void vgic_v5_irs_cache_id_regs(const struct gic_kvm_info *info) +{ + irs_caps.irs_base = info->gicv5_irs.base; + irs_caps.non_coherent = info->gicv5_irs.non_coherent; + + irs_caps.idr2 = irs_readl_relaxed(GICV5_IRS_IDR2); + irs_caps.idr3 = irs_readl_relaxed(GICV5_IRS_IDR3); + irs_caps.idr4 = irs_readl_relaxed(GICV5_IRS_IDR4); +} + /* * Probe for a vGICv5 compatible interrupt controller, returning 0 on success. */ @@ -43,6 +81,8 @@ int vgic_v5_probe(const struct gic_kvm_info *info) int ret; kvm_vgic_global_state.type = VGIC_V5; + kvm_vgic_global_state.max_gic_vcpus = 0; + kvm_vgic_global_state.max_gicv5_vcpus = 0; kvm_vgic_global_state.vcpu_base = 0; kvm_vgic_global_state.vctrl_base = NULL; @@ -60,17 +100,65 @@ int vgic_v5_probe(const struct gic_kvm_info *info) goto skip_v5; } - kvm_vgic_global_state.max_gic_vcpus = VGIC_V5_MAX_CPUS; + if (!gicv5_global_data.lpi_domain) { + kvm_err("GICv5 LPI domain unavailable\n"); + return -ENODEV; + } + vgic_v5_irs_cache_id_regs(info); vgic_v5_get_implemented_ppis(); + /* + * Even if the HW supports more per-VM vCPUs, artificially cap as we + * can't use them all. + */ + kvm_vgic_global_state.max_gicv5_vcpus = min(vgic_v5_irs_max_vpes(&irs_caps), + VGIC_V5_MAX_CPUS); + + /* + * GICv5 requires a set of tables to be allocated in order to manage + * VMs. We allocate them in advance here, which alas means that we + * already have to make a decisions regarding the maximum number of VMs + * we want to run. For now, we match the maximum number offered by the + * hardware, but this might not be a wise choice in the long term. + */ + ret = vgic_v5_vmt_allocate(kvm_vgic_global_state.max_gicv5_vcpus); + if (ret) { + kvm_err("Failed to allocate the GICv5 VM tables; no GICv5 support\n"); + return -ENODEV; + } + + /* + * We've now allocated the VM table, but the host's IRS doesn't know + * about it yet. Provide the base address of the VMT to the IRS, as well + * as the number of ID bits that it covers and the structure used + * (linear/two-level). + */ + ret = vgic_v5_irs_assign_vmt(vgic_v5_irs_two_level_vmt_support(&irs_caps), + ilog2(vgic_v5_irs_max_vms(&irs_caps)), + vgic_v5_get_vmt_base()); + if (ret) { + kvm_err("Failed to assign the GICv5 VM tables to the IRS; no GICv5 support\n"); + if (!vgic_v5_irs_clear_vmt()) + vgic_v5_vmt_free(); + return -ENODEV; + } + ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V5); if (ret) { kvm_err("Cannot register GICv5 KVM device.\n"); - goto skip_v5; + /* + * Don't free the VMT itself if the hardware still has a valid + * pointer to it. + */ + if (!vgic_v5_irs_clear_vmt()) + vgic_v5_vmt_free(); + return -ENODEV; } v5_registered = true; + kvm_vgic_global_state.max_gic_vcpus = + kvm_vgic_global_state.max_gicv5_vcpus; kvm_info("GCIE system register CPU interface\n"); skip_v5: @@ -92,12 +180,12 @@ skip_v5: ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V3); if (ret) { kvm_err("Cannot register GICv3-legacy KVM device.\n"); - return ret; + /* vGICv5 should still work */ + return v5_registered ? 0 : ret; } - /* We potentially limit the max VCPUs further than we need to here */ - kvm_vgic_global_state.max_gic_vcpus = min(VGIC_V3_MAX_CPUS, - VGIC_V5_MAX_CPUS); + kvm_vgic_global_state.max_gic_vcpus = max(kvm_vgic_global_state.max_gic_vcpus, + VGIC_V3_MAX_CPUS); static_branch_enable(&kvm_vgic_global_state.gicv3_cpuif); kvm_info("GCIE legacy system register CPU interface\n"); @@ -107,6 +195,411 @@ skip_v5: return 0; } +/* + * Wait for completion of a change in any of IRS_VMT_BASER, IRS_VMAP_L2_VMTR, + * IRS_VMAP_VMR, IRS_VMAP_VPER, IRS_VMAP_VISTR, IRS_VMAP_L2_VISTR. + */ +static int vgic_v5_irs_wait_for_vm_op(void) +{ + return gicv5_wait_for_op_atomic(irs_caps.irs_base, + GICV5_IRS_VMT_STATUSR, + GICV5_IRS_VMT_STATUSR_IDLE, + NULL); +} + +/* + * Wait for completion of a change in any of IRS_VPE_SELR, IRS_VPE_DBR, + * IRS_VPE_CR0. + */ +static int vgic_v5_irs_wait_for_vpe_op(void) +{ + return gicv5_wait_for_op_atomic(irs_caps.irs_base, + GICV5_IRS_VPE_STATUSR, + GICV5_IRS_VPE_STATUSR_IDLE, + NULL); +} + +/* + * Wait for a write to IRS_SAVE_VMR to complete. + */ +static int vgic_v5_irs_wait_for_save_vm_op(u32 *statusr) +{ + return gicv5_wait_for_op_atomic(irs_caps.irs_base, + GICV5_IRS_SAVE_VM_STATUSR, + GICV5_IRS_SAVE_VM_STATUSR_IDLE, + statusr); +} + +static int vgic_v5_irs_write_vm_mmio_reg(u64 val, u32 offset) +{ + int ret; + + lockdep_assert_held(&vgic_v5_irs_lock); + + /* Make sure that we are idle to begin with */ + ret = vgic_v5_irs_wait_for_vm_op(); + if (ret) + return ret; + + irs_writeq_relaxed(val, offset); + + return vgic_v5_irs_wait_for_vm_op(); +} + +static int vgic_v5_irs_assign_vmt(bool two_level, u8 vm_id_bits, + phys_addr_t vmt_base) +{ + u64 vmt_baser; + u32 vmt_cfgr; + int ret; + + guard(raw_spinlock_irqsave)(&vgic_v5_irs_lock); + + ret = vgic_v5_irs_wait_for_vm_op(); + if (ret) + return ret; + + vmt_baser = irs_readq_relaxed(GICV5_IRS_VMT_BASER); + if (!!FIELD_GET(GICV5_IRS_VMT_BASER_VALID, vmt_baser)) + return -EBUSY; + + vmt_cfgr = FIELD_PREP(GICV5_IRS_VMT_CFGR_VM_ID_BITS, vm_id_bits); + if (two_level) + vmt_cfgr |= FIELD_PREP(GICV5_IRS_VMT_CFGR_STRUCTURE, + GICV5_IRS_VMT_CFGR_STRUCTURE_TWO_LEVEL); + + irs_writel_relaxed(vmt_cfgr, GICV5_IRS_VMT_CFGR); + + /* The base address is intentionally only masked and not shifted */ + vmt_baser = FIELD_PREP(GICV5_IRS_VMT_BASER_VALID, true) | + (vmt_base & GICV5_IRS_VMT_BASER_ADDR); + irs_writeq_relaxed(vmt_baser, GICV5_IRS_VMT_BASER); + + return vgic_v5_irs_wait_for_vm_op(); +} + +static int vgic_v5_irs_clear_vmt(void) +{ + guard(raw_spinlock_irqsave)(&vgic_v5_irs_lock); + + return vgic_v5_irs_write_vm_mmio_reg(0, GICV5_IRS_VMT_BASER); +} + +static int vgic_v5_irs_vmap_l2_vmt(u16 vm_id) +{ + u64 val = FIELD_PREP(GICV5_IRS_VMAP_L2_VMTR_VM_ID, vm_id) | + GICV5_IRS_VMAP_L2_VMTR_M; + + return vgic_v5_irs_write_vm_mmio_reg(val, GICV5_IRS_VMAP_L2_VMTR); +} + +static int __vgic_v5_irs_vmap_vm(u16 vm_id, bool unmap) +{ + u64 val = FIELD_PREP(GICV5_IRS_VMAP_VMR_VM_ID, vm_id) | + FIELD_PREP(GICV5_IRS_VMAP_VMR_U, unmap) | + GICV5_IRS_VMAP_VMR_M; + + return vgic_v5_irs_write_vm_mmio_reg(val, GICV5_IRS_VMAP_VMR); +} + +static int vgic_v5_irs_set_vm_valid(u16 vm_id) +{ + return __vgic_v5_irs_vmap_vm(vm_id, false); +} + +static int vgic_v5_irs_set_vm_invalid(u16 vm_id) +{ + return __vgic_v5_irs_vmap_vm(vm_id, true); +} + +static int __vgic_v5_irs_update_vist_validity(u16 vm_id, bool spi_ist, bool unmap) +{ + u8 type = spi_ist ? 0b011 : 0b010; + u64 val = FIELD_PREP(GICV5_IRS_VMAP_VISTR_TYPE, type) | + FIELD_PREP(GICV5_IRS_VMAP_VISTR_VM_ID, vm_id) | + FIELD_PREP(GICV5_IRS_VMAP_VISTR_U, unmap) | + GICV5_IRS_VMAP_VISTR_M; + + return vgic_v5_irs_write_vm_mmio_reg(val, GICV5_IRS_VMAP_VISTR); +} + +static int vgic_v5_irs_set_vist_valid(u16 vm_id, bool spi_ist) +{ + return __vgic_v5_irs_update_vist_validity(vm_id, spi_ist, false); +} + +/* + * LPI ISTs can be invalidated explicitly. SPI ISTs are invalidated by making + * the VMTE invalid during teardown. + */ +static int vgic_v5_irs_set_vist_invalid(u16 vm_id, bool spi_ist) +{ + return __vgic_v5_irs_update_vist_validity(vm_id, spi_ist, true); +} + +static int vgic_v5_irs_set_up_vpe(u16 vm_id, u16 vpe_id, + irq_hw_number_t db_hwirq) +{ + u64 vmap_vper, dbr, selr; + u32 statusr, cr0; + int ret; + + lockdep_assert_held(&vgic_v5_irs_lock); + + /* Make sure that we are idle to begin with */ + ret = vgic_v5_irs_wait_for_vm_op(); + if (ret) + return ret; + + /* Mark the VPE as valid */ + vmap_vper = FIELD_PREP(GICV5_IRS_VMAP_VPER_VPE_ID, vpe_id) | + FIELD_PREP(GICV5_IRS_VMAP_VPER_VM_ID, vm_id) | + GICV5_IRS_VMAP_VPER_M; + irs_writeq_relaxed(vmap_vper, GICV5_IRS_VMAP_VPER); + + /* Wait for the VPE to be marked valid in the VPET */ + ret = vgic_v5_irs_wait_for_vm_op(); + if (ret) + return ret; + + selr = FIELD_PREP(GICV5_IRS_VPE_SELR_VPE_ID, vpe_id) | + FIELD_PREP(GICV5_IRS_VPE_SELR_VM_ID, vm_id) | + GICV5_IRS_VPE_SELR_S; + irs_writeq_relaxed(selr, GICV5_IRS_VPE_SELR); + + ret = vgic_v5_irs_wait_for_vpe_op(); + if (ret) + return ret; + + statusr = irs_readl_relaxed(GICV5_IRS_VPE_STATUSR); + if (!FIELD_GET(GICV5_IRS_VPE_STATUSR_V, statusr)) + return -EINVAL; + + /* Set targeted only routing (disable 1ofN vPE selection) */ + cr0 = GICV5_IRS_VPE_CR0_DPS; + irs_writel_relaxed(cr0, GICV5_IRS_VPE_CR0); + + ret = vgic_v5_irs_wait_for_vpe_op(); + if (ret) + return ret; + + /* + * The VPE has not yet run. Therefore, make sure that all interrupts + * will generate a doorbell. + */ + dbr = FIELD_PREP(GICV5_IRS_VPE_DBR_INTID, db_hwirq) | + GICV5_IRS_VPE_DBR_DBV; + irs_writeq_relaxed(dbr, GICV5_IRS_VPE_DBR); + + ret = vgic_v5_irs_wait_for_vpe_op(); + if (ret) + return ret; + + return 0; +} + +static int vgic_v5_irs_save_vm_op(u16 vm_id, bool save, u32 *statusr) +{ + u64 save_vmr; + int ret; + + save_vmr = FIELD_PREP(GICV5_IRS_SAVE_VMR_VM_ID, vm_id); + save_vmr |= GICV5_IRS_SAVE_VMR_Q; + save_vmr |= FIELD_PREP(GICV5_IRS_SAVE_VMR_S, save); + + guard(raw_spinlock_irqsave)(&vgic_v5_irs_lock); + + /* Make sure that we are idle to begin with. */ + ret = vgic_v5_irs_wait_for_save_vm_op(NULL); + if (ret) + return ret; + + irs_writeq_relaxed(save_vmr, GICV5_IRS_SAVE_VMR); + + return vgic_v5_irs_wait_for_save_vm_op(statusr); +} + +static irqreturn_t db_handler(int irq, void *data) +{ + struct kvm_vcpu *vcpu = data; + + WRITE_ONCE(vcpu->arch.vgic_cpu.vgic_v5.gicv5_vpe.db_fired, true); + + kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu); + kvm_vcpu_kick(vcpu); + + return IRQ_HANDLED; +} + +static int vgic_v5_send_command(struct kvm_vcpu *vcpu, enum gicv5_vcpu_cmd cmd) +{ + int irq = vgic_v5_vpe_db(vcpu); + + if (!irq) + return -ENXIO; + + return irq_set_vcpu_affinity(irq, &cmd); +} + +static int vgic_v5_db_set_vcpu_affinity(struct irq_data *data, void *vcpu_info) +{ + struct vgic_v5_vm *vm = data->domain->host_data; + enum gicv5_vcpu_cmd *cmd = vcpu_info; + unsigned int vcpu_idx = data->hwirq; + struct kvm_vcpu *vcpu; + u16 vpe_id; + + guard(raw_spinlock_irqsave)(&vgic_v5_irs_lock); + + switch (*cmd) { + case VMT_L2_MAP: + return vgic_v5_irs_vmap_l2_vmt(vm->vm_id); + case VMTE_MAKE_VALID: + return vgic_v5_irs_set_vm_valid(vm->vm_id); + case VMTE_MAKE_INVALID: + return vgic_v5_irs_set_vm_invalid(vm->vm_id); + case VPE_MAKE_VALID: + /* + * The index in the doorbell domain aligns with our flat + * vcpu_idx index. However, we need the actual VPE ID, which + * means we first need to resolve the actual vcpu. + */ + vcpu = kvm_get_vcpu(vm->kvm, vcpu_idx); + if (!vcpu) + return -EINVAL; + + vpe_id = vgic_v5_vpe_id(vcpu); + + /* + * We need the actual LPI ID which lives in the top-most parent + * domain. This hwirq won't include the type (LPI) but that's + * not required for the IRS_VPE_DBR. + */ + while (data->parent_data) + data = data->parent_data; + return vgic_v5_irs_set_up_vpe(vm->vm_id, vpe_id, data->hwirq); + case SPI_VIST_MAKE_VALID: + return vgic_v5_irs_set_vist_valid(vm->vm_id, true); + case LPI_VIST_MAKE_VALID: + return vgic_v5_irs_set_vist_valid(vm->vm_id, false); + case LPI_VIST_MAKE_INVALID: + return vgic_v5_irs_set_vist_invalid(vm->vm_id, false); + default: + return -EINVAL; + } +} + +/* + * This set of irq_chip functions is specific for doorbells. + */ +static const struct irq_chip vgic_v5_db_irq_chip = { + .name = "GICv5-DB", + .irq_mask = irq_chip_mask_parent, + .irq_unmask = irq_chip_unmask_parent, + .irq_eoi = irq_chip_eoi_parent, + .irq_set_affinity = irq_chip_set_affinity_parent, + .irq_get_irqchip_state = irq_chip_get_parent_state, + .irq_set_irqchip_state = irq_chip_set_parent_state, + .irq_set_vcpu_affinity = vgic_v5_db_set_vcpu_affinity, + .flags = IRQCHIP_SET_TYPE_MASKED | IRQCHIP_SKIP_SET_WAKE | + IRQCHIP_MASK_ON_SUSPEND, +}; + +static void vgic_v5_irq_db_domain_free(struct irq_domain *domain, + unsigned int virq, unsigned int nr_irqs) +{ + int i; + + for (i = 0; i < nr_irqs; i++) { + struct irq_data *d = irq_domain_get_irq_data(domain, virq + i); + + irq_set_handler(virq + i, NULL); + irq_domain_reset_irq_data(d); + } + + irq_domain_free_irqs_parent(domain, virq, nr_irqs); +} + +static int vgic_v5_irq_db_domain_alloc(struct irq_domain *domain, + unsigned int virq, unsigned int nr_irqs, + void *arg) +{ + const struct irq_chip *chip = &vgic_v5_db_irq_chip; + struct irq_data *irqd; + int ret; + + ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, NULL); + if (ret) + return ret; + + for (int i = 0; i < nr_irqs; i++) { + irq_domain_set_hwirq_and_chip(domain, virq + i, i, chip, + domain->host_data); + irqd = irq_desc_get_irq_data(irq_to_desc(virq + i)); + irqd_set_single_target(irqd); + } + + return 0; +} + +static const struct irq_domain_ops vgic_v5_irq_db_domain_ops = { + .alloc = vgic_v5_irq_db_domain_alloc, + .free = vgic_v5_irq_db_domain_free, +}; + +static int vgic_v5_create_per_vm_domain(struct kvm *kvm) +{ + struct vgic_v5_vm *vm = &kvm->arch.vgic.gicv5_vm; + int nr_vcpus = atomic_read(&kvm->online_vcpus); + int id = task_pid_nr(current); + int ret, db_virq = 0; + + vm->fwnode = irq_domain_alloc_named_id_fwnode("GICv5-vpe-db", id); + if (!vm->fwnode) + return -ENOMEM; + + vm->domain = irq_domain_create_hierarchy(gicv5_global_data.lpi_domain, + 0, nr_vcpus, vm->fwnode, + &vgic_v5_irq_db_domain_ops, vm); + if (!vm->domain) { + ret = -ENOMEM; + goto err; + } + + db_virq = irq_domain_alloc_irqs(vm->domain, nr_vcpus, NUMA_NO_NODE, vm); + if (db_virq <= 0) { + ret = db_virq; + goto err; + } + + kvm->arch.vgic.gicv5_vm.vpe_db_base = db_virq; + + return 0; + +err: + if (vm->domain) + irq_domain_remove(vm->domain); + if (vm->fwnode) + irq_domain_free_fwnode(vm->fwnode); + + kvm->arch.vgic.gicv5_vm.vpe_db_base = 0; + vm->domain = NULL; + vm->fwnode = NULL; + + return ret; +} + +static void vgic_v5_teardown_per_vm_domain(struct vgic_v5_vm *vm) +{ + if (!vm->domain) + return; + + irq_domain_remove(vm->domain); + irq_domain_free_fwnode(vm->fwnode); + vm->domain = NULL; + vm->fwnode = NULL; +} + void vgic_v5_reset(struct kvm_vcpu *vcpu) { /* @@ -120,17 +613,117 @@ void vgic_v5_reset(struct kvm_vcpu *vcpu) * CPUIF (but potentially fewer in the IRS). */ vcpu->arch.vgic_cpu.num_pri_bits = 5; + + /* Make the VPE valid in the VPET */ + if (WARN_ON(vgic_v5_send_command(vcpu, VPE_MAKE_VALID))) + return; } -int vgic_v5_init(struct kvm *kvm) +static void vgic_v5_free_doorbells(struct kvm *kvm, unsigned int nr_dbs) { + struct vgic_v5_vm *vm = &kvm->arch.vgic.gicv5_vm; struct kvm_vcpu *vcpu; - unsigned long idx; + unsigned long i; + int db; - if (vgic_initialized(kvm)) - return 0; + for (i = 0; i < nr_dbs; i++) { + vcpu = kvm_get_vcpu(kvm, i); + db = vgic_v5_vpe_db(vcpu); + if (!db) + continue; + + free_irq(db, vcpu); + vcpu->arch.vgic_cpu.vgic_v5.gicv5_vpe.db = 0; + } + + if (vm->vpe_db_base) { + irq_domain_free_irqs(vm->vpe_db_base, + atomic_read(&kvm->online_vcpus)); + vm->vpe_db_base = 0; + } +} + +void vgic_v5_teardown(struct kvm *kvm) +{ + struct vgic_dist *dist = &kvm->arch.vgic; + struct kvm_vcpu *vcpu, *vcpu0; + bool release_vm_id = true; + unsigned long i; + int rc; + + lockdep_assert_held(&kvm->arch.config_lock); - kvm_for_each_vcpu(idx, vcpu, kvm) { + /* + * If the VM's ID isn't valid, then we either failed init very early or + * we've been called a second time. Nothing to do here in either case. + */ + if (kvm->arch.vgic.gicv5_vm.vm_id == VGIC_V5_VM_ID_INVAL) + return; + + if (kvm->arch.vgic.gicv5_vm.vmte_allocated) { + /* Make the VM invalid */ + vcpu0 = kvm_get_vcpu(kvm, 0); + rc = vgic_v5_send_command(vcpu0, VMTE_MAKE_INVALID); + if (rc) { + kvm_err("could not make VMTE invalid\n"); + release_vm_id = false; + goto out_free_doorbells; + } + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vgic_v5_vmte_free_vpe(vcpu)) { + kvm_err("Failed to free VPE\n"); + release_vm_id = false; + goto out_free_doorbells; + } + } + + if (vgic_v5_vmte_release(kvm)) { + kvm_err("Failed to release VM 0x%x\n", dist->gicv5_vm.vm_id); + release_vm_id = false; + } + } + +out_free_doorbells: + vgic_v5_free_doorbells(kvm, atomic_read(&kvm->online_vcpus)); + vgic_v5_teardown_per_vm_domain(&kvm->arch.vgic.gicv5_vm); + + /* + * We only release the VM ID itself if we didn't fail earlier. It does + * mean that we might lose the VM ID (and associated VMTE, etc), but + * given that we've failed to tear them down correctly there's no way to + * safely reuse them. The VM ID allocating IDA will make sure we don't + * accidentally reuse this partially torn down state. + */ + if (release_vm_id) + vgic_v5_release_vm_id(kvm); +} + +/* + * Claim and populate a VMTE (optionally making a new L2 VMT valid), create VPE + * doorbells, allocate VPET and populate for each VPE. Finally, we also init the + * vIRS, which means allocating and making the virtual SPI IST valid. + * + * Note: We do need to put the cart before the horse here. The VPE doorbells are + * our conduit for communication with the IRS, which means we need to have those + * before making the VMTE valid. + * + * On failure, we clean up in the teardown path (vgic_v5_teardown()). + */ +int vgic_v5_init(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu, *vcpu0; + int nr_vcpus, ret = 0; + unsigned int db_virq; + unsigned long i; + + lockdep_assert_held(&kvm->arch.config_lock); + + nr_vcpus = atomic_read(&kvm->online_vcpus); + if (nr_vcpus == 0) + return -ENODEV; + + kvm_for_each_vcpu(i, vcpu, kvm) { if (vcpu_has_nv(vcpu)) { kvm_err("Nested GICv5 VMs are currently unsupported\n"); return -EINVAL; @@ -145,20 +738,84 @@ int vgic_v5_init(struct kvm *kvm) kvm->arch.vgic.gicv5_vm.userspace_ppis, ppi_caps.impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); + ret = vgic_v5_allocate_vm_id(kvm); + if (ret) + return ret; + + /* + * Stash a backpointer to struct kvm. It is required to resolve the VPE + * ID from the doorbell index, which matches the flat vcpu_idx and not + * the vcpu_id. + */ + kvm->arch.vgic.gicv5_vm.kvm = kvm; + + ret = vgic_v5_create_per_vm_domain(kvm); + if (ret) + goto err; + + db_virq = kvm->arch.vgic.gicv5_vm.vpe_db_base; + kvm_for_each_vcpu(i, vcpu, kvm) { + ret = request_irq(db_virq + i, db_handler, 0, "vcpu", vcpu); + if (ret) + goto err; + + /* Stash it with the VCPU for easy retrieval */ + vcpu->arch.vgic_cpu.vgic_v5.gicv5_vpe.db = db_virq + i; + } + + /* Populate VMTE (with VPET and VM descriptor) */ + ret = vgic_v5_vmte_init(kvm); + if (ret) + goto err; + + /* We pick the first vcpu to make the VMTE valid - any would do */ + vcpu0 = kvm_get_vcpu(kvm, 0); + ret = vgic_v5_send_command(vcpu0, VMTE_MAKE_VALID); + if (ret) + goto err; + + /* Populate the VPETE for each VPE. */ + kvm_for_each_vcpu(i, vcpu, kvm) { + ret = vgic_v5_vmte_alloc_vpe(vcpu); + if (ret) + goto err; + } + + /* Init IRS (and alloc SPI IST) */ + ret = kvm_vgic_v5_irs_init(kvm, kvm->arch.vgic.nr_spis); + if (ret) + goto err; + return 0; + +err: + /* + * Explicitly tear everything down on failure. The teardown function is + * written to handle any partial state we might have, so we don't need + * to do any clean-up first. Teardown will be called a second time on VM + * destruction, but that's fine - it is better to leave things in a + * clean state now, and doubly so because userspace could actually go + * and retry init. + */ + vgic_v5_teardown(kvm); + + return ret; } int vgic_v5_map_resources(struct kvm *kvm) { if (!vgic_initialized(kvm)) return -EBUSY; + if (kvm->arch.vgic.vgic_v5_irs_data->lpi_ist_restore_pending) + return -EINVAL; return 0; } -int vgic_v5_finalize_ppi_state(struct kvm *kvm) +int vgic_v5_finalize_ppi_state(struct kvm_vcpu *vcpu) { - struct kvm_vcpu *vcpu0; + struct kvm *kvm = vcpu->kvm; + struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; int i; if (!vgic_is_v5(kvm)) @@ -167,35 +824,65 @@ int vgic_v5_finalize_ppi_state(struct kvm *kvm) guard(mutex)(&kvm->arch.config_lock); /* - * If SW_PPI has been advertised, then we know we already - * initialised the whole thing, and we can return early. Yes, - * this is pretty hackish as far as state tracking goes... + * Discover the set of PPIs that are exposed to the guest once per VM. + * Once known, apply that mask to each VCPU's restored PPI state as the + * VCPUs are first run. */ - if (test_bit(GICV5_ARCH_PPI_SW_PPI, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask)) - return 0; - - /* The PPI state for all VCPUs should be the same. Pick the first. */ - vcpu0 = kvm_get_vcpu(kvm, 0); + if (!test_bit(GICV5_ARCH_PPI_SW_PPI, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask)) { + bitmap_zero(kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, + VGIC_V5_NR_PRIVATE_IRQS); + bitmap_zero(kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr, + VGIC_V5_NR_PRIVATE_IRQS); + + for_each_set_bit(i, ppi_caps.impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) { + const u32 intid = vgic_v5_make_ppi(i); + struct vgic_irq *irq; + + irq = vgic_get_vcpu_irq(vcpu, intid); + + /* Expose PPIs with an owner or the SW_PPI, only */ + scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { + if (irq->owner || i == GICV5_ARCH_PPI_SW_PPI) { + __set_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask); + __assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr, + irq->config == VGIC_CONFIG_LEVEL); + } + } - bitmap_zero(kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); - bitmap_zero(kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr, VGIC_V5_NR_PRIVATE_IRQS); + vgic_put_irq(kvm, irq); + } + } - for_each_set_bit(i, ppi_caps.impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) { + /* + * Apply the mask to Enable, Active. Skip pending as that's calculated + * on guest entry. + */ + bitmap_and(cpu_if->vgic_ppi_enabler, cpu_if->vgic_ppi_enabler, + kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); + bitmap_and(cpu_if->vgic_ppi_activer, cpu_if->vgic_ppi_activer, + kvm->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); + + /* Also update the vgic_irqs */ + for (i = 0; i < VGIC_V5_NR_PRIVATE_IRQS; i++) { + bool visible = test_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask); const u32 intid = vgic_v5_make_ppi(i); struct vgic_irq *irq; - irq = vgic_get_vcpu_irq(vcpu0, intid); + irq = vgic_get_vcpu_irq(vcpu, intid); - /* Expose PPIs with an owner or the SW_PPI, only */ scoped_guard(raw_spinlock_irqsave, &irq->irq_lock) { - if (irq->owner || i == GICV5_ARCH_PPI_SW_PPI) { - __set_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_mask); - __assign_bit(i, kvm->arch.vgic.gicv5_vm.vgic_ppi_hmr, - irq->config == VGIC_CONFIG_LEVEL); + if (!visible) { + irq->enabled = false; + irq->active = false; + irq->pending_latch = false; + irq->line_level = false; + } else { + irq->enabled = test_bit(i, cpu_if->vgic_ppi_enabler); + irq->active = test_bit(i, cpu_if->vgic_ppi_activer); } } - vgic_put_irq(vcpu0->kvm, irq); + vgic_put_irq(kvm, irq); } return 0; @@ -436,9 +1123,53 @@ void vgic_v5_flush_ppi_state(struct kvm_vcpu *vcpu) VGIC_V5_NR_PRIVATE_IRQS); } +static bool vgic_v5_set_spi_pending_state(struct kvm_vcpu *vcpu, + struct vgic_irq *irq) +{ + vgic_v5_set_irq_pend(irq->target_vcpu, irq); + return true; +} + +bool vgic_v5_spi_queue_irq_unlock(struct kvm *kvm, + struct vgic_irq *irq, + unsigned long flags) + __releases(&irq->irq_lock) +{ + lockdep_assert_held(&irq->irq_lock); + + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + return true; +} + +static const struct irq_ops vgic_v5_spi_irq_ops = { + .set_pending_state = vgic_v5_set_spi_pending_state, + .queue_irq_unlock = vgic_v5_spi_queue_irq_unlock, +}; + +void vgic_v5_set_spi_ops(struct vgic_irq *irq) +{ + if (WARN_ON(!irq) || WARN_ON(irq->ops)) + return; + + irq->ops = &vgic_v5_spi_irq_ops; +} + +/* Set the pending state for GICv5 SPIs and LPIs */ +void vgic_v5_set_irq_pend(struct kvm_vcpu *vcpu, struct vgic_irq *irq) +{ + if (WARN_ON(__irq_is_ppi(KVM_DEV_TYPE_ARM_VGIC_V5, irq->intid))) + return; + + kvm_call_hyp(__vgic_v5_vdpend, irq->intid, irq_is_pending(irq), + vcpu->kvm->arch.vgic.gicv5_vm.vm_id); +} + void vgic_v5_load(struct kvm_vcpu *vcpu) { + bool irichppidis = !READ_ONCE(vcpu->kvm->arch.vgic.enabled); struct vgic_v5_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v5; + u16 vm = vgic_v5_vm_id(vcpu->kvm); + u16 vpe = vgic_v5_vpe_id(vcpu); /* * On the WFI path, vgic_load is called a second time. The first is when @@ -451,7 +1182,16 @@ void vgic_v5_load(struct kvm_vcpu *vcpu) kvm_call_hyp(__vgic_v5_restore_vmcr_apr, cpu_if); - cpu_if->gicv5_vpe.resident = true; + cpu_if->vgic_contextr = FIELD_PREP(ICH_CONTEXTR_EL2_V, true) | + FIELD_PREP(ICH_CONTEXTR_EL2_IRICHPPIDIS, irichppidis) | + FIELD_PREP(ICH_CONTEXTR_EL2_VPE, vpe) | + FIELD_PREP(ICH_CONTEXTR_EL2_VM, vm); + + kvm_call_hyp(__vgic_v5_make_resident, cpu_if); + + /* Failed to make the VPE resident? Bang! */ + if (WARN_ON(!!FIELD_GET(ICH_CONTEXTR_EL2_F, cpu_if->vgic_contextr))) + kvm_vm_dead(vcpu->kvm); } void vgic_v5_put(struct kvm_vcpu *vcpu) @@ -469,7 +1209,58 @@ void vgic_v5_put(struct kvm_vcpu *vcpu) kvm_call_hyp(__vgic_v5_save_apr, cpu_if); - cpu_if->gicv5_vpe.resident = false; + cpu_if->vgic_contextr = 0; + + /* Request a doorbell if entering WFI, unless the IRS is disabled */ + if (vcpu_get_flag(vcpu, IN_WFI) && + READ_ONCE(vcpu->kvm->arch.vgic.enabled)) { + u32 priority_mask; + int dbpm; + + /* + * Find the virtual running priority and use this to calculate + * the doorbell priority mask. We combine the highest active + * priority and the CPU's priority mask. The guest can't handle + * interrupts with priorities less than or equal to the virtual + * running priority, so there's literally no point in waking the + * guest for these. + * + * The priority needs to be higher than the mask to signal, so + * pick the next higher priority (subtract 1). + */ + priority_mask = vgic_v5_get_effective_priority_mask(vcpu); + + /* + * Request a doorbell *unless* the priority is 0, indicating + * that no interrupt can wake the CPU up. + */ + if (priority_mask) { + int db_irq = vgic_v5_vpe_db(vcpu); + struct irq_data *d = irq_get_irq_data(db_irq); + const struct cpumask *aff = irq_data_get_effective_affinity_mask(d); + int cpu = smp_processor_id(); + + dbpm = priority_mask - 1; + cpu_if->vgic_contextr = FIELD_PREP(ICH_CONTEXTR_EL2_DB, 1) | + FIELD_PREP(ICH_CONTEXTR_EL2_DBPM, dbpm); + + /* + * Make the doorbell affine to this CPU, if it isn't + * already. Actively check the cpumask first as it is + * cheaper than changing the affinity every time. + */ + if (!cpumask_test_cpu(cpu, aff)) { + int ret; + + ret = irq_set_affinity(db_irq, cpumask_of(cpu)); + if (ret) + kvm_debug_ratelimited("Failed to move doorbell IRQ %d to CPU %d: %d\n", + db_irq, cpu, ret); + } + } + } + + kvm_call_hyp(__vgic_v5_make_non_resident, cpu_if); /* The shadow priority is only updated on entering WFI */ if (vcpu_get_flag(vcpu, IN_WFI)) @@ -513,3 +1304,300 @@ void vgic_v5_save_state(struct kvm_vcpu *vcpu) __vgic_v5_save_ppi_state(cpu_if); dsb(sy); } + +static int vgic_v5_irs_status_is_quiesced(u32 statusr) +{ + if (statusr & GICV5_IRS_SAVE_VM_STATUSR_Q) + return 0; + + return -EBUSY; +} + +static int vgic_v5_irs_is_quiesced(u16 vm_id) +{ + u32 statusr; + int ret; + + /* + * Ensure that the IST snapshot is complete before asking the IRS + * whether the VM remained quiescent while it was copied. + */ + mb(); + + ret = vgic_v5_irs_save_vm_op(vm_id, false, &statusr); + if (ret) + return ret; + + return vgic_v5_irs_status_is_quiesced(statusr); +} + +static int vgic_v5_copy_ist_attr(struct kvm_device_attr *attr, + struct kvm_vgic_v5_ist *ist_attr) +{ + void __user *uaddr = (void __user *)(unsigned long)attr->addr; + + if (!uaddr) + return -EINVAL; + + if (copy_from_user(ist_attr, uaddr, sizeof(*ist_attr))) + return -EFAULT; + + return 0; +} + +static int vgic_v5_validate_ist_user_buffer(__u64 addr, __u64 size, + size_t expected) +{ + if (!addr || size != expected) + return -EINVAL; + + return 0; +} + +static int vgic_v5_validate_ist_attr(struct kvm *kvm, + const struct kvm_vgic_v5_ist *ist_attr) +{ + unsigned int id_bits; + int ret; + + /* We always have SPIs to save */ + ret = vgic_v5_validate_ist_user_buffer(ist_attr->spi_ist_addr, + ist_attr->spi_ist_size, + kvm->arch.vgic.nr_spis * sizeof(__u32)); + if (ret) + return ret; + + /* We don't always have LPIs to save */ + ret = vgic_v5_irs_lpi_ist_id_bits(kvm, &id_bits); + if (ret < 0) + return ret; + + /* No LPI IST */ + if (!ret) { + if (ist_attr->lpi_ist_addr || ist_attr->lpi_ist_size) + return -EINVAL; + + return 0; + } + + return vgic_v5_validate_ist_user_buffer(ist_attr->lpi_ist_addr, + ist_attr->lpi_ist_size, + BIT(id_bits) * sizeof(__u32)); +} + +int vgic_v5_irs_save_ists(struct kvm *kvm, struct kvm_device_attr *attr) +{ + struct kvm_vgic_v5_ist ist_attr; + u16 vm_id = vgic_v5_vm_id(kvm); + u32 statusr; + int ret = 0; + + mutex_lock(&kvm->lock); + + if (kvm_trylock_all_vcpus(kvm)) { + mutex_unlock(&kvm->lock); + return -EBUSY; + } + + mutex_lock(&kvm->arch.config_lock); + + if (!vgic_initialized(kvm)) { + ret = -EBUSY; + goto out_unlock; + } + + ret = vgic_v5_copy_ist_attr(attr, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_validate_ist_attr(kvm, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_irs_save_vm_op(vm_id, true, &statusr); + if (ret) { + kvm_err("Failed to save GICv5 IRS VM state: %d\n", ret); + goto out_unlock; + } + + ret = vgic_v5_irs_status_is_quiesced(statusr); + if (ret) + goto out_unlock; + + /* Save the SPI IST to the userspace buffer. */ + ret = vgic_v5_save_spi_ist(kvm, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_irs_is_quiesced(vm_id); + if (ret) + goto out_unlock; + + /* Save the LPI IST to the userspace buffer. */ + ret = vgic_v5_save_lpi_ist(kvm, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_irs_is_quiesced(vm_id); + if (ret) + goto out_unlock; + +out_unlock: + mutex_unlock(&kvm->arch.config_lock); + kvm_unlock_all_vcpus(kvm); + mutex_unlock(&kvm->lock); + + return ret; +} + +/* Allocate the LPI IST to restore into */ +static int vgic_v5_restore_lpi_ist_alloc(struct kvm *kvm, bool *allocated) +{ + unsigned int id_bits; + int ret; + + *allocated = false; + + ret = vgic_v5_irs_lpi_ist_id_bits(kvm, &id_bits); + if (ret <= 0) + return ret; + + ret = vgic_v5_lpi_ist_alloc(kvm, id_bits); + if (ret) + return ret; + + *allocated = true; + + return 0; +} + +/* + * Clean up the LPI IST if we allocated it, and restore the VMTE to the + * original, valid state. + */ +static void vgic_v5_restore_cleanup(struct kvm *kvm, + struct kvm_vcpu *vcpu, + bool lpi_ist_allocated) +{ + /* + * We are on the restore failure path, so we do a best-effort + * cleanup. These commands might fail, but at this stage this is the + * best we can realistically do. + */ + if (lpi_ist_allocated) { + if (!vgic_v5_send_command(vcpu, VMTE_MAKE_INVALID)) + vgic_v5_lpi_ist_free(kvm); + } + + vgic_v5_send_command(vcpu, VMTE_MAKE_VALID); +} + +int vgic_v5_irs_restore_ists(struct kvm *kvm, struct kvm_device_attr *attr) +{ + bool lpi_ist_allocated = false, vmte_invalid = false; + struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0); + struct kvm_vgic_v5_ist ist_attr; + int ret = 0; + + mutex_lock(&kvm->lock); + + if (kvm_trylock_all_vcpus(kvm)) { + mutex_unlock(&kvm->lock); + return -EBUSY; + } + + mutex_lock(&kvm->arch.config_lock); + + if (!vgic_initialized(kvm)) { + ret = -EBUSY; + goto out_unlock; + } + + if (kvm_vm_has_ran_once(kvm)) { + ret = -EBUSY; + goto out_unlock; + } + + ret = vgic_v5_copy_ist_attr(attr, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_validate_ist_attr(kvm, &ist_attr); + if (ret) + goto out_unlock; + + ret = vgic_v5_lpi_ist_exists(kvm); + if (ret) { + if (ret > 0) + ret = -EBUSY; + goto out_unlock; + } + + /* + * If the guest has previously allocated an IST (which we check based on + * the IRS_IST_BASER), extract the number of LPI ID bits from the + * IRS_IST_CFGR. Else, do nothing. + * + * We do this before making the VMTE invalid as we rely on + * IRS_VMAP_VISTR to mark the IST as valid in the VMTE. This can only + * happen while the VMTE is valid. + */ + ret = vgic_v5_restore_lpi_ist_alloc(kvm, &lpi_ist_allocated); + if (ret) + goto out_unlock; + + /* + * Host ISTs are updated while the VMTE is invalid, so the GIC cannot + * observe partially restored state. + */ + ret = vgic_v5_send_command(vcpu0, VMTE_MAKE_INVALID); + if (ret) { + /* + * If invalidation fails, the restore cannot safely update host + * IST state. + */ + goto out_unlock; + } + vmte_invalid = true; + + /* Restore the SPI IST from the userspace buffer. */ + ret = vgic_v5_restore_spi_ist(kvm, &ist_attr); + if (ret) + goto out_unlock; + + /* Restore the LPI IST from the userspace buffer. */ + if (lpi_ist_allocated) { + ret = vgic_v5_restore_lpi_ist(kvm, &ist_attr); + if (ret) + goto out_unlock; + } + + /* And make the VM Valid again */ + ret = vgic_v5_send_command(vcpu0, VMTE_MAKE_VALID); + if (ret) + goto out_unlock; + vmte_invalid = false; + + /* + * As part of restoring the ISTs, and previously pending interrupts have + * been tracked and made non-pending. Now that the ISTs have been + * restored, and the VM is valid again, restore the pending interrupts. + */ + ret = vgic_v5_restore_pending_irqs(kvm); + if (ret) + goto out_unlock; + + kvm->arch.vgic.vgic_v5_irs_data->lpi_ist_restore_pending = false; + +out_unlock: + if (ret && (vmte_invalid || lpi_ist_allocated)) { + vgic_v5_discard_pending_irqs(kvm); + vgic_v5_restore_cleanup(kvm, vcpu0, lpi_ist_allocated); + } + + mutex_unlock(&kvm->arch.config_lock); + kvm_unlock_all_vcpus(kvm); + mutex_unlock(&kvm->lock); + + return ret; +} diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 5cf5a1ef86cd..c2726d4c06e6 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -86,20 +86,32 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid) */ struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid) { - /* Non-private IRQs are not yet implemented for GICv5 */ - if (vgic_is_v5(kvm)) - return NULL; + enum kvm_device_type type = kvm->arch.vgic.vgic_model; /* SPIs */ - if (intid >= VGIC_NR_PRIVATE_IRQS && - intid < (kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS)) { - intid -= VGIC_NR_PRIVATE_IRQS; - intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis); - return &kvm->arch.vgic.spis[intid]; + if (__irq_is_spi(type, intid)) { + switch (type) { + case KVM_DEV_TYPE_ARM_VGIC_V5: + intid = vgic_v5_get_hwirq_id(intid); + + if (intid >= kvm->arch.vgic.nr_spis) + return NULL; + + intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis); + return &kvm->arch.vgic.spis[intid]; + default: { + u32 max_intid = kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS; + + if (intid < max_intid) { + intid -= VGIC_NR_PRIVATE_IRQS; + intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis); + return &kvm->arch.vgic.spis[intid]; + } + }} } /* LPIs */ - if (irq_is_lpi(kvm, intid)) + if (__irq_is_lpi(type, intid)) return vgic_get_lpi(kvm, intid); return NULL; @@ -565,6 +577,9 @@ int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu, else irq->pending_latch = true; + if (irq->ops && irq->ops->set_pending_state) + WARN_ON_ONCE(!irq->ops->set_pending_state(vcpu, irq)); + vgic_queue_irq_unlock(kvm, irq, flags); vgic_put_irq(kvm, irq); @@ -1247,8 +1262,12 @@ int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu) unsigned long flags; struct vgic_vmcr vmcr; - if (vgic_is_v5(vcpu->kvm)) + if (vgic_is_v5(vcpu->kvm)) { + if (READ_ONCE(vcpu->arch.vgic_cpu.vgic_v5.gicv5_vpe.db_fired)) + return true; + return vgic_v5_has_pending_ppi(vcpu); + } if (!vcpu->kvm->arch.vgic.enabled) return false; diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index b71d486ae514..97fc3bdf520e 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -269,6 +269,8 @@ struct ap_list_summary { #define irqs_active_outside_lrs(s) \ ((s)->nr_act && irqs_outside_lrs(s)) +int vgic_v5_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr, + struct vgic_reg_attr *reg_attr); int vgic_v3_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr, struct vgic_reg_attr *reg_attr); int vgic_v2_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr, @@ -383,8 +385,14 @@ void vgic_debug_destroy(struct kvm *kvm); int vgic_v5_probe(const struct gic_kvm_info *info); void vgic_v5_reset(struct kvm_vcpu *vcpu); int vgic_v5_init(struct kvm *kvm); +int kvm_vgic_v5_irs_init(struct kvm *kvm, unsigned int nr_spis); +void vgic_v5_teardown(struct kvm *kvm); int vgic_v5_map_resources(struct kvm *kvm); void vgic_v5_set_ppi_ops(struct kvm_vcpu *vcpu, u32 vintid); +void vgic_v5_set_spi_ops(struct vgic_irq *irq); +bool vgic_v5_spi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, + unsigned long flags); +void vgic_v5_set_irq_pend(struct kvm_vcpu *vcpu, struct vgic_irq *irq); bool vgic_v5_has_pending_ppi(struct kvm_vcpu *vcpu); void vgic_v5_flush_ppi_state(struct kvm_vcpu *vcpu); void vgic_v5_fold_ppi_state(struct kvm_vcpu *vcpu); @@ -394,6 +402,19 @@ void vgic_v5_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr); void vgic_v5_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr); void vgic_v5_restore_state(struct kvm_vcpu *vcpu); void vgic_v5_save_state(struct kvm_vcpu *vcpu); +int vgic_v5_register_irs_iodev(struct kvm *kvm, gpa_t irs_base_address); +int vgic_v5_irs_lpi_ist_id_bits(struct kvm *kvm, unsigned int *id_bits); + +int vgic_v5_cpu_sysregs_uaccess(struct kvm_vcpu *vcpu, + struct kvm_device_attr *attr, bool is_write); +int vgic_v5_has_cpu_sysregs_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); +const struct sys_reg_desc *vgic_v5_get_sysreg_table(unsigned int *sz); +int vgic_v5_irs_save_ists(struct kvm *kvm, struct kvm_device_attr *attr); +int vgic_v5_irs_restore_ists(struct kvm *kvm, struct kvm_device_attr *attr); +int vgic_v5_irs_attr_regs_access(struct kvm_device *dev, + struct kvm_device_attr *attr, + u64 *reg, bool is_write); +int vgic_v5_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr); #define for_each_visible_v5_ppi(__i, __k) \ for_each_set_bit(__i, (__k)->arch.vgic.gicv5_vm.vgic_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index e2d37ee221b8..893b5f9df575 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -401,6 +401,57 @@ Res0 3:1 Field 0 SO EndSysreg +SysregFields SPSR_ELx +Res0 63:37 +Field 36 UINJ +Field 35 PACM +Field 34 EXLOCK +Res0 33 +Field 32 PM +Field 31 N +Field 30 Z +Field 29 C +Field 28 V +Res0 27:26 +Field 25 TCO +Field 24 DIT +Field 23 UAO +Field 22 PAN +Field 21 SS +Field 20 IL +Res0 19:14 +Field 13 ALLINT +Field 12 SSBS +Field 11:10 BTYPE +Field 9 D +Field 8 A +Field 7 I +Field 6 F +Res0 5 +Field 4 M4 +Enum 3:0 M + 0b0000 EL0 + 0b0100 EL1t + 0b0101 EL1h + 0b1000 EL2t + 0b1001 EL2h + 0b1100 EL3t + 0b1101 EL3h +EndEnum +EndSysregFields + +Sysreg SPSR_EL1 3 0 4 0 0 +Fields SPSR_ELx +EndSysreg + +Sysreg SPSR_EL2 3 4 4 0 0 +Fields SPSR_ELx +EndSysreg + +Sysreg SPSR_EL12 3 5 4 0 0 +Mapping SPSR_EL1 +EndSysreg + Sysreg ID_PFR0_EL1 3 0 0 1 0 Res0 63:32 UnsignedEnum 31:28 RAS diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index dbac3573b2c8..db090f4c5e95 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -269,7 +269,7 @@ static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu) * stolen time for a vcore when it is inactive, or for a vcpu * when it is in state RUNNING or NOTREADY. NOTREADY is a bit of * a misnomer; it means that the vcpu task is not executing in - * the KVM_VCPU_RUN ioctl, i.e. it is in userspace or elsewhere in + * the KVM_RUN ioctl, i.e. it is in userspace or elsewhere in * the kernel. We don't have any way of dividing up that time * between time that the vcpu is genuinely stopped, time that * the task is actively working on behalf of the vcpu, and time diff --git a/arch/s390/kvm/s390/s390.c b/arch/s390/kvm/s390/s390.c index efd77b9cd36c..b14706c70946 100644 --- a/arch/s390/kvm/s390/s390.c +++ b/arch/s390/kvm/s390/s390.c @@ -5783,6 +5783,14 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old, { } +int kvm_arch_pre_fault_allowed(struct kvm_vcpu *vcpu) +{ + if (kvm_is_ucontrol(vcpu->kvm)) + return -EINVAL; + + return 0; +} + /** * kvm_arch_vcpu_pre_fault_memory() -- pre-fault and link gmap dat tables * @vcpu: the vcpu that shall appear to have generated the fault-in. @@ -5809,9 +5817,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_ gpa_t end; int rc; - if (kvm_is_ucontrol(vcpu->kvm)) - return -EINVAL; - rc = kvm_s390_faultin_gfn(vcpu, NULL, &f); if (rc == PGM_ADDRESSING) return -ENOENT; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 064ecc33b926..c35fd2868c20 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5086,6 +5086,14 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa, } } +int kvm_arch_pre_fault_allowed(struct kvm_vcpu *vcpu) +{ + if (!vcpu->kvm->arch.pre_fault_allowed) + return -EOPNOTSUPP; + + return 0; +} + long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range) { @@ -5095,9 +5103,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, u64 end; int r; - if (!vcpu->kvm->arch.pre_fault_allowed) - return -EOPNOTSUPP; - if (kvm_is_gfn_alias(vcpu->kvm, gpa_to_gfn(range->gpa))) return -EINVAL; diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b272c20586a7..a4d4479e2b28 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1708,7 +1708,7 @@ static int tdx_sept_map_leaf_spte(struct kvm *kvm, gfn_t gfn, enum pg_level leve WARN_ON_ONCE((new_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK); /* - * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() + * Ensure pre_fault_allowed is read by kvm_arch_pre_fault_allowed() * before kvm_tdx->state. Userspace must not be allowed to pre-fault * arbitrary memory until the initial memory image is finalized. Pairs * with the smp_wmb() in tdx_td_finalize(). diff --git a/drivers/irqchip/irq-gic-v5-irs.c b/drivers/irqchip/irq-gic-v5-irs.c index 4486645a4b6f..abf92b1c619d 100644 --- a/drivers/irqchip/irq-gic-v5-irs.c +++ b/drivers/irqchip/irq-gic-v5-irs.c @@ -21,8 +21,6 @@ */ #define LPI_ID_BITS_LINEAR 12 -#define IRS_FLAGS_NON_COHERENT BIT(0) - static DEFINE_PER_CPU_READ_MOSTLY(struct gicv5_irs_chip_data *, per_cpu_irs_data); static LIST_HEAD(irs_nodes); @@ -50,6 +48,11 @@ static void irs_writeq_relaxed(struct gicv5_irs_chip_data *irs_data, writeq_relaxed(val, irs_data->irs_base + reg_offset); } +struct gicv5_irs_chip_data *gicv5_irs_get_chip_data(void) +{ + return per_cpu(per_cpu_irs_data, 0); +} + /* * The polling wait (in gicv5_wait_for_op_s_atomic()) on a GIC register * provides the memory barriers (through MMIO accessors) @@ -266,24 +269,24 @@ int gicv5_irs_iste_alloc(const u32 lpi) * itself is not supported) again serves to make it easier to find physically * contiguous blocks of memory. */ -static unsigned int gicv5_irs_l2_sz(u32 idr2) +unsigned int gicv5_irs_l2_sz(u32 l2sz) { switch (PAGE_SIZE) { case SZ_64K: - if (GICV5_IRS_IST_L2SZ_SUPPORT_64KB(idr2)) + if (GICV5_IRS_IST_L2SZ_SUPPORT_64KB(l2sz)) return GICV5_IRS_IST_CFGR_L2SZ_64K; fallthrough; case SZ_4K: - if (GICV5_IRS_IST_L2SZ_SUPPORT_4KB(idr2)) + if (GICV5_IRS_IST_L2SZ_SUPPORT_4KB(l2sz)) return GICV5_IRS_IST_CFGR_L2SZ_4K; fallthrough; case SZ_16K: - if (GICV5_IRS_IST_L2SZ_SUPPORT_16KB(idr2)) + if (GICV5_IRS_IST_L2SZ_SUPPORT_16KB(l2sz)) return GICV5_IRS_IST_CFGR_L2SZ_16K; break; } - if (GICV5_IRS_IST_L2SZ_SUPPORT_4KB(idr2)) + if (GICV5_IRS_IST_L2SZ_SUPPORT_4KB(l2sz)) return GICV5_IRS_IST_CFGR_L2SZ_4K; return GICV5_IRS_IST_CFGR_L2SZ_64K; @@ -331,7 +334,7 @@ static int __init gicv5_irs_init_ist(struct gicv5_irs_chip_data *irs_data) lpi_id_bits = min(lpi_id_bits, gicv5_global_data.cpuif_id_bits); if (two_levels) - l2sz = gicv5_irs_l2_sz(idr2); + l2sz = gicv5_irs_l2_sz(FIELD_GET(GICV5_IRS_IDR2_IST_L2SZ, idr2)); istmd = !!FIELD_GET(GICV5_IRS_IDR2_ISTMD, idr2); diff --git a/drivers/irqchip/irq-gic-v5.c b/drivers/irqchip/irq-gic-v5.c index 5f2551cf077d..0341b05c888b 100644 --- a/drivers/irqchip/irq-gic-v5.c +++ b/drivers/irqchip/irq-gic-v5.c @@ -1122,8 +1122,10 @@ static void gicv5_set_cpuif_idbits(void) #ifdef CONFIG_KVM static struct gic_kvm_info gic_v5_kvm_info __initdata; -static void __init gic_of_setup_kvm_info(struct device_node *node) +static int __init gic_setup_kvm_info(unsigned int maint_irq) { + struct gicv5_irs_chip_data *irs_data = gicv5_irs_get_chip_data(); + /* * If we don't have native GICv5 virtualisation support, then * we also don't have FEAT_GCIE_LEGACY - the architecture @@ -1131,25 +1133,120 @@ static void __init gic_of_setup_kvm_info(struct device_node *node) */ if (!gicv5_global_data.virt_capable) { pr_info("GIC implementation is not virtualization capable\n"); - return; + return -ENODEV; } + if (WARN_ON(!irs_data)) + return -ENODEV; + gic_v5_kvm_info.type = GIC_V5; + gic_v5_kvm_info.gicv5_irs.base = irs_data->irs_base; + gic_v5_kvm_info.gicv5_irs.non_coherent = !!(irs_data->flags & IRS_FLAGS_NON_COHERENT); /* GIC Virtual CPU interface maintenance interrupt */ gic_v5_kvm_info.no_maint_irq_mask = false; - gic_v5_kvm_info.maint_irq = irq_of_parse_and_map(node, 0); - if (!gic_v5_kvm_info.maint_irq) { - pr_warn("cannot find GICv5 virtual CPU interface maintenance interrupt\n"); + gic_v5_kvm_info.maint_irq = maint_irq; + + /* + * We require an MI if we have legacy support, but don't, otherwise. + * Given that there's an existing flag to convey that an MI isn't + * needed, we (ab)use it to tell KVM that the MI isn't needed if we + * don't support legacy. + * + * The check for ARM64_HAS_GICV5_LEGACY explicitly doesn't use + * cpus_have_final_cap() here as we run too early. + */ + if (!cpus_have_cap(ARM64_HAS_GICV5_LEGACY) && !gic_v5_kvm_info.maint_irq) + gic_v5_kvm_info.no_maint_irq_mask = true; + + vgic_set_kvm_info(&gic_v5_kvm_info); + return 0; +} + +static void __init gic_of_setup_kvm_info(struct device_node *node) +{ + unsigned int maint_irq; + + /* GIC Virtual CPU interface maintenance interrupt */ + maint_irq = irq_of_parse_and_map(node, 0); + if (gic_setup_kvm_info(maint_irq)) + irq_dispose_mapping(maint_irq); +} + +#ifdef CONFIG_ACPI +struct gicv5_acpi_kvm_info { + u32 maint_irq; + bool valid; +}; + +static struct gicv5_acpi_kvm_info acpi_v5_kvm_info __initdata; + +static int __init gic_acpi_parse_virt_madt_gicc(union acpi_subtable_headers *header, + const unsigned long end) +{ + struct acpi_madt_generic_interrupt *gicc = + (struct acpi_madt_generic_interrupt *)header; + + if (!(gicc->flags & + (ACPI_MADT_ENABLED | ACPI_MADT_GICC_ONLINE_CAPABLE))) + return 0; + + if (gicc->flags & ACPI_MADT_VGIC_IRQ_MODE) + pr_warn_once(FW_BUG "MI wrongly advertised as Edge-triggered\n"); + + if (!acpi_v5_kvm_info.valid) { + acpi_v5_kvm_info.valid = true; + acpi_v5_kvm_info.maint_irq = gicc->vgic_interrupt; + return 0; + } + + if (acpi_v5_kvm_info.maint_irq != gicc->vgic_interrupt) + pr_warn_once(FW_BUG "MI not the same on all CPUs\n"); + + return 0; +} + +static bool __init gic_acpi_collect_virt_info(void) +{ + acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT, + gic_acpi_parse_virt_madt_gicc, 0); + + return acpi_v5_kvm_info.valid; +} + +static void __init gic_acpi_setup_kvm_info(void) +{ + int irq = 0; + + if (!gic_acpi_collect_virt_info()) { + pr_warn("Unable to get hardware information used for virtualization\n"); return; } - vgic_set_kvm_info(&gic_v5_kvm_info); + if (acpi_v5_kvm_info.maint_irq) { + irq = acpi_register_gsi(NULL, acpi_v5_kvm_info.maint_irq, + ACPI_LEVEL_SENSITIVE, + ACPI_ACTIVE_HIGH); + if (irq <= 0) { + pr_warn("Failed to register GSI for GICv5 maintenance IRQ\n"); + irq = 0; + } + } + + if (gic_setup_kvm_info(irq) && irq) + acpi_unregister_gsi(acpi_v5_kvm_info.maint_irq); } +#endif // CONFIG_ACPI #else static inline void __init gic_of_setup_kvm_info(struct device_node *node) { } + +#ifdef CONFIG_ACPI +static inline void __init gic_acpi_setup_kvm_info(void) +{ +} +#endif #endif // CONFIG_KVM static int __init gicv5_init_common(struct fwnode_handle *parent_domain) @@ -1259,6 +1356,8 @@ static int __init gic_acpi_init(union acpi_subtable_headers *header, const unsig acpi_set_irq_model(ACPI_IRQ_MODEL_GIC_V5, gic_v5_get_gsi_domain_id, gic_v5_get_gsi_handle); + gic_acpi_setup_kvm_info(); + return 0; out_irs: diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index bc6f2fdd7ad3..80f96ea59f1a 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -195,4 +195,19 @@ static inline void timer_set_offset(struct arch_timer_context *ctxt, u64 offset) WRITE_ONCE(*ctxt->offset.vm_offset, offset); } +/* + * CVAL to program when the hardware won't apply the timer's offset, so that + * its compare against the raw counter matches the guest's at 'now'. An + * expired timer stays expired; one past the wrap never fires. + */ +static inline u64 timer_apply_offset(u64 cval, u64 offset, u64 now) +{ + u64 hw = cval + offset; + + if (now - offset >= cval) + return 0; + + return hw > now ? hw : U64_MAX; +} + #endif diff --git a/include/kvm/arm_psci.h b/include/kvm/arm_psci.h index f86a006d6713..06c20612e9e7 100644 --- a/include/kvm/arm_psci.h +++ b/include/kvm/arm_psci.h @@ -10,6 +10,8 @@ #include <linux/kvm_host.h> #include <uapi/linux/psci.h> +#include <asm/kvm_emulate.h> + #define KVM_ARM_PSCI_0_1 PSCI_VERSION(0, 1) #define KVM_ARM_PSCI_0_2 PSCI_VERSION(0, 2) #define KVM_ARM_PSCI_1_0 PSCI_VERSION(1, 0) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 086b7578e5f3..7be81b936bf1 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -31,7 +31,12 @@ #define VGIC_MAX_SPI 1019 #define VGIC_MAX_RESERVED 1023 #define VGIC_MIN_LPI 8192 -#define KVM_IRQCHIP_NUM_PINS (1020 - 32) + +/* + * The common IRQ routing table must accommodate all GIC types. GICv2 and + * GICv3 need 988 pins, while GICv5 needs 1024 pins. + */ +#define KVM_IRQCHIP_NUM_PINS 1024 /* * GICv5 supports 128 PPIs, but only the first 64 are architected. We only @@ -40,6 +45,8 @@ * in KVM for now. At a future stage, this can be bumped up to 128, if required. */ #define VGIC_V5_NR_PRIVATE_IRQS 64 +#define VGIC_V5_DEFAULT_NR_SPIS 32 +#define VGIC_V5_MAX_NR_SPIS BIT(10) #define is_v5_type(t, i) (FIELD_GET(GICV5_HWIRQ_TYPE, (i)) == (t)) @@ -132,6 +139,19 @@ enum vgic_type { VGIC_V5, /* Newer, fancier GICv5 */ }; +struct vgic_v5_irs_caps { + /* Base address of the host IRS's CONFIG_FRAME */ + void __iomem *irs_base; + + /* Raw host IRS ID registers */ + u32 idr2; + u32 idr3; + u32 idr4; + + /* Is the IRS coherent with us, or not? */ + bool non_coherent; +}; + /* same for all guests, as depending only on the _host's_ GIC model */ struct vgic_global { /* type of the host GIC */ @@ -159,9 +179,16 @@ struct vgic_global { /* Maintenance IRQ number */ unsigned int maint_irq; - /* maximum number of VCPUs allowed (GICv2 limits us to 8) */ + /* + * Maximum number of VCPUs exposed before userspace has selected a + * VGIC model. Individual VGIC models can impose a lower limit + * (GICv2 limits us to 8). + */ int max_gic_vcpus; + /* Maximum number of VCPUs allowed for a GICv5 VM. */ + int max_gicv5_vcpus; + /* Only needed for the legacy KVM_CREATE_IRQCHIP */ bool can_emulate_gicv2; @@ -182,6 +209,9 @@ struct vgic_global { struct { DECLARE_BITMAP(impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); } vgic_v5_ppi_caps; + + /* GICv5 IRS capabilities */ + struct vgic_v5_irs_caps vgic_v5_irs_caps; }; extern struct vgic_global kvm_vgic_global_state; @@ -217,6 +247,12 @@ struct irq_ops { bool (*get_input_level)(int vintid); /* + * Function pointer to directly update hardware pending state after the + * VGIC shadow pending state has changed. + */ + bool (*set_pending_state)(struct kvm_vcpu *vcpu, struct vgic_irq *irq); + + /* * Function pointer to override the queuing of an IRQ. */ bool (*queue_irq_unlock)(struct kvm *kvm, struct vgic_irq *irq, @@ -287,7 +323,8 @@ enum iodev_type { IODEV_CPUIF, IODEV_DIST, IODEV_REDIST, - IODEV_ITS + IODEV_ITS, + IODEV_GICV5_IRS }; struct vgic_io_device { @@ -345,7 +382,11 @@ struct vgic_redist_region { struct list_head list; }; +#define VGIC_V5_VM_ID_INVAL (-1) + struct vgic_v5_vm { + struct kvm *kvm; + /* * We only expose a subset of PPIs to the guest. This subset is a * combination of the PPIs that are actually implemented and what we @@ -363,6 +404,96 @@ struct vgic_v5_vm { * convenient way to do that). */ DECLARE_BITMAP(vgic_ppi_hmr, VGIC_V5_NR_PRIVATE_IRQS); + + struct fwnode_handle *fwnode; + struct irq_domain *domain; + int vpe_db_base; + u32 vm_id; + bool vmte_allocated; +}; + +/*** GICv5 ***/ +struct vgic_v5_irs { + /* base addresses in guest physical address space: */ + gpa_t vgic_v5_irs_base; + + struct vgic_io_device iodev; + struct kvm_device *dev; + + /* IRS state - used for registers etc */ + struct { + u8 domain; + u8 pa_range; + bool virt; + bool setlpi; + bool mec; + bool mpam; + bool swe; + u16 irs_id; + } idr0; + + struct { + /* PE_CNT is populated from online_vcpus at runtime */ + u8 priority_bits; + } idr1; + + struct { + u8 id_bits; + u8 min_lpi_id_bits; + bool ist_levels; + u8 ist_l2sz; + bool istmd; + u8 istmd_sz; + } idr2; + + struct { + u32 spi_range; + } idr5; + + struct { + u32 spi_irs_range; + } idr6; + + struct { + u32 spi_base; + } idr7; + + struct { + u8 sh; + u8 oc; + u8 ic; + bool ist_ra; + bool ist_wa; + bool vmt_ra; + bool vpet_ra; + bool vmd_ra; + bool vmd_wa; + bool vped_ra; + bool vped_wa; + } cr1; + + struct { + u32 id; + } spi_selr; + + struct { + u32 iaffid; + } pe_selr; + + struct { + u8 lpi_id_bits; + u8 l2sz; + u8 istsz; + bool structure; + } ist_cfgr; + + struct { + bool valid; + u64 addr; + } ist_baser; + + /* A valid userspace-restored BASER still needs its host LPI IST. */ + bool lpi_ist_restore_pending; }; struct vgic_dist { @@ -442,6 +573,11 @@ struct vgic_dist { * GICv5 per-VM data. */ struct vgic_v5_vm gicv5_vm; + + /* + * GICv5 IRS data. Dynamically allocated due to the size. + */ + struct vgic_v5_irs *vgic_v5_irs_data; }; struct vgic_v2_cpu_if { @@ -492,6 +628,9 @@ struct vgic_v5_cpu_if { */ u64 vgic_icsr; + /* The contextr used to make VPEs resident and non-resident */ + u64 vgic_contextr; + struct gicv5_vpe gicv5_vpe; }; @@ -598,10 +737,11 @@ void kvm_vgic_process_async_update(struct kvm_vcpu *vcpu); void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1); /** - * kvm_vgic_get_max_vcpus - Get the maximum number of VCPUs allowed by HW + * kvm_vgic_get_max_vcpus - Get the pre-VGIC-selection VCPU limit * - * The host's GIC naturally limits the maximum amount of VCPUs a guest - * can use. + * Userspace can query KVM_CAP_MAX_VCPUS before selecting a VGIC model, so + * expose the highest model-specific limit and let kvm_vgic_create() enforce + * the selected model's actual limit. */ static inline int kvm_vgic_get_max_vcpus(void) { @@ -627,7 +767,66 @@ int vgic_v4_load(struct kvm_vcpu *vcpu); void vgic_v4_commit(struct kvm_vcpu *vcpu); int vgic_v4_put(struct kvm_vcpu *vcpu); -int vgic_v5_finalize_ppi_state(struct kvm *kvm); +static inline u8 vgic_v5_irs_ist_id_bits(const struct vgic_v5_irs_caps *caps) +{ + return FIELD_GET(GICV5_IRS_IDR2_ID_BITS, caps->idr2); +} + +static inline u8 vgic_v5_irs_min_lpi_id_bits(const struct vgic_v5_irs_caps *caps) +{ + return FIELD_GET(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, caps->idr2); +} + +static inline bool vgic_v5_irs_ist_levels(const struct vgic_v5_irs_caps *caps) +{ + return !!(caps->idr2 & GICV5_IRS_IDR2_IST_LEVELS); +} + +static inline u8 vgic_v5_irs_ist_l2sz(const struct vgic_v5_irs_caps *caps) +{ + return FIELD_GET(GICV5_IRS_IDR2_IST_L2SZ, caps->idr2); +} + +static inline bool vgic_v5_irs_istmd(const struct vgic_v5_irs_caps *caps) +{ + return !!(caps->idr2 & GICV5_IRS_IDR2_ISTMD); +} + +static inline u8 vgic_v5_irs_istmd_sz(const struct vgic_v5_irs_caps *caps) +{ + return FIELD_GET(GICV5_IRS_IDR2_ISTMD_SZ, caps->idr2); +} + +static inline u32 vgic_v5_irs_max_vms(const struct vgic_v5_irs_caps *caps) +{ + return BIT(FIELD_GET(GICV5_IRS_IDR3_VM_ID_BITS, caps->idr3)); +} + +static inline bool vgic_v5_irs_two_level_vmt_support(const struct vgic_v5_irs_caps *caps) +{ + return !!(caps->idr3 & GICV5_IRS_IDR3_VMT_LEVELS); +} + +static inline u16 vgic_v5_irs_vmd_size(const struct vgic_v5_irs_caps *caps) +{ + if (!(caps->idr3 & GICV5_IRS_IDR3_VMD)) + return 0; + + return BIT(FIELD_GET(GICV5_IRS_IDR3_VMD_SZ, caps->idr3)); +} + +static inline u16 vgic_v5_irs_vped_size(const struct vgic_v5_irs_caps *caps) +{ + return BIT(FIELD_GET(GICV5_IRS_IDR4_VPED_SZ, caps->idr4)); +} + +static inline u32 vgic_v5_irs_max_vpes(const struct vgic_v5_irs_caps *caps) +{ + /* Field stores VPE_ID_BITS - 1 */ + return BIT(FIELD_GET(GICV5_IRS_IDR4_VPE_ID_BITS, caps->idr4) + 1); +} + +int vgic_v5_finalize_ppi_state(struct kvm_vcpu *vcpu); bool vgic_v5_ppi_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, unsigned long flags); void vgic_v5_set_ppi_dvi(struct kvm_vcpu *vcpu, struct vgic_irq *irq, bool dvi); diff --git a/include/linux/irqchip/arm-gic-v5.h b/include/linux/irqchip/arm-gic-v5.h index 2c2fb39f049c..299e75bc9525 100644 --- a/include/linux/irqchip/arm-gic-v5.h +++ b/include/linux/irqchip/arm-gic-v5.h @@ -63,20 +63,34 @@ #define GICV5_OUTER_SHARE 0b10 #define GICV5_INNER_SHARE 0b11 +#define GICV5_AIDR_COMPONENT_IRS 0b00 +#define GICV5_AIDR_COMPONENT_ITS 0b01 +#define GICV5_AIDR_COMPONENT_IWB 0b10 + +#define GICV5_AIDR_ARCH_MAJ_REV_V5 0 +#define GICV5_AIDR_ARCH_MIN_REV_V0 0 + /* * IRS registers and tables structures */ #define GICV5_IRS_IDR0 0x0000 #define GICV5_IRS_IDR1 0x0004 #define GICV5_IRS_IDR2 0x0008 +#define GICV5_IRS_IDR3 0x000c +#define GICV5_IRS_IDR4 0x0010 #define GICV5_IRS_IDR5 0x0014 #define GICV5_IRS_IDR6 0x0018 #define GICV5_IRS_IDR7 0x001c +#define GICV5_IRS_IIDR 0x0040 +#define GICV5_IRS_AIDR 0x0044 #define GICV5_IRS_CR0 0x0080 #define GICV5_IRS_CR1 0x0084 #define GICV5_IRS_SYNCR 0x00c0 #define GICV5_IRS_SYNC_STATUSR 0x00c4 +#define GICV5_IRS_SPI_VMR 0x0100 #define GICV5_IRS_SPI_SELR 0x0108 +#define GICV5_IRS_SPI_DOMAINR 0x010c +#define GICV5_IRS_SPI_RESAMPLER 0x0110 #define GICV5_IRS_SPI_CFGR 0x0114 #define GICV5_IRS_SPI_STATUSR 0x0118 #define GICV5_IRS_PE_SELR 0x0140 @@ -86,11 +100,51 @@ #define GICV5_IRS_IST_CFGR 0x0190 #define GICV5_IRS_IST_STATUSR 0x0194 #define GICV5_IRS_MAP_L2_ISTR 0x01c0 - +#define GICV5_IRS_VMT_BASER 0x0200 +#define GICV5_IRS_VMT_CFGR 0x0210 +#define GICV5_IRS_VMT_STATUSR 0x0214 +#define GICV5_IRS_VPE_SELR 0x0240 +#define GICV5_IRS_VPE_DBR 0x0248 +#define GICV5_IRS_VPE_HPPIR 0x0250 +#define GICV5_IRS_VPE_CR0 0x0258 +#define GICV5_IRS_VPE_STATUSR 0x025c +#define GICV5_IRS_VM_DBR 0x0280 +#define GICV5_IRS_VM_SELR 0x0288 +#define GICV5_IRS_VM_STATUSR 0x028c +#define GICV5_IRS_VMAP_L2_VMTR 0x02c0 +#define GICV5_IRS_VMAP_VMR 0x02c8 +#define GICV5_IRS_VMAP_VISTR 0x02d0 +#define GICV5_IRS_VMAP_L2_VISTR 0x02d8 +#define GICV5_IRS_VMAP_VPER 0x02e0 +#define GICV5_IRS_SAVE_VMR 0x0300 +#define GICV5_IRS_SAVE_VM_STATUSR 0x0308 +#define GICV5_IRS_MEC_IDR 0x0340 +#define GICV5_IRS_MEC_MECID_R 0x0344 +#define GICV5_IRS_MPAM_IDR 0x0380 +#define GICV5_IRS_MPAM_PARTID_R 0x0384 +#define GICV5_IRS_SWERR_STATUSR 0x03c0 +#define GICV5_IRS_SWERR_SYNDROMER0 0x03c8 +#define GICV5_IRS_SWERR_SYNDROMER1 0x03d0 + +#define GICV5_IRS_IDR0_IRSID GENMASK(31, 16) +#define GICV5_IRS_IDR0_SWE BIT(12) +#define GICV5_IRS_IDR0_MPAM BIT(11) +#define GICV5_IRS_IDR0_MEC BIT(10) +#define GICV5_IRS_IDR0_SETLPI BIT(9) +#define GICV5_IRS_IDR0_VIRT_ONE_N BIT(8) +#define GICV5_IRS_IDR0_ONE_N BIT(7) #define GICV5_IRS_IDR0_VIRT BIT(6) +#define GICV5_IRS_IDR0_PA_RANGE GENMASK(5, 2) +#define GICV5_IRS_IDR0_INT_DOM GENMASK(1, 0) + +#define GICV5_IRS_IDR0_INT_DOM_SECURE 0b00 +#define GICV5_IRS_IDR0_INT_DOM_NON_SECURE 0b01 +#define GICV5_IRS_IDR0_INT_DOM_EL3 0b10 +#define GICV5_IRS_IDR0_INT_DOM_REALM 0b11 #define GICV5_IRS_IDR1_PRIORITY_BITS GENMASK(22, 20) #define GICV5_IRS_IDR1_IAFFID_BITS GENMASK(19, 16) +#define GICV5_IRS_IDR1_PE_CNT GENMASK(15, 0) #define GICV5_IRS_IDR1_PRIORITY_BITS_1BITS 0b000 #define GICV5_IRS_IDR1_PRIORITY_BITS_2BITS 0b001 @@ -106,13 +160,30 @@ #define GICV5_IRS_IDR2_LPI BIT(5) #define GICV5_IRS_IDR2_ID_BITS GENMASK(4, 0) +#define GICV5_IRS_IST_L2SZ_SUPPORT_4KB(r) FIELD_GET(BIT(0), (r)) +#define GICV5_IRS_IST_L2SZ_SUPPORT_16KB(r) FIELD_GET(BIT(1), (r)) +#define GICV5_IRS_IST_L2SZ_SUPPORT_64KB(r) FIELD_GET(BIT(2), (r)) + +#define GICV5_IRS_IDR3_VMT_LEVELS BIT(10) +#define GICV5_IRS_IDR3_VM_ID_BITS GENMASK(9, 5) +#define GICV5_IRS_IDR3_VMD_SZ GENMASK(4, 1) +#define GICV5_IRS_IDR3_VMD BIT(0) + +#define GICV5_IRS_IDR4_VPE_ID_BITS GENMASK(9, 6) +#define GICV5_IRS_IDR4_VPED_SZ GENMASK(5, 0) + #define GICV5_IRS_IDR5_SPI_RANGE GENMASK(24, 0) #define GICV5_IRS_IDR6_SPI_IRS_RANGE GENMASK(24, 0) #define GICV5_IRS_IDR7_SPI_BASE GENMASK(23, 0) -#define GICV5_IRS_IST_L2SZ_SUPPORT_4KB(r) FIELD_GET(BIT(11), (r)) -#define GICV5_IRS_IST_L2SZ_SUPPORT_16KB(r) FIELD_GET(BIT(12), (r)) -#define GICV5_IRS_IST_L2SZ_SUPPORT_64KB(r) FIELD_GET(BIT(13), (r)) +#define GICV5_IRS_IIDR_PRODUCT_ID GENMASK(31, 20) +#define GICV5_IRS_IIDR_VARIANT GENMASK(19, 16) +#define GICV5_IRS_IIDR_REVISION GENMASK(15, 12) +#define GICV5_IRS_IIDR_IMPLEMENTER GENMASK(11, 0) + +#define GICV5_IRS_AIDR_COMPONENT GENMASK(11, 8) +#define GICV5_IRS_AIDR_ARCHMAJORREV GENMASK(7, 4) +#define GICV5_IRS_AIDR_ARCHMINORREV GENMASK(3, 0) #define GICV5_IRS_CR0_IDLE BIT(1) #define GICV5_IRS_CR0_IRSEN BIT(0) @@ -135,21 +206,39 @@ #define GICV5_IRS_SYNC_STATUSR_IDLE BIT(0) -#define GICV5_IRS_SPI_STATUSR_V BIT(1) -#define GICV5_IRS_SPI_STATUSR_IDLE BIT(0) +#define GICV5_IRS_SPI_VMR_VIRT BIT_ULL(63) +#define GICV5_IRS_SPI_VMR_VM_ID GENMASK_ULL(15, 0) #define GICV5_IRS_SPI_SELR_ID GENMASK(23, 0) +#define GICV5_IRS_SPI_DOMAINR_DOMAIN GENMASK(1, 0) + +#define GICV5_IRS_SPI_DOMAINR_DOMAIN_SECURE 0b00 +#define GICV5_IRS_SPI_DOMAINR_DOMAIN_NON_SECURE 0b01 +#define GICV5_IRS_SPI_DOMAINR_DOMAIN_EL3 0b10 +#define GICV5_IRS_SPI_DOMAINR_DOMAIN_REALM 0b11 + +#define GICV5_IRS_SPI_RESAMPLER_ID GENMASK(23, 0) + #define GICV5_IRS_SPI_CFGR_TM BIT(0) +#define GICV5_IRS_SPI_CFGR_TM_EDGE 0b0 +#define GICV5_IRS_SPI_CFGR_TM_LEVEL 0b1 + +#define GICV5_IRS_SPI_STATUSR_V BIT(1) +#define GICV5_IRS_SPI_STATUSR_IDLE BIT(0) + #define GICV5_IRS_PE_SELR_IAFFID GENMASK(15, 0) +#define GICV5_IRS_PE_STATUSR_ONLINE BIT(2) #define GICV5_IRS_PE_STATUSR_V BIT(1) #define GICV5_IRS_PE_STATUSR_IDLE BIT(0) #define GICV5_IRS_PE_CR0_DPS BIT(0) -#define GICV5_IRS_IST_STATUSR_IDLE BIT(0) +#define GICV5_IRS_IST_BASER_ADDR_MASK GENMASK_ULL(55, 6) +#define GICV5_IRS_IST_BASER_VALID BIT_ULL(0) +#define GICV5_IRS_IST_BASER_ADDR_SHIFT 6ULL #define GICV5_IRS_IST_CFGR_STRUCTURE BIT(16) #define GICV5_IRS_IST_CFGR_ISTSZ GENMASK(8, 7) @@ -167,15 +256,111 @@ #define GICV5_IRS_IST_CFGR_L2SZ_16K 0b01 #define GICV5_IRS_IST_CFGR_L2SZ_64K 0b10 -#define GICV5_IRS_IST_BASER_ADDR_MASK GENMASK_ULL(55, 6) -#define GICV5_IRS_IST_BASER_VALID BIT_ULL(0) +#define GICV5_IRS_IST_STATUSR_IDLE BIT(0) #define GICV5_IRS_MAP_L2_ISTR_ID GENMASK(23, 0) +#define GICV5_IRS_VMT_BASER_ADDR GENMASK_ULL(55, 3) +#define GICV5_IRS_VMT_BASER_ADDR_SHIFT 3ULL +#define GICV5_IRS_VMT_BASER_VALID BIT_ULL(0) + +#define GICV5_IRS_VMT_CFGR_STRUCTURE_TWO_LEVEL 0b1 +#define GICV5_IRS_VMT_CFGR_STRUCTURE_LINEAR 0b0 + +#define GICV5_IRS_VMT_CFGR_STRUCTURE BIT(16) +#define GICV5_IRS_VMT_CFGR_VM_ID_BITS GENMASK(4, 0) + +#define GICV5_IRS_VMT_STATUSR_IDLE BIT(0) + +#define GICV5_IRS_VPE_SELR_S BIT_ULL(63) +#define GICV5_IRS_VPE_SELR_VPE_ID GENMASK_ULL(47, 32) +#define GICV5_IRS_VPE_SELR_VM_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_VPE_DBR_DBV BIT_ULL(63) +#define GICV5_IRS_VPE_DBR_REQ_DB BIT_ULL(62) +#define GICV5_IRS_VPE_DBR_DBPM GENMASK_ULL(36, 32) +#define GICV5_IRS_VPE_DBR_INTID GENMASK_ULL(23, 0) + +#define GICV5_IRS_VPE_HPPIR_HPPIV BIT_ULL(32) +#define GICV5_IRS_VPE_HPPIR_TYPE GENMASK_ULL(31, 29) +#define GICV5_IRS_VPE_HPPIR_ID GENMASK_ULL(23, 0) + +#define GICV5_IRS_VPE_CR0_DPS BIT(0) + +#define GICV5_IRS_VPE_STATUSR_V BIT(1) +#define GICV5_IRS_VPE_STATUSR_IDLE BIT(0) + +#define GICV5_IRS_VM_DBR_EN BIT_ULL(63) +#define GICV5_IRS_VM_DBR_VPE_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_VM_SELR_VM_ID GENMASK(15, 0) + +#define GICV5_IRS_VM_STATUSR_V BIT(1) +#define GICV5_IRS_VM_STATUSR_IDLE BIT(0) + +#define GICV5_IRS_VMAP_L2_VMTR_M BIT_ULL(63) +#define GICV5_IRS_VMAP_L2_VMTR_VM_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_VMAP_VMR_M BIT_ULL(63) +#define GICV5_IRS_VMAP_VMR_U BIT_ULL(62) +#define GICV5_IRS_VMAP_VMR_VM_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_VMAP_VISTR_M BIT_ULL(63) +#define GICV5_IRS_VMAP_VISTR_U BIT_ULL(62) +#define GICV5_IRS_VMAP_VISTR_VM_ID GENMASK_ULL(47, 32) +#define GICV5_IRS_VMAP_VISTR_TYPE GENMASK_ULL(31, 29) + +#define GICV5_IRS_VMAP_L2_VISTR_M BIT_ULL(63) +#define GICV5_IRS_VMAP_L2_VISTR_VM_ID GENMASK_ULL(47, 32) +#define GICV5_IRS_VMAP_L2_VISTR_TYPE GENMASK_ULL(31, 29) +#define GICV5_IRS_VMAP_L2_VISTR_ID GENMASK_ULL(23, 0) + +#define GICV5_IRS_VMAP_VPER_M BIT_ULL(63) +#define GICV5_IRS_VMAP_VPER_VM_ID GENMASK_ULL(47, 32) +#define GICV5_IRS_VMAP_VPER_VPE_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_SAVE_VMR_VM_ID GENMASK_ULL(15, 0) +#define GICV5_IRS_SAVE_VMR_Q BIT_ULL(62) +#define GICV5_IRS_SAVE_VMR_S BIT_ULL(63) + +#define GICV5_IRS_SAVE_VM_STATUSR_IDLE BIT(0) +#define GICV5_IRS_SAVE_VM_STATUSR_Q BIT(1) + +#define GICV5_IRS_MEC_IDR_MECIDSIZE GENMASK(3, 0) + +#define GICV5_IRS_MEC_MECID_R_MECID GENMASK(15, 0) + +#define GICV5_IRS_MPAM_IDR_HAS_MPAM_SP BIT(24) +#define GICV5_IRS_MPAM_IDR_PMG_MAX GENMASK(23, 16) +#define GICV5_IRS_MPAM_IDR_PARTID_MAX GENMASK(15, 0) + +#define GICV5_IRS_MPAM_PARTID_R_IDLE BIT(31) +#define GICV5_IRS_MPAM_PARTID_R_MPAM_SP GENMASK(25, 24) +#define GICV5_IRS_MPAM_PARTID_R_PMG GENMASK(23, 16) +#define GICV5_IRS_MPAM_PARTID_R_PARTID GENMASK(15, 0) + +#define GICV5_IRS_SWERR_STATUSR_IMP_EC GENMASK_ULL(31, 24) +#define GICV5_IRS_SWERR_STATUSR_EC GENMASK_ULL(23, 16) +#define GICV5_IRS_SWERR_STATUSR_OF BIT_ULL(3) +#define GICV5_IRS_SWERR_STATUSR_S1V BIT_ULL(2) +#define GICV5_IRS_SWERR_STATUSR_S0V BIT_ULL(1) +#define GICV5_IRS_SWERR_STATUSR_V BIT_ULL(0) + +#define GICV5_IRS_SWERR_SYNDROMER0_VIRTUAL BIT_ULL(63) +#define GICV5_IRS_SWERR_SYNDROMER0_TYPE GENMASK_ULL(62, 60) +#define GICV5_IRS_SWERR_SYNDROMER0_ID GENMASK_ULL(55, 32) +#define GICV5_IRS_SWERR_SYNDROMER0_VM_ID GENMASK_ULL(15, 0) + +#define GICV5_IRS_SWERR_SYNDROMER1_ADDR GENMASK_ULL(55, 3) + #define GICV5_ISTL1E_VALID BIT_ULL(0) +#define GICV5_IRS_ISTL1E_SIZE 8UL #define GICV5_ISTL1E_L2_ADDR_MASK GENMASK_ULL(55, 12) +#define GICV5_IRS_SETLPIR 0x0000 +#define GICV5_IRS_SETLPIR_ID GENMASK(23, 0) + /* * ITS registers and tables structures */ @@ -298,6 +483,42 @@ #define GICV5_GSI_IWB_WIRE GENMASK(15, 0) /* + * CoreSight identification registers - ordered by increasing offset. + */ +#define GICV5_CORESIGHT_DEVARCH 0xffbc +#define GICV5_CORESIGHT_PIDR4 0xffd0 +#define GICV5_CORESIGHT_PIDR5 0xffd4 +#define GICV5_CORESIGHT_PIDR6 0xffd8 +#define GICV5_CORESIGHT_PIDR7 0xffdc +#define GICV5_CORESIGHT_PIDR0 0xffe0 +#define GICV5_CORESIGHT_PIDR1 0xffe4 +#define GICV5_CORESIGHT_PIDR2 0xffe8 +#define GICV5_CORESIGHT_PIDR3 0xffec +#define GICV5_CORESIGHT_CIDR0 0xfff0 +#define GICV5_CORESIGHT_CIDR1 0xfff4 +#define GICV5_CORESIGHT_CIDR2 0xfff8 +#define GICV5_CORESIGHT_CIDR3 0xfffc + +#define GICV5_CORESIGHT_DEVARCH_VAL \ + (FIELD_PREP(GENMASK(31, 21), 0x23b) | \ + BIT(20) | \ + 0x5a19) + +#define GICV5_CORESIGHT_PIDR4_JEP106_CONT 0x04 +#define GICV5_CORESIGHT_PIDR5_RES0 0x00 +#define GICV5_CORESIGHT_PIDR6_RES0 0x00 +#define GICV5_CORESIGHT_PIDR7_RES0 0x00 +#define GICV5_CORESIGHT_PIDR0_PART_0 0x4b +#define GICV5_CORESIGHT_PIDR1_DES_0_PART_1 0xb0 +#define GICV5_CORESIGHT_PIDR2_DES_1 0x0b +#define GICV5_CORESIGHT_PIDR3_REVAND_CMOD 0x00 + +#define GICV5_CORESIGHT_CIDR0_VAL 0x0d +#define GICV5_CORESIGHT_CIDR1_VAL 0xf0 +#define GICV5_CORESIGHT_CIDR2_VAL 0x05 +#define GICV5_CORESIGHT_CIDR3_VAL 0xb1 + +/* * Global Data structures and functions */ struct gicv5_chip_data { @@ -332,6 +553,8 @@ struct gicv5_irs_chip_data { raw_spinlock_t spi_config_lock; }; +#define IRS_FLAGS_NON_COHERENT BIT(0) + static inline int gicv5_wait_for_op_s_atomic(void __iomem *addr, u32 offset, const char *reg_s, u32 mask, u32 *val) @@ -379,6 +602,7 @@ void __init gicv5_free_lpi_domain(void); int gicv5_irs_of_probe(struct device_node *parent); int gicv5_irs_acpi_probe(void); +struct gicv5_irs_chip_data *gicv5_irs_get_chip_data(void); void gicv5_irs_remove(void); int gicv5_irs_enable(void); void gicv5_irs_its_probe(void); @@ -387,10 +611,13 @@ int gicv5_irs_cpu_to_iaffid(int cpu_id, u16 *iaffid); struct gicv5_irs_chip_data *gicv5_irs_lookup_by_spi_id(u32 spi_id); int gicv5_spi_irq_set_type(struct irq_data *d, unsigned int type); int gicv5_irs_iste_alloc(u32 lpi); +unsigned int gicv5_irs_l2_sz(u32 l2sz); void gicv5_irs_syncr(void); /* Embedded in kvm.arch */ struct gicv5_vpe { + int db; + bool db_fired; bool resident; }; @@ -429,4 +656,15 @@ void gicv5_deinit_lpis(void); void __init gicv5_its_of_probe(struct device_node *parent); void __init gicv5_its_acpi_probe(void); + +enum gicv5_vcpu_cmd { + VMT_L2_MAP, /* Map in a L2 VMT - *may* happen on VM init */ + VMTE_MAKE_VALID, /* Make the VMTE valid */ + VMTE_MAKE_INVALID, /* Make the VMTE (et al.) invalid */ + VPE_MAKE_VALID, /* No corresponding invalid */ + SPI_VIST_MAKE_VALID, /* No corresponding invalid */ + LPI_VIST_MAKE_VALID, /* Triggered by a guest */ + LPI_VIST_MAKE_INVALID, /* Triggered by a guest */ +}; + #endif diff --git a/include/linux/irqchip/arm-vgic-info.h b/include/linux/irqchip/arm-vgic-info.h index 67d9d960273b..f05370e2debf 100644 --- a/include/linux/irqchip/arm-vgic-info.h +++ b/include/linux/irqchip/arm-vgic-info.h @@ -38,6 +38,11 @@ struct gic_kvm_info { bool has_v4_1; /* Deactivation impared, subpar stuff */ bool no_hw_deactivation; + /* GICv5 IRS base */ + struct { + void __iomem *base; + bool non_coherent; + } gicv5_irs; }; #ifdef CONFIG_KVM diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index c8a29162b3f9..afecd0fd55ad 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -566,7 +566,7 @@ static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu) /* * The memory barrier ensures a previous write to vcpu->requests cannot * be reordered with the read of vcpu->mode. It pairs with the general - * memory barrier following the write of vcpu->mode in VCPU RUN. + * memory barrier following the write of vcpu->mode in KVM_RUN. */ smp_mb__before_atomic(); return cmpxchg(&vcpu->mode, IN_GUEST_MODE, EXITING_GUEST_MODE); @@ -2670,6 +2670,7 @@ void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range #endif #ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY +int kvm_arch_pre_fault_allowed(struct kvm_vcpu *vcpu); long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range); #endif diff --git a/include/linux/trace_remote_event.h b/include/linux/trace_remote_event.h index c8ae1e1f5e72..e4cc2d4497bc 100644 --- a/include/linux/trace_remote_event.h +++ b/include/linux/trace_remote_event.h @@ -3,6 +3,8 @@ #ifndef _LINUX_TRACE_REMOTE_EVENTS_H #define _LINUX_TRACE_REMOTE_EVENTS_H +#include <linux/types.h> + struct trace_remote; struct trace_event_fields; struct trace_seq; diff --git a/include/trace/define_remote_events.h b/include/trace/define_remote_events.h index 676e803dc144..78346ac77585 100644 --- a/include/trace/define_remote_events.h +++ b/include/trace/define_remote_events.h @@ -35,17 +35,26 @@ do { \ #define RE_PRINTK(__args...) __args -#define REMOTE_EVENT(__name, __id, __struct, __printk) \ - REMOTE_EVENT_FORMAT(__name, __struct); \ +#define REMOTE_EVENT_PRINT_FUNC(__name, __printk) \ static void remote_event_print_##__name(void *evt, struct trace_seq *seq) \ { \ struct remote_event_format_##__name __maybe_unused *__entry = evt; \ trace_seq_puts(seq, #__name); \ - remote_printk(__printk); \ + __printk; \ } + +#define REMOTE_EVENT(__name, __id, __struct, __printk) \ + REMOTE_EVENT_FORMAT(__name, __struct); \ + REMOTE_EVENT_PRINT_FUNC(__name, remote_printk(__printk)) + +#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk) \ + REMOTE_EVENT_FORMAT(__name, __struct); \ + REMOTE_EVENT_PRINT_FUNC(__name, __printk) + #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE) #undef REMOTE_EVENT +#undef REMOTE_EVENT_CUSTOM_PRINTK #undef RE_PRINTK #undef re_field #define re_field(__type, __field) \ @@ -70,4 +79,8 @@ do { \ .print_fmt = remote_event_print_fmt_##__name, \ .print = remote_event_print_##__name, \ } + +#define REMOTE_EVENT_CUSTOM_PRINTK(__name, __id, __struct, __printk) \ + REMOTE_EVENT(__name, __id, RE_STRUCT(__struct), "\"Unknown\"") + #include REMOTE_EVENT_INCLUDE(REMOTE_EVENT_INCLUDE_FILE) diff --git a/tools/arch/arm64/include/uapi/asm/kvm.h b/tools/arch/arm64/include/uapi/asm/kvm.h index 1c13bfa2d38a..9998252ea8ec 100644 --- a/tools/arch/arm64/include/uapi/asm/kvm.h +++ b/tools/arch/arm64/include/uapi/asm/kvm.h @@ -97,6 +97,11 @@ struct kvm_regs { #define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K) #define KVM_VGIC_V3_ITS_SIZE (2 * SZ_64K) +/* Supported VGICv5 address types */ +#define KVM_VGIC_V5_ADDR_TYPE_IRS 6 + +#define KVM_VGIC_V5_IRS_SIZE (2 * SZ_64K) + #define KVM_ARM_VCPU_POWER_OFF 0 /* CPU is started in OFF state */ #define KVM_ARM_VCPU_EL1_32BIT 1 /* CPU running a 32bit VM */ #define KVM_ARM_VCPU_PSCI_0_2 2 /* CPU uses PSCI v0.2 */ @@ -417,6 +422,16 @@ enum { #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 #define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 +#define KVM_DEV_ARM_VGIC_GRP_IRS_REGS 10 +#define KVM_DEV_ARM_VGIC_GRP_IST 11 + +struct kvm_vgic_v5_ist { + __u64 spi_ist_addr; + __u64 spi_ist_size; + __u64 lpi_ist_addr; + __u64 lpi_ist_size; +}; + #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm index 51b98f47044f..fa10eb9e8fa7 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -178,6 +178,7 @@ TEST_GEN_PROGS_arm64 += arm64/debug-exceptions TEST_GEN_PROGS_arm64 += arm64/hello_el2 TEST_GEN_PROGS_arm64 += arm64/host_sve TEST_GEN_PROGS_arm64 += arm64/hypercalls +TEST_GEN_PROGS_arm64 += arm64/nv_pre_fault_memory_test TEST_GEN_PROGS_arm64 += arm64/external_aborts TEST_GEN_PROGS_arm64 += arm64/mmio_sign_ext TEST_GEN_PROGS_arm64 += arm64/page_fault_test @@ -206,6 +207,7 @@ TEST_GEN_PROGS_arm64 += guest_memfd_test TEST_GEN_PROGS_arm64 += mmu_stress_test TEST_GEN_PROGS_arm64 += rseq_test TEST_GEN_PROGS_arm64 += steal_time +TEST_GEN_PROGS_arm64 += pre_fault_memory_test TEST_GEN_PROGS_s390 = $(TEST_GEN_PROGS_COMMON) TEST_GEN_PROGS_s390 += s390/memop diff --git a/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c b/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c index d9c9377a6325..d46799ff91dc 100644 --- a/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c +++ b/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c @@ -787,6 +787,30 @@ static void test_timers_in_the_past(enum arch_timer timer) } } +/* + * A cval within long_wait_ms of the counter's wrap while the counter sits at + * wait_ms: a deadline further away than the counter can reach. Unmasked, so + * an early ISTATUS shows up as an IRQ. + */ +static void test_timers_past_counter_wrap(enum arch_timer timer) +{ + u64 cval = -msec_to_cycles(test_args.long_wait_ms); + int i; + + for (i = 0; i < ARRAY_SIZE(sleep_method); i++) { + sleep_method_t sm = sleep_method[i]; + + local_irq_disable(); + set_counter(timer, msec_to_cycles(test_args.wait_ms)); + set_cval_irq(timer, cval, CTL_ENABLE); + sm(timer, msecs_to_usecs(test_args.wait_ms) + TIMEOUT_NO_IRQ_US); + local_irq_enable(); + isb(); + assert_irqs_handled(0); + timer_set_ctl(timer, CTL_IMASK); + } +} + static void test_long_timer_delays(enum arch_timer timer) { s32 tval = (s32)msec_to_cycles(test_args.long_wait_ms); @@ -806,6 +830,7 @@ static void guest_run_iteration(enum arch_timer timer) test_basic_functionality(timer); test_timers_sanity_checks(timer); + test_timers_past_counter_wrap(timer); test_timers_above_tval_max(timer); test_timers_in_the_past(timer); diff --git a/tools/testing/selftests/kvm/arm64/debug-exceptions.c b/tools/testing/selftests/kvm/arm64/debug-exceptions.c index 7dc5f0b4f6ad..9da6dc5bcfe6 100644 --- a/tools/testing/selftests/kvm/arm64/debug-exceptions.c +++ b/tools/testing/selftests/kvm/arm64/debug-exceptions.c @@ -387,7 +387,7 @@ static void guest_code_ss(int test_cnt) * Enable Single Step execution. Note! This _must_ be a bare * ucall as the ucall() path uses atomic operations to manage * the ucall structures, and the built-in "atomics" are usually - * implemented via exclusive access instructions. The exlusive + * implemented via exclusive access instructions. The exclusive * monitor is cleared on ERET, and so taking debug exceptions * during a LDREX=>STREX sequence will prevent forward progress * and hang the guest/test. diff --git a/tools/testing/selftests/kvm/arm64/external_aborts.c b/tools/testing/selftests/kvm/arm64/external_aborts.c index d8fe17a6cc59..1a858d0d8296 100644 --- a/tools/testing/selftests/kvm/arm64/external_aborts.c +++ b/tools/testing/selftests/kvm/arm64/external_aborts.c @@ -75,6 +75,17 @@ static void vcpu_inject_serror(struct kvm_vcpu *vcpu) vcpu_events_set(vcpu, &events); } +static void vcpu_assert_serror_pending(struct kvm_vcpu *vcpu, bool pending) +{ + struct kvm_vcpu_events events; + + vcpu_events_get(vcpu, &events); + TEST_ASSERT_EQ(events.exception.serror_pending, pending); + + if (pending && vcpu_has_ras(vcpu)) + TEST_ASSERT_EQ(events.exception.serror_esr, EXPECTED_SERROR_ISS); +} + static void __vcpu_run_expect(struct kvm_vcpu *vcpu, unsigned int cmd) { struct ucall uc; @@ -134,6 +145,7 @@ static void test_mmio_abort(void) vcpu_inject_sea(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -162,6 +174,7 @@ static void test_mmio_nisv(void) TEST_ASSERT(_vcpu_run(vcpu), "Expected nonzero return code from KVM_RUN"); TEST_ASSERT_EQ(errno, ENOSYS); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -185,6 +198,7 @@ static void test_mmio_nisv_abort(void) vcpu_inject_sea(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -212,6 +226,7 @@ static void test_serror_masked(void) vcpu_inject_serror(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, true); kvm_vm_free(vm); } @@ -247,6 +262,7 @@ static void test_serror(void) vcpu_inject_serror(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -288,6 +304,7 @@ static void test_s1ptw_abort(void) *ptep |= bad_pa; vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -313,6 +330,7 @@ static void test_serror_emulated(void) vcpu_run_expect_sync(vcpu); vcpu_inject_serror(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } @@ -339,7 +357,7 @@ static void test_mmio_ease(void) pfr1 = vcpu_get_reg(vcpu, KVM_ARM64_SYS_REG(SYS_ID_AA64PFR1_EL1)); if (!SYS_FIELD_GET(ID_AA64PFR1_EL1, DF2, pfr1)) { pr_debug("Skipping %s\n", __func__); - return; + goto done; } /* @@ -356,6 +374,8 @@ static void test_mmio_ease(void) vcpu_inject_sea(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); +done: kvm_vm_free(vm); } @@ -394,6 +414,7 @@ static void test_serror_amo(void) vcpu_run_expect_sync(vcpu); vcpu_inject_serror(vcpu); vcpu_run_expect_done(vcpu); + vcpu_assert_serror_pending(vcpu, false); kvm_vm_free(vm); } diff --git a/tools/testing/selftests/kvm/arm64/hypercalls.c b/tools/testing/selftests/kvm/arm64/hypercalls.c index 5d96cdf382c4..391e2ba58b0b 100644 --- a/tools/testing/selftests/kvm/arm64/hypercalls.c +++ b/tools/testing/selftests/kvm/arm64/hypercalls.c @@ -1,12 +1,13 @@ // SPDX-License-Identifier: GPL-2.0-only -/* hypercalls: Check the ARM64's psuedo-firmware bitmap register interface. +/* hypercalls: Check the ARM64's pseudo-firmware bitmap register interface. * * The test validates the basic hypercall functionalities that are exposed - * via the psuedo-firmware bitmap register. This includes the registers' + * via the pseudo-firmware bitmap register. This includes the registers' * read/write behavior before and after the VM has started, and if the * hypercalls are properly masked or unmasked to the guest when disabled or - * enabled from the KVM userspace, respectively. + * enabled from the KVM userspace, respectively. It also checks the two + * SMCCC_ARCH_FEATURES queries whose answer SMCCC pins to SUCCESS. */ #include <errno.h> #include <linux/arm-smccc.h> @@ -97,6 +98,12 @@ static const struct test_hvc_info false_hvc_info[] = { TEST_HVC_INFO(ARM_SMCCC_HV_PV_TIME_FEATURES, ARM_SMCCC_TRNG_RND64), }; +/* SMCCC pins the answer to SUCCESS, whatever the bitmaps hold. */ +static const struct test_hvc_info always_hvc_info[] = { + TEST_HVC_INFO(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, ARM_SMCCC_VERSION_FUNC_ID), + TEST_HVC_INFO(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, ARM_SMCCC_ARCH_FEATURES_FUNC_ID), +}; + static void guest_test_hvc(const struct test_hvc_info *hc_info) { unsigned int i; @@ -128,6 +135,23 @@ static void guest_test_hvc(const struct test_hvc_info *hc_info) } } +static void guest_test_always_hvc(void) +{ + struct arm_smccc_res res; + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(always_hvc_info); i++) { + memset(&res, 0, sizeof(res)); + do_smccc(always_hvc_info[i].func_id, always_hvc_info[i].arg1, + 0, 0, 0, 0, 0, 0, &res); + + __GUEST_ASSERT(res.a0 == SMCCC_RET_SUCCESS, + "a0 = 0x%lx, func_id = 0x%x, arg1 = 0x%lx, stage = %u", + res.a0, always_hvc_info[i].func_id, + always_hvc_info[i].arg1, stage); + } +} + static void guest_code(void) { while (stage != TEST_STAGE_END) { @@ -137,6 +161,7 @@ static void guest_code(void) case TEST_STAGE_HVC_IFACE_FEAT_DISABLED: case TEST_STAGE_HVC_IFACE_FEAT_ENABLED: guest_test_hvc(hvc_info); + guest_test_always_hvc(); break; case TEST_STAGE_HVC_IFACE_FALSE_INFO: guest_test_hvc(false_hvc_info); diff --git a/tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c b/tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c new file mode 100644 index 000000000000..09c1db303856 --- /dev/null +++ b/tools/testing/selftests/kvm/arm64/nv_pre_fault_memory_test.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * nv_pre_fault_memory_test - Test KVM_PRE_FAULT_MEMORY on a vCPU whose + * last-run context is nested. + * + * The guest enters vEL2, sets up its EL2 translation configuration into the + * real EL1 registers then ERETs to vEL1 and exits to userspace so its vCPU + * last-run context is nested backed by a shadow stage 2 MMU. + * + * Assert that pre-faulting ignores that and targets the canonical stage-2 + * page tables only. + */ +#include "kvm_util.h" +#include "processor.h" +#include "test_util.h" +#include "ucall.h" + +#include <asm/sysreg.h> +#include <linux/sizes.h> + +#define TEST_MEM_SLOT 10 +#define TEST_MEM_SIZE SZ_2M +#define TEST_MEM_GPA SZ_1G + +static void guest_el1_code(void) +{ + u64 offset; + + GUEST_ASSERT_EQ(get_current_el(), 1); + + /* Exit to userspace with the vEL1 (nested) context live. */ + GUEST_SYNC(1); + + /* + * Touch the prefaulted range. vstage-2 is disabled, so the shadow + * stage-2 is a 1:1 view of the canonical IPA space. + */ + for (offset = 0; offset < TEST_MEM_SIZE; offset += SZ_4K) + READ_ONCE(*(u64 *)(TEST_MEM_GPA + offset)); + + GUEST_DONE(); +} + +static void guest_code(void) +{ + u64 sp; + + GUEST_ASSERT_EQ(get_current_el(), 2); + + /* + * Mirror the EL2 translation regime into the real EL1 registers so + * that vEL1 runs on the test's stage-1 page tables. With E2H=1, the + * _EL1 accessors read the EL2 registers, and the _EL12 accessors + * write the real EL1 registers. + */ + write_sysreg_s(read_sysreg(sctlr_el1), SYS_SCTLR_EL12); + write_sysreg_s(read_sysreg(tcr_el1), SYS_TCR_EL12); + write_sysreg_s(read_sysreg(ttbr0_el1), SYS_TTBR0_EL12); + write_sysreg_s(read_sysreg(mair_el1), SYS_MAIR_EL12); + write_sysreg_s(read_sysreg(cpacr_el1), SYS_CPACR_EL12); + + /* Run vEL1 on the same stack. */ + asm volatile("mov %0, sp" : "=r"(sp)); + write_sysreg(sp, sp_el1); + + /* + * Drop TGE so that vEL1 is a nested context rather than host EL0. + * KVM backs it with a shadow stage-2 MMU even though vstage-2 is + * disabled (HCR_EL2.VM=0). + */ + write_sysreg(read_sysreg(hcr_el2) & ~HCR_EL2_TGE, hcr_el2); + isb(); + + write_sysreg(PSR_MODE_EL1h | PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | + PSR_D_BIT, spsr_el2); + write_sysreg((u64)guest_el1_code, elr_el2); + asm volatile("eret"); + + GUEST_ASSERT(false); +} + +static void pre_fault(struct kvm_vcpu *vcpu, u64 gpa, u64 size) +{ + struct kvm_pre_fault_memory range = { + .gpa = gpa, + .size = size, + }; + int ret; + + do { + ret = __vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range); + } while ((!ret && range.size) || + (ret < 0 && (errno == EINTR || errno == EAGAIN))); + + TEST_ASSERT(!ret, "KVM_PRE_FAULT_MEMORY failed, ret: %d errno: %d", + ret, errno); + TEST_ASSERT_EQ(range.size, 0); +} + +int main(void) +{ + struct kvm_vcpu_init init; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + u64 npages; + + TEST_REQUIRE(test_supports_el2()); + TEST_REQUIRE(kvm_check_cap(KVM_CAP_PRE_FAULT_MEMORY)); + + vm = vm_create(1); + + kvm_get_default_vcpu_target(vm, &init); + init.features[0] |= BIT(KVM_ARM_VCPU_HAS_EL2); + vcpu = aarch64_vcpu_add(vm, 0, &init, guest_code); + kvm_arch_vm_finalize_vcpus(vm); + + npages = TEST_MEM_SIZE / vm->page_size; + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, TEST_MEM_GPA, + TEST_MEM_SLOT, npages, 0); + virt_map(vm, TEST_MEM_GPA, TEST_MEM_GPA, npages); + + /* Run the guest until it has ERET'd from vEL2 to vEL1. */ + vcpu_run(vcpu); + switch (get_ucall(vcpu, &uc)) { + case UCALL_SYNC: + TEST_ASSERT_EQ(uc.args[1], 1); + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + break; + default: + TEST_FAIL("Unhandled ucall: %ld", uc.cmd); + } + + /* + * The vCPU's last-run context is vEL1, so its hw_mmu is a shadow + * stage-2 MMU. + * + * Pre-faulting must ignore that and populate the canonical stage-2. + */ + pre_fault(vcpu, TEST_MEM_GPA, TEST_MEM_SIZE); + + /* Resume at vEL1 and touch the prefaulted range. */ + vcpu_run(vcpu); + switch (get_ucall(vcpu, &uc)) { + case UCALL_DONE: + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + break; + default: + TEST_FAIL("Unhandled ucall: %ld", uc.cmd); + } + + kvm_vm_free(vm); + return 0; +} diff --git a/tools/testing/selftests/kvm/arm64/sea_to_user.c b/tools/testing/selftests/kvm/arm64/sea_to_user.c index e96d8982c28b..d4af6e0eed28 100644 --- a/tools/testing/selftests/kvm/arm64/sea_to_user.c +++ b/tools/testing/selftests/kvm/arm64/sea_to_user.c @@ -81,25 +81,46 @@ static u64 translate_hva_to_hpa(unsigned long hva) static void write_einj_entry(const char *einj_path, u64 val) { - char cmd[256] = {0}; - FILE *cmdfile = NULL; + char buf[32]; + int fd, len, ret; - sprintf(cmd, "echo %#lx > %s", val, einj_path); - cmdfile = popen(cmd, "r"); + fd = open(einj_path, O_WRONLY); + if (fd < 0) + ksft_exit_fail_perror(einj_path); - if (pclose(cmdfile) == 0) - ksft_print_msg("echo %#lx > %s - done\n", val, einj_path); - else - ksft_exit_fail_perror("Failed to write EINJ entry"); + len = snprintf(buf, sizeof(buf), "%#lx\n", val); + ret = write(fd, buf, len); + if (ret != len) + ksft_exit_fail_perror(einj_path); + + close(fd); + ksft_print_msg("%#lx > %s - done\n", val, einj_path); } static void inject_uer(u64 hpa) { - if (access("/sys/firmware/acpi/tables/EINJ", R_OK) == -1) - ksft_test_result_skip("EINJ table no available in firmware"); + int fd; - if (access(EINJ_ETYPE, R_OK | W_OK) == -1) - ksft_test_result_skip("EINJ module probably not loaded?"); + /* + * EINJ is exposed only through debugfs, and opening it tells us why it + * is unusable far more reliably than access()-ing the ACPI EINJ table: + * reading that table needs CAP_SYS_ADMIN, so access() gives a false + * negative for unprivileged runs. Open the injection interface and let + * errno say whether the test cannot run (skip) or genuinely failed. + */ + fd = open(EINJ_ETYPE, O_WRONLY); + if (fd < 0) { + switch (errno) { + case ENOENT: + ksft_exit_skip("need CONFIG_ACPI_APEI_EINJ and firmware EINJ support\n"); + case EACCES: + case EPERM: + ksft_exit_skip("EINJ requires running as root\n"); + default: + ksft_exit_fail_perror(EINJ_ETYPE); + } + } + close(fd); write_einj_entry(EINJ_ETYPE, ERROR_TYPE_MEMORY_UER); write_einj_entry(EINJ_FLAGS, MASK_MEMORY_UER); @@ -194,13 +215,33 @@ static void run_vm(struct kvm_vm *vm, struct kvm_vcpu *vcpu) ksft_print_msg("Dump kvm_run info about KVM_EXIT_%s\n", exit_reason_str(run->exit_reason)); + + /* + * The guest's read of the injected location is expected to trap to KVM + * as an SEA. If it does not, the injected error was never placed as + * consumable poison: some firmware honours EINJ's notrigger request by + * arming the poison only as part of the (now skipped) trigger step, so + * nothing is left in memory for the guest to consume. The guest then + * reads back the sentinel value and reports it via GUEST_FAIL, which + * arm64 delivers as a ucall over MMIO (hence a KVM_EXIT_MMIO here). + * Treat that as "this platform cannot host the test" and skip, matching + * the requirement documented at the top of this file, rather than + * failing on a hardware/firmware limitation the test cannot control. + */ + if (run->exit_reason != KVM_EXIT_ARM_SEA && + get_ucall(vcpu, &uc) == UCALL_ABORT) { + ksft_print_msg("Guest consumed no SEA: %s", uc.buffer); + ksft_exit_skip("EINJ notrigger placed no consumable poison on this platform\n"); + } + + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_ARM_SEA); + + /* arm_sea holds valid data only for a KVM_EXIT_ARM_SEA exit. */ ksft_print_msg("kvm_run.arm_sea: esr=%#llx, flags=%#llx\n", run->arm_sea.esr, run->arm_sea.flags); ksft_print_msg("kvm_run.arm_sea: gva=%#llx, gpa=%#llx\n", run->arm_sea.gva, run->arm_sea.gpa); - TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_ARM_SEA); - esr = run->arm_sea.esr; TEST_ASSERT_EQ(ESR_ELx_EC(esr), ESR_ELx_EC_DABT_LOW); TEST_ASSERT_EQ(esr & ESR_ELx_FSC_TYPE, ESR_ELx_FSC_EXTABT); @@ -260,6 +301,12 @@ static struct kvm_vm *vm_create_with_sea_handler(struct kvm_vcpu **vcpu) alignment = max(backing_page_size, guest_page_size); num_guest_pages = VM_MEM_SIZE / guest_page_size; + /* + * The region is backed by 1GB hugepages; skip gracefully rather than + * failing with mmap() -ENOMEM if the host has none reserved. + */ + TEST_REQUIRE(get_free_hugepages(backing_page_size) >= VM_MEM_SIZE); + vm = __vm_create_with_one_vcpu(vcpu, num_guest_pages, guest_code); vm_init_descriptor_tables(vm); vcpu_init_descriptor_tables(*vcpu); diff --git a/tools/testing/selftests/kvm/arm64/set_id_regs.c b/tools/testing/selftests/kvm/arm64/set_id_regs.c index 7429a1055df5..10849d21c0dd 100644 --- a/tools/testing/selftests/kvm/arm64/set_id_regs.c +++ b/tools/testing/selftests/kvm/arm64/set_id_regs.c @@ -13,6 +13,7 @@ #include "kvm_util.h" #include "processor.h" #include "test_util.h" +#include "vgic.h" #include <linux/bitfield.h> enum ftr_type { @@ -803,6 +804,107 @@ static void test_reset_preserves_id_regs(struct kvm_vcpu *vcpu) ksft_test_result_pass("%s\n", __func__); } +/* + * ID registers must stay immutable even when a vCPU's first KVM_RUN fails + * after finalization but before KVM_ARCH_FLAG_HAS_RAN_ONCE is set. + */ +static void test_idreg_frozen_after_failed_run(void) +{ + static const u32 imp_id_regs[] = { + SYS_MIDR_EL1, + SYS_REVIDR_EL1, + SYS_AIDR_EL1, + }; + struct kvm_vcpu_init init; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + int r; + + if (!kvm_has_cap(KVM_CAP_ARM_PMU_V3)) { + ksft_print_msg("PMUv3 unsupported, cannot fail the first run\n"); + ksft_test_result_skip("%s\n", __func__); + return; + } + + /* Skip the default vGIC so the KVM_CREATE_DEVICE gate is reachable. */ + test_disable_default_vgic(); + + vm = vm_create(1); + vm_enable_cap(vm, KVM_CAP_ARM_WRITABLE_IMP_ID_REGS, 0); + kvm_get_default_vcpu_target(vm, &init); + init.features[0] |= (1 << KVM_ARM_VCPU_PMU_V3); + vcpu = aarch64_vcpu_add(vm, 0, &init, guest_code); + kvm_arch_vm_finalize_vcpus(vm); + + /* + * A PMUv3 vCPU left without PMU init is rejected by + * kvm_arm_pmu_v3_enable(), which runs after sysreg finalization. + */ + r = _vcpu_run(vcpu); + TEST_ASSERT(r < 0 && errno == EINVAL, + "first KVM_RUN should fail post-finalization: r=%d errno=%d", + r, errno); + + /* + * Feature ID registers: use values that would have been accepted before + * finalization, so that a rejection means the registers are final + * rather than the value being invalid. + */ + for (int i = 0; i < ARRAY_SIZE(test_regs); i++) { + const struct reg_ftr_bits *ftr_bits = test_regs[i].ftr_bits; + u64 reg = KVM_ARM64_SYS_REG(test_regs[i].reg); + u64 val = vcpu_get_reg(vcpu, reg); + + for (int j = 0; ftr_bits[j].type != FTR_END; j++) { + u64 ftr = (val & ftr_bits[j].mask) >> ftr_bits[j].shift; + u64 safe = get_safe_value(&ftr_bits[j], ftr); + u64 new_val; + + if (safe == ftr) + continue; + + new_val = (val & ~ftr_bits[j].mask) | + (safe << ftr_bits[j].shift); + + r = __vcpu_set_reg(vcpu, reg, new_val); + TEST_ASSERT(r < 0 && errno == EBUSY, + "%s write after failed first run: r=%d errno=%d", + ftr_bits[j].name, r, errno); + TEST_ASSERT_EQ(vcpu_get_reg(vcpu, reg), val); + } + + /* A write matching the finalized value is still accepted. */ + vcpu_set_reg(vcpu, reg, val); + } + + /* + * The VM-wide implementation ID registers are gated separately. Bit 0 + * is within the writable mask of all three, so flipping it is a change + * KVM would otherwise accept. + */ + for (int i = 0; i < ARRAY_SIZE(imp_id_regs); i++) { + u64 reg = KVM_ARM64_SYS_REG(imp_id_regs[i]); + u64 val = vcpu_get_reg(vcpu, reg); + + r = __vcpu_set_reg(vcpu, reg, val ^ 1); + TEST_ASSERT(r < 0 && errno == EBUSY, + "implementation ID reg write after failed first run: r=%d errno=%d", + r, errno); + TEST_ASSERT_EQ(vcpu_get_reg(vcpu, reg), val); + } + + /* Creating an in-kernel irqchip would change the ID registers too. */ + if (kvm_supports_vgic_v3()) { + r = __kvm_create_device(vm, KVM_DEV_TYPE_ARM_VGIC_V3); + TEST_ASSERT(r < 0 && errno == EBUSY, + "vGIC creation after failed first run: r=%d errno=%d", + r, errno); + } + + kvm_vm_free(vm); + ksft_test_result_pass("%s\n", __func__); +} + int main(void) { struct kvm_vcpu *vcpu; @@ -828,7 +930,7 @@ int main(void) ksft_print_header(); - test_cnt = 3 + MPAM_IDREG_TEST + MTE_IDREG_TEST; + test_cnt = 4 + MPAM_IDREG_TEST + MTE_IDREG_TEST; for (i = 0; i < ARRAY_SIZE(test_regs); i++) for (j = 0; test_regs[i].ftr_bits[j].type != FTR_END; j++) test_cnt++; @@ -847,5 +949,7 @@ int main(void) kvm_vm_free(vm); + test_idreg_frozen_after_failed_run(); + ksft_finished(); } diff --git a/tools/testing/selftests/kvm/arm64/vgic_init.c b/tools/testing/selftests/kvm/arm64/vgic_init.c index 47e34b43afb2..5a30f3cb039b 100644 --- a/tools/testing/selftests/kvm/arm64/vgic_init.c +++ b/tools/testing/selftests/kvm/arm64/vgic_init.c @@ -5,6 +5,7 @@ * Copyright (C) 2020, Red Hat, Inc. */ #include <linux/kernel.h> +#include <linux/sizes.h> #include <sys/syscall.h> #include <asm/kvm.h> #include <asm/kvm_para.h> @@ -13,12 +14,21 @@ #include "test_util.h" #include "kvm_util.h" +#include "gic.h" #include "processor.h" #include "vgic.h" #include "gic_v3.h" #define NR_VCPUS 4 +#define REDIST_RETRY_REGION0_BASE GICR_BASE_GPA +#define REDIST_RETRY_REGION1_BASE \ + (REDIST_RETRY_REGION0_BASE + 2 * KVM_VGIC_V3_REDIST_SIZE) +#define REDIST_RETRY_DIST_BASE \ + (REDIST_RETRY_REGION1_BASE + KVM_VGIC_V3_REDIST_SIZE) +#define REDIST_RETRY_REGION2_BASE \ + (REDIST_RETRY_DIST_BASE + KVM_VGIC_V3_DIST_SIZE) + #define REG_OFFSET(vcpu, offset) (((u64)vcpu << 32) | offset) #define VGIC_DEV_IS_V2(_d) ((_d) == KVM_DEV_TYPE_ARM_VGIC_V2) @@ -65,6 +75,23 @@ static void guest_code(void) GUEST_DONE(); } +static void guest_check_redist_retry(void) +{ + unsigned int i; + + /* The first three redistributors span adjacent regions 0 and 1. */ + for (i = 0; i < NR_VCPUS; i++) { + u64 base = i < 3 ? REDIST_RETRY_REGION0_BASE + + i * KVM_VGIC_V3_REDIST_SIZE : + REDIST_RETRY_REGION2_BASE; + u64 typer = readq((void *)(unsigned long)(base + GICR_TYPER)); + + GUEST_ASSERT_EQ(GICR_TYPER_CPU_NUMBER(typer), i); + } + + GUEST_DONE(); +} + /* we don't want to assert on run execution, hence that helper */ static int run_vcpu(struct kvm_vcpu *vcpu) { @@ -73,6 +100,7 @@ static int run_vcpu(struct kvm_vcpu *vcpu) static struct vm_gic vm_gic_create_with_vcpus(u32 gic_dev_type, u32 nr_vcpus, + void *guest_code, struct kvm_vcpu *vcpus[]) { struct vm_gic v; @@ -338,7 +366,7 @@ static void test_vgic_then_vcpus(u32 gic_dev_type) struct vm_gic v; int ret, i; - v = vm_gic_create_with_vcpus(gic_dev_type, 1, vcpus); + v = vm_gic_create_with_vcpus(gic_dev_type, 1, guest_code, vcpus); subtest_dist_rdist(&v); @@ -359,7 +387,8 @@ static void test_vcpus_then_vgic(u32 gic_dev_type) struct vm_gic v; int ret; - v = vm_gic_create_with_vcpus(gic_dev_type, NR_VCPUS, vcpus); + v = vm_gic_create_with_vcpus(gic_dev_type, NR_VCPUS, guest_code, + vcpus); subtest_dist_rdist(&v); @@ -411,7 +440,8 @@ static void test_v3_new_redist_regions(void) u64 addr; int ret; - v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, vcpus); + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_code, vcpus); subtest_v3_redist_regions(&v); kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); @@ -422,7 +452,8 @@ static void test_v3_new_redist_regions(void) /* step2 */ - v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, vcpus); + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_code, vcpus); subtest_v3_redist_regions(&v); addr = REDIST_REGION_ATTR_ADDR(1, 0x280000, 0, 2); @@ -436,7 +467,8 @@ static void test_v3_new_redist_regions(void) /* step 3 */ - v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, vcpus); + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_code, vcpus); subtest_v3_redist_regions(&v); ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, @@ -457,6 +489,70 @@ static void test_v3_new_redist_regions(void) vm_gic_destroy(&v); } +static void test_v3_redist_region_retry(void) +{ + struct kvm_vcpu *vcpus[NR_VCPUS]; + struct vm_gic v; + struct ucall uc; + u64 addr; + int ret; + + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_check_redist_retry, vcpus); + + addr = REDIST_REGION_ATTR_ADDR(2, REDIST_RETRY_REGION0_BASE, 0, 0); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, &addr); + + addr = REDIST_REGION_ATTR_ADDR(1, REDIST_RETRY_REGION1_BASE, 0, 1); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, &addr); + + addr = REDIST_RETRY_DIST_BASE; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V3_ADDR_TYPE_DIST, &addr); + + addr = REDIST_REGION_ATTR_ADDR(1, REDIST_RETRY_DIST_BASE, 0, 2); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, + &addr); + TEST_ASSERT(ret && errno == EINVAL, + "register redist region colliding with dist"); + + addr = REDIST_REGION_ATTR_ADDR(1, REDIST_RETRY_REGION2_BASE, 0, 2); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, &addr); + + virt_map(v.vm, REDIST_RETRY_REGION0_BASE, REDIST_RETRY_REGION0_BASE, + vm_calc_num_guest_pages(v.vm->mode, + 3 * KVM_VGIC_V3_REDIST_SIZE)); + virt_map(v.vm, REDIST_RETRY_REGION2_BASE, REDIST_RETRY_REGION2_BASE, + vm_calc_num_guest_pages(v.vm->mode, + KVM_VGIC_V3_REDIST_SIZE)); + + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + vcpu_run(vcpus[0]); + switch (get_ucall(vcpus[0], &uc)) { + case UCALL_DONE: + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + break; + case UCALL_NONE: + if (vcpus[0]->run->exit_reason == KVM_EXIT_MMIO) + TEST_FAIL("Unexpected MMIO exit at 0x%llx", + vcpus[0]->run->mmio.phys_addr); + fallthrough; + default: + TEST_FAIL("Unexpected ucall %lu, exit_reason %u", + uc.cmd, vcpus[0]->run->exit_reason); + } + + vm_gic_destroy(&v); +} + static void test_v3_typer_accesses(void) { struct vm_gic v; @@ -608,7 +704,8 @@ static void test_v3_redist_ipa_range_check_at_vcpu_run(void) int ret, i; u64 addr; - v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, 1, vcpus); + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, 1, guest_code, + vcpus); /* Set space for 3 redists, we have 1 vcpu, so this succeeds. */ addr = max_phys_size - (3 * 2 * 0x10000); @@ -641,7 +738,8 @@ static void test_v3_its_region(void) u64 addr; int its_fd, ret; - v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, vcpus); + v = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_code, vcpus); its_fd = kvm_create_device(v.vm, KVM_DEV_TYPE_ARM_VGIC_ITS); addr = 0x401000; @@ -684,7 +782,8 @@ static void test_v3_nassgicap(void) u32 typer2; int ret; - vm = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, vcpus); + vm = vm_gic_create_with_vcpus(KVM_DEV_TYPE_ARM_VGIC_V3, NR_VCPUS, + guest_code, vcpus); kvm_device_attr_get(vm.gic_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, GICD_TYPER2, &typer2); has_nassgicap = typer2 & GICD_TYPER2_nASSGIcap; @@ -978,6 +1077,7 @@ void run_tests(u32 gic_dev_type) if (VGIC_DEV_IS_V3(gic_dev_type)) { test_v3_new_redist_regions(); + test_v3_redist_region_retry(); test_v3_typer_accesses(); test_v3_last_bit_redist_regions(); test_v3_last_bit_single_rdist(); diff --git a/tools/testing/selftests/kvm/arm64/vgic_v5.c b/tools/testing/selftests/kvm/arm64/vgic_v5.c index 96cfd6bb32f6..d5df742b2300 100644 --- a/tools/testing/selftests/kvm/arm64/vgic_v5.c +++ b/tools/testing/selftests/kvm/arm64/vgic_v5.c @@ -13,6 +13,32 @@ #include "vgic.h" #define NR_VCPUS 1 +#define VGIC_V5_LPI_NR_VCPUS 2 +#define VGIC_V5_SPARSE_NR_VCPUS 2 +#define VGIC_V5_SPARSE_VCPU0_ID 17 +#define VGIC_V5_SPARSE_VCPU1_ID 3 +#define VGIC_V5_LPI_MEMSLOT 1 +#define VGIC_V5_NR_PRIVATE_IRQS 64 +#define VGIC_V5_DEFAULT_NR_SPIS 32 +#define VGIC_V5_MAX_NR_SPIS BIT(10) +#define VGIC_V5_IRS_SIZE 0x20000 +#define VGIC_V5_IRS_WAIT_RETRIES 1000000 +#define VGIC_V5_LPI_IST_BASE_GPA (GICV5_IRS_CONFIG_BASE_GPA + VGIC_V5_IRS_SIZE) +#define VGIC_V5_LPI_IST_SIZE 0x10000 +#define LPI_TEST_TO_VPE1 0 +#define LPI_TEST_TO_VPE0 1 +#define RESTORE_TEST_LPI_PENDING 0 +#define RESTORE_TEST_LPI_ENABLED 1 +#define RESTORE_TEST_SPI_PRIO_HIGH 0 +#define RESTORE_TEST_SPI_PRIO_LOW 1 +#define RESTORE_TEST_SPI_PENDING_DISABLED 2 +#define RESTORE_TEST_SPI_ENABLED 3 +#define VGIC_V5_IST_PRIO_HIGH 0x08 +#define VGIC_V5_IST_PRIO_LOW 0x10 + +static u64 max_phys_size; + +#define VGIC_V5_TEST_IST_BASER_GPA 0x10000000ULL struct vm_gic { struct kvm_vm *vm; @@ -24,8 +50,200 @@ struct vm_gic { #define GUEST_CMD_IRQ_DIEOI 11 #define GUEST_CMD_IS_AWAKE 12 #define GUEST_CMD_IS_READY 13 +#define GUEST_CMD_LPI_SENT 14 +#define GUEST_CMD_LPI_REPLIED 15 +#define GUEST_CMD_IST_SOURCE_READY 16 +#define GUEST_CMD_IST_SPI_INJECT_READY 17 +#define GUEST_CMD_IST_LPI_INJECT_READY 18 + +static struct kvm_vgic_v5_ist vgic_v5_ist_attr(void *spi_ist, size_t spi_size, + void *lpi_ist, size_t lpi_size) +{ + return (struct kvm_vgic_v5_ist) { + .spi_ist_addr = (uintptr_t)spi_ist, + .spi_ist_size = spi_size, + .lpi_ist_addr = (uintptr_t)lpi_ist, + .lpi_ist_size = lpi_size, + }; +} + +static u32 spi_line_expected; +static bool spi_line_level_sensitive; +static bool lpi_ist_ready; +static u32 ist_restore_irq_count[VGIC_V5_LPI_NR_VCPUS]; + +static u64 gicv5_hwirq(u32 type, u32 intid) +{ + return FIELD_PREP(GICV5_HWIRQ_TYPE, type) | + FIELD_PREP(GICV5_HWIRQ_ID, intid); +} + +static u64 gicv5_lpi_hwirq(u32 lpi) +{ + return gicv5_hwirq(GICV5_HWIRQ_TYPE_LPI, lpi); +} + +static u64 gicv5_spi_hwirq(u32 spi) +{ + return gicv5_hwirq(GICV5_HWIRQ_TYPE_SPI, spi); +} + +static void gicv5_setup_and_enable_hwirq(u64 hwirq, u32 target_vpe) +{ + u64 val; + + val = hwirq | FIELD_PREP(GICV5_GIC_CDPRI_PRIORITY_MASK, + GICV5_IRQ_DEFAULT_PRI); + gic_insn(val, CDPRI); + + val = hwirq | FIELD_PREP(GICV5_GIC_CDAFF_IAFFID_MASK, target_vpe); + gic_insn(val, CDAFF); + + gic_insn(hwirq, CDEN); +} + +static void gicv5_set_hwirq_priority(u64 hwirq, u32 priority) +{ + u64 val; + + val = hwirq | FIELD_PREP(GICV5_GIC_CDPRI_PRIORITY_MASK, priority); + gic_insn(val, CDPRI); +} + +static void gicv5_set_hwirq_affinity(u64 hwirq, u32 target_vpe) +{ + u64 val; + + val = hwirq | FIELD_PREP(GICV5_GIC_CDAFF_IAFFID_MASK, target_vpe); + gic_insn(val, CDAFF); +} + +static void gicv5_enable_hwirq(u64 hwirq) +{ + gic_insn(hwirq, CDEN); +} + +static void gicv5_enable_spi(u32 spi, u32 target_vpe) +{ + gicv5_setup_and_enable_hwirq(gicv5_spi_hwirq(spi), target_vpe); +} + +static void gicv5_enable_lpi(u32 lpi, u32 target_vpe) +{ + gicv5_setup_and_enable_hwirq(gicv5_lpi_hwirq(lpi), target_vpe); +} + +static void gicv5_send_lpi(u32 lpi) +{ + u64 hwirq = gicv5_lpi_hwirq(lpi); + + gic_insn(hwirq | GICV5_GIC_CDPEND_PENDING_MASK, CDPEND); +} + +static void gicv5_wait_for_irs_idle(u32 reg, u32 idle) +{ + int i; + + for (i = 0; i < VGIC_V5_IRS_WAIT_RETRIES; i++) { + if (readl(GICV5_IRS_CONFIG_BASE_GVA + reg) & idle) + return; + + cpu_relax(); + } + + GUEST_FAIL("IRS operation did not become idle"); +} + +static void gicv5_configure_spi(u32 spi, bool level) +{ + u32 val; + + val = FIELD_PREP(GICV5_IRS_SPI_SELR_ID, spi); + writel(val, GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_SPI_SELR); + gicv5_wait_for_irs_idle(GICV5_IRS_SPI_STATUSR, + GICV5_IRS_SPI_STATUSR_IDLE); + + val = level ? GICV5_IRS_SPI_CFGR_TM : 0; + writel(val, GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_SPI_CFGR); + gicv5_wait_for_irs_idle(GICV5_IRS_SPI_STATUSR, + GICV5_IRS_SPI_STATUSR_IDLE); +} + +static u32 gicv5_lpi_istsz(u32 idr2, u32 lpi_id_bits) +{ + if (!(idr2 & GICV5_IRS_IDR2_ISTMD)) + return GICV5_IRS_IST_CFGR_ISTSZ_4; + + if (lpi_id_bits >= FIELD_GET(GICV5_IRS_IDR2_ISTMD_SZ, idr2)) + return GICV5_IRS_IST_CFGR_ISTSZ_16; + + return GICV5_IRS_IST_CFGR_ISTSZ_8; +} + +static void gicv5_configure_lpi_ist(void) +{ + u32 idr2, min_lpi_id_bits, lpi_id_bits, id_bits, istsz; + u64 val; + + idr2 = readl(GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_IDR2); + GUEST_ASSERT(idr2 & GICV5_IRS_IDR2_LPI); + + min_lpi_id_bits = FIELD_GET(GICV5_IRS_IDR2_MIN_LPI_ID_BITS, idr2); + id_bits = FIELD_GET(GICV5_IRS_IDR2_ID_BITS, idr2); + /* Default to 32 LPIs, unless the min requirement is higher */ + lpi_id_bits = max(5U, min_lpi_id_bits); + GUEST_ASSERT(lpi_id_bits <= id_bits); + + istsz = gicv5_lpi_istsz(idr2, lpi_id_bits); + /* + * We allocate a fixed-size IST buffer on the host to avoid dynamic + * allocation from the guest. Make sure it fits the linear IST described + * by IDR2 before programming IRS_IST_BASER. + */ + GUEST_ASSERT((BIT_ULL(lpi_id_bits) << (istsz + 2)) <= VGIC_V5_LPI_IST_SIZE); + + val = FIELD_PREP(GICV5_IRS_IST_CFGR_LPI_ID_BITS, lpi_id_bits) | + FIELD_PREP(GICV5_IRS_IST_CFGR_ISTSZ, istsz); + writel(val, GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_IST_CFGR); + + val = FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, + VGIC_V5_LPI_IST_BASE_GPA >> GICV5_IRS_IST_BASER_ADDR_SHIFT); + val |= GICV5_IRS_IST_BASER_VALID; + writeq(val, GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_IST_BASER); + + GUEST_ASSERT_EQ(readl(GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_IST_STATUSR), + GICV5_IRS_IST_STATUSR_IDLE); + + /* The address is immutable while the BASER is valid. */ + writeq(val ^ FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, 1), + GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_IST_BASER); + GUEST_ASSERT_EQ(readq(GICV5_IRS_CONFIG_BASE_GVA + + GICV5_IRS_IST_BASER), val); +} + +static void gicv5_enable_irs(void) +{ + writel(GICV5_IRS_CR0_IRSEN, + GICV5_IRS_CONFIG_BASE_GVA + GICV5_IRS_CR0); + gicv5_wait_for_irs_idle(GICV5_IRS_CR0, GICV5_IRS_CR0_IDLE); +} + +static void gicv5_configure_test_lpis(void) +{ + gicv5_enable_lpi(LPI_TEST_TO_VPE1, 1); + gicv5_enable_lpi(LPI_TEST_TO_VPE0, 0); +} + +static void gicv5_set_pcr(u32 priority) +{ + u64 pcr; -static void guest_irq_handler(struct ex_regs *regs) + pcr = FIELD_PREP(ICC_PCR_EL1_PRIORITY, priority); + write_sysreg_s(pcr, SYS_ICC_PCR_EL1); + isb(); +} + +static void guest_ppi_irq_handler(struct ex_regs *regs) { bool valid; u32 hwirq; @@ -78,9 +296,284 @@ static void guest_code(void) GUEST_SYNC(GUEST_CMD_IS_READY); /* Loop forever waiting for interrupts */ - while (1); + for (;;) + cpu_relax(); +} + +static void guest_iaffid_code(void) +{ + GUEST_SYNC(FIELD_GET(ICC_IAFFIDR_EL1_IAFFID, + read_sysreg_s(SYS_ICC_IAFFIDR_EL1))); + GUEST_DONE(); +} + +static void guest_spi_irq_handler(struct ex_regs *regs) +{ + bool valid; + u32 hwirq; + u64 ia; + + ia = gicr_insn(CDIA); + valid = GICV5_GICR_CDIA_VALID(ia); + + if (!valid) + return; + + gsb_ack(); + isb(); + + hwirq = FIELD_GET(GICV5_GICR_CDIA_INTID, ia); + + GUEST_ASSERT_EQ(hwirq, gicv5_spi_hwirq(READ_ONCE(spi_line_expected))); + + gic_insn(hwirq, CDDI); + gic_insn(0, CDEOI); + + GUEST_DONE(); +} + +static void guest_spi_line_code(void) +{ + local_irq_disable(); + + gicv5_enable_irs(); + gicv5_cpu_enable_interrupts(); + gicv5_configure_spi(READ_ONCE(spi_line_expected), + READ_ONCE(spi_line_level_sensitive)); + gicv5_enable_spi(READ_ONCE(spi_line_expected), 0); + + local_irq_enable(); + + GUEST_SYNC(GUEST_CMD_IS_READY); + + while (1) + wfi(); +} + +static void guest_lpi_irq_handler(struct ex_regs *regs) +{ + u32 vcpu_id = guest_get_vcpuid(); + u32 expected_lpi = vcpu_id ? LPI_TEST_TO_VPE1 : LPI_TEST_TO_VPE0; + u64 expected_hwirq = gicv5_lpi_hwirq(expected_lpi); + u32 hwirq; + u64 ia; + + ia = gicr_insn(CDIA); + if (!GICV5_GICR_CDIA_VALID(ia)) + return; + + gsb_ack(); + isb(); + + hwirq = FIELD_GET(GICV5_GICR_CDIA_INTID, ia); + GUEST_ASSERT_EQ(hwirq, expected_hwirq); + + gic_insn(hwirq, CDDI); + gic_insn(0, CDEOI); + + if (vcpu_id) { + gicv5_send_lpi(LPI_TEST_TO_VPE0); + GUEST_SYNC(GUEST_CMD_LPI_REPLIED); + while (1) + wfi(); + } + + GUEST_DONE(); +} + +static void guest_unexpected_irq_handler(struct ex_regs *regs) +{ + GUEST_FAIL("Unexpected IRQ"); +} + +static void guest_lpi_code(void) +{ + u32 vcpu_id = guest_get_vcpuid(); + + local_irq_disable(); + + if (!vcpu_id) { + gicv5_enable_irs(); + gicv5_configure_lpi_ist(); + gicv5_configure_test_lpis(); + WRITE_ONCE(lpi_ist_ready, true); + } + + /* Go bang if we run VCPU1 before VCPU0 */ + GUEST_ASSERT(READ_ONCE(lpi_ist_ready)); + + gicv5_cpu_enable_interrupts(); + local_irq_enable(); + + GUEST_SYNC(GUEST_CMD_IS_READY); + + if (!vcpu_id) { + gicv5_send_lpi(LPI_TEST_TO_VPE1); + GUEST_SYNC(GUEST_CMD_LPI_SENT); + } + + while (1) + wfi(); } +static void restore_test_setup_hwirq(u64 hwirq, u32 target_vpe, + u32 priority, bool enable) +{ + gicv5_set_hwirq_priority(hwirq, priority); + gicv5_set_hwirq_affinity(hwirq, target_vpe); + + if (enable) + gicv5_enable_hwirq(hwirq); +} + +/* + * The source VM configures SPI and LPI state for the IST save/restore test. + * It leaves interrupts disabled, allowing userspace to save the state before + * the restore VM enables interrupts and checks delivery. + */ +static void guest_ist_source_code(void) +{ + local_irq_disable(); + + gicv5_enable_irs(); + gicv5_configure_lpi_ist(); + + restore_test_setup_hwirq(gicv5_lpi_hwirq(RESTORE_TEST_LPI_PENDING), + 1, VGIC_V5_IST_PRIO_HIGH, true); + restore_test_setup_hwirq(gicv5_lpi_hwirq(RESTORE_TEST_LPI_ENABLED), + 1, VGIC_V5_IST_PRIO_HIGH, true); + restore_test_setup_hwirq(gicv5_spi_hwirq(RESTORE_TEST_SPI_PRIO_HIGH), + 0, VGIC_V5_IST_PRIO_HIGH, true); + restore_test_setup_hwirq(gicv5_spi_hwirq(RESTORE_TEST_SPI_PRIO_LOW), + 0, VGIC_V5_IST_PRIO_LOW, true); + restore_test_setup_hwirq(gicv5_spi_hwirq(RESTORE_TEST_SPI_ENABLED), + 0, VGIC_V5_IST_PRIO_HIGH, true); + restore_test_setup_hwirq(gicv5_spi_hwirq(RESTORE_TEST_SPI_PENDING_DISABLED), + 0, VGIC_V5_IST_PRIO_HIGH, false); + + gicv5_send_lpi(RESTORE_TEST_LPI_PENDING); + + GUEST_SYNC(GUEST_CMD_IST_SOURCE_READY); + + while (1) + wfi(); +} + +/* + * This is the restore VM for the IST save/restore test. The IRS regs and ISTs + * have been restored before this code runs. Based on which VPE is running, a + * different set of interrupts is expected. We check that we get what we + * expected before terminating the test. + * + * The expected interrupts for VPE0 are: + * - High-priority SPI, pending and enabled before save. + * - Low-priority SPI, pending and enabled before save. This is only delivered + * after the guest lowers the masked priority via ICC_PCR_EL1. + * - SPI that was pending but disabled before save. This is only delivered + * after the guest enables it after restore. + * - SPI that was enabled but not pending before save. Userspace injects this + * after restore to check that the enable state and affinity were restored. + * + * The expected interrupts for VPE1 are: + * - LPI that was pending and enabled before save. + * - LPI that was enabled but not pending before save. The guest injects this + * after restore to check that the enable state and affinity were restored. + */ +static void guest_ist_restore_irq_handler(struct ex_regs *regs) +{ + u32 vcpu_id = guest_get_vcpuid(); + u32 count = ist_restore_irq_count[vcpu_id]; + u64 hwirq, expected_hwirq = 0; + u64 ia; + + ia = gicr_insn(CDIA); + if (!GICV5_GICR_CDIA_VALID(ia)) + return; + + gsb_ack(); + isb(); + + hwirq = FIELD_GET(GICV5_GICR_CDIA_INTID, ia); + + if (!vcpu_id) { + switch (count) { + case 0: + expected_hwirq = gicv5_spi_hwirq(RESTORE_TEST_SPI_PRIO_HIGH); + break; + case 1: + expected_hwirq = gicv5_spi_hwirq(RESTORE_TEST_SPI_PRIO_LOW); + break; + case 2: + expected_hwirq = gicv5_spi_hwirq(RESTORE_TEST_SPI_PENDING_DISABLED); + break; + case 3: + expected_hwirq = gicv5_spi_hwirq(RESTORE_TEST_SPI_ENABLED); + break; + default: + GUEST_FAIL("Unexpected VPE0 IST restore interrupt"); + } + } else { + switch (count) { + case 0: + expected_hwirq = gicv5_lpi_hwirq(RESTORE_TEST_LPI_PENDING); + break; + case 1: + expected_hwirq = gicv5_lpi_hwirq(RESTORE_TEST_LPI_ENABLED); + break; + default: + GUEST_FAIL("Unexpected VPE1 IST restore interrupt"); + } + } + + GUEST_ASSERT_EQ(hwirq, expected_hwirq); + + gic_insn(hwirq, CDDI); + gic_insn(0, CDEOI); + ist_restore_irq_count[vcpu_id]++; + + if (!vcpu_id) { + switch (count) { + case 0: + gicv5_set_pcr(VGIC_V5_IST_PRIO_LOW); + break; + case 1: + gicv5_enable_hwirq( + gicv5_spi_hwirq(RESTORE_TEST_SPI_PENDING_DISABLED)); + break; + case 2: + GUEST_SYNC(GUEST_CMD_IST_SPI_INJECT_READY); + break; + case 3: + GUEST_DONE(); + } + } else { + switch (count) { + case 0: + gicv5_send_lpi(RESTORE_TEST_LPI_ENABLED); + GUEST_SYNC(GUEST_CMD_IST_LPI_INJECT_READY); + break; + case 1: + GUEST_DONE(); + } + } +} + +static void guest_ist_restore_code(void) +{ + local_irq_disable(); + + gicv5_cpu_enable_interrupts(); + + if (!guest_get_vcpuid()) + gicv5_set_pcr(VGIC_V5_IST_PRIO_HIGH); + + GUEST_SYNC(GUEST_CMD_IS_READY); + + local_irq_enable(); + + while (1) + wfi(); +} /* we don't want to assert on run execution, hence that helper */ static int run_vcpu(struct kvm_vcpu *vcpu) @@ -94,12 +587,1005 @@ static void vm_gic_destroy(struct vm_gic *v) kvm_vm_free(v->vm); } +static void vgic_v5_map_irs(struct kvm_vm *vm) +{ + unsigned int nr_irs_pages; + + nr_irs_pages = vm_calc_num_guest_pages(vm->mode, VGIC_V5_IRS_SIZE); + + /* Map the IRS at VA == IPA so guest MMIO writes hit the IRS IODEV. */ + virt_map(vm, GICV5_IRS_CONFIG_BASE_GPA, + GICV5_IRS_CONFIG_BASE_GPA, nr_irs_pages); +} + +static u32 vgic_v5_irq_line_payload(u32 type, u32 num) +{ + return (type << KVM_ARM_IRQ_TYPE_SHIFT) | + FIELD_PREP(KVM_ARM_IRQ_NUM_MASK, num); +} + +static int __vgic_v5_irq_line(struct kvm_vm *vm, u32 type, u32 num, int level) +{ + return _kvm_irq_line(vm, vgic_v5_irq_line_payload(type, num), level); +} + +static void vgic_v5_spi_line(struct kvm_vm *vm, u32 spi, int level) +{ + int ret = __vgic_v5_irq_line(vm, KVM_ARM_IRQ_TYPE_SPI, spi, level); + + TEST_ASSERT(!ret, "KVM_IRQ_LINE failed for SPI %u level %d", + spi, level); +} + +static void vgic_v5_spi_line_vm_create(struct vm_gic *v, + struct kvm_vcpu **vcpu, + u32 nr_spis) +{ + u64 attr; + + v->gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v->vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v->gic_fd = kvm_create_device(v->vm, v->gic_dev_type); + *vcpu = vm_vcpu_add(v->vm, 0, guest_spi_line_code); + TEST_ASSERT(*vcpu, "Failed to create vCPU"); + + vm_init_descriptor_tables(v->vm); + vm_install_exception_handler(v->vm, VECTOR_IRQ_CURRENT, + guest_spi_irq_handler); + vcpu_init_descriptor_tables(*vcpu); + + kvm_device_attr_set(v->gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0, + &nr_spis); + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v->gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + vgic_v5_map_irs(v->vm); + kvm_device_attr_set(v->gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); +} + +static void vgic_v5_lpi_ist_vm_create(struct vm_gic *v, + struct kvm_vcpu *vcpus[VGIC_V5_LPI_NR_VCPUS], + void (*guest_code)(void), + void (*irq_handler)(struct ex_regs *)) +{ + unsigned int nr_lpi_ist_pages; + u64 attr; + int i; + + v->gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v->vm = __vm_create(VM_SHAPE_DEFAULT, VGIC_V5_LPI_NR_VCPUS, 0); + v->gic_fd = kvm_create_device(v->vm, v->gic_dev_type); + + for (i = 0; i < VGIC_V5_LPI_NR_VCPUS; i++) { + vcpus[i] = vm_vcpu_add(v->vm, i, guest_code); + TEST_ASSERT(vcpus[i], "Failed to create vCPU"); + } + + vm_init_descriptor_tables(v->vm); + vm_install_exception_handler(v->vm, VECTOR_IRQ_CURRENT, + irq_handler); + + for (i = 0; i < VGIC_V5_LPI_NR_VCPUS; i++) + vcpu_init_descriptor_tables(vcpus[i]); + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v->gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + vgic_v5_map_irs(v->vm); + + nr_lpi_ist_pages = vm_calc_num_guest_pages(v->vm->mode, + VGIC_V5_LPI_IST_SIZE); + vm_userspace_mem_region_add(v->vm, VM_MEM_SRC_ANONYMOUS, + VGIC_V5_LPI_IST_BASE_GPA, + VGIC_V5_LPI_MEMSLOT, + nr_lpi_ist_pages, 0); + /* + * Map the IST at VA == IPA so the guest can program the same BASER + * address. + */ + virt_map(v->vm, VGIC_V5_LPI_IST_BASE_GPA, + VGIC_V5_LPI_IST_BASE_GPA, nr_lpi_ist_pages); + + kvm_device_attr_set(v->gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); +} + +static void vgic_v5_lpi_vm_create(struct vm_gic *v, + struct kvm_vcpu *vcpus[VGIC_V5_LPI_NR_VCPUS]) +{ + vgic_v5_lpi_ist_vm_create(v, vcpus, guest_lpi_code, + guest_lpi_irq_handler); +} + +static void vgic_v5_ist_restore_vm_create(struct vm_gic *v, + struct kvm_vcpu *vcpus[VGIC_V5_LPI_NR_VCPUS], + void (*guest_code)(void), + void (*irq_handler)(struct ex_regs *)) +{ + vgic_v5_lpi_ist_vm_create(v, vcpus, guest_code, irq_handler); +} + +static void vgic_v5_run_spi_line_test(u32 nr_spis, u32 expected_spi, + bool level_sensitive, + bool lower_before_run) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + struct ucall uc; + int ret; + + spi_line_expected = expected_spi; + spi_line_level_sensitive = level_sensitive; + + vgic_v5_spi_line_vm_create(&v, &vcpu, nr_spis); + + sync_global_to_guest(v.vm, spi_line_expected); + sync_global_to_guest(v.vm, spi_line_level_sensitive); + + ret = run_vcpu(vcpu); + TEST_ASSERT(!ret, "Failed to run GICv5 IRQ_LINE VM"); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC && + uc.args[1] == GUEST_CMD_IS_READY, + "GICv5 IRQ_LINE guest did not become ready"); + + vgic_v5_spi_line(v.vm, expected_spi, 1); + /* + * For edge-triggered SPIs, a following low transition must be ignored: + * once the edge has made the SPI pending, it cannot be recalled. This + * allows us to test that. + */ + if (lower_before_run) + vgic_v5_spi_line(v.vm, expected_spi, 0); + + while (1) { + ret = run_vcpu(vcpu); + TEST_ASSERT(!ret, "Failed to run GICv5 IRQ_LINE VM"); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + break; + case UCALL_DONE: + goto done; + default: + TEST_FAIL("Unknown ucall %lu", uc.cmd); + } + } + +done: + vm_gic_destroy(&v); +} + +static void vgic_v5_expect_sync(struct kvm_vcpu *vcpu, u64 cmd, + const char *msg) +{ + struct ucall uc; + + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC && uc.args[1] == cmd, + "%s", msg); +} + +struct vgic_region_attr { + u64 attr; + u64 size; + u64 alignment; +}; + +static const struct vgic_region_attr gic_v5_irs_region = { + .attr = KVM_VGIC_V5_ADDR_TYPE_IRS, + .size = GICV5_IRS_SIZE, + .alignment = GICV5_IRS_ALIGN, +}; + +struct vgic_sysreg_attr { + const char *name; + u32 encoding; +}; + +struct vgic_irs_reg_attr { + const char *name; + u64 attr; +}; + +#define PACK_SR(r) \ + ((sys_reg_Op0(r) << 14) | \ + (sys_reg_Op1(r) << 11) | \ + (sys_reg_CRn(r) << 7) | \ + (sys_reg_CRm(r) << 3) | \ + (sys_reg_Op2(r))) + +#define SR(r) \ + { \ + .name = #r, \ + .encoding = r, \ + } + +#define IRS_REG(r) \ + { \ + .name = #r, \ + .attr = r, \ + } + +static const struct vgic_irs_reg_attr gic_v5_irs_regs[] = { + IRS_REG(GICV5_IRS_IDR0), + IRS_REG(GICV5_IRS_IDR1), + IRS_REG(GICV5_IRS_IDR2), + IRS_REG(GICV5_IRS_IDR3), + IRS_REG(GICV5_IRS_IDR4), + IRS_REG(GICV5_IRS_IDR5), + IRS_REG(GICV5_IRS_IDR6), + IRS_REG(GICV5_IRS_IDR7), + IRS_REG(GICV5_IRS_IIDR), + IRS_REG(GICV5_IRS_AIDR), + IRS_REG(GICV5_IRS_CR0), + IRS_REG(GICV5_IRS_CR1), + IRS_REG(GICV5_IRS_SYNCR), + IRS_REG(GICV5_IRS_SYNC_STATUSR), + IRS_REG(GICV5_IRS_SPI_VMR), + IRS_REG(GICV5_IRS_SPI_SELR), + IRS_REG(GICV5_IRS_SPI_DOMAINR), + IRS_REG(GICV5_IRS_SPI_RESAMPLER), + IRS_REG(GICV5_IRS_SPI_CFGR), + IRS_REG(GICV5_IRS_SPI_STATUSR), + IRS_REG(GICV5_IRS_PE_SELR), + IRS_REG(GICV5_IRS_PE_STATUSR), + IRS_REG(GICV5_IRS_PE_CR0), + IRS_REG(GICV5_IRS_IST_BASER), + IRS_REG(GICV5_IRS_IST_CFGR), + IRS_REG(GICV5_IRS_IST_STATUSR), + IRS_REG(GICV5_IRS_MAP_L2_ISTR), + IRS_REG(GICV5_IRS_VMT_BASER), + IRS_REG(GICV5_IRS_VMT_CFGR), + IRS_REG(GICV5_IRS_VMT_STATUSR), + IRS_REG(GICV5_IRS_VPE_SELR), + IRS_REG(GICV5_IRS_VPE_DBR), + IRS_REG(GICV5_IRS_VPE_HPPIR), + IRS_REG(GICV5_IRS_VPE_CR0), + IRS_REG(GICV5_IRS_VPE_STATUSR), + IRS_REG(GICV5_IRS_VM_DBR), + IRS_REG(GICV5_IRS_VM_SELR), + IRS_REG(GICV5_IRS_VM_STATUSR), + IRS_REG(GICV5_IRS_VMAP_L2_VMTR), + IRS_REG(GICV5_IRS_VMAP_VMR), + IRS_REG(GICV5_IRS_VMAP_VISTR), + IRS_REG(GICV5_IRS_VMAP_L2_VISTR), + IRS_REG(GICV5_IRS_VMAP_VPER), + IRS_REG(GICV5_IRS_SAVE_VMR), + IRS_REG(GICV5_IRS_SAVE_VM_STATUSR), + IRS_REG(GICV5_IRS_MEC_IDR), + IRS_REG(GICV5_IRS_MEC_MECID_R), + IRS_REG(GICV5_IRS_MPAM_IDR), + IRS_REG(GICV5_IRS_MPAM_PARTID_R), + IRS_REG(GICV5_IRS_SWERR_STATUSR), + IRS_REG(GICV5_IRS_SWERR_SYNDROMER0), + IRS_REG(GICV5_IRS_SWERR_SYNDROMER1), +}; + +static const struct vgic_sysreg_attr gic_v5_cpu_sysregs[] = { + SR(SYS_ICC_ICSR_EL1), + SR(SYS_ICC_PPI_ENABLER0_EL1), + SR(SYS_ICC_PPI_ENABLER1_EL1), + SR(SYS_ICC_PPI_SACTIVER0_EL1), + SR(SYS_ICC_PPI_SACTIVER1_EL1), + SR(SYS_ICC_PPI_SPENDR0_EL1), + SR(SYS_ICC_PPI_SPENDR1_EL1), + SR(SYS_ICC_PPI_PRIORITYR0_EL1), + SR(SYS_ICC_PPI_PRIORITYR1_EL1), + SR(SYS_ICC_PPI_PRIORITYR2_EL1), + SR(SYS_ICC_PPI_PRIORITYR3_EL1), + SR(SYS_ICC_PPI_PRIORITYR4_EL1), + SR(SYS_ICC_PPI_PRIORITYR5_EL1), + SR(SYS_ICC_PPI_PRIORITYR6_EL1), + SR(SYS_ICC_PPI_PRIORITYR7_EL1), + SR(SYS_ICC_PPI_PRIORITYR8_EL1), + SR(SYS_ICC_PPI_PRIORITYR9_EL1), + SR(SYS_ICC_PPI_PRIORITYR10_EL1), + SR(SYS_ICC_PPI_PRIORITYR11_EL1), + SR(SYS_ICC_PPI_PRIORITYR12_EL1), + SR(SYS_ICC_PPI_PRIORITYR13_EL1), + SR(SYS_ICC_PPI_PRIORITYR14_EL1), + SR(SYS_ICC_PPI_PRIORITYR15_EL1), + SR(SYS_ICC_APR_EL1), + SR(SYS_ICC_CR0_EL1), + SR(SYS_ICC_PCR_EL1), +}; + +static void vgic_v5_save_irs_regs(struct vm_gic *v, u64 *regs) +{ + int ret, i; + + for (i = 0; i < ARRAY_SIZE(gic_v5_irs_regs); i++) { + ret = __kvm_device_attr_get(v->gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + gic_v5_irs_regs[i].attr, + ®s[i]); + TEST_ASSERT(!ret, "GICv5 IRS_REGS save failed for %s", + gic_v5_irs_regs[i].name); + } +} + +static void vgic_v5_restore_irs_regs(struct vm_gic *v, u64 *regs) +{ + int ret, i; + + for (i = 0; i < ARRAY_SIZE(gic_v5_irs_regs); i++) { + ret = __kvm_device_attr_set(v->gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + gic_v5_irs_regs[i].attr, + ®s[i]); + TEST_ASSERT(!ret, "GICv5 IRS_REGS restore failed for %s", + gic_v5_irs_regs[i].name); + } +} + +static u64 vgic_v5_saved_irs_reg(u64 *regs, u64 attr) +{ + for (int i = 0; i < ARRAY_SIZE(gic_v5_irs_regs); i++) { + if (gic_v5_irs_regs[i].attr == attr) + return regs[i]; + } + + TEST_FAIL("GICv5 IRS register 0x%lx was not saved", attr); +} + +static size_t vgic_v5_lpi_ist_state_size(u64 *irs_regs) +{ + u64 cfgr = vgic_v5_saved_irs_reg(irs_regs, GICV5_IRS_IST_CFGR); + u32 lpi_id_bits = FIELD_GET(GICV5_IRS_IST_CFGR_LPI_ID_BITS, cfgr); + + return BIT(lpi_id_bits) * sizeof(__u32); +} + +static void test_vgic_v5_addr_attrs(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + u64 addr; + int ret; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + + /* Check existing group/attributes */ + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, gic_v5_irs_region.attr); + + /* check non existing attribute */ + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, -1); + TEST_ASSERT(ret && errno == ENXIO, "attribute not supported"); + + /* get IRS base address before setting*/ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(!ret && addr == (-1ULL), "GICv5 IRS returns VGIC_ADDR_UNDEF"); + + /* misaligned IRS address settings */ + addr = gic_v5_irs_region.alignment / 0x10; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(ret && errno == EINVAL, "GIC IRS base not aligned"); + + /* out of range address */ + addr = max_phys_size; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(ret && errno == E2BIG, "IRS address beyond IPA limit"); + + /* Space for half an IRS (an IRS is: 2 * irs.alignment). */ + addr = max_phys_size - gic_v5_irs_region.alignment; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(ret && errno == E2BIG, + "half of the IRS is beyond IPA limit"); + + /* set IRS base address @0x0*/ + addr = 0x00000; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(!ret, "GICv5 IRS base correctly set"); + + /* get IRS base address */ + addr = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(!ret && addr == 0, "GICv5 IRS base correctly set"); + + /* Attempt to create a second IRS region */ + addr = 0xE0000; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &addr); + TEST_ASSERT(ret && errno == EEXIST, "GICv5 IRS base set again"); + + vm_gic_destroy(&v); + + /* Try running a VM without ever setting the IRS base addr */ + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + ret = run_vcpu(vcpu); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IRS base not set"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_nr_irqs_attrs(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + u32 nr_irqs; + int ret; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + + /* Check existing group/attribute */ + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0); + + /* Before userspace sets NR_IRQS, no SPI count has been selected. */ + nr_irqs = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(!ret && nr_irqs == 0, "GICv5 NR_IRQS defaults to 0 before init"); + + /* Too few SPIs */ + nr_irqs = VGIC_V5_DEFAULT_NR_SPIS - 1; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 NR_IRQS below minimum"); + + /* Not a multiple of 32 */ + nr_irqs = VGIC_V5_DEFAULT_NR_SPIS + 1; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 NR_IRQS not 32-aligned"); + + /* Larger than KVM's supported VGICv5 SPI count */ + nr_irqs = VGIC_V5_MAX_NR_SPIS + 32; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 NR_IRQS above maximum"); + + /* Valid custom SPI count */ + nr_irqs = VGIC_V5_MAX_NR_SPIS; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(!ret, "GICv5 NR_IRQS accepts valid custom SPI count"); + + nr_irqs = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(!ret && nr_irqs == VGIC_V5_MAX_NR_SPIS, + "GICv5 NR_IRQS returns SPI count only"); + + /* A second successful configuration attempt must be rejected. */ + nr_irqs = VGIC_V5_DEFAULT_NR_SPIS; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 NR_IRQS set twice"); + + /* The maximum supported count must also initialize successfully. */ + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + vm_gic_destroy(&v); + + /* If userspace does not set NR_IRQS, init selects the default. */ + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + nr_irqs = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(!ret && nr_irqs == VGIC_V5_DEFAULT_NR_SPIS, + "GICv5 NR_IRQS defaults to 32 SPIs after init"); + + nr_irqs = VGIC_V5_DEFAULT_NR_SPIS * 2; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, + 0, &nr_irqs); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 NR_IRQS set after init"); + + vm_gic_destroy(&v); + +} + +static void test_vgic_v5_irs_regs_attrs(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + u64 attr, val; + int ret, i; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + + /* IRS_REGS attributes can be probed before the IRS base is set. */ + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0); + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + + /* Check existing group/attribute */ + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0); + + /* IRS_REGS are not accessible before the VGIC is initialized. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 IRS_REGS get before init"); + + val = 0; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 IRS_REGS set before init"); + + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + /* Read all supported IRS regs and write the value back. */ + for (i = 0; i < ARRAY_SIZE(gic_v5_irs_regs); i++) { + attr = gic_v5_irs_regs[i].attr; + ret = __kvm_has_device_attr(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr); + TEST_ASSERT(!ret, "GICv5 IRS_REGS missing %s", + gic_v5_irs_regs[i].name); + + val = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get failed for %s", + gic_v5_irs_regs[i].name); + + ret = __kvm_device_attr_set(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS set failed for %s", + gic_v5_irs_regs[i].name); + } + + /* Check bad offsets */ + attr = 0x10000; + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IRS_REGS accepted bad offset"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr, &val); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IRS_REGS get bad offset"); + + /* Check alignment for 32-bit and 64-bit IRS regs. */ + attr = GICV5_IRS_IDR0 + 2; + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted unaligned 32-bit attr"); + + attr = GICV5_IRS_IST_BASER + 4; + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted unaligned 64-bit attr"); + + /* Check bad user pointers */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, NULL); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 IRS_REGS get with bad pointer"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, NULL); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 IRS_REGS set with bad pointer"); + + /* ID restore validation rejects unsupported values. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR0 failed"); + val &= ~GICV5_IRS_IDR0_INT_DOM; + val |= FIELD_PREP(GICV5_IRS_IDR0_INT_DOM, + GICV5_IRS_IDR0_INT_DOM_SECURE); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR0 domain"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR0 failed"); + val |= GICV5_IRS_IDR0_SETLPI; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR0, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted unsupported IDR0"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR1, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR1 failed"); + val |= FIELD_PREP(GICV5_IRS_IDR1_PRIORITY_BITS, 0x7); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR1, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR1"); + + val = 0; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR2, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR2"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR5, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR5 failed"); + val++; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR5, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR5"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR6, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR6 failed"); + val++; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR6, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR6"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR7, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS get IDR7 failed"); + val++; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IDR7, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IRS_REGS accepted bad IDR7"); + + /* Status registers read as idle through userspace accessors. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_SYNC_STATUSR, &val); + TEST_ASSERT(!ret && val == GICV5_IRS_SYNC_STATUSR_IDLE, + "GICv5 IRS_REGS SYNC_STATUSR is not idle"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_SPI_STATUSR, &val); + TEST_ASSERT(!ret && val == GICV5_IRS_SPI_STATUSR_IDLE, + "GICv5 IRS_REGS SPI_STATUSR is not idle"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_PE_STATUSR, &val); + TEST_ASSERT(!ret && val == GICV5_IRS_PE_STATUSR_IDLE, + "GICv5 IRS_REGS PE_STATUSR is not idle"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_STATUSR, &val); + TEST_ASSERT(!ret && val == GICV5_IRS_IST_STATUSR_IDLE, + "GICv5 IRS_REGS IST_STATUSR is not idle"); + + /* + * Userspace restores IST_BASER without allocating or accessing an IST. + * Supply a synthetic, aligned guest address only to verify the register + * round trip. + */ + val = FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, + VGIC_V5_TEST_IST_BASER_GPA >> GICV5_IRS_IST_BASER_ADDR_SHIFT); + val |= GICV5_IRS_IST_BASER_VALID; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &val); + TEST_ASSERT(!ret, "GICv5 IRS_REGS failed to restore IST_BASER"); + + attr = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &attr); + TEST_ASSERT(!ret && attr == val, "GICv5 IRS_REGS IST_BASER restore mismatch"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_ist_attrs(void) +{ + struct kvm_vgic_v5_ist ist_attr; + struct kvm_vcpu *vcpu; + struct vm_gic v; + u32 spi_ist[VGIC_V5_DEFAULT_NR_SPIS]; + u64 attr; + int ret; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + + /* Check existing group/attribute */ + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, 0); + + /* Check non-existing attribute */ + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, 1); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IST accepted bad attr"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), NULL, 0); + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 1, &ist_attr); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IST get accepted bad attr"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 1, &ist_attr); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 IST set accepted bad attr"); + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + + /* IST save/restore is not accessible before the VGIC is initialized. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 IST get before init"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 IST set before init"); + + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + /* A VM with SPIs must provide a userspace IST descriptor. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, NULL); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST get accepted NULL descriptor"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, NULL); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST set accepted NULL descriptor"); + + /* Check bad userspace IST descriptors. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, (void *)1); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 IST get with bad descriptor"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, (void *)1); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 IST set with bad descriptor"); + + /* Check missing and incorrectly sized SPI IST buffers. */ + ist_attr = vgic_v5_ist_attr(NULL, 0, NULL, 0); + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST get accepted missing SPI buffer"); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST set accepted missing SPI buffer"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist) - sizeof(__u32), + NULL, 0); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST set accepted bad SPI size"); + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST get accepted bad SPI size"); + + /* LPI storage must be absent when no LPI IST is configured. */ + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), spi_ist, + sizeof(__u32)); + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST get accepted unexpected LPI buffer"); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST set accepted unexpected LPI buffer"); + + vm_gic_destroy(&v); + + /* IST restore is rejected after the VM has run. */ + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, guest_code); + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + ret = run_vcpu(vcpu); + TEST_ASSERT(!ret, "Failed to run GICv5 VM before IST restore test"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), NULL, 0); + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 IST restore after run"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_userspace_ppis_attrs(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + u64 user_ppis[2]; + int ret; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT); + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_USERSPACE_PPIS); + + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + user_ppis[0] = 0; + user_ppis[1] = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_USERSPACE_PPIS, user_ppis); + TEST_ASSERT(!ret, "GICv5 USERSPACE_PPIS get failed"); + TEST_ASSERT(user_ppis[0] & BIT(GICV5_ARCH_PPI_SW_PPI), + "GICv5 USERSPACE_PPIS does not expose SW_PPI"); + TEST_ASSERT(!user_ppis[1], "GICv5 USERSPACE_PPIS upper word is not zero"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_USERSPACE_PPIS, user_ppis); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 USERSPACE_PPIS set accepted"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_USERSPACE_PPIS, NULL); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 USERSPACE_PPIS get with bad pointer"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_cpu_sysreg_attrs(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + u64 attr, val; + int ret, i; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + vcpu = vm_vcpu_add(v.vm, 0, NULL); + TEST_ASSERT(vcpu, "Failed to create vCPU"); + + /* Check existing group/attribute */ + attr = PACK_SR(SYS_ICC_CR0_EL1); + kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, attr); + + /* CPU sysregs are not accessible before the VGIC is initialized. */ + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 CPU_SYSREGS get before init"); + + val = 0; + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(ret && errno == EBUSY, "GICv5 CPU_SYSREGS set before init"); + + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + /* Read all exposed CPU sysregs and write the value back. */ + for (i = 0; i < ARRAY_SIZE(gic_v5_cpu_sysregs); i++) { + attr = PACK_SR(gic_v5_cpu_sysregs[i].encoding); + ret = __kvm_has_device_attr(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr); + TEST_ASSERT(!ret, "GICv5 CPU_SYSREGS missing %s", + gic_v5_cpu_sysregs[i].name); + + val = 0xbad; + ret = __kvm_device_attr_get(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(!ret, "GICv5 CPU_SYSREGS get failed for %s", + gic_v5_cpu_sysregs[i].name); + + ret = __kvm_device_attr_set(v.gic_fd, + KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(!ret, "GICv5 CPU_SYSREGS set failed for %s", + gic_v5_cpu_sysregs[i].name); + } + + /* Check non existent GICv3 sysreg */ + attr = PACK_SR(SYS_ICC_CTLR_EL1); + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr); + TEST_ASSERT(ret && errno == ENXIO, "GICv5 CPU_SYSREGS accepted bad sysreg"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(ret && errno == ENOENT, "GICv5 CPU_SYSREGS get bad sysreg"); + + /* Check non existing vCPU */ + attr = PACK_SR(SYS_ICC_CR0_EL1) | + (1ULL << KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT); + ret = __kvm_has_device_attr(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 CPU_SYSREGS accepted bad MPIDR"); + + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, &val); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 CPU_SYSREGS get bad MPIDR"); + + /* Check bad user pointers */ + attr = PACK_SR(SYS_ICC_CR0_EL1); + ret = __kvm_device_attr_get(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, NULL); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 CPU_SYSREGS get with bad pointer"); + + ret = __kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, + attr, NULL); + TEST_ASSERT(ret && errno == EFAULT, "GICv5 CPU_SYSREGS set with bad pointer"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_sparse_vcpu_ids(void) +{ + struct kvm_vcpu *vcpus[VGIC_V5_SPARSE_NR_VCPUS]; + const u32 vcpu_ids[VGIC_V5_SPARSE_NR_VCPUS] = { + VGIC_V5_SPARSE_VCPU0_ID, + VGIC_V5_SPARSE_VCPU1_ID, + }; + int ret, i; + struct ucall uc; + struct vm_gic v; + u64 attr; + + v.gic_dev_type = KVM_DEV_TYPE_ARM_VGIC_V5; + v.vm = __vm_create(VM_SHAPE_DEFAULT, VGIC_V5_SPARSE_NR_VCPUS, 0); + v.gic_fd = kvm_create_device(v.vm, v.gic_dev_type); + + /* Create two vCPUs with legal, albeit sparse, IDs. */ + for (i = 0; i < ARRAY_SIZE(vcpus); i++) { + vcpus[i] = vm_vcpu_add(v.vm, vcpu_ids[i], guest_iaffid_code); + TEST_ASSERT(vcpus[i], "Failed to create vCPU %u", vcpu_ids[i]); + } + + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + vgic_v5_map_irs(v.vm); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); + + /* Check both are runnable, and read back their correct IAFFIDs */ + for (i = 0; i < ARRAY_SIZE(vcpus); i++) { + /* IAFFID is the VPE ID and must match the userspace vCPU ID. */ + ret = run_vcpu(vcpus[i]); + TEST_ASSERT(!ret, "Failed to run GICv5 vCPU %u", vcpu_ids[i]); + TEST_ASSERT(get_ucall(vcpus[i], &uc) == UCALL_SYNC && + uc.args[1] == vcpu_ids[i], + "GICv5 vCPU %u IAFFID mismatch", vcpu_ids[i]); + + ret = run_vcpu(vcpus[i]); + TEST_ASSERT(!ret, "Failed to complete GICv5 vCPU %u", vcpu_ids[i]); + TEST_ASSERT(get_ucall(vcpus[i], NULL) == UCALL_DONE, + "GICv5 vCPU %u did not complete", vcpu_ids[i]); + } + + vm_gic_destroy(&v); +} + static void test_vgic_v5_ppis(u32 gic_dev_type) { struct kvm_vcpu *vcpus[NR_VCPUS]; struct ucall uc; u64 user_ppis[2]; struct vm_gic v; + uint64_t attr; int ret, i; v.gic_dev_type = gic_dev_type; @@ -111,11 +1597,17 @@ static void test_vgic_v5_ppis(u32 gic_dev_type) vcpus[i] = vm_vcpu_add(v.vm, i, guest_code); vm_init_descriptor_tables(v.vm); - vm_install_exception_handler(v.vm, VECTOR_IRQ_CURRENT, guest_irq_handler); + vm_install_exception_handler(v.vm, VECTOR_IRQ_CURRENT, + guest_ppi_irq_handler); for (i = 0; i < NR_VCPUS; i++) vcpu_init_descriptor_tables(vcpus[i]); + /* Set the address of the IRS before initialising the GIC */ + attr = GICV5_IRS_CONFIG_BASE_GPA; + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR, + KVM_VGIC_V5_ADDR_TYPE_IRS, &attr); + kvm_device_attr_set(v.gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_CTRL_INIT, NULL); @@ -127,6 +1619,20 @@ static void test_vgic_v5_ppis(u32 gic_dev_type) TEST_ASSERT(user_ppis[0] & BIT(GICV5_ARCH_PPI_SW_PPI), "SW_PPI is not drivable by userspace"); + /* PPIs not explicitly exposed to userspace must be rejected. */ + for (i = 0; i < VGIC_V5_NR_PRIVATE_IRQS; i++) { + if (user_ppis[i / 64] & BIT_ULL(i % 64)) + continue; + + ret = __vgic_v5_irq_line(v.vm, KVM_ARM_IRQ_TYPE_PPI, i, 1); + TEST_ASSERT(ret && errno == EINVAL, + "GICv5 accepted non-userspace PPI %d", i); + } + + ret = __vgic_v5_irq_line(v.vm, KVM_ARM_IRQ_TYPE_PPI, + VGIC_V5_NR_PRIVATE_IRQS, 1); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 accepted out-of-range PPI"); + while (1) { ret = run_vcpu(vcpus[0]); if (ret) @@ -148,9 +1654,9 @@ static void test_vgic_v5_ppis(u32 gic_dev_type) kvm_irq_line(v.vm, irq, level); } else if (uc.args[1] == GUEST_CMD_IS_AWAKE) { - pr_info("Guest skipping WFI due to pending IRQ\n"); + pr_debug("Guest skipping WFI due to pending IRQ\n"); } else if (uc.args[1] == GUEST_CMD_IRQ_CDIA) { - pr_info("Guest acknowledged IRQ\n"); + pr_debug("Guest acknowledged IRQ\n"); } continue; @@ -170,6 +1676,304 @@ done: vm_gic_destroy(&v); } +static void test_vgic_v5_spis(void) +{ + struct kvm_vcpu *vcpu; + struct vm_gic v; + int ret; + + /* Default NR_IRQS exposes 32 SPIs, numbered 0..31 in KVM_IRQ_LINE. */ + vgic_v5_spi_line_vm_create(&v, &vcpu, VGIC_V5_DEFAULT_NR_SPIS); + ret = __vgic_v5_irq_line(v.vm, KVM_ARM_IRQ_TYPE_SPI, + VGIC_V5_DEFAULT_NR_SPIS, 1); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 accepted first invalid SPI"); + + vm_gic_destroy(&v); + + /* Basic SPI injection through KVM_IRQ_LINE. */ + vgic_v5_run_spi_line_test(VGIC_V5_DEFAULT_NR_SPIS, 0, false, false); + + /* The last valid SPI in the maximum configured range is injectable. */ + vgic_v5_run_spi_line_test(VGIC_V5_MAX_NR_SPIS, + VGIC_V5_MAX_NR_SPIS - 1, false, false); + + vgic_v5_spi_line_vm_create(&v, &vcpu, VGIC_V5_MAX_NR_SPIS); + ret = __vgic_v5_irq_line(v.vm, KVM_ARM_IRQ_TYPE_SPI, + VGIC_V5_MAX_NR_SPIS, 1); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 accepted configured invalid SPI"); + vm_gic_destroy(&v); + + /* + * Edge SPIs remain pending after the line is lowered. The low + * transition is injected before the guest runs and must be ignored: + * once an edge interrupt is pending, lowering the line cannot recall it. + */ + vgic_v5_run_spi_line_test(VGIC_V5_DEFAULT_NR_SPIS, 1, false, true); + + /* Level SPIs can be raised and delivered through KVM_IRQ_LINE. */ + vgic_v5_run_spi_line_test(VGIC_V5_DEFAULT_NR_SPIS, 2, true, false); +} + +static void test_vgic_v5_lpis(void) +{ + struct kvm_vcpu *vcpus[VGIC_V5_LPI_NR_VCPUS]; + struct ucall uc; + struct vm_gic v; + int ret; + + lpi_ist_ready = false; + vgic_v5_lpi_vm_create(&v, vcpus); + sync_global_to_guest(v.vm, lpi_ist_ready); + + /* VPE0 programs a linear LPI IST from the virtual IRS ID registers. */ + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to run GICv5 LPI vCPU0"); + TEST_ASSERT(get_ucall(vcpus[0], &uc) == UCALL_SYNC && + uc.args[1] == GUEST_CMD_IS_READY, + "GICv5 LPI vCPU0 did not become ready"); + + /* VPE1 waits for the shared LPI setup and enables its CPU interface. */ + ret = run_vcpu(vcpus[1]); + TEST_ASSERT(!ret, "Failed to run GICv5 LPI vCPU1"); + TEST_ASSERT(get_ucall(vcpus[1], &uc) == UCALL_SYNC && + uc.args[1] == GUEST_CMD_IS_READY, + "GICv5 LPI vCPU1 did not become ready"); + + /* VPE0 sends an LPI to VPE1. */ + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to run GICv5 LPI vCPU0 sender"); + TEST_ASSERT(get_ucall(vcpus[0], &uc) == UCALL_SYNC && + uc.args[1] == GUEST_CMD_LPI_SENT, + "GICv5 LPI vCPU0 did not send LPI"); + + /* VPE1 consumes the LPI and replies with another LPI. */ + ret = run_vcpu(vcpus[1]); + TEST_ASSERT(!ret, "Failed to run GICv5 LPI vCPU1 receiver"); + TEST_ASSERT(get_ucall(vcpus[1], &uc) == UCALL_SYNC && + uc.args[1] == GUEST_CMD_LPI_REPLIED, + "GICv5 LPI vCPU1 did not reply"); + + /* VPE0 consumes the reply LPI and ends the test. */ + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to run GICv5 LPI vCPU0 receiver"); + TEST_ASSERT(get_ucall(vcpus[0], &uc) == UCALL_DONE, + "GICv5 LPI vCPU0 did not complete"); + + vm_gic_destroy(&v); +} + +static void test_vgic_v5_ist_save_restore(void) +{ + static u32 lpi_ist[VGIC_V5_LPI_IST_SIZE / sizeof(u32)]; + static u32 lpi_ist_resave[VGIC_V5_LPI_IST_SIZE / sizeof(u32)]; + static u32 spi_ist[VGIC_V5_DEFAULT_NR_SPIS]; + static u32 spi_ist_resave[VGIC_V5_DEFAULT_NR_SPIS]; + static u64 irs_regs[ARRAY_SIZE(gic_v5_irs_regs)]; + struct kvm_vcpu *vcpus[VGIC_V5_LPI_NR_VCPUS]; + struct kvm_vgic_v5_ist ist_attr; + struct vm_gic src, dst; + u64 ist_baser, bad_ist_baser; + u64 ist_cfgr, bad_ist_cfgr; + size_t lpi_ist_size; + int ret; + + memset(ist_restore_irq_count, 0, sizeof(ist_restore_irq_count)); + + vgic_v5_ist_restore_vm_create(&src, vcpus, guest_ist_source_code, + guest_unexpected_irq_handler); + + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to run GICv5 IST save source VM"); + vgic_v5_expect_sync(vcpus[0], GUEST_CMD_IST_SOURCE_READY, + "GICv5 IST source guest did not become ready"); + + /* + * The guest configured enable/priority/affinity state. Make selected + * SPIs pending from userspace so the SPI IST save path sees them. + */ + vgic_v5_spi_line(src.vm, RESTORE_TEST_SPI_PRIO_HIGH, 1); + vgic_v5_spi_line(src.vm, RESTORE_TEST_SPI_PRIO_LOW, 1); + vgic_v5_spi_line(src.vm, RESTORE_TEST_SPI_PENDING_DISABLED, 1); + + vgic_v5_save_irs_regs(&src, irs_regs); + lpi_ist_size = vgic_v5_lpi_ist_state_size(irs_regs); + + /* + * VCPU0 has configured the initial state. KVM must reject missing or + * incorrectly sized LPI storage before accepting the complete IST image. + */ + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), NULL, 0); + ret = __kvm_device_attr_get(src.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST save accepted missing LPI buffer"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), lpi_ist, + lpi_ist_size - sizeof(__u32)); + ret = __kvm_device_attr_get(src.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST save accepted bad LPI size"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), lpi_ist, + lpi_ist_size); + ret = __kvm_device_attr_get(src.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(!ret, "GICv5 IST save failed"); + + vm_gic_destroy(&src); + + /* Create the guest that we are restoring state into */ + memset(ist_restore_irq_count, 0, sizeof(ist_restore_irq_count)); + vgic_v5_ist_restore_vm_create(&dst, vcpus, guest_ist_restore_code, + guest_ist_restore_irq_handler); + sync_global_to_guest(dst.vm, ist_restore_irq_count); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), lpi_ist, + lpi_ist_size); + ret = __kvm_device_attr_set(dst.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST restore before IRS state"); + + vgic_v5_restore_irs_regs(&dst, irs_regs); + + /* BASER remains mutable until the host LPI IST has been allocated. */ + ist_baser = vgic_v5_saved_irs_reg(irs_regs, GICV5_IRS_IST_BASER); + bad_ist_baser = ist_baser ^ + FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, 1); + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &bad_ist_baser); + TEST_ASSERT(!ret, "GICv5 rejected changed unallocated IST_BASER"); + + bad_ist_baser = ist_baser & ~GICV5_IRS_IST_BASER_VALID; + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &bad_ist_baser); + TEST_ASSERT(!ret, "GICv5 rejected invalidated unallocated IST_BASER"); + + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &ist_baser); + TEST_ASSERT(!ret, "GICv5 failed to restore IST_BASER"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), NULL, 0); + ret = __kvm_device_attr_set(dst.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST restore accepted missing LPI buffer"); + + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), lpi_ist, + lpi_ist_size - sizeof(__u32)); + ret = __kvm_device_attr_set(dst.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(ret && errno == EINVAL, "GICv5 IST restore accepted bad LPI size"); + + /* Restore the VMM-owned SPI and LPI IST images. */ + ist_attr = vgic_v5_ist_attr(spi_ist, sizeof(spi_ist), lpi_ist, + lpi_ist_size); + ret = __kvm_device_attr_set(dst.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(!ret, "GICv5 IST restore failed"); + + /* A live IST makes BASER and the guest-visible configuration immutable. */ + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &ist_baser); + TEST_ASSERT(!ret, "GICv5 rejected unchanged live IST_BASER"); + + bad_ist_baser = ist_baser ^ + FIELD_PREP(GICV5_IRS_IST_BASER_ADDR_MASK, 1); + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &bad_ist_baser); + TEST_ASSERT(ret && errno == EINVAL, + "GICv5 accepted changed live IST_BASER"); + + bad_ist_baser = ist_baser & ~GICV5_IRS_IST_BASER_VALID; + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_BASER, &bad_ist_baser); + TEST_ASSERT(ret && errno == EINVAL, + "GICv5 accepted invalidated live IST_BASER"); + + ist_cfgr = vgic_v5_saved_irs_reg(irs_regs, GICV5_IRS_IST_CFGR); + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_CFGR, &ist_cfgr); + TEST_ASSERT(!ret, "GICv5 rejected unchanged live IST_CFGR"); + + bad_ist_cfgr = ist_cfgr ^ + FIELD_PREP(GICV5_IRS_IST_CFGR_LPI_ID_BITS, 1); + ret = __kvm_device_attr_set(dst.gic_fd, + KVM_DEV_ARM_VGIC_GRP_IRS_REGS, + GICV5_IRS_IST_CFGR, &bad_ist_cfgr); + TEST_ASSERT(ret && errno == EINVAL, + "GICv5 accepted changed live IST_CFGR"); + + /* + * Re-save the restored IST image before the destination VM runs, so a + * dropped serialized field fails independently of later interrupt + * delivery. + */ + ist_attr = vgic_v5_ist_attr(spi_ist_resave, sizeof(spi_ist_resave), + lpi_ist_resave, lpi_ist_size); + ret = __kvm_device_attr_get(dst.gic_fd, KVM_DEV_ARM_VGIC_GRP_IST, + 0, &ist_attr); + TEST_ASSERT(!ret, "GICv5 IST re-save failed"); + TEST_ASSERT(!memcmp(spi_ist, spi_ist_resave, sizeof(spi_ist)), + "GICv5 SPI IST state changed across restore"); + TEST_ASSERT(!memcmp(lpi_ist, lpi_ist_resave, lpi_ist_size), + "GICv5 LPI IST state changed across restore"); + + /* + * At this stage, the guest's IST state should match what was saved. Run + * each vcpu once to let it enable interrupts, etc. + */ + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to run GICv5 IST restore vCPU0"); + vgic_v5_expect_sync(vcpus[0], GUEST_CMD_IS_READY, + "GICv5 IST restore vCPU0 did not become ready"); + + ret = run_vcpu(vcpus[1]); + TEST_ASSERT(!ret, "Failed to run GICv5 IST restore vCPU1"); + vgic_v5_expect_sync(vcpus[1], GUEST_CMD_IS_READY, + "GICv5 IST restore vCPU1 did not become ready"); + + /* + * VPE0 consumes two pending SPIs in restored priority order, then + * enables a restored pending-but-disabled SPI. + */ + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to consume restored GICv5 SPIs"); + vgic_v5_expect_sync(vcpus[0], GUEST_CMD_IST_SPI_INJECT_READY, + "GICv5 IST restore vCPU0 did not consume restored SPIs"); + + /* + * This SPI was enabled before save, but not pending. Inject it after + * restore to prove enable, priority and affinity were restored. + */ + vgic_v5_spi_line(dst.vm, RESTORE_TEST_SPI_ENABLED, 1); + + ret = run_vcpu(vcpus[0]); + TEST_ASSERT(!ret, "Failed to consume post-restore GICv5 SPI"); + TEST_ASSERT(get_ucall(vcpus[0], NULL) == UCALL_DONE, + "GICv5 IST restore vCPU0 did not complete"); + + /* + * VPE1 consumes a restored pending LPI, then pends another LPI that + * was enabled before save but not pending. + */ + ret = run_vcpu(vcpus[1]); + TEST_ASSERT(!ret, "Failed to consume restored GICv5 LPI"); + vgic_v5_expect_sync(vcpus[1], GUEST_CMD_IST_LPI_INJECT_READY, + "GICv5 IST restore vCPU1 did not consume restored LPI"); + + ret = run_vcpu(vcpus[1]); + TEST_ASSERT(!ret, "Failed to consume post-restore GICv5 LPI"); + TEST_ASSERT(get_ucall(vcpus[1], NULL) == UCALL_DONE, + "GICv5 IST restore vCPU1 did not complete"); + + vm_gic_destroy(&dst); +} + /* * Returns 0 if it's possible to create GIC device of a given type (V5). */ @@ -201,13 +2005,46 @@ int test_kvm_device(u32 gic_dev_type) void run_tests(u32 gic_dev_type) { + pr_info("Test VGICv5 address attrs\n"); + test_vgic_v5_addr_attrs(); + + pr_info("Test VGICv5 NR_IRQS attrs\n"); + test_vgic_v5_nr_irqs_attrs(); + + pr_info("Test VGICv5 IRS_REGS attrs\n"); + test_vgic_v5_irs_regs_attrs(); + + pr_info("Test VGICv5 IST attrs\n"); + test_vgic_v5_ist_attrs(); + + pr_info("Test VGICv5 userspace PPI attrs\n"); + test_vgic_v5_userspace_ppis_attrs(); + + pr_info("Test VGICv5 CPU sysreg attrs\n"); + test_vgic_v5_cpu_sysreg_attrs(); + + pr_info("Test VGICv5 sparse vCPU IDs\n"); + test_vgic_v5_sparse_vcpu_ids(); + pr_info("Test VGICv5 PPIs\n"); test_vgic_v5_ppis(gic_dev_type); + + pr_info("Test VGICv5 SPIs\n"); + test_vgic_v5_spis(); + + pr_info("Test VGICv5 LPIs\n"); + test_vgic_v5_lpis(); + + pr_info("Test VGICv5 IST save/restore\n"); + test_vgic_v5_ist_save_restore(); } int main(int ac, char **av) { - int ret; + int pa_bits, ret; + + pa_bits = vm_guest_mode_params[VM_MODE_DEFAULT].pa_bits; + max_phys_size = 1ULL << pa_bits; test_disable_default_vgic(); diff --git a/tools/testing/selftests/kvm/include/arm64/gic_v5.h b/tools/testing/selftests/kvm/include/arm64/gic_v5.h index eb523d9277cf..2d455515563e 100644 --- a/tools/testing/selftests/kvm/include/arm64/gic_v5.h +++ b/tools/testing/selftests/kvm/include/arm64/gic_v5.h @@ -10,6 +10,106 @@ #include "processor.h" +/* GIC component base address is guest PA space */ +#define GICV5_IRS_CONFIG_BASE_GPA 0x8000000ULL +#define GICV5_IRS_CONFIG_BASE_GVA ((volatile void *)GICV5_IRS_CONFIG_BASE_GPA) + +#define GICV5_IRS_SIZE 0x20000 +#define GICV5_IRS_ALIGN 0x10000 + +/* + * IRS register offsets. Keep this selftest copy scoped to the offsets and + * fields exercised by the userspace API tests. + */ +#define GICV5_IRS_IDR0 0x0000 +#define GICV5_IRS_IDR1 0x0004 +#define GICV5_IRS_IDR2 0x0008 +#define GICV5_IRS_IDR3 0x000c +#define GICV5_IRS_IDR4 0x0010 +#define GICV5_IRS_IDR5 0x0014 +#define GICV5_IRS_IDR6 0x0018 +#define GICV5_IRS_IDR7 0x001c +#define GICV5_IRS_IIDR 0x0040 +#define GICV5_IRS_AIDR 0x0044 +#define GICV5_IRS_CR0 0x0080 +#define GICV5_IRS_CR1 0x0084 +#define GICV5_IRS_SYNCR 0x00c0 +#define GICV5_IRS_SYNC_STATUSR 0x00c4 +#define GICV5_IRS_SPI_VMR 0x0100 +#define GICV5_IRS_SPI_SELR 0x0108 +#define GICV5_IRS_SPI_DOMAINR 0x010c +#define GICV5_IRS_SPI_RESAMPLER 0x0110 +#define GICV5_IRS_SPI_CFGR 0x0114 +#define GICV5_IRS_SPI_STATUSR 0x0118 +#define GICV5_IRS_PE_SELR 0x0140 +#define GICV5_IRS_PE_STATUSR 0x0144 +#define GICV5_IRS_PE_CR0 0x0148 +#define GICV5_IRS_IST_BASER 0x0180 +#define GICV5_IRS_IST_CFGR 0x0190 +#define GICV5_IRS_IST_STATUSR 0x0194 +#define GICV5_IRS_MAP_L2_ISTR 0x01c0 +#define GICV5_IRS_VMT_BASER 0x0200 +#define GICV5_IRS_VMT_CFGR 0x0210 +#define GICV5_IRS_VMT_STATUSR 0x0214 +#define GICV5_IRS_VPE_SELR 0x0240 +#define GICV5_IRS_VPE_DBR 0x0248 +#define GICV5_IRS_VPE_HPPIR 0x0250 +#define GICV5_IRS_VPE_CR0 0x0258 +#define GICV5_IRS_VPE_STATUSR 0x025c +#define GICV5_IRS_VM_DBR 0x0280 +#define GICV5_IRS_VM_SELR 0x0288 +#define GICV5_IRS_VM_STATUSR 0x028c +#define GICV5_IRS_VMAP_L2_VMTR 0x02c0 +#define GICV5_IRS_VMAP_VMR 0x02c8 +#define GICV5_IRS_VMAP_VISTR 0x02d0 +#define GICV5_IRS_VMAP_L2_VISTR 0x02d8 +#define GICV5_IRS_VMAP_VPER 0x02e0 +#define GICV5_IRS_SAVE_VMR 0x0300 +#define GICV5_IRS_SAVE_VM_STATUSR 0x0308 +#define GICV5_IRS_MEC_IDR 0x0340 +#define GICV5_IRS_MEC_MECID_R 0x0344 +#define GICV5_IRS_MPAM_IDR 0x0380 +#define GICV5_IRS_MPAM_PARTID_R 0x0384 +#define GICV5_IRS_SWERR_STATUSR 0x03c0 +#define GICV5_IRS_SWERR_SYNDROMER0 0x03c8 +#define GICV5_IRS_SWERR_SYNDROMER1 0x03d0 + +#define GICV5_IRS_IDR0_SETLPI BIT(9) +#define GICV5_IRS_IDR0_INT_DOM GENMASK(1, 0) +#define GICV5_IRS_IDR0_INT_DOM_SECURE 0b00 + +#define GICV5_IRS_IDR1_PRIORITY_BITS GENMASK(22, 20) + +#define GICV5_IRS_IDR2_ISTMD_SZ GENMASK(19, 15) +#define GICV5_IRS_IDR2_ISTMD BIT(14) +#define GICV5_IRS_IDR2_IST_L2SZ GENMASK(13, 11) +#define GICV5_IRS_IDR2_IST_LEVELS BIT(10) +#define GICV5_IRS_IDR2_MIN_LPI_ID_BITS GENMASK(9, 6) +#define GICV5_IRS_IDR2_LPI BIT(5) +#define GICV5_IRS_IDR2_ID_BITS GENMASK(4, 0) + +#define GICV5_IRS_CR0_IDLE BIT(1) +#define GICV5_IRS_CR0_IRSEN BIT(0) + +#define GICV5_IRS_SPI_SELR_ID GENMASK(23, 0) +#define GICV5_IRS_SPI_CFGR_TM BIT(0) + +#define GICV5_IRS_SYNC_STATUSR_IDLE BIT(0) +#define GICV5_IRS_SPI_STATUSR_IDLE BIT(0) +#define GICV5_IRS_PE_STATUSR_IDLE BIT(0) +#define GICV5_IRS_IST_STATUSR_IDLE BIT(0) + +#define GICV5_IRS_IST_BASER_ADDR_MASK GENMASK_ULL(55, 6) +#define GICV5_IRS_IST_BASER_VALID BIT_ULL(0) +#define GICV5_IRS_IST_BASER_ADDR_SHIFT 6ULL +#define GICV5_IRS_IST_CFGR_STRUCTURE BIT(16) +#define GICV5_IRS_IST_CFGR_ISTSZ GENMASK(8, 7) +#define GICV5_IRS_IST_CFGR_L2SZ GENMASK(6, 5) +#define GICV5_IRS_IST_CFGR_LPI_ID_BITS GENMASK(4, 0) +#define GICV5_IRS_IST_CFGR_ISTSZ_4 0b00 +#define GICV5_IRS_IST_CFGR_ISTSZ_8 0b01 +#define GICV5_IRS_IST_CFGR_ISTSZ_16 0b10 + /* * Definitions for GICv5 instructions for the Current Domain */ @@ -77,6 +177,11 @@ #define GICV5_GICR_CDNMIA_TYPE_MASK GENMASK_ULL(31, 29) #define GICV5_GICR_CDNMIA_ID_MASK GENMASK_ULL(23, 0) +#define GICV5_HWIRQ_TYPE GENMASK_ULL(31, 29) +#define GICV5_HWIRQ_ID GENMASK_ULL(23, 0) +#define GICV5_HWIRQ_TYPE_LPI 0x2 +#define GICV5_HWIRQ_TYPE_SPI 0x3 + #define gicr_insn(insn) read_sysreg_s(GICV5_OP_GICR_##insn) #define gic_insn(v, insn) write_sysreg_s(v, GICV5_OP_GIC_##insn) diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index a6a3e1657895..76bb11bb79f3 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -172,6 +172,7 @@ struct vm_mem_backing_src_alias { bool thp_configured(void); size_t get_trans_hugepagesz(void); size_t get_def_hugetlb_pagesz(void); +size_t get_free_hugepages(size_t page_size); const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(u32 i); size_t get_backing_src_pagesz(u32 i); bool is_backing_src_hugetlb(u32 i); diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index 4dff72f6bd34..f9263a715704 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -255,6 +255,21 @@ size_t get_def_hugetlb_pagesz(void) TEST_FAIL("Error in reading /proc/meminfo"); } +size_t get_free_hugepages(size_t page_size) +{ + char path[128]; + size_t free; + + snprintf(path, sizeof(path), + "/sys/kernel/mm/hugepages/hugepages-%zukB/free_hugepages", + page_size >> 10); + if (!test_sysfs_path(path)) + return 0; + + free = get_sysfs_val(path); + return free * page_size; +} + #define ANON_FLAGS (MAP_PRIVATE | MAP_ANONYMOUS) #define ANON_HUGE_FLAGS (ANON_FLAGS | MAP_HUGETLB) diff --git a/tools/testing/selftests/kvm/pre_fault_memory_test.c b/tools/testing/selftests/kvm/pre_fault_memory_test.c index c57631aab3d3..3082fe09b95f 100644 --- a/tools/testing/selftests/kvm/pre_fault_memory_test.c +++ b/tools/testing/selftests/kvm/pre_fault_memory_test.c @@ -12,19 +12,29 @@ #include <processor.h> #include <pthread.h> #include <ucall_common.h> +#include <guest_modes.h> /* Arbitrarily chosen values */ -#define TEST_SIZE (SZ_2M + PAGE_SIZE) -#define TEST_NPAGES (TEST_SIZE / PAGE_SIZE) +#define TEST_BASE_SIZE SZ_2M #define TEST_SLOT 10 +/* Storage of test info to share with guest code */ +struct test_config { + u64 page_size; + u64 test_size; + u64 test_num_pages; +}; + +static struct test_config test_config; + static void guest_code(u64 base_gva) { volatile u64 val __used; + struct test_config *config = &test_config; int i; - for (i = 0; i < TEST_NPAGES; i++) { - u64 *src = (u64 *)(base_gva + i * PAGE_SIZE); + for (i = 0; i < config->test_num_pages; i++) { + u64 *src = (u64 *)(base_gva + i * config->page_size); val = *src; } @@ -36,6 +46,7 @@ struct slot_worker_data { struct kvm_vm *vm; gpa_t gpa; u32 flags; + enum vm_mem_backing_src_type mem_backing_src; bool worker_ready; bool prefault_ready; bool recreate_slot; @@ -56,14 +67,16 @@ static void *delete_slot_worker(void *__data) while (!READ_ONCE(data->recreate_slot)) cpu_relax(); - vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, data->gpa, - TEST_SLOT, TEST_NPAGES, data->flags); + vm_userspace_mem_region_add(vm, data->mem_backing_src, data->gpa, + TEST_SLOT, test_config.test_num_pages, data->flags); return NULL; } static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, - u64 size, u64 expected_left, bool private) + u64 size, u64 expected_left, + enum vm_mem_backing_src_type mem_backing_src, + bool private) { struct kvm_pre_fault_memory range = { .gpa = base_gpa + offset, @@ -74,6 +87,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, .vm = vcpu->vm, .gpa = base_gpa, .flags = private ? KVM_MEM_GUEST_MEMFD : 0, + .mem_backing_src = mem_backing_src, }; bool slot_recreated = false; pthread_t slot_worker; @@ -150,8 +164,8 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, /* * Assert success if prefaulting the entire range should succeed, i.e. * complete with no bytes remaining. Otherwise prefaulting should have - * failed due to ENOENT (due to RET_PF_EMULATE for emulated MMIO when - * no memslot exists). + * failed due to ENOENT (no memslot exists for the GPA; on x86 this + * surfaces via RET_PF_EMULATE). */ if (!expected_left) TEST_ASSERT_VM_VCPU_IOCTL(!ret, KVM_PRE_FAULT_MEMORY, ret, vcpu->vm); @@ -160,39 +174,82 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, KVM_PRE_FAULT_MEMORY, ret, vcpu->vm); } -static void __test_pre_fault_memory(unsigned long vm_type, bool private) +struct test_params { + unsigned long vm_type; + bool private; + enum vm_mem_backing_src_type mem_backing_src; +}; + +static void __test_pre_fault_memory(enum vm_guest_mode guest_mode, void *arg) { - gpa_t gpa, gva, alignment, guest_page_size; + gpa_t gpa, gva, alignment, guest_page_size, host_page_size; + gpa_t backing_src_pagesz, mem_page_size; + struct test_params *p = arg; const struct vm_shape shape = { - .mode = VM_MODE_DEFAULT, - .type = vm_type, + .mode = guest_mode, + .type = p->vm_type, }; struct kvm_vcpu *vcpu; + struct kvm_run *run; struct kvm_vm *vm; struct ucall uc; + pr_info("Testing guest mode: %s\n", vm_guest_mode_string(guest_mode)); + pr_info("Testing memory backing src type: %s\n", + vm_mem_backing_src_alias(p->mem_backing_src)->name); + vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code); - alignment = guest_page_size = vm_guest_mode_params[VM_MODE_DEFAULT].page_size; - gpa = (vm->max_gfn - TEST_NPAGES) * guest_page_size; + guest_page_size = vm_guest_mode_params[guest_mode].page_size; + host_page_size = getpagesize(); + backing_src_pagesz = get_backing_src_pagesz(p->mem_backing_src); + mem_page_size = max(host_page_size, backing_src_pagesz); + + test_config.page_size = guest_page_size; + test_config.test_size = align_up(TEST_BASE_SIZE + test_config.page_size, + mem_page_size); + test_config.test_num_pages = vm_calc_num_guest_pages(vm->mode, test_config.test_size); + + gpa = (vm->max_gfn - test_config.test_num_pages) * test_config.page_size; alignment = SZ_2M; + alignment = max(alignment, mem_page_size); gpa = align_down(gpa, alignment); gva = gpa & ((1ULL << (vm->va_bits - 1)) - 1); - vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa, TEST_SLOT, - TEST_NPAGES, private ? KVM_MEM_GUEST_MEMFD : 0); - virt_map(vm, gva, gpa, TEST_NPAGES); + vm_userspace_mem_region_add(vm, p->mem_backing_src, + gpa, TEST_SLOT, test_config.test_num_pages, + p->private ? KVM_MEM_GUEST_MEMFD : 0); + virt_map(vm, gva, gpa, test_config.test_num_pages); + + if (p->private) + vm_mem_set_private(vm, gpa, test_config.test_size); + + pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0, + p->mem_backing_src, p->private); + /* Retry the same range after the first prefault attempt. */ + pre_fault_memory(vcpu, gpa, 0, test_config.test_size, 0, + p->mem_backing_src, p->private); + pre_fault_memory(vcpu, gpa, + test_config.test_size - host_page_size, + host_page_size * 2, host_page_size, + p->mem_backing_src, p->private); + pre_fault_memory(vcpu, gpa, test_config.test_size, + host_page_size, host_page_size, + p->mem_backing_src, p->private); - if (private) - vm_mem_set_private(vm, gpa, TEST_SIZE); + vcpu_args_set(vcpu, 1, gva); - pre_fault_memory(vcpu, gpa, 0, SZ_2M, 0, private); - pre_fault_memory(vcpu, gpa, SZ_2M, PAGE_SIZE * 2, PAGE_SIZE, private); - pre_fault_memory(vcpu, gpa, TEST_SIZE, PAGE_SIZE, PAGE_SIZE, private); + /* Export the shared variables to the guest. */ + sync_global_to_guest(vm, test_config); - vcpu_args_set(vcpu, 1, gva); vcpu_run(vcpu); + run = vcpu->run; + TEST_ASSERT(run->exit_reason == UCALL_EXIT_REASON, + "Wanted %s, got exit reason: %u (%s)", + exit_reason_str(UCALL_EXIT_REASON), + run->exit_reason, exit_reason_str(run->exit_reason)); + switch (get_ucall(vcpu, &uc)) { case UCALL_ABORT: REPORT_GUEST_ASSERT(uc); @@ -207,24 +264,61 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) kvm_vm_free(vm); } -static void test_pre_fault_memory(unsigned long vm_type, bool private) +static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_src_type backing_src, + bool private) { + struct test_params p = { + .vm_type = vm_type, + .private = private, + .mem_backing_src = backing_src, + }; + if (vm_type && !(kvm_check_cap(KVM_CAP_VM_TYPES) & BIT(vm_type))) { pr_info("Skipping tests for vm_type 0x%lx\n", vm_type); return; } - __test_pre_fault_memory(vm_type, private); + for_each_guest_mode(__test_pre_fault_memory, &p); +} + +static void help(char *name) +{ + puts(""); + printf("usage: %s [-h] [-m mode] [-s mem-type]\n", name); + puts(""); + guest_modes_help(); + backing_src_help("-s"); + puts(""); } int main(int argc, char *argv[]) { + enum vm_mem_backing_src_type backing = DEFAULT_VM_MEM_SRC; + int opt; + + guest_modes_append_default(); + + while ((opt = getopt(argc, argv, "hm:s:")) != -1) { + switch (opt) { + case 'm': + guest_modes_cmdline(optarg); + break; + case 's': + backing = parse_backing_src_type(optarg); + break; + case 'h': + default: + help(argv[0]); + exit(0); + } + } + TEST_REQUIRE(kvm_check_cap(KVM_CAP_PRE_FAULT_MEMORY)); - test_pre_fault_memory(0, false); + test_pre_fault_memory(0, backing, false); #ifdef __x86_64__ - test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, false); - test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, true); + test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, false); + test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, true); #endif return 0; } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ba546c1a92f5..ed4340c62518 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -472,7 +472,7 @@ static void kvm_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_dirty_ring_free(&vcpu->dirty_ring); /* - * No need for rcu_read_lock as VCPU_RUN is the only place that changes + * No need for rcu_read_lock as KVM_RUN is the only place that changes * the vcpu->pid pointer, and at destruction time all file descriptors * are already gone. */ @@ -4367,7 +4367,7 @@ static int kvm_vcpu_ioctl_get_stats_fd(struct kvm_vcpu *vcpu) static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range) { - int idx; + int idx, err; long r; u64 full_size; @@ -4379,6 +4379,14 @@ static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, range->gpa + range->size <= range->gpa) return -EINVAL; + /* + * Certain architectures (e.g. arm64) need to reject the ioctl 'early' + * before vcpu_load(). + */ + err = kvm_arch_pre_fault_allowed(vcpu); + if (err) + return err; + vcpu_load(vcpu); idx = srcu_read_lock(&vcpu->kvm->srcu); |