diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-20 07:55:19 -1000 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-20 07:55:19 -1000 |
| commit | 303bff3637d13f0ecbb280c7cb403470c440a2a6 (patch) | |
| tree | 6023750f7a533ec10ad7ed440d1cb529b6165dce | |
| parent | c3134590a8c51b34ce66fa471ffa4b4fe38daa1f (diff) | |
| parent | d0de3a73194a250a2e5ffbf1dc604c0bcbb16e8a (diff) | |
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 updates from Catalin Marinas:
"Here are the arm64 updates for 7.4.
The more interesting bits are the removal of big-endian kernel support
(already marked BROKEN for some time), preemptible this_cpu_*()
operations and MPAM support for the firmware-backed (Fb) interface.
CCA (Arm's confidential compute) is slowly trickling in with host RMI
(Realm Management Interface) support and some refactoring, including a
dedicated SMCCC bus to advertise firmware services. There are also the
usual PMU updates for new hardware and a few fixes/cleanups.
Core features:
- Remove big-endian kernel support: CONFIG_CPU_BIG_ENDIAN has
depended on BROKEN for a while, so drop the remaining code (BE
hypervisor object, BPF JIT, assembly/crypto/library routines,
endianness helper macros, selftests) and force a little-endian
build
- Add preemptible this_cpu_*() operations (read/write, RMW, xchg,
cmpxchg and cmpxchg128) that no longer need to disable preemption
around the per-CPU address generation and access
- Preparation patches for implementing CONFIG_HOTPLUG_PARALLEL on
arm64
Memory management:
- Treat faults taken by EFI runtime firmware as kernel faults
- Clear PTE_WRITE for EFI_MEMORY_RO mappings (with hardware dirty
state management enabled, this is not truly read-only)
- Raise the default ARCH_MMAP_RND_BITS for 48-bit and 52-bit VA
configurations (28/26/24 bits for 4K/16K/64K pages)
- Implement clear_page[s]() using memset()
- Enable BBML2 no-abort (a.k.a. BBML3) for the NVIDIA Rigel CPU
Perf and PMU:
- Support for HiSilicon's ITS PMU
- Add support for some new events: INST_SIMD_ALU_VEC for Apple PMUs
and a cycle counter for the latest version of HiSilicon's MN PMU
hardware
- A bunch of fixes to the CXL PMU driver to address pre-existing
issues from Sashiko
- Fix generation of PERF_RECORD_ITRACE_START records for SPE and ETM
drivers
- Reduce excessive stack consumption in the Marvell DDR PMU driver
- Minor driver updates and cleanups all over
Arm Confidential Compute Architecture (CCA):
- Add an Arm SMCCC bus for firmware services discovered through SMCCC
and move the TRNG and CCA guest (TSM provider) drivers onto it,
removing the dummy CCA platform device and adding a sysfs ABI for
Realm guests
- Move the Realm Service Interface (RSI) code out of arch/arm64/ into
drivers/firmware/arm_rmm/
- Add the host side of the Realm Management Interface (RMI): detect,
configure and activate the RMM, add SRO support and wrappers for
the Realm RMI commands
- Block hibernation (new arch hook in the PM core) and kexec while
the RMM is active. To be revisited in the future with proper
support for deactivation
- Handle Granule Protection Faults (GPFs) taken by the host, caused
by accesses to memory delegated to the RMM. In a future patch, the
exception table lookup will be limited to is_kdump_kernel() cases
MPAM (Memory Partitioning And Monitoring):
- Add support for the MPAM firmware-backed (Fb) interface, allowing
MSC accesses to be mediated by firmware
- Fix a list of theoretical issues with the driver (MSC access error
propagation, MBWU monitor save/restore and locking, resctrl class
handling, error interrupt restore, maximum number of PARTIDs)
- Add the mailing list and a missing file to the MAINTAINERS entry
- Document when to set arm64.nompam
Errata and CPU features:
- Add the NXP i.MX8QM/8QP workaround for the Cortex-A53 cache
coherency issue (ERR050104), using a new ARM64_CPUCAP_SOC_ERRATUM
capability type for SoC-level errata
Miscellaneous:
- Mask debug exceptions (DAIF) around the hardware breakpoint
register manipulations when called from generic kernel code
- Don't zero the SVE state buffer when the SVE state is live
- Return -EPERM instead of -EBUSY from the GCS prctl() locking
failures (matching x86), with the selftest updated. Slight ABI
change but no software deployment yet
- kselftest: fix unsound MTE fault skipping by introducing MTE-safe
memory accessors, avoid copying each signal test binary more than
once
- Update sysreg definitions (fix three enum values and two field
names, mark architecturally dropped fields as RES0) and remove the
TME transaction state check from the BRBE driver
- Fix the description of ARM64_HAS_LPA2 and remove a redundant
ID_AA64MMFR0 define"
* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (144 commits)
arm64: cpufeature: Enable BBML2 for NVIDIA Rigel
arm64: cputype: Add NVIDIA Rigel definitions
arm64: raise default ARCH_MMAP_RND_BITS for 48-bit and 52-bit VA
arm64/debug: mask debug exceptions when called from common code
kselftest/arm64: signal: copy each test binary only once
kselftest/arm64: Check for -EPERM not -EBUSY in the locking test
arm64: gcs: Return -EPERM not -EBUSY for prctl() locking failures
perf: arm_brbe: Remove the check for TME transaction states
arm64: sysreg: Mark fields the architecture dropped as RES0
arm64: sysreg: Fix the names of two misspelt fields
arm64: sysreg: Fix three incorrect enum values
arm64: smp: Defer RCU registration during secondary CPU bringup
arm64: smp: Don't bother printing the I-cache policy for each CPU
arm64: smp: Defer update of secondary CPU capabilities
arm64: smp: Tidy up cpuinfo init and cpufeature updates
arm64: smp: Tidy up smp_prepare_cpus()
arm64: cpufeature: Store every ID register with its overrides applied
arm64: cpufeature: Read MPAMIDR_EL1 in __cpuinfo_store_cpu()
arm64: cpufeature: Add read_cpuid_with_overrides()
arm64: cpufeature: Check arm64_ftr_regs[] before the first store
...
169 files changed, 5822 insertions, 1962 deletions
diff --git a/Documentation/ABI/testing/sysfs-firmware-cca b/Documentation/ABI/testing/sysfs-firmware-cca new file mode 100644 index 000000000000..d545b7523530 --- /dev/null +++ b/Documentation/ABI/testing/sysfs-firmware-cca @@ -0,0 +1,7 @@ +What: /sys/firmware/cca/realm_guest +Date: September 2026 +Contact: Linux ARM Kernel Mailing list <linux-arm-kernel@lists.infradead.org> +Description: Read-only. Present only when the kernel is running as an + Arm Confidential Compute Architecture (CCA) Realm guest, in + which case it reads 1. If the file does not exist, the kernel + is not running as a Realm guest. diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index e75344f4e0cd..df4021c2dbe0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -575,7 +575,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension support diff --git a/Documentation/admin-guide/perf/hisi-pmu.rst b/Documentation/admin-guide/perf/hisi-pmu.rst index d56b2d690709..8cb4711c9c37 100644 --- a/Documentation/admin-guide/perf/hisi-pmu.rst +++ b/Documentation/admin-guide/perf/hisi-pmu.rst @@ -128,6 +128,22 @@ channel with this option. The current supported channels are as follows: 7. tt_en: NoC PMU supports counting only transactions that have tracetag set if this option is set. See the 2nd list for more information about tracetag. +8. int_id: ITS PMU supports filtering by interrupt id, which is defined by +hardware. Interrupt id takes up to 32 bits, and can be divided into 2 parts: + +- Upper 16 bits: DeviceID if counting LPI, PEID if counting SGI/PPI. +- Lower 16 bits: EventID if counting LPI, IntID if counting SGI/PPI. + +int_id is a global configuration for each PMU instance: the filter applies to +all counters, so the PMU can only run events that agree on it. An unfiltered +event (int_en not set) cannot run concurrently with a filtered one, and two +filtered events can run concurrently only if they filter the same int_id. +perf time-multiplexes any events that cannot coexist, rather than corrupting +the counts. + +9. int_en: A one-bit flag to tell if int_id is used to filter the statistics. It +allows filtering 0 DeviceID and EventID. + For HiSilicon uncore PMU v3 whose identifier is 0x40, some uncore PMUs are further divided into parts for finer granularity of tracing, each part has its own dedicated PMU, and all such PMUs together cover the monitoring job of events diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst index 3fea4b14ef7c..dae5ff9fae26 100644 --- a/Documentation/arch/arm64/booting.rst +++ b/Documentation/arch/arm64/booting.rst @@ -589,7 +589,7 @@ following manner: - CPUs with a "spin-table" enable-method must have a 'cpu-release-addr' property in their cpu node. This property identifies a - naturally-aligned 64-bit zero-initalised memory location. + naturally-aligned 64-bit zero-initialised memory location. These CPUs should spin outside of the kernel in a reserved area of memory (communicated to the kernel by a /memreserve/ region in the diff --git a/Documentation/arch/arm64/mpam.rst b/Documentation/arch/arm64/mpam.rst index 67fe515ed501..e445f4078ee1 100644 --- a/Documentation/arch/arm64/mpam.rst +++ b/Documentation/arch/arm64/mpam.rst @@ -87,6 +87,32 @@ The supported features are: MBWU monitors can be exposed to the user after support for more monitoring scopes is added to resctrl. +Command line parameters +======================= + +arm64.nompam +------------ +Firmware controls MPAM through two bits of MPAM3_EL3. MPAMEN enables +it: while set, the PARTID and PMG in the MPAMn_ELx registers label the +CPU's memory requests. TRAPLOWER, which resets to 1, traps accesses to +the MPAM system registers from the lower exception levels to EL3. +Firmware must clear it, or handle the trap and emulate MPAM as disabled. +Where it does neither, the CPUs still advertise MPAM in the ID registers, +the kernel's MPAM register accesses trap to EL3, and the boot fails. +``arm64.nompam`` exists for that firmware: it makes the kernel treat the +CPUs as not implementing MPAM, so no MPAM system register is accessed. +Set it only on a machine that does not boot without it. + +It is not a way to turn MPAM off. Where firmware has cleared TRAPLOWER, +the option leaves the trap controls in MPAM2_EL2 and MPAMHCR_EL2 +unwritten, and their reset values are UNKNOWN. KVM still hides MPAM from +guests but no longer enables the traps that stop a guest from using it, +so a guest may be able to read and write MPAM0_EL1, MPAM1_EL1 and +MPAMSM_EL1. KVM does not save or restore them, so what one guest writes +is still there when the next guest runs on that CPU, and when the host +does. With MPAMEN set, EL0 and EL1 requests then carry that PARTID and +PMG. + Reporting Bugs ============== If you are not seeing the counters or controls you expect please share the diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index 99a1eb4b833d..bd7e6c5a17b7 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -329,6 +329,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+ +| Freescale/NXP | i.MX 8QM/8QP | ERR050104 | NXP_IMX8QM_ERRATUM_ERR050104| ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Hisilicon | Hip0{5,6,7} | #161010101 | HISILICON_ERRATUM_161010101 | +----------------+-----------------+-----------------+-----------------------------+ diff --git a/MAINTAINERS b/MAINTAINERS index 54b5106ed4e3..56c38a9fdc93 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3949,10 +3949,16 @@ R: Mark Rutland <mark.rutland@arm.com> L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git +F: Documentation/ABI/testing/sysfs-firmware-cca F: Documentation/arch/arm64/ F: arch/arm64/ +F: drivers/firmware/arm_rmm/ F: drivers/virt/coco/arm-cca-guest/ F: drivers/virt/coco/pkvm-guest/ +F: include/linux/arm-rmi-cmds.h +F: include/linux/arm-rsi-cmds.h +F: include/linux/arm-smccc-rmi.h +F: include/linux/arm-smccc-rsi.h F: tools/testing/selftests/arm64/ X: arch/arm64/boot/dts/ X: arch/arm64/configs/defconfig @@ -18446,7 +18452,9 @@ M: James Morse <james.morse@arm.com> M: Ben Horgan <ben.horgan@arm.com> R: Reinette Chatre <reinette.chatre@intel.com> R: Fenghua Yu <fenghuay@nvidia.com> +L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained +F: Documentation/arch/arm64/mpam.rst F: drivers/resctrl/mpam_* F: drivers/resctrl/test_mpam_* F: include/linux/arm_mpam.h @@ -24756,7 +24764,9 @@ M: Sudeep Holla <sudeep.holla@kernel.org> L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: drivers/firmware/smccc/ +F: include/linux/arm-smccc-bus.h F: include/linux/arm-smccc.h +F: include/linux/device-id/arm_smccc.h SECURITY CONTACT M: Security Officers <security@kernel.org> diff --git a/arch/arm/include/asm/archrandom.h b/arch/arm/include/asm/archrandom.h index cc4714eb1a75..ee39a03ddf8a 100644 --- a/arch/arm/include/asm/archrandom.h +++ b/arch/arm/include/asm/archrandom.h @@ -2,7 +2,7 @@ #ifndef _ASM_ARCHRANDOM_H #define _ASM_ARCHRANDOM_H -static inline bool __init smccc_probe_trng(void) +static inline bool smccc_probe_trng(void) { return false; } diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cab741a695f5..6a38cb1ca3a1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -73,7 +73,7 @@ config ARM64 select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_MEMORY_FAILURE select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK - select ARCH_SUPPORTS_LTO_CLANG if CPU_LITTLE_ENDIAN + select ARCH_SUPPORTS_LTO_CLANG select ARCH_SUPPORTS_LTO_CLANG_THIN select ARCH_SUPPORTS_CFI select ARCH_SUPPORTS_ATOMIC_RMW @@ -83,6 +83,7 @@ config ARM64 select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_RMM select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_SCHED_SMT select ARCH_SUPPORTS_SCHED_CLUSTER @@ -224,7 +225,7 @@ config ARM64 select HAVE_FUNCTION_ARG_ACCESS_API select MMU_GATHER_RCU_TABLE_FREE select HAVE_RSEQ - select HAVE_RUST if RUSTC_SUPPORTS_ARM64 + select HAVE_RUST select HAVE_STACKPROTECTOR select HAVE_STATIC_CALL if CFI select HAVE_SYSCALL_TRACEPOINTS @@ -263,10 +264,6 @@ config ARM64 help ARM 64-bit (AArch64) Linux support. -config RUSTC_SUPPORTS_ARM64 - def_bool y - depends on CPU_LITTLE_ENDIAN - config CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS def_bool CC_IS_CLANG # https://github.com/ClangBuiltLinux/linux/issues/1507 @@ -313,6 +310,14 @@ config ARCH_MMAP_RND_BITS_MAX default 16 if ARM64_16K_PAGES default 18 +# default to 1 TiB (5 bits below ARCH_MMAP_RND_BITS_MAX) on 48-bit and +# 52-bit VA; smaller VA sizes fall back to ARCH_MMAP_RND_BITS_MIN in +# arch/Kconfig +config ARCH_MMAP_RND_BITS + default 24 if (ARM64_VA_BITS=48 || ARM64_VA_BITS=52) && ARM64_64K_PAGES + default 26 if (ARM64_VA_BITS=48 || ARM64_VA_BITS=52) && ARM64_16K_PAGES + default 28 if (ARM64_VA_BITS=48 || ARM64_VA_BITS=52) + config ARCH_MMAP_RND_COMPAT_BITS_MIN default 7 if ARM64_64K_PAGES default 9 if ARM64_16K_PAGES @@ -1485,6 +1490,28 @@ config ROCKCHIP_ERRATUM_3588001 If unsure, say Y. +config NXP_IMX8QM_ERRATUM_ERR050104 + bool "NXP iMX8QM/QP ERR050104: broken cache/TLB invalidation broadcast" + default y + help + On iMX8QM (QuadMax) and iMX8QP (QuadPlus), address bits above bit 35 + in the A53 subsystem ARADDR and ACADDR buses are incorrectly + connected. This corrupts targeted TLBI and IC broadcasts exchanged + between the A53 and A72 core clusters. + + Work around this by redirecting targeted TLBI operations to + broadcast-all variants (VMALLE1IS) in the high-level TLB flush + functions, upgrading IC IVAU to IC IALLUIS for both kernel and + user-space, and disabling KVM since correct TLB maintenance + cannot be guaranteed for guests. + + The user-space IC IVAU is upgraded by trapping it to the kernel + (SCTLR_EL1.UCI cleared). This also traps the user DC CVAU, DC CVAC, + DC CVAP, DC CVADP and DC CIVAC instructions, making user cache + maintenance slower on the affected SoCs. + + If unsure, say Y. + config SOCIONEXT_SYNQUACER_PREITS bool "Socionext Synquacer: Workaround for GICv3 pre-ITS" default y @@ -1626,28 +1653,6 @@ config ARM64_LPA2 def_bool y depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES -choice - prompt "Endianness" - default CPU_LITTLE_ENDIAN - help - Select the endianness of data accesses performed by the CPU. Userspace - applications will need to be compiled and linked for the endianness - that is selected here. - -config CPU_BIG_ENDIAN - bool "Build big-endian kernel" - depends on BROKEN - help - Say Y if you plan on running a kernel with a big-endian userspace. - -config CPU_LITTLE_ENDIAN - bool "Build little-endian kernel" - help - Say Y if you plan on running a kernel with a little-endian userspace. - This is usually the case for distributions targeting arm64. - -endchoice - config NR_CPUS int "Maximum number of CPUs (2-4096)" range 2 4096 @@ -1885,7 +1890,6 @@ config KUSER_HELPERS config COMPAT_VDSO bool "Enable vDSO for 32-bit applications" - depends on !CPU_BIG_ENDIAN depends on (CC_IS_CLANG && LD_IS_LLD) || "$(CROSS_COMPILE_COMPAT)" != "" default y help @@ -2349,7 +2353,7 @@ menu "ARMv9.6 architectural features" config ARM64_LSUI bool "Support Unprivileged Load Store Instructions (LSUI)" default y - depends on AS_HAS_LSUI && !CPU_BIG_ENDIAN + depends on AS_HAS_LSUI help The Unprivileged Load Store Instructions (LSUI) provides variants load/store instructions that access user-space memory @@ -2554,7 +2558,7 @@ config EFI_STUB config EFI bool "UEFI runtime support" - depends on OF && !CPU_BIG_ENDIAN + depends on OF depends on KERNEL_MODE_NEON select ARCH_SUPPORTS_ACPI select LIBFDT diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6b005c8fef70..04e1a0d12f8f 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -119,20 +119,12 @@ KBUILD_CFLAGS += -ffixed-x18 KBUILD_RUSTFLAGS += -Zfixed-x18 endif -ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) -KBUILD_CPPFLAGS += -mbig-endian -CHECKFLAGS += -D__AARCH64EB__ -# Prefer the baremetal ELF build target, but not all toolchains include -# it so fall back to the standard linux version if needed. -KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb -z norelro) -UTS_MACHINE := aarch64_be -else KBUILD_CPPFLAGS += -mlittle-endian CHECKFLAGS += -D__AARCH64EL__ -# Same as above, prefer ELF but fall back to linux target if needed. +# Prefer the baremetal ELF build target, but not all toolchains include +# it so fall back to the standard linux version if needed. KBUILD_LDFLAGS += -EL $(call ld-option, -maarch64elf, -maarch64linux -z norelro) UTS_MACHINE := aarch64 -endif ifeq ($(CONFIG_LD_IS_LLD), y) KBUILD_LDFLAGS += -z norelro diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S index f2624238fd95..c2e8f500da14 100644 --- a/arch/arm64/crypto/aes-ce-ccm-core.S +++ b/arch/arm64/crypto/aes-ce-ccm-core.S @@ -50,7 +50,7 @@ ld1 {v0.16b}, [x5] /* load mac */ cbz x2, ce_aes_ccm_final ldr x8, [x6, #8] /* load lower ctr */ -CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ + rev x8, x8 /* keep swabbed ctr in reg */ 0: /* outer loop */ ld1 {v1.8b}, [x6] /* load upper ctr */ prfm pldl1strm, [x1] @@ -73,7 +73,7 @@ CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ eor v0.16b, v0.16b, v2.16b /* xor mac with pt ^ rk[last] */ st1 {v6.16b}, [x0], #16 /* write output block */ bne 0b -CPU_LE( rev x8, x8 ) + rev x8, x8 str x8, [x6, #8] /* store lsb end of ctr (BE) */ cbnz x7, ce_aes_ccm_final st1 {v0.16b}, [x5] /* store mac */ diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S index baf450717b24..ba1f981a5464 100644 --- a/arch/arm64/crypto/aes-neonbs-core.S +++ b/arch/arm64/crypto/aes-neonbs-core.S @@ -824,8 +824,8 @@ SYM_FUNC_START(aesbs_ctr_encrypt) frame_push 0 ldp x7, x8, [x5] ld1 {v0.16b}, [x5] -CPU_LE( rev x7, x7 ) -CPU_LE( rev x8, x8 ) + rev x7, x7 + rev x8, x8 adds x8, x8, #1 adc x7, x7, xzr diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S index 33772d8fe6b5..ba350d57fc76 100644 --- a/arch/arm64/crypto/ghash-ce-core.S +++ b/arch/arm64/crypto/ghash-ce-core.S @@ -148,7 +148,7 @@ SYM_FUNC_START(pmull_ghash_update_p64) sub w0, w0, #1 3: /* multiply XL by SHASH in GF(2^128) */ -CPU_LE( rev64 T1.16b, T1.16b ) + rev64 T1.16b, T1.16b ext T2.16b, XL.16b, XL.16b, #8 ext IN1.16b, T1.16b, T1.16b, #8 @@ -272,7 +272,7 @@ SYM_FUNC_END(pmull_ghash_update_p64) cbz x0, 3f // tag only? ldr w8, [x5, #12] // load lower counter -CPU_LE( rev w8, w8 ) + rev w8, w8 0: mov w9, #4 // max blocks per round add x10, x0, #0xf @@ -380,7 +380,7 @@ CPU_LE( rev w8, w8 ) ret 5: -CPU_LE( rev w8, w8 ) + rev w8, w8 str w8, [x5, #12] // store lower counter st1 {XL.2d}, [x4] b 4b diff --git a/arch/arm64/crypto/sm4-ce-cipher-core.S b/arch/arm64/crypto/sm4-ce-cipher-core.S index 4ac6cfbc5797..1682d727e097 100644 --- a/arch/arm64/crypto/sm4-ce-cipher-core.S +++ b/arch/arm64/crypto/sm4-ce-cipher-core.S @@ -18,7 +18,7 @@ SYM_FUNC_START(sm4_ce_do_crypt) ld1 {v8.4s}, [x2] ld1 {v0.4s-v3.4s}, [x0], #64 -CPU_LE( rev32 v8.16b, v8.16b ) + rev32 v8.16b, v8.16b ld1 {v4.4s-v7.4s}, [x0] sm4e v8.4s, v0.4s sm4e v8.4s, v1.4s @@ -30,7 +30,7 @@ CPU_LE( rev32 v8.16b, v8.16b ) sm4e v8.4s, v7.4s rev64 v8.4s, v8.4s ext v8.16b, v8.16b, v8.16b, #8 -CPU_LE( rev32 v8.16b, v8.16b ) + rev32 v8.16b, v8.16b st1 {v8.4s}, [x1] ret SYM_FUNC_END(sm4_ce_do_crypt) diff --git a/arch/arm64/include/asm/alternative-macros.h b/arch/arm64/include/asm/alternative-macros.h index 862416624852..ee75a2f1a575 100644 --- a/arch/arm64/include/asm/alternative-macros.h +++ b/arch/arm64/include/asm/alternative-macros.h @@ -53,8 +53,7 @@ * * Alternatives with callbacks do not generate replacement instructions. */ -#define __ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, cfg_enabled) \ - ".if "__stringify(cfg_enabled)" == 1\n" \ +#define __ALTERNATIVE(oldinstr, newinstr, cpucap) \ "661:\n\t" \ oldinstr "\n" \ "662:\n" \ @@ -68,10 +67,8 @@ ".org . - (664b-663b) + (662b-661b)\n\t" \ ".org . - (662b-661b) + (664b-663b)\n\t" \ ".previous\n" \ - ".endif\n" -#define __ALTERNATIVE_CFG_CB(oldinstr, cpucap, cfg_enabled, cb) \ - ".if "__stringify(cfg_enabled)" == 1\n" \ +#define __ALTERNATIVE_CB(oldinstr, cpucap, cb) \ "661:\n\t" \ oldinstr "\n" \ "662:\n" \ @@ -80,13 +77,12 @@ ".popsection\n" \ "663:\n\t" \ "664:\n\t" \ - ".endif\n" -#define _ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, cfg, ...) \ - __ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, IS_ENABLED(cfg)) +#define _ALTERNATIVE(oldinstr, newinstr, cpucap) \ + __ALTERNATIVE(oldinstr, newinstr, cpucap) #define ALTERNATIVE_CB(oldinstr, cpucap, cb) \ - __ALTERNATIVE_CFG_CB(oldinstr, (1 << ARM64_CB_SHIFT) | (cpucap), 1, cb) + __ALTERNATIVE_CB(oldinstr, (1 << ARM64_CB_SHIFT) | (cpucap), cb) #else #include <asm/assembler.h> @@ -99,8 +95,7 @@ .byte \alt_len .endm -.macro alternative_insn insn1, insn2, cap, enable = 1 - .if \enable +.macro alternative_insn insn1, insn2, cap 661: \insn1 662: .pushsection .altinstructions, "a" altinstruction_entry 661b, 663f, \cap, 662b-661b, 664f-663f @@ -110,7 +105,6 @@ 664: .org . - (664b-663b) + (662b-661b) .org . - (662b-661b) + (664b-663b) .previous - .endif .endm /* @@ -204,20 +198,16 @@ alternative_else alternative_endif .endm -#define _ALTERNATIVE_CFG(insn1, insn2, cap, cfg, ...) \ - alternative_insn insn1, insn2, cap, IS_ENABLED(cfg) +#define _ALTERNATIVE(insn1, insn2, cap) \ + alternative_insn insn1, insn2, cap #endif /* __ASSEMBLER__ */ /* * Usage: asm(ALTERNATIVE(oldinstr, newinstr, cpucap)); - * - * Usage: asm(ALTERNATIVE(oldinstr, newinstr, cpucap, CONFIG_FOO)); - * N.B. If CONFIG_FOO is specified, but not selected, the whole block - * will be omitted, including oldinstr. */ -#define ALTERNATIVE(oldinstr, newinstr, ...) \ - _ALTERNATIVE_CFG(oldinstr, newinstr, __VA_ARGS__, 1) +#define ALTERNATIVE(oldinstr, newinstr, cpucap) \ + _ALTERNATIVE(oldinstr, newinstr, cpucap) #ifndef __ASSEMBLER__ diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index 8babfbe31f95..7605dd81bd1e 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -12,7 +12,7 @@ extern bool smccc_trng_available; -static inline bool __init smccc_probe_trng(void) +static inline bool smccc_probe_trng(void) { struct arm_smccc_res res; diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0b58b550e8dc..020670814bda 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -130,37 +130,6 @@ lr .req x30 // link register .endm /* - * Select code when configured for BE. - */ -#ifdef CONFIG_CPU_BIG_ENDIAN -#define CPU_BE(code...) code -#else -#define CPU_BE(code...) -#endif - -/* - * Select code when configured for LE. - */ -#ifdef CONFIG_CPU_BIG_ENDIAN -#define CPU_LE(code...) -#else -#define CPU_LE(code...) code -#endif - -/* - * Define a macro that constructs a 64-bit value by concatenating two - * 32-bit registers. Note that on big endian systems the order of the - * registers is swapped. - */ -#ifndef CONFIG_CPU_BIG_ENDIAN - .macro regs_to_64, rd, lbits, hbits -#else - .macro regs_to_64, rd, hbits, lbits -#endif - orr \rd, \lbits, \hbits, lsl #32 - .endm - -/* * Pseudo-ops for PC-relative adr/ldr/str <reg>, <symbol> where * <symbol> is within the range +/- 4 GB of the PC. */ @@ -451,6 +420,10 @@ alternative_else_nop_endif * Corrupts: tmp1, tmp2 */ .macro invalidate_icache_by_line start, end, tmp1, tmp2, fixup +alternative_if ARM64_WORKAROUND_NXP_ERR050104 + ic ialluis + b .Licache_done\@ +alternative_else_nop_endif icache_line_size \tmp1, \tmp2 sub \tmp2, \tmp1, #1 bic \tmp2, \start, \tmp2 @@ -459,6 +432,7 @@ alternative_else_nop_endif add \tmp2, \tmp2, \tmp1 cmp \tmp2, \end b.lo .Licache_op\@ +.Licache_done\@: dsb ish isb diff --git a/arch/arm64/include/asm/atomic_ll_sc.h b/arch/arm64/include/asm/atomic_ll_sc.h index 89d2ba272359..160c1251f0ec 100644 --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -11,6 +11,7 @@ #define __ASM_ATOMIC_LL_SC_H #include <linux/stringify.h> +#include <asm/xwreg.h> #ifndef CONFIG_CC_HAS_K_CONSTRAINT #define K @@ -239,20 +240,17 @@ __ll_sc_atomic64_dec_if_positive(atomic64_t *v) #define __CMPXCHG_CASE(w, sfx, name, sz, mb, acq, rel, cl, constraint) \ static __always_inline u##sz \ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ - unsigned long old, \ + u##sz old, \ u##sz new) \ { \ + /* \ + * Sub-word sizes require zero extension so that EOR+CBNZ won't \ + * consume non-zero upper bits of the register containing "old".\ + */ \ + xwreg_t(w) cmpval = xwreg_zero_extend(old, w, sz); \ unsigned long tmp; \ u##sz oldval; \ \ - /* \ - * Sub-word sizes require explicit casting so that the compare \ - * part of the cmpxchg doesn't end up interpreting non-zero \ - * upper bits of the register containing "old". \ - */ \ - if (sz < 32) \ - old = (u##sz)old; \ - \ asm volatile( \ " prfm pstl1strm, %[v]\n" \ "1: ld" #acq "xr" #sfx "\t%" #w "[oldval], %[v]\n" \ @@ -264,7 +262,8 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ "2:" \ : [tmp] "=&r" (tmp), [oldval] "=&r" (oldval), \ [v] "+Q" (*(u##sz *)ptr) \ - : [old] __stringify(constraint) "r" (old), [new] "r" (new) \ + : [old] __stringify(constraint) "r" (cmpval), \ + [new] "r" (new) \ : cl); \ \ return oldval; \ @@ -278,19 +277,19 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ __CMPXCHG_CASE(w, b, , 8, , , , , K) __CMPXCHG_CASE(w, h, , 16, , , , , K) __CMPXCHG_CASE(w, , , 32, , , , , K) -__CMPXCHG_CASE( , , , 64, , , , , L) +__CMPXCHG_CASE(x, , , 64, , , , , L) __CMPXCHG_CASE(w, b, acq_, 8, , a, , "memory", K) __CMPXCHG_CASE(w, h, acq_, 16, , a, , "memory", K) __CMPXCHG_CASE(w, , acq_, 32, , a, , "memory", K) -__CMPXCHG_CASE( , , acq_, 64, , a, , "memory", L) +__CMPXCHG_CASE(x, , acq_, 64, , a, , "memory", L) __CMPXCHG_CASE(w, b, rel_, 8, , , l, "memory", K) __CMPXCHG_CASE(w, h, rel_, 16, , , l, "memory", K) __CMPXCHG_CASE(w, , rel_, 32, , , l, "memory", K) -__CMPXCHG_CASE( , , rel_, 64, , , l, "memory", L) +__CMPXCHG_CASE(x, , rel_, 64, , , l, "memory", L) __CMPXCHG_CASE(w, b, mb_, 8, dmb ish, , l, "memory", K) __CMPXCHG_CASE(w, h, mb_, 16, dmb ish, , l, "memory", K) __CMPXCHG_CASE(w, , mb_, 32, dmb ish, , l, "memory", K) -__CMPXCHG_CASE( , , mb_, 64, dmb ish, , l, "memory", L) +__CMPXCHG_CASE(x, , mb_, 64, dmb ish, , l, "memory", L) #undef __CMPXCHG_CASE diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h index afad1849c4cf..d588af056533 100644 --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -291,15 +291,13 @@ __lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ register unsigned long x1 asm ("x1") = o.high; \ register unsigned long x2 asm ("x2") = n.low; \ register unsigned long x3 asm ("x3") = n.high; \ - register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ \ asm volatile( \ __LSE_PREAMBLE \ " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ : [old1] "+&r" (x0), [old2] "+&r" (x1), \ [v] "+Q" (*(u128 *)ptr) \ - : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ - [oldval1] "r" (o.low), [oldval2] "r" (o.high) \ + : [new1] "r" (x2), [new2] "r" (x3) \ : cl); \ \ r.low = x0; r.high = x1; \ diff --git a/arch/arm64/include/asm/compat.h b/arch/arm64/include/asm/compat.h index ae904a1ad529..5e4cc4d2f872 100644 --- a/arch/arm64/include/asm/compat.h +++ b/arch/arm64/include/asm/compat.h @@ -28,34 +28,20 @@ typedef u16 compat_ipc_pid_t; #include <linux/sched.h> #include <linux/sched/task_stack.h> -#ifdef __AARCH64EB__ -#define COMPAT_UTS_MACHINE "armv8b\0\0" -#else #define COMPAT_UTS_MACHINE "armv8l\0\0" -#endif typedef u16 __compat_uid16_t; typedef u16 __compat_gid16_t; typedef s32 compat_nlink_t; struct compat_stat { -#ifdef __AARCH64EB__ - short st_dev; - short __pad1; -#else compat_dev_t st_dev; -#endif compat_ino_t st_ino; compat_mode_t st_mode; compat_ushort_t st_nlink; __compat_uid16_t st_uid; __compat_gid16_t st_gid; -#ifdef __AARCH64EB__ - short st_rdev; - short __pad2; -#else compat_dev_t st_rdev; -#endif compat_off_t st_size; compat_off_t st_blksize; compat_off_t st_blocks; diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index 3c008821219c..3d1693aaf4e0 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -73,11 +73,8 @@ struct cpuinfo_arm64 { DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data); void cpuinfo_store_cpu(void); -void __init cpuinfo_store_boot_cpu(void); +void update_cpu_features(int cpu); -void __init init_cpu_features(struct cpuinfo_arm64 *info); -void update_cpu_features(int cpu, struct cpuinfo_arm64 *info, - struct cpuinfo_arm64 *boot); -bool gmid_el1_accessible(const struct cpuinfo_arm64 *info); +void __init validate_ftr_regs(void); #endif /* __ASM_CPU_H */ diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 76350b38f0d7..0c3bfbe99aad 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -66,6 +66,8 @@ cpucap_is_possible(const unsigned int cap) return IS_ENABLED(CONFIG_ARM64_ERRATUM_3194386); case ARM64_WORKAROUND_4193714: return IS_ENABLED(CONFIG_ARM64_ERRATUM_4193714); + case ARM64_WORKAROUND_NXP_ERR050104: + return IS_ENABLED(CONFIG_NXP_IMX8QM_ERRATUM_ERR050104); case ARM64_MPAM: /* * KVM MPAM support doesn't rely on the host kernel supporting MPAM. diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4f04ad82ea34..e3c8a1493dc8 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -187,6 +187,10 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; * b) SCOPE_LOCAL_CPU, SCOPE_SYSTEM: All CPUs hotplugged on by the * user after the kernel boot are verified against the capability. * + * SCOPE_BOOT_CPU and SCOPE_SYSTEM capabilities with + * ARM64_CPUCAP_PROBE_ONCE are not matched again on these CPUs. They + * take the state already detected, so there cannot be a conflict. + * * If there is a conflict, the kernel takes an action, based on the * severity (e.g, a CPU could be prevented from booting or cause a * kernel panic). The CPU is allowed to "affect" the state of the @@ -267,6 +271,13 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; #define SCOPE_ALL ARM64_CPUCAP_SCOPE_MASK /* + * matches() is called only once, when the capability is detected, and the + * result applies to all CPUs. Secondary and late CPUs are not checked for + * conflicts but cpu_enable() is still called on each of them. It has no + * effect on SCOPE_LOCAL_CPU capabilities. + */ +#define ARM64_CPUCAP_PROBE_ONCE ((u16)BIT(3)) +/* * Is it permitted for a late CPU to have this capability when system * hasn't already enabled it ? */ @@ -294,6 +305,13 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; #define ARM64_CPUCAP_LOCAL_CPU_ERRATUM \ (ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU) /* + * SoC (not CPU) errata workarounds. The erratum is probed once on the boot + * CPU, before the secondary CPUs are brought up, and no secondary or late CPU + * can conflict with it. + */ +#define ARM64_CPUCAP_SOC_ERRATUM \ + (ARM64_CPUCAP_SCOPE_BOOT_CPU | ARM64_CPUCAP_PROBE_ONCE) +/* * CPU feature detected at boot time based on system-wide value of a * feature. It is safe for a late CPU to have this feature even though * the system hasn't enabled it, although the feature will not be used @@ -414,6 +432,12 @@ static inline bool cpucap_match_all_early_cpus(const struct arm64_cpu_capabiliti return cap->type & ARM64_CPUCAP_MATCH_ALL_EARLY_CPUS; } +static inline bool cpucap_probe_once(const struct arm64_cpu_capabilities *cap) +{ + return (cap->type & ARM64_CPUCAP_PROBE_ONCE) && + !(cap->type & ARM64_CPUCAP_SCOPE_LOCAL_CPU); +} + /* * Generic helper for handling capabilities with multiple (match,enable) pairs * of call backs, sharing the same capability bit. @@ -627,6 +651,13 @@ static inline bool id_aa64pfr1_mpamfrac(u64 pfr1) return val > 0; } +static inline bool id_aa64pfr1_mte(u64 pfr1) +{ + u32 val = cpuid_feature_extract_unsigned_field(pfr1, ID_AA64PFR1_EL1_MTE_SHIFT); + + return val >= ID_AA64PFR1_EL1_MTE_MTE2; +} + void __init setup_boot_cpu_features(void); void __init setup_system_features(void); void __init setup_user_features(void); @@ -957,6 +988,10 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +u64 arm64_ftr_reg_override(u32 sys_id, u64 val); + +/* Read this CPU's ID register with each valid command-line override applied. */ +#define read_cpuid_with_overrides(reg) arm64_ftr_reg_override(SYS_##reg, read_cpuid(reg)) extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 1fa29616e586..6002d1d5f55f 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -136,6 +136,7 @@ #define NVIDIA_CPU_PART_DENVER 0x003 #define NVIDIA_CPU_PART_CARMEL 0x004 #define NVIDIA_CPU_PART_OLYMPUS 0x010 +#define NVIDIA_CPU_PART_RIGEL 0x011 #define FUJITSU_CPU_PART_A64FX 0x001 @@ -234,6 +235,7 @@ #define MIDR_NVIDIA_DENVER MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_DENVER) #define MIDR_NVIDIA_CARMEL MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_CARMEL) #define MIDR_NVIDIA_OLYMPUS MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_OLYMPUS) +#define MIDR_NVIDIA_RIGEL MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_RIGEL) #define MIDR_FUJITSU_A64FX MIDR_CPU_MODEL(ARM_CPU_IMP_FUJITSU, FUJITSU_CPU_PART_A64FX) #define MIDR_HISI_TSV110 MIDR_CPU_MODEL(ARM_CPU_IMP_HISI, HISI_CPU_PART_TSV110) #define MIDR_HISI_HIP09 MIDR_CPU_MODEL(ARM_CPU_IMP_HISI, HISI_CPU_PART_HIP09) diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h index d2779d604c7b..5a0bd3d38b1a 100644 --- a/arch/arm64/include/asm/elf.h +++ b/arch/arm64/include/asm/elf.h @@ -72,11 +72,7 @@ * These are used to set parameters in the core dumps. */ #define ELF_CLASS ELFCLASS64 -#ifdef __AARCH64EB__ -#define ELF_DATA ELFDATA2MSB -#else #define ELF_DATA ELFDATA2LSB -#endif #define ELF_ARCH EM_AARCH64 /* @@ -85,11 +81,7 @@ * intent than poking at uname or /proc/cpuinfo. */ #define ELF_PLATFORM_SIZE 16 -#ifdef __AARCH64EB__ -#define ELF_PLATFORM ("aarch64_be") -#else #define ELF_PLATFORM ("aarch64") -#endif /* * This is used to ensure we don't load something for the wrong architecture. @@ -195,11 +187,7 @@ extern int arch_setup_additional_pages(struct linux_binprm *bprm, #define STACK_RND_MASK (0x3ffff >> (PAGE_SHIFT - 12)) #endif -#ifdef __AARCH64EB__ -#define COMPAT_ELF_PLATFORM ("v8b") -#else #define COMPAT_ELF_PLATFORM ("v8l") -#endif /* AArch32 registers. */ #define COMPAT_ELF_NGREG 18 diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 8fa0707069e8..bbc22e382cfe 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -76,7 +76,7 @@ static inline int gcs_check_locked(struct task_struct *task, new_val &= task->thread.gcs_el0_locked; if (cur_val != new_val) - return -EBUSY; + return -EPERM; return 0; } diff --git a/arch/arm64/include/asm/gpr-num.h b/arch/arm64/include/asm/gpr-num.h index a114e4f8209b..30fb8a67b19f 100644 --- a/arch/arm64/include/asm/gpr-num.h +++ b/arch/arm64/include/asm/gpr-num.h @@ -11,6 +11,8 @@ .equ .L__gpr_num_xzr, 31 .equ .L__gpr_num_wzr, 31 +#define __GPR_NUM(gpr) (.L__gpr_num_##gpr) + #else /* __ASSEMBLER__ */ #define __DEFINE_ASM_GPR_NUMS \ @@ -21,6 +23,13 @@ " .equ .L__gpr_num_xzr, 31\n" \ " .equ .L__gpr_num_wzr, 31\n" +#define __DEFINE_ASM_GPR_ALIASES \ +" .irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n" \ +" wx\\n .req w\\n\n" \ +" .endr\n" + +#define __GPR_NUM(gpr) "(.L__gpr_num_" gpr ")" + #endif /* __ASSEMBLER__ */ #endif /* __ASM_GPR_NUM_H */ diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h index 9aa0bb08ab60..076224734c53 100644 --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h @@ -8,6 +8,7 @@ #ifndef __ASM_IO_H #define __ASM_IO_H +#include <linux/arm-rsi-cmds.h> #include <linux/types.h> #include <linux/pgtable.h> @@ -17,7 +18,6 @@ #include <asm/early_ioremap.h> #include <asm/alternative.h> #include <asm/cpufeature.h> -#include <asm/rsi.h> /* * Generic IO read/write. These perform native-endian accesses. diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index cd9b9d2462f9..406c57c99e84 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -73,8 +73,10 @@ enum kvm_mode { }; #ifdef CONFIG_KVM enum kvm_mode kvm_get_mode(void); +void kvm_force_disabled(void); #else static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; }; +static inline void kvm_force_disabled(void) { }; #endif extern unsigned int __ro_after_init kvm_sve_max_vl; diff --git a/arch/arm64/include/asm/mem_encrypt.h b/arch/arm64/include/asm/mem_encrypt.h index 636f45b4d8af..ef8b8463e52b 100644 --- a/arch/arm64/include/asm/mem_encrypt.h +++ b/arch/arm64/include/asm/mem_encrypt.h @@ -2,8 +2,8 @@ #ifndef __ASM_MEM_ENCRYPT_H #define __ASM_MEM_ENCRYPT_H +#include <linux/arm-rsi-cmds.h> #include <asm/hypervisor.h> -#include <asm/rsi.h> struct device; @@ -16,8 +16,7 @@ int arm64_mem_crypt_ops_register(const struct arm64_mem_crypt_ops *ops); int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); - -int realm_register_memory_enc_ops(void); +int __set_memory_enc_dec(unsigned long addr, int numpages, bool encrypt); static inline bool force_dma_unencrypted(struct device *dev) { diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 352139271918..de910f1c19e1 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -53,14 +53,16 @@ static inline bool system_uses_mte_async_or_asymm_mode(void) */ static inline void mte_disable_tco(void) { - asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(0), - ARM64_MTE, CONFIG_KASAN_HW_TAGS)); + if (!IS_ENABLED(CONFIG_KASAN_HW_TAGS)) + return; + asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(0), ARM64_MTE)); } static inline void mte_enable_tco(void) { - asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(1), - ARM64_MTE, CONFIG_KASAN_HW_TAGS)); + if (!IS_ENABLED(CONFIG_KASAN_HW_TAGS)) + return; + asm volatile(ALTERNATIVE("nop", SET_PSTATE_TCO(1), ARM64_MTE)); } /* diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 58200de8a221..b2eaa63d9056 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -13,6 +13,7 @@ #ifndef __ASSEMBLER__ #include <linux/personality.h> /* for READ_IMPLIES_EXEC */ +#include <linux/string.h> /* for memset() */ #include <linux/types.h> /* for gfp_t */ #include <asm/pgtable-types.h> @@ -20,7 +21,17 @@ struct page; struct vm_area_struct; extern void copy_page(void *to, const void *from); -extern void clear_page(void *to); + +static inline void clear_pages(void *addr, unsigned int npages) +{ + memset(addr, 0, npages * PAGE_SIZE); +} +#define clear_pages clear_pages + +static inline void clear_page(void *addr) +{ + clear_pages(addr, 1); +} void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma); diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 8cf4068ce1b5..3cc316091d66 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -5,10 +5,13 @@ #ifndef __ASM_PERCPU_H #define __ASM_PERCPU_H +#include <linux/bits.h> #include <linux/preempt.h> +#include <linux/stringify.h> #include <asm/alternative.h> #include <asm/cmpxchg.h> +#include <asm/gpr-num.h> #include <asm/stack_pointer.h> #include <asm/sysreg.h> @@ -29,6 +32,11 @@ static inline unsigned long __hyp_my_cpu_offset(void) return read_sysreg(tpidr_el2); } +#define __KERN_ASM_CPU_OFFSET(dst) \ + ALTERNATIVE("mrs " dst ", tpidr_el1", \ + "mrs " dst ", tpidr_el2", \ + ARM64_HAS_VIRT_HOST_EXTN) + static inline unsigned long __kern_my_cpu_offset(void) { unsigned long off; @@ -37,73 +45,192 @@ static inline unsigned long __kern_my_cpu_offset(void) * We want to allow caching the value, so avoid using volatile and * instead use a fake stack read to hazard against barrier(). */ - asm(ALTERNATIVE("mrs %0, tpidr_el1", - "mrs %0, tpidr_el2", - ARM64_HAS_VIRT_HOST_EXTN) - : "=r" (off) : - "Q" (*(const unsigned long *)current_stack_pointer)); + asm( + __KERN_ASM_CPU_OFFSET("%0") + : "=r" (off) + : "Q" (*(const unsigned long *)current_stack_pointer) + ); return off; } +#define PCPU_GPR_PCP_SHIFT 0 +#define PCPU_GPR_PCP GENMASK(4, 0) +#define PCPU_GPR_OFF_SHIFT 5 +#define PCPU_GPR_OFF GENMASK(9, 5) +#define PCPU_GPR_ADDR_SHIFT 10 +#define PCPU_GPR_ADDR GENMASK(14, 10) + +#define __VAL_PCPU_GPRS(pcp, off, addr) \ + "(" \ + "(" __GPR_NUM(pcp) " << " __stringify(PCPU_GPR_PCP_SHIFT) ") | " \ + "(" __GPR_NUM(off) " << " __stringify(PCPU_GPR_OFF_SHIFT) ") | " \ + "(" __GPR_NUM(addr) " << " __stringify(PCPU_GPR_ADDR_SHIFT) ")" \ + ")" + +#define __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \ + ".if (" \ + "(" __GPR_NUM(pcp) " == " __GPR_NUM(off) ") || " \ + "(" __GPR_NUM(pcp) " == " __GPR_NUM(addr) ") || " \ + "(" __GPR_NUM(off) " == " __GPR_NUM(addr) ")" \ + " )\n" \ + ".error \"PCPU GPRS overlap: {" pcp "," off "," addr "}\"\n" \ + ".endif\n" + +#define ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + "// ____PCPU_GPRS_BEGIN(" gprs ", " pcp ", " off ", " addr")\n" \ + __DEFINE_ASM_GPR_NUMS \ + __DEFINE_ASM_GPR_ALIASES \ + __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \ + " mov w" off ", #" __VAL_PCPU_GPRS(pcp, off, addr) "\n" \ + " strh w" off ", " gprs "\n" \ + __KERN_ASM_CPU_OFFSET(off) "\n" + +/* + * Begin a PCPU GPR critical section which requires <addr> (and <off>). + * + * At the start of the critical section, and upon any (preemptible) exception + * until __PCPU_GPRS_END(): + * - <off> will be set to the current CPU's percpu offset. + * - <addr> will be set to <pcp> + <off>. + * + * The <pcp>, <off>, and <addr> registers must be distinct GPRs. + * + * <gprs> must be '¤t_thread_info()->pcpu_gprs', as a memory operand + * which can be written both at the start and end of the critical section + * (e.g. using "=Qo" constraints). + */ +#define __PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + " add " addr ", " pcp ", " off "\n" + +/* + * Begin a PCPU GPR critical section which only requires <off> and does not + * require <addr>. + * + * This is only for operations that can use register-offset addressing, + * e.g. STR <Xt>, [<Xn>, <Xm>]. + * + * All other details are the same as __PCPU_GPRS_BEGIN(). + */ +#define __PCPU_GPRS_BEGIN_OFFSET(gprs, pcp, off) \ + ____PCPU_GPRS_BEGIN(gprs, pcp, off, "xzr") + +/* + * End a PCPU GPR critical section. + */ +#define __PCPU_GPRS_END(gprs) \ + " strh wzr, " gprs "\n" + #ifdef __KVM_NVHE_HYPERVISOR__ #define __my_cpu_offset __hyp_my_cpu_offset() #else #define __my_cpu_offset __kern_my_cpu_offset() #endif -#define PERCPU_RW_OPS(sz) \ -static inline unsigned long __percpu_read_##sz(void *ptr) \ -{ \ - return READ_ONCE(*(u##sz *)ptr); \ -} \ - \ -static inline void __percpu_write_##sz(void *ptr, unsigned long val) \ -{ \ - WRITE_ONCE(*(u##sz *)ptr, (u##sz)val); \ +#define PERCPU_RW_OPS(w, sfx, sz) \ +static inline unsigned long __percpu_read_##sz(void __percpu *pcp) \ +{ \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long off; \ + unsigned long val; \ + \ + asm volatile( \ + __PCPU_GPRS_BEGIN_OFFSET("%[gprs]", "%[pcp]", "%[off]") \ + " ldr" #sfx "\t%" #w "[val], [%[pcp], %[off]]\n" \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [off] "=&r" (off), \ + [val] "=&r" (val) \ + : [pcp] "r" (pcp) \ + : "memory" \ + ); \ + \ + return val; \ +} \ + \ +static inline void __percpu_write_##sz(void __percpu *pcp, unsigned long val) \ +{ \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long off; \ + \ + asm volatile( \ + __PCPU_GPRS_BEGIN_OFFSET("%[gprs]", "%[pcp]", "%[off]") \ + " str" #sfx "\t%" #w "[val], [%[pcp], %[off]]\n" \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [off] "=&r" (off) \ + : [pcp] "r" (pcp), \ + [val] "r" ((u##sz)val) \ + : "memory" \ + ); \ } #define __PERCPU_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \ static inline void \ -__percpu_##name##_case_##sz(void *ptr, unsigned long val) \ +__percpu_##name##_case_##sz(void __percpu *pcp, unsigned long val) \ { \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long addr; \ + unsigned long off; \ unsigned int loop; \ u##sz tmp; \ \ - asm volatile (ARM64_LSE_ATOMIC_INSN( \ + asm volatile ( \ + __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \ + ARM64_LSE_ATOMIC_INSN( \ /* LL/SC */ \ - "1: ldxr" #sfx "\t%" #w "[tmp], %[ptr]\n" \ + "1: ldxr" #sfx "\t%" #w "[tmp], [%[addr]]\n" \ #op_llsc "\t%" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \ - " stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n" \ + " stxr" #sfx "\t%w[loop], %" #w "[tmp], [%[addr]]\n" \ " cbnz %w[loop], 1b", \ /* LSE atomics */ \ - #op_lse #sfx "\t%" #w "[val], %" #w "[tmp], %[ptr]\n" \ + #op_lse #sfx "\t%" #w "[val], %" #w "[tmp], [%[addr]]\n"\ __nops(3)) \ - : [loop] "=&r" (loop), [tmp] "=&r" (tmp), \ - [ptr] "+Q"(*(u##sz *)ptr) \ - : [val] "r" ((u##sz)(val))); \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [addr] "=&r" (addr), \ + [off] "=&r" (off), \ + [loop] "=&r" (loop), \ + [tmp] "=&r" (tmp) \ + : [pcp] "r" (pcp), \ + [val] "r" ((u##sz)(val)) \ + : "memory" \ + ); \ } #define __PERCPU_RET_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \ static inline u##sz \ -__percpu_##name##_return_case_##sz(void *ptr, unsigned long val) \ +__percpu_##name##_return_case_##sz(void __percpu *pcp, unsigned long val) \ { \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long addr; \ + unsigned long off; \ unsigned int loop; \ u##sz ret; \ \ - asm volatile (ARM64_LSE_ATOMIC_INSN( \ + asm volatile ( \ + __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \ + ARM64_LSE_ATOMIC_INSN( \ /* LL/SC */ \ - "1: ldxr" #sfx "\t%" #w "[ret], %[ptr]\n" \ + "1: ldxr" #sfx "\t%" #w "[ret], [%[addr]]\n" \ #op_llsc "\t%" #w "[ret], %" #w "[ret], %" #w "[val]\n" \ - " stxr" #sfx "\t%w[loop], %" #w "[ret], %[ptr]\n" \ + " stxr" #sfx "\t%w[loop], %" #w "[ret], [%[addr]]\n" \ " cbnz %w[loop], 1b", \ /* LSE atomics */ \ - #op_lse #sfx "\t%" #w "[val], %" #w "[ret], %[ptr]\n" \ + #op_lse #sfx "\t%" #w "[val], %" #w "[ret], [%[addr]]\n"\ #op_llsc "\t%" #w "[ret], %" #w "[ret], %" #w "[val]\n" \ __nops(2)) \ - : [loop] "=&r" (loop), [ret] "=&r" (ret), \ - [ptr] "+Q"(*(u##sz *)ptr) \ - : [val] "r" ((u##sz)(val))); \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [addr] "=&r" (addr), \ + [off] "=&r" (off), \ + [loop] "=&r" (loop), \ + [ret] "=&r" (ret) \ + : [pcp] "r" (pcp), \ + [val] "r" ((u##sz)(val)) \ + : "memory" \ + ); \ \ return ret; \ } @@ -120,10 +247,10 @@ __percpu_##name##_return_case_##sz(void *ptr, unsigned long val) \ __PERCPU_RET_OP_CASE(w, , name, 32, op_llsc, op_lse) \ __PERCPU_RET_OP_CASE( , , name, 64, op_llsc, op_lse) -PERCPU_RW_OPS(8) -PERCPU_RW_OPS(16) -PERCPU_RW_OPS(32) -PERCPU_RW_OPS(64) +PERCPU_RW_OPS(w, b, 8) +PERCPU_RW_OPS(w, h, 16) +PERCPU_RW_OPS(w, , 32) +PERCPU_RW_OPS( , , 64) /* * Use value-returning atomics for CPU-local ops as they are more likely @@ -142,120 +269,257 @@ PERCPU_RET_OP(add, add, ldadd) #undef PERCPU_OP #undef PERCPU_RET_OP -/* - * It would be nice to avoid the conditional call into the scheduler when - * re-enabling preemption for preemptible kernels, but doing that in a way - * which builds inside a module would mean messing directly with the preempt - * count. If you do this, peterz and tglx will hunt you down. - * - * Not to mention it'll break the actual preemption model for missing a - * preemption point when TIF_NEED_RESCHED gets set while preemption is - * disabled. - */ +#define PERCPU_XCHG_OP(w, sfx, sz) \ +static inline unsigned long \ +__percpu_xchg_case_##sz(void __percpu *pcp, u##sz val) \ +{ \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long addr; \ + unsigned long off; \ + unsigned int loop; \ + unsigned long ret; \ + \ + asm volatile ( \ + __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \ + ARM64_LSE_ATOMIC_INSN( \ + /* LL/SC */ \ + " prfm pstl1strm, [%[addr]]\n" \ + "1: ldxr" #sfx "\t%" #w "[ret], [%[addr]]\n" \ + " stxr" #sfx "\t%w[loop], %" #w "[val], [%[addr]]\n" \ + " cbnz %w[loop], 1b\n" \ + , \ + /* LSE atomics */ \ + " swp" #sfx "\t%" #w "[val], %" #w "[ret], [%[addr]]\n" \ + __nops(3) \ + ) \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [addr] "=&r" (addr), \ + [off] "=&r" (off), \ + [loop] "=&r" (loop), \ + [ret] "=&r" (ret) \ + : [pcp] "r" (pcp), \ + [val] "r" (val) \ + : "memory" \ + ); \ + \ + return ret; \ +} + +#define PERCPU_CMPXCHG_OP(w, sfx, sz) \ +static inline unsigned long \ +__percpu_cmpxchg_case_##sz(void __percpu *pcp, \ + u##sz old, \ + u##sz new) \ +{ \ + /* \ + * Sub-word sizes require zero extension so that EOR+CBNZ don't \ + * consume non-zero upper bits of the register containing "old".\ + */ \ + xwreg_t(w) cmpval = xwreg_zero_extend(old, w, sz); \ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; \ + unsigned long addr; \ + unsigned long off; \ + unsigned long tmp; \ + unsigned long oldval; \ + \ + asm volatile ( \ + __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \ + ARM64_LSE_ATOMIC_INSN( \ + /* LL/SC */ \ + " prfm pstl1strm, [%[addr]]\n" \ + "1: ldxr" #sfx "\t%" #w "[oldval], [%[addr]]\n" \ + " eor %" #w "[tmp], %" #w "[oldval], %" #w "[old]\n" \ + " cbnz %" #w "[tmp], 2f\n" \ + " stxr" #sfx "\t%w[tmp], %" #w "[new], [%[addr]]\n" \ + " cbnz %w[tmp], 1b\n" \ + "2:\n" \ + , \ + /* LSE atomics */ \ + " mov %" #w "[oldval], %" #w "[old]\n" \ + " cas" #sfx "\t%" #w "[oldval], %" #w "[new], [%[addr]]\n"\ + __nops(4) \ + ) \ + __PCPU_GPRS_END("%[gprs]") \ + : [gprs] "=Qo" (*gprs), \ + [addr] "=&r" (addr), \ + [off] "=&r" (off), \ + [tmp] "=&r" (tmp), \ + [oldval] "=&r" (oldval) \ + : [pcp] "r" (pcp), \ + [old] "r" (cmpval), \ + [new] "r" (new) \ + : "memory" \ + ); \ + \ + return oldval; \ +} + +PERCPU_XCHG_OP(w, b, 8) +PERCPU_XCHG_OP(w, h, 16) +PERCPU_XCHG_OP(w, , 32) +PERCPU_XCHG_OP(x, , 64) + +PERCPU_CMPXCHG_OP(w, b, 8) +PERCPU_CMPXCHG_OP(w, h, 16) +PERCPU_CMPXCHG_OP(w, , 32) +PERCPU_CMPXCHG_OP(x, , 64) -#define _pcp_protect(op, pcp, ...) \ +#undef PERCPU_XCHG_OP +#undef PERCPU_CMPXCHG_OP + +#define _pcp_wrap(op, pcp, ...) \ +({ \ + op(&(pcp), __VA_ARGS__); \ +}) + +#define _pcp_wrap_return(op, pcp, args...) \ +({ \ + (typeof(pcp))op(&(pcp), ##args); \ +}) + +#define _pcp_wrap_xchg(op, pcp, val) \ ({ \ - preempt_disable_notrace(); \ - op(raw_cpu_ptr(&(pcp)), __VA_ARGS__); \ - preempt_enable_notrace(); \ + (typeof(pcp))op(&(pcp), (unsigned long)(val)); \ }) -#define _pcp_protect_return(op, pcp, args...) \ +#define _pcp_wrap_cmpxchg(op, pcp, old, new) \ ({ \ - typeof(pcp) __retval; \ - preempt_disable_notrace(); \ - __retval = (typeof(pcp))op(raw_cpu_ptr(&(pcp)), ##args); \ - preempt_enable_notrace(); \ - __retval; \ + (typeof(pcp))op(&(pcp), (unsigned long)(old), \ + (unsigned long)(new)); \ }) #define this_cpu_read_1(pcp) \ - _pcp_protect_return(__percpu_read_8, pcp) + _pcp_wrap_return(__percpu_read_8, pcp) #define this_cpu_read_2(pcp) \ - _pcp_protect_return(__percpu_read_16, pcp) + _pcp_wrap_return(__percpu_read_16, pcp) #define this_cpu_read_4(pcp) \ - _pcp_protect_return(__percpu_read_32, pcp) + _pcp_wrap_return(__percpu_read_32, pcp) #define this_cpu_read_8(pcp) \ - _pcp_protect_return(__percpu_read_64, pcp) + _pcp_wrap_return(__percpu_read_64, pcp) #define this_cpu_write_1(pcp, val) \ - _pcp_protect(__percpu_write_8, pcp, (unsigned long)(val)) + _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val)) #define this_cpu_write_2(pcp, val) \ - _pcp_protect(__percpu_write_16, pcp, (unsigned long)(val)) + _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val)) #define this_cpu_write_4(pcp, val) \ - _pcp_protect(__percpu_write_32, pcp, (unsigned long)(val)) + _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val)) #define this_cpu_write_8(pcp, val) \ - _pcp_protect(__percpu_write_64, pcp, (unsigned long)(val)) + _pcp_wrap(__percpu_write_64, pcp, (unsigned long)(val)) #define this_cpu_add_1(pcp, val) \ - _pcp_protect(__percpu_add_case_8, pcp, val) + _pcp_wrap(__percpu_add_case_8, pcp, val) #define this_cpu_add_2(pcp, val) \ - _pcp_protect(__percpu_add_case_16, pcp, val) + _pcp_wrap(__percpu_add_case_16, pcp, val) #define this_cpu_add_4(pcp, val) \ - _pcp_protect(__percpu_add_case_32, pcp, val) + _pcp_wrap(__percpu_add_case_32, pcp, val) #define this_cpu_add_8(pcp, val) \ - _pcp_protect(__percpu_add_case_64, pcp, val) + _pcp_wrap(__percpu_add_case_64, pcp, val) #define this_cpu_add_return_1(pcp, val) \ - _pcp_protect_return(__percpu_add_return_case_8, pcp, val) + _pcp_wrap_return(__percpu_add_return_case_8, pcp, val) #define this_cpu_add_return_2(pcp, val) \ - _pcp_protect_return(__percpu_add_return_case_16, pcp, val) + _pcp_wrap_return(__percpu_add_return_case_16, pcp, val) #define this_cpu_add_return_4(pcp, val) \ - _pcp_protect_return(__percpu_add_return_case_32, pcp, val) + _pcp_wrap_return(__percpu_add_return_case_32, pcp, val) #define this_cpu_add_return_8(pcp, val) \ - _pcp_protect_return(__percpu_add_return_case_64, pcp, val) + _pcp_wrap_return(__percpu_add_return_case_64, pcp, val) #define this_cpu_and_1(pcp, val) \ - _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val)) + _pcp_wrap(__percpu_andnot_case_8, pcp, ~(u8)(val)) #define this_cpu_and_2(pcp, val) \ - _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val)) + _pcp_wrap(__percpu_andnot_case_16, pcp, ~(u16)(val)) #define this_cpu_and_4(pcp, val) \ - _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val)) + _pcp_wrap(__percpu_andnot_case_32, pcp, ~(u32)(val)) #define this_cpu_and_8(pcp, val) \ - _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val)) + _pcp_wrap(__percpu_andnot_case_64, pcp, ~(u64)(val)) #define this_cpu_or_1(pcp, val) \ - _pcp_protect(__percpu_or_case_8, pcp, val) + _pcp_wrap(__percpu_or_case_8, pcp, val) #define this_cpu_or_2(pcp, val) \ - _pcp_protect(__percpu_or_case_16, pcp, val) + _pcp_wrap(__percpu_or_case_16, pcp, val) #define this_cpu_or_4(pcp, val) \ - _pcp_protect(__percpu_or_case_32, pcp, val) + _pcp_wrap(__percpu_or_case_32, pcp, val) #define this_cpu_or_8(pcp, val) \ - _pcp_protect(__percpu_or_case_64, pcp, val) + _pcp_wrap(__percpu_or_case_64, pcp, val) #define this_cpu_xchg_1(pcp, val) \ - _pcp_protect_return(xchg_relaxed, pcp, val) + _pcp_wrap_xchg(__percpu_xchg_case_8, pcp, val) #define this_cpu_xchg_2(pcp, val) \ - _pcp_protect_return(xchg_relaxed, pcp, val) + _pcp_wrap_xchg(__percpu_xchg_case_16, pcp, val) #define this_cpu_xchg_4(pcp, val) \ - _pcp_protect_return(xchg_relaxed, pcp, val) + _pcp_wrap_xchg(__percpu_xchg_case_32, pcp, val) #define this_cpu_xchg_8(pcp, val) \ - _pcp_protect_return(xchg_relaxed, pcp, val) + _pcp_wrap_xchg(__percpu_xchg_case_64, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_8, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_16, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_32, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_64, pcp, o, n) #define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) -#define this_cpu_cmpxchg128(pcp, o, n) \ -({ \ - typedef typeof(pcp) pcp_op_T__; \ - u128 old__, new__, ret__; \ - pcp_op_T__ *ptr__; \ - old__ = o; \ - new__ = n; \ - preempt_disable_notrace(); \ - ptr__ = raw_cpu_ptr(&(pcp)); \ - ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \ - preempt_enable_notrace(); \ - ret__; \ -}) +static inline u128 +__percpu_cmpxchg_128(void __percpu *pcp, u128 old, u128 new) +{ + u16 *gprs = ¤t_thread_info()->pcpu_gprs; + unsigned long addr; + unsigned long off; + union __u128_halves r, o = { .full = (old) }, + n = { .full = (new) }; + register unsigned long ol asm ("x0") = o.low; + register unsigned long oh asm ("x1") = o.high; + register unsigned long nl asm ("x2") = n.low; + register unsigned long nh asm ("x3") = n.high; + unsigned long rl, rh; + unsigned long tmp; + + asm volatile ( + __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " prfm pstl1strm, [%[addr]]\n" + "1: ldxp %[rl], %[rh], [%[addr]]\n" + " cmp %[rl], %[ol]\n" + " ccmp %[rh], %[oh], 0, eq\n" + " b.ne 2f\n" + " stxp %w[tmp], %[nl], %[nh], [%[addr]]\n" + " cbnz %w[tmp], 1b\n" + "2:\n" + , + /* LSE atomics */ + " casp %[ol], %[oh], %[nl], %[nh], [%[addr]]\n" + " mov %[rl], %[ol]\n" + " mov %[rh], %[oh]\n" + __nops(4) + ) + __PCPU_GPRS_END("%[gprs]") + : [gprs] "=Qo" (*gprs), + [addr] "=&r" (addr), + [off] "=&r" (off), + [tmp] "=&r" (tmp), + [ol] "+&r" (ol), + [oh] "+&r" (oh), + [rl] "=&r" (rl), + [rh] "=&r" (rh) + : [pcp] "r" (pcp), + [nl] "r" (nl), + [nh] "r" (nh) + : "memory", "cc" + ); + + r.low = rl; + r.high = rh; + + return r.full; +} + +#define this_cpu_cmpxchg128(pcp, o, n) \ + _pcp_wrap_return(__percpu_cmpxchg_128, pcp, o, n) #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 09d7c00cf405..590ef4328f12 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -66,9 +66,9 @@ #ifndef __ASSEMBLER__ +#include <linux/arm-rsi-cmds.h> #include <asm/cpufeature.h> #include <asm/pgtable-types.h> -#include <asm/rsi.h> extern bool arm64_use_ng_mappings; extern unsigned long prot_ns_shared; @@ -139,7 +139,7 @@ static inline bool __pure lpa2_is_enabled(void) /* * Page types used via Permission Indirection Extension (PIE). PIE uses - * the USER, DBM, PXN and UXN bits to to generate an index which is used + * the USER, DBM, PXN and UXN bits to generate an index which is used * to look up the actual permission in PIR_ELx and PIRE0_EL1. We define * combinations we use on non-PIE systems with the same encoding, for * convenience these are listed here as comments as are the unallocated diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h index 9ecc2766a9f2..3c85cacc1d19 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -75,28 +75,21 @@ static inline int __preempt_count_sub_return(int val) return pc; } -static inline bool __preempt_count_dec_and_test(void) +static inline bool should_resched(int preempt_offset) { - struct thread_info *ti = current_thread_info(); - u64 pc = READ_ONCE(ti->preempt_count); - - /* Update only the count field, leaving need_resched unchanged */ - WRITE_ONCE(ti->preempt.count, --pc); - - /* - * If we wrote back all zeroes, then we're preemptible and in - * need of a reschedule. Otherwise, we need to reload the - * preempt_count in case the need_resched flag was cleared by an - * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE - * pair. - */ - return !pc || !READ_ONCE(ti->preempt_count); + u64 pc = READ_ONCE(current_thread_info()->preempt_count); + return unlikely(pc == preempt_offset); } -static inline bool should_resched(int preempt_offset) +static inline bool __preempt_count_dec_and_test(void) { - u64 pc = READ_ONCE(current_thread_info()->preempt_count); - return pc == preempt_offset; + /* + * We must load the combined 'preempt_count' after decrementing + * 'preempt.count' as an interrupt could modify 'need_resched' before + * __preempt_count_sub() writes back to 'preempt.count'. + */ + __preempt_count_sub(1); + return should_resched(0); } #ifdef CONFIG_PREEMPTION diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 6dfbcacd9ba0..481f1999f7c1 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -351,8 +351,6 @@ static inline void compat_start_thread(struct pt_regs *regs, unsigned long pc, unsigned long pstate = PSR_AA32_MODE_USR; if (pc & 1) pstate |= PSR_AA32_T_BIT; - if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN)) - pstate |= PSR_AA32_E_BIT; start_thread_common(regs, pc, pstate); spectre_v4_enable_task_mitigation(current); diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h index f7dc5fb9427d..bd291bd6cfac 100644 --- a/arch/arm64/include/asm/ptrace.h +++ b/arch/arm64/include/asm/ptrace.h @@ -66,12 +66,6 @@ #define PSR_AA32_IT_MASK 0x0600fc00 /* If-Then execution state mask */ #define PSR_AA32_GE_MASK 0x000f0000 -#ifdef CONFIG_CPU_BIG_ENDIAN -#define PSR_AA32_ENDSTATE PSR_AA32_E_BIT -#else -#define PSR_AA32_ENDSTATE 0 -#endif - /* AArch32 CPSR bits, as seen in AArch32 */ #define COMPAT_PSR_DIT_BIT 0x00200000 @@ -167,6 +161,11 @@ struct pt_regs { u64 sdei_ttbr1; struct frame_record_meta stackframe; + + u16 pcpu_gprs; + u16 __unused1; + u32 __unused2; + u64 __unused3; }; /* For correct stack alignment, pt_regs has to be a multiple of 16 bytes. */ @@ -266,7 +265,8 @@ static inline u64 regs_get_register(struct pt_regs *regs, unsigned int offset) * Read a register given an architectural register index r. * This handles the common case where 31 means XZR, not SP. */ -static inline unsigned long pt_regs_read_reg(const struct pt_regs *regs, int r) +static __always_inline unsigned long +pt_regs_read_reg(const struct pt_regs *regs, int r) { return (r == 31) ? 0 : regs->regs[r]; } @@ -275,8 +275,8 @@ static inline unsigned long pt_regs_read_reg(const struct pt_regs *regs, int r) * Write a register given an architectural register index r. * This handles the common case where 31 means XZR, not SP. */ -static inline void pt_regs_write_reg(struct pt_regs *regs, int r, - unsigned long val) +static __always_inline void +pt_regs_write_reg(struct pt_regs *regs, int r, unsigned long val) { if (r != 31) regs->regs[r] = val; diff --git a/arch/arm64/include/asm/rsi.h b/arch/arm64/include/asm/rsi.h deleted file mode 100644 index 88b50d660e85..000000000000 --- a/arch/arm64/include/asm/rsi.h +++ /dev/null @@ -1,70 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright (C) 2024 ARM Ltd. - */ - -#ifndef __ASM_RSI_H_ -#define __ASM_RSI_H_ - -#include <linux/errno.h> -#include <linux/jump_label.h> -#include <asm/rsi_cmds.h> - -#define RSI_PDEV_NAME "arm-cca-dev" - -DECLARE_STATIC_KEY_FALSE(rsi_present); - -void __init arm64_rsi_init(void); - -bool arm64_rsi_is_protected(phys_addr_t base, size_t size); - -static inline bool is_realm_world(void) -{ - return static_branch_unlikely(&rsi_present); -} - -static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end, - enum ripas state, unsigned long flags) -{ - unsigned long ret; - phys_addr_t top; - - while (start != end) { - ret = rsi_set_addr_range_state(start, end, state, flags, &top); - if (ret || top < start || top > end) - return -EINVAL; - start = top; - } - - return 0; -} - -/* - * Convert the specified range to RAM. Do not use this if you rely on the - * contents of a page that may already be in RAM state. - */ -static inline int rsi_set_memory_range_protected(phys_addr_t start, - phys_addr_t end) -{ - return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, - RSI_CHANGE_DESTROYED); -} - -/* - * Convert the specified range to RAM. Do not convert any pages that may have - * been DESTROYED, without our permission. - */ -static inline int rsi_set_memory_range_protected_safe(phys_addr_t start, - phys_addr_t end) -{ - return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, - RSI_NO_CHANGE_DESTROYED); -} - -static inline int rsi_set_memory_range_shared(phys_addr_t start, - phys_addr_t end) -{ - return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY, - RSI_CHANGE_DESTROYED); -} -#endif /* __ASM_RSI_H_ */ diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 23d27623e478..cf57da90b5d1 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -17,7 +17,7 @@ * * This implies, the total number of page table levels required for * IPA_SHIFT at stage2 expected by the hardware can be calculated using - * the same logic used for the (non-collapsable) stage1 page tables but for + * the same logic used for the (non-collapsible) stage1 page tables but for * (IPA_SHIFT - 4). */ #define stage2_pgtable_levels(ipa) ARM64_HW_PGTABLE_LEVELS((ipa) - 4) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 7aa08d59d494..cec36606f6a4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -62,19 +62,10 @@ #else /* CONFIG_BROKEN_GAS_INST */ -#ifndef CONFIG_CPU_BIG_ENDIAN -#define __INSTR_BSWAP(x) (x) -#else /* CONFIG_CPU_BIG_ENDIAN */ -#define __INSTR_BSWAP(x) ((((x) << 24) & 0xff000000) | \ - (((x) << 8) & 0x00ff0000) | \ - (((x) >> 8) & 0x0000ff00) | \ - (((x) >> 24) & 0x000000ff)) -#endif /* CONFIG_CPU_BIG_ENDIAN */ - #ifdef __ASSEMBLER__ -#define __emit_inst(x) .long __INSTR_BSWAP(x) +#define __emit_inst(x) .long (x) #else /* __ASSEMBLER__ */ -#define __emit_inst(x) ".long " __stringify(__INSTR_BSWAP(x)) "\n\t" +#define __emit_inst(x) ".long " __stringify(x) "\n\t" #endif /* __ASSEMBLER__ */ #endif /* CONFIG_BROKEN_GAS_INST */ @@ -173,7 +164,7 @@ * come from here. The header relies on the definition of sys_reg() * earlier in this file. */ -#include "asm/sysreg-defs.h" +#include <asm/sysreg-defs.h> /* * System registers, organised loosely by encoding but grouped together @@ -835,39 +826,26 @@ #define SCTLR_ELx_A (BIT(1)) #define SCTLR_ELx_M (BIT(0)) -#ifdef CONFIG_CPU_BIG_ENDIAN -#define ENDIAN_SET_EL2 SCTLR_ELx_EE -#else -#define ENDIAN_SET_EL2 0 -#endif - #define INIT_SCTLR_EL2_MMU_ON \ (SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA | SCTLR_ELx_I | \ - SCTLR_ELx_IESB | SCTLR_ELx_WXN | ENDIAN_SET_EL2 | \ - SCTLR_ELx_ITFSB | SCTLR_ELx_EIS | SCTLR_ELx_EOS | SCTLR_EL2_RES1) + SCTLR_ELx_IESB | SCTLR_ELx_WXN | SCTLR_ELx_ITFSB | \ + SCTLR_ELx_EIS | SCTLR_ELx_EOS | SCTLR_EL2_RES1) -#define INIT_SCTLR_EL2_MMU_OFF \ - (SCTLR_EL2_RES1 | ENDIAN_SET_EL2) +#define INIT_SCTLR_EL2_MMU_OFF (SCTLR_EL2_RES1) /* SCTLR_EL1 specific flags. */ -#ifdef CONFIG_CPU_BIG_ENDIAN -#define ENDIAN_SET_EL1 (SCTLR_EL1_E0E | SCTLR_ELx_EE) -#else -#define ENDIAN_SET_EL1 0 -#endif - #define INIT_SCTLR_EL1_MMU_OFF \ - (ENDIAN_SET_EL1 | SCTLR_EL1_LSMAOE | SCTLR_EL1_nTLSMD | \ - SCTLR_EL1_EIS | SCTLR_EL1_TSCXT | SCTLR_EL1_EOS) + (SCTLR_EL1_LSMAOE | SCTLR_EL1_nTLSMD | SCTLR_EL1_EIS | \ + SCTLR_EL1_TSCXT | SCTLR_EL1_EOS) #define INIT_SCTLR_EL1_MMU_ON \ - (SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA | \ - SCTLR_EL1_SA0 | SCTLR_EL1_SED | SCTLR_ELx_I | \ - SCTLR_EL1_DZE | SCTLR_EL1_UCT | SCTLR_EL1_nTWE | \ - SCTLR_ELx_IESB | SCTLR_EL1_SPAN | SCTLR_ELx_ITFSB | \ - ENDIAN_SET_EL1 | SCTLR_EL1_UCI | SCTLR_EL1_EPAN | \ - SCTLR_EL1_LSMAOE | SCTLR_EL1_nTLSMD | SCTLR_EL1_EIS | \ - SCTLR_EL1_TSCXT | SCTLR_EL1_EOS) + (SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA | \ + SCTLR_EL1_SA0 | SCTLR_EL1_SED | SCTLR_ELx_I | \ + SCTLR_EL1_DZE | SCTLR_EL1_UCT | SCTLR_EL1_nTWE | \ + SCTLR_ELx_IESB | SCTLR_EL1_SPAN | SCTLR_ELx_ITFSB | \ + SCTLR_EL1_UCI | SCTLR_EL1_EPAN | SCTLR_EL1_LSMAOE | \ + SCTLR_EL1_nTLSMD | SCTLR_EL1_EIS | SCTLR_EL1_TSCXT | \ + SCTLR_EL1_EOS) /* MAIR_ELx memory attributes (used by Linux) */ #define MAIR_ATTR_DEVICE_nGnRnE UL(0x00) diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 5d7fe3e153c8..6cb976c1c693 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -29,13 +29,8 @@ struct thread_info { union { u64 preempt_count; /* 0 => preemptible, <0 => bug */ struct { -#ifdef CONFIG_CPU_BIG_ENDIAN - u32 need_resched; - u32 count; -#else u32 count; u32 need_resched; -#endif } preempt; }; #ifdef CONFIG_SHADOW_CALL_STACK @@ -46,6 +41,7 @@ struct thread_info { u64 mpam_partid_pmg; #endif u32 cpu; + u16 pcpu_gprs; }; #define thread_saved_pc(tsk) \ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 14a78ac0f800..a05520d3a013 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -378,9 +378,13 @@ static inline void flush_tlb_mm(struct mm_struct *mm) unsigned long asid; dsb(ishst); - asid = __TLBI_VADDR(0, ASID(mm)); - __tlbi(aside1is, asid); - __tlbi_user(aside1is, asid); + if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104)) { + __tlbi(vmalle1is); + } else { + asid = __TLBI_VADDR(0, ASID(mm)); + __tlbi(aside1is, asid); + __tlbi_user(aside1is, asid); + } __tlbi_sync_s1ish(mm); mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } @@ -580,23 +584,28 @@ static __always_inline void __do_flush_tlb_range(struct vm_area_struct *vma, asid = ASID(mm); - switch (flags & (TLBF_NOWALKCACHE | TLBF_NOBROADCAST)) { - case TLBF_NONE: - __flush_s1_tlb_range_op(vae1is, start, pages, stride, - asid, tlb_level); - break; - case TLBF_NOWALKCACHE: - __flush_s1_tlb_range_op(vale1is, start, pages, stride, - asid, tlb_level); - break; - case TLBF_NOBROADCAST: - /* Combination unused */ - BUG(); - break; - case TLBF_NOWALKCACHE | TLBF_NOBROADCAST: - __flush_s1_tlb_range_op(vale1, start, pages, stride, - asid, tlb_level); - break; + if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104) && + !(flags & TLBF_NOBROADCAST)) { + __tlbi(vmalle1is); + } else { + switch (flags & (TLBF_NOWALKCACHE | TLBF_NOBROADCAST)) { + case TLBF_NONE: + __flush_s1_tlb_range_op(vae1is, start, pages, stride, + asid, tlb_level); + break; + case TLBF_NOWALKCACHE: + __flush_s1_tlb_range_op(vale1is, start, pages, stride, + asid, tlb_level); + break; + case TLBF_NOBROADCAST: + /* Combination unused */ + BUG(); + break; + case TLBF_NOWALKCACHE | TLBF_NOBROADCAST: + __flush_s1_tlb_range_op(vale1, start, pages, stride, + asid, tlb_level); + break; + } } if (!(flags & TLBF_NONOTIFY)) @@ -657,7 +666,8 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end end = round_up(end, stride); pages = (end - start) >> PAGE_SHIFT; - if (__flush_tlb_range_limit_excess(pages, stride)) { + if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104) || + __flush_tlb_range_limit_excess(pages, stride)) { flush_tlb_all(); return; } @@ -675,10 +685,14 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end */ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) { - unsigned long addr = __TLBI_VADDR(kaddr, 0); - dsb(ishst); - __tlbi(vaae1is, addr); + if (alternative_has_cap_unlikely(ARM64_WORKAROUND_NXP_ERR050104)) { + __tlbi(vmalle1is); + } else { + unsigned long addr = __TLBI_VADDR(kaddr, 0); + + __tlbi(vaae1is, addr); + } __tlbi_sync_s1ish_kernel(); isb(); } diff --git a/arch/arm64/include/asm/word-at-a-time.h b/arch/arm64/include/asm/word-at-a-time.h index 824ca6987a51..d7a3b69dc8c6 100644 --- a/arch/arm64/include/asm/word-at-a-time.h +++ b/arch/arm64/include/asm/word-at-a-time.h @@ -7,8 +7,6 @@ #include <linux/uaccess.h> -#ifndef __AARCH64EB__ - #include <linux/bitops.h> #include <linux/wordpart.h> @@ -36,10 +34,6 @@ static inline unsigned long zero_bytemask(unsigned long bits) return bits >> 7; } -#else /* __AARCH64EB__ */ -#include <asm-generic/word-at-a-time.h> -#endif - /* * Load an unaligned word from kernel space. * diff --git a/arch/arm64/include/asm/xwreg.h b/arch/arm64/include/asm/xwreg.h new file mode 100644 index 000000000000..d55e3ac68f7e --- /dev/null +++ b/arch/arm64/include/asm/xwreg.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_XWREG_H +#define __ASM_XWREG_H + +#include <asm/types.h> + +#define __xwreg_t_x u64 +#define __xwreg_t_w u32 +#define xwreg_t(xw) __xwreg_t_##xw + +/* + * Zero extend 'v' from 'sz' bits (8/16/32/64) to fill an X or W register. + */ +#define xwreg_zero_extend(v, xw, sz) ((xwreg_t(xw))(u##sz)(v)) + +#endif /* __ASM_XWREG_H */ diff --git a/arch/arm64/include/uapi/asm/byteorder.h b/arch/arm64/include/uapi/asm/byteorder.h index ca9cfdab33fc..b078e60f0506 100644 --- a/arch/arm64/include/uapi/asm/byteorder.h +++ b/arch/arm64/include/uapi/asm/byteorder.h @@ -17,10 +17,6 @@ #ifndef __ASM_BYTEORDER_H #define __ASM_BYTEORDER_H -#ifdef __AARCH64EB__ -#include <linux/byteorder/big_endian.h> -#else #include <linux/byteorder/little_endian.h> -#endif #endif /* __ASM_BYTEORDER_H */ diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h index 15649a253a57..0c8c6a126c3b 100644 --- a/arch/arm64/include/uapi/asm/ptrace.h +++ b/arch/arm64/include/uapi/asm/ptrace.h @@ -113,7 +113,7 @@ struct user_hwdebug_state { struct user_sve_header { __u32 size; /* total meaningful regset content in bytes */ - __u32 max_size; /* maxmium possible size for this thread */ + __u32 max_size; /* maximum possible size for this thread */ __u16 vl; /* current vector length */ __u16 max_vl; /* maximum possible vector length */ __u16 flags; @@ -272,7 +272,7 @@ struct user_pac_generic_keys { struct user_za_header { __u32 size; /* total meaningful regset content in bytes */ - __u32 max_size; /* maxmium possible size for this thread */ + __u32 max_size; /* maximum possible size for this thread */ __u16 vl; /* current vector length */ __u16 max_vl; /* maximum possible vector length */ __u16 flags; diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index ea66339435bf..fc5d1a0c38b3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -37,7 +37,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ syscall.o proton-pack.o idle.o patching.o pi/ \ - rsi.o jump_label.o + jump_label.o obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 9c853ed3ceab..9d603317eeb9 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -36,6 +36,7 @@ int main(void) DEFINE(TSK_TI_SCS_BASE, offsetof(struct task_struct, thread_info.scs_base)); DEFINE(TSK_TI_SCS_SP, offsetof(struct task_struct, thread_info.scs_sp)); #endif + DEFINE(TSK_TI_PCPU_GPRS, offsetof(struct task_struct, thread_info.pcpu_gprs)); DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); @@ -78,6 +79,7 @@ int main(void) DEFINE(S_PMR, offsetof(struct pt_regs, pmr)); DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); DEFINE(S_STACKFRAME_TYPE, offsetof(struct pt_regs, stackframe.type)); + DEFINE(S_PCPU_GPRS, offsetof(struct pt_regs, pcpu_gprs)); DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs)); BLANK(); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index e0c09402540c..38abd123c740 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -6,6 +6,7 @@ */ #include <linux/arm-smccc.h> +#include <linux/of.h> #include <linux/types.h> #include <linux/cpu.h> #include <asm/cpu.h> @@ -13,6 +14,7 @@ #include <asm/cpufeature.h> #include <asm/fpsimd.h> #include <asm/kvm_asm.h> +#include <asm/kvm_host.h> #include <asm/smp_plat.h> static u64 target_impl_cpu_num; @@ -203,6 +205,33 @@ cpu_enable_cache_maint_trap(const struct arm64_cpu_capabilities *__unused) sysreg_clear_set(sctlr_el1, SCTLR_EL1_UCI, 0); } +#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104 +static bool +is_imx8qm_soc(const struct arm64_cpu_capabilities *entry, int scope) +{ + static const char * const compats[] = { + "fsl,imx8qm", + "fsl,imx8qp", + NULL + }; + + return of_machine_compatible_match(compats); +} + +static void +cpu_enable_imx8qm_err050104(const struct arm64_cpu_capabilities *__unused) +{ + /* Trap the EL0 IC IVAU so that it can be upgraded to IC IALLUIS */ + sysreg_clear_set(sctlr_el1, SCTLR_EL1_UCI, 0); + + /* + * TLB maintenance cannot be guaranteed correct for guests, so + * disable KVM as if kvm-arm.mode=none was passed on the command line. + */ + kvm_force_disabled(); +} +#endif + #define CAP_MIDR_RANGE(model, v_min, r_min, v_max, r_max) \ .matches = is_affected_midr_range, \ .midr_range = MIDR_RANGE(model, v_min, r_min, v_max, r_max) @@ -986,7 +1015,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = { #ifdef CONFIG_ARM64_ERRATUM_4311569 { .capability = ARM64_WORKAROUND_4311569, - .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .type = ARM64_CPUCAP_SOC_ERRATUM, .matches = need_arm_si_l1_workaround_4311569, }, #endif @@ -1045,6 +1074,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_broken_gic_v3_seis, }, +#ifdef CONFIG_NXP_IMX8QM_ERRATUM_ERR050104 + { + .desc = "NXP erratum ERR050104", + .capability = ARM64_WORKAROUND_NXP_ERR050104, + .type = ARM64_CPUCAP_SOC_ERRATUM, + .matches = is_imx8qm_soc, + .cpu_enable = cpu_enable_imx8qm_err050104, + }, +#endif { } }; diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 3ae3ad1a1cdf..34c5c03c0c9b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -117,6 +117,7 @@ EXPORT_SYMBOL(system_cpucaps); static struct arm64_cpu_capabilities const __ro_after_init *cpucap_ptrs[ARM64_NCAPS]; DECLARE_BITMAP(boot_cpucaps, ARM64_NCAPS); +static struct cpuinfo_arm64 boot_cpu_data; /* * arm64_use_ng_mappings must be placed in the .data section, otherwise it @@ -293,6 +294,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar3[] = { static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_CSV3_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_CSV2_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_RME_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_DIT_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_AMU_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_MPAM_SHIFT, 4, 0), @@ -888,7 +890,7 @@ static int search_cmp_ftr_reg(const void *id, const void *regp) * entry. * * returns - Upon success, matching ftr_reg entry for id. - * - NULL on failure. It is upto the caller to decide + * - NULL on failure. It is up to the caller to decide * the impact of a failure. */ static struct arm64_ftr_reg *get_arm64_ftr_reg_nowarn(u32 sys_id) @@ -962,7 +964,33 @@ s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, return ret; } -static void __init sort_ftr_regs(void) +u64 arm64_ftr_reg_override(u32 sys_id, u64 val) +{ + struct arm64_ftr_reg *reg = get_arm64_ftr_reg(sys_id); + const struct arm64_ftr_bits *ftrp; + + if (!reg || !reg->override->mask) + return val; + + for (ftrp = reg->ftr_bits; ftrp->width; ftrp++) { + u64 ftr_mask = arm64_ftr_mask(ftrp); + s64 ftr_new, ftr_ovr; + + if ((ftr_mask & reg->override->mask) != ftr_mask) + continue; + + ftr_new = arm64_ftr_value(ftrp, val); + ftr_ovr = arm64_ftr_value(ftrp, reg->override->val); + + /* An unsafe override is left for init_cpu_ftr_reg() to remove. */ + if (arm64_ftr_safe_value(ftrp, ftr_ovr, ftr_new) == ftr_ovr) + val = arm64_ftr_set_value(ftrp, val, ftr_ovr); + } + + return val; +} + +void __init validate_ftr_regs(void) { unsigned int i; @@ -1017,6 +1045,8 @@ static void __init sort_ftr_regs(void) * Also initialises the strict_mask for the register. * Any bits that are not covered by an arm64_ftr_bits entry are considered * RES0 for the system-wide value, and must strictly match. + * @new has been through arm64_ftr_reg_override(), so a valid override is + * already applied to it. Only an unsafe one is left to remove here. */ static void init_cpu_ftr_reg(u32 sys_reg, u64 new) { @@ -1037,29 +1067,20 @@ static void init_cpu_ftr_reg(u32 sys_reg, u64 new) s64 ftr_ovr = arm64_ftr_value(ftrp, reg->override->val); if ((ftr_mask & reg->override->mask) == ftr_mask) { - s64 tmp = arm64_ftr_safe_value(ftrp, ftr_ovr, ftr_new); - char *str = NULL; + char *str = "forced"; - if (ftr_ovr != tmp) { + if (arm64_ftr_safe_value(ftrp, ftr_ovr, ftr_new) != ftr_ovr) { /* Unsafe, remove the override */ reg->override->mask &= ~ftr_mask; reg->override->val &= ~ftr_mask; - tmp = ftr_ovr; str = "ignoring override"; - } else if (ftr_new != tmp) { - /* Override was valid */ - ftr_new = tmp; - str = "forced"; - } else { - /* Override was the safe value */ - str = "already set"; } pr_warn("%s[%d:%d]: %s to %llx\n", reg->name, ftrp->shift + ftrp->width - 1, ftrp->shift, str, - tmp & (BIT(ftrp->width) - 1)); + ftr_ovr & (BIT(ftrp->width) - 1)); } else if ((ftr_mask & reg->override->val) == ftr_mask) { reg->override->val &= ~ftr_mask; pr_warn("%s[%d:%d]: impossible override, ignored\n", @@ -1178,37 +1199,15 @@ static bool detect_ftr_has_mpam(void) return id_aa64pfr0_mpam(pfr0) || id_aa64pfr1_mpamfrac(pfr1); } -bool gmid_el1_accessible(const struct cpuinfo_arm64 *info) +static void __init init_cpu_features(void) { - const struct arm64_ftr_bits *ftrp; - s64 mte, ovr; - u64 ftr_mask; - - /* No ID register reflects CONFIG_ARM64_MTE. */ - if (!IS_ENABLED(CONFIG_ARM64_MTE)) - return false; - - for (ftrp = ftr_id_aa64pfr1; ftrp->width; ftrp++) { - if (ftrp->shift == ID_AA64PFR1_EL1_MTE_SHIFT) - break; - } + struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0); - ftr_mask = arm64_ftr_mask(ftrp); - mte = arm64_ftr_value(ftrp, info->reg_id_aa64pfr1); - - /* The boot CPU runs before init_cpu_ftr_reg() strips unsafe overrides. */ - if ((id_aa64pfr1_override.mask & ftr_mask) == ftr_mask) { - ovr = arm64_ftr_value(ftrp, id_aa64pfr1_override.val); - mte = arm64_ftr_safe_value(ftrp, ovr, mte); - } - - return mte >= ID_AA64PFR1_EL1_MTE_MTE2; -} - -void __init init_cpu_features(struct cpuinfo_arm64 *info) -{ - /* Before we start using the tables, make sure it is sorted */ - sort_ftr_regs(); + /* + * We keep a copy of the boot CPU registers so that physical hotplug + * of CPU 0 can still be properly checked. + */ + boot_cpu_data = *info; init_cpu_ftr_reg(SYS_CTR_EL0, info->reg_ctr); init_cpu_ftr_reg(SYS_DCZID_EL0, info->reg_dczid); @@ -1252,12 +1251,10 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) cpacr_restore(cpacr); } - if (detect_ftr_has_mpam()) { - info->reg_mpamidr = read_cpuid(MPAMIDR_EL1); + if (id_aa64pfr0_mpam(info->reg_id_aa64pfr0) || id_aa64pfr1_mpamfrac(info->reg_id_aa64pfr1)) init_cpu_ftr_reg(SYS_MPAMIDR_EL1, info->reg_mpamidr); - } - if (gmid_el1_accessible(info)) + if (IS_ENABLED(CONFIG_ARM64_MTE) && id_aa64pfr1_mte(info->reg_id_aa64pfr1)) init_cpu_ftr_reg(SYS_GMID_EL1, info->reg_gmid); } @@ -1399,154 +1396,6 @@ static int update_32bit_cpu_features(int cpu, struct cpuinfo_32bit *info, return taint; } -/* - * Update system wide CPU feature registers with the values from a - * non-boot CPU. Also performs SANITY checks to make sure that there - * aren't any insane variations from that of the boot CPU. - */ -void update_cpu_features(int cpu, - struct cpuinfo_arm64 *info, - struct cpuinfo_arm64 *boot) -{ - int taint = 0; - - /* - * The kernel can handle differing I-cache policies, but otherwise - * caches should look identical. Userspace JITs will make use of - * *minLine. - */ - taint |= check_update_ftr_reg(SYS_CTR_EL0, cpu, - info->reg_ctr, boot->reg_ctr); - - /* - * Userspace may perform DC ZVA instructions. Mismatched block sizes - * could result in too much or too little memory being zeroed if a - * process is preempted and migrated between CPUs. - */ - taint |= check_update_ftr_reg(SYS_DCZID_EL0, cpu, - info->reg_dczid, boot->reg_dczid); - - /* If different, timekeeping will be broken (especially with KVM) */ - taint |= check_update_ftr_reg(SYS_CNTFRQ_EL0, cpu, - info->reg_cntfrq, boot->reg_cntfrq); - - /* - * The kernel uses self-hosted debug features and expects CPUs to - * support identical debug features. We presently need CTX_CMPs, WRPs, - * and BRPs to be identical. - * ID_AA64DFR1 is currently RES0. - */ - taint |= check_update_ftr_reg(SYS_ID_AA64DFR0_EL1, cpu, - info->reg_id_aa64dfr0, boot->reg_id_aa64dfr0); - taint |= check_update_ftr_reg(SYS_ID_AA64DFR1_EL1, cpu, - info->reg_id_aa64dfr1, boot->reg_id_aa64dfr1); - /* - * Even in big.LITTLE, processors should be identical instruction-set - * wise. - */ - taint |= check_update_ftr_reg(SYS_ID_AA64ISAR0_EL1, cpu, - info->reg_id_aa64isar0, boot->reg_id_aa64isar0); - taint |= check_update_ftr_reg(SYS_ID_AA64ISAR1_EL1, cpu, - info->reg_id_aa64isar1, boot->reg_id_aa64isar1); - taint |= check_update_ftr_reg(SYS_ID_AA64ISAR2_EL1, cpu, - info->reg_id_aa64isar2, boot->reg_id_aa64isar2); - taint |= check_update_ftr_reg(SYS_ID_AA64ISAR3_EL1, cpu, - info->reg_id_aa64isar3, boot->reg_id_aa64isar3); - - /* - * Differing PARange support is fine as long as all peripherals and - * memory are mapped within the minimum PARange of all CPUs. - * Linux should not care about secure memory. - */ - taint |= check_update_ftr_reg(SYS_ID_AA64MMFR0_EL1, cpu, - info->reg_id_aa64mmfr0, boot->reg_id_aa64mmfr0); - taint |= check_update_ftr_reg(SYS_ID_AA64MMFR1_EL1, cpu, - info->reg_id_aa64mmfr1, boot->reg_id_aa64mmfr1); - taint |= check_update_ftr_reg(SYS_ID_AA64MMFR2_EL1, cpu, - info->reg_id_aa64mmfr2, boot->reg_id_aa64mmfr2); - taint |= check_update_ftr_reg(SYS_ID_AA64MMFR3_EL1, cpu, - info->reg_id_aa64mmfr3, boot->reg_id_aa64mmfr3); - taint |= check_update_ftr_reg(SYS_ID_AA64MMFR4_EL1, cpu, - info->reg_id_aa64mmfr4, boot->reg_id_aa64mmfr4); - - taint |= check_update_ftr_reg(SYS_ID_AA64PFR0_EL1, cpu, - info->reg_id_aa64pfr0, boot->reg_id_aa64pfr0); - taint |= check_update_ftr_reg(SYS_ID_AA64PFR1_EL1, cpu, - info->reg_id_aa64pfr1, boot->reg_id_aa64pfr1); - taint |= check_update_ftr_reg(SYS_ID_AA64PFR2_EL1, cpu, - info->reg_id_aa64pfr2, boot->reg_id_aa64pfr2); - - taint |= check_update_ftr_reg(SYS_ID_AA64ZFR0_EL1, cpu, - info->reg_id_aa64zfr0, boot->reg_id_aa64zfr0); - - taint |= check_update_ftr_reg(SYS_ID_AA64SMFR0_EL1, cpu, - info->reg_id_aa64smfr0, boot->reg_id_aa64smfr0); - - taint |= check_update_ftr_reg(SYS_ID_AA64FPFR0_EL1, cpu, - info->reg_id_aa64fpfr0, boot->reg_id_aa64fpfr0); - - /* Probe vector lengths */ - if (IS_ENABLED(CONFIG_ARM64_SVE) && - id_aa64pfr0_sve(read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1))) { - if (!system_capabilities_finalized()) { - unsigned long cpacr = cpacr_save_enable_kernel_sve(); - - vec_update_vq_map(ARM64_VEC_SVE); - - cpacr_restore(cpacr); - } - } - - if (IS_ENABLED(CONFIG_ARM64_SME) && - id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1))) { - unsigned long cpacr = cpacr_save_enable_kernel_sme(); - - /* Probe vector lengths */ - if (!system_capabilities_finalized()) - vec_update_vq_map(ARM64_VEC_SME); - - cpacr_restore(cpacr); - } - - if (detect_ftr_has_mpam()) { - info->reg_mpamidr = read_cpuid(MPAMIDR_EL1); - taint |= check_update_ftr_reg(SYS_MPAMIDR_EL1, cpu, - info->reg_mpamidr, boot->reg_mpamidr); - } - - /* - * The kernel uses the LDGM/STGM instructions and the number of tags - * they read/write depends on the GMID_EL1.BS field. Check that the - * value is the same on all CPUs. - */ - if (gmid_el1_accessible(info)) - taint |= check_update_ftr_reg(SYS_GMID_EL1, cpu, - info->reg_gmid, boot->reg_gmid); - - /* - * If we don't have AArch32 at all then skip the checks entirely - * as the register values may be UNKNOWN and we're not going to be - * using them for anything. - * - * This relies on a sanitised view of the AArch64 ID registers - * (e.g. SYS_ID_AA64PFR0_EL1), so we call it last. - */ - if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) { - lazy_init_32bit_cpu_features(info, boot); - taint |= update_32bit_cpu_features(cpu, &info->aarch32, - &boot->aarch32); - } - - /* - * Mismatched CPU features are a recipe for disaster. Don't even - * pretend to support them. - */ - if (taint) { - pr_warn_once("Unsupported CPU feature variation detected.\n"); - add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK); - } -} - u64 read_sanitised_ftr_reg(u32 id) { struct arm64_ftr_reg *regp = get_arm64_ftr_reg(id); @@ -1931,7 +1780,7 @@ static bool has_nv1(const struct arm64_cpu_capabilities *entry, int scope) is_midr_in_range_list(nv1_ni_list))); } -#if defined(ID_AA64MMFR0_EL1_TGRAN_LPA2) && defined(ID_AA64MMFR0_EL1_TGRAN_2_SUPPORTED_LPA2) +#if defined(ID_AA64MMFR0_EL1_TGRAN_LPA2) static bool has_lpa2_at_stage1(u64 mmfr0) { unsigned int tgran; @@ -2166,6 +2015,7 @@ bool cpu_supports_bbml3(void) MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), MIDR_REV_RANGE(MIDR_NEOVERSE_V3AE, 0, 2, 0xf), MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS), + MIDR_ALL_VERSIONS(MIDR_NVIDIA_RIGEL), MIDR_ALL_VERSIONS(MIDR_AMPERE1), MIDR_ALL_VERSIONS(MIDR_AMPERE1A), MIDR_ALL_VERSIONS(MIDR_CORTEX_A520AE), @@ -3103,7 +2953,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_bbml3, }, { - .desc = "52-bit Virtual Addressing for KVM (LPA2)", + .desc = "52-bit Virtual-Physical Addressing for S1 and S2 (LPA2)", .capability = ARM64_HAS_LPA2, .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_lpa2, @@ -3694,8 +3544,11 @@ static void verify_local_cpu_caps(u16 scope_mask) if (!caps || !(caps->type & scope_mask)) continue; - cpu_has_cap = caps->matches(caps, SCOPE_LOCAL_CPU); system_has_cap = cpus_have_cap(caps->capability); + if (cpucap_probe_once(caps)) + cpu_has_cap = system_has_cap; + else + cpu_has_cap = caps->matches(caps, SCOPE_LOCAL_CPU); if (system_has_cap) { /* @@ -3707,7 +3560,7 @@ static void verify_local_cpu_caps(u16 scope_mask) /* * We have to issue cpu_enable() irrespective of * whether the CPU has it or not, as it is enabeld - * system wide. It is upto the call back to take + * system wide. It is up to the call back to take * appropriate action on this CPU. */ if (caps->cpu_enable) @@ -3892,15 +3745,166 @@ void check_local_cpu_capabilities(void) check_early_cpu_features(); /* + * Verify that this CPU has all the system advertised + * capabilities. + */ + if (system_capabilities_finalized()) + verify_local_cpu_capabilities(); +} + +/* + * Update system wide CPU feature registers with the values from a + * non-boot CPU. Also performs SANITY checks to make sure that there + * aren't any insane variations from that of the boot CPU. + */ +void update_cpu_features(int cpu) +{ + struct cpuinfo_arm64 *boot, *info; + int taint = 0; + + /* * If we haven't finalised the system capabilities, this CPU gets * a chance to update the errata work arounds and local features. - * Otherwise, this CPU should verify that it has all the system - * advertised capabilities. */ if (!system_capabilities_finalized()) update_cpu_capabilities(SCOPE_LOCAL_CPU); - else - verify_local_cpu_capabilities(); + + boot = &boot_cpu_data; + info = per_cpu_ptr(&cpu_data, cpu); + + /* + * The kernel can handle differing I-cache policies, but otherwise + * caches should look identical. Userspace JITs will make use of + * *minLine. + */ + taint |= check_update_ftr_reg(SYS_CTR_EL0, cpu, + info->reg_ctr, boot->reg_ctr); + + /* + * Userspace may perform DC ZVA instructions. Mismatched block sizes + * could result in too much or too little memory being zeroed if a + * process is preempted and migrated between CPUs. + */ + taint |= check_update_ftr_reg(SYS_DCZID_EL0, cpu, + info->reg_dczid, boot->reg_dczid); + + /* If different, timekeeping will be broken (especially with KVM) */ + taint |= check_update_ftr_reg(SYS_CNTFRQ_EL0, cpu, + info->reg_cntfrq, boot->reg_cntfrq); + + /* + * The kernel uses self-hosted debug features and expects CPUs to + * support identical debug features. We presently need CTX_CMPs, WRPs, + * and BRPs to be identical. + * ID_AA64DFR1 is currently RES0. + */ + taint |= check_update_ftr_reg(SYS_ID_AA64DFR0_EL1, cpu, + info->reg_id_aa64dfr0, boot->reg_id_aa64dfr0); + taint |= check_update_ftr_reg(SYS_ID_AA64DFR1_EL1, cpu, + info->reg_id_aa64dfr1, boot->reg_id_aa64dfr1); + /* + * Even in big.LITTLE, processors should be identical instruction-set + * wise. + */ + taint |= check_update_ftr_reg(SYS_ID_AA64ISAR0_EL1, cpu, + info->reg_id_aa64isar0, boot->reg_id_aa64isar0); + taint |= check_update_ftr_reg(SYS_ID_AA64ISAR1_EL1, cpu, + info->reg_id_aa64isar1, boot->reg_id_aa64isar1); + taint |= check_update_ftr_reg(SYS_ID_AA64ISAR2_EL1, cpu, + info->reg_id_aa64isar2, boot->reg_id_aa64isar2); + taint |= check_update_ftr_reg(SYS_ID_AA64ISAR3_EL1, cpu, + info->reg_id_aa64isar3, boot->reg_id_aa64isar3); + + /* + * Differing PARange support is fine as long as all peripherals and + * memory are mapped within the minimum PARange of all CPUs. + * Linux should not care about secure memory. + */ + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR0_EL1, cpu, + info->reg_id_aa64mmfr0, boot->reg_id_aa64mmfr0); + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR1_EL1, cpu, + info->reg_id_aa64mmfr1, boot->reg_id_aa64mmfr1); + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR2_EL1, cpu, + info->reg_id_aa64mmfr2, boot->reg_id_aa64mmfr2); + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR3_EL1, cpu, + info->reg_id_aa64mmfr3, boot->reg_id_aa64mmfr3); + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR4_EL1, cpu, + info->reg_id_aa64mmfr4, boot->reg_id_aa64mmfr4); + + taint |= check_update_ftr_reg(SYS_ID_AA64PFR0_EL1, cpu, + info->reg_id_aa64pfr0, boot->reg_id_aa64pfr0); + taint |= check_update_ftr_reg(SYS_ID_AA64PFR1_EL1, cpu, + info->reg_id_aa64pfr1, boot->reg_id_aa64pfr1); + taint |= check_update_ftr_reg(SYS_ID_AA64PFR2_EL1, cpu, + info->reg_id_aa64pfr2, boot->reg_id_aa64pfr2); + + taint |= check_update_ftr_reg(SYS_ID_AA64ZFR0_EL1, cpu, + info->reg_id_aa64zfr0, boot->reg_id_aa64zfr0); + + taint |= check_update_ftr_reg(SYS_ID_AA64SMFR0_EL1, cpu, + info->reg_id_aa64smfr0, boot->reg_id_aa64smfr0); + + taint |= check_update_ftr_reg(SYS_ID_AA64FPFR0_EL1, cpu, + info->reg_id_aa64fpfr0, boot->reg_id_aa64fpfr0); + + /* Probe vector lengths */ + if (IS_ENABLED(CONFIG_ARM64_SVE) && + id_aa64pfr0_sve(read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1))) { + if (!system_capabilities_finalized()) { + unsigned long cpacr = cpacr_save_enable_kernel_sve(); + + vec_update_vq_map(ARM64_VEC_SVE); + + cpacr_restore(cpacr); + } + } + + if (IS_ENABLED(CONFIG_ARM64_SME) && + id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1))) { + unsigned long cpacr = cpacr_save_enable_kernel_sme(); + + /* Probe vector lengths */ + if (!system_capabilities_finalized()) + vec_update_vq_map(ARM64_VEC_SME); + + cpacr_restore(cpacr); + } + + if (id_aa64pfr0_mpam(info->reg_id_aa64pfr0) || id_aa64pfr1_mpamfrac(info->reg_id_aa64pfr1)) + taint |= check_update_ftr_reg(SYS_MPAMIDR_EL1, cpu, + info->reg_mpamidr, boot->reg_mpamidr); + + /* + * The kernel uses the LDGM/STGM instructions and the number of tags + * they read/write depends on the GMID_EL1.BS field. Check that the + * value is the same on all CPUs. + */ + if (IS_ENABLED(CONFIG_ARM64_MTE) && id_aa64pfr1_mte(info->reg_id_aa64pfr1)) + taint |= check_update_ftr_reg(SYS_GMID_EL1, cpu, + info->reg_gmid, boot->reg_gmid); + + /* + * If we don't have AArch32 at all then skip the checks entirely + * as the register values may be UNKNOWN and we're not going to be + * using them for anything. + * + * This relies on a sanitised view of the AArch64 ID registers + * (e.g. SYS_ID_AA64PFR0_EL1), so we call it last. + */ + if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) { + lazy_init_32bit_cpu_features(info, boot); + taint |= update_32bit_cpu_features(cpu, &info->aarch32, + &boot->aarch32); + } + + /* + * Mismatched CPU features are a recipe for disaster. Don't even + * pretend to support them. + */ + if (taint) { + pr_warn_once("Unsupported CPU feature variation detected.\n"); + add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK); + } } bool this_cpu_has_cap(unsigned int n) @@ -3978,6 +3982,8 @@ static void __init setup_boot_cpu_capabilities(void) void __init setup_boot_cpu_features(void) { + init_cpu_features(); + /* * Initialize the indirect array of CPU capabilities pointers before we * handle the boot CPU. diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index 45c63f3d75c5..94e6cd211d70 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -31,20 +31,6 @@ * values depending on configuration at or after reset. */ DEFINE_PER_CPU(struct cpuinfo_arm64, cpu_data); -static struct cpuinfo_arm64 boot_cpu_data; - -static inline const char *icache_policy_str(int l1ip) -{ - switch (l1ip) { - case CTR_EL0_L1Ip_VIPT: - return "VIPT"; - case CTR_EL0_L1Ip_PIPT: - return "PIPT"; - default: - return "RESERVED/UNKNOWN"; - } -} - unsigned long __icache_flags; static const char *const hwcap_str[] = { @@ -425,7 +411,6 @@ device_initcall(cpuinfo_regs_init); static void cpuinfo_detect_icache_policy(struct cpuinfo_arm64 *info) { - unsigned int cpu = smp_processor_id(); u32 l1ip = CTR_L1IP(info->reg_ctr); switch (l1ip) { @@ -437,39 +422,37 @@ static void cpuinfo_detect_icache_policy(struct cpuinfo_arm64 *info) set_bit(ICACHEF_ALIASING, &__icache_flags); break; } - - pr_info("Detected %s I-cache on CPU%d\n", icache_policy_str(l1ip), cpu); } static void __cpuinfo_store_cpu_32bit(struct cpuinfo_32bit *info) { - info->reg_id_dfr0 = read_cpuid(ID_DFR0_EL1); - info->reg_id_dfr1 = read_cpuid(ID_DFR1_EL1); - info->reg_id_isar0 = read_cpuid(ID_ISAR0_EL1); - info->reg_id_isar1 = read_cpuid(ID_ISAR1_EL1); - info->reg_id_isar2 = read_cpuid(ID_ISAR2_EL1); - info->reg_id_isar3 = read_cpuid(ID_ISAR3_EL1); - info->reg_id_isar4 = read_cpuid(ID_ISAR4_EL1); - info->reg_id_isar5 = read_cpuid(ID_ISAR5_EL1); - info->reg_id_isar6 = read_cpuid(ID_ISAR6_EL1); - info->reg_id_mmfr0 = read_cpuid(ID_MMFR0_EL1); - info->reg_id_mmfr1 = read_cpuid(ID_MMFR1_EL1); - info->reg_id_mmfr2 = read_cpuid(ID_MMFR2_EL1); - info->reg_id_mmfr3 = read_cpuid(ID_MMFR3_EL1); - info->reg_id_mmfr4 = read_cpuid(ID_MMFR4_EL1); - info->reg_id_mmfr5 = read_cpuid(ID_MMFR5_EL1); - info->reg_id_pfr0 = read_cpuid(ID_PFR0_EL1); - info->reg_id_pfr1 = read_cpuid(ID_PFR1_EL1); - info->reg_id_pfr2 = read_cpuid(ID_PFR2_EL1); - - info->reg_mvfr0 = read_cpuid(MVFR0_EL1); - info->reg_mvfr1 = read_cpuid(MVFR1_EL1); - info->reg_mvfr2 = read_cpuid(MVFR2_EL1); + info->reg_id_dfr0 = read_cpuid_with_overrides(ID_DFR0_EL1); + info->reg_id_dfr1 = read_cpuid_with_overrides(ID_DFR1_EL1); + info->reg_id_isar0 = read_cpuid_with_overrides(ID_ISAR0_EL1); + info->reg_id_isar1 = read_cpuid_with_overrides(ID_ISAR1_EL1); + info->reg_id_isar2 = read_cpuid_with_overrides(ID_ISAR2_EL1); + info->reg_id_isar3 = read_cpuid_with_overrides(ID_ISAR3_EL1); + info->reg_id_isar4 = read_cpuid_with_overrides(ID_ISAR4_EL1); + info->reg_id_isar5 = read_cpuid_with_overrides(ID_ISAR5_EL1); + info->reg_id_isar6 = read_cpuid_with_overrides(ID_ISAR6_EL1); + info->reg_id_mmfr0 = read_cpuid_with_overrides(ID_MMFR0_EL1); + info->reg_id_mmfr1 = read_cpuid_with_overrides(ID_MMFR1_EL1); + info->reg_id_mmfr2 = read_cpuid_with_overrides(ID_MMFR2_EL1); + info->reg_id_mmfr3 = read_cpuid_with_overrides(ID_MMFR3_EL1); + info->reg_id_mmfr4 = read_cpuid_with_overrides(ID_MMFR4_EL1); + info->reg_id_mmfr5 = read_cpuid_with_overrides(ID_MMFR5_EL1); + info->reg_id_pfr0 = read_cpuid_with_overrides(ID_PFR0_EL1); + info->reg_id_pfr1 = read_cpuid_with_overrides(ID_PFR1_EL1); + info->reg_id_pfr2 = read_cpuid_with_overrides(ID_PFR2_EL1); + + info->reg_mvfr0 = read_cpuid_with_overrides(MVFR0_EL1); + info->reg_mvfr1 = read_cpuid_with_overrides(MVFR1_EL1); + info->reg_mvfr2 = read_cpuid_with_overrides(MVFR2_EL1); } static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) { - info->reg_cntfrq = arch_timer_get_cntfrq(); + info->reg_cntfrq = arm64_ftr_reg_override(SYS_CNTFRQ_EL0, arch_timer_get_cntfrq()); /* * Use the effective value of the CTR_EL0 than the raw value * exposed by the CPU. CTR_EL0.IDC field value must be interpreted @@ -478,42 +461,39 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) * effective value of the CTR_EL0 in our internal records for * accurate sanity check and feature enablement. */ - info->reg_ctr = read_cpuid_effective_cachetype(); - info->reg_dczid = read_cpuid(DCZID_EL0); + info->reg_ctr = arm64_ftr_reg_override(SYS_CTR_EL0, read_cpuid_effective_cachetype()); + info->reg_dczid = read_cpuid_with_overrides(DCZID_EL0); info->reg_midr = read_cpuid_id(); info->reg_revidr = read_cpuid(REVIDR_EL1); info->reg_aidr = read_cpuid(AIDR_EL1); - info->reg_id_aa64dfr0 = read_cpuid(ID_AA64DFR0_EL1); - info->reg_id_aa64dfr1 = read_cpuid(ID_AA64DFR1_EL1); - info->reg_id_aa64isar0 = read_cpuid(ID_AA64ISAR0_EL1); - info->reg_id_aa64isar1 = read_cpuid(ID_AA64ISAR1_EL1); - info->reg_id_aa64isar2 = read_cpuid(ID_AA64ISAR2_EL1); - info->reg_id_aa64isar3 = read_cpuid(ID_AA64ISAR3_EL1); - info->reg_id_aa64mmfr0 = read_cpuid(ID_AA64MMFR0_EL1); - info->reg_id_aa64mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); - info->reg_id_aa64mmfr2 = read_cpuid(ID_AA64MMFR2_EL1); - info->reg_id_aa64mmfr3 = read_cpuid(ID_AA64MMFR3_EL1); - info->reg_id_aa64mmfr4 = read_cpuid(ID_AA64MMFR4_EL1); - info->reg_id_aa64pfr0 = read_cpuid(ID_AA64PFR0_EL1); - info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1); - info->reg_id_aa64pfr2 = read_cpuid(ID_AA64PFR2_EL1); - info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1); - info->reg_id_aa64smfr0 = read_cpuid(ID_AA64SMFR0_EL1); - info->reg_id_aa64fpfr0 = read_cpuid(ID_AA64FPFR0_EL1); - - if (gmid_el1_accessible(info)) - info->reg_gmid = read_cpuid(GMID_EL1); + info->reg_id_aa64dfr0 = read_cpuid_with_overrides(ID_AA64DFR0_EL1); + info->reg_id_aa64dfr1 = read_cpuid_with_overrides(ID_AA64DFR1_EL1); + info->reg_id_aa64isar0 = read_cpuid_with_overrides(ID_AA64ISAR0_EL1); + info->reg_id_aa64isar1 = read_cpuid_with_overrides(ID_AA64ISAR1_EL1); + info->reg_id_aa64isar2 = read_cpuid_with_overrides(ID_AA64ISAR2_EL1); + info->reg_id_aa64isar3 = read_cpuid_with_overrides(ID_AA64ISAR3_EL1); + info->reg_id_aa64mmfr0 = read_cpuid_with_overrides(ID_AA64MMFR0_EL1); + info->reg_id_aa64mmfr1 = read_cpuid_with_overrides(ID_AA64MMFR1_EL1); + info->reg_id_aa64mmfr2 = read_cpuid_with_overrides(ID_AA64MMFR2_EL1); + info->reg_id_aa64mmfr3 = read_cpuid_with_overrides(ID_AA64MMFR3_EL1); + info->reg_id_aa64mmfr4 = read_cpuid_with_overrides(ID_AA64MMFR4_EL1); + info->reg_id_aa64pfr0 = read_cpuid_with_overrides(ID_AA64PFR0_EL1); + info->reg_id_aa64pfr1 = read_cpuid_with_overrides(ID_AA64PFR1_EL1); + info->reg_id_aa64pfr2 = read_cpuid_with_overrides(ID_AA64PFR2_EL1); + info->reg_id_aa64zfr0 = read_cpuid_with_overrides(ID_AA64ZFR0_EL1); + info->reg_id_aa64smfr0 = read_cpuid_with_overrides(ID_AA64SMFR0_EL1); + info->reg_id_aa64fpfr0 = read_cpuid_with_overrides(ID_AA64FPFR0_EL1); + + if (IS_ENABLED(CONFIG_ARM64_MTE) && id_aa64pfr1_mte(info->reg_id_aa64pfr1)) + info->reg_gmid = read_cpuid_with_overrides(GMID_EL1); + + if (id_aa64pfr0_mpam(info->reg_id_aa64pfr0) || id_aa64pfr1_mpamfrac(info->reg_id_aa64pfr1)) + info->reg_mpamidr = read_cpuid_with_overrides(MPAMIDR_EL1); if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) __cpuinfo_store_cpu_32bit(&info->aarch32); - /* - * info->reg_mpamidr deferred to {init,update}_cpu_features because we - * don't want to read it (and trigger a trap on buggy firmware) if - * using an aa64pfr0_el1 override to unconditionally disable MPAM. - */ - if (IS_ENABLED(CONFIG_ARM64_SME) && id_aa64pfr1_sme(info->reg_id_aa64pfr1)) { /* @@ -531,14 +511,4 @@ void cpuinfo_store_cpu(void) { struct cpuinfo_arm64 *info = this_cpu_ptr(&cpu_data); __cpuinfo_store_cpu(info); - update_cpu_features(smp_processor_id(), info, &boot_cpu_data); -} - -void __init cpuinfo_store_boot_cpu(void) -{ - struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0); - __cpuinfo_store_cpu(info); - - boot_cpu_data = *info; - init_cpu_features(&boot_cpu_data); } diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 0ec90fd1754e..6f34aef3d04e 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -117,8 +117,14 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) const efi_memory_desc_t *md = spd->md; pte_t pte = __ptep_get(ptep); + /* + * Clear PTE_WRITE as well: it doubles as the DBM bit, and leaving it + * set would permit the hardware to make the mapping writable again + * when TCR_EL1.HD is set. + */ if (md->attribute & EFI_MEMORY_RO) - pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); + pte = set_pte_bit(clear_pte_bit(pte, __pgprot(PTE_WRITE)), + __pgprot(PTE_RDONLY)); if (md->attribute & EFI_MEMORY_XP) pte = set_pte_bit(pte, __pgprot(PTE_PXN)); else if (system_supports_bti_kernel() && spd->has_bti) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 72c03ccea59f..11d3f888fcd8 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -25,6 +25,7 @@ #include <asm/irq_regs.h> #include <asm/kprobes.h> #include <asm/mmu.h> +#include <asm/percpu.h> #include <asm/processor.h> #include <asm/sdei.h> #include <asm/stacktrace.h> @@ -32,6 +33,42 @@ #include <asm/system_misc.h> /* + * Where the context being returned to had an active percpu GPR critical + * section, ensure that the offset and address GPRs are updated to match the + * current CPU. + * + * For simplicity we always update the GPRs when a critical section is active + * and preemption was *possible*, regardless of whether preemption actually + * occurred. Where preemption did not occur, the updates are redundant but not + * harmful. + */ +static __always_inline void irqentry_exit_pcpu_adjust(struct pt_regs *regs) +{ + int reg_pcp, reg_off, reg_addr; + unsigned long pcp, off, addr; + u16 gprs = regs->pcpu_gprs; + + /* + * Zero means no active PCPU GPRs. As the PCPU GPRs must be distinct, + * a PCPU critical section cannot possibly use {x0,x0,x0}. + */ + if (likely(!gprs)) + return; + + reg_pcp = FIELD_GET(PCPU_GPR_PCP, gprs); + reg_off = FIELD_GET(PCPU_GPR_OFF, gprs); + reg_addr = FIELD_GET(PCPU_GPR_ADDR, gprs); + + pcp = pt_regs_read_reg(regs, reg_pcp); + + off = __kern_my_cpu_offset(); + pt_regs_write_reg(regs, reg_off, off); + + addr = pcp + off; + pt_regs_write_reg(regs, reg_addr, addr); +} + +/* * Handle IRQ/context state management when entering from kernel mode. * Before this function is called it is not safe to call regular kernel code, * instrumentable code, or any code which may trigger an exception. @@ -56,6 +93,7 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs, irqentry_state_t state) { local_daif_mask(); + irqentry_exit_pcpu_adjust(regs); mte_check_tfsr_exit(); irqentry_exit_to_kernel_mode_after_preempt(regs, state); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index f63049ac32dc..8d17f61d1b5b 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -194,6 +194,17 @@ alternative_cb_end #endif .endm + .macro pcpu_gprs_entry, tsk:req, regs:req, tmp:req + ldrh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS] + strh w\tmp, [\regs, #S_PCPU_GPRS] + strh wzr, [\tsk, #TSK_TI_PCPU_GPRS] + .endm + + .macro pcpu_gprs_exit, tsk:req, regs:req, tmp:req + ldrh w\tmp, [\regs, #S_PCPU_GPRS] + strh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS] + .endm + .macro kernel_entry, el, regsize = 64 .if \el == 0 alternative_insn nop, SET_PSTATE_DIT(1), ARM64_HAS_DIT @@ -277,6 +288,7 @@ alternative_else_nop_endif .else add x21, sp, #PT_REGS_SIZE get_current_task tsk + pcpu_gprs_entry tsk, sp, x0 .endif /* \el == 0 */ mrs x22, elr_el1 mrs x23, spsr_el1 @@ -333,6 +345,10 @@ alternative_else_nop_endif .endm .macro kernel_exit, el + .if \el != 0 + pcpu_gprs_exit tsk, sp, x0 + .endif + #ifdef CONFIG_ARM64_PSEUDO_NMI alternative_if_not ARM64_HAS_GIC_PRIO_MASKING b .Lskip_pmr_restore\@ @@ -1023,11 +1039,12 @@ SYM_CODE_START(__sdei_asm_handler) /* * We may have interrupted userspace, or a guest, or exit-from or - * return-to either of these. We can't trust sp_el0, restore it. + * return-to either of these. Preserve the interrupted sp_el0, and + * initialize sp_el0 to the current task. */ - mrs x28, sp_el0 - ldr_this_cpu dst=x0, sym=__entry_task, tmp=x1 - msr sp_el0, x0 + mrs x20, sp_el0 + ldr_this_cpu dst=tsk, sym=__entry_task, tmp=x1 + msr sp_el0, tsk /* If we interrupted the kernel point to the previous stack/frame. */ and x0, x3, #0xc @@ -1039,15 +1056,25 @@ SYM_CODE_START(__sdei_asm_handler) stp x29, x4, [sp, #-16]! mov x29, sp + add x16, x19, #SDEI_EVENT_INTREGS + pcpu_gprs_entry tsk, x16, x17 + add x0, x19, #SDEI_EVENT_INTREGS mov x1, x19 bl __sdei_handler - msr sp_el0, x28 - /* restore regs >x17 that we clobbered */ + add x16, x19, #SDEI_EVENT_INTREGS + pcpu_gprs_exit tsk, x16, x17 + + msr sp_el0, x20 + /* restore regs >x17 that firmware won't restore */ mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline - ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14] ldp x18, x19, [x4, #SDEI_EVENT_INTREGS + 16 * 9] + ldp x20, x21, [x4, #SDEI_EVENT_INTREGS + 16 * 10] + ldp x22, x23, [x4, #SDEI_EVENT_INTREGS + 16 * 11] + ldp x24, x25, [x4, #SDEI_EVENT_INTREGS + 16 * 12] + ldp x26, x27, [x4, #SDEI_EVENT_INTREGS + 16 * 13] + ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14] ldp lr, x1, [x4, #SDEI_EVENT_INTREGS + S_LR] mov sp, x1 diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e7f1682a3059..ecfa21615f16 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -621,23 +621,6 @@ static int __init sme_sysctl_init(void) { return 0; } #define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) -#ifdef CONFIG_CPU_BIG_ENDIAN -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - u64 a = swab64(x); - u64 b = swab64(x >> 64); - - return ((__uint128_t)a << 64) | b; -} -#else -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - return x; -} -#endif - -#define arm64_le128_to_cpu(x) arm64_cpu_to_le128(x) - static void __fpsimd_to_sve(struct arm64_sve_state *sst, struct user_fpsimd_state const *fst, unsigned int vq) @@ -647,7 +630,7 @@ static void __fpsimd_to_sve(struct arm64_sve_state *sst, for (i = 0; i < SVE_NUM_ZREGS; ++i) { p = (__uint128_t *)ZREG(sst, vq, i); - *p = arm64_cpu_to_le128(fst->vregs[i]); + *p = fst->vregs[i]; } } @@ -702,7 +685,7 @@ static inline void sve_to_fpsimd(struct task_struct *task) vq = sve_vq_from_vl(vl); for (i = 0; i < SVE_NUM_ZREGS; ++i) { p = (__uint128_t const *)ZREG(sst, vq, i); - fst->vregs[i] = arm64_le128_to_cpu(*p); + fst->vregs[i] = *p; } } @@ -1316,7 +1299,7 @@ void do_sve_acc(unsigned long esr, struct pt_regs *regs) return; } - sve_alloc(current, true); + sve_alloc(current, false); if (!current->thread.sve_state) { force_sig(SIGKILL); return; @@ -1341,6 +1324,7 @@ void do_sve_acc(unsigned long esr, struct pt_regs *regs) sve_flush_live(); fpsimd_bind_task_to_cpu(); } else { + memset(current->thread.sve_state, 0, sve_state_size(current)); fpsimd_to_sve(current); current->thread.fp_type = FP_STATE_SVE; fpsimd_flush_task_state(current); diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 87a822e5c4ca..8951ce693552 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -138,8 +138,7 @@ SYM_CODE_START_LOCAL(record_mmu_state) b.ne 0f mrs x19, sctlr_el2 0: -CPU_LE( tbnz x19, #SCTLR_ELx_EE_SHIFT, 1f ) -CPU_BE( tbz x19, #SCTLR_ELx_EE_SHIFT, 1f ) + tbnz x19, #SCTLR_ELx_EE_SHIFT, 1f tst x19, #SCTLR_ELx_C // Z := (C == 0) and x19, x19, #SCTLR_ELx_M // isolate M bit csel x19, xzr, x19, eq // clear x19 if Z diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 424291c547f0..8c39863d2d63 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -10,6 +10,8 @@ * Copyright (C) 2006 Rafael J. Wysocki <rjw@sisk.pl> */ #define pr_fmt(x) "hibernate: " x + +#include <linux/arm-rmi-cmds.h> #include <linux/cpu.h> #include <linux/kvm_host.h> #include <linux/pm.h> @@ -105,6 +107,18 @@ void notrace restore_processor_state(void) { } +bool arch_hibernation_available(void) +{ + /* + * If we have activated the RMM, there could be pages that are + * delegated to the RMM. Trying to save them to the image will be fatal. + * Also, we donate pages to the RMM at activation and restoring data + * to those pages are going to be fatal. + * Hence, disable the hibernation when the RMM is active + */ + return !is_rmm_active(); +} + int arch_hibernation_header_save(void *addr, unsigned int max_size) { struct arch_hibernate_hdr *hdr = addr; diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c index 73cce8ac8368..32b38967433e 100644 --- a/arch/arm64/kernel/hw_breakpoint.c +++ b/arch/arm64/kernel/hw_breakpoint.c @@ -20,6 +20,7 @@ #include <linux/uaccess.h> #include <asm/current.h> +#include <asm/daifflags.h> #include <asm/debug-monitors.h> #include <asm/esr.h> #include <asm/exception.h> @@ -221,6 +222,12 @@ static int hw_breakpoint_slot_setup(struct perf_event **slots, int max_slots, return -ENOSPC; } +/* + * Configure hardware breakpoint registers from a perf event, + * according to operation. + * + * Must be called with debug exceptions masked. + */ static int hw_breakpoint_control(struct perf_event *bp, enum hw_breakpoint_ops ops) { @@ -286,15 +293,24 @@ static int hw_breakpoint_control(struct perf_event *bp, /* * Install a perf counter breakpoint. + * + * Perf can and will call this with debug exceptions unmasked. + * Mask them to avoid triggering one while manipulating their registers. */ int arch_install_hw_breakpoint(struct perf_event *bp) { - return hw_breakpoint_control(bp, HW_BREAKPOINT_INSTALL); + unsigned long daif = local_daif_save(); + int ret = hw_breakpoint_control(bp, HW_BREAKPOINT_INSTALL); + + local_daif_restore(daif); + return ret; } void arch_uninstall_hw_breakpoint(struct perf_event *bp) { + unsigned long daif = local_daif_save(); hw_breakpoint_control(bp, HW_BREAKPOINT_UNINSTALL); + local_daif_restore(daif); } static int get_hbp_len(u8 hbp_len) @@ -586,7 +602,10 @@ int hw_breakpoint_arch_parse(struct perf_event *bp, /* * Enable/disable all of the breakpoints active at the specified * exception level at the register level. - * This is used when single-stepping after a breakpoint exception. + * This is used when single-stepping after a breakpoint exception or switching + * threads. + * + * Must be called with debug exceptions masked. */ static void toggle_bp_registers(int reg, enum dbg_active_el el, int enable) { @@ -908,7 +927,11 @@ bool try_step_suspended_breakpoints(struct pt_regs *regs) NOKPROBE_SYMBOL(try_step_suspended_breakpoints); /* - * Context-switcher for restoring suspended breakpoints. + * Context-switcher for restoring suspended CPU-bound breakpoints. + * + * Breakpoints of the previous task are uninstalled before + * this function is called, in perf_event_task_sched_out(), and those + * of the next task are installed after, in perf_event_task_sched_in(). */ void hw_breakpoint_thread_switch(struct task_struct *next) { @@ -922,6 +945,7 @@ void hw_breakpoint_thread_switch(struct task_struct *next) */ struct debug_info *current_debug_info, *next_debug_info; + unsigned long daif = local_daif_save(); current_debug_info = ¤t->thread.debug; next_debug_info = &next->thread.debug; @@ -937,6 +961,8 @@ void hw_breakpoint_thread_switch(struct task_struct *next) toggle_bp_registers(AARCH64_DBG_REG_WCR, DBG_ACTIVE_EL0, !next_debug_info->wps_disabled); + + local_daif_restore(daif); } /* diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 14beb7b9d304..facda8137d9b 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -119,7 +119,6 @@ KVM_NVHE_ALIAS(__start___kvm_ex_table); KVM_NVHE_ALIAS(__stop___kvm_ex_table); /* Position-independent library routines */ -KVM_NVHE_ALIAS_HYP(clear_page, __pi_clear_page); KVM_NVHE_ALIAS_HYP(copy_page, __pi_copy_page); KVM_NVHE_ALIAS_HYP(memcpy, __pi_memcpy); KVM_NVHE_ALIAS_HYP(memset, __pi_memset); diff --git a/arch/arm64/kernel/image.h b/arch/arm64/kernel/image.h index 7bc3ba897901..5c10ca773363 100644 --- a/arch/arm64/kernel/image.h +++ b/arch/arm64/kernel/image.h @@ -24,28 +24,14 @@ * when PIE is in effect. So we need to split them up in 32-bit high and low * words. */ -#ifdef CONFIG_CPU_BIG_ENDIAN -#define DATA_LE32(data) \ - ((((data) & 0x000000ff) << 24) | \ - (((data) & 0x0000ff00) << 8) | \ - (((data) & 0x00ff0000) >> 8) | \ - (((data) & 0xff000000) >> 24)) -#else -#define DATA_LE32(data) ((data) & 0xffffffff) -#endif - #define DEFINE_IMAGE_LE64(sym, data) \ - sym##_lo32 = DATA_LE32((data) & 0xffffffff); \ - sym##_hi32 = DATA_LE32((data) >> 32) + sym##_lo32 = (data) & 0xffffffff; \ + sym##_hi32 = (data) >> 32 #define __HEAD_FLAG(field) (__HEAD_FLAG_##field << \ ARM64_IMAGE_FLAG_##field##_SHIFT) -#ifdef CONFIG_CPU_BIG_ENDIAN -#define __HEAD_FLAG_BE ARM64_IMAGE_FLAG_BE -#else #define __HEAD_FLAG_BE ARM64_IMAGE_FLAG_LE -#endif #define __HEAD_FLAG_PAGE_SIZE ((PAGE_SHIFT - 10) / 2) diff --git a/arch/arm64/kernel/kexec_image.c b/arch/arm64/kernel/kexec_image.c index b70f4df15a1a..b8d0fb0a9daf 100644 --- a/arch/arm64/kernel/kexec_image.c +++ b/arch/arm64/kernel/kexec_image.c @@ -40,7 +40,7 @@ static void *image_load(struct kimage *image, { struct arm64_image_header *h; u64 flags, value; - bool be_image, be_kernel; + bool be_image; struct kexec_buf kbuf = {}; unsigned long text_offset, kernel_segment_number; struct kexec_segment *kernel_segment; @@ -58,8 +58,7 @@ static void *image_load(struct kimage *image, /* Check cpu features */ flags = le64_to_cpu(h->flags); be_image = arm64_image_flag_field(flags, ARM64_IMAGE_FLAG_BE); - be_kernel = IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - if ((be_image != be_kernel) && !system_supports_mixed_endian()) + if (be_image && !system_supports_mixed_endian()) return ERR_PTR(-EINVAL); value = arm64_image_flag_field(flags, ARM64_IMAGE_FLAG_PAGE_SIZE); diff --git a/arch/arm64/kernel/kgdb.c b/arch/arm64/kernel/kgdb.c index 968324a79a89..3426d0908973 100644 --- a/arch/arm64/kernel/kgdb.c +++ b/arch/arm64/kernel/kgdb.c @@ -60,11 +60,7 @@ struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = { * 32-bits. Look for the big comment in asm/kgdb.h for more * detail. */ - { "pstate", 4, offsetof(struct pt_regs, pstate) -#ifdef CONFIG_CPU_BIG_ENDIAN - + 4 -#endif - }, + { "pstate", 4, offsetof(struct pt_regs, pstate)}, { "v0", 16, -1 }, { "v1", 16, -1 }, { "v2", 16, -1 }, diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 8f9bc2327dc8..48f343704cb5 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -6,6 +6,7 @@ * Copyright (C) Huawei Futurewei Technologies. */ +#include <linux/arm-rmi-cmds.h> #include <linux/interrupt.h> #include <linux/irq.h> #include <linux/kernel.h> @@ -59,6 +60,16 @@ int machine_kexec_prepare(struct kimage *kimage) return -EBUSY; } + /* + * We will be able to allow kdump to proceed, once we have the support + * for handling GPF from vmcore accesses to delegated pages. Until then + * block kexec completely. + */ + if (is_rmm_active()) { + pr_err("Can't kexec: RMM is active.\n"); + return -EBUSY; + } + return 0; } diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index f743cbec1c3a..7405c76724bf 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -2517,12 +2517,8 @@ static int valid_compat_regs(struct user_pt_regs *regs) { regs->pstate &= ~SPSR_EL1_AARCH32_RES0_BITS; - if (!system_supports_mixed_endian_el0()) { - if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN)) - regs->pstate |= PSR_AA32_E_BIT; - else - regs->pstate &= ~PSR_AA32_E_BIT; - } + if (!system_supports_mixed_endian_el0()) + regs->pstate &= ~PSR_AA32_E_BIT; if (user_mode(regs) && (regs->pstate & PSR_MODE32_BIT) && (regs->pstate & PSR_AA32_A_BIT) == 0 && diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 29c6100f0c50..a707deba6ae7 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -7,6 +7,7 @@ */ #include <linux/acpi.h> +#include <linux/arm-rsi-cmds.h> #include <linux/export.h> #include <linux/kernel.h> #include <linux/stddef.h> @@ -43,7 +44,6 @@ #include <asm/cpu_ops.h> #include <asm/kasan.h> #include <asm/numa.h> -#include <asm/rsi.h> #include <asm/scs.h> #include <asm/sections.h> #include <asm/setup.h> diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c index bb3b526ff43f..3e1ce0b00a68 100644 --- a/arch/arm64/kernel/signal32.c +++ b/arch/arm64/kernel/signal32.c @@ -70,22 +70,12 @@ static inline int get_sigset_t(sigset_t *set, /* * VFP save/restore code. - * - * We have to be careful with endianness, since the fpsimd context-switch - * code operates on 128-bit (Q) register values whereas the compat ABI - * uses an array of 64-bit (D) registers. Consequently, we need to swap - * the two halves of each Q register when running on a big-endian CPU. */ union __fpsimd_vreg { __uint128_t raw; struct { -#ifdef __AARCH64EB__ - u64 hi; - u64 lo; -#else u64 lo; u64 hi; -#endif }; }; @@ -334,9 +324,6 @@ static void compat_setup_return(struct pt_regs *regs, struct k_sigaction *ka, /* The IT state must be cleared for both ARM and Thumb-2 */ spsr &= ~PSR_AA32_IT_MASK; - /* Restore the original endianness */ - spsr |= PSR_AA32_ENDSTATE; - if (ka->sa.sa_flags & SA_RESTORER) { retcode = ptr_to_compat(ka->sa.sa_restorer); } else { diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index a61dc3016a11..014b163ce658 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -217,8 +217,7 @@ asmlinkage notrace void secondary_start_kernel(void) if (system_uses_irq_prio_masking()) init_gic_priority_masking(); - rcutree_report_cpu_starting(cpu); - trace_hardirqs_off(); + lockdep_hardirqs_off(CALLER_ADDR0); /* * If the system has established the capabilities, make sure @@ -235,12 +234,15 @@ asmlinkage notrace void secondary_start_kernel(void) * Log the CPU info before it is marked online and might get read. */ cpuinfo_store_cpu(); + rcutree_report_cpu_starting(cpu); + update_cpu_features(cpu); store_cpu_topology(cpu); /* * Enable GIC and timers. */ notify_cpu_starting(cpu); + trace_hardirqs_off_finish(); ipi_setup(cpu); @@ -410,7 +412,6 @@ void __noreturn cpu_die_early(void) /* Mark this CPU absent */ set_cpu_present(cpu, 0); - rcutree_report_cpu_dead(); if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) { update_cpu_boot_status(CPU_KILL_ME); @@ -455,7 +456,9 @@ void __init smp_prepare_boot_cpu(void) */ set_my_cpu_offset(per_cpu_offset(smp_processor_id())); - cpuinfo_store_boot_cpu(); + /* Check arm64_ftr_regs[] before anything looks registers up in it. */ + validate_ftr_regs(); + cpuinfo_store_cpu(); setup_boot_cpu_features(); /* Conditionally switch to GIC PMR for interrupt masking */ @@ -779,16 +782,14 @@ void __init smp_init_cpus(void) void __init smp_prepare_cpus(unsigned int max_cpus) { const struct cpu_operations *ops; - int err; unsigned int cpu; - unsigned int this_cpu; + int err; init_cpu_topology(); - this_cpu = smp_processor_id(); - store_cpu_topology(this_cpu); - numa_store_cpu_info(this_cpu); - numa_add_cpu(this_cpu); + store_cpu_topology(0); + numa_store_cpu_info(0); + numa_add_cpu(0); /* * If UP is mandated by "nosmp" (which implies "maxcpus=0"), don't set @@ -803,8 +804,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus) * secondaries from the bootloader. */ for_each_possible_cpu(cpu) { - - if (cpu == smp_processor_id()) + if (cpu == 0) continue; ops = get_cpu_ops(cpu); diff --git a/arch/arm64/kernel/sys32.c b/arch/arm64/kernel/sys32.c index 12a948f3a504..8ac07aeb018d 100644 --- a/arch/arm64/kernel/sys32.c +++ b/arch/arm64/kernel/sys32.c @@ -59,12 +59,7 @@ COMPAT_SYSCALL_DEFINE6(aarch32_mmap2, unsigned long, addr, unsigned long, len, return ksys_mmap_pgoff(addr, len, prot, flags, fd, off_4k); } -#ifdef CONFIG_CPU_BIG_ENDIAN -#define arg_u32p(name) u32, name##_hi, u32, name##_lo -#else #define arg_u32p(name) u32, name##_lo, u32, name##_hi -#endif - #define arg_u64(name) (((u64)name##_hi << 32) | name##_lo) COMPAT_SYSCALL_DEFINE6(aarch32_pread64, unsigned int, fd, char __user *, buf, diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 914282016069..d6391ab20db3 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -586,6 +586,8 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs) break; case ESR_ELx_SYS64_ISS_CRM_IC_IVAU: /* IC IVAU */ __user_cache_maint("ic ivau", address, ret); + if (cpus_have_final_cap(ARM64_WORKAROUND_NXP_ERR050104) && !ret) + asm volatile("ic ialluis"); break; default: force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0); diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile index bea3675fa668..00efdc5d2354 100644 --- a/arch/arm64/kernel/vdso32/Makefile +++ b/arch/arm64/kernel/vdso32/Makefile @@ -45,11 +45,7 @@ endif # From arm Makefile VDSO_CAFLAGS += $(call cc32-option,-fno-dwarf2-cfi-asm) VDSO_CAFLAGS += -mabi=aapcs-linux -mfloat-abi=soft -ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) -VDSO_CAFLAGS += -mbig-endian -else VDSO_CAFLAGS += -mlittle-endian -endif # From arm vDSO Makefile VDSO_CAFLAGS += -fPIC -fno-builtin -fno-stack-protector diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 0576c2022ef5..f2478b561459 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -3163,6 +3163,11 @@ static int __init early_kvm_mode_cfg(char *arg) } early_param("kvm-arm.mode", early_kvm_mode_cfg); +void kvm_force_disabled(void) +{ + kvm_mode = KVM_MODE_NONE; +} + static int __init early_kvm_wfx_trap_policy_cfg(char *arg, enum kvm_wfx_trap_policy *p) { if (!arg) diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c index 1053676551af..e93919cadf86 100644 --- a/arch/arm64/kvm/config.c +++ b/arch/arm64/kvm/config.c @@ -804,7 +804,7 @@ static const struct reg_bits_to_feat_map hfgrtr2_feat_map[] = { HFGRTR2_EL2_nCPACRALIAS_EL1 | HFGRTR2_EL2_nCPACRMASK_EL1 | HFGRTR2_EL2_nSCTLR2MASK_EL1 | - HFGRTR2_EL2_nSCTLRALIAS2_EL1 | + HFGRTR2_EL2_nSCTLR2ALIAS_EL1 | HFGRTR2_EL2_nSCTLRALIAS_EL1 | HFGRTR2_EL2_nSCTLRMASK_EL1 | HFGRTR2_EL2_nTCR2ALIAS_EL1 | @@ -825,7 +825,7 @@ static const struct reg_bits_to_feat_map hfgwtr2_feat_map[] = { HFGWTR2_EL2_nCPACRALIAS_EL1 | HFGWTR2_EL2_nCPACRMASK_EL1 | HFGWTR2_EL2_nSCTLR2MASK_EL1 | - HFGWTR2_EL2_nSCTLRALIAS2_EL1 | + HFGWTR2_EL2_nSCTLR2ALIAS_EL1 | HFGWTR2_EL2_nSCTLRALIAS_EL1 | HFGWTR2_EL2_nSCTLRMASK_EL1 | HFGWTR2_EL2_nTCR2ALIAS_EL1 | diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index f59d89d04eca..fc23c793ef17 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -1431,7 +1431,7 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initconst = { SR_FGT(SYS_CPACRMASK_EL1, HFGRTR2, nCPACRMASK_EL1, 0), SR_FGT(SYS_PFAR_EL1, HFGRTR2, nPFAR_EL1, 0), SR_FGT(SYS_RCWSMASK_EL1, HFGRTR2, nRCWSMASK_EL1, 0), - SR_FGT(SYS_SCTLR2ALIAS_EL1, HFGRTR2, nSCTLRALIAS2_EL1, 0), + SR_FGT(SYS_SCTLR2ALIAS_EL1, HFGRTR2, nSCTLR2ALIAS_EL1, 0), SR_FGT(SYS_SCTLR2MASK_EL1, HFGRTR2, nSCTLR2MASK_EL1, 0), SR_FGT(SYS_SCTLRALIAS_EL1, HFGRTR2, nSCTLRALIAS_EL1, 0), SR_FGT(SYS_SCTLRMASK_EL1, HFGRTR2, nSCTLRMASK_EL1, 0), diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h index 7c7ea8c55405..b6978f03e86a 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -20,11 +20,7 @@ typedef union hyp_spinlock { u32 __val; struct { -#ifdef __AARCH64EB__ - u16 next, owner; -#else u16 owner, next; -#endif }; } hyp_spinlock_t; diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index f57450ebcb49..2ed489bf544e 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -17,7 +17,7 @@ ccflags-y += -fno-stack-protector \ hostprogs := gen-hyprel HOST_EXTRACFLAGS += -I$(objtree)/include -lib-objs := clear_page.o copy_page.o memcpy.o memset.o tishift.o +lib-objs := copy_page.o memcpy.o memset.o tishift.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) CFLAGS_switch.nvhe.o += -Wno-override-init diff --git a/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c b/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c index b63f4e1c1033..99243e0fefb0 100644 --- a/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c +++ b/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c @@ -139,28 +139,12 @@ static struct { const char *sh_string; } elf; -#if defined(CONFIG_CPU_LITTLE_ENDIAN) - #define elf16toh(x) le16toh(x) #define elf32toh(x) le32toh(x) #define elf64toh(x) le64toh(x) #define ELFENDIAN ELFDATA2LSB -#elif defined(CONFIG_CPU_BIG_ENDIAN) - -#define elf16toh(x) be16toh(x) -#define elf32toh(x) be32toh(x) -#define elf64toh(x) be64toh(x) - -#define ELFENDIAN ELFDATA2MSB - -#else - -#error PDP-endian sadly unsupported... - -#endif - #define fatal_error(fmt, ...) \ ({ \ fprintf(stderr, "error: %s: " fmt "\n", \ diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index a2f4e769a428..5e6b04c5567d 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2164,6 +2164,8 @@ static u64 sanitise_id_aa64pfr0_el1(const struct kvm_vcpu *vcpu, u64 val) */ val &= ~ID_AA64PFR0_EL1_MPAM_MASK; + val &= ~ID_AA64PFR0_EL1_RME_MASK; + return val; } diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index b33e1ca4a781..38ffd359123c 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -5,7 +5,7 @@ KCSAN_SANITIZE_delay.o := n lib-y := clear_user.o delay.o copy_from_user.o \ copy_to_user.o copy_page.o \ - clear_page.o csum.o insn.o memchr.o memcpy.o \ + csum.o insn.o memchr.o memcpy.o \ memset.o memcmp.o strcmp.o strncmp.o strlen.o \ strnlen.o strchr.o strrchr.o tishift.o diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S deleted file mode 100644 index bd6f7d5eb6eb..000000000000 --- a/arch/arm64/lib/clear_page.S +++ /dev/null @@ -1,53 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright (C) 2012 ARM Ltd. - */ - -#include <linux/linkage.h> -#include <linux/const.h> -#include <asm/assembler.h> -#include <asm/page.h> - -/* - * Clear page @dest - * - * Parameters: - * x0 - dest - */ -SYM_FUNC_START(__pi_clear_page) -#ifdef CONFIG_AS_HAS_MOPS - .arch_extension mops -alternative_if_not ARM64_HAS_MOPS - b .Lno_mops -alternative_else_nop_endif - - mov x1, #PAGE_SIZE - setpn [x0]!, x1!, xzr - setmn [x0]!, x1!, xzr - seten [x0]!, x1!, xzr - ret -.Lno_mops: -#endif - mrs x1, dczid_el0 - tbnz x1, #4, 2f /* Branch if DC ZVA is prohibited */ - and w1, w1, #0xf - mov x2, #4 - lsl x1, x2, x1 - -1: dc zva, x0 - add x0, x0, x1 - tst x0, #(PAGE_SIZE - 1) - b.ne 1b - ret - -2: stnp xzr, xzr, [x0] - stnp xzr, xzr, [x0, #16] - stnp xzr, xzr, [x0, #32] - stnp xzr, xzr, [x0, #48] - add x0, x0, #64 - tst x0, #(PAGE_SIZE - 1) - b.ne 2b - ret -SYM_FUNC_END(__pi_clear_page) -SYM_FUNC_ALIAS(clear_page, __pi_clear_page) -EXPORT_SYMBOL(clear_page) diff --git a/arch/arm64/lib/csum.c b/arch/arm64/lib/csum.c index 2432683e48a6..75db718e5b6a 100644 --- a/arch/arm64/lib/csum.c +++ b/arch/arm64/lib/csum.c @@ -47,11 +47,7 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len) */ shift = offset * 8; data = *ptr++; -#ifdef __LITTLE_ENDIAN data = (data >> shift) << shift; -#else - data = (data << shift) >> shift; -#endif /* * Body: straightforward aligned loads from here on (the paired loads @@ -94,13 +90,8 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len) len -= 16; ptr += 2; -#ifdef __LITTLE_ENDIAN data = tmp >> 64; sum64 = accumulate(sum64, tmp); -#else - data = tmp; - sum64 = accumulate(sum64, tmp >> 64); -#endif } if (len > 0) { sum64 = accumulate(sum64, data); @@ -112,11 +103,7 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len) * preserving odd/even alignment. */ shift = len * -8; -#ifdef __LITTLE_ENDIAN data = (data << shift) >> shift; -#else - data = (data >> shift) << shift; -#endif sum64 = accumulate(sum64, data); /* Finally, folding */ @@ -140,11 +127,7 @@ __sum16 csum_ipv6_magic(const struct in6_addr *saddr, dst = *(const __uint128_t *)daddr->s6_addr; sum += (__force u32)htonl(len); -#ifdef __LITTLE_ENDIAN sum += (u32)proto << 24; -#else - sum += proto; -#endif src += (src >> 64) | (src << 64); dst += (dst >> 64) | (dst << 64); diff --git a/arch/arm64/lib/memchr.S b/arch/arm64/lib/memchr.S index 37a9f2a4f7f4..b14b27cb8419 100644 --- a/arch/arm64/lib/memchr.S +++ b/arch/arm64/lib/memchr.S @@ -63,7 +63,7 @@ L(byte_loop): sub srcin, srcin, #1 ret L(found_word): -CPU_LE( rev tmp, tmp) + rev tmp, tmp clz tmp, tmp sub tmp, tmp, #64 add result, srcin, tmp, asr #3 diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S index a5ccf2c55f91..1f528ce046e4 100644 --- a/arch/arm64/lib/memcmp.S +++ b/arch/arm64/lib/memcmp.S @@ -103,10 +103,8 @@ L(last_bytes): /* Compare data bytes and set return value to 0, -1 or 1. */ L(return): -#ifndef __AARCH64EB__ rev data1, data1 rev data2, data2 -#endif cmp data1, data2 L(ret_eq): cset result, ne diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S index 9b89b4533607..2389b47dca22 100644 --- a/arch/arm64/lib/strcmp.S +++ b/arch/arm64/lib/strcmp.S @@ -38,14 +38,6 @@ #define shift x9 #define off2 x10 -/* On big-endian early bytes are at MSB and on little-endian LSB. - LS_FW means shifting towards early bytes. */ -#ifdef __AARCH64EB__ -# define LS_FW lsl -#else -# define LS_FW lsr -#endif - /* NUL detection works on the principle that (X - 1) & (~X) & 0x80 (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and can be done in parallel across the entire word. @@ -67,28 +59,17 @@ L(loop_aligned): ldr data2, [src1, off2] ldr data1, [src1], 8 L(start_realigned): -#ifdef __AARCH64EB__ - rev tmp, data1 - sub has_nul, tmp, zeroones - orr tmp, tmp, REP8_7f -#else sub has_nul, data1, zeroones orr tmp, data1, REP8_7f -#endif bics has_nul, has_nul, tmp /* Non-zero if NUL terminator. */ ccmp data1, data2, 0, eq b.eq L(loop_aligned) -#ifdef __AARCH64EB__ - rev has_nul, has_nul -#endif eor diff, data1, data2 orr syndrome, diff, has_nul L(end): -#ifndef __AARCH64EB__ rev syndrome, syndrome rev data1, data1 rev data2, data2 -#endif clz shift, syndrome /* The most-significant-non-zero bit of the syndrome marks either the first bit that is different, or the top bit of the first zero byte. @@ -113,7 +94,7 @@ L(mutual_align): ldr data1, [src1], 8 neg shift, src2, lsl 3 /* Bits to alignment -64. */ mov tmp, -1 - LS_FW tmp, tmp, shift + lsr tmp, tmp, shift orr data1, data1, tmp orr data2, data2, tmp b L(start_realigned) @@ -135,9 +116,6 @@ L(src1_aligned): neg shift, src2, lsl 3 bic src2, src2, 7 ldr data3, [src2], 8 -#ifdef __AARCH64EB__ - rev data3, data3 -#endif lsr tmp, zeroones, shift orr data3, data3, tmp sub has_nul, data3, zeroones @@ -152,9 +130,6 @@ L(src1_aligned): L(loop_unaligned): ldr data3, [src1, off1] ldr data2, [src1, off2] -#ifdef __AARCH64EB__ - rev data3, data3 -#endif sub has_nul, data3, zeroones orr tmp, data3, REP8_7f ldr data1, [src1], 8 @@ -163,9 +138,6 @@ L(loop_unaligned): b.eq L(loop_unaligned) lsl tmp, has_nul, shift -#ifdef __AARCH64EB__ - rev tmp, tmp -#endif eor diff, data1, data2 orr syndrome, diff, tmp cbnz syndrome, L(end) @@ -174,10 +146,6 @@ L(tail): neg shift, shift lsr data2, data3, shift lsr has_nul, has_nul, shift -#ifdef __AARCH64EB__ - rev data2, data2 - rev has_nul, has_nul -#endif eor diff, data1, data2 orr syndrome, diff, has_nul b L(end) diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S index 4919fe81ae54..a8c49a3886f6 100644 --- a/arch/arm64/lib/strlen.S +++ b/arch/arm64/lib/strlen.S @@ -85,14 +85,6 @@ SYM_FUNC_START(__pi_strlen) cmp tmp1, MIN_PAGE_SIZE - 16 b.gt L(page_cross) ldp data1, data2, [srcin] -#ifdef __AARCH64EB__ - /* For big-endian, carry propagation (if the final byte in the - string is 0x01) means we cannot use has_nul1/2 directly. - Since we expect strings to be small and early-exit, - byte-swap the data now so has_null1/2 will be correct. */ - rev data1, data1 - rev data2, data2 -#endif sub tmp1, data1, zeroones orr tmp2, data1, REP8_7f sub tmp3, data2, zeroones @@ -144,19 +136,7 @@ L(page_cross_entry): /* Enter with C = has_nul1 == 0. */ L(tail): -#ifdef __AARCH64EB__ - /* For big-endian, carry propagation (if the final byte in the - string is 0x01) means we cannot use has_nul1/2 directly. The - easiest way to get the correct byte is to byte-swap the data - and calculate the syndrome a second time. */ - csel data1, data1, data2, cc - rev data1, data1 - sub tmp1, data1, zeroones - orr tmp2, data1, REP8_7f - bic has_nul1, tmp1, tmp2 -#else csel has_nul1, has_nul1, has_nul2, cc -#endif sub len, src, srcin rev has_nul1, has_nul1 add tmp2, len, 8 @@ -194,13 +174,7 @@ L(page_cross): ldp data1, data2, [src] lsl tmp1, srcin, 3 mov tmp4, -1 -#ifdef __AARCH64EB__ - /* Big-endian. Early bytes are at MSB. */ - lsr tmp1, tmp4, tmp1 /* Shift (tmp1 & 63). */ -#else - /* Little-endian. Early bytes are at LSB. */ lsl tmp1, tmp4, tmp1 /* Shift (tmp1 & 63). */ -#endif orr tmp1, tmp1, REP8_80 orn data1, data1, tmp1 orn tmp2, data2, tmp1 diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S index fe7bbc0b42a7..e4fde8586089 100644 --- a/arch/arm64/lib/strncmp.S +++ b/arch/arm64/lib/strncmp.S @@ -45,19 +45,6 @@ #define offset pos #define neg_offset x15 -/* Define endian dependent shift operations. - On big-endian early bytes are at MSB and on little-endian LSB. - LS_FW means shifting towards early bytes. - LS_BK means shifting towards later bytes. - */ -#ifdef __AARCH64EB__ -#define LS_FW lsl -#define LS_BK lsr -#else -#define LS_FW lsr -#define LS_BK lsl -#endif - SYM_FUNC_START(__pi_strncmp) cbz limit, L(ret0) eor tmp1, src1, src2 @@ -86,7 +73,6 @@ L(start_realigned): /* End of main loop */ L(full_check): -#ifndef __AARCH64EB__ orr syndrome, diff, has_nul add limit, limit, 8 /* Rewind limit to before last subs. */ L(syndrome_check): @@ -105,55 +91,6 @@ L(syndrome_check): sub result, data1, data2, lsr #56 csel result, result, xzr, hi ret -#else - /* Not reached the limit, must have found the end or a diff. */ - tbz limit, #63, L(not_limit) - add tmp1, limit, 8 - cbz limit, L(not_limit) - - lsl limit, tmp1, #3 /* Bits -> bytes. */ - mov mask, #~0 - lsr mask, mask, limit - bic data1, data1, mask - bic data2, data2, mask - - /* Make sure that the NUL byte is marked in the syndrome. */ - orr has_nul, has_nul, mask - -L(not_limit): - /* For big-endian we cannot use the trick with the syndrome value - as carry-propagation can corrupt the upper bits if the trailing - bytes in the string contain 0x01. */ - /* However, if there is no NUL byte in the dword, we can generate - the result directly. We can't just subtract the bytes as the - MSB might be significant. */ - cbnz has_nul, 1f - cmp data1, data2 - cset result, ne - cneg result, result, lo - ret -1: - /* Re-compute the NUL-byte detection, using a byte-reversed value. */ - rev tmp3, data1 - sub tmp1, tmp3, zeroones - orr tmp2, tmp3, #REP8_7f - bic has_nul, tmp1, tmp2 - rev has_nul, has_nul - orr syndrome, diff, has_nul - clz pos, syndrome - /* The most-significant-non-zero bit of the syndrome marks either the - first bit that is different, or the top bit of the first zero byte. - Shifting left now will bring the critical information into the - top bits. */ -L(end_quick): - lsl data1, data1, pos - lsl data2, data2, pos - /* But we need to zero-extend (char is unsigned) the value and then - perform a signed 32-bit subtraction. */ - lsr data1, data1, #56 - sub result, data1, data2, lsr #56 - ret -#endif L(mutual_align): /* Sources are mutually aligned, but are not currently at an @@ -167,7 +104,7 @@ L(mutual_align): neg tmp3, count, lsl #3 /* 64 - bits(bytes beyond align). */ ldr data2, [src2], #8 mov tmp2, #~0 - LS_FW tmp2, tmp2, tmp3 /* Shift (count & 63). */ + lsr tmp2, tmp2, tmp3 /* Shift (count & 63). */ /* Adjust the limit and ensure it doesn't overflow. */ adds limit, limit, count csinv limit, limit, xzr, lo @@ -236,15 +173,15 @@ L(src1_aligned): neg neg_offset, offset ldr data1, [src1], #8 ldp tmp1, tmp2, [src2], #16 - LS_BK mask, mask, neg_offset + lsl mask, mask, neg_offset and neg_offset, neg_offset, #63 /* Need actual value for cmp later. */ /* Skip the first compare if data in tmp1 is irrelevant. */ tbnz offset, 6, L(misaligned_mid_loop) L(loop_misaligned): /* STEP_A: Compare full 8 bytes when there is enough data from SRC2.*/ - LS_FW data2, tmp1, offset - LS_BK tmp1, tmp2, neg_offset + lsr data2, tmp1, offset + lsl tmp1, tmp2, neg_offset subs limit, limit, #8 orr data2, data2, tmp1 /* 8 bytes from SRC2 combined from two regs.*/ sub has_nul, data1, zeroones @@ -258,23 +195,12 @@ L(loop_misaligned): ldr data1, [src1], #8 L(misaligned_mid_loop): /* STEP_B: Compare first part of data1 to second part of tmp2. */ - LS_FW data2, tmp2, offset -#ifdef __AARCH64EB__ - /* For big-endian we do a byte reverse to avoid carry-propagation - problem described above. This way we can reuse the has_nul in the - next step and also use syndrome value trick at the end. */ - rev tmp3, data1 - #define data1_fixed tmp3 -#else + lsr data2, tmp2, offset #define data1_fixed data1 -#endif sub has_nul, data1_fixed, zeroones orr tmp3, data1_fixed, #REP8_7f eor diff, data2, data1 /* Non-zero if differences found. */ bic has_nul, has_nul, tmp3 /* Non-zero if NUL terminator. */ -#ifdef __AARCH64EB__ - rev has_nul, has_nul -#endif cmp limit, neg_offset, lsr #3 orr syndrome, diff, has_nul bic syndrome, syndrome, mask /* Ignore later bytes. */ @@ -284,7 +210,7 @@ L(misaligned_mid_loop): /* STEP_C: Compare second part of data1 to first part of tmp1. */ ldp tmp1, tmp2, [src2], #16 cmp limit, #8 - LS_BK data2, tmp1, neg_offset + lsl data2, tmp1, neg_offset eor diff, data2, data1 /* Non-zero if differences found. */ orr syndrome, diff, has_nul and syndrome, syndrome, mask /* Ignore earlier bytes. */ @@ -295,13 +221,6 @@ L(misaligned_mid_loop): sub limit, limit, #8 b L(loop_misaligned) -#ifdef __AARCH64EB__ -L(syndrome_check): - clz pos, syndrome - cmp pos, limit, lsl #3 - b.lo L(end_quick) -#endif - L(ret0): mov result, #0 ret diff --git a/arch/arm64/lib/strnlen.S b/arch/arm64/lib/strnlen.S index d5ac0e10a01d..b20011d559f1 100644 --- a/arch/arm64/lib/strnlen.S +++ b/arch/arm64/lib/strnlen.S @@ -91,22 +91,10 @@ SYM_FUNC_START(__pi_strnlen) */ sub len, src, srcin cbz has_nul1, .Lnul_in_data2 -CPU_BE( mov data2, data1 ) /*perpare data to re-calculate the syndrome*/ sub len, len, #8 mov has_nul2, has_nul1 .Lnul_in_data2: - /* - * For big-endian, carry propagation (if the final byte in the - * string is 0x01) means we cannot use has_nul directly. The - * easiest way to get the correct byte is to byte-swap the data - * and calculate the syndrome a second time. - */ -CPU_BE( rev data2, data2 ) -CPU_BE( sub tmp1, data2, zeroones ) -CPU_BE( orr tmp2, data2, #REP8_7f ) -CPU_BE( bic has_nul2, tmp1, tmp2 ) - sub len, len, #8 rev has_nul2, has_nul2 clz pos, has_nul2 @@ -139,10 +127,8 @@ CPU_BE( bic has_nul2, tmp1, tmp2 ) lsl tmp4, tmp4, #3 /* Bytes beyond alignment -> bits. */ mov tmp2, #~0 - /* Big-endian. Early bytes are at MSB. */ -CPU_BE( lsl tmp2, tmp2, tmp4 ) /* Shift (tmp1 & 63). */ /* Little-endian. Early bytes are at LSB. */ -CPU_LE( lsr tmp2, tmp2, tmp4 ) /* Shift (tmp1 & 63). */ + lsr tmp2, tmp2, tmp4 /* Shift (tmp1 & 63). */ cmp tmp1, #8 diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c index 6e0528831cd3..299a244f68e6 100644 --- a/arch/arm64/mm/extable.c +++ b/arch/arm64/mm/extable.c @@ -79,12 +79,7 @@ ex_handler_load_unaligned_zeropad(const struct exception_table_entry *ex, addr &= ~0x7UL; data = *(unsigned long*)addr; - -#ifndef __AARCH64EB__ data >>= 8 * offset; -#else - data <<= 8 * offset; -#endif pt_regs_write_reg(regs, reg_data, data); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 75c3e463df2e..d82118afdf4b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -620,9 +620,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, /* * If we're in an interrupt or have no user context, we must not take - * the fault. + * the fault. EFI runtime services may execute with preemption enabled + * and mm == &efi_mm, but faults need to be treated as kernel faults. */ - if (faulthandler_disabled() || !mm) + if (faulthandler_disabled() || !mm || mm_is_efi(mm)) goto no_context; if (user_mode(regs)) @@ -914,6 +915,29 @@ static int do_tag_check_fault(unsigned long far, unsigned long esr, return 0; } +static int do_gpf_ptw(unsigned long far, unsigned long esr, struct pt_regs *regs) +{ + const struct fault_info *inf = esr_to_fault_info(esr); + unsigned long addr = untagged_addr(far); + + die_kernel_fault(inf->name, addr, esr, regs); + return 0; +} + +static int do_gpf(unsigned long far, unsigned long esr, struct pt_regs *regs) +{ + /* + * Userspace must not have a delegated page mapped in. If the kernel + * is made to access it, then we have a serious problem. + * Only fixup if the access came via kernel VA. e.g., load_unaligned_zeropad() + */ + if (!user_mode(regs) && !is_el1_instruction_abort(esr) && + !is_ttbr0_addr(untagged_addr(far)) && fixup_exception(regs, esr)) + return 0; + + return 1; +} + static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "ttbr address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "level 1 address size fault" }, @@ -950,12 +974,12 @@ static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "unknown 32" }, { do_alignment_fault, SIGBUS, BUS_ADRALN, "alignment fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 34" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 35" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 36" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 37" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 38" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 39" }, - { do_bad, SIGKILL, SI_KERNEL, "unknown 40" }, + { do_gpf_ptw, SIGKILL, SI_KERNEL, "level -1 granule protection fault (translation table walk)" }, + { do_gpf_ptw, SIGKILL, SI_KERNEL, "level 0 granule protection fault (translation table walk)" }, + { do_gpf_ptw, SIGKILL, SI_KERNEL, "level 1 granule protection fault (translation table walk)" }, + { do_gpf_ptw, SIGKILL, SI_KERNEL, "level 2 granule protection fault (translation table walk)" }, + { do_gpf_ptw, SIGKILL, SI_KERNEL, "level 3 granule protection fault (translation table walk)" }, + { do_gpf, SIGBUS, BUS_OBJERR, "granule protection fault" }, { do_bad, SIGKILL, SI_KERNEL, "level -1 address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 42" }, { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level -1 translation fault" }, diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index fbf215ecc7d0..2a8de20d2b44 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -34,6 +34,8 @@ #include <linux/acpi_iort.h> #include <linux/kmemleak.h> #include <linux/execmem.h> +#include <linux/arm-rsi-cmds.h> + #include <asm/boot.h> #include <asm/fixmap.h> @@ -43,7 +45,6 @@ #include <asm/kvm_host.h> #include <asm/memory.h> #include <asm/numa.h> -#include <asm/rsi.h> #include <asm/sections.h> #include <asm/setup.h> #include <linux/sizes.h> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bbe98ac9ad8c..14b2a3801f40 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -275,9 +275,7 @@ int set_direct_map_default_noflush(struct page *page) PAGE_SIZE, set_mask, clear_mask); } -static int __set_memory_enc_dec(unsigned long addr, - int numpages, - bool encrypt) +int __set_memory_enc_dec(unsigned long addr, int numpages, bool encrypt) { unsigned long set_prot = 0, clear_prot = 0; phys_addr_t start, end; @@ -321,40 +319,6 @@ static int __set_memory_enc_dec(unsigned long addr, __pgprot(PTE_PRESENT_INVALID)); } -static int realm_set_memory_encrypted(unsigned long addr, int numpages) -{ - int ret = __set_memory_enc_dec(addr, numpages, true); - - /* - * If the request to change state fails, then the only sensible cause - * of action for the caller is to leak the memory - */ - WARN(ret, "Failed to encrypt memory, %d pages will be leaked", - numpages); - - return ret; -} - -static int realm_set_memory_decrypted(unsigned long addr, int numpages) -{ - int ret = __set_memory_enc_dec(addr, numpages, false); - - WARN(ret, "Failed to decrypt memory, %d pages will be leaked", - numpages); - - return ret; -} - -static const struct arm64_mem_crypt_ops realm_crypt_ops = { - .encrypt = realm_set_memory_encrypted, - .decrypt = realm_set_memory_decrypted, -}; - -int realm_register_memory_enc_ops(void) -{ - return arm64_mem_crypt_ops_register(&realm_crypt_ops); -} - int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) { unsigned long addr = (unsigned long)page_address(page); diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c1279fabce47..8108012cd7e4 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1441,13 +1441,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn case BPF_ALU | BPF_END | BPF_FROM_LE: case BPF_ALU | BPF_END | BPF_FROM_BE: case BPF_ALU64 | BPF_END | BPF_FROM_LE: -#ifdef CONFIG_CPU_BIG_ENDIAN - if (BPF_CLASS(code) == BPF_ALU && BPF_SRC(code) == BPF_FROM_BE) - goto emit_bswap_uxt; -#else /* !CONFIG_CPU_BIG_ENDIAN */ if (BPF_CLASS(code) == BPF_ALU && BPF_SRC(code) == BPF_FROM_LE) goto emit_bswap_uxt; -#endif switch (imm) { case 16: emit(A64_REV16(is64, dst, dst), ctx); @@ -2568,15 +2563,9 @@ static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes) { if (effective_bytes) { int garbage_bits = 64 - 8 * effective_bytes; -#ifdef CONFIG_CPU_BIG_ENDIAN - /* garbage bits are at the right end */ - emit(A64_LSR(1, reg, reg, garbage_bits), ctx); - emit(A64_LSL(1, reg, reg, garbage_bits), ctx); -#else /* garbage bits are at the left end */ emit(A64_LSL(1, reg, reg, garbage_bits), ctx); emit(A64_LSR(1, reg, reg, garbage_bits), ctx); -#endif } } diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 58b6c084f9f2..20cc185649f9 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -124,6 +124,7 @@ WORKAROUND_DEVICE_LOAD_ACQUIRE WORKAROUND_DISABLE_CNP WORKAROUND_GICv3_BROKEN_SEIS WORKAROUND_NVIDIA_OLYMPUS_1027 +WORKAROUND_NXP_ERR050104 WORKAROUND_PMUV3_IMPDEF_TRAPS WORKAROUND_QCOM_FALKOR_E1003 WORKAROUND_QCOM_ORYON_CNTVOFF diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index e2d37ee221b8..b15a59b9e95d 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -434,7 +434,7 @@ EndEnum UnsignedEnum 7:4 State1 0b0000 NI 0b0001 THUMB - 0b0010 THUMB2 + 0b0011 THUMB2 EndEnum UnsignedEnum 3:0 State0 0b0000 NI @@ -469,7 +469,7 @@ UnsignedEnum 15:12 Virtualization EndEnum UnsignedEnum 11:8 MProgMod 0b0000 NI - 0b0001 IMP + 0b0010 IMP EndEnum Enum 7:4 Security 0b0000 NI @@ -1338,10 +1338,7 @@ Res0 63:47 Field 46 CCU Field 45:40 CC_EXP Field 39:32 CC_MANT -Res0 31:18 -Field 17 LASTFAILED -Field 16 T -Res0 15:14 +Res0 31:14 Enum 13:8 TYPE 0b000000 DIRECT_UNCOND 0b000001 INDIRECT @@ -1420,8 +1417,7 @@ Field 17 DIRECT Field 16 EnI Res0 15:8 Field 7 PAUSED -Field 6 LASTFAILED -Res0 5:0 +Res0 6:0 EndSysreg Sysreg BRBTS_EL1 2 1 9 0 2 @@ -2663,11 +2659,7 @@ Field 57 EPAN Field 56 EnALS Field 55 EnAS0 Field 54 EnASR -Field 53 TME -Field 52 TME0 -Field 51 TMT -Field 50 TMT0 -Res0 49:47 +Res0 53:47 Field 46 TWEDEL Field 45 TWEDEn Field 44 DSSBS @@ -2985,7 +2977,7 @@ Field 27 NSH Field 26 M Res0 25 Field 24 SH -Field 23 T +Res0 23 Field 22 RLK Field 21 RLU Field 20 RLH @@ -4336,7 +4328,7 @@ Field 14 nACTLRALIAS_EL1 Field 13 nACTLRMASK_EL1 Field 12 nTCR2ALIAS_EL1 Field 11 nTCRALIAS_EL1 -Field 10 nSCTLRALIAS2_EL1 +Field 10 nSCTLR2ALIAS_EL1 Field 9 nSCTLRALIAS_EL1 Field 8 nCPACRALIAS_EL1 Field 7 nTCR2MASK_EL1 @@ -4355,7 +4347,7 @@ Field 14 nACTLRALIAS_EL1 Field 13 nACTLRMASK_EL1 Field 12 nTCR2ALIAS_EL1 Field 11 nTCRALIAS_EL1 -Field 10 nSCTLRALIAS2_EL1 +Field 10 nSCTLR2ALIAS_EL1 Field 9 nSCTLRALIAS_EL1 Field 8 nCPACRALIAS_EL1 Field 7 nTCR2MASK_EL1 @@ -4592,7 +4584,7 @@ Field 55:52 P13 Field 51:48 P12 Field 47:44 P11 Field 43:40 P10 -Field 39:36 F9 +Field 39:36 P9 Field 35:32 P8 Field 31:28 P7 Field 27:24 P6 @@ -4652,7 +4644,7 @@ Enum 15:14 TG0 EndEnum Enum 13:12 SH0 0b00 NONE - 0b01 OUTER + 0b10 OUTER 0b11 INNER EndEnum Enum 11:10 ORGN0 diff --git a/drivers/acpi/arm64/mpam.c b/drivers/acpi/arm64/mpam.c index 84963a20c3e7..c57bd367922b 100644 --- a/drivers/acpi/arm64/mpam.c +++ b/drivers/acpi/arm64/mpam.c @@ -220,8 +220,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_ struct platform_device *pdev __free(platform_device_put) = platform_device_alloc("mpam_msc", tbl_msc->identifier); int next_res = 0, next_prop = 0, err; - /* pcc, nrdy, affinity and a sentinel */ - struct property_entry props[4] = { 0 }; + /* pcc, msc-id, nrdy, affinity and a sentinel */ + struct property_entry props[5] = { 0 }; /* mmio, 2xirq, no sentinel. */ struct resource res[3] = { 0 }; struct acpi_device *companion; @@ -256,6 +256,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_ } else if (iface == MPAM_IFACE_PCC) { props[next_prop++] = PROPERTY_ENTRY_U32("pcc-channel", tbl_msc->base_address); + props[next_prop++] = PROPERTY_ENTRY_U32("mpam-fb-msc-id", + tbl_msc->identifier); } acpi_mpam_parse_irqs(pdev, tbl_msc, res, &next_res); diff --git a/drivers/char/hw_random/arm_smccc_trng.c b/drivers/char/hw_random/arm_smccc_trng.c index dcb8e7f37f25..01c1a30cbab8 100644 --- a/drivers/char/hw_random/arm_smccc_trng.c +++ b/drivers/char/hw_random/arm_smccc_trng.c @@ -16,8 +16,10 @@ #include <linux/device.h> #include <linux/hw_random.h> #include <linux/module.h> -#include <linux/platform_device.h> #include <linux/arm-smccc.h> +#include <linux/arm-smccc-bus.h> + +#include <asm/archrandom.h> #ifdef CONFIG_ARM64 #define ARM_SMCCC_TRNG_RND ARM_SMCCC_TRNG_RND64 @@ -94,29 +96,37 @@ static int smccc_trng_read(struct hwrng *rng, void *data, size_t max, bool wait) return copied; } -static int smccc_trng_probe(struct platform_device *pdev) +static int smccc_trng_probe(struct arm_smccc_device *sdev) { struct hwrng *trng; - trng = devm_kzalloc(&pdev->dev, sizeof(*trng), GFP_KERNEL); + /* validate the minimum version requirement */ + if (!smccc_probe_trng()) + return -ENODEV; + + trng = devm_kzalloc(&sdev->dev, sizeof(*trng), GFP_KERNEL); if (!trng) return -ENOMEM; trng->name = "smccc_trng"; trng->read = smccc_trng_read; - return devm_hwrng_register(&pdev->dev, trng); + return devm_hwrng_register(&sdev->dev, trng); } -static struct platform_driver smccc_trng_driver = { - .driver = { - .name = "smccc_trng", - }, - .probe = smccc_trng_probe, +static const struct arm_smccc_device_id smccc_trng_id_table[] = { + { .func_id = ARM_SMCCC_TRNG_VERSION }, + {} +}; +MODULE_DEVICE_TABLE(arm_smccc, smccc_trng_id_table); + +static struct arm_smccc_driver smccc_trng_driver = { + .name = KBUILD_MODNAME, + .probe = smccc_trng_probe, + .id_table = smccc_trng_id_table, }; -module_platform_driver(smccc_trng_driver); +module_arm_smccc_driver(smccc_trng_driver); -MODULE_ALIAS("platform:smccc_trng"); MODULE_AUTHOR("Andre Przywara"); MODULE_DESCRIPTION("Arm SMCCC TRNG firmware interface support"); MODULE_LICENSE("GPL"); diff --git a/drivers/crypto/hisilicon/sec2/sec_main.c b/drivers/crypto/hisilicon/sec2/sec_main.c index 752565200c16..03523cfbb28e 100644 --- a/drivers/crypto/hisilicon/sec2/sec_main.c +++ b/drivers/crypto/hisilicon/sec2/sec_main.c @@ -484,7 +484,7 @@ static void sec_set_endian(struct hisi_qm *qm) if (!IS_ENABLED(CONFIG_64BIT)) reg |= BIT(1); - if (!IS_ENABLED(CONFIG_CPU_LITTLE_ENDIAN)) + if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN)) reg |= BIT(0); writel_relaxed(reg, qm->io_base + SEC_CONTROL_REG); diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index c7c91e8dc51d..b6dcfcf6eb5b 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -892,6 +892,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) dev_dbg(&pdev->dev, "No CXL FWCTL setup\n"); + rc = cxl_event_config(host_bridge, mds, irq_avail); + if (rc) + return rc; + pmu_count = cxl_count_regblock(pdev, CXL_REGLOC_RBI_PMU); if (pmu_count < 0) return pmu_count; @@ -918,13 +922,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) } } - rc = cxl_event_config(host_bridge, mds, irq_avail); - if (rc) - return rc; - pci_save_state(pdev); - return rc; + /* A missing PMU is not fatal, the memdev is still usable */ + return 0; } static const struct pci_device_id cxl_mem_pci_tbl[] = { diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig index b7cc11e4fbfa..f1a76d482429 100644 --- a/drivers/firmware/Kconfig +++ b/drivers/firmware/Kconfig @@ -297,6 +297,7 @@ config TURRIS_MOX_RWTM_KEYCTL endif # TURRIS_MOX_RWTM source "drivers/firmware/arm_ffa/Kconfig" +source "drivers/firmware/arm_rmm/Kconfig" source "drivers/firmware/broadcom/Kconfig" source "drivers/firmware/cirrus/Kconfig" source "drivers/firmware/google/Kconfig" diff --git a/drivers/firmware/Makefile b/drivers/firmware/Makefile index be46f1e1dc77..54755db0cf6e 100644 --- a/drivers/firmware/Makefile +++ b/drivers/firmware/Makefile @@ -25,6 +25,7 @@ obj-$(CONFIG_TRUSTED_FOUNDATIONS) += trusted_foundations.o obj-$(CONFIG_TURRIS_MOX_RWTM) += turris-mox-rwtm.o obj-y += arm_ffa/ +obj-y += arm_rmm/ obj-y += arm_scmi/ obj-y += broadcom/ obj-y += cirrus/ diff --git a/drivers/firmware/arm_rmm/Kconfig b/drivers/firmware/arm_rmm/Kconfig new file mode 100644 index 000000000000..ad81330d056b --- /dev/null +++ b/drivers/firmware/arm_rmm/Kconfig @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: GPL-2.0-only +config ARCH_SUPPORTS_RMM + bool + +config ARM_RMM_RMI + bool "Realm Management Interface (RMI) Support" + depends on ARCH_SUPPORTS_RMM + help + Support the Realm Management Monitor (RMM) on Arm systems that + implement the Realm Management Extension (RME), as defined by the + Arm Confidential Compute Architecture. + + The RMM runs at EL2 in the Realm world and provides the Realm + Management Interface (RMI) used by a Normal World host to create, + manage and run protected virtual machines called Realms. The RMM can + also act as a TSM, as defined by the PCIe TDISP and can manage the + PCI IDE setup for securing the PCIe links. + + This option builds the host-side RMI support used by KVM to detect a + compatible RMM, configure it, manage delegated memory and enable + Realm guests. + + Selecting this option does not by itself make Realm guests available: + the system must also provide RME-capable hardware and firmware with a + compatible RMM implementation. + +config ARM_RMM_RSI + bool "Realm Service Interface (RSI) Support" + depends on ARCH_SUPPORTS_RMM + default y + help + Enable support for running Linux as a Realm guest under a Realm + Management Monitor (RMM), as defined by the Arm Confidential + Compute Architecture. + + The RMM exposes the Realm Service Interface (RSI) to Realm guests. + Linux uses RSI to negotiate the interface version, obtain the Realm + configuration, and manage the protected or shared state of guest + memory. diff --git a/drivers/firmware/arm_rmm/Makefile b/drivers/firmware/arm_rmm/Makefile new file mode 100644 index 000000000000..8b7c8bc6beb0 --- /dev/null +++ b/drivers/firmware/arm_rmm/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_ARM_RMM_RMI) += rmi.o +obj-$(CONFIG_ARM_RMM_RSI) += rsi.o diff --git a/drivers/firmware/arm_rmm/rmi.c b/drivers/firmware/arm_rmm/rmi.c new file mode 100644 index 000000000000..77255a90e0a6 --- /dev/null +++ b/drivers/firmware/arm_rmm/rmi.c @@ -0,0 +1,1135 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2023-2026 ARM Ltd. + */ + +#include <linux/arm-rmi-cmds.h> +#include <linux/cpufeature.h> +#include <linux/memblock.h> +#include <linux/memory.h> +#include <linux/mmzone.h> +#include <linux/slab.h> + +#include <asm/memory.h> +#include <asm/pgtable-hwdef.h> + +/* RMM v2.0 defines RmiFeatureRegister0 to RmiFeatureRegister4. */ +static unsigned long rmi_feat_reg_cache[5] __ro_after_init; +static bool arm64_rmm_active; +static bool arm64_rmi_is_available; + +/* is_rmm_active: Returns if the RMM is ACTIVE */ +bool is_rmm_active(void) +{ + return READ_ONCE(arm64_rmm_active); +} +EXPORT_SYMBOL_GPL(is_rmm_active); + +/* + * is_rmi_available: Returns if the RMM is ACTIVE and is configured with static + * carveout for metadata nad L1GPTs. + */ +bool is_rmi_available(void) +{ + return READ_ONCE(arm64_rmi_is_available); +} +EXPORT_SYMBOL_GPL(is_rmi_available); + +/* + * rmi_granule_range_delegate() - Delegate granules + * @base: PA of the first granule of the range + * @top: PA of the first granule after the range + * @out_top: PA of the first granule not delegated + * + * Delegate a range of granule for use by the realm world. If the entire range + * was delegated then @out_top == @top, otherwise the function should be called + * again with @base == @out_top. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static long rmi_granule_range_delegate(unsigned long base, + unsigned long top, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_GRANULE_RANGE_DELEGATE, base, top + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +/* + * rmi_delegate_range: Delegate a physically contiguous range. + * We iterate over the range until we hit an error. So we may + * return an error, but with a partially delegated range. The + * caller must always look at the @out_phys to figure out, how + * much progress was made. + * + * @phys: Base of the physical address range + * @size: Size of the physical address range + * @out_phys: Top of the range that was completed. This is always + * valid, irrespective of the result. + * + * Returns RMI_SUCCESS on successful completion. Otherwise, returns + * the Linux error number or the RMI status code as described + * by the RMM spec for RMI_GRANULE_DELEGATE_RANGE or RMI_BLOCKED. + */ +int rmi_delegate_range(phys_addr_t phys, + unsigned long size, + phys_addr_t *out_phys) +{ + long ret = 0; + unsigned long top = phys + size; + unsigned long out_top; + + while (phys < top) { + ret = rmi_granule_range_delegate(phys, top, &out_top); + + if (ret != RMI_SUCCESS) + break; + /* + * Buggy RMM ? Let the caller handle the failure. + * We can't know how far the RMM delegated in this + * iteration, so we return the best known good limit. + * RMM can deal with granules already in "undelegated" + * in a given range. So, it is fine for the caller to + * try the range we return. + */ + if (WARN_ON(out_top <= phys)) { + ret = -ENXIO; + break; + } + phys = out_top; + } + + if (out_phys) + *out_phys = phys; + + return ret; +} +EXPORT_SYMBOL_GPL(rmi_delegate_range); + +/* + * rmi_granule_range_undelegate() - Undelegate a range of granules + * @base: Base PA of the target range + * @top: Top PA of the target range + * @out_top: Returns the top PA of range whose state is undelegated + * + * Undelegate a range of granules to allow use by the normal world. Will fail + * if the granules are in use by RMM. RMM can ignore granules that are already + * undelegated and thus is safe to be called on a range with a mix of delegated + * and undelegated granules. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static long rmi_granule_range_undelegate(unsigned long base, + unsigned long top, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_GRANULE_RANGE_UNDELEGATE, base, top + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +int rmi_undelegate_range(phys_addr_t phys, + unsigned long size) +{ + long ret = 0; + unsigned long top = phys + size; + unsigned long out_top; + + while (phys < top) { + ret = rmi_granule_range_undelegate(phys, top, &out_top); + + if (ret != RMI_SUCCESS) + break; + /* Buggy RMM ? Let the caller leak the pages */ + if (WARN_ON(out_top <= phys)) + return -ENXIO; + phys = out_top; + } + + return ret; +} +EXPORT_SYMBOL_GPL(rmi_undelegate_range); + +/* + * rmi_free_delegated_page: Undelegate and free a page that has been previously + * delegated to the Realm world. If we are unable to undelegate it, the page is + * leaked. + * NOTE: Do not use this helper if the page could be concurrently operated by + * another thread, as it may get leaked if the undelegation fails due to RMI_BLOCKED + */ +int rmi_free_delegated_page(phys_addr_t phys) +{ + if (WARN_ON_ONCE(rmi_undelegate_page(phys))) { + /* Undelegate failed: leak the page */ + return -EBUSY; + } + + free_page((unsigned long)phys_to_virt(phys)); + + return 0; +} +EXPORT_SYMBOL_GPL(rmi_free_delegated_page); + +/* + * Convert the RmiAddrBlockSize to actual size. This is used in RmiDonateReq + * and RmiAddrRangeDesc*. + */ +static unsigned long rmi_addr_block_size_to_bytes(unsigned long block_size_fld) +{ + return BIT(ARM64_HW_PGTABLE_LEVEL_SHIFT(3 - block_size_fld)); +} + +/* + * free_addr_range: Free memory described by the address range entry, which may + * be partially consumed by RMM. + * + * @entry: RMI_ADDR_RANGE descriptor + * @consumed_size: Page aligned size consumed by the RMM from the address range. + * + * If the state of the address is DELEGATED, undelegate it back, before freeing. + * Leaks the memory if we cannot undelegate the range. + */ +static void free_addr_range(unsigned long entry, unsigned long consumed_size) +{ + unsigned long phys = RMI_ADDR_RANGE_ADDR(entry); + unsigned long block_size_fld = RMI_ADDR_RANGE_BLOCK_SIZE(entry); + unsigned long count = RMI_ADDR_RANGE_COUNT(entry); + unsigned long state = RMI_ADDR_RANGE_STATE(entry); + unsigned long size = rmi_addr_block_size_to_bytes(block_size_fld) * count; + + WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(consumed_size)); + + /* We shouldn't see this in reclaim path, leak it for now */ + if (WARN_ON((state != RMI_OP_MEM_DELEGATED) && + (state != RMI_OP_MEM_UNDELEGATED))) + return; + + /* Adjust the address and size for partially consumed entry */ + phys += consumed_size; + size -= consumed_size; + /* + * Undelegate the pages back if required. If we can't + * change them back, leak the pages. + */ + if (state == RMI_OP_MEM_DELEGATED && + WARN_ON(rmi_undelegate_range(phys, size))) + return; + free_pages_exact(phys_to_virt(phys), size); +} + +static void rmi_op_continue(unsigned long sro_handle, unsigned long flags, + struct arm_smccc_1_2_regs *out_regs) +{ + *out_regs = (struct arm_smccc_1_2_regs) { + SMC_RMI_OP_CONTINUE, sro_handle, flags + }; + + rmi_smccc_invoke(out_regs); +} + +static void rmi_op_cancel(unsigned long sro_handle, + struct arm_smccc_1_2_regs *out_regs) +{ + *out_regs = (struct arm_smccc_1_2_regs) { + SMC_RMI_OP_CANCEL, sro_handle + }; + + rmi_smccc_invoke(out_regs); +} + +static void rmi_op_mem_donate(unsigned long sro_handle, unsigned long list_addr, + unsigned long list_count, unsigned long flags, + struct arm_smccc_1_2_regs *out_regs) +{ + *out_regs = (struct arm_smccc_1_2_regs) { + SMC_RMI_OP_MEM_DONATE, sro_handle, list_addr, list_count, flags + }; + + /* + * The output donated count (a1) is always valid, irrespective + * of the return result. i.e., 0 if there was an error + */ + rmi_smccc_invoke(out_regs); +} + +static void rmi_op_mem_reclaim(unsigned long sro_handle, + unsigned long list_addr, + unsigned long list_count, + struct arm_smccc_1_2_regs *out_regs) +{ + *out_regs = (struct arm_smccc_1_2_regs) { + SMC_RMI_OP_MEM_RECLAIM, sro_handle, list_addr, list_count + }; + + rmi_smccc_invoke(out_regs); +} + +static int rmi_sro_ensure_capacity(struct rmi_sro_state *sro, + unsigned long count) +{ + if (WARN_ON_ONCE(sro->addr_count > RMI_MAX_ADDR_LIST)) + return -EOVERFLOW; + + if (count > RMI_MAX_ADDR_LIST - sro->addr_count) + return -ENOSPC; + + return 0; +} + +static int rmi_sro_donate_contig(struct rmi_sro_state *sro, + unsigned long sro_handle, + unsigned long donatereq, + struct arm_smccc_1_2_regs *out_regs, + gfp_t gfp) +{ + unsigned long block_size_fld = RMI_DONATE_BLOCK_SIZE(donatereq); + unsigned long block_size = rmi_addr_block_size_to_bytes(block_size_fld); + unsigned long count = RMI_DONATE_COUNT(donatereq); + unsigned long state = RMI_DONATE_STATE(donatereq); + unsigned long size = block_size * count; + unsigned long addr_range; + unsigned long donated_size; + int ret; + void *virt; + phys_addr_t phys; + + /* + * The RMM specification requires contiguous allocations are always a + * power of 2 + */ + if (WARN_ON_ONCE(!is_power_of_2(size))) + return -EINVAL; + /* + * RMM clamps the Maximum value of RmiOpMemDonateReq:count to prevent + * overflow in the RMI_ADDR_RANGE_COUNT field. + */ + if (WARN_ON_ONCE(count > FIELD_MAX(RMI_ADDR_RANGE_COUNT_MASK))) + return -EINVAL; + + /* + * We can't allocate blocks more than what the buddy allocator can + * satisfy. TODO: Add support for larger allocations. + */ + if (get_order(size) > MAX_PAGE_ORDER) + return -ENOMEM; + + /* Reuse the cached address range if we have one */ + for (int i = 0; i < sro->addr_count; i++) { + unsigned long entry = sro->addr_list[i]; + + if (RMI_ADDR_RANGE_BLOCK_SIZE(entry) == block_size_fld && + RMI_ADDR_RANGE_COUNT(entry) == count && + RMI_ADDR_RANGE_STATE(entry) == state && + IS_ALIGNED(RMI_ADDR_RANGE_ADDR(entry), size)) { + sro->addr_count--; + swap(sro->addr_list[sro->addr_count], + sro->addr_list[i]); + + goto mem_donate; + } + } + + ret = rmi_sro_ensure_capacity(sro, 1); + /* In the odd case we have filled up the cache, free up the entire list */ + if (ret) + rmi_sro_free(sro); + + virt = alloc_pages_exact(size, gfp); + if (!virt) + return -ENOMEM; + phys = virt_to_phys(virt); + + if (state == RMI_OP_MEM_DELEGATED) { + phys_addr_t delegated_phys; + + if (rmi_delegate_range(phys, size, &delegated_phys)) { + if (!rmi_undelegate_range(phys, delegated_phys - phys)) + free_pages_exact(virt, size); + return -ENXIO; + } + } + + addr_range = phys & RMI_ADDR_RANGE_ADDR_MASK; + FIELD_MODIFY(RMI_ADDR_RANGE_BLOCK_SIZE_MASK, &addr_range, block_size_fld); + FIELD_MODIFY(RMI_ADDR_RANGE_COUNT_MASK, &addr_range, count); + FIELD_MODIFY(RMI_ADDR_RANGE_STATE_MASK, &addr_range, state); + + sro->addr_list[sro->addr_count] = addr_range; + +mem_donate: + rmi_op_mem_donate(sro_handle, + virt_to_phys(&sro->addr_list[sro->addr_count]), 1, + 0, out_regs); + donated_size = out_regs->a1 << PAGE_SHIFT; + + if (WARN_ON(out_regs->a1 > (size >> PAGE_SHIFT))) + donated_size = size; + + /* All granules consumed by the RMM */ + if (donated_size == size) + return 0; + /* + * No granules were consumed by the RMM, cache them. This is a valid + * case, where the RMM was donated memory by another competing thread. + */ + if (donated_size == 0) { + sro->addr_count++; + return 0; + } + + /* The granules were partially consumed, reclaim the unused ones. */ + free_addr_range(sro->addr_list[sro->addr_count], donated_size); + + return 0; +} + +static int rmi_sro_donate_noncontig(struct rmi_sro_state *sro, + unsigned long sro_handle, + unsigned long donatereq, + struct arm_smccc_1_2_regs *out_regs, + gfp_t gfp) +{ + unsigned long block_size_fld = RMI_DONATE_BLOCK_SIZE(donatereq); + unsigned long block_size = rmi_addr_block_size_to_bytes(block_size_fld); + unsigned long count = RMI_DONATE_COUNT(donatereq); + unsigned long state = RMI_DONATE_STATE(donatereq); + unsigned long found = 0; + unsigned long donated_granules; + unsigned long granules_per_block = block_size >> PAGE_SHIFT; + unsigned long consumed_blocks; + int addr_list_start = sro->addr_count; + int ret, i, src; + + /* + * We can't allocate blocks more than what the buddy allocator can + * satisfy. TODO: Add support for larger allocations. + */ + if (get_order(block_size) > MAX_PAGE_ORDER) + return -ENOMEM; + /* + * Maximum value of RmiOpMemDonateReq:count is 2^(PAGE_SHIFT-2) - 1. + * But we further clamp it down by the number of entries we can do + * in one go. The RMM can request the remaining in the next iteration. + */ + if (count > RMI_MAX_ADDR_LIST) + count = RMI_MAX_ADDR_LIST; + + /* Gather the suitable entries to the end of the list */ + i = 0; + while (i < addr_list_start && found < count) { + unsigned long entry = sro->addr_list[i]; + + if (RMI_ADDR_RANGE_BLOCK_SIZE(entry) == block_size_fld && + RMI_ADDR_RANGE_COUNT(entry) == 1 && + RMI_ADDR_RANGE_STATE(entry) == state) { + addr_list_start--; + swap(sro->addr_list[addr_list_start], + sro->addr_list[i]); + found++; + /* Continue from the swapped in entry */ + continue; + } + /* Skip past the entry */ + i++; + } + + ret = rmi_sro_ensure_capacity(sro, count - found); + if (ret) { + /* If we have found some entries, donate them and try again */ + if (found) + goto mem_donate; + /* Otherwise free up the list and start again */ + rmi_sro_free(sro); + /* Reset the addr_list_start to match sro->addr_count */ + addr_list_start = 0; + } + + for (; found < count; found++) { + unsigned long addr_range; + void *virt = alloc_pages_exact(block_size, gfp); + phys_addr_t phys; + + if (!virt) + return -ENOMEM; + + phys = virt_to_phys(virt); + + if (state == RMI_OP_MEM_DELEGATED) { + phys_addr_t delegated_phys; + + if (rmi_delegate_range(phys, block_size, &delegated_phys)) { + if (!rmi_undelegate_range(phys, delegated_phys - phys)) + free_pages_exact(virt, block_size); + return -ENXIO; + } + } + + addr_range = phys & RMI_ADDR_RANGE_ADDR_MASK; + FIELD_MODIFY(RMI_ADDR_RANGE_BLOCK_SIZE_MASK, &addr_range, block_size_fld); + FIELD_MODIFY(RMI_ADDR_RANGE_COUNT_MASK, &addr_range, 1); + FIELD_MODIFY(RMI_ADDR_RANGE_STATE_MASK, &addr_range, state); + + sro->addr_list[sro->addr_count++] = addr_range; + } + +mem_donate: + rmi_op_mem_donate(sro_handle, + virt_to_phys(&sro->addr_list[addr_list_start]), + found, 0, out_regs); + + donated_granules = out_regs->a1; + /* + * The RMM shouldn't report more granules than we provided, but clamp + * just in case. + */ + if (WARN_ON_ONCE(donated_granules > found * granules_per_block)) + donated_granules = found * granules_per_block; + + /* + * The RMM reports the consumed memory in terms of granules, but we + * track in the address lists in block-sized ranges. So divide to get + * the number of (complete) consumed blocks. + */ + consumed_blocks = donated_granules / granules_per_block; + if (donated_granules % granules_per_block) { + /* + * A block has been partially consumed, the start is owned by + * the RMM, the tail is owned by the host + */ + unsigned long entry = + sro->addr_list[addr_list_start + consumed_blocks]; + unsigned long donated_size = + (donated_granules % granules_per_block) << PAGE_SHIFT; + + free_addr_range(entry, donated_size); + /* + * This block is now fully 'consumed' (either held by the RMM or + * freed) + */ + consumed_blocks++; + } + + /* + * Keep just the blocks the RMM didn't use in addr_list + * RMM claimed consumed_blocks entries from addr_list_start. + * Move the entries left out at the end i.e., + * [ addr_list_start + consumed_blocks, addr_list_start + found) + * to the rest of the valid entries and adjust the addr_count to + * reflect the available entries. + */ + for (i = 0, src = addr_list_start + consumed_blocks; + i < found - consumed_blocks; i++) + sro->addr_list[addr_list_start + i] = sro->addr_list[src + i]; + + sro->addr_count -= consumed_blocks; + + return 0; +} + +static int rmi_sro_donate(struct rmi_sro_state *sro, + unsigned long sro_handle, + unsigned long donatereq, + struct arm_smccc_1_2_regs *regs, + gfp_t gfp) +{ + unsigned long state = RMI_DONATE_STATE(donatereq); + + if (WARN_ON_ONCE(!RMI_DONATE_COUNT(donatereq))) + return -EINVAL; + + /* + * We do not support RMI_OP_MEM_CONDITIONAL yet. This is only required + * for use in RMI_GRANULE_TRACKING_SET, which we don't support yet. + */ + if (WARN_ON_ONCE((state != RMI_OP_MEM_DELEGATED) && + (state != RMI_OP_MEM_UNDELEGATED))) + return -EINVAL; + + if (RMI_DONATE_CONTIG(donatereq) == RMI_OP_MEM_CONTIG) + return rmi_sro_donate_contig(sro, sro_handle, donatereq, regs, gfp); + else + return rmi_sro_donate_noncontig(sro, sro_handle, donatereq, regs, gfp); +} + +static int rmi_sro_reclaim(struct rmi_sro_state *sro, + unsigned long sro_handle, + struct arm_smccc_1_2_regs *out_regs) +{ + unsigned long capacity; + + /* + * We don't do a partial free of the entries. So for now free the + * entire address list as we prepare to reclaim more from the RMM. + */ + if (rmi_sro_ensure_capacity(sro, 1)) + rmi_sro_free(sro); + + capacity = RMI_MAX_ADDR_LIST - sro->addr_count; + + rmi_op_mem_reclaim(sro_handle, + virt_to_phys(&sro->addr_list[sro->addr_count]), + capacity, out_regs); + + /* + * RMI_OP_MEM_RECLAIM always return RMI_INCOMPLETE, except when the + * input parameters were invalid. + */ + if (WARN_ON_ONCE(RMI_RESULT_STATUS(out_regs->a0) != RMI_INCOMPLETE)) + return -EINVAL; + if (WARN_ON_ONCE(out_regs->a1 > capacity)) + out_regs->a1 = capacity; + + sro->addr_count += out_regs->a1; + + return 0; +} + +void rmi_sro_free(struct rmi_sro_state *sro) +{ + /* Handle the worse */ + if (WARN_ON(sro->addr_count < 0)) + return; + + if (WARN_ON(sro->addr_count > RMI_MAX_ADDR_LIST)) + sro->addr_count = RMI_MAX_ADDR_LIST; + + for (int i = 0; i < sro->addr_count; i++) + free_addr_range(sro->addr_list[i], 0); + + sro->addr_count = 0; +} +EXPORT_SYMBOL_GPL(rmi_sro_free); + +long rmi_sro_memxfer_execute(struct rmi_sro_state *sro, gfp_t gfp) +{ + struct arm_smccc_1_2_regs *regs = &sro->regs; + bool cancelled = false; + unsigned long sro_handle; + + rmi_smccc_invoke(regs); + + sro_handle = regs->a1; + while (RMI_RESULT_STATUS(regs->a0) == RMI_INCOMPLETE) { + bool can_cancel = RMI_RESULT_CAN_CANCEL(regs->a0) == RMI_OP_CAN_CANCEL; + int ret = 0; + + switch (RMI_RESULT_MEMREQ(regs->a0)) { + case RMI_OP_MEM_REQ_NONE: + rmi_op_continue(sro_handle, RMI_CONTINUE_KEEP_GOING, + regs); + break; + case RMI_OP_MEM_REQ_DONATE: + ret = rmi_sro_donate(sro, sro_handle, regs->a2, regs, + gfp); + break; + case RMI_OP_MEM_REQ_RECLAIM: + ret = rmi_sro_reclaim(sro, sro_handle, regs); + break; + default: + WARN_ON_ONCE(1); + ret = -ENXIO; + } + + if (ret) { + /* + * All memory donating SROs must be cancellable. So a + * failure in memory allocation shouldn't be an issue. + * However, if we encounter a random failure (e.g., + * buggy RMM), don't loop forever, just give up. + */ + if (WARN_ON_ONCE(!can_cancel)) + return ret; + /* + * If we have already cancelled, and came back here due + * to an error in MEMREQ, then there is no point + * in going in loops. + */ + if (WARN_ON_ONCE(cancelled)) + break; + rmi_op_cancel(sro_handle, regs); + cancelled = true; + + if (WARN_ON_ONCE(RMI_RESULT_STATUS(regs->a0) != RMI_INCOMPLETE)) + return ret; + } + } + + if (cancelled) + return -ECANCELED; + + return regs->a0; +} +EXPORT_SYMBOL_GPL(rmi_sro_memxfer_execute); + +/* + * rmi_sro_execute: Execute an RMI command that is stateful but not memory + * transferring. Takes regs, filled with the FIDs and the arguments in place. + * + * Returns : + * -ECANCELED - If the operation had to be aborted and SRO was cancellable. + * Otherwise, returns the result of the RMI command. + */ +long rmi_sro_execute(struct arm_smccc_1_2_regs *regs) +{ + bool cancelled = false; + unsigned long sro_handle; + + rmi_smccc_invoke(regs); + + sro_handle = regs->a1; + while (RMI_RESULT_STATUS(regs->a0) == RMI_INCOMPLETE) { + bool can_cancel = RMI_RESULT_CAN_CANCEL(regs->a0) == RMI_OP_CAN_CANCEL; + + switch (RMI_RESULT_MEMREQ(regs->a0)) { + case RMI_OP_MEM_REQ_NONE: + rmi_op_continue(sro_handle, RMI_CONTINUE_KEEP_GOING, + regs); + break; + default: + WARN_ON_ONCE(1); + if (!can_cancel) + return regs->a0; + /* + * We can't get here normally, but handle this anyway + * for a buggy RMM implementation. + */ + if (cancelled) + return -ECANCELED; + rmi_op_cancel(sro_handle, regs); + cancelled = true; + } + } + + if (cancelled) + return -ECANCELED; + + return regs->a0; +} +EXPORT_SYMBOL_GPL(rmi_sro_execute); + +static int rmi_check_version(void) +{ + unsigned short version_major, version_minor; + unsigned long host_version = RMI_ABI_VERSION(RMI_ABI_MAJOR_VERSION, + RMI_ABI_MINOR_VERSION); + unsigned long aa64pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1); + struct arm_smccc_1_2_regs res = { + SMC_RMI_VERSION, host_version, + }; + + /* If RME isn't supported, then RMI can't be */ + if (cpuid_feature_extract_unsigned_field(aa64pfr0, ID_AA64PFR0_EL1_RME_SHIFT) == 0) + return -ENXIO; + + if (arm_smccc_1_1_get_conduit() != SMCCC_CONDUIT_SMC) + return -ENXIO; + + rmi_smccc_invoke(&res); + if (res.a0 == SMCCC_RET_NOT_SUPPORTED) + return -ENXIO; + + version_major = RMI_ABI_VERSION_GET_MAJOR(res.a1); + version_minor = RMI_ABI_VERSION_GET_MINOR(res.a1); + + if (res.a0 != RMI_SUCCESS) { + unsigned short high_version_major, high_version_minor; + + high_version_major = RMI_ABI_VERSION_GET_MAJOR(res.a2); + high_version_minor = RMI_ABI_VERSION_GET_MINOR(res.a2); + + pr_err("Unsupported RMI ABI (v%d.%d - v%d.%d) we want v%d.%d\n", + version_major, version_minor, + high_version_major, high_version_minor, + RMI_ABI_MAJOR_VERSION, + RMI_ABI_MINOR_VERSION); + return -ENXIO; + } + + pr_info("RMI ABI version %d.%d\n", version_major, version_minor); + + return 0; +} + +static int rmi_read_features(void) +{ + /* + * Since we've negotiated a compatible version these feature registers + * should always be available + */ + for (int i = 0; i < ARRAY_SIZE(rmi_feat_reg_cache); i++) { + struct arm_smccc_1_2_regs args = { + SMC_RMI_FEATURES, i, + }; + + rmi_smccc_invoke(&args); + if (WARN_ON(args.a0 != RMI_SUCCESS)) + return -EINVAL; + + rmi_feat_reg_cache[i] = args.a1; + } + + return 0; +} + +unsigned long rmi_feat_reg(unsigned int index) +{ + if (WARN_ON(index >= ARRAY_SIZE(rmi_feat_reg_cache))) + return 0; + + return rmi_feat_reg_cache[index]; +} +EXPORT_SYMBOL_GPL(rmi_feat_reg); + +static int rmi_rmm_config_set(unsigned long cfg_ptr) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RMM_CONFIG_SET, cfg_ptr, + }; + + rmi_smccc_invoke(®s); + + return regs.a0; +} + +static int rmi_configure(void) +{ + unsigned long granule_feature; + unsigned long granule_size; + int ret; + + switch (PAGE_SIZE) { + case SZ_4K: + granule_size = RMI_GRANULE_SIZE_4KB; + granule_feature = RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_4KB; + break; + case SZ_16K: + granule_size = RMI_GRANULE_SIZE_16KB; + granule_feature = RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_16KB; + break; + case SZ_64K: + granule_size = RMI_GRANULE_SIZE_64KB; + granule_feature = RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_64KB; + break; + default: + BUILD_BUG(); + } + + if (!(rmi_feat_reg(1) & granule_feature)) { + pr_err("RMM does not support %luKB granules\n", + PAGE_SIZE >> 10); + return -ENXIO; + } + + struct rmm_config *config __free(free_page) = + (struct rmm_config *)get_zeroed_page(GFP_KERNEL); + + if (!config) { + pr_err("Unable to allocate memory for RMM config\n"); + return -ENOMEM; + } + + config->rmi_granule_size = granule_size; + + /* + * For now we set the tracking_region_size to 0 which is the only option + * for 4KB PAGE_SIZE (1GB for 4KB PAGE_SIZE, 32MB/512MB for 16KB/64KB). + * TODO: Support other tracking sizes via Kconfig option for other + * PAGE_SIZES + */ + config->tracking_region_size = 0; + + ret = rmi_rmm_config_set(virt_to_phys(config)); + if (ret) { + pr_err("RMM config set failed (%d)\n", ret); + return -EINVAL; + } + + return 0; +} + +/** + * rmi_granule_tracking_get() - Get configuration of a Granule tracking region + * @start: Base PA of the tracking region + * @end: End of the PA region + * @out_category: Memory category + * @out_state: Tracking region state + * @out_top: Top of the memory region + * + * Return: RMI return code + */ +static int rmi_granule_tracking_get(unsigned long start, + unsigned long end, + unsigned long *out_category, + unsigned long *out_state, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_GRANULE_TRACKING_GET, start, end, + }; + + rmi_smccc_invoke(®s); + + if (regs.a0 != RMI_SUCCESS) + return regs.a0; + + if (out_category) + *out_category = regs.a1; + if (out_state) + *out_state = regs.a2; + if (out_top) + *out_top = regs.a3; + + return RMI_SUCCESS; +} + +/* + * Make sure the area is tracked by RMM at FINE granularity. + * We do not support changing the tracking yet. + */ +static int rmi_verify_memory_tracking(phys_addr_t start, phys_addr_t end) +{ + while (start < end) { + unsigned long ret, category, state, next; + + ret = rmi_granule_tracking_get(start, end, &category, &state, &next); + if (ret != RMI_SUCCESS) { + pr_err("RMI_GRANULE_TRACKING_GET failed for %llx-%llx: (%lu)\n", + start, end, ret); + return -ENXIO; + } + + if (WARN_ON(next <= start)) + return -ENXIO; + + if (state != RMI_TRACKING_FINE || + category != RMI_MEM_CATEGORY_CONVENTIONAL) { + /* TODO: Set granule tracking in this case */ + pr_err("Granule tracking for region isn't fine/conventional: %llx-%lx\n", + start, next); + return -ENODEV; + } + start = next; + } + + return 0; +} + +/* + * rmi_gpt_info - Query the GPT info for the given PAR. + * @start: Base of the physical address region + * @end: Top of the physical address region + * @out_top: Top of the physical address region for which the GPT @out_gpt_par_state + * is valid + * @out_gpt_par_state: State of the GPT covered by [start, out_top) + */ +static long rmi_gpt_info(unsigned long start, unsigned long end, + unsigned long *out_top, + unsigned long *out_gpt_par_state) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_GPT_INFO, start, end, + }; + + rmi_smccc_invoke(®s); + if (regs.a0 != RMI_SUCCESS) + return regs.a0; + + if (out_top) + *out_top = regs.a1; + if (out_gpt_par_state) + *out_gpt_par_state = regs.a2; + + return RMI_SUCCESS; +} + +/* + * We do not support creating L1 GPTs yet. So, make sure that all the regions + * are managed by the firmware. + */ +static int rmi_verify_gpt_firmware_managed(phys_addr_t start, phys_addr_t end) +{ + unsigned long l0gpt_sz; + unsigned long next, par_state; + + l0gpt_sz = 1UL << (30 + FIELD_GET(RMI_FEATURE_REGISTER_1_L0GPTSZ, + rmi_feat_reg(1))); + start = ALIGN_DOWN(start, l0gpt_sz); + end = ALIGN(end, l0gpt_sz); + + while (start < end) { + long ret = rmi_gpt_info(start, end, &next, &par_state); + + if (ret != RMI_SUCCESS) { + pr_err("RMI_GPT_INFO failed for %llx-%llx: (%ld)\n", + start, end, ret); + return -ENXIO; + } + + if (WARN_ON(next <= start)) + return -ENXIO; + + if (par_state != RMI_GPT_PAR_PLAT) { + pr_err("GPT for the region is not managed by firmware %llx-%lx\n", + start, next); + return -ENODEV; + } + start = next; + } + + return 0; +} + +static int rmi_prepare_memory(phys_addr_t start, phys_addr_t end) +{ + int ret; + + if (start >= end) + return -EINVAL; + + ret = rmi_verify_memory_tracking(start, end); + if (ret) + return ret; + + return rmi_verify_gpt_firmware_managed(start, end); +} + +static int rmi_init_metadata(void) +{ + phys_addr_t start, end; + struct memblock_region *r; + + for_each_mem_region(r) { + int ret; + + /* Firmware-reserved NOMAP regions are not usable system RAM */ + if (memblock_is_nomap(r)) + continue; + + start = PAGE_ALIGN(r->base); + /* + * We always deal with PAGE_SIZE and if in the odd case this + * region boundary is not PAGE aligned, we stick to the page + * that we can use from the region. + */ + end = PAGE_ALIGN_DOWN(r->base + r->size); + /* Too small ? */ + if (start >= end) + continue; + + ret = rmi_prepare_memory(start, end); + if (ret) + return ret; + } + + return 0; +} + +static int rmi_memory_notifier(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct memory_notify *arg = data; + phys_addr_t start, end; + int ret; + + if (action != MEM_GOING_ONLINE) + return NOTIFY_DONE; + + start = PFN_PHYS(arg->start_pfn); + end = PFN_PHYS(arg->start_pfn + arg->nr_pages); + + /* + * If the hotplugged memory region is in ZONE_MOVABLE, neither the + * guest-memfd nor the kernel allocations can come from there. So we are + * fine to skip this. + * For boot time memory, we don't allow this discount, as the + * decision to move boot memory pages to ZONE_MOVABLE is purely + * a kernel choice and the firmware must have decided what it covers. + */ + if (is_zone_movable_page(pfn_to_page(arg->start_pfn))) + return NOTIFY_DONE; + + ret = rmi_prepare_memory(start, end); + + return notifier_from_errno(ret); +} + +static struct notifier_block rmi_memory_nb = { + .notifier_call = rmi_memory_notifier, +}; + +static int rmi_init_memory(void) +{ + int ret; + + ret = rmi_init_metadata(); + if (ret) + return ret; + + return register_memory_notifier(&rmi_memory_nb); +} + +static int __init arm64_init_rmi(void) +{ + int ret; + + /* If we can't agree on the RMI ABI version, don't proceed further */ + ret = rmi_check_version(); + if (ret) + return ret; + + ret = rmi_read_features(); + if (ret) + return ret; + + ret = rmi_configure(); + if (ret) + return ret; + + /* Activate the RMM */ + struct rmi_sro_state *sro __free(kfree) = kmalloc_obj(*sro); + if (!sro) + return -ENOMEM; + + ret = rmi_sro_memxfer_cmd(sro, GFP_KERNEL, SMC_RMI_RMM_ACTIVATE); + if (ret) { + pr_err("RMM activate failed (%d)\n", ret); + ret = ret < 0 ? ret : -ENXIO; + return ret; + } + + WRITE_ONCE(arm64_rmm_active, true); + + ret = rmi_init_memory(); + if (ret) { + /* Deactivate the RMM */ + if (!WARN_ON(rmi_sro_memxfer_cmd(sro, GFP_KERNEL, SMC_RMI_RMM_DEACTIVATE))) + WRITE_ONCE(arm64_rmm_active, false); + return ret; + } + + /* Mark the RMM is usable */ + WRITE_ONCE(arm64_rmi_is_available, true); + pr_info("RMI configured\n"); + + return 0; +} + +/* + * Note arm64_init_rmi() must be called before kvm_init_rmi() otherwise KVM + * will not support realm guests. subsys_initcall() is called before + * module_init() (used for KVM) so this is OK. + */ +subsys_initcall(arm64_init_rmi); diff --git a/arch/arm64/kernel/rsi.c b/drivers/firmware/arm_rmm/rsi.c index 25ca75ce1a4d..0fcc4e33bd92 100644 --- a/arch/arm64/kernel/rsi.c +++ b/drivers/firmware/arm_rmm/rsi.c @@ -7,12 +7,13 @@ #include <linux/memblock.h> #include <linux/psci.h> #include <linux/swiotlb.h> -#include <linux/platform_device.h> +#include <linux/arm-rsi-cmds.h> +#include <linux/kobject.h> +#include <linux/sysfs.h> #include <asm/io.h> #include <asm/mem_encrypt.h> #include <asm/pgtable.h> -#include <asm/rsi.h> static struct realm_config config; @@ -127,6 +128,40 @@ static int realm_ioremap_hook(phys_addr_t phys, size_t size, pgprot_t *prot) return 0; } +static int realm_set_memory_encrypted(unsigned long addr, int numpages) +{ + int ret = __set_memory_enc_dec(addr, numpages, true); + + /* + * If the request to change state fails, then the only sensible cause + * of action for the caller is to leak the memory + */ + WARN(ret, "Failed to encrypt memory, %d pages will be leaked", + numpages); + + return ret; +} + +static int realm_set_memory_decrypted(unsigned long addr, int numpages) +{ + int ret = __set_memory_enc_dec(addr, numpages, false); + + WARN(ret, "Failed to decrypt memory, %d pages will be leaked", + numpages); + + return ret; +} + +static const struct arm64_mem_crypt_ops realm_crypt_ops = { + .encrypt = realm_set_memory_encrypted, + .decrypt = realm_set_memory_decrypted, +}; + +static int realm_register_memory_enc_ops(void) +{ + return arm64_mem_crypt_ops_register(&realm_crypt_ops); +} + void __init arm64_rsi_init(void) { if (arm_smccc_1_1_get_conduit() != SMCCC_CONDUIT_SMC) @@ -148,17 +183,35 @@ void __init arm64_rsi_init(void) static_branch_enable(&rsi_present); } -static struct platform_device rsi_dev = { - .name = RSI_PDEV_NAME, - .id = PLATFORM_DEVID_NONE +static ssize_t realm_guest_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "1\n"); +} + +static struct kobj_attribute cca_realm_guest = __ATTR_RO(realm_guest); +static const struct attribute *cca_realm_attrs[] = { + &cca_realm_guest.attr, + NULL }; -static int __init arm64_create_dummy_rsi_dev(void) +static int __init realm_sysfs_init(void) { - if (is_realm_world() && - platform_device_register(&rsi_dev)) - pr_err("failed to register rsi platform device\n"); - return 0; -} + int ret; + struct kobject *cca_kobj; + + if (!is_realm_world()) + return 0; -arch_initcall(arm64_create_dummy_rsi_dev) + cca_kobj = kobject_create_and_add("cca", firmware_kobj); + if (!cca_kobj) + return -ENOMEM; + + ret = sysfs_create_files(cca_kobj, cca_realm_attrs); + if (!ret) + return 0; + + kobject_put(cca_kobj); + return ret; +} +device_initcall(realm_sysfs_init); diff --git a/drivers/firmware/smccc/Makefile b/drivers/firmware/smccc/Makefile index 40d19144a860..68bbff1407b8 100644 --- a/drivers/firmware/smccc/Makefile +++ b/drivers/firmware/smccc/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 # -obj-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smccc.o kvm_guest.o +obj-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += bus.o smccc.o kvm_guest.o obj-$(CONFIG_ARM_SMCCC_SOC_ID) += soc_id.o diff --git a/drivers/firmware/smccc/bus.c b/drivers/firmware/smccc/bus.c new file mode 100644 index 000000000000..0357a7c48af0 --- /dev/null +++ b/drivers/firmware/smccc/bus.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026 Arm Limited + */ + +#include <linux/arm-smccc-bus.h> +#include <linux/slab.h> + +static int arm_smccc_bus_match(struct device *dev, + const struct device_driver *drv) +{ + const struct arm_smccc_device_id *id_table; + const struct arm_smccc_device *smccc_dev = to_arm_smccc_device(dev); + + id_table = to_arm_smccc_driver(drv)->id_table; + if (!id_table) + return 0; + + while (id_table->func_id) { + if (smccc_dev->func_id == id_table->func_id) + return 1; + id_table++; + } + + return 0; +} + +static int arm_smccc_bus_probe(struct device *dev) +{ + struct arm_smccc_driver *smccc_drv = to_arm_smccc_driver(dev->driver); + + return smccc_drv->probe(to_arm_smccc_device(dev)); +} + +static void arm_smccc_bus_remove(struct device *dev) +{ + struct arm_smccc_driver *smcc_drv = to_arm_smccc_driver(dev->driver); + + if (smcc_drv->remove) + smcc_drv->remove(to_arm_smccc_device(dev)); +} + +static int arm_smccc_bus_uevent(const struct device *dev, + struct kobj_uevent_env *env) +{ + const struct arm_smccc_device *smccc_dev = to_arm_smccc_device(dev); + + return add_uevent_var(env, "MODALIAS=" ARM_SMCCC_MODULE_PREFIX "f%08X", + smccc_dev->func_id); +} + +static ssize_t modalias_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct arm_smccc_device *smccc_dev = to_arm_smccc_device(dev); + + return sysfs_emit(buf, ARM_SMCCC_MODULE_PREFIX "f%08X\n", + smccc_dev->func_id); +} +static DEVICE_ATTR_RO(modalias); + +static struct attribute *arm_smccc_device_attrs[] = { + &dev_attr_modalias.attr, + NULL, +}; +ATTRIBUTE_GROUPS(arm_smccc_device); + +const struct bus_type arm_smccc_bus_type = { + .name = "arm_smccc", + .match = arm_smccc_bus_match, + .probe = arm_smccc_bus_probe, + .remove = arm_smccc_bus_remove, + .uevent = arm_smccc_bus_uevent, + .dev_groups = arm_smccc_device_groups, +}; +EXPORT_SYMBOL_GPL(arm_smccc_bus_type); + +int arm_smccc_driver_register(struct arm_smccc_driver *driver, + struct module *owner, const char *mod_name) +{ + if (!driver->probe || !driver->id_table) + return -EINVAL; + + driver->driver.bus = &arm_smccc_bus_type; + driver->driver.name = driver->name; + driver->driver.owner = owner; + driver->driver.mod_name = mod_name; + + return driver_register(&driver->driver); +} +EXPORT_SYMBOL_GPL(arm_smccc_driver_register); + +void arm_smccc_driver_unregister(struct arm_smccc_driver *driver) +{ + driver_unregister(&driver->driver); +} +EXPORT_SYMBOL_GPL(arm_smccc_driver_unregister); + +static void arm_smccc_release_device(struct device *dev) +{ + struct arm_smccc_device *smccc_dev = to_arm_smccc_device(dev); + + kfree(smccc_dev); +} + +static struct device *arm_smccc_device_alloc(u32 func_id) +{ + struct arm_smccc_device *smccc_dev; + + smccc_dev = kzalloc_obj(*smccc_dev); + if (!smccc_dev) + return NULL; + + smccc_dev->func_id = func_id; + smccc_dev->dev.bus = &arm_smccc_bus_type; + smccc_dev->dev.release = arm_smccc_release_device; + device_initialize(&smccc_dev->dev); + + return &smccc_dev->dev; +} + +struct arm_smccc_device *arm_smccc_device_register(const char *name, u32 func_id) +{ + int ret; + + if (!name) + return ERR_PTR(-EINVAL); + + struct device *dev __free(put_device) = arm_smccc_device_alloc(func_id); + if (!dev) + return ERR_PTR(-ENOMEM); + + ret = dev_set_name(dev, "%s", name); + if (ret) + return ERR_PTR(ret); + + ret = device_add(dev); + if (ret) + return ERR_PTR(ret); + + return to_arm_smccc_device(no_free_ptr(dev)); +} +EXPORT_SYMBOL_GPL(arm_smccc_device_register); + +void arm_smccc_device_unregister(struct arm_smccc_device *smccc_dev) +{ + if (!smccc_dev) + return; + + device_unregister(&smccc_dev->dev); +} +EXPORT_SYMBOL_GPL(arm_smccc_device_unregister); + +static int __init arm_smccc_bus_init(void) +{ + return bus_register(&arm_smccc_bus_type); +} +subsys_initcall(arm_smccc_bus_init); diff --git a/drivers/firmware/smccc/smccc.c b/drivers/firmware/smccc/smccc.c index bdee057db2fd..5dc768ac4143 100644 --- a/drivers/firmware/smccc/smccc.c +++ b/drivers/firmware/smccc/smccc.c @@ -9,7 +9,9 @@ #include <linux/init.h> #include <linux/arm-smccc.h> #include <linux/kernel.h> -#include <linux/platform_device.h> +#include <linux/arm-smccc-bus.h> +#include <linux/arm-smccc-rsi.h> + #include <asm/archrandom.h> static u32 smccc_version = ARM_SMCCC_VERSION_1_0; @@ -81,16 +83,63 @@ bool arm_smccc_hypervisor_has_uuid(const uuid_t *hyp_uuid) } EXPORT_SYMBOL_GPL(arm_smccc_hypervisor_has_uuid); +struct smccc_device_info { + u32 func_id; + bool requires_smc; + const char *device_name; +}; + +static const struct smccc_device_info smccc_devices[] __initconst = { + { + .func_id = ARM_SMCCC_TRNG_VERSION, + .requires_smc = false, + .device_name = "arm-smccc-trng", + }, + { + .func_id = SMC_RSI_ABI_VERSION, + .requires_smc = true, + .device_name = "arm-rsi", + }, +}; + +static bool __init smccc_probe_smccc_device(const struct smccc_device_info *smccc_dev) +{ + int ret; + struct arm_smccc_res res = {}; + + if (smccc_conduit == SMCCC_CONDUIT_NONE) + return false; + + if (smccc_dev->requires_smc && smccc_conduit != SMCCC_CONDUIT_SMC) + return false; + + if (IS_ENABLED(CONFIG_ARM) && ARM_SMCCC_IS_64(smccc_dev->func_id)) + return false; + + arm_smccc_1_1_invoke(smccc_dev->func_id, 0, &res); + ret = res.a0; + + if (ret == SMCCC_RET_NOT_SUPPORTED) + return false; + + return true; +} + static int __init smccc_devices_init(void) { - struct platform_device *pdev; - - if (smccc_trng_available) { - pdev = platform_device_register_simple("smccc_trng", -1, - NULL, 0); - if (IS_ERR(pdev)) - pr_err("smccc_trng: could not register device: %ld\n", - PTR_ERR(pdev)); + struct arm_smccc_device *sdev; + const struct smccc_device_info *smccc_dev; + + for (int i = 0; i < ARRAY_SIZE(smccc_devices); i++) { + smccc_dev = &smccc_devices[i]; + + if (!smccc_probe_smccc_device(smccc_dev)) + continue; + + sdev = arm_smccc_device_register(smccc_dev->device_name, smccc_dev->func_id); + if (IS_ERR(sdev)) + pr_err("%s: could not register device: %pe\n", + smccc_dev->device_name, sdev); } return 0; diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig index aa11bcefddf2..36445c2619ed 100644 --- a/drivers/hv/Kconfig +++ b/drivers/hv/Kconfig @@ -4,8 +4,7 @@ menu "Microsoft Hyper-V guest support" config HYPERV bool "Microsoft Hyper-V core hypervisor support" - depends on (X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) \ - || (ARM64 && !CPU_BIG_ENDIAN) + depends on (X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) || ARM64 select PARAVIRT select X86_HV_CALLBACK_VECTOR if X86 select OF_EARLY_FLATTREE if OF diff --git a/drivers/hwtracing/coresight/coresight-etm-perf.c b/drivers/hwtracing/coresight/coresight-etm-perf.c index 09b21a711a87..80502e8b0840 100644 --- a/drivers/hwtracing/coresight/coresight-etm-perf.c +++ b/drivers/hwtracing/coresight/coresight-etm-perf.c @@ -613,6 +613,8 @@ static void etm_event_start(struct perf_event *event, int flags) perf_report_aux_output_id(event, hw_id); } + perf_event_itrace_started(event); + out: /* Tell the perf core the event is alive */ event->hw.state = 0; diff --git a/drivers/misc/vmw_vmci/Kconfig b/drivers/misc/vmw_vmci/Kconfig index b6d4d7fd686a..ca03cbfdab1e 100644 --- a/drivers/misc/vmw_vmci/Kconfig +++ b/drivers/misc/vmw_vmci/Kconfig @@ -5,7 +5,7 @@ config VMWARE_VMCI tristate "VMware VMCI Driver" - depends on (X86 || ARM64) && !CPU_BIG_ENDIAN && PCI + depends on (X86 || ARM64) && PCI help This is VMware's Virtual Machine Communication Interface. It enables high-speed communication between host and guest in a virtual diff --git a/drivers/net/ethernet/google/Kconfig b/drivers/net/ethernet/google/Kconfig index 14c9431e15e5..a68327f47acb 100644 --- a/drivers/net/ethernet/google/Kconfig +++ b/drivers/net/ethernet/google/Kconfig @@ -17,7 +17,7 @@ if NET_VENDOR_GOOGLE config GVE tristate "Google Virtual NIC (gVNIC) support" - depends on (PCI_MSI && (X86 || CPU_LITTLE_ENDIAN)) + depends on (PCI_MSI && (X86 || ARM64 || CPU_LITTLE_ENDIAN)) depends on PTP_1588_CLOCK_OPTIONAL select PAGE_POOL help diff --git a/drivers/net/ethernet/microsoft/Kconfig b/drivers/net/ethernet/microsoft/Kconfig index e9be18c92ca5..c4ad3d85875a 100644 --- a/drivers/net/ethernet/microsoft/Kconfig +++ b/drivers/net/ethernet/microsoft/Kconfig @@ -18,7 +18,7 @@ if NET_VENDOR_MICROSOFT config MICROSOFT_MANA tristate "Microsoft Azure Network Adapter (MANA) support" depends on PCI_MSI - depends on X86_64 || (ARM64 && !CPU_BIG_ENDIAN) + depends on X86_64 || ARM64 depends on PCI_HYPERV select AUXILIARY_BUS select DIMLIB diff --git a/drivers/perf/apple_m1_cpu_pmu.c b/drivers/perf/apple_m1_cpu_pmu.c index 81b6f1a62349..1f73c8a77462 100644 --- a/drivers/perf/apple_m1_cpu_pmu.c +++ b/drivers/perf/apple_m1_cpu_pmu.c @@ -81,7 +81,7 @@ enum m1_pmu_events { M1_PMU_PERFCTR_INST_SIMD_ALU = 0x9a, M1_PMU_PERFCTR_INST_LDST = 0x9b, M1_PMU_PERFCTR_INST_BARRIER = 0x9c, - M1_PMU_PERFCTR_UNKNOWN_9f = 0x9f, + M1_PMU_PERFCTR_INST_SIMD_ALU_VEC = 0x9f, M1_PMU_PERFCTR_L1D_TLB_ACCESS = 0xa0, M1_PMU_PERFCTR_L1D_TLB_MISS = 0xa1, M1_PMU_PERFCTR_L1D_CACHE_MISS_ST = 0xa2, @@ -150,7 +150,7 @@ static const u16 m1_pmu_event_affinity[M1_PMU_PERFCTR_LAST + 1] = { [M1_PMU_PERFCTR_INST_SIMD_ALU] = BIT(7), [M1_PMU_PERFCTR_INST_LDST] = BIT(7), [M1_PMU_PERFCTR_INST_BARRIER] = ONLY_5_6_7, - [M1_PMU_PERFCTR_UNKNOWN_9f] = BIT(7), + [M1_PMU_PERFCTR_INST_SIMD_ALU_VEC] = BIT(7), [M1_PMU_PERFCTR_L1D_CACHE_MISS_LD_NONSPEC] = ONLY_5_6_7, [M1_PMU_PERFCTR_L1D_CACHE_MISS_ST_NONSPEC] = ONLY_5_6_7, [M1_PMU_PERFCTR_L1D_TLB_MISS_NONSPEC] = ONLY_5_6_7, diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index 254be4da8ae2..073d14584741 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -166,21 +166,11 @@ static bool brbe_record_is_target_only(u64 brbinf) return brbe_record_valid(brbinf) == BRBINFx_EL1_VALID_TARGET; } -static int brbinf_get_in_tx(u64 brbinf) -{ - return FIELD_GET(BRBINFx_EL1_T_MASK, brbinf); -} - static int brbinf_get_mispredict(u64 brbinf) { return FIELD_GET(BRBINFx_EL1_MPRED_MASK, brbinf); } -static int brbinf_get_lastfailed(u64 brbinf) -{ - return FIELD_GET(BRBINFx_EL1_LASTFAILED_MASK, brbinf); -} - static u16 brbinf_get_cycles(u64 brbinf) { u32 exp, mant, cycles; @@ -627,14 +617,6 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent entry->mispred = brbinf_get_mispredict(brbinf); entry->predicted = !entry->mispred; } - - /* - * Currently TME feature is neither implemented in any hardware - * nor it is being supported in the kernel. Just warn here once - * if TME related information shows up rather unexpectedly. - */ - if (brbinf_get_lastfailed(brbinf) || brbinf_get_in_tx(brbinf)) - pr_warn_once("Unknown transaction states\n"); } /* diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c index b70f3fedf028..40314ef8b731 100644 --- a/drivers/perf/arm_spe_pmu.c +++ b/drivers/perf/arm_spe_pmu.c @@ -890,6 +890,8 @@ static void arm_spe_pmu_start(struct perf_event *event, int flags) return; } + perf_event_itrace_started(event); + reg = arm_spe_event_to_pmsfcr(event); write_sysreg_s(reg, SYS_PMSFCR_EL1); diff --git a/drivers/perf/cxl_pmu.c b/drivers/perf/cxl_pmu.c index 2c8a03cc11e4..7e4113717e01 100644 --- a/drivers/perf/cxl_pmu.c +++ b/drivers/perf/cxl_pmu.c @@ -111,7 +111,10 @@ struct cxl_pmu_info { struct hlist_node node; bool filter_hdm; bool filter_crb; + int msi_vec; int irq; + /* Set between pmu_enable() and pmu_disable(), read by the IRQ handler */ + bool enabled; }; #define pmu_to_cxl_pmu_info(_pmu) container_of(_pmu, struct cxl_pmu_info, pmu) @@ -144,15 +147,24 @@ static int cxl_pmu_parse_caps(struct device *dev, struct cxl_pmu_info *info) info->num_counters = FIELD_GET(CXL_PMU_CAP_NUM_COUNTERS_MSK, val) + 1; info->counter_width = FIELD_GET(CXL_PMU_CAP_COUNTER_WIDTH_MSK, val); + /* + * The Counter Data register is 64 bits wide, so a Counter Width of 0 or + * >64 is invalid. Reject it rather than let GENMASK_ULL(width - 1, 0) in + * the read path shift out of range. + */ + if (info->counter_width == 0 || info->counter_width > 64) { + dev_err(dev, "Invalid counter width %d\n", info->counter_width); + return -ENODEV; + } info->num_event_capabilities = FIELD_GET(CXL_PMU_CAP_NUM_EVN_CAP_REG_SUP_MSK, val) + 1; info->filter_hdm = FIELD_GET(CXL_PMU_CAP_FILTERS_SUP_MSK, val) & CXL_PMU_FILTER_HDM; info->filter_crb = FIELD_GET(CXL_PMU_CAP_FILTERS_SUP_MSK, val) & CXL_PMU_FILTER_CHAN_RANK_BANK; if (FIELD_GET(CXL_PMU_CAP_INT, val)) - info->irq = FIELD_GET(CXL_PMU_CAP_MSI_N_MSK, val); + info->msi_vec = FIELD_GET(CXL_PMU_CAP_MSI_N_MSK, val); else - info->irq = -1; + info->msi_vec = -1; /* First handle fixed function counters; note if configurable counters found */ for (i = 0; i < info->num_counters; i++) { @@ -588,7 +600,10 @@ static const struct attribute_group *cxl_pmu_attr_groups[] = { NULL }; -/* If counter_idx == NULL, don't try to allocate a counter. */ +/* + * If counter_idx == NULL, don't try to allocate a counter. Callers that + * allocate pass both counter_idx and event_idx. + */ static int cxl_pmu_get_event_idx(struct perf_event *event, int *counter_idx, int *event_idx) { @@ -616,7 +631,7 @@ static int cxl_pmu_get_event_idx(struct perf_event *event, int *counter_idx, pmu_ev = cxl_pmu_find_config_counter_ev_cap(info, vid, gid, mask); if (!IS_ERR(pmu_ev)) { - if (!counter_idx) + if (!counter_idx || !event_idx) return 0; bitmap_andnot(configurable_and_free, info->conf_counter_bm, @@ -627,6 +642,7 @@ static int cxl_pmu_get_event_idx(struct perf_event *event, int *counter_idx, return -EINVAL; *counter_idx = i; + *event_idx = pmu_ev->event_idx; return 0; } @@ -696,6 +712,7 @@ static void cxl_pmu_enable(struct pmu *pmu) void __iomem *base = info->base; /* Can assume frozen at this stage */ + WRITE_ONCE(info->enabled, true); writeq(0, base + CXL_PMU_FREEZE_REG); } @@ -704,6 +721,7 @@ static void cxl_pmu_disable(struct pmu *pmu) struct cxl_pmu_info *info = pmu_to_cxl_pmu_info(pmu); void __iomem *base = info->base; + WRITE_ONCE(info->enabled, false); /* * Whilst bits above number of counters are RsvdZ * they are unlikely to be repurposed given @@ -755,17 +773,17 @@ static void cxl_pmu_event_start(struct perf_event *event, int flags) cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_INT_ON_OVRFLW, 1); cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_FREEZE_ON_OVRFLW, 1); cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_ENABLE, 1); - cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_EDGE, - cxl_pmu_config1_get_edge(event) ? 1 : 0); - cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_INVERT, - cxl_pmu_config1_get_invert(event) ? 1 : 0); + FIELD_MODIFY(CXL_PMU_COUNTER_CFG_EDGE, &cfg, + cxl_pmu_config1_get_edge(event) ? 1 : 0); + FIELD_MODIFY(CXL_PMU_COUNTER_CFG_INVERT, &cfg, + cxl_pmu_config1_get_invert(event) ? 1 : 0); /* Fixed purpose counters have next two fields RO */ if (test_bit(hwc->idx, info->conf_counter_bm)) { - cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_EVENT_GRP_ID_IDX_MSK, - hwc->event_base); - cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_EVENTS_MSK, - cxl_pmu_config_get_mask(event)); + FIELD_MODIFY(CXL_PMU_COUNTER_CFG_EVENT_GRP_ID_IDX_MSK, &cfg, + hwc->event_base); + FIELD_MODIFY(CXL_PMU_COUNTER_CFG_EVENTS_MSK, &cfg, + cxl_pmu_config_get_mask(event)); } cfg &= ~CXL_PMU_COUNTER_CFG_THRESHOLD_MSK; /* @@ -778,6 +796,15 @@ static void cxl_pmu_event_start(struct perf_event *event, int flags) */ cfg |= FIELD_PREP(CXL_PMU_COUNTER_CFG_THRESHOLD_MSK, cxl_pmu_config1_get_threshold(event)); + + /* + * Drop any overflow the previous owner of this counter left pending; + * cxl_pmu_event_stop() does not. Otherwise an interrupt latched over + * the handover gets charged to this event. Do it before arming Interrupt + * on Overflow below, and note RW1C leaves the other counters alone. + */ + writeq(BIT_ULL(hwc->idx), base + CXL_PMU_OVERFLOW_REG); + writeq(cfg, base + CXL_PMU_COUNTER_CFG_REG(hwc->idx)); local64_set(&hwc->prev_count, 0); @@ -798,7 +825,7 @@ static void __cxl_pmu_read(struct perf_event *event, bool overflow) { struct cxl_pmu_info *info = pmu_to_cxl_pmu_info(event->pmu); struct hw_perf_event *hwc = &event->hw; - u64 new_cnt, prev_cnt, delta; + u64 new_cnt, prev_cnt, delta, mask; do { prev_cnt = local64_read(&hwc->prev_count); @@ -806,12 +833,16 @@ static void __cxl_pmu_read(struct perf_event *event, bool overflow) } while (local64_cmpxchg(&hwc->prev_count, prev_cnt, new_cnt) != prev_cnt); /* - * If we know an overflow occur then take that into account. - * Note counter is not reset as that would lose events + * The mask discards the bit that says the counter wrapped, so a delta of + * one whole period reads back as 0 - the same as no events at all. Only + * the overflow status separates them, and new_cnt >= prev_cnt is that + * case, so add the period back. mask + 1 is 2^counter_width, which comes + * out as 0 for a 64-bit counter and avoids an undefined 1 << 64. */ - delta = (new_cnt - prev_cnt) & GENMASK_ULL(info->counter_width - 1, 0); - if (overflow && delta < GENMASK_ULL(info->counter_width - 1, 0)) - delta += (1UL << info->counter_width); + mask = GENMASK_ULL(info->counter_width - 1, 0); + delta = (new_cnt - prev_cnt) & mask; + if (overflow && new_cnt >= prev_cnt) + delta += mask + 1; local64_add(delta, &event->count); } @@ -887,7 +918,7 @@ static irqreturn_t cxl_pmu_irq(int irq, void *data) overflowed = readq(base + CXL_PMU_OVERFLOW_REG); - /* Interrupt may be shared, so maybe it isn't ours */ + /* Nothing overflowed, so the device did not raise this */ if (!overflowed) return IRQ_NONE; @@ -896,7 +927,7 @@ static irqreturn_t cxl_pmu_irq(int irq, void *data) struct perf_event *event = info->hw_events[i]; if (!event) { - dev_dbg(info->pmu.dev, + dev_dbg(info->pmu.parent, "overflow but on non enabled counter %d\n", i); continue; } @@ -906,6 +937,21 @@ static irqreturn_t cxl_pmu_irq(int irq, void *data) writeq(overflowed, base + CXL_PMU_OVERFLOW_REG); + /* + * An overflow freezes every counter in the CPMU, so unfreeze once the + * overflowed ones have been read and their status cleared. Otherwise + * they stay frozen until the next pmu_enable() and events are lost. + * + * Not while the PMU is disabled, so as not to undo an intentional freeze. + * The check is advisory, not exclusive: pmu_disable() normally runs on + * info->on_cpu with interrupts off, where the pinned handler cannot + * preempt it. In the one window where it does not - the migration in + * cxl_pmu_offline_cpu() - the counters are legitimately running again, + * so unfreezing is correct there anyway. + */ + if (READ_ONCE(info->enabled)) + writeq(0, base + CXL_PMU_FREEZE_REG); + return IRQ_HANDLED; } @@ -978,10 +1024,10 @@ static int cxl_pmu_probe(struct device *dev) .capabilities = PERF_PMU_CAP_NO_EXCLUDE, }; - if (info->irq <= 0) + if (info->msi_vec < 0) return -EINVAL; - rc = pci_irq_vector(pdev, info->irq); + rc = pci_irq_vector(pdev, info->msi_vec); if (rc < 0) return rc; irq = rc; @@ -990,7 +1036,22 @@ static int cxl_pmu_probe(struct device *dev) if (!irq_name) return -ENOMEM; - rc = devm_request_irq(dev, irq, cxl_pmu_irq, IRQF_SHARED | IRQF_NO_THREAD, + /* + * Same for counters no event owns yet: clear whatever firmware or a + * previous kernel left set before the handler goes live. Bits above the + * implemented counters are reserved, so only write those. + */ + writeq(GENMASK_ULL(info->num_counters - 1, 0), + info->base + CXL_PMU_OVERFLOW_REG); + + /* + * The handler must run on info->on_cpu, so the interrupt cannot be + * shared - IRQF_NOBALANCING is only honoured for the first action on a + * line, and a co-owner would keep taking the interrupt wherever its own + * affinity points. + */ + rc = devm_request_irq(dev, irq, cxl_pmu_irq, + IRQF_NO_THREAD | IRQF_NOBALANCING, irq_name, info); if (rc) return rc; diff --git a/drivers/perf/dwc_pcie_pmu.c b/drivers/perf/dwc_pcie_pmu.c index 6b9ed59de26e..eb86e7468c49 100644 --- a/drivers/perf/dwc_pcie_pmu.c +++ b/drivers/perf/dwc_pcie_pmu.c @@ -747,16 +747,16 @@ static struct notifier_block dwc_pcie_pmu_nb = { static int dwc_pcie_pmu_probe(struct platform_device *plat_dev) { - struct pci_dev *pdev; + u32 sbdf = plat_dev->id; + struct pci_dev *pdev __free(pci_dev_put) = + pci_get_domain_bus_and_slot(sbdf >> 16, + PCI_BUS_NUM(sbdf & 0xffff), + sbdf & 0xff); struct dwc_pcie_pmu *pcie_pmu; char *name; - u32 sbdf; u16 vsec; int ret; - sbdf = plat_dev->id; - pdev = pci_get_domain_bus_and_slot(sbdf >> 16, PCI_BUS_NUM(sbdf & 0xffff), - sbdf & 0xff); if (!pdev) { pr_err("No pdev found for the sbdf 0x%x\n", sbdf); return -ENODEV; @@ -766,7 +766,6 @@ static int dwc_pcie_pmu_probe(struct platform_device *plat_dev) if (!vsec) return -ENODEV; - pci_dev_put(pdev); name = devm_kasprintf(&plat_dev->dev, GFP_KERNEL, "dwc_rootport_%x", sbdf); if (!name) return -ENOMEM; diff --git a/drivers/perf/hisilicon/Makefile b/drivers/perf/hisilicon/Makefile index 186be3d02238..5f28cfdb8a72 100644 --- a/drivers/perf/hisilicon/Makefile +++ b/drivers/perf/hisilicon/Makefile @@ -2,7 +2,7 @@ obj-$(CONFIG_HISI_PMU) += hisi_uncore_pmu.o hisi_uncore_l3c_pmu.o \ hisi_uncore_hha_pmu.o hisi_uncore_ddrc_pmu.o hisi_uncore_sllc_pmu.o \ hisi_uncore_pa_pmu.o hisi_uncore_cpa_pmu.o hisi_uncore_uc_pmu.o \ - hisi_uncore_noc_pmu.o hisi_uncore_mn_pmu.o + hisi_uncore_noc_pmu.o hisi_uncore_mn_pmu.o hisi_uncore_its_pmu.o obj-$(CONFIG_HISI_PCIE_PMU) += hisi_pcie_pmu.o obj-$(CONFIG_HNS3_PMU) += hns3_pmu.o diff --git a/drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c b/drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c index b879b81adfdd..83967725c51c 100644 --- a/drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c @@ -295,7 +295,7 @@ static int hisi_cpa_pmu_probe(struct platform_device *pdev) /* Power Management should be disabled before using CPA PMU. */ hisi_cpa_pmu_disable_pm(cpa_pmu); - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE, + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &cpa_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); @@ -307,7 +307,7 @@ static int hisi_cpa_pmu_probe(struct platform_device *pdev) if (ret) { dev_err(cpa_pmu->dev, "PMU register failed\n"); cpuhp_state_remove_instance_nocalls( - CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE, &cpa_pmu->node); + hisi_uncore_pmu_cpuhp_state, &cpa_pmu->node); hisi_cpa_pmu_enable_pm(cpa_pmu); return ret; } @@ -321,7 +321,7 @@ static void hisi_cpa_pmu_remove(struct platform_device *pdev) struct hisi_pmu *cpa_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&cpa_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &cpa_pmu->node); hisi_cpa_pmu_enable_pm(cpa_pmu); } @@ -336,33 +336,7 @@ static struct platform_driver hisi_cpa_pmu_driver = { .remove = hisi_cpa_pmu_remove, }; -static int __init hisi_cpa_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE, - "AP_PERF_ARM_HISI_CPA_ONLINE", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret) { - pr_err("setup hotplug failed: %d\n", ret); - return ret; - } - - ret = platform_driver_register(&hisi_cpa_pmu_driver); - if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE); - - return ret; -} -module_init(hisi_cpa_pmu_module_init); - -static void __exit hisi_cpa_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_cpa_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE); -} -module_exit(hisi_cpa_pmu_module_exit); +module_platform_driver(hisi_cpa_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC CPA PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c b/drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c index 21c494881ca0..e905160523ee 100644 --- a/drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c @@ -401,7 +401,7 @@ static int hisi_ddrc_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &ddrc_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug;\n", ret); @@ -414,7 +414,7 @@ static int hisi_ddrc_pmu_probe(struct platform_device *pdev) if (ret) { dev_err(ddrc_pmu->dev, "DDRC PMU register failed!\n"); cpuhp_state_remove_instance_nocalls( - CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, &ddrc_pmu->node); + hisi_uncore_pmu_cpuhp_state, &ddrc_pmu->node); } return ret; @@ -425,7 +425,7 @@ static void hisi_ddrc_pmu_remove(struct platform_device *pdev) struct hisi_pmu *ddrc_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&ddrc_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &ddrc_pmu->node); } @@ -501,33 +501,7 @@ static struct platform_driver hisi_ddrc_pmu_driver = { .remove = hisi_ddrc_pmu_remove, }; -static int __init hisi_ddrc_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, - "AP_PERF_ARM_HISI_DDRC_ONLINE", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret) { - pr_err("DDRC PMU: setup hotplug, ret = %d\n", ret); - return ret; - } - - ret = platform_driver_register(&hisi_ddrc_pmu_driver); - if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE); - - return ret; -} -module_init(hisi_ddrc_pmu_module_init); - -static void __exit hisi_ddrc_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_ddrc_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE); -} -module_exit(hisi_ddrc_pmu_module_exit); +module_platform_driver(hisi_ddrc_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC DDRC uncore PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_hha_pmu.c b/drivers/perf/hisilicon/hisi_uncore_hha_pmu.c index 97cfaa586a87..93a250b7abb8 100644 --- a/drivers/perf/hisilicon/hisi_uncore_hha_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_hha_pmu.c @@ -490,7 +490,7 @@ static int hisi_hha_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &hha_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); @@ -503,7 +503,7 @@ static int hisi_hha_pmu_probe(struct platform_device *pdev) if (ret) { dev_err(hha_pmu->dev, "HHA PMU register failed!\n"); cpuhp_state_remove_instance_nocalls( - CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, &hha_pmu->node); + hisi_uncore_pmu_cpuhp_state, &hha_pmu->node); } return ret; @@ -514,7 +514,7 @@ static void hisi_hha_pmu_remove(struct platform_device *pdev) struct hisi_pmu *hha_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&hha_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &hha_pmu->node); } @@ -528,33 +528,7 @@ static struct platform_driver hisi_hha_pmu_driver = { .remove = hisi_hha_pmu_remove, }; -static int __init hisi_hha_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, - "AP_PERF_ARM_HISI_HHA_ONLINE", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret) { - pr_err("HHA PMU: Error setup hotplug, ret = %d;\n", ret); - return ret; - } - - ret = platform_driver_register(&hisi_hha_pmu_driver); - if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE); - - return ret; -} -module_init(hisi_hha_pmu_module_init); - -static void __exit hisi_hha_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_hha_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE); -} -module_exit(hisi_hha_pmu_module_exit); +module_platform_driver(hisi_hha_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC HHA uncore PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_its_pmu.c b/drivers/perf/hisilicon/hisi_uncore_its_pmu.c new file mode 100644 index 000000000000..66311657db89 --- /dev/null +++ b/drivers/perf/hisilicon/hisi_uncore_its_pmu.c @@ -0,0 +1,385 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Driver for HiSilicon Uncore ITS PMU device + * + * Copyright (c) 2026 HiSilicon Technologies Co., Ltd. + * Author: Yushan Wang <wangyushan12@huawei.com> + */ +#include <linux/bitops.h> +#include <linux/cpuhotplug.h> +#include <linux/device.h> +#include <linux/io.h> +#include <linux/module.h> +#include <linux/platform_device.h> +#include <linux/property.h> +#include <linux/spinlock.h> +#include <linux/sysfs.h> + +#include "hisi_uncore_pmu.h" + +#define ITS_PMU_VERSION 0x21000 +#define ITS_PMU_GLOBAL_CTRL 0x21004 +#define ITS_PMU_GLOBAL_CTRL_PMU_EN BIT(0) +#define ITS_PMU_COUNTER_CTRL 0x21008 +#define ITS_PMU_EVENT_CTRL 0x2100c +#define ITS_PMU_COUNTER0 0x21010 + +#define ITS_PMU_INT_ID_MASK 0x20008 +#define ITS_PMU_INT_ID_CTRL 0x20084 + +#define ITS_PMU_NR_COUNTERS 4 + +#define ITS_PMU_EVENT_CNTRn(cntr0, n) ((cntr0) + 8 * (n)) +#define ITS_PMU_CNTR_CTRL_MASK(n) GENMASK(8 * ((n) + 1) - 1, 8 * (n)) +#define ITS_PMU_CNTR_EVENT_CFG(n, e) ((e) << ((n) * 8)) +#define ITS_PMU_EVENT_CTRL_TYPE GENMASK(12, 0) + +HISI_PMU_EVENT_ATTR_EXTRACTOR(int_id, config1, 31, 0); +HISI_PMU_EVENT_ATTR_EXTRACTOR(int_en, config1, 32, 32); + +struct hisi_its_pmu_regs { + u32 version; + u32 pmu_ctrl; + u32 event_ctrl0; + u32 event_cntr0; + u32 cntr_ctrl; +}; + +static void hisi_its_pmu_write_evtype(struct hisi_pmu *pmu, int idx, u32 type) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + u32 reg; + + reg = readl(pmu->base + reg_info->event_ctrl0); + reg &= ~ITS_PMU_CNTR_CTRL_MASK(idx); + reg |= ITS_PMU_CNTR_EVENT_CFG(idx, type); + writel(reg, pmu->base + reg_info->event_ctrl0); +} + +static u64 hisi_its_pmu_read_counter(struct hisi_pmu *pmu, + struct hw_perf_event *hwc) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + + return readq(pmu->base + ITS_PMU_EVENT_CNTRn(reg_info->event_cntr0, hwc->idx)); +} + +static void hisi_its_pmu_write_counter(struct hisi_pmu *pmu, + struct hw_perf_event *hwc, u64 val) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + + writeq(val, pmu->base + ITS_PMU_EVENT_CNTRn(reg_info->event_cntr0, hwc->idx)); +} + +static void hisi_its_pmu_enable_counter(struct hisi_pmu *pmu, + struct hw_perf_event *hwc) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + u32 reg; + + reg = readl(pmu->base + reg_info->cntr_ctrl); + reg |= BIT(hwc->idx); + writel(reg, pmu->base + reg_info->cntr_ctrl); +} + +static void hisi_its_pmu_disable_counter(struct hisi_pmu *pmu, + struct hw_perf_event *hwc) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + u32 reg; + + reg = readl(pmu->base + reg_info->cntr_ctrl); + reg &= ~BIT(hwc->idx); + writel(reg, pmu->base + reg_info->cntr_ctrl); +} + +static void hisi_its_pmu_enable_counter_int(struct hisi_pmu *pmu, + struct hw_perf_event *hwc) +{ + /* No overflow interrupt; counters rely on the framework's priming. */ +} + +static void hisi_its_pmu_disable_counter_int(struct hisi_pmu *pmu, + struct hw_perf_event *hwc) +{ +} + +static void hisi_its_pmu_start_counters(struct hisi_pmu *pmu) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + u32 reg; + + reg = readl(pmu->base + reg_info->pmu_ctrl); + reg |= ITS_PMU_GLOBAL_CTRL_PMU_EN; + writel(reg, pmu->base + reg_info->pmu_ctrl); +} + +static void hisi_its_pmu_stop_counters(struct hisi_pmu *pmu) +{ + struct hisi_its_pmu_regs *reg_info = pmu->dev_info->private; + u32 reg; + + reg = readl(pmu->base + reg_info->pmu_ctrl); + reg &= ~ITS_PMU_GLOBAL_CTRL_PMU_EN; + writel(reg, pmu->base + reg_info->pmu_ctrl); +} + +static void hisi_its_pmu_enable_filter(struct perf_event *event) +{ + struct hisi_pmu *pmu = to_hisi_pmu(event->pmu); + u32 int_id = hisi_get_int_id(event); + u32 int_en = hisi_get_int_en(event); + + if (!int_en) + return; + + writel(int_id, pmu->base + ITS_PMU_INT_ID_CTRL); + /* Write 0 to this register to enable filtering with int_id. */ + writel(0, pmu->base + ITS_PMU_INT_ID_MASK); +} + +static void hisi_its_pmu_disable_filter(struct perf_event *event) +{ + struct hisi_pmu *pmu = to_hisi_pmu(event->pmu); + u32 int_en = hisi_get_int_en(event); + + if (!int_en) + return; + + /* + * The INT_ID filter is global, leave it alone if there are still + * active counters. + */ + if (bitmap_weight(pmu->pmu_events.used_mask, pmu->num_counters) > 1) + return; + + writel(0, pmu->base + ITS_PMU_INT_ID_CTRL); + writel(-1U, pmu->base + ITS_PMU_INT_ID_MASK); +} + +/* + * The INT_ID filter is global to the PMU. Events can coexist only if they + * agree on it: both unfiltered, or both filtering the same int_id. + */ +static bool hisi_its_pmu_check_global_filter(struct perf_event *curr, + struct perf_event *new) +{ + u32 curr_en = hisi_get_int_en(curr); + u32 new_en = hisi_get_int_en(new); + + if (curr_en != new_en) + return false; + + if (curr_en) + return hisi_get_int_id(curr) == hisi_get_int_id(new); + + return true; +} + +static int hisi_its_pmu_get_event_idx(struct perf_event *event) +{ + struct hisi_pmu *pmu = to_hisi_pmu(event->pmu); + struct hisi_pmu_hwevents *pmu_events = &pmu->pmu_events; + int cur_idx; + + cur_idx = find_first_bit(pmu_events->used_mask, pmu->num_counters); + if (cur_idx != pmu->num_counters && + !hisi_its_pmu_check_global_filter(pmu_events->hw_events[cur_idx], event)) + return -EAGAIN; + + return hisi_uncore_pmu_get_event_idx(event); +} + +static const struct hisi_uncore_ops hisi_uncore_its_ops = { + .write_evtype = hisi_its_pmu_write_evtype, + .get_event_idx = hisi_its_pmu_get_event_idx, + .read_counter = hisi_its_pmu_read_counter, + .write_counter = hisi_its_pmu_write_counter, + .enable_counter = hisi_its_pmu_enable_counter, + .disable_counter = hisi_its_pmu_disable_counter, + .enable_counter_int = hisi_its_pmu_enable_counter_int, + .disable_counter_int = hisi_its_pmu_disable_counter_int, + .start_counters = hisi_its_pmu_start_counters, + .stop_counters = hisi_its_pmu_stop_counters, + .enable_filter = hisi_its_pmu_enable_filter, + .disable_filter = hisi_its_pmu_disable_filter, +}; + +static struct attribute *hisi_its_pmu_format_attrs[] = { + HISI_PMU_FORMAT_ATTR(event, "config:0-7"), + HISI_PMU_FORMAT_ATTR(int_id, "config1:0-31"), + HISI_PMU_FORMAT_ATTR(int_en, "config1:32-32"), + NULL +}; + +static const struct attribute_group hisi_its_pmu_format_group = { + .name = "format", + .attrs = hisi_its_pmu_format_attrs, +}; + +static struct attribute *hisi_its_pmu_events_attrs[] = { + HISI_PMU_EVENT_ATTR(lpi_num, 0xc0), + HISI_PMU_EVENT_ATTR(lpi_time, 0x80), + HISI_PMU_EVENT_ATTR(sgi_num, 0xc1), + HISI_PMU_EVENT_ATTR(sgi_time, 0x81), + HISI_PMU_EVENT_ATTR(ppi_num, 0xc2), + HISI_PMU_EVENT_ATTR(ppi_time, 0x82), + HISI_PMU_EVENT_ATTR(sl3_lpi_num, 0xc3), + HISI_PMU_EVENT_ATTR(sl3_sgi_num, 0xc4), + HISI_PMU_EVENT_ATTR(sl3_ppi_num, 0xc5), + HISI_PMU_EVENT_ATTR(sl0_ddr_read, 0xc9), + HISI_PMU_EVENT_ATTR(sl0_ddr_time, 0x89), + HISI_PMU_EVENT_ATTR(sl1_ddr_read, 0xca), + HISI_PMU_EVENT_ATTR(sl1_ddr_time, 0x8a), + HISI_PMU_EVENT_ATTR(sl2_ddr_read, 0xcb), + HISI_PMU_EVENT_ATTR(sl2_ddr_time, 0x8b), + HISI_PMU_EVENT_ATTR(cycles, 0xcc), + NULL +}; + +static const struct attribute_group hisi_its_pmu_events_group = { + .name = "events", + .attrs = hisi_its_pmu_events_attrs, +}; + +static const struct attribute_group *hisi_its_pmu_attr_groups[] = { + &hisi_its_pmu_format_group, + &hisi_its_pmu_events_group, + &hisi_pmu_cpumask_attr_group, + &hisi_pmu_identifier_group, + NULL +}; + +static int hisi_its_pmu_dev_init(struct platform_device *pdev, + struct hisi_pmu *pmu) +{ + struct hisi_its_pmu_regs *reg_info; + + hisi_uncore_pmu_init_topology(pmu, &pdev->dev); + + if (pmu->topo.scl_id < 0) + return dev_err_probe(&pdev->dev, -EINVAL, "failed to get scl-id\n"); + + if (pmu->topo.index_id < 0) + return dev_err_probe(&pdev->dev, -EINVAL, "failed to get idx-id\n"); + + pmu->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(pmu->base)) + return dev_err_probe(&pdev->dev, PTR_ERR(pmu->base), + "fail to remap io memory\n"); + + pmu->dev_info = device_get_match_data(&pdev->dev); + if (!pmu->dev_info) + return -ENODEV; + + pmu->pmu_events.attr_groups = pmu->dev_info->attr_groups; + pmu->counter_bits = pmu->dev_info->counter_bits; + pmu->check_event = pmu->dev_info->check_event; + pmu->num_counters = ITS_PMU_NR_COUNTERS; + pmu->ops = &hisi_uncore_its_ops; + pmu->dev = &pdev->dev; + pmu->on_cpu = -1; + + reg_info = pmu->dev_info->private; + pmu->identifier = readl(pmu->base + reg_info->version); + + /* Reset the global INT_ID filter to disabled. */ + writel(0, pmu->base + ITS_PMU_INT_ID_CTRL); + writel(-1U, pmu->base + ITS_PMU_INT_ID_MASK); + + return 0; +} + +static void hisi_its_pmu_remove_cpuhp_instance(void *hotplug_node) +{ + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, + hotplug_node); +} + +static void hisi_its_pmu_unregister_pmu(void *pmu) +{ + perf_pmu_unregister(pmu); +} + +static int hisi_its_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct hisi_pmu *pmu; + char *name; + int ret; + + pmu = devm_kzalloc(dev, sizeof(*pmu), GFP_KERNEL); + if (!pmu) + return -ENOMEM; + + /* + * HiSilicon Uncore PMU framework needs to get common hisi_pmu device + * from device's drvdata. + */ + platform_set_drvdata(pdev, pmu); + + ret = hisi_its_pmu_dev_init(pdev, pmu); + if (ret) + return ret; + + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &pmu->node); + if (ret) + return dev_err_probe(dev, ret, "Fail to register cpuhp instance\n"); + + ret = devm_add_action_or_reset(dev, hisi_its_pmu_remove_cpuhp_instance, + &pmu->node); + if (ret) + return ret; + + hisi_pmu_init(pmu, THIS_MODULE); + + name = devm_kasprintf(dev, GFP_KERNEL, "hisi_scl%d_its%d", + pmu->topo.scl_id, pmu->topo.index_id); + if (!name) + return -ENOMEM; + + ret = perf_pmu_register(&pmu->pmu, name, -1); + if (ret) + return dev_err_probe(dev, ret, "Fail to register PMU\n"); + + return devm_add_action_or_reset(dev, hisi_its_pmu_unregister_pmu, + &pmu->pmu); +} + +static struct hisi_its_pmu_regs hisi_its_v1_pmu_regs = { + .version = ITS_PMU_VERSION, + .pmu_ctrl = ITS_PMU_GLOBAL_CTRL, + .event_ctrl0 = ITS_PMU_EVENT_CTRL, + .event_cntr0 = ITS_PMU_COUNTER0, + .cntr_ctrl = ITS_PMU_COUNTER_CTRL, +}; + +static const struct hisi_pmu_dev_info hisi_its_v1 = { + .attr_groups = hisi_its_pmu_attr_groups, + .counter_bits = 48, + .check_event = ITS_PMU_EVENT_CTRL_TYPE, + .private = &hisi_its_v1_pmu_regs, +}; + +static const struct acpi_device_id hisi_its_pmu_ids[] = { + { "HISI0591", (kernel_ulong_t)&hisi_its_v1 }, + { } +}; +MODULE_DEVICE_TABLE(acpi, hisi_its_pmu_ids); + +static struct platform_driver hisi_its_pmu_driver = { + .driver = { + .name = "hisi_its_pmu", + .acpi_match_table = hisi_its_pmu_ids, + .suppress_bind_attrs = true, + }, + .probe = hisi_its_pmu_probe, +}; + +module_platform_driver(hisi_its_pmu_driver); + +MODULE_IMPORT_NS("HISI_PMU"); +MODULE_DESCRIPTION("HiSilicon SoC Uncore ITS PMU driver"); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Yushan Wang <wangyushan12@huawei.com>"); diff --git a/drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c b/drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c index 56a88fb0d3c2..e40506e09702 100644 --- a/drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c @@ -93,6 +93,8 @@ struct hisi_l3c_pmu_ext { bool support_ext; }; +static enum cpuhp_state hisi_l3c_pmu_cpuhp_state; + static bool support_ext(struct hisi_l3c_pmu *pmu) { struct hisi_l3c_pmu_ext *l3c_pmu_ext = pmu->l3c_pmu.dev_info->private; @@ -844,8 +846,7 @@ static int hisi_l3c_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, - &l3c_pmu->node); + ret = cpuhp_state_add_instance(hisi_l3c_pmu_cpuhp_state, &l3c_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); return ret; @@ -857,7 +858,7 @@ static int hisi_l3c_pmu_probe(struct platform_device *pdev) if (ret) { dev_err(l3c_pmu->dev, "L3C PMU register failed!\n"); cpuhp_state_remove_instance_nocalls( - CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, &l3c_pmu->node); + hisi_l3c_pmu_cpuhp_state, &l3c_pmu->node); } return ret; @@ -868,7 +869,7 @@ static void hisi_l3c_pmu_remove(struct platform_device *pdev) struct hisi_pmu *l3c_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&l3c_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_l3c_pmu_cpuhp_state, &l3c_pmu->node); } @@ -938,20 +939,20 @@ static int hisi_l3c_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) static int __init hisi_l3c_pmu_module_init(void) { - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, + int ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "AP_PERF_ARM_HISI_L3_ONLINE", hisi_l3c_pmu_online_cpu, hisi_l3c_pmu_offline_cpu); - if (ret) { + if (ret < 0) { pr_err("L3C PMU: Error setup hotplug, ret = %d\n", ret); return ret; } + hisi_l3c_pmu_cpuhp_state = ret; + ret = platform_driver_register(&hisi_l3c_pmu_driver); if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE); + cpuhp_remove_multi_state(hisi_l3c_pmu_cpuhp_state); return ret; } @@ -960,7 +961,7 @@ module_init(hisi_l3c_pmu_module_init); static void __exit hisi_l3c_pmu_module_exit(void) { platform_driver_unregister(&hisi_l3c_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE); + cpuhp_remove_multi_state(hisi_l3c_pmu_cpuhp_state); } module_exit(hisi_l3c_pmu_module_exit); diff --git a/drivers/perf/hisilicon/hisi_uncore_mn_pmu.c b/drivers/perf/hisilicon/hisi_uncore_mn_pmu.c index 246cc0333099..1ded8a6b2358 100644 --- a/drivers/perf/hisilicon/hisi_uncore_mn_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_mn_pmu.c @@ -13,9 +13,6 @@ #include "hisi_uncore_pmu.h" -/* Dynamic CPU hotplug state used by MN PMU */ -static enum cpuhp_state hisi_mn_pmu_online; - /* MN register definition */ #define HISI_MN_DYNAMIC_CTRL_REG 0x400 #define HISI_MN_DYNAMIC_CTRL_EN BIT(0) @@ -191,7 +188,47 @@ static const struct attribute_group hisi_mn_pmu_format_group = { .attrs = hisi_mn_pmu_format_attr, }; -static struct attribute *hisi_mn_pmu_events_attr[] = { +static struct attribute *hisi_mn_pmu_events_attr_v1[] = { + HISI_PMU_EVENT_ATTR(req_eobarrier_num, 0x00), + HISI_PMU_EVENT_ATTR(req_ecbarrier_num, 0x01), + HISI_PMU_EVENT_ATTR(req_dvmop_num, 0x02), + HISI_PMU_EVENT_ATTR(req_dvmsync_num, 0x03), + HISI_PMU_EVENT_ATTR(req_retry_num, 0x04), + HISI_PMU_EVENT_ATTR(req_writenosnp_num, 0x05), + HISI_PMU_EVENT_ATTR(req_readnosnp_num, 0x06), + HISI_PMU_EVENT_ATTR(snp_dvm_num, 0x07), + HISI_PMU_EVENT_ATTR(snp_dvmsync_num, 0x08), + HISI_PMU_EVENT_ATTR(l3t_req_dvm_num, 0x09), + HISI_PMU_EVENT_ATTR(l3t_req_dvmsync_num, 0x0A), + HISI_PMU_EVENT_ATTR(mn_req_dvm_num, 0x0B), + HISI_PMU_EVENT_ATTR(mn_req_dvmsync_num, 0x0C), + HISI_PMU_EVENT_ATTR(pa_req_dvm_num, 0x0D), + HISI_PMU_EVENT_ATTR(pa_req_dvmsync_num, 0x0E), + HISI_PMU_EVENT_ATTR(snp_dvm_latency, 0x80), + HISI_PMU_EVENT_ATTR(snp_dvmsync_latency, 0x81), + HISI_PMU_EVENT_ATTR(l3t_req_dvm_latency, 0x82), + HISI_PMU_EVENT_ATTR(l3t_req_dvmsync_latency, 0x83), + HISI_PMU_EVENT_ATTR(mn_req_dvm_latency, 0x84), + HISI_PMU_EVENT_ATTR(mn_req_dvmsync_latency, 0x85), + HISI_PMU_EVENT_ATTR(pa_req_dvm_latency, 0x86), + HISI_PMU_EVENT_ATTR(pa_req_dvmsync_latency, 0x87), + NULL +}; + +static const struct attribute_group hisi_mn_pmu_events_group_v1 = { + .name = "events", + .attrs = hisi_mn_pmu_events_attr_v1, +}; + +static const struct attribute_group *hisi_mn_pmu_attr_groups_v1[] = { + &hisi_mn_pmu_format_group, + &hisi_mn_pmu_events_group_v1, + &hisi_pmu_cpumask_attr_group, + &hisi_pmu_identifier_group, + NULL +}; + +static struct attribute *hisi_mn_pmu_events_attr_v2[] = { HISI_PMU_EVENT_ATTR(req_eobarrier_num, 0x00), HISI_PMU_EVENT_ATTR(req_ecbarrier_num, 0x01), HISI_PMU_EVENT_ATTR(req_dvmop_num, 0x02), @@ -207,6 +244,7 @@ static struct attribute *hisi_mn_pmu_events_attr[] = { HISI_PMU_EVENT_ATTR(mn_req_dvmsync_num, 0x0C), HISI_PMU_EVENT_ATTR(pa_req_dvm_num, 0x0D), HISI_PMU_EVENT_ATTR(pa_req_dvmsync_num, 0x0E), + HISI_PMU_EVENT_ATTR(cycles, 0x0F), HISI_PMU_EVENT_ATTR(snp_dvm_latency, 0x80), HISI_PMU_EVENT_ATTR(snp_dvmsync_latency, 0x81), HISI_PMU_EVENT_ATTR(l3t_req_dvm_latency, 0x82), @@ -218,14 +256,14 @@ static struct attribute *hisi_mn_pmu_events_attr[] = { NULL }; -static const struct attribute_group hisi_mn_pmu_events_group = { +static const struct attribute_group hisi_mn_pmu_events_group_v2 = { .name = "events", - .attrs = hisi_mn_pmu_events_attr, + .attrs = hisi_mn_pmu_events_attr_v2, }; -static const struct attribute_group *hisi_mn_pmu_attr_groups[] = { +static const struct attribute_group *hisi_mn_pmu_attr_groups_v2[] = { &hisi_mn_pmu_format_group, - &hisi_mn_pmu_events_group, + &hisi_mn_pmu_events_group_v2, &hisi_pmu_cpumask_attr_group, &hisi_pmu_identifier_group, NULL @@ -291,7 +329,7 @@ static int hisi_mn_pmu_dev_init(struct platform_device *pdev, static void hisi_mn_pmu_remove_cpuhp(void *hotplug_node) { - cpuhp_state_remove_instance_nocalls(hisi_mn_pmu_online, hotplug_node); + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, hotplug_node); } static void hisi_mn_pmu_unregister(void *pmu) @@ -320,7 +358,7 @@ static int hisi_mn_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(hisi_mn_pmu_online, &mn_pmu->node); + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &mn_pmu->node); if (ret) return dev_err_probe(&pdev->dev, ret, "Failed to register cpu hotplug\n"); @@ -350,14 +388,22 @@ static struct hisi_mn_pmu_regs hisi_mn_v1_pmu_regs = { }; static const struct hisi_pmu_dev_info hisi_mn_v1 = { - .attr_groups = hisi_mn_pmu_attr_groups, + .attr_groups = hisi_mn_pmu_attr_groups_v1, + .counter_bits = 48, + .check_event = HISI_MN_EVTYPE_MASK, + .private = &hisi_mn_v1_pmu_regs, +}; + +static const struct hisi_pmu_dev_info hisi_mn_v2 = { + .attr_groups = hisi_mn_pmu_attr_groups_v2, .counter_bits = 48, .check_event = HISI_MN_EVTYPE_MASK, .private = &hisi_mn_v1_pmu_regs, }; static const struct acpi_device_id hisi_mn_pmu_acpi_match[] = { - { "HISI0222", (kernel_ulong_t) &hisi_mn_v1 }, + { "HISI0222", (kernel_ulong_t)&hisi_mn_v1 }, + { "HISI0224", (kernel_ulong_t)&hisi_mn_v2 }, { } }; MODULE_DEVICE_TABLE(acpi, hisi_mn_pmu_acpi_match); @@ -376,33 +422,7 @@ static struct platform_driver hisi_mn_pmu_driver = { .probe = hisi_mn_pmu_probe, }; -static int __init hisi_mn_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "perf/hisi/mn:online", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret < 0) { - pr_err("hisi_mn_pmu: Failed to setup MN PMU hotplug: %d\n", ret); - return ret; - } - hisi_mn_pmu_online = ret; - - ret = platform_driver_register(&hisi_mn_pmu_driver); - if (ret) - cpuhp_remove_multi_state(hisi_mn_pmu_online); - - return ret; -} -module_init(hisi_mn_pmu_module_init); - -static void __exit hisi_mn_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_mn_pmu_driver); - cpuhp_remove_multi_state(hisi_mn_pmu_online); -} -module_exit(hisi_mn_pmu_module_exit); +module_platform_driver(hisi_mn_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC MN uncore PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_noc_pmu.c b/drivers/perf/hisilicon/hisi_uncore_noc_pmu.c index 616f4af57db7..abf44fa46d17 100644 --- a/drivers/perf/hisilicon/hisi_uncore_noc_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_noc_pmu.c @@ -41,9 +41,6 @@ HISI_PMU_EVENT_ATTR_EXTRACTOR(ch, config1, 2, 0); HISI_PMU_EVENT_ATTR_EXTRACTOR(tt_en, config1, 3, 3); -/* Dynamic CPU hotplug state used by this PMU driver */ -static enum cpuhp_state hisi_noc_pmu_cpuhp_state; - struct hisi_noc_pmu_regs { u32 version; u32 pmu_ctrl; @@ -324,7 +321,7 @@ static int hisi_noc_pmu_dev_init(struct platform_device *pdev, struct hisi_pmu * static void hisi_noc_pmu_remove_cpuhp_instance(void *hotplug_node) { - cpuhp_state_remove_instance_nocalls(hisi_noc_pmu_cpuhp_state, hotplug_node); + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, hotplug_node); } static void hisi_noc_pmu_unregister_pmu(void *pmu) @@ -353,7 +350,7 @@ static int hisi_noc_pmu_probe(struct platform_device *pdev) if (ret) return ret; - ret = cpuhp_state_add_instance(hisi_noc_pmu_cpuhp_state, &noc_pmu->node); + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &noc_pmu->node); if (ret) return dev_err_probe(dev, ret, "Fail to register cpuhp instance\n"); @@ -408,33 +405,7 @@ static struct platform_driver hisi_noc_pmu_driver = { .probe = hisi_noc_pmu_probe, }; -static int __init hisi_noc_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "perf/hisi/noc:online", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret < 0) { - pr_err("hisi_noc_pmu: Fail to setup cpuhp callbacks, ret = %d\n", ret); - return ret; - } - hisi_noc_pmu_cpuhp_state = ret; - - ret = platform_driver_register(&hisi_noc_pmu_driver); - if (ret) - cpuhp_remove_multi_state(hisi_noc_pmu_cpuhp_state); - - return ret; -} -module_init(hisi_noc_pmu_module_init); - -static void __exit hisi_noc_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_noc_pmu_driver); - cpuhp_remove_multi_state(hisi_noc_pmu_cpuhp_state); -} -module_exit(hisi_noc_pmu_module_exit); +module_platform_driver(hisi_noc_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC Uncore NoC PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c index 80108c63cb60..3d1e0af64260 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c @@ -468,7 +468,7 @@ static int hisi_pa_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE, + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &pa_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); @@ -479,7 +479,7 @@ static int hisi_pa_pmu_probe(struct platform_device *pdev) ret = perf_pmu_register(&pa_pmu->pmu, name, -1); if (ret) { dev_err(pa_pmu->dev, "PMU register failed, ret = %d\n", ret); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &pa_pmu->node); return ret; } @@ -493,7 +493,7 @@ static void hisi_pa_pmu_remove(struct platform_device *pdev) struct hisi_pmu *pa_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&pa_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &pa_pmu->node); } @@ -515,33 +515,7 @@ static struct platform_driver hisi_pa_pmu_driver = { .remove = hisi_pa_pmu_remove, }; -static int __init hisi_pa_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE, - "AP_PERF_ARM_HISI_PA_ONLINE", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret) { - pr_err("PA PMU: cpuhp state setup failed, ret = %d\n", ret); - return ret; - } - - ret = platform_driver_register(&hisi_pa_pmu_driver); - if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE); - - return ret; -} -module_init(hisi_pa_pmu_module_init); - -static void __exit hisi_pa_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_pa_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE); -} -module_exit(hisi_pa_pmu_module_exit); +module_platform_driver(hisi_pa_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon Protocol Adapter uncore PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_pmu.c b/drivers/perf/hisilicon/hisi_uncore_pmu.c index 77390d033d08..346c87594614 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_pmu.c @@ -23,6 +23,9 @@ #define HISI_MAX_PERIOD(nr) (GENMASK_ULL((nr) - 1, 0)) +enum cpuhp_state hisi_uncore_pmu_cpuhp_state; +EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_cpuhp_state, "HISI_PMU"); + /* * PMU event attributes */ @@ -618,5 +621,27 @@ void hisi_pmu_init(struct hisi_pmu *hisi_pmu, struct module *module) } EXPORT_SYMBOL_NS_GPL(hisi_pmu_init, "HISI_PMU"); +static int __init hisi_uncore_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "perf/hisi/uncore:online", + hisi_uncore_pmu_online_cpu, + hisi_uncore_pmu_offline_cpu); + if (ret < 0) + return ret; + + hisi_uncore_pmu_cpuhp_state = ret; + + return 0; +} +module_init(hisi_uncore_pmu_init); + +static void __exit hisi_uncore_pmu_exit(void) +{ + cpuhp_remove_multi_state(hisi_uncore_pmu_cpuhp_state); +} +module_exit(hisi_uncore_pmu_exit); + MODULE_DESCRIPTION("HiSilicon SoC uncore Performance Monitor driver framework"); MODULE_LICENSE("GPL v2"); diff --git a/drivers/perf/hisilicon/hisi_uncore_pmu.h b/drivers/perf/hisilicon/hisi_uncore_pmu.h index 3ffe6acda653..e5a4b3898865 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pmu.h +++ b/drivers/perf/hisilicon/hisi_uncore_pmu.h @@ -171,4 +171,6 @@ int hisi_uncore_pmu_init_irq(struct hisi_pmu *hisi_pmu, void hisi_uncore_pmu_init_topology(struct hisi_pmu *hisi_pmu, struct device *dev); void hisi_pmu_init(struct hisi_pmu *hisi_pmu, struct module *module); + +extern enum cpuhp_state hisi_uncore_pmu_cpuhp_state; #endif /* __HISI_UNCORE_PMU_H__ */ diff --git a/drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c b/drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c index cd32d606df05..d01340ff2e5c 100644 --- a/drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c @@ -503,7 +503,7 @@ static int hisi_sllc_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE, + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &sllc_pmu->node); if (ret) { dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); @@ -515,7 +515,7 @@ static int hisi_sllc_pmu_probe(struct platform_device *pdev) ret = perf_pmu_register(&sllc_pmu->pmu, name, -1); if (ret) { dev_err(sllc_pmu->dev, "PMU register failed, ret = %d\n", ret); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &sllc_pmu->node); return ret; } @@ -530,7 +530,7 @@ static void hisi_sllc_pmu_remove(struct platform_device *pdev) struct hisi_pmu *sllc_pmu = platform_get_drvdata(pdev); perf_pmu_unregister(&sllc_pmu->pmu); - cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE, + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, &sllc_pmu->node); } @@ -551,33 +551,7 @@ static struct platform_driver hisi_sllc_pmu_driver = { .remove = hisi_sllc_pmu_remove, }; -static int __init hisi_sllc_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE, - "AP_PERF_ARM_HISI_SLLC_ONLINE", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret) { - pr_err("SLLC PMU: cpuhp state setup failed, ret = %d\n", ret); - return ret; - } - - ret = platform_driver_register(&hisi_sllc_pmu_driver); - if (ret) - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE); - - return ret; -} -module_init(hisi_sllc_pmu_module_init); - -static void __exit hisi_sllc_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_sllc_pmu_driver); - cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE); -} -module_exit(hisi_sllc_pmu_module_exit); +module_platform_driver(hisi_sllc_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SLLC uncore PMU driver"); diff --git a/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c b/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c index e8186b6e1687..b4d017a5d688 100644 --- a/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c @@ -13,9 +13,6 @@ #include "hisi_uncore_pmu.h" -/* Dynamic CPU hotplug state used by UC PMU */ -static enum cpuhp_state hisi_uc_pmu_online; - /* UC register definition */ #define HISI_UC_INT_MASK_REG 0x0800 #define HISI_UC_INT_STS_REG 0x0808 @@ -488,7 +485,7 @@ static int hisi_uc_pmu_dev_probe(struct platform_device *pdev, static void hisi_uc_pmu_remove_cpuhp_instance(void *hotplug_node) { - cpuhp_state_remove_instance_nocalls(hisi_uc_pmu_online, hotplug_node); + cpuhp_state_remove_instance_nocalls(hisi_uncore_pmu_cpuhp_state, hotplug_node); } static void hisi_uc_pmu_unregister_pmu(void *pmu) @@ -518,7 +515,7 @@ static int hisi_uc_pmu_probe(struct platform_device *pdev) if (!name) return -ENOMEM; - ret = cpuhp_state_add_instance(hisi_uc_pmu_online, &uc_pmu->node); + ret = cpuhp_state_add_instance(hisi_uncore_pmu_cpuhp_state, &uc_pmu->node); if (ret) return dev_err_probe(&pdev->dev, ret, "Error registering hotplug\n"); @@ -559,34 +556,7 @@ static struct platform_driver hisi_uc_pmu_driver = { .probe = hisi_uc_pmu_probe, }; -static int __init hisi_uc_pmu_module_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, - "perf/hisi/uc:online", - hisi_uncore_pmu_online_cpu, - hisi_uncore_pmu_offline_cpu); - if (ret < 0) { - pr_err("UC PMU: Error setup hotplug, ret = %d\n", ret); - return ret; - } - hisi_uc_pmu_online = ret; - - ret = platform_driver_register(&hisi_uc_pmu_driver); - if (ret) - cpuhp_remove_multi_state(hisi_uc_pmu_online); - - return ret; -} -module_init(hisi_uc_pmu_module_init); - -static void __exit hisi_uc_pmu_module_exit(void) -{ - platform_driver_unregister(&hisi_uc_pmu_driver); - cpuhp_remove_multi_state(hisi_uc_pmu_online); -} -module_exit(hisi_uc_pmu_module_exit); +module_platform_driver(hisi_uc_pmu_driver); MODULE_IMPORT_NS("HISI_PMU"); MODULE_DESCRIPTION("HiSilicon SoC UC uncore PMU driver"); diff --git a/drivers/perf/marvell_cn10k_ddr_pmu.c b/drivers/perf/marvell_cn10k_ddr_pmu.c index 0a8cb6166408..9b68d6c1c31c 100644 --- a/drivers/perf/marvell_cn10k_ddr_pmu.c +++ b/drivers/perf/marvell_cn10k_ddr_pmu.c @@ -1165,6 +1165,47 @@ static const struct ddr_pmu_platform_data odyssey_ddr_pmu_pdata = { }; #endif +static const struct pmu cn10k_pmu = { + .module = THIS_MODULE, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE, + .task_ctx_nr = perf_invalid_context, + .attr_groups = cn10k_attr_groups, + .event_init = cn10k_ddr_perf_event_init, + .add = cn10k_ddr_perf_event_add, + .del = cn10k_ddr_perf_event_del, + .start = cn10k_ddr_perf_event_start, + .stop = cn10k_ddr_perf_event_stop, + .read = cn10k_ddr_perf_event_update, + .pmu_enable = cn10k_ddr_perf_pmu_enable, + .pmu_disable = cn10k_ddr_perf_pmu_disable, +}; + +static const struct pmu ody_pmu = { + .module = THIS_MODULE, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE, + .task_ctx_nr = perf_invalid_context, + .attr_groups = odyssey_attr_groups, + .event_init = cn10k_ddr_perf_event_init, + .add = cn10k_ddr_perf_event_add, + .del = cn10k_ddr_perf_event_del, + .start = cn10k_ddr_perf_event_start, + .stop = cn10k_ddr_perf_event_stop, + .read = cn10k_ddr_perf_event_update, +}; + +static const struct pmu cn20k_pmu = { + .module = THIS_MODULE, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE, + .task_ctx_nr = perf_invalid_context, + .attr_groups = cn20k_attr_groups, + .event_init = cn10k_ddr_perf_event_init, + .add = cn10k_ddr_perf_event_add, + .del = cn10k_ddr_perf_event_del, + .start = cn10k_ddr_perf_event_start, + .stop = cn10k_ddr_perf_event_stop, + .read = cn10k_ddr_perf_event_update, +}; + static int cn10k_ddr_perf_probe(struct platform_device *pdev) { const struct ddr_pmu_platform_data *dev_data; @@ -1202,56 +1243,20 @@ static int cn10k_ddr_perf_probe(struct platform_device *pdev) /* Setup the PMU counter to work in manual mode */ writeq_relaxed(OP_MODE_CTRL_VAL_MANUAL, ddr_pmu->base + ddr_pmu->p_data->cnt_op_mode_ctrl); + ddr_pmu->pmu = cn10k_pmu; - ddr_pmu->pmu = (struct pmu) { - .module = THIS_MODULE, - .capabilities = PERF_PMU_CAP_NO_EXCLUDE, - .task_ctx_nr = perf_invalid_context, - .attr_groups = cn10k_attr_groups, - .event_init = cn10k_ddr_perf_event_init, - .add = cn10k_ddr_perf_event_add, - .del = cn10k_ddr_perf_event_del, - .start = cn10k_ddr_perf_event_start, - .stop = cn10k_ddr_perf_event_stop, - .read = cn10k_ddr_perf_event_update, - .pmu_enable = cn10k_ddr_perf_pmu_enable, - .pmu_disable = cn10k_ddr_perf_pmu_disable, - }; } if (silicon_flags & IS_ODY) { ddr_pmu->ops = &ddr_pmu_ody_ops; - - ddr_pmu->pmu = (struct pmu) { - .module = THIS_MODULE, - .capabilities = PERF_PMU_CAP_NO_EXCLUDE, - .task_ctx_nr = perf_invalid_context, - .attr_groups = odyssey_attr_groups, - .event_init = cn10k_ddr_perf_event_init, - .add = cn10k_ddr_perf_event_add, - .del = cn10k_ddr_perf_event_del, - .start = cn10k_ddr_perf_event_start, - .stop = cn10k_ddr_perf_event_stop, - .read = cn10k_ddr_perf_event_update, - }; + ddr_pmu->pmu = ody_pmu; } if (silicon_flags & IS_CN20K) { ddr_pmu->ops = &ddr_pmu_ody_ops; - - ddr_pmu->pmu = (struct pmu) { - .module = THIS_MODULE, - .capabilities = PERF_PMU_CAP_NO_EXCLUDE, - .task_ctx_nr = perf_invalid_context, - .attr_groups = cn20k_attr_groups, - .event_init = cn10k_ddr_perf_event_init, - .add = cn10k_ddr_perf_event_add, - .del = cn10k_ddr_perf_event_del, - .start = cn10k_ddr_perf_event_start, - .stop = cn10k_ddr_perf_event_stop, - .read = cn10k_ddr_perf_event_update, - }; + ddr_pmu->pmu = cn20k_pmu; } + /* Choose this cpu to collect perf data */ ddr_pmu->cpu = raw_smp_processor_id(); diff --git a/drivers/perf/marvell_cn10k_tad_pmu.c b/drivers/perf/marvell_cn10k_tad_pmu.c index 54909d0031b7..301be4d92b1b 100644 --- a/drivers/perf/marvell_cn10k_tad_pmu.c +++ b/drivers/perf/marvell_cn10k_tad_pmu.c @@ -424,6 +424,7 @@ static const struct of_device_id tad_pmu_of_match[] = { { .compatible = "marvell,cn10k-tad-pmu", .data = &tad_pmu_data }, {}, }; +MODULE_DEVICE_TABLE(of, tad_pmu_of_match); #endif #ifdef CONFIG_ACPI diff --git a/drivers/perf/xgene_pmu.c b/drivers/perf/xgene_pmu.c index ab3623fab914..0aa2b605fd53 100644 --- a/drivers/perf/xgene_pmu.c +++ b/drivers/perf/xgene_pmu.c @@ -1873,6 +1873,8 @@ static int xgene_pmu_probe(struct platform_device *pdev) if (irq < 0) return -EINVAL; + raw_spin_lock_init(&xgene_pmu->lock); + rc = devm_request_irq(&pdev->dev, irq, xgene_pmu_isr, IRQF_NOBALANCING | IRQF_NO_THREAD, dev_name(&pdev->dev), xgene_pmu); @@ -1881,8 +1883,6 @@ static int xgene_pmu_probe(struct platform_device *pdev) xgene_pmu->irq = irq; - raw_spin_lock_init(&xgene_pmu->lock); - /* Check for active MCBs and MCUs */ rc = xgene_pmu_probe_active_mcb_mcu_l3c(xgene_pmu, pdev); if (rc) { diff --git a/drivers/resctrl/Kconfig b/drivers/resctrl/Kconfig index 672abea3b03c..f5ff9b1f6c48 100644 --- a/drivers/resctrl/Kconfig +++ b/drivers/resctrl/Kconfig @@ -2,6 +2,8 @@ menuconfig ARM64_MPAM_DRIVER bool "MPAM driver" depends on ARM64 && ARM64_MPAM select ACPI_MPAM if ACPI + select PCC if ACPI + select MAILBOX help Memory System Resource Partitioning and Monitoring (MPAM) driver for System IP, e.g. caches and memory controllers. diff --git a/drivers/resctrl/Makefile b/drivers/resctrl/Makefile index 4f6d0e81f9b8..097c036724e9 100644 --- a/drivers/resctrl/Makefile +++ b/drivers/resctrl/Makefile @@ -1,5 +1,5 @@ obj-$(CONFIG_ARM64_MPAM_DRIVER) += mpam.o -mpam-y += mpam_devices.o +mpam-y += mpam_devices.o mpam_fb.o mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS) += mpam_resctrl.o ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG) += -DDEBUG diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index dd422c56fbb1..48adc49b833c 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -12,6 +12,7 @@ #include <linux/cpu.h> #include <linux/cpumask.h> #include <linux/device.h> +#include <linux/errname.h> #include <linux/errno.h> #include <linux/gfp.h> #include <linux/interrupt.h> @@ -19,14 +20,19 @@ #include <linux/irqdesc.h> #include <linux/list.h> #include <linux/lockdep.h> +#include <linux/mailbox_client.h> #include <linux/mutex.h> #include <linux/platform_device.h> #include <linux/printk.h> +#include <linux/property.h> #include <linux/srcu.h> #include <linux/spinlock.h> #include <linux/types.h> #include <linux/workqueue.h> +#include <acpi/pcc.h> +#include <acpi/acpi_io.h> + #include "mpam_internal.h" /* Values for the T241 errata workaround */ @@ -49,6 +55,87 @@ static LIST_HEAD(mpam_all_msc); struct srcu_struct mpam_srcu; +/* PCC channels might be serving multiple MSCs, so keep a refcounted list. */ +static DEFINE_MUTEX(pcc_chan_list_lock); +static LIST_HEAD(pcc_chan_list); + +static void mpam_pcc_chan_release(struct kref *ref) +{ + struct mpam_pcc_chan *cur = container_of(ref, struct mpam_pcc_chan, + refcount); + + pcc_mbox_free_channel(cur->pcc_chan); + list_del(&cur->pcc_chans); + mutex_destroy(&cur->pcc_chan_lock); + kfree(cur); +} + +static struct mpam_pcc_chan *mpam_pcc_chan_get(struct device *dev, + int subspace_id) +{ + struct mpam_pcc_chan *cur; + + guard(mutex)(&pcc_chan_list_lock); + + list_for_each_entry(cur, &pcc_chan_list, pcc_chans) { + if (cur->subspace_id == subspace_id) { + kref_get(&cur->refcount); + + return cur; + } + } + + cur = kzalloc_obj(*cur); + if (!cur) + return ERR_PTR(-ENOMEM); + + cur->pcc_cl.tx_block = true; + + cur->pcc_chan = pcc_mbox_request_channel(&cur->pcc_cl, subspace_id); + if (IS_ERR(cur->pcc_chan)) { + long err = PTR_ERR(cur->pcc_chan); + + kfree(cur); + return ERR_PTR(err); + } + + /* + * Timeout based on the "nominal latency" in us, from the + * PCC ACPI table. tx_tout is in ms. + * Add some margin here to be on the safe side. + */ + cur->pcc_cl.tx_tout = DIV_ROUND_UP(cur->pcc_chan->latency * 5, 1000); + + mutex_init(&cur->pcc_chan_lock); + + cur->subspace_id = subspace_id; + kref_init(&cur->refcount); + + list_add_tail(&cur->pcc_chans, &pcc_chan_list); + + return cur; +} + +static int mpam_pcc_chan_put(struct mpam_pcc_chan *pcc_chan) +{ + struct mpam_pcc_chan *cur, *tmp; + + if (!pcc_chan) + return 0; + + guard(mutex)(&pcc_chan_list_lock); + + list_for_each_entry_safe(cur, tmp, &pcc_chan_list, pcc_chans) { + if (cur == pcc_chan) { + kref_put(&cur->refcount, mpam_pcc_chan_release); + + return 0; + } + } + + return -ENOENT; +} + /* * Number of MSCs that have been probed. Once all MSCs have been probed MPAM * can be enabled. @@ -82,6 +169,21 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable); /* When mpam is disabled, the printed reason to aid debugging */ static char *mpam_disable_reason; +static int mpam_disable_errno; +static int mpam_disable_mpam_fb_err; + +void mpam_fb_disable_mpam(int err, int mpam_fb_err) +{ + /* Prevent repeated calls when mpam_disable() does MSC accesses. */ + if (!mpam_is_enabled()) + return; + + mpam_disable_errno = err; + mpam_disable_mpam_fb_err = mpam_fb_err; + + mpam_disable_reason = "MPAM-Fb error"; + schedule_work(&mpam_broken_work); +} /* * Whether resctrl has been setup. Used by cpuhp in preference to @@ -157,7 +259,7 @@ static void mpam_free_garbage(void) if (iter->pdev) devm_kfree(&iter->pdev->dev, iter->to_free); else - kfree(iter->to_free); + kvfree(iter->to_free); } } @@ -177,59 +279,74 @@ static void mpam_assert_partid_sizes_fixed(void) WARN_ON_ONCE(!partid_max_published); } -static u32 __mpam_read_reg(struct mpam_msc *msc, u16 reg) +static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res) { + + if (msc->iface == MPAM_IFACE_PCC) + return mpam_fb_send_read_request(msc, reg, res); + WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility)); + *res = readl_relaxed(msc->mapped_hwpage + reg); - return readl_relaxed(msc->mapped_hwpage + reg); + return 0; } -static inline u32 _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg) +static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg, + u32 *res) { lockdep_assert_held_once(&msc->part_sel_lock); - return __mpam_read_reg(msc, reg); + return __mpam_read_reg(msc, reg, res); } -#define mpam_read_partsel_reg(msc, reg) _mpam_read_partsel_reg(msc, MPAMF_##reg) +#define mpam_read_partsel_reg(msc, reg, res) _mpam_read_partsel_reg(msc, MPAMF_##reg, res) -static void __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val) +static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val) { - WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz); - WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility)); + if (msc->iface == MPAM_IFACE_PCC) + return mpam_fb_send_write_request(msc, reg, val); + + WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility)); + WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz); writel_relaxed(val, msc->mapped_hwpage + reg); + + return 0; } -static inline void _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val) +static inline int _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val) { lockdep_assert_held_once(&msc->part_sel_lock); - __mpam_write_reg(msc, reg, val); + return __mpam_write_reg(msc, reg, val); } #define mpam_write_partsel_reg(msc, reg, val) _mpam_write_partsel_reg(msc, MPAMCFG_##reg, val) -static inline u32 _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg) +static inline int _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg, + u32 *res) { mpam_mon_sel_lock_held(msc); - return __mpam_read_reg(msc, reg); + return __mpam_read_reg(msc, reg, res); } -#define mpam_read_monsel_reg(msc, reg) _mpam_read_monsel_reg(msc, MSMON_##reg) +#define mpam_read_monsel_reg(msc, reg, res) _mpam_read_monsel_reg(msc, MSMON_##reg, res) -static inline void _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val) +static inline int _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val) { mpam_mon_sel_lock_held(msc); - __mpam_write_reg(msc, reg, val); + return __mpam_write_reg(msc, reg, val); } #define mpam_write_monsel_reg(msc, reg, val) _mpam_write_monsel_reg(msc, MSMON_##reg, val) static bool mpam_msc_check_aidr(struct mpam_msc *msc) { - u32 aidr = __mpam_read_reg(msc, MPAMF_AIDR); - u32 major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr); - u32 minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr); + u32 aidr, major, minor; + + if (__mpam_read_reg(msc, MPAMF_AIDR, &aidr)) + return false; + major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr); + minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr); /* * v0.0 and >v2.x aren't supported, but anything else should be backward * compatible to v0.1 or v1.0. @@ -242,25 +359,41 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc) return true; } -static u64 mpam_msc_read_idr(struct mpam_msc *msc) +static int mpam_msc_read_idr(struct mpam_msc *msc, u64 *res) { - u64 idr_high = 0, idr_low; + u32 idr_high, idr_low; + int ret; lockdep_assert_held(&msc->part_sel_lock); - idr_low = mpam_read_partsel_reg(msc, IDR); - if (FIELD_GET(MPAMF_IDR_EXT, idr_low)) - idr_high = mpam_read_partsel_reg(msc, IDR + 4); + ret = mpam_read_partsel_reg(msc, IDR, &idr_low); + if (ret) + return ret; - return (idr_high << 32) | idr_low; + if (FIELD_GET(MPAMF_IDR_EXT, idr_low)) { + ret = mpam_read_partsel_reg(msc, IDR + 4, &idr_high); + if (ret) + return ret; + } else { + idr_high = 0; + } + + *res = ((u64)idr_high << 32) | idr_low; + + return 0; } -static void mpam_msc_clear_esr(struct mpam_msc *msc) +static int mpam_msc_clear_esr(struct mpam_msc *msc) { - u64 esr_low = __mpam_read_reg(msc, MPAMF_ESR); + u32 esr_low; + int ret; + + ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low); + if (ret) + return ret; if (!esr_low) - return; + return 0; /* * Clearing the high/low bits of MPAMF_ESR can not be atomic. @@ -268,44 +401,59 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc) * lower half prevent hardware from updating either half of the * register. */ - if (msc->has_extd_esr) - __mpam_write_reg(msc, MPAMF_ESR + 4, 0); - __mpam_write_reg(msc, MPAMF_ESR, 0); + if (msc->has_extd_esr) { + ret = __mpam_write_reg(msc, MPAMF_ESR + 4, 0); + if (ret) + return ret; + } + + return __mpam_write_reg(msc, MPAMF_ESR, 0); } -static u64 mpam_msc_read_esr(struct mpam_msc *msc) +static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res) { - u64 esr_high = 0, esr_low; + u32 esr_high, esr_low; + int ret; - esr_low = __mpam_read_reg(msc, MPAMF_ESR); - if (msc->has_extd_esr) - esr_high = __mpam_read_reg(msc, MPAMF_ESR + 4); + ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low); + if (ret) + return ret; - return (esr_high << 32) | esr_low; + if (msc->has_extd_esr) { + ret = __mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high); + if (ret) + return ret; + } else { + esr_high = 0; + } + + *res = ((u64)esr_high << 32) | esr_low; + + return 0; } -static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc) +static int __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc) { lockdep_assert_held(&msc->part_sel_lock); - mpam_write_partsel_reg(msc, PART_SEL, partsel); + return mpam_write_partsel_reg(msc, PART_SEL, partsel); } -static void __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc) +static int __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc) { u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) | FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, partid); - __mpam_part_sel_raw(partsel, msc); + return __mpam_part_sel_raw(partsel, msc); } -static void __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc) +static int __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc) { u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) | FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, intpartid) | MPAMCFG_PART_SEL_INTERNAL; - __mpam_part_sel_raw(partsel, msc); + return __mpam_part_sel_raw(partsel, msc); } int mpam_register_requestor(u16 partid_max, u8 pmg_max) @@ -752,44 +900,62 @@ static void mpam_enable_quirks(struct mpam_msc *msc) * Try and see what values stick in this bit. If we can write either value, * its probably not implemented by hardware. */ -static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) +static int mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) { u32 now, mon_sel, ctl_val; bool can_set, can_clear; struct mpam_msc *msc = ris->vmsc->msc; + int ret; - if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) - return false; + ACQUIRE(mon_sel_lock, guard)(msc); + ret = ACQUIRE_ERR(mon_sel_lock, &guard); + if (ret) + return ret; mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) | FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx); - mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); + ret = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); + if (ret) + return ret; /* Hardware might ignore nrdy if it's not enabled */ ctl_val = MSMON_CFG_CSU_CTL_TYPE_CSU; ctl_val |= MSMON_CFG_x_CTL_MATCH_PARTID; ctl_val |= MSMON_CFG_x_CTL_MATCH_PMG; ctl_val |= MSMON_CFG_x_CTL_EN; - mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0); - mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val); - - _mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY); - now = _mpam_read_monsel_reg(msc, MSMON_CSU); + ret = mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0); + if (ret) + return ret; + ret = mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val); + if (ret) + return ret; + + ret = _mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY); + if (ret) + return ret; + ret = _mpam_read_monsel_reg(msc, MSMON_CSU, &now); + if (ret) + return ret; can_set = now & MSMON___NRDY; - _mpam_write_monsel_reg(msc, MSMON_CSU, 0); + ret = _mpam_write_monsel_reg(msc, MSMON_CSU, 0); + if (ret) + return ret; /* Configuration change to try and coax hardware into setting nrdy */ - mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1); - now = _mpam_read_monsel_reg(msc, MSMON_CSU); + ret = mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1); + if (ret) + return ret; + ret = _mpam_read_monsel_reg(msc, MSMON_CSU, &now); + if (ret) + return ret; can_clear = !(now & MSMON___NRDY); - mpam_mon_sel_unlock(msc); return (!can_set || !can_clear); } -static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) +static int mpam_ris_hw_probe(struct mpam_msc_ris *ris) { - int err; + int fw_has_nrdy_err, err; struct mpam_msc *msc = ris->vmsc->msc; struct device *dev = &msc->pdev->dev; struct mpam_props *props = &ris->props; @@ -800,7 +966,11 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Cache Capacity Partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) { - u32 ccap_features = mpam_read_partsel_reg(msc, CCAP_IDR); + u32 ccap_features; + + err = mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features); + if (err) + return err; props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features); if (props->cmax_wd && @@ -823,7 +993,11 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Cache Portion partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) { - u32 cpor_features = mpam_read_partsel_reg(msc, CPOR_IDR); + u32 cpor_features; + + err = mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features); + if (err) + return err; props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features); if (props->cpbm_wd) @@ -832,7 +1006,10 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Memory bandwidth partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) { - u32 mbw_features = mpam_read_partsel_reg(msc, MBW_IDR); + u32 mbw_features; + err = mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features); + if (err) + return err; /* portion bitmap resolution */ props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features); @@ -860,7 +1037,10 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Priority partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) { - u32 pri_features = mpam_read_partsel_reg(msc, PRI_IDR); + u32 pri_features; + err = mpam_read_partsel_reg(msc, PRI_IDR, &pri_features); + if (err) + return err; props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features); if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) { @@ -879,20 +1059,27 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Performance Monitoring */ if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) { - u32 msmon_features = mpam_read_partsel_reg(msc, MSMON_IDR); + u32 msmon_features; + + err = mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features); + if (err) + return err; /* * If the firmware max-nrdy-us property is missing, the * CSU counters can't be used. Should we wait forever? */ - err = device_property_read_u32(&msc->pdev->dev, - "arm,not-ready-us", - &msc->nrdy_usec); + fw_has_nrdy_err = device_property_read_u32(&msc->pdev->dev, + "arm,not-ready-us", + &msc->nrdy_usec); if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) { u32 csumonidr; - csumonidr = mpam_read_partsel_reg(msc, CSUMON_IDR); + err = mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr); + if (err) + return err; + props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr); if (props->num_csu_mon) { bool hw_managed; @@ -903,19 +1090,26 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) mpam_set_feature(mpam_feat_msmon_csu_xcl, props); /* Is NRDY hardware managed? */ - hw_managed = mpam_ris_hw_probe_csu_nrdy(ris); + err = mpam_ris_hw_probe_csu_nrdy(ris); + if (err < 0) + return err; + hw_managed = err; /* * Accept the missing firmware property if NRDY appears * un-implemented. */ - if (err && hw_managed) + if (fw_has_nrdy_err && hw_managed) dev_err_once(dev, "Counters are not usable because not-ready timeout was not provided by firmware."); } } if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_MBWU, msmon_features)) { bool has_long; - u32 mbwumon_idr = mpam_read_partsel_reg(msc, MBWUMON_IDR); + u32 mbwumon_idr; + + err = mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr); + if (err) + return err; props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr); if (props->num_mbwu_mon) { @@ -930,9 +1124,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) mpam_set_feature(mpam_feat_msmon_mbwu_63counter, props); else mpam_set_feature(mpam_feat_msmon_mbwu_44counter, props); - } else { - mpam_set_feature(mpam_feat_msmon_mbwu_31counter, props); } + + mpam_set_feature(mpam_feat_msmon_mbwu_31counter, props); } } } @@ -945,16 +1139,25 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) */ if (FIELD_GET(MPAMF_IDR_HAS_PARTID_NRW, ris->idr) && class->type != MPAM_CLASS_UNKNOWN) { - u32 nrwidr = mpam_read_partsel_reg(msc, PARTID_NRW_IDR); - u16 partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr); + u16 partid_max; + u32 nrwidr; + + err = mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr); + if (err) + return err; + + partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr); mpam_set_feature(mpam_feat_partid_nrw, props); msc->partid_max = min(msc->partid_max, partid_max); } + + return 0; } static int mpam_msc_hw_probe(struct mpam_msc *msc) { + int ret; u64 idr; u16 partid_max; u8 ris_idx, pmg_max; @@ -969,10 +1172,15 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc) } /* Grab an IDR value to find out how many RIS there are */ - mutex_lock(&msc->part_sel_lock); - idr = mpam_msc_read_idr(msc); - msc->iidr = mpam_read_partsel_reg(msc, IIDR); - mutex_unlock(&msc->part_sel_lock); + scoped_guard(mutex, &msc->part_sel_lock) { + ret = mpam_msc_read_idr(msc, &idr); + if (ret) + return ret; + + ret = mpam_read_partsel_reg(msc, IIDR, &msc->iidr); + if (ret) + return ret; + } mpam_enable_quirks(msc); @@ -983,10 +1191,15 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc) msc->pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr); for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) { - mutex_lock(&msc->part_sel_lock); - __mpam_part_sel(ris_idx, 0, msc); - idr = mpam_msc_read_idr(msc); - mutex_unlock(&msc->part_sel_lock); + scoped_guard(mutex, &msc->part_sel_lock) { + ret = __mpam_part_sel(ris_idx, 0, msc); + if (ret) + return ret; + + ret = mpam_msc_read_idr(msc, &idr); + if (ret) + return ret; + } partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr); pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr); @@ -994,21 +1207,28 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc) msc->pmg_max = min(msc->pmg_max, pmg_max); msc->has_extd_esr = FIELD_GET(MPAMF_IDR_HAS_EXTD_ESR, idr); - mutex_lock(&mpam_list_lock); - ris = mpam_get_or_create_ris(msc, ris_idx); - mutex_unlock(&mpam_list_lock); - if (IS_ERR(ris)) - return PTR_ERR(ris); + scoped_guard(mutex, &mpam_list_lock) { + ris = mpam_get_or_create_ris(msc, ris_idx); + if (IS_ERR(ris)) + return PTR_ERR(ris); + } ris->idr = idr; - mutex_lock(&msc->part_sel_lock); - __mpam_part_sel(ris_idx, 0, msc); - mpam_ris_hw_probe(ris); - mutex_unlock(&msc->part_sel_lock); + scoped_guard(mutex, &msc->part_sel_lock) { + ret = __mpam_part_sel(ris_idx, 0, msc); + if (ret) + return ret; + + ret = mpam_ris_hw_probe(ris); + if (ret) + return ret; + } } /* Clear any stale errors */ - mpam_msc_clear_esr(msc); + ret = mpam_msc_clear_esr(msc); + if (ret) + return ret; spin_lock(&partid_max_lock); mpam_partid_max = min(mpam_partid_max, msc->partid_max); @@ -1029,48 +1249,64 @@ struct mon_read { bool waited_timeout; }; -static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris) -{ - return (mpam_has_feature(mpam_feat_msmon_mbwu_63counter, &ris->props) || - mpam_has_feature(mpam_feat_msmon_mbwu_44counter, &ris->props)); -} - -static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc) +static int mpam_msc_read_mbwu_l(struct mpam_msc *msc, u64 *res) { + int ret; int retry = 3; u32 mbwu_l_low; - u64 mbwu_l_high1, mbwu_l_high2; + u32 mbwu_l_high1, mbwu_l_high2; mpam_mon_sel_lock_held(msc); - WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz); - WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility)); + if (msc->iface == MPAM_IFACE_MMIO) { + WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz); + WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &msc->accessibility)); + } + + ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2); + if (ret) + return ret; - mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4); do { mbwu_l_high1 = mbwu_l_high2; - mbwu_l_low = __mpam_read_reg(msc, MSMON_MBWU_L); - mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4); + ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low); + if (ret) + return ret; + ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2); + if (ret) + return ret; retry--; } while (mbwu_l_high1 != mbwu_l_high2 && retry > 0); - if (mbwu_l_high1 == mbwu_l_high2) - return (mbwu_l_high1 << 32) | mbwu_l_low; + if (mbwu_l_high1 == mbwu_l_high2) { + *res = ((u64)mbwu_l_high1 << 32) | mbwu_l_low; + } else { + pr_warn("Failed to read a stable value\n"); + *res = MSMON___L_NRDY; + } - pr_warn("Failed to read a stable value\n"); - return MSMON___L_NRDY; + return 0; } -static void mpam_msc_zero_mbwu_l(struct mpam_msc *msc) +static int mpam_msc_zero_mbwu_l(struct mpam_msc *msc) { + int ret; + mpam_mon_sel_lock_held(msc); - WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz); - WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility)); + if (msc->iface == MPAM_IFACE_MMIO) { + WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz); + WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &msc->accessibility)); + } - __mpam_write_reg(msc, MSMON_MBWU_L, 0); - __mpam_write_reg(msc, MSMON_MBWU_L + 4, 0); + ret = __mpam_write_reg(msc, MSMON_MBWU_L, 0); + if (ret) + return ret; + + return __mpam_write_reg(msc, MSMON_MBWU_L + 4, 0); } static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val, @@ -1113,24 +1349,28 @@ static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val, } } -static void read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val, - u32 *flt_val) +static int read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val, + u32 *flt_val) { struct mpam_msc *msc = m->ris->vmsc->msc; + int ret; switch (m->type) { case mpam_feat_msmon_csu: - *ctl_val = mpam_read_monsel_reg(msc, CFG_CSU_CTL); - *flt_val = mpam_read_monsel_reg(msc, CFG_CSU_FLT); - break; + ret = mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val); + if (ret) + return ret; + return mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val); case mpam_feat_msmon_mbwu_31counter: case mpam_feat_msmon_mbwu_44counter: case mpam_feat_msmon_mbwu_63counter: - *ctl_val = mpam_read_monsel_reg(msc, CFG_MBWU_CTL); - *flt_val = mpam_read_monsel_reg(msc, CFG_MBWU_FLT); - break; + ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val); + if (ret) + return ret; + return mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val); default: pr_warn("Unexpected monitor type %d\n", m->type); + return -EINVAL; } } @@ -1143,10 +1383,11 @@ static inline void clean_msmon_ctl_val(u32 *cur_ctl) *cur_ctl &= ~MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L; } -static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val, - u32 flt_val) +static int write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val, + u32 flt_val) { struct mpam_msc *msc = m->ris->vmsc->msc; + int ret; /* * Write the ctl_val with the enable bit cleared, reset the counter, @@ -1154,25 +1395,37 @@ static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val, */ switch (m->type) { case mpam_feat_msmon_csu: - mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val); - mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val); - mpam_write_monsel_reg(msc, CSU, 0); - mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN); - break; + ret = mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val); + if (ret) + return ret; + ret = mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val); + if (ret) + return ret; + ret = mpam_write_monsel_reg(msc, CSU, 0); + if (ret) + return ret; + return mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN); case mpam_feat_msmon_mbwu_31counter: case mpam_feat_msmon_mbwu_44counter: case mpam_feat_msmon_mbwu_63counter: - mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val); - mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val); - mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val | MSMON_CFG_x_CTL_EN); + ret = mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val); + if (ret) + return ret; + ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val); + if (ret) + return ret; + ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, + ctl_val | MSMON_CFG_x_CTL_EN); + if (ret) + return ret; /* Counting monitors require NRDY to be reset by software */ if (m->type == mpam_feat_msmon_mbwu_31counter) - mpam_write_monsel_reg(msc, MBWU, 0); - else - mpam_msc_zero_mbwu_l(m->ris->vmsc->msc); - break; + return mpam_write_monsel_reg(msc, MBWU, 0); + + return mpam_msc_zero_mbwu_l(m->ris->vmsc->msc); default: pr_warn("Unexpected monitor type %d\n", m->type); + return -EINVAL; } } @@ -1202,13 +1455,17 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type, return overflow_val; } -static void __ris_msmon_read(void *arg) +/* + * This function might be called via smp_call_function_any(), so propagate + * errors inside the arg struct. + */ +static void __ris_msmon_read_locked(struct mon_read *m) { u64 now; + u32 now32; bool nrdy = false; bool config_mismatch; bool overflow = false; - struct mon_read *m = arg; struct mon_cfg *ctx = m->ctx; bool reset_on_next_read = false; struct mpam_msc_ris *ris = m->ris; @@ -1216,13 +1473,13 @@ static void __ris_msmon_read(void *arg) struct mpam_msc *msc = m->ris->vmsc->msc; u32 mon_sel, ctl_val, flt_val, cur_ctl, cur_flt; - if (!mpam_mon_sel_lock(msc)) { - m->err = -EIO; - return; - } + mpam_mon_sel_lock_held(msc); + mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, ctx->mon) | FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx); - mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); + m->err = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); + if (m->err) + return; switch (m->type) { case mpam_feat_msmon_mbwu_31counter: @@ -1242,7 +1499,9 @@ static void __ris_msmon_read(void *arg) * Read the existing configuration to avoid re-writing the same values. * This saves waiting for 'nrdy' on subsequent reads. */ - read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt); + m->err = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt); + if (m->err) + return; if (mpam_feat_msmon_mbwu_31counter == m->type) overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS; @@ -1256,20 +1515,26 @@ static void __ris_msmon_read(void *arg) cur_ctl != (ctl_val | MSMON_CFG_x_CTL_EN); if (config_mismatch || reset_on_next_read) { - write_msmon_ctl_flt_vals(m, ctl_val, flt_val); + m->err = write_msmon_ctl_flt_vals(m, ctl_val, flt_val); + if (m->err) + return; overflow = false; } else if (overflow) { - mpam_write_monsel_reg(msc, CFG_MBWU_CTL, - cur_ctl & - ~(MSMON_CFG_x_CTL_OFLOW_STATUS | - MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L)); + m->err = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, + cur_ctl & + ~(MSMON_CFG_x_CTL_OFLOW_STATUS | + MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L)); + if (m->err) + return; } switch (m->type) { case mpam_feat_msmon_csu: - now = mpam_read_monsel_reg(msc, CSU); - nrdy = now & MSMON___NRDY; - now = FIELD_GET(MSMON___VALUE, now); + m->err = mpam_read_monsel_reg(msc, CSU, &now32); + if (m->err) + return; + nrdy = now32 & MSMON___NRDY; + now = FIELD_GET(MSMON___VALUE, now32); if (mpam_has_quirk(IGNORE_CSU_NRDY, msc) && m->waited_timeout) nrdy = false; @@ -1279,7 +1544,9 @@ static void __ris_msmon_read(void *arg) case mpam_feat_msmon_mbwu_44counter: case mpam_feat_msmon_mbwu_63counter: if (m->type != mpam_feat_msmon_mbwu_31counter) { - now = mpam_msc_read_mbwu_l(msc); + m->err = mpam_msc_read_mbwu_l(msc, &now); + if (m->err) + return; nrdy = now & MSMON___L_NRDY; if (m->type == mpam_feat_msmon_mbwu_63counter) @@ -1287,9 +1554,11 @@ static void __ris_msmon_read(void *arg) else now = FIELD_GET(MSMON___L_VALUE, now); } else { - now = mpam_read_monsel_reg(msc, MBWU); - nrdy = now & MSMON___NRDY; - now = FIELD_GET(MSMON___VALUE, now); + m->err = mpam_read_monsel_reg(msc, MBWU, &now32); + if (m->err) + return; + nrdy = now32 & MSMON___NRDY; + now = FIELD_GET(MSMON___VALUE, now32); } if (mpam_has_quirk(T241_MBW_COUNTER_SCALE_64, msc)) @@ -1312,7 +1581,6 @@ static void __ris_msmon_read(void *arg) default: m->err = -EINVAL; } - mpam_mon_sel_unlock(msc); if (nrdy) m->err = -EBUSY; @@ -1323,6 +1591,21 @@ static void __ris_msmon_read(void *arg) *m->val += now; } +static void __ris_msmon_read(void *arg) +{ + struct mon_read *m = arg; + struct mpam_msc *msc = m->ris->vmsc->msc; + + if (!mpam_mon_sel_lock(msc)) { + m->err = -EIO; + return; + } + + __ris_msmon_read_locked(m); + + mpam_mon_sel_unlock(msc); +} + static int _msmon_read(struct mpam_component *comp, struct mon_read *arg) { int err, any_err = 0; @@ -1338,11 +1621,16 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg) srcu_read_lock_held(&mpam_srcu)) { arg->ris = ris; - err = smp_call_function_any(&msc->accessibility, - __ris_msmon_read, arg, - true); - if (!err && arg->err) + if (msc->iface == MPAM_IFACE_MMIO) { + err = smp_call_function_any(&msc->accessibility, + __ris_msmon_read, + arg, true); + if (!err) + err = arg->err; + } else { + __ris_msmon_read(arg); err = arg->err; + } /* * Save one error to be returned to the caller, but @@ -1449,7 +1737,7 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx) } } -static void mpam_reset_msc_bitmap(struct mpam_msc *msc, u16 reg, u16 wd) +static int mpam_reset_msc_bitmap(struct mpam_msc *msc, u16 reg, u16 wd) { u32 num_words, msb; u32 bm = ~0; @@ -1458,15 +1746,20 @@ static void mpam_reset_msc_bitmap(struct mpam_msc *msc, u16 reg, u16 wd) lockdep_assert_held(&msc->part_sel_lock); if (wd == 0) - return; + return 0; /* * Write all ~0 to all but the last 32bit-word, which may * have fewer bits... */ num_words = DIV_ROUND_UP(wd, 32); - for (i = 0; i < num_words - 1; i++, reg += sizeof(bm)) - __mpam_write_reg(msc, reg, bm); + for (i = 0; i < num_words - 1; i++, reg += sizeof(bm)) { + int ret; + + ret = __mpam_write_reg(msc, reg, bm); + if (ret) + return ret; + } /* * ....and then the last (maybe) partial 32bit word. When wd is a @@ -1474,7 +1767,7 @@ static void mpam_reset_msc_bitmap(struct mpam_msc *msc, u16 reg, u16 wd) */ msb = (wd - 1) % 32; bm = GENMASK(msb, 0); - __mpam_write_reg(msc, reg, bm); + return __mpam_write_reg(msc, reg, bm); } static void mpam_apply_t241_erratum(struct mpam_msc_ris *ris, u16 partid) @@ -1547,40 +1840,59 @@ static u16 mpam_wa_t241_calc_min_from_max(struct mpam_props *props, } /* Called via IPI. Call while holding an SRCU reference */ -static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, - struct mpam_config *cfg) +static int mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, + struct mpam_config *cfg) { u16 cmax = MPAMCFG_CMAX_CMAX; struct mpam_msc *msc = ris->vmsc->msc; struct mpam_props *rprops = &ris->props; + int ret; - mutex_lock(&msc->part_sel_lock); - __mpam_part_sel(ris->ris_idx, partid, msc); + guard(mutex)(&msc->part_sel_lock); + ret = __mpam_part_sel(ris->ris_idx, partid, msc); + if (ret) + return ret; if (mpam_has_feature(mpam_feat_partid_nrw, rprops)) { /* Update the intpartid mapping */ - mpam_write_partsel_reg(msc, INTPARTID, - MPAMCFG_INTPARTID_INTERNAL | partid); + ret = mpam_write_partsel_reg(msc, INTPARTID, + MPAMCFG_INTPARTID_INTERNAL | partid); + if (ret) + return ret; /* * Then switch to the 'internal' partid to update the * configuration. */ - __mpam_intpart_sel(ris->ris_idx, partid, msc); + ret = __mpam_intpart_sel(ris->ris_idx, partid, msc); + if (ret) + return ret; } if (mpam_has_feature(mpam_feat_cpor_part, rprops)) { - if (mpam_has_feature(mpam_feat_cpor_part, cfg)) - mpam_write_partsel_reg(msc, CPBM, cfg->cpbm); - else - mpam_reset_msc_bitmap(msc, MPAMCFG_CPBM, rprops->cpbm_wd); + if (mpam_has_feature(mpam_feat_cpor_part, cfg)) { + ret = mpam_write_partsel_reg(msc, CPBM, cfg->cpbm); + if (ret) + return ret; + } else { + ret = mpam_reset_msc_bitmap(msc, MPAMCFG_CPBM, + rprops->cpbm_wd); + if (ret) + return ret; + } } if (mpam_has_feature(mpam_feat_mbw_part, rprops)) { - if (mpam_has_feature(mpam_feat_mbw_part, cfg)) - mpam_write_partsel_reg(msc, MBW_PBM, cfg->mbw_pbm); - else - mpam_reset_msc_bitmap(msc, MPAMCFG_MBW_PBM, rprops->mbw_pbm_bits); + if (mpam_has_feature(mpam_feat_mbw_part, cfg)) { + ret = mpam_write_partsel_reg(msc, MBW_PBM, cfg->mbw_pbm); + if (ret) + return ret; + } else { + ret = mpam_reset_msc_bitmap(msc, MPAMCFG_MBW_PBM, + rprops->mbw_pbm_bits); + if (ret) + return ret; + } } if (mpam_has_feature(mpam_feat_mbw_min, rprops)) { @@ -1593,27 +1905,47 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, val = max(val, min); } - mpam_write_partsel_reg(msc, MBW_MIN, val); + ret = mpam_write_partsel_reg(msc, MBW_MIN, val); + if (ret) + return ret; } if (mpam_has_feature(mpam_feat_mbw_max, rprops)) { - if (mpam_has_feature(mpam_feat_mbw_max, cfg)) - mpam_write_partsel_reg(msc, MBW_MAX, cfg->mbw_max); - else - mpam_write_partsel_reg(msc, MBW_MAX, MPAMCFG_MBW_MAX_MAX); + if (mpam_has_feature(mpam_feat_mbw_max, cfg)) { + ret = mpam_write_partsel_reg(msc, MBW_MAX, cfg->mbw_max); + if (ret) + return ret; + } else { + ret = mpam_write_partsel_reg(msc, MBW_MAX, + MPAMCFG_MBW_MAX_MAX); + if (ret) + return ret; + } } - if (mpam_has_feature(mpam_feat_mbw_prop, rprops)) - mpam_write_partsel_reg(msc, MBW_PROP, 0); + if (mpam_has_feature(mpam_feat_mbw_prop, rprops)) { + ret = mpam_write_partsel_reg(msc, MBW_PROP, 0); + if (ret) + return ret; + } - if (mpam_has_feature(mpam_feat_cmax_cmax, rprops)) - mpam_write_partsel_reg(msc, CMAX, cmax); + if (mpam_has_feature(mpam_feat_cmax_cmax, rprops)) { + ret = mpam_write_partsel_reg(msc, CMAX, cmax); + if (ret) + return ret; + } - if (mpam_has_feature(mpam_feat_cmax_cmin, rprops)) - mpam_write_partsel_reg(msc, CMIN, 0); + if (mpam_has_feature(mpam_feat_cmax_cmin, rprops)) { + ret = mpam_write_partsel_reg(msc, CMIN, 0); + if (ret) + return ret; + } - if (mpam_has_feature(mpam_feat_cmax_cassoc, rprops)) - mpam_write_partsel_reg(msc, CASSOC, MPAMCFG_CASSOC_CASSOC); + if (mpam_has_feature(mpam_feat_cmax_cassoc, rprops)) { + ret = mpam_write_partsel_reg(msc, CASSOC, MPAMCFG_CASSOC_CASSOC); + if (ret) + return ret; + } if (mpam_has_feature(mpam_feat_intpri_part, rprops) || mpam_has_feature(mpam_feat_dspri_part, rprops)) { @@ -1637,12 +1969,14 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, pri_val |= FIELD_PREP(MPAMCFG_PRI_DSPRI, dspri); } - mpam_write_partsel_reg(msc, PRI, pri_val); + ret = mpam_write_partsel_reg(msc, PRI, pri_val); + if (ret) + return ret; } mpam_quirk_post_config_change(ris, partid, cfg); - mutex_unlock(&msc->part_sel_lock); + return 0; } /* Call with msc cfg_lock held */ @@ -1650,19 +1984,39 @@ static int mpam_restore_mbwu_state(void *_ris) { int i; u64 val; - struct mon_read mwbu_arg; + struct mon_read mbwu_arg; struct mpam_msc_ris *ris = _ris; + struct msmon_mbwu_state *mbwu_state; + struct mpam_msc *msc = ris->vmsc->msc; struct mpam_class *class = ris->vmsc->comp->class; for (i = 0; i < ris->props.num_mbwu_mon; i++) { - if (ris->mbwu_state[i].enabled) { - mwbu_arg.ris = ris; - mwbu_arg.ctx = &ris->mbwu_state[i].cfg; - mwbu_arg.type = mpam_msmon_choose_counter(class); - mwbu_arg.val = &val; + if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) + return -EIO; - __ris_msmon_read(&mwbu_arg); + mbwu_state = &ris->mbwu_state[i]; + + if (!mbwu_state->enabled) { + mpam_mon_sel_unlock(msc); + continue; } + + val = 0; + mbwu_arg = (struct mon_read) { + .ris = ris, + .ctx = &mbwu_state->cfg, + .type = mpam_msmon_choose_counter(class), + .val = &val, + }; + + mbwu_state->reset_on_next_read = true; + + __ris_msmon_read_locked(&mbwu_arg); + + mpam_mon_sel_unlock(msc); + + if (mbwu_arg.err && mbwu_arg.err != -EBUSY) + return mbwu_arg.err; } return 0; @@ -1672,42 +2026,69 @@ static int mpam_restore_mbwu_state(void *_ris) static int mpam_save_mbwu_state(void *arg) { int i; + int ret; u64 val; struct mon_cfg *cfg; + struct mon_read mbwu_arg; u32 cur_flt, cur_ctl, mon_sel; struct mpam_msc_ris *ris = arg; struct msmon_mbwu_state *mbwu_state; struct mpam_msc *msc = ris->vmsc->msc; + struct mpam_class *class = ris->vmsc->comp->class; for (i = 0; i < ris->props.num_mbwu_mon; i++) { - mbwu_state = &ris->mbwu_state[i]; - cfg = &mbwu_state->cfg; - if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) return -EIO; + mbwu_state = &ris->mbwu_state[i]; + cfg = &mbwu_state->cfg; + mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) | FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx); - mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); - - cur_flt = mpam_read_monsel_reg(msc, CFG_MBWU_FLT); - cur_ctl = mpam_read_monsel_reg(msc, CFG_MBWU_CTL); - mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0); - - if (mpam_ris_has_mbwu_long_counter(ris)) { - val = mpam_msc_read_mbwu_l(msc); - mpam_msc_zero_mbwu_l(msc); - } else { - val = mpam_read_monsel_reg(msc, MBWU); - mpam_write_monsel_reg(msc, MBWU, 0); - } + ret = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel); + if (ret) + return ret; + ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt); + if (ret) + return ret; + ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl); + if (ret) + return ret; cfg->mon = i; cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt); cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl); cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt); - mbwu_state->correction += val; mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl); + + if (!mbwu_state->enabled) { + mpam_mon_sel_unlock(msc); + continue; + } + + val = 0; + mbwu_arg = (struct mon_read) { + .ris = ris, + .ctx = cfg, + .type = mpam_msmon_choose_counter(class), + .val = &val, + }; + + __ris_msmon_read_locked(&mbwu_arg); + + mbwu_state->reset_on_next_read = true; + if (!mbwu_arg.err) { + /* + * __ris_msmon_read_locked() already included the + * previous correction value. + */ + mbwu_state->correction = val; + } + + ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0); + if (ret) + return ret; + mpam_mon_sel_unlock(msc); } @@ -1720,9 +2101,10 @@ static int mpam_save_mbwu_state(void *arg) */ static int mpam_reset_ris(void *arg) { - u16 partid, partid_max; + u16 partid_max; struct mpam_config reset_cfg = {}; struct mpam_msc_ris *ris = arg; + int ret; if (ris->in_reset_state) return 0; @@ -1730,8 +2112,11 @@ static int mpam_reset_ris(void *arg) spin_lock(&partid_max_lock); partid_max = mpam_partid_max; spin_unlock(&partid_max_lock); - for (partid = 0; partid <= partid_max; partid++) - mpam_reprogram_ris_partid(ris, partid, &reset_cfg); + for (u32 partid = 0; partid <= partid_max; partid++) { + ret = mpam_reprogram_ris_partid(ris, partid, &reset_cfg); + if (ret) + return ret; + } return 0; } @@ -1753,6 +2138,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc) static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg) { + if (msc->iface != MPAM_IFACE_MMIO) + return fn(arg); + lockdep_assert_irqs_enabled(); lockdep_assert_cpus_held(); WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu))); @@ -1770,14 +2158,12 @@ static int __write_config(void *arg) { struct mpam_write_config_arg *c = arg; - mpam_reprogram_ris_partid(c->ris, c->partid, &c->comp->cfg[c->partid]); - - return 0; + return mpam_reprogram_ris_partid(c->ris, c->partid, + &c->comp->cfg[c->partid]); } static void mpam_reprogram_msc(struct mpam_msc *msc) { - u16 partid; bool reset; struct mpam_config *cfg; struct mpam_msc_ris *ris; @@ -1801,7 +2187,7 @@ static void mpam_reprogram_msc(struct mpam_msc *msc) arg.comp = ris->vmsc->comp; arg.ris = ris; reset = true; - for (partid = 0; partid <= mpam_partid_max; partid++) { + for (u32 partid = 0; partid <= mpam_partid_max; partid++) { cfg = &ris->vmsc->comp->cfg[partid]; if (!bitmap_empty(cfg->features, MPAM_FEATURE_LAST)) reset = false; @@ -1817,6 +2203,21 @@ static void mpam_reprogram_msc(struct mpam_msc *msc) mutex_unlock(&msc->cfg_lock); } +static int mpam_enable_msc_ecr(void *_msc) +{ + struct mpam_msc *msc = _msc; + + return __mpam_write_reg(msc, MPAMF_ECR, MPAMF_ECR_INTEN); +} + +/* This can run in mpam_disable(), and the interrupt handler on the same CPU */ +static int mpam_disable_msc_ecr(void *_msc) +{ + struct mpam_msc *msc = _msc; + + return __mpam_write_reg(msc, MPAMF_ECR, 0); +} + static void _enable_percpu_irq(void *_irq) { int *irq = _irq; @@ -1837,8 +2238,14 @@ static int mpam_cpu_online(unsigned int cpu) if (msc->reenable_error_ppi) _enable_percpu_irq(&msc->reenable_error_ppi); - if (atomic_fetch_inc(&msc->online_refs) == 0) + if (atomic_fetch_inc(&msc->online_refs) == 0) { + mutex_lock(&msc->error_irq_lock); + if (msc->error_irq_hw_enabled) + mpam_touch_msc(msc, mpam_enable_msc_ecr, msc); + mutex_unlock(&msc->error_irq_lock); + mpam_reprogram_msc(msc); + } } if (mpam_resctrl_enabled) @@ -2014,6 +2421,8 @@ static void mpam_msc_destroy(struct mpam_msc *msc) mpam_ris_destroy(ris); list_del_rcu(&msc->all_msc_list); + synchronize_srcu(&mpam_srcu); + mpam_pcc_chan_put(msc->pcc_chan); platform_set_drvdata(pdev, NULL); add_to_garbage(msc); @@ -2036,7 +2445,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev) static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) { int err; - u32 tmp; + u32 pcc_subspace_id; struct mpam_msc *msc; struct resource *msc_res; struct device *dev = &pdev->dev; @@ -2065,7 +2474,6 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) if (err) return ERR_PTR(err); - mpam_mon_sel_lock_init(msc); msc->id = pdev->id; msc->pdev = pdev; INIT_LIST_HEAD_RCU(&msc->all_msc_list); @@ -2081,11 +2489,16 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) if (err) return ERR_PTR(err); - if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp)) + if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id)) msc->iface = MPAM_IFACE_MMIO; else msc->iface = MPAM_IFACE_PCC; + /* Lock type depends on MSC interface used */ + err = mpam_mon_sel_lock_init(dev, msc); + if (err) + return ERR_PTR(err); + if (msc->iface == MPAM_IFACE_MMIO) { void __iomem *io; @@ -2097,6 +2510,33 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) } msc->mapped_hwpage_sz = msc_res->end - msc_res->start; msc->mapped_hwpage = io; + } else if (msc->iface == MPAM_IFACE_PCC) { + u32 mpam_fb_msc_id; + + msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id); + if (IS_ERR(msc->pcc_chan)) { + pr_err("Failed to request MSC PCC channel\n"); + return ERR_CAST(msc->pcc_chan); + } + + err = mpam_fb_check_shared_buffer_size(msc); + if (err) { + mpam_pcc_chan_put(msc->pcc_chan); + + return ERR_PTR(err); + } + + err = mpam_fb_check_protocol_version(msc); + if (err) { + mpam_pcc_chan_put(msc->pcc_chan); + + return ERR_PTR(err); + } + + if (device_property_read_u32(&pdev->dev, "mpam-fb-msc-id", + &mpam_fb_msc_id)) + mpam_fb_msc_id = msc->id; + msc->fb_id = mpam_fb_msc_id; } else { return ERR_PTR(-EINVAL); } @@ -2443,43 +2883,42 @@ static char *mpam_errcode_names[16] = { [12 ... 15] = "Reserved" }; -static int mpam_enable_msc_ecr(void *_msc) -{ - struct mpam_msc *msc = _msc; - - __mpam_write_reg(msc, MPAMF_ECR, MPAMF_ECR_INTEN); - - return 0; -} - -/* This can run in mpam_disable(), and the interrupt handler on the same CPU */ -static int mpam_disable_msc_ecr(void *_msc) -{ - struct mpam_msc *msc = _msc; - - __mpam_write_reg(msc, MPAMF_ECR, 0); - - return 0; -} - +/* + * This will run as the threaded IRQ handler part when using MPAM-Fb, but + * as the sole hard-IRQ handler for MMIO based accesses. + */ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) { u64 reg; + int ret; u16 partid; u8 errcode, pmg, ris; - if (WARN_ON_ONCE(!msc) || + if (WARN_ON_ONCE(!msc)) + return IRQ_NONE; + + if (msc->iface == MPAM_IFACE_MMIO && WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility))) return IRQ_NONE; - reg = mpam_msc_read_esr(msc); + ret = mpam_msc_read_esr(msc, ®); + if (ret) { + pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id); + + /* Try out best here ... */ + goto out_disable; + } errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg); if (!errcode) return IRQ_NONE; - /* Clear level triggered irq */ + /* + * Clear the level triggered IRQ. If that fails, we cannot do anything + * about it, so ignore any errors. We will disable the IRQ either on + * the device side or on the irqchip level next anyway. + */ mpam_msc_clear_esr(msc); partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg); @@ -2490,16 +2929,24 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) msc->id, mpam_errcode_names[errcode], partid, pmg, ris); - /* Disable this interrupt. */ - mpam_disable_msc_ecr(msc); +out_disable: + /* + * Disable this interrupt on the device side. If that fails, disable + * the IRQ on the irqchip level, as we must prevent further handler + * invocations. We only take the interrupt once anyway, as we + * are going to free the IRQ next, in mpam_disable(). + */ + ret = mpam_disable_msc_ecr(msc); + if (ret) + disable_irq_nosync(irq); - /* Are we racing with the thread disabling MPAM? */ + /* Check whether we are racing with the thread disabling MPAM. */ if (!mpam_is_enabled()) return IRQ_HANDLED; /* - * Schedule the teardown work. Don't use a threaded IRQ as we can't - * unregister the interrupt from the threaded part of the handler. + * Schedule the teardown work. We have to defer it as we can't + * unregister the interrupt from the threaded part of a handler. */ mpam_disable_reason = "hardware error interrupt"; schedule_work(&mpam_broken_work); @@ -2514,10 +2961,30 @@ static irqreturn_t mpam_ppi_handler(int irq, void *dev_id) return __mpam_irq_handler(irq, msc); } -static irqreturn_t mpam_spi_handler(int irq, void *dev_id) +/* + * MMIO based MSC accesses must run in non-preemptible context, as they + * might have affinity requirements to check. + * MPAM-Fb based MSC accesses must NOT run in hard-IRQ context, as they + * can sleep. + * So split the IRQ handling up, depending on the MSC access type. + */ +static irqreturn_t mpam_shared_hard_irq(int irq, void *dev_id) { struct mpam_msc *msc = dev_id; + if (msc->iface == MPAM_IFACE_MMIO) + return __mpam_irq_handler(irq, msc); + + return IRQ_WAKE_THREAD; +} + +static irqreturn_t mpam_shared_thread_irq(int irq, void *dev_id) +{ + struct mpam_msc *msc = dev_id; + + if (msc->iface == MPAM_IFACE_MMIO) + return IRQ_HANDLED; + return __mpam_irq_handler(irq, msc); } @@ -2538,6 +3005,11 @@ static int mpam_register_irqs(void) /* The MPAM spec says the interrupt can be SPI, PPI or LPI */ /* We anticipate sharing the interrupt with other MSCs */ if (irq_is_percpu(irq)) { + if (msc->iface != MPAM_IFACE_MMIO) { + dev_err(&msc->pdev->dev, + "Only MMIO MSCs can use per-CPU interrupts\n"); + return -EINVAL; + } err = request_percpu_irq(irq, &mpam_ppi_handler, "mpam:msc:error", msc->error_dev_id); @@ -2549,18 +3021,21 @@ static int mpam_register_irqs(void) &_enable_percpu_irq, &irq, true); } else { - err = devm_request_irq(&msc->pdev->dev, irq, - &mpam_spi_handler, IRQF_SHARED, - "mpam:msc:error", msc); + err = devm_request_threaded_irq(&msc->pdev->dev, irq, + &mpam_shared_hard_irq, + &mpam_shared_thread_irq, + IRQF_SHARED | IRQF_ONESHOT, + "mpam:msc:error", msc); if (err) return err; } - mutex_lock(&msc->error_irq_lock); + guard(mutex)(&msc->error_irq_lock); msc->error_irq_req = true; - mpam_touch_msc(msc, mpam_enable_msc_ecr, msc); + err = mpam_touch_msc(msc, mpam_enable_msc_ecr, msc); + if (err) + return err; msc->error_irq_hw_enabled = true; - mutex_unlock(&msc->error_irq_lock); } return 0; @@ -2581,8 +3056,8 @@ static void mpam_unregister_irqs(void) mutex_lock(&msc->error_irq_lock); if (msc->error_irq_hw_enabled) { - mpam_touch_msc(msc, mpam_disable_msc_ecr, msc); - msc->error_irq_hw_enabled = false; + if (!mpam_touch_msc(msc, mpam_disable_msc_ecr, msc)) + msc->error_irq_hw_enabled = false; } if (msc->error_irq_req) { @@ -2653,7 +3128,7 @@ static int __allocate_component_cfg(struct mpam_component *comp) if (comp->cfg) return 0; - comp->cfg = kzalloc_objs(*comp->cfg, mpam_partid_max + 1); + comp->cfg = kvzalloc_objs(*comp->cfg, mpam_partid_max + 1); if (!comp->cfg) return -ENOMEM; @@ -2774,7 +3249,7 @@ static void mpam_enable_once(void) mpam_partid_max + 1, mpam_pmg_max + 1); } -static void mpam_reset_component_locked(struct mpam_component *comp) +static int mpam_reset_component_locked(struct mpam_component *comp) { struct mpam_vmsc *vmsc; @@ -2788,26 +3263,38 @@ static void mpam_reset_component_locked(struct mpam_component *comp) srcu_read_lock_held(&mpam_srcu)) { struct mpam_msc *msc = vmsc->msc; struct mpam_msc_ris *ris; + int ret; list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list, srcu_read_lock_held(&mpam_srcu)) { - if (!ris->in_reset_state) - mpam_touch_msc(msc, mpam_reset_ris, ris); + if (!ris->in_reset_state) { + ret = mpam_touch_msc(msc, mpam_reset_ris, ris); + if (ret) + return ret; + } ris->in_reset_state = true; } } + + return 0; } -void mpam_reset_class_locked(struct mpam_class *class) +int mpam_reset_class_locked(struct mpam_class *class) { struct mpam_component *comp; + int ret; lockdep_assert_cpus_held(); guard(srcu)(&mpam_srcu); list_for_each_entry_srcu(comp, &class->components, class_list, - srcu_read_lock_held(&mpam_srcu)) - mpam_reset_component_locked(comp); + srcu_read_lock_held(&mpam_srcu)) { + ret = mpam_reset_component_locked(comp); + if (ret) + return ret; + } + + return 0; } static void mpam_reset_class(struct mpam_class *class) @@ -2866,7 +3353,12 @@ void mpam_disable(struct work_struct *ignored) mutex_unlock(&mpam_list_lock); mpam_free_garbage(); - pr_err_once("MPAM disabled due to %s\n", mpam_disable_reason); + if (mpam_disable_errno || mpam_disable_mpam_fb_err) + pr_err_once("MPAM disabled due to %s: %s, MPAM-Fb error %d\n", + mpam_disable_reason, errname(mpam_disable_errno), + mpam_disable_mpam_fb_err); + else + pr_err_once("MPAM disabled due to %s\n", mpam_disable_reason); } /* @@ -2926,6 +3418,7 @@ int mpam_apply_config(struct mpam_component *comp, u16 partid, struct mpam_msc_ris *ris; struct mpam_vmsc *vmsc; struct mpam_msc *msc; + int ret; lockdep_assert_cpus_held(); @@ -2943,14 +3436,15 @@ int mpam_apply_config(struct mpam_component *comp, u16 partid, srcu_read_lock_held(&mpam_srcu)) { msc = vmsc->msc; - mutex_lock(&msc->cfg_lock); + guard(mutex)(&msc->cfg_lock); list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list, srcu_read_lock_held(&mpam_srcu)) { arg.ris = ris; - mpam_touch_msc(msc, __write_config, &arg); + ret = mpam_touch_msc(msc, __write_config, &arg); + if (ret) + return ret; ris->in_reset_state = false; } - mutex_unlock(&msc->cfg_lock); } return 0; diff --git a/drivers/resctrl/mpam_fb.c b/drivers/resctrl/mpam_fb.c new file mode 100644 index 000000000000..54834052cbde --- /dev/null +++ b/drivers/resctrl/mpam_fb.c @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2024-2026 Arm Ltd. + +#include <linux/arm_mpam.h> +#include <linux/cleanup.h> +#include <linux/errno.h> +#include <linux/mailbox_client.h> +#include <linux/mutex.h> +#include <linux/platform_device.h> +#include <linux/types.h> + +#include <acpi/pcc.h> +#include <asm/mpam.h> + +#include "mpam_internal.h" + +#define MPAM_MSC_TOKEN_MASK GENMASK_U32(27, 18) +#define MPAM_MSC_PROT_ID_MASK GENMASK_U32(17, 10) + +#define MPAM_FB_PROTOCOL_ID 0x1a + +#define MPAM_PROTOCOL_VERSION_CMD 0x0 +#define MPAM_FB_VERSION_MAJOR_MASK GENMASK_U32(31, 16) +#define MPAM_FB_VERSION_MINOR_MASK GENMASK_U32(15, 0) + +#define MPAM_MSC_READ_CMD 0x4 +#define MPAM_MSC_WRITE_CMD 0x5 + +#define MPAM_FB_PROT_HEADER_LEN sizeof(u32) +/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */ +#define MPAM_FB_MAX_MSG_SIZE (4 * sizeof(u32)) + +#define MPAM_FB_SUCCESS 0 +#define MPAM_FB_ERR_NOT_SUPPORTED -1 +#define MPAM_FB_ERR_INVALID_PARAMETERS -2 +#define MPAM_FB_ERR_DENIED -3 +#define MPAM_FB_ERR_NOT_FOUND -4 +#define MPAM_FB_ERR_OUT_OF_RANGE -5 +#define MPAM_FB_ERR_BUSY -6 +#define MPAM_FB_ERR_COMMS_ERROR -7 +#define MPAM_FB_ERR_GENERIC_ERROR -8 +#define MPAM_FB_ERR_HW_ERROR -9 +#define MPAM_FB_ERR_PROTOCOL_ERROR -10 +#define MPAM_FB_ERR_IN_USE -11 + +static atomic_t mpam_fb_token = ATOMIC_INIT(0); + +static int mpam_fb_to_linux_errno(int mpam_fb_code) +{ + switch (mpam_fb_code) { + case MPAM_FB_ERR_NOT_SUPPORTED: + return -EOPNOTSUPP; + case MPAM_FB_ERR_INVALID_PARAMETERS: + return -EINVAL; + case MPAM_FB_ERR_NOT_FOUND: + return -ENOENT; + case MPAM_FB_ERR_OUT_OF_RANGE: + return -ERANGE; + case MPAM_FB_ERR_BUSY: + return -EBUSY; + default: + return -EINVAL; + } +} + +static void mpam_fb_build_version_message(unsigned int token, + void __iomem *msg_buf) +{ + struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf; + + /* .signature is filled by the platform */ + writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags); + writel_relaxed(MPAM_FB_PROT_HEADER_LEN, &pcc_shmem->length); + writel_relaxed(MPAM_PROTOCOL_VERSION_CMD | + FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) | + FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID), + &pcc_shmem->command); +} + +static void mpam_fb_build_read_message(int msc_fb_id, int reg, + unsigned int token, + void __iomem *msg_buf) +{ + struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf; + struct mpam_fb_read_payload { + u32 msc_fb_id; + u32 flags; + u32 reg_offset; + } __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem); + int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload); + + /* .signature is filled by the platform */ + writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags); + writel_relaxed(msg_size, &pcc_shmem->length); + writel_relaxed(MPAM_MSC_READ_CMD | + FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) | + FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID), + &pcc_shmem->command); + + writel_relaxed(msc_fb_id, &payload->msc_fb_id); + writel_relaxed(0, &payload->flags); + writel_relaxed(reg, &payload->reg_offset); +} + +static void mpam_fb_build_write_message(int msc_fb_id, int reg, u32 val, + unsigned int token, + void __iomem *msg_buf) +{ + struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf; + struct mpam_fb_write_payload { + u32 msc_fb_id; + u32 flags; + u32 reg_offset; + u32 value; + } __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem); + int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload); + + /* .signature is filled by the platform */ + writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags); + writel_relaxed(msg_size, &pcc_shmem->length); + writel_relaxed(MPAM_MSC_WRITE_CMD | + FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) | + FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID), + &pcc_shmem->command); + + writel_relaxed(msc_fb_id, &payload->msc_fb_id); + writel_relaxed(0, &payload->flags); + writel_relaxed(reg, &payload->reg_offset); + writel_relaxed(val, &payload->value); +} + +static int mpam_fb_send_request(struct mpam_msc *msc, u32 msc_fb_id, + u16 reg, u32 *result, int mpam_fb_command) +{ + unsigned int token = atomic_inc_return(&mpam_fb_token); + struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem; + struct mpam_pcc_chan *pcc_chan; + struct pcc_mbox_chan *chan; + void __iomem *payload_ofs; + int mpam_fb_err = 0; + u32 status; + int ret; + + pcc_chan = msc->pcc_chan; + if (!pcc_chan) + return -ENODEV; + + chan = pcc_chan->pcc_chan; + + /* prune token to fit into the 10 bits inside the command register */ + token = FIELD_GET(MPAM_MSC_TOKEN_MASK, + FIELD_PREP(MPAM_MSC_TOKEN_MASK, token)); + + mutex_lock(&pcc_chan->pcc_chan_lock); + + switch (mpam_fb_command) { + case MPAM_PROTOCOL_VERSION_CMD: + mpam_fb_build_version_message(token, chan->shmem); + break; + case MPAM_MSC_READ_CMD: + mpam_fb_build_read_message(msc_fb_id, reg, token, chan->shmem); + break; + case MPAM_MSC_WRITE_CMD: + mpam_fb_build_write_message(msc_fb_id, reg, *result, + token, chan->shmem); + break; + default: + dev_err(&msc->pdev->dev, "unsupported MPAM-Fb command %d\n", + mpam_fb_command); + ret = -EINVAL; + goto out_err; + } + + dma_wmb(); + ret = mbox_send_message(chan->mchan, NULL); + if (ret < 0) + goto out_err; + + pcc_shmem = chan->shmem; + payload_ofs = chan->shmem + sizeof(*pcc_shmem); + status = readl(&pcc_shmem->command); + if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) { + ret = -ETIMEDOUT; + goto out_err; + } + + mpam_fb_err = readl(payload_ofs + 0x0); + if (mpam_fb_err < 0) { + ret = mpam_fb_to_linux_errno(mpam_fb_err); + goto out_err; + } + + if (mpam_fb_command != MPAM_MSC_WRITE_CMD) + *result = readl(payload_ofs + 0x4); + + mutex_unlock(&pcc_chan->pcc_chan_lock); + + return 0; + +out_err: + mutex_unlock(&pcc_chan->pcc_chan_lock); + + mpam_fb_disable_mpam(ret, mpam_fb_err); + + /* + * TODO: Error codes are filtered in higher layers, return something + * that is known there for now, to actually report them. + */ + return -EIO; +} + +int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result) +{ + return mpam_fb_send_request(msc, msc->fb_id, reg, result, + MPAM_MSC_READ_CMD); +} + +int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value) +{ + return mpam_fb_send_request(msc, msc->fb_id, reg, &value, + MPAM_MSC_WRITE_CMD); +} + +/* We only support MPAM-Fb protocol version 1.x */ +int mpam_fb_check_protocol_version(struct mpam_msc *msc) +{ + u32 version; + int ret; + + ret = mpam_fb_send_request(msc, 0, 0, &version, + MPAM_PROTOCOL_VERSION_CMD); + if (ret) + return ret; + + if (FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, version) != 1) { + pr_err("Incompatible MPAM-Fb protocol version %d.%d\n", + FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, version), + FIELD_GET(MPAM_FB_VERSION_MINOR_MASK, version)); + + return -EINVAL; + } + + return 0; +} + +int mpam_fb_check_shared_buffer_size(struct mpam_msc *msc) +{ + int min_buffer_size = MPAM_FB_MAX_MSG_SIZE + + sizeof(struct acpi_pcct_ext_pcc_shared_memory); + + if (msc->pcc_chan->pcc_chan->shmem_size < min_buffer_size) { + pr_err("MPAM-Fb PCC channel size too small.\n"); + + return -ENOMEM; + } + + return 0; +} diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index def0e3a65c23..1f86cc158100 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -11,6 +11,7 @@ #include <linux/io.h> #include <linux/jump_label.h> #include <linux/llist.h> +#include <linux/mailbox_client.h> #include <linux/mutex.h> #include <linux/resctrl.h> #include <linux/spinlock.h> @@ -57,6 +58,15 @@ struct mpam_garbage { struct platform_device *pdev; }; +struct mpam_pcc_chan { + struct list_head pcc_chans; + struct mbox_client pcc_cl; + struct pcc_mbox_chan *pcc_chan; + struct mutex pcc_chan_lock; /* only one message at a time */ + struct kref refcount; + int subspace_id; +}; + struct mpam_msc { /* member of mpam_all_msc */ struct list_head all_msc_list; @@ -66,6 +76,8 @@ struct mpam_msc { /* Not modified after mpam_is_enabled() becomes true */ enum mpam_msc_iface iface; + struct mpam_pcc_chan *pcc_chan; + int fb_id; u32 nrdy_usec; cpumask_t accessibility; bool has_extd_esr; @@ -126,6 +138,12 @@ struct mpam_msc { */ raw_spinlock_t _mon_sel_lock; unsigned long _mon_sel_flags; + /* + * mon_sel_mutex is the mutex version of the _mon_sel_lock above. + * Always use the mpam_mon_sel_lock() helpers when taking the lock, + * as this will select the correct lock type automatically. + */ + struct mutex mon_sel_mutex; void __iomem *mapped_hwpage; size_t mapped_hwpage_sz; @@ -139,29 +157,57 @@ struct mpam_msc { /* Returning false here means accesses to mon_sel must fail and report an error. */ static inline bool __must_check mpam_mon_sel_lock(struct mpam_msc *msc) { - /* Locking will require updating to support a firmware backed interface */ - if (WARN_ON_ONCE(msc->iface != MPAM_IFACE_MMIO)) + if (msc->iface == MPAM_IFACE_MMIO) { + raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags); + + return true; + } + + if (!preemptible()) return false; - raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags); + mutex_lock(&msc->mon_sel_mutex); + return true; } static inline void mpam_mon_sel_unlock(struct mpam_msc *msc) { - raw_spin_unlock_irqrestore(&msc->_mon_sel_lock, msc->_mon_sel_flags); + if (msc->iface == MPAM_IFACE_MMIO) { + raw_spin_unlock_irqrestore(&msc->_mon_sel_lock, + msc->_mon_sel_flags); + + return; + } + + mutex_unlock(&msc->mon_sel_mutex); } static inline void mpam_mon_sel_lock_held(struct mpam_msc *msc) { - lockdep_assert_held_once(&msc->_mon_sel_lock); + if (msc->iface == MPAM_IFACE_MMIO) + lockdep_assert_held_once(&msc->_mon_sel_lock); + else + lockdep_assert_held_once(&msc->mon_sel_mutex); } -static inline void mpam_mon_sel_lock_init(struct mpam_msc *msc) +static inline int mpam_mon_sel_lock_init(struct device *dev, + struct mpam_msc *msc) { - raw_spin_lock_init(&msc->_mon_sel_lock); + if (msc->iface == MPAM_IFACE_MMIO) { + raw_spin_lock_init(&msc->_mon_sel_lock); + + return 0; + } + + return devm_mutex_init(dev, &msc->mon_sel_mutex); } +DEFINE_CLASS(mon_sel_lock, struct mpam_msc *, + if (_T) mpam_mon_sel_unlock(_T), + mpam_mon_sel_lock(msc) ? msc : NULL, struct mpam_msc *msc); +DEFINE_CLASS_IS_COND_GUARD(mon_sel_lock); + /* Bits for mpam features bitmaps */ enum mpam_device_features { mpam_feat_cpor_part, @@ -460,8 +506,11 @@ extern u8 mpam_pmg_max; void mpam_enable(struct work_struct *work); void mpam_disable(struct work_struct *work); +/* helper function to call from outside mpam_devices.c */ +void mpam_fb_disable_mpam(int err, int mpam_fb_err); + /* Reset all the RIS in a class under cpus_read_lock() */ -void mpam_reset_class_locked(struct mpam_class *class); +int mpam_reset_class_locked(struct mpam_class *class); int mpam_apply_config(struct mpam_component *comp, u16 partid, struct mpam_config *cfg); @@ -487,6 +536,12 @@ static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { } static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { } #endif /* CONFIG_RESCTRL_FS */ +/* MPAM-Fb Firmware-backed protocol wrappers */ +int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result); +int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value); +int mpam_fb_check_protocol_version(struct mpam_msc *msc); +int mpam_fb_check_shared_buffer_size(struct mpam_msc *msc); + /* * MPAM MSCs have the following register layout. See: * Arm Memory System Resource Partitioning and Monitoring (MPAM) System diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 9d223057953a..9cec5843e8fa 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -497,7 +497,7 @@ static int read_mon_cdp_safe(struct mpam_resctrl_mon *mon, struct mpam_component if (err) return err; - *val += code_val + data_val; + *val = code_val + data_val; return 0; } @@ -1076,7 +1076,7 @@ static void counter_update_class(enum resctrl_event_id evt_id, struct mpam_class *existing_class = mpam_resctrl_counters[evt_id].class; if (existing_class) { - if (class->level == 3) { + if (existing_class->level == 3) { pr_debug("Existing class is L3 - L3 wins\n"); return; } @@ -1943,10 +1943,6 @@ static void mpam_resctrl_teardown_mon(struct mpam_resctrl_mon *mon, struct mpam_ mon->mbwu_idx_to_mon = NULL; } -/* - * The driver is detaching an MSC from this class, if resctrl was using it, - * pull on resctrl_exit(). - */ void mpam_resctrl_teardown_class(struct mpam_class *class) { struct mpam_resctrl_res *res; @@ -1957,17 +1953,14 @@ void mpam_resctrl_teardown_class(struct mpam_class *class) might_sleep(); for_each_mpam_resctrl_control(res, rid) { - if (res->class == class) { + if (res->class == class) res->class = NULL; - break; - } } for_each_mpam_resctrl_mon(mon, eventid) { if (mon->class == class) { mon->class = NULL; mpam_resctrl_teardown_mon(mon, class); - break; } } } diff --git a/drivers/soc/tegra/Kconfig b/drivers/soc/tegra/Kconfig index 8b6fdb9ad1ea..318e7b604342 100644 --- a/drivers/soc/tegra/Kconfig +++ b/drivers/soc/tegra/Kconfig @@ -102,7 +102,6 @@ config ARCH_TEGRA_210_SOC config ARCH_TEGRA_186_SOC bool "NVIDIA Tegra186 SoC" default ARCH_TEGRA - depends on !CPU_BIG_ENDIAN select PINCTRL_TEGRA186 select MAILBOX select SOC_TEGRA_PMC @@ -117,7 +116,6 @@ config ARCH_TEGRA_186_SOC config ARCH_TEGRA_194_SOC bool "NVIDIA Tegra194 SoC" default ARCH_TEGRA - depends on !CPU_BIG_ENDIAN select MAILBOX select PINCTRL_TEGRA194 select SOC_TEGRA_PMC @@ -127,7 +125,6 @@ config ARCH_TEGRA_194_SOC config ARCH_TEGRA_234_SOC bool "NVIDIA Tegra234 SoC" default ARCH_TEGRA - depends on !CPU_BIG_ENDIAN select MAILBOX select PINCTRL_TEGRA234 select SOC_TEGRA_PMC @@ -137,7 +134,6 @@ config ARCH_TEGRA_234_SOC config ARCH_TEGRA_238_SOC bool "NVIDIA Tegra238 SoC" default ARCH_TEGRA - depends on !CPU_BIG_ENDIAN select MAILBOX select SOC_TEGRA_PMC help @@ -152,7 +148,6 @@ config ARCH_TEGRA_241_SOC config ARCH_TEGRA_264_SOC bool "NVIDIA Tegra264 SoC" default ARCH_TEGRA - depends on !CPU_BIG_ENDIAN select MAILBOX select SOC_TEGRA_PMC help diff --git a/drivers/virt/coco/arm-cca-guest/Kconfig b/drivers/virt/coco/arm-cca-guest/Kconfig index 3f0f013f03f1..0d4ce72e2d86 100644 --- a/drivers/virt/coco/arm-cca-guest/Kconfig +++ b/drivers/virt/coco/arm-cca-guest/Kconfig @@ -1,6 +1,7 @@ config ARM_CCA_GUEST tristate "Arm CCA Guest driver" - depends on ARM64 + depends on ARM_RMM_RSI + depends on HAVE_ARM_SMCCC_DISCOVERY select TSM_REPORTS help The driver provides userspace interface to request and diff --git a/drivers/virt/coco/arm-cca-guest/Makefile b/drivers/virt/coco/arm-cca-guest/Makefile index 69eeba08e98a..7564f4fcc4a4 100644 --- a/drivers/virt/coco/arm-cca-guest/Makefile +++ b/drivers/virt/coco/arm-cca-guest/Makefile @@ -1,2 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_ARM_CCA_GUEST) += arm-cca-guest.o + +arm-cca-guest-y += main.o diff --git a/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c b/drivers/virt/coco/arm-cca-guest/main.c index 0eeddd1ff05b..f97f593da6e3 100644 --- a/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c +++ b/drivers/virt/coco/arm-cca-guest/main.c @@ -3,7 +3,9 @@ * Copyright (C) 2023 ARM Ltd. */ -#include <linux/arm-smccc.h> +#include <linux/arm-rsi-cmds.h> +#include <linux/arm-smccc-bus.h> +#include <linux/arm-smccc-rsi.h> #include <linux/cc_platform.h> #include <linux/kernel.h> #include <linux/device-id/platform.h> @@ -12,8 +14,6 @@ #include <linux/tsm.h> #include <linux/types.h> -#include <asm/rsi.h> - /** * struct arm_cca_token_info - a descriptor for the token buffer. * @granule: PA of the granule to which the token will be written @@ -157,52 +157,44 @@ exit_migrate_enable: return ret; } -static const struct tsm_report_ops arm_cca_tsm_ops = { +static const struct tsm_report_ops arm_cca_tsm_report_ops = { .name = KBUILD_MODNAME, .report_new = arm_cca_report_new, }; -/** - * arm_cca_guest_init - Register with the Trusted Security Module (TSM) - * interface. - * - * Return: - * * %0 - Registered successfully with the TSM interface. - * * %-ENODEV - The execution context is not an Arm Realm. - * * %-EBUSY - Already registered. - */ -static int __init arm_cca_guest_init(void) +static int cca_tsm_probe(struct arm_smccc_device *sdev) { int ret; if (!is_realm_world()) return -ENODEV; - ret = tsm_report_register(&arm_cca_tsm_ops, NULL); + ret = tsm_report_register(&arm_cca_tsm_report_ops, NULL); if (ret < 0) - pr_err("Error %d registering with TSM\n", ret); + return dev_err_probe(&sdev->dev, + ret, "Error registering with TSM\n"); - return ret; + return 0; } -module_init(arm_cca_guest_init); -/** - * arm_cca_guest_exit - unregister with the Trusted Security Module (TSM) - * interface. - */ -static void __exit arm_cca_guest_exit(void) +static void cca_tsm_remove(struct arm_smccc_device *sdev) { - tsm_report_unregister(&arm_cca_tsm_ops); + tsm_report_unregister(&arm_cca_tsm_report_ops); } -module_exit(arm_cca_guest_exit); -/* modalias, so userspace can autoload this module when RSI is available */ -static const struct platform_device_id arm_cca_match[] __maybe_unused = { - { .name = RSI_PDEV_NAME }, - { } +static const struct arm_smccc_device_id cca_tsm_id_table[] = { + { .func_id = SMC_RSI_ABI_VERSION }, + {} }; +MODULE_DEVICE_TABLE(arm_smccc, cca_tsm_id_table); -MODULE_DEVICE_TABLE(platform, arm_cca_match); +static struct arm_smccc_driver cca_tsm_driver = { + .name = KBUILD_MODNAME, + .probe = cca_tsm_probe, + .remove = cca_tsm_remove, + .id_table = cca_tsm_id_table, +}; +module_arm_smccc_driver(cca_tsm_driver); MODULE_AUTHOR("Sami Mujawar <sami.mujawar@arm.com>"); MODULE_DESCRIPTION("Arm CCA Guest TSM Driver"); MODULE_LICENSE("GPL"); diff --git a/include/linux/arm-rmi-cmds.h b/include/linux/arm-rmi-cmds.h new file mode 100644 index 000000000000..a84e50ac851c --- /dev/null +++ b/include/linux/arm-rmi-cmds.h @@ -0,0 +1,581 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2026 ARM Ltd. + */ + +#ifndef __LINUX_ARM_RMI_CMDS_H_ +#define __LINUX_ARM_RMI_CMDS_H_ + +#include <linux/arm-smccc-rmi.h> +#include <linux/bug.h> +#include <linux/gfp.h> +#include <linux/processor.h> +#include <linux/string.h> +#include <linux/types.h> + +#define RMI_MAX_ADDR_LIST 256 + +struct rmi_sro_state { + struct arm_smccc_1_2_regs regs; + int addr_count; + unsigned long addr_list[RMI_MAX_ADDR_LIST]; +}; + +#define RMM_BLOCKED_RETRY_COUNT 2 +/* + * rmi_smccc_invoke: Invoke the RMI call and return the results, retrying the + * command when status is RMI_BUSY. If we encounter RMI_BLOCKED, we retry + * it one more time before we give up. The caller is supposed to handle the + * result and reissue if required. + * + * We don't expect to see RMI_BLOCKED on a practical system, except when + * there are parallel requests that results in long standing operation, + * with one blocking the other. + * + * @regs: Input parameters filled in. Updated with the output results + * after the call. + */ +static inline void rmi_smccc_invoke(struct arm_smccc_1_2_regs *regs) +{ + struct arm_smccc_1_2_regs args = *regs; + long status; + int i = 0; + + while (i < RMM_BLOCKED_RETRY_COUNT) { + arm_smccc_1_2_smc(&args, regs); + + status = RMI_RESULT_STATUS(regs->a0); + if (status != RMI_BUSY && status != RMI_BLOCKED) + break; + if (status == RMI_BLOCKED) + i++; + cpu_relax(); + } +} + +/* + * rmi_smccc_invoke_once: Invoke the RMI call and return the results. Do not + * retry the command. Let the caller deal with RMI_BUSY or RMI_BLOCKED. + * + * @regs: Input parameters filled in. Updated with the output results + * after the call. + */ +static __always_inline void rmi_smccc_invoke_once(struct arm_smccc_1_2_regs *regs) +{ + arm_smccc_1_2_smc(regs, regs); +} + +unsigned long rmi_feat_reg(unsigned int index); + +int rmi_delegate_range(phys_addr_t phys, unsigned long size, + phys_addr_t *out_phys); +int rmi_undelegate_range(phys_addr_t phys, unsigned long size); +int rmi_free_delegated_page(phys_addr_t phys); + +static inline int rmi_delegate_page(phys_addr_t phys) +{ + return rmi_delegate_range(phys, PAGE_SIZE, NULL); +} + +static inline int rmi_undelegate_page(phys_addr_t phys) +{ + return rmi_undelegate_range(phys, PAGE_SIZE); +} + +long rmi_sro_memxfer_execute(struct rmi_sro_state *sro, gfp_t gfp); +void rmi_sro_free(struct rmi_sro_state *sro); +long rmi_sro_execute(struct arm_smccc_1_2_regs *regs); + +/* + * Resetting the addr_count is sufficient to ignore the addr_list contents. + */ +#define rmi_sro_memxfer_cmd(sro, gfp, ...) ({ \ + struct rmi_sro_state *__sro = (sro); \ + __sro->addr_count = 0; \ + __sro->regs = (struct arm_smccc_1_2_regs){ __VA_ARGS__ }; \ + long __ret = rmi_sro_memxfer_execute(__sro, gfp); \ + rmi_sro_free(__sro); \ + __ret; \ +}) + +#ifdef CONFIG_ARM_RMM_RMI + +bool is_rmm_active(void); +bool is_rmi_available(void); + +#else + +static inline bool is_rmm_active(void) +{ + return false; +} + +static inline bool is_rmi_available(void) +{ + return false; +} +#endif /* CONFIG_ARM_RMM_RMI */ + +/** + * rmi_rtt_data_map_init() - Create a mapping at protected IPA, copying contents + * from a given non-secure source granule. + * @rd: PA of the RD + * @data: PA of the target granule mapped in the guest + * @ipa: IPA at which the granule @data will be mapped in the guest + * @src: PA of the source granule with contents + * @flags: RMI_MEASURE_CONTENT if the contents should be measured + * + * Create a mapping from Protected IPA space to conventional memory, copying + * contents from a Non-secure Granule provided by the caller. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_data_map_init(unsigned long rd, unsigned long data, + unsigned long ipa, unsigned long src, + unsigned long flags) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_DATA_MAP_INIT, rd, data, ipa, src, flags + }; + + return rmi_sro_execute(®s); +} + +/** + * rmi_rtt_data_map() - Create mappings in protected IPA range with unknown contents + * @rd: PA of the RD + * @base: Base of the target IPA range + * @top: Top of the target IPA range + * @flags: Flags + * @oaddr: Output address set descriptor + * @out_top: Top address of range which was processed. + * + * Return: 0 on success, positive RMI result code or negative Linux error code. + * NOTE: out_top is only valid with ret == RMI_SUCCESS + */ +static inline long rmi_rtt_data_map(unsigned long rd, + unsigned long base, + unsigned long top, + unsigned long flags, + unsigned long oaddr, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_DATA_MAP, rd, base, top, flags, oaddr + }; + long ret; + + ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +/** + * rmi_rtt_data_unmap() - Remove mappings to conventional memory at a protected + * IPA range + * @rd: PA of the RD + * @base: Base of the target IPA range + * @top: Top of the target IPA range + * @flags: Flags + * @oaddr: Output address set descriptor + * @out_top: Returns top IPA of range which has been unmapped + * @out_range: Output address range + * @out_count: Number of entries in output address list + * + * Removes mappings to convention memory with a target Protected IPA range. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + * NOTE: out_top is only valid with ret == RMI_SUCCESS + */ +static inline long rmi_rtt_data_unmap(unsigned long rd, + unsigned long base, + unsigned long top, + unsigned long flags, + unsigned long oaddr, + unsigned long *out_top, + unsigned long *out_range, + unsigned long *out_count) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_DATA_UNMAP, rd, base, top, flags, oaddr + }; + long ret; + + ret = rmi_sro_execute(®s); + + if (ret != RMI_SUCCESS) + return ret; + + if (out_top) + *out_top = regs.a1; + if (out_range) + *out_range = regs.a2; + if (out_count) + *out_count = regs.a3; + + return RMI_SUCCESS; +} + +/** + * rmi_psci_complete() - Complete pending PSCI command + * @calling_rec: PA of the calling REC + * @status: Status of the PSCI request + * + * Completes a pending PSCI command. + * + * Return: RMI return code + */ +static inline long rmi_psci_complete(unsigned long calling_rec, + unsigned long status) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_PSCI_COMPLETE, calling_rec, status, + }; + + rmi_smccc_invoke(®s); + return regs.a0; +} + +/** + * rmi_realm_activate() - Activate a realm + * @rd: PA of the RD + * + * Mark a realm as Active, signalling that creation is completed, allowing + * execution of the realm. + * + * Return: RMI return code + */ +static inline long rmi_realm_activate(unsigned long rd) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_REALM_ACTIVATE, rd, + }; + + rmi_smccc_invoke(®s); + return regs.a0; +} + +/** + * rmi_realm_create() - Create a realm + * @rd: PA of the RD + * @params: PA of realm parameters + * @sro: Preallocated SRO context + * + * Create a new realm using the given parameters. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_realm_create(unsigned long rd, unsigned long params, + struct rmi_sro_state *sro) +{ + return rmi_sro_memxfer_cmd(sro, GFP_KERNEL, + SMC_RMI_REALM_CREATE, rd, params); +} + +/** + * rmi_realm_terminate() - Terminate a realm + * @rd: PA of the RD + * @sro: Preallocated SRO context + * + * Terminates a realm, moving it into a ZOMBIE state + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_realm_terminate(unsigned long rd, + struct rmi_sro_state *sro) +{ + return rmi_sro_memxfer_cmd(sro, GFP_KERNEL, + SMC_RMI_REALM_TERMINATE, rd); +} + +/** + * rmi_realm_destroy() - Destroy a realm + * @rd: PA of the RD + * @sro: Preallocated SRO context + * + * Destroys a realm, all objects belonging to the realm must be destroyed first. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_realm_destroy(unsigned long rd, + struct rmi_sro_state *sro) +{ + return rmi_sro_memxfer_cmd(sro, GFP_KERNEL, + SMC_RMI_REALM_DESTROY, rd); +} + +/** + * rmi_rec_create() - Create a REC + * @rd: PA of the RD + * @rec: PA of the target REC + * @params: PA of REC parameters + * @sro: Allocated SRO context to be used + * + * Create a REC using the parameters specified in the struct rec_params pointed + * to by @params. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rec_create(unsigned long rd, + unsigned long rec, + unsigned long params, + struct rmi_sro_state *sro) +{ + return rmi_sro_memxfer_cmd(sro, GFP_KERNEL, + SMC_RMI_REC_CREATE, rd, rec, params); +} + +/** + * rmi_rec_destroy() - Destroy a REC + * @rec: PA of the target REC + * @sro: Allocated SRO context to be used + * + * Destroys a REC. The REC must not be running. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static __always_inline long rmi_rec_destroy(unsigned long rec, + struct rmi_sro_state *sro) +{ + return rmi_sro_memxfer_cmd(sro, GFP_KERNEL, SMC_RMI_REC_DESTROY, rec); +} + +/** + * rmi_rec_enter() - Enter a REC + * @rec: PA of the target REC + * @run_ptr: PA of RecRun structure + * + * Starts (or continues) execution within a REC. + * + * Return: RMI result + */ +static __always_inline long rmi_rec_enter(unsigned long rec, unsigned long run_ptr) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_REC_ENTER, rec, run_ptr, + }; + + rmi_smccc_invoke_once(®s); + return regs.a0; +} + +/** + * rmi_rtt_create() - Creates an RTT + * @rd: PA of the RD + * @rtt: PA of the target RTT + * @ipa: Base of the IPA range described by the RTT + * @level: Depth of the RTT within the tree + * + * Creates an RTT (Realm Translation Table) at the specified level for the + * translation of the specified address within the realm. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_create(unsigned long rd, unsigned long rtt, + unsigned long ipa, long level) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_CREATE, rd, rtt, ipa, level + }; + + return rmi_sro_execute(®s); +} + +/** + * rmi_rtt_destroy() - Destroy an RTT + * @rd: PA of the RD + * @ipa: Base of the IPA range described by the RTT + * @level: RTT level + * @out_rtt: Pointer to write the PA of the RTT which was destroyed + * @out_top: Pointer to write the top IPA of non-live RTT entries, from entry + * at which the RTT walk terminated. + * + * Destroys an RTT. The RTT must be non-live, i.e. none of the entries in the + * table are in ASSIGNED or TABLE state. + * + * Return: 0 on success, positive RMI result code or negative Linux error code. + */ +static inline long rmi_rtt_destroy(unsigned long rd, + unsigned long ipa, + long level, + unsigned long *out_rtt, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_DESTROY, rd, ipa, level + }; + long ret = rmi_sro_execute(®s); + + switch (RMI_RESULT_STATUS(ret)) { + case RMI_SUCCESS: + if (out_rtt) + *out_rtt = regs.a1; + fallthrough; + case RMI_ERROR_RTT: + if (out_top) + *out_top = regs.a2; + break; + default: + break; + } + + return ret; +} + +/** + * rmi_rtt_fold() - Fold an RTT + * @rd: PA of the RD + * @ipa: Base of the IPA range described by the RTT + * @level: Depth of the RTT within the tree + * @out_rtt: Pointer to write the PA of the RTT which was destroyed + * + * Folds an RTT. If all entries with the RTT are 'homogeneous' the RTT can be + * folded into the parent and the RTT destroyed. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_fold(unsigned long rd, unsigned long ipa, + long level, unsigned long *out_rtt) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_FOLD, rd, ipa, level + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_rtt) + *out_rtt = regs.a1; + + return ret; +} + +/** + * rmi_rtt_init_ripas() - Set RIPAS for new realm + * @rd: PA of the RD + * @base: Base of target IPA region + * @top: Top of target IPA region + * @out_top: Top IPA of range whose RIPAS was modified + * + * Sets the RIPAS of a target IPA range to RAM, for a realm in the NEW state. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_init_ripas(unsigned long rd, unsigned long base, + unsigned long top, unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_INIT_RIPAS, rd, base, top + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +/** + * rmi_rtt_unprot_map() - Map unprotected granules into a realm + * @rd: PA of the RD + * @base: Base IPA of the mapping + * @top: Top of the target IPA range + * @flags: Flags + * @oaddr: Output address set descriptor + * @out_top: Top IPA of range which has been mapped + * + * Create mappings to memory within a target unprotected IPA range. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_unprot_map(unsigned long rd, + unsigned long base, + unsigned long top, + unsigned long flags, + unsigned long oaddr, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_UNPROT_MAP, rd, base, top, flags, oaddr + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +/** + * rmi_rtt_set_ripas() - Set RIPAS for an running realm + * @rd: PA of the RD + * @rec: PA of the REC making the request + * @base: Base of target IPA region + * @top: Top of target IPA region + * @out_top: Pointer to write top IPA of range whose RIPAS was modified + * + * Completes a request made by the realm to change the RIPAS of a target IPA + * range. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + */ +static inline long rmi_rtt_set_ripas(unsigned long rd, unsigned long rec, + unsigned long base, unsigned long top, + unsigned long *out_top) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_SET_RIPAS, rd, rec, base, top + }; + long ret = rmi_sro_execute(®s); + + if (ret == RMI_SUCCESS && out_top) + *out_top = regs.a1; + + return ret; +} + +/** + * rmi_rtt_unprot_unmap() - Remove mappings within an unprotected IPA range + * @rd: PA of the RD + * @base: Base IPA of the mapping + * @top: Top of the target IPA range + * @flags: Flags + * @oaddr: Output address set descriptor + * @out_top: Top IPA which has been unmapped + * @out_range: Output address range + * @out_count: Number of entries in output address list + * + * Removes mappings to memory within a target unprotected IPA range. + * + * Return: 0 on success, positive RMI result code or negative Linux error code + * NOTE: out_top is only valid with ret == RMI_SUCCESS. + */ +static inline long rmi_rtt_unprot_unmap(unsigned long rd, + unsigned long base, + unsigned long top, + unsigned long flags, + unsigned long oaddr, + unsigned long *out_top, + unsigned long *out_range, + unsigned long *out_count) +{ + struct arm_smccc_1_2_regs regs = { + SMC_RMI_RTT_UNPROT_UNMAP, rd, base, top, flags, oaddr + }; + long ret = rmi_sro_execute(®s); + + if (ret != RMI_SUCCESS) + return ret; + + if (out_top) + *out_top = regs.a1; + if (out_range) + *out_range = regs.a2; + if (out_count) + *out_count = regs.a3; + + return RMI_SUCCESS; +} + +#endif /* __LINUX_ARM_RMI_CMDS_H_ */ diff --git a/arch/arm64/include/asm/rsi_cmds.h b/include/linux/arm-rsi-cmds.h index c1fab41f671e..3f7a6a833993 100644 --- a/arch/arm64/include/asm/rsi_cmds.h +++ b/include/linux/arm-rsi-cmds.h @@ -3,14 +3,35 @@ * Copyright (C) 2023 ARM Ltd. */ -#ifndef __ASM_RSI_CMDS_H -#define __ASM_RSI_CMDS_H +#ifndef __LINUX_ARM_RSI_CMDS_H_ +#define __LINUX_ARM_RSI_CMDS_H_ -#include <linux/arm-smccc.h> +#include <linux/arm-smccc-rsi.h> +#include <linux/jump_label.h> #include <linux/string.h> #include <asm/memory.h> -#include <asm/rsi_smc.h> +#ifdef CONFIG_ARM_RMM_RSI +DECLARE_STATIC_KEY_FALSE(rsi_present); + +void __init arm64_rsi_init(void); + +bool arm64_rsi_is_protected(phys_addr_t base, size_t size); + +static inline bool is_realm_world(void) +{ + return static_branch_unlikely(&rsi_present); +} +#else +static inline void arm64_rsi_init(void) { } + +static inline bool arm64_rsi_is_protected(phys_addr_t base, size_t size) +{ + return false; +} + +static inline bool is_realm_world(void) { return false; } +#endif #define RSI_GRANULE_SHIFT 12 #define RSI_GRANULE_SIZE (_AC(1, UL) << RSI_GRANULE_SHIFT) @@ -88,6 +109,51 @@ static inline long rsi_set_addr_range_state(phys_addr_t start, return res.a0; } +static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end, + enum ripas state, unsigned long flags) +{ + unsigned long ret; + phys_addr_t top; + + while (start != end) { + ret = rsi_set_addr_range_state(start, end, state, flags, &top); + if (ret || top < start || top > end) + return -EINVAL; + start = top; + } + + return 0; +} + +/* + * Convert the specified range to RAM. Do not use this if you rely on the + * contents of a page that may already be in RAM state. + */ +static inline int rsi_set_memory_range_protected(phys_addr_t start, + phys_addr_t end) +{ + return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, + RSI_CHANGE_DESTROYED); +} + +/* + * Convert the specified range to RAM. Do not convert any pages that may have + * been DESTROYED, without our permission. + */ +static inline int rsi_set_memory_range_protected_safe(phys_addr_t start, + phys_addr_t end) +{ + return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, + RSI_NO_CHANGE_DESTROYED); +} + +static inline int rsi_set_memory_range_shared(phys_addr_t start, + phys_addr_t end) +{ + return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY, + RSI_CHANGE_DESTROYED); +} + #define RSI_ATTEST_CHALLENGE_MIN_SIZE 32 #define RSI_ATTEST_CHALLENGE_MAX_SIZE 64 @@ -170,4 +236,4 @@ static inline unsigned long rsi_attestation_token_continue(phys_addr_t granule, return res.a0; } -#endif /* __ASM_RSI_CMDS_H */ +#endif /* __LINUX_ARM_RSI_CMDS_H_ */ diff --git a/include/linux/arm-smccc-bus.h b/include/linux/arm-smccc-bus.h new file mode 100644 index 000000000000..05c0c850510e --- /dev/null +++ b/include/linux/arm-smccc-bus.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 Arm Limited + */ +#ifndef __LINUX_ARM_SMCCC_BUS_H +#define __LINUX_ARM_SMCCC_BUS_H + +#include <linux/device.h> +#include <linux/device-id/arm_smccc.h> +#include <linux/module.h> + +struct arm_smccc_device { + struct device dev; + u32 func_id; +}; + +#define to_arm_smccc_device(d) container_of_const(d, struct arm_smccc_device, dev) + +struct arm_smccc_driver { + struct device_driver driver; + + const char *name; + int (*probe)(struct arm_smccc_device *sdev); + void (*remove)(struct arm_smccc_device *sdev); + const struct arm_smccc_device_id *id_table; +}; + +#define to_arm_smccc_driver(d) \ + container_of_const(d, struct arm_smccc_driver, driver) + +int arm_smccc_driver_register(struct arm_smccc_driver *driver, + struct module *owner, const char *mod_name); +void arm_smccc_driver_unregister(struct arm_smccc_driver *driver); +struct arm_smccc_device *arm_smccc_device_register(const char *name, u32 func_id); +void arm_smccc_device_unregister(struct arm_smccc_device *smcc_dev); + +#define arm_smccc_register(driver) \ + arm_smccc_driver_register(driver, THIS_MODULE, KBUILD_MODNAME) +#define arm_smccc_unregister(driver) \ + arm_smccc_driver_unregister(driver) + +#define module_arm_smccc_driver(__arm_smccc_driver) \ + module_driver(__arm_smccc_driver, arm_smccc_register, \ + arm_smccc_unregister) + +extern const struct bus_type arm_smccc_bus_type; + +#endif /* __LINUX_ARM_SMCCC_BUS_H */ diff --git a/include/linux/arm-smccc-rmi.h b/include/linux/arm-smccc-rmi.h new file mode 100644 index 000000000000..655726b37462 --- /dev/null +++ b/include/linux/arm-smccc-rmi.h @@ -0,0 +1,516 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023-2026 ARM Ltd. + * + * The values and structures in this file are from the Realm Management Monitor + * specification (DEN0137) version 2.0-bet3: + * https://developer.arm.com/documentation/den0137/2-0bet3/ + */ + +#ifndef __LINUX_ARM_SMCCC_RMI_H_ +#define __LINUX_ARM_SMCCC_RMI_H_ + +#include <linux/arm-smccc.h> +#include <linux/bitfield.h> +#include <linux/bits.h> +#include <linux/build_bug.h> +#include <linux/sizes.h> +#include <linux/stddef.h> + +#include <asm/page.h> + +#define SMC_RMI_CALL(func) \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_STANDARD, \ + (func)) + +#define SMC_RMI_VERSION SMC_RMI_CALL(0x0150) + +#define SMC_RMI_RTT_DATA_MAP_INIT SMC_RMI_CALL(0x0153) + +#define SMC_RMI_REALM_ACTIVATE SMC_RMI_CALL(0x0157) +#define SMC_RMI_REALM_CREATE SMC_RMI_CALL(0x0158) +#define SMC_RMI_REALM_DESTROY SMC_RMI_CALL(0x0159) +#define SMC_RMI_REC_CREATE SMC_RMI_CALL(0x015a) +#define SMC_RMI_REC_DESTROY SMC_RMI_CALL(0x015b) +#define SMC_RMI_REC_ENTER SMC_RMI_CALL(0x015c) +#define SMC_RMI_RTT_CREATE SMC_RMI_CALL(0x015d) +#define SMC_RMI_RTT_DESTROY SMC_RMI_CALL(0x015e) + +#define SMC_RMI_RTT_READ_ENTRY SMC_RMI_CALL(0x0161) + +#define SMC_RMI_RTT_DEV_VALIDATE SMC_RMI_CALL(0x0163) +#define SMC_RMI_PSCI_COMPLETE SMC_RMI_CALL(0x0164) +#define SMC_RMI_FEATURES SMC_RMI_CALL(0x0165) +#define SMC_RMI_RTT_FOLD SMC_RMI_CALL(0x0166) + +#define SMC_RMI_RTT_INIT_RIPAS SMC_RMI_CALL(0x0168) +#define SMC_RMI_RTT_SET_RIPAS SMC_RMI_CALL(0x0169) +#define SMC_RMI_VSMMU_CREATE SMC_RMI_CALL(0x016a) +#define SMC_RMI_VSMMU_DESTROY SMC_RMI_CALL(0x016b) + +#define SMC_RMI_RMM_CONFIG_SET SMC_RMI_CALL(0x016e) +#define SMC_RMI_PSMMU_IRQ_NOTIFY SMC_RMI_CALL(0x016f) +#define SMC_RMI_ATTEST_PLAT_TOKEN_REFRESH SMC_RMI_CALL(0x0170) + +#define SMC_RMI_PDEV_ABORT SMC_RMI_CALL(0x0174) +#define SMC_RMI_PDEV_COMMUNICATE SMC_RMI_CALL(0x0175) +#define SMC_RMI_PDEV_CREATE SMC_RMI_CALL(0x0176) +#define SMC_RMI_PDEV_DESTROY SMC_RMI_CALL(0x0177) +#define SMC_RMI_PDEV_GET_STATE SMC_RMI_CALL(0x0178) + +#define SMC_RMI_PDEV_STREAM_KEY_REFRESH SMC_RMI_CALL(0x017a) +#define SMC_RMI_PDEV_SET_PUBKEY SMC_RMI_CALL(0x017b) +#define SMC_RMI_PDEV_STOP SMC_RMI_CALL(0x017c) +#define SMC_RMI_RTT_AUX_CREATE SMC_RMI_CALL(0x017d) +#define SMC_RMI_RTT_AUX_DESTROY SMC_RMI_CALL(0x017e) +#define SMC_RMI_RTT_AUX_FOLD SMC_RMI_CALL(0x017f) + +#define SMC_RMI_VDEV_ABORT SMC_RMI_CALL(0x0185) +#define SMC_RMI_VDEV_COMMUNICATE SMC_RMI_CALL(0x0186) +#define SMC_RMI_VDEV_CREATE SMC_RMI_CALL(0x0187) +#define SMC_RMI_VDEV_DESTROY SMC_RMI_CALL(0x0188) +#define SMC_RMI_VDEV_GET_STATE SMC_RMI_CALL(0x0189) +#define SMC_RMI_VDEV_UNLOCK SMC_RMI_CALL(0x018a) +#define SMC_RMI_RTT_SET_S2AP SMC_RMI_CALL(0x018b) + +#define SMC_RMI_VDEV_GET_INTERFACE_REPORT SMC_RMI_CALL(0x01d0) +#define SMC_RMI_VDEV_GET_MEASUREMENTS SMC_RMI_CALL(0x01d1) +#define SMC_RMI_VDEV_LOCK SMC_RMI_CALL(0x01d2) +#define SMC_RMI_VDEV_START SMC_RMI_CALL(0x01d3) + +#define SMC_RMI_VSMMU_EVENT_HANDLE SMC_RMI_CALL(0x01d6) +#define SMC_RMI_PSMMU_ACTIVATE SMC_RMI_CALL(0x01d7) +#define SMC_RMI_PSMMU_DEACTIVATE SMC_RMI_CALL(0x01d8) + +#define SMC_RMI_PSMMU_ST_L2_CREATE SMC_RMI_CALL(0x01db) +#define SMC_RMI_PSMMU_ST_L2_DESTROY SMC_RMI_CALL(0x01dc) +#define SMC_RMI_DPT_L0_CREATE SMC_RMI_CALL(0x01dd) +#define SMC_RMI_DPT_L0_DESTROY SMC_RMI_CALL(0x01de) +#define SMC_RMI_DPT_L1_CREATE SMC_RMI_CALL(0x01df) +#define SMC_RMI_DPT_L1_DESTROY SMC_RMI_CALL(0x01e0) +#define SMC_RMI_GRANULE_TRACKING_GET SMC_RMI_CALL(0x01e1) + +#define SMC_RMI_GRANULE_TRACKING_SET SMC_RMI_CALL(0x01e3) + +#define SMC_RMI_RMM_CONFIG_GET SMC_RMI_CALL(0x01ec) + +#define SMC_RMI_RMM_STATE_GET SMC_RMI_CALL(0x01ee) + +#define SMC_RMI_PSMMU_EVENT_CONSUME SMC_RMI_CALL(0x01f0) +#define SMC_RMI_GRANULE_RANGE_DELEGATE SMC_RMI_CALL(0x01f1) +#define SMC_RMI_GRANULE_RANGE_UNDELEGATE SMC_RMI_CALL(0x01f2) +#define SMC_RMI_GPT_L1_CREATE SMC_RMI_CALL(0x01f3) +#define SMC_RMI_GPT_L1_DESTROY SMC_RMI_CALL(0x01f4) +#define SMC_RMI_RTT_DATA_MAP SMC_RMI_CALL(0x01f5) +#define SMC_RMI_RTT_DATA_UNMAP SMC_RMI_CALL(0x01f6) +#define SMC_RMI_RTT_DEV_MAP SMC_RMI_CALL(0x01f7) +#define SMC_RMI_RTT_DEV_UNMAP SMC_RMI_CALL(0x01f8) +#define SMC_RMI_RTT_ARCH_DEV_MAP SMC_RMI_CALL(0x01f9) +#define SMC_RMI_RTT_ARCH_DEV_UNMAP SMC_RMI_CALL(0x01fa) +#define SMC_RMI_RTT_UNPROT_MAP SMC_RMI_CALL(0x01fb) +#define SMC_RMI_RTT_UNPROT_UNMAP SMC_RMI_CALL(0x01fc) +#define SMC_RMI_RTT_AUX_PROT_MAP SMC_RMI_CALL(0x01fd) +#define SMC_RMI_RTT_AUX_PROT_UNMAP SMC_RMI_CALL(0x01fe) +#define SMC_RMI_RTT_AUX_UNPROT_MAP SMC_RMI_CALL(0x01ff) +#define SMC_RMI_RTT_AUX_UNPROT_UNMAP SMC_RMI_CALL(0x0200) +#define SMC_RMI_REALM_TERMINATE SMC_RMI_CALL(0x0201) +#define SMC_RMI_RMM_ACTIVATE SMC_RMI_CALL(0x0202) +#define SMC_RMI_OP_CONTINUE SMC_RMI_CALL(0x0203) +#define SMC_RMI_PDEV_STREAM_CONNECT SMC_RMI_CALL(0x0204) +#define SMC_RMI_PDEV_STREAM_DISCONNECT SMC_RMI_CALL(0x0205) +#define SMC_RMI_PDEV_STREAM_COMPLETE SMC_RMI_CALL(0x0206) +#define SMC_RMI_PDEV_STREAM_KEY_PURGE SMC_RMI_CALL(0x0207) +#define SMC_RMI_OP_MEM_DONATE SMC_RMI_CALL(0x0208) +#define SMC_RMI_OP_MEM_RECLAIM SMC_RMI_CALL(0x0209) +#define SMC_RMI_OP_CANCEL SMC_RMI_CALL(0x020a) +#define SMC_RMI_VSMMU_FEATURES SMC_RMI_CALL(0x020b) +#define SMC_RMI_VSMMU_CMD_GET SMC_RMI_CALL(0x020c) +#define SMC_RMI_VSMMU_CMD_COMPLETE SMC_RMI_CALL(0x020d) +#define SMC_RMI_PSMMU_INFO SMC_RMI_CALL(0x020e) +#define SMC_RMI_RMM_DEACTIVATE SMC_RMI_CALL(0x020f) +#define SMC_RMI_PDEV_STREAM_INFO SMC_RMI_CALL(0x0210) +#define SMC_RMI_GPT_INFO SMC_RMI_CALL(0x0211) + +#define RMI_ABI_MAJOR_VERSION 2 +#define RMI_ABI_MINOR_VERSION 0 + +#define RMI_ABI_VERSION_MAJOR_MASK GENMASK(30, 16) +#define RMI_ABI_VERSION_MINOR_MASK GENMASK(15, 0) + +#define RMI_ABI_VERSION_GET_MAJOR(v) FIELD_GET(RMI_ABI_VERSION_MAJOR_MASK, (v)) +#define RMI_ABI_VERSION_GET_MINOR(v) FIELD_GET(RMI_ABI_VERSION_MINOR_MASK, (v)) +#define RMI_ABI_VERSION(major, minor) \ + (FIELD_PREP(RMI_ABI_VERSION_MAJOR_MASK, (major)) | \ + FIELD_PREP(RMI_ABI_VERSION_MINOR_MASK, (minor))) + +/* RmiResult Type definitions */ +#define RMI_RESULT_STATUS_MASK GENMASK(7, 0) +/* RmiResultDataLevel field */ +#define RMI_RESULT_DATA_LEVEL_MASK GENMASK(15, 8) +/* RmiResultDataIncomplete fields */ +#define RMI_RESULT_MEMREQ_MASK GENMASK(9, 8) +#define RMI_RESULT_CAN_CANCEL_MASK BIT(10) + +#define RMI_RESULT_STATUS(ret) FIELD_GET(RMI_RESULT_STATUS_MASK, ret) +#define RMI_RESULT_DATA_LEVEL(ret) FIELD_GET(RMI_RESULT_DATA_LEVEL_MASK, ret) +#define RMI_RESULT_MEMREQ(ret) FIELD_GET(RMI_RESULT_MEMREQ_MASK, ret) +#define RMI_RESULT_CAN_CANCEL(ret) FIELD_GET(RMI_RESULT_CAN_CANCEL_MASK, ret) + +#define RMI_SUCCESS 0 +#define RMI_ERROR_INPUT 1 +#define RMI_ERROR_REALM 2 +#define RMI_ERROR_REC 3 +#define RMI_ERROR_RTT 4 +#define RMI_ERROR_NOT_SUPPORTED 5 +#define RMI_ERROR_DEVICE 6 +#define RMI_ERROR_RTT_AUX 7 +#define RMI_ERROR_PSMMU_ST 8 +#define RMI_ERROR_DPT 9 +#define RMI_BUSY 10 +#define RMI_ERROR_GLOBAL 11 +#define RMI_ERROR_TRACKING 12 +#define RMI_INCOMPLETE 13 +#define RMI_BLOCKED 14 +#define RMI_ERROR_GPT 15 +#define RMI_ERROR_GRANULE 16 + +#define RMI_CONTINUE_KEEP_GOING 0 +#define RMI_CONTINUE_STOP 1 + +#define RMI_OP_MEM_REQ_NONE 0 +#define RMI_OP_MEM_REQ_DONATE 1 +#define RMI_OP_MEM_REQ_RECLAIM 2 + +#define RMI_OP_CANNOT_CANCEL 0 +#define RMI_OP_CAN_CANCEL 1 + +#define RMI_DONATE_STATE_MASK GENMASK(18, 17) +#define RMI_DONATE_CONTIG_MASK BIT(16) +#define RMI_DONATE_COUNT_MASK GENMASK(15, 2) +#define RMI_DONATE_BLOCK_SIZE_MASK GENMASK(1, 0) + +#define RMI_DONATE_STATE(req) FIELD_GET(RMI_DONATE_STATE_MASK, req) +#define RMI_DONATE_CONTIG(req) FIELD_GET(RMI_DONATE_CONTIG_MASK, req) +#define RMI_DONATE_COUNT(req) FIELD_GET(RMI_DONATE_COUNT_MASK, req) +#define RMI_DONATE_BLOCK_SIZE(req) FIELD_GET(RMI_DONATE_BLOCK_SIZE_MASK, req) + +#define RMI_OP_MEM_DELEGATED 0 +#define RMI_OP_MEM_UNDELEGATED 1 +#define RMI_OP_MEM_CONDITIONAL 2 + +#define RMI_OP_MEM_NON_CONTIG 0 +#define RMI_OP_MEM_CONTIG 1 + +#define RMI_ADDR_TYPE_NONE 0 +#define RMI_ADDR_TYPE_SINGLE 1 +#define RMI_ADDR_TYPE_LIST 2 + +#define RMI_ADDR_RANGE_STATE_MASK GENMASK_ULL(63, 62) +#define RMI_ADDR_RANGE_ADDR_MASK GENMASK_ULL(51, PAGE_SHIFT) +#define RMI_ADDR_RANGE_COUNT_MASK GENMASK(PAGE_SHIFT - 1, 2) +#define RMI_ADDR_RANGE_BLOCK_SIZE_MASK GENMASK(1, 0) + +#define RMI_ADDR_RANGE_STATE(r) FIELD_GET(RMI_ADDR_RANGE_STATE_MASK, (r)) +#define RMI_ADDR_RANGE_ADDR(r) ((r) & RMI_ADDR_RANGE_ADDR_MASK) +#define RMI_ADDR_RANGE_COUNT(r) FIELD_GET(RMI_ADDR_RANGE_COUNT_MASK, (r)) +#define RMI_ADDR_RANGE_BLOCK_SIZE(r) FIELD_GET(RMI_ADDR_RANGE_BLOCK_SIZE_MASK, (r)) + +enum rmi_ripas { + RMI_EMPTY = 0, + RMI_RAM = 1, + RMI_DESTROYED = 2, + RMI_DEV = 3, +}; + +#define RMI_NO_MEASURE_CONTENT 0 +#define RMI_MEASURE_CONTENT 1 + +#define RMI_FEATURE_REGISTER_0_S2OASZ GENMASK_ULL(40, 33) +#define RMI_FEATURE_REGISTER_0_L0GPT_BLOCK_DELEGATE BIT_ULL(32) +#define RMI_FEATURE_REGISTER_0_PMU_NUM_CTRS GENMASK(31, 27) +#define RMI_FEATURE_REGISTER_0_PMU BIT(26) +#define RMI_FEATURE_REGISTER_0_NUM_WPS GENMASK(25, 20) +#define RMI_FEATURE_REGISTER_0_NUM_BPS GENMASK(19, 14) +#define RMI_FEATURE_REGISTER_0_SVE_VL GENMASK(13, 10) +#define RMI_FEATURE_REGISTER_0_SVE BIT(9) +#define RMI_FEATURE_REGISTER_0_LPA2 BIT(8) +#define RMI_FEATURE_REGISTER_0_S2SZ GENMASK(7, 0) + +#define RMI_FEATURE_REGISTER_1_PPS GENMASK(16, 14) +#define RMI_FEATURE_REGISTER_1_L0GPTSZ GENMASK(13, 10) +#define RMI_FEATURE_REGISTER_1_MAX_RECS_ORDER GENMASK(9, 6) +#define RMI_FEATURE_REGISTER_1_HASH_SHA_512 BIT(5) +#define RMI_FEATURE_REGISTER_1_HASH_SHA_384 BIT(4) +#define RMI_FEATURE_REGISTER_1_HASH_SHA_256 BIT(3) +#define RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_64KB BIT(2) +#define RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_16KB BIT(1) +#define RMI_FEATURE_REGISTER_1_RMI_GRAN_SZ_4KB BIT(0) + +#define RMI_FEATURE_REGISTER_2_REALM_MAX_VDEVS_ORDER GENMASK(14, 10) +#define RMI_FEATURE_REGISTER_2_NON_TEE_STREAM BIT(9) +#define RMI_FEATURE_REGISTER_2_VDEV_KROU BIT(8) +#define RMI_FEATURE_REGISTER_2_PDEV_MAX_VDEVS_ORDER GENMASK(7, 4) +#define RMI_FEATURE_REGISTER_2_ATS BIT(3) +#define RMI_FEATURE_REGISTER_2_VSMMU BIT(2) +#define RMI_FEATURE_REGISTER_2_DA_COH BIT(1) +#define RMI_FEATURE_REGISTER_2_DA BIT(0) + +#define RMI_FEATURE_REGISTER_3_RTT_S2AP_INDIRECT BIT(6) +#define RMI_FEATURE_REGISTER_3_RTT_PLANE GENMASK(5, 4) +#define RMI_FEATURE_REGISTER_3_MAX_NUM_AUX_PLANES GENMASK(3, 0) + +#define RMI_FEATURE_REGISTER_4_MEC_COUNT GENMASK_ULL(63, 0) + +#define RMI_MEM_CATEGORY_CONVENTIONAL 0 +#define RMI_MEM_CATEGORY_DEV_NCOH 1 +#define RMI_MEM_CATEGORY_DEV_COH 2 +#define RMI_MEM_CATEGORY_NONE 3 + +#define RMI_TRACKING_RESERVED 0 +#define RMI_TRACKING_NONE 1 +#define RMI_TRACKING_FINE 2 +#define RMI_TRACKING_COARSE 3 +#define RMI_TRACKING_INTERMEDIATE 4 + +#define RMI_GRANULE_SIZE_4KB 0 +#define RMI_GRANULE_SIZE_16KB 1 +#define RMI_GRANULE_SIZE_64KB 2 + +#define RMI_GPT_PAR_RESERVED 0U +#define RMI_GPT_PAR_PLAT 1U +#define RMI_GPT_PAR_HOST_NOT_CREATED 2U +#define RMI_GPT_PAR_HOST_CREATED 3U + +/* + * Note many of these fields are smaller than u64 but all fields have u64 + * alignment, so use u64 to ensure correct alignment. + */ +struct rmm_config { + union { /* 0x0 */ + struct { + u64 tracking_region_size; + u64 rmi_granule_size; + }; + u8 sizer[SZ_4K]; + }; +}; + +static_assert(sizeof(struct rmm_config) == SZ_4K); + +#define RMI_REALM_PARAM_FLAG_MEC_POLICY_MASK GENMASK(8, 7) +#define RMI_REALM_PARAM_FLAG_LFA_POLICY_MASK GENMASK(6, 5) +#define RMI_REALM_PARAM_FLAG_DA BIT(3) +#define RMI_REALM_PARAM_FLAG_PMU BIT(2) +#define RMI_REALM_PARAM_FLAG_SVE BIT(1) + +#define RMI_HASH_SHA_256 0 +#define RMI_HASH_SHA_512 1 +#define RMI_HASH_SHA_384 2 + +struct realm_params { + union { /* 0x0 */ + struct { + u64 flags0; + u64 s2sz; + u64 sve_vl; + u64 num_bps; + u64 num_wps; + u64 pmu_num_ctrs; + u64 hash_algo; + u64 num_aux_planes; + }; + u8 sizer0[0x400]; + }; + union { /* 0x400 */ + struct { + u8 rpv[64]; + u64 ats_plane; + }; + u8 sizer1[0x400]; + }; + union { /* 0x800 */ + struct { + u64 sizer2; + u64 rtt_base; + s64 rtt_level_start; + u64 rtt_num_start; + u64 flags1; + u64 max_num_vdevs; + }; + u8 sizer3[0x700]; + }; + union { /* 0xf00 */ + struct { + u8 sizer4[0x80]; + u64 aux_rtt_base[3]; + }; + u8 sizer5[0x100]; + }; +}; + +static_assert(offsetof(struct realm_params, rpv) == 0x400); +static_assert(offsetof(struct realm_params, rtt_base) == 0x808); +static_assert(offsetof(struct realm_params, aux_rtt_base) == 0xf80); +static_assert(sizeof(struct realm_params) == SZ_4K); + +/* + * The number of GPRs (starting from X0) that are configured by the host when + * a REC is created. + */ +#define REC_CREATE_NR_GPRS 8 + +#define REC_PARAMS_FLAG_RUNNABLE BIT(0) + +struct rec_params { + union { /* 0x0 */ + u64 flags; + u8 sizer0[0x100]; + }; + union { /* 0x100 */ + u64 mpidr; + u8 sizer1[0x100]; + }; + union { /* 0x200 */ + u64 pc; + u8 sizer2[0x100]; + }; + union { /* 0x300 */ + u64 gprs[REC_CREATE_NR_GPRS]; + u8 sizer3[0xd00]; + }; +}; + +static_assert(offsetof(struct rec_params, mpidr) == 0x100); +static_assert(offsetof(struct rec_params, pc) == 0x200); +static_assert(offsetof(struct rec_params, gprs) == 0x300); +static_assert(sizeof(struct rec_params) == SZ_4K); + +#define REC_ENTER_FLAG_FORCE_P0 BIT(7) +#define REC_ENTER_FLAG_DEV_MEM_RESPONSE BIT(6) +#define REC_ENTER_FLAG_S2AP_RESPONSE BIT(5) +#define REC_ENTER_FLAG_RIPAS_RESPONSE BIT(4) +#define REC_ENTER_FLAG_TRAP_WFE BIT(3) +#define REC_ENTER_FLAG_TRAP_WFI BIT(2) +#define REC_ENTER_FLAG_INJECT_SEA BIT(1) +#define REC_ENTER_FLAG_EMULATED_MMIO BIT(0) + +#define REC_RUN_GPRS 31 + +struct rec_enter { + union { /* 0x000 */ + u64 flags; + u8 sizer0[0x200]; + }; + union { /* 0x200 */ + u64 gprs[REC_RUN_GPRS]; + u8 sizer1[0x600]; + }; +}; + +static_assert(offsetof(struct rec_enter, gprs) == 0x200); +static_assert(sizeof(struct rec_enter) == SZ_2K); + +#define RMI_EXIT_SYNC 0x00 +#define RMI_EXIT_IRQ 0x01 +#define RMI_EXIT_FIQ 0x02 +#define RMI_EXIT_PSCI 0x03 +#define RMI_EXIT_RIPAS_CHANGE 0x04 +#define RMI_EXIT_HOST_CALL 0x05 +#define RMI_EXIT_SERROR 0x06 +#define RMI_EXIT_S2AP_CHANGE 0x07 +#define RMI_EXIT_VDEV_VALIDATE_MAPPING 0x08 +#define RMI_EXIT_VSMMU_COMMAND 0x0a + +struct rec_exit { + union { /* 0x000 */ + u8 exit_reason; + u8 sizer0[0x100]; + }; + union { /* 0x100 */ + struct { + u64 esr; + u64 far; + u64 hpfar; + u64 rtt_tree; + }; + u8 sizer1[0x100]; + }; + union { /* 0x200 */ + u64 gprs[REC_RUN_GPRS]; + u8 sizer2[0x200]; + }; + union { /* 0x400 */ + struct { + u64 cntp_ctl; + u64 cntp_cval; + u64 cntv_ctl; + u64 cntv_cval; + }; + u8 sizer4[0x100]; + }; + union { /* 0x500 */ + struct { + u64 ripas_base; + u64 ripas_top; + u8 ripas_value; + u8 sizer5[0xf]; + u64 s2ap_base; + u64 s2ap_top; + u64 vdev_id_1; + u64 vdev_id_2; + u64 dev_mem_base; + u64 dev_mem_top; + u64 dev_mem_pa; + }; + u8 sizer6[0x100]; + }; + union { /* 0x600 */ + struct { + u16 imm; + u8 sizer7[0x6]; + u64 plane; + }; + u8 sizer8[0x100]; + }; + union { /* 0x700 */ + struct { + u8 pmu_ovf_status; + u8 sizer9[0xf]; + u64 vsmmu; + }; + u8 sizer10[0x100]; + }; +}; + +static_assert(sizeof(struct rec_exit) == SZ_2K); +static_assert(offsetof(struct rec_exit, esr) == 0x100); +static_assert(offsetof(struct rec_exit, gprs) == 0x200); +static_assert(offsetof(struct rec_exit, cntp_ctl) == 0x400); +static_assert(offsetof(struct rec_exit, ripas_base) == 0x500); +static_assert(offsetof(struct rec_exit, imm) == 0x600); +static_assert(offsetof(struct rec_exit, pmu_ovf_status) == 0x700); + +struct rec_run { + struct rec_enter enter; + struct rec_exit exit; +}; + +static_assert(sizeof(struct rec_run) == SZ_4K); + +/* RMI_RTT_UNPROT_MAP_FLAGS definitions */ +#define RMI_RTT_UNPROT_MAP_FLAGS_S2AP_MASK GENMASK(22, 19) +#define RMI_RTT_UNPROT_MAP_FLAGS_MEMATTR_MASK GENMASK(18, 16) +#define RMI_RTT_UNPROT_MAP_FLAGS_LIST_COUNT_MASK GENMASK(15, 2) +#define RMI_RTT_UNPROT_MAP_FLAGS_OADDR_TYPE_MASK GENMASK(1, 0) + +/* RMI_RTT_PROT_MAP_FLAGS definitions */ +#define RMI_RTT_PROT_MAP_FLAGS_LIST_COUNT_MASK GENMASK(15, 2) +#define RMI_RTT_PROT_MAP_FLAGS_OADDR_TYPE_MASK GENMASK(1, 0) + +/* S2AP Direct Encodings, used in RMI_RTT_UNPROT_MAP_FLAGS_S2AP */ +#define RMI_S2AP_DIRECT_READ BIT(1) +#define RMI_S2AP_DIRECT_WRITE BIT(0) + +#endif /* __LINUX_ARM_SMCCC_RMI_H_ */ diff --git a/arch/arm64/include/asm/rsi_smc.h b/include/linux/arm-smccc-rsi.h index e19253f96c94..cca881ddfdba 100644 --- a/arch/arm64/include/asm/rsi_smc.h +++ b/include/linux/arm-smccc-rsi.h @@ -3,8 +3,8 @@ * Copyright (C) 2023 ARM Ltd. */ -#ifndef __ASM_RSI_SMC_H_ -#define __ASM_RSI_SMC_H_ +#ifndef __LINUX_ARM_SMCCC_RSI_H_ +#define __LINUX_ARM_SMCCC_RSI_H_ #include <linux/arm-smccc.h> @@ -128,7 +128,7 @@ struct realm_config { union { struct { unsigned long ipa_bits; /* Width of IPA in bits */ - unsigned long hash_algo; /* Hash algorithm */ + u8 hash_algo; /* Hash algorithm */ }; u8 pad[0x200]; }; @@ -190,4 +190,4 @@ struct realm_config { */ #define SMC_RSI_HOST_CALL SMC_RSI_FID(0x199) -#endif /* __ASM_RSI_SMC_H_ */ +#endif /* __LINUX_ARM_SMCCC_RSI_H_ */ diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index f92a36187a52..002f56e15362 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -12,7 +12,7 @@ struct mpam_msc; enum mpam_msc_iface { MPAM_IFACE_MMIO, /* a real MPAM MSC */ - MPAM_IFACE_PCC, /* a fake MPAM MSC */ + MPAM_IFACE_PCC, /* using the MPAM-Fb firmware redirection */ }; enum mpam_class_types { diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index feb32949aeea..1b6dba159bb2 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -209,12 +209,6 @@ enum cpuhp_state { CPUHP_AP_PERF_S390_SF_ONLINE, CPUHP_AP_PERF_ARM_CCI_ONLINE, CPUHP_AP_PERF_ARM_CCN_ONLINE, - CPUHP_AP_PERF_ARM_HISI_CPA_ONLINE, - CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, - CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, - CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, - CPUHP_AP_PERF_ARM_HISI_PA_ONLINE, - CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE, CPUHP_AP_PERF_ARM_HISI_PCIE_PMU_ONLINE, CPUHP_AP_PERF_ARM_HNS3_PMU_ONLINE, CPUHP_AP_PERF_ARM_L2X0_ONLINE, diff --git a/include/linux/device-id/arm_smccc.h b/include/linux/device-id/arm_smccc.h new file mode 100644 index 000000000000..a8579832fc1c --- /dev/null +++ b/include/linux/device-id/arm_smccc.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_DEVICE_ID_ARM_SMCCC_H +#define __LINUX_DEVICE_ID_ARM_SMCCC_H + +#ifdef __KERNEL__ +#include <linux/types.h> +#endif + +#define ARM_SMCCC_MODULE_PREFIX "arm_smccc:" + +/** + * struct arm_smccc_device_id - Arm SMCCC bus device identifier + * @func_id: SMCCC function identifier + */ +struct arm_smccc_device_id { + __u32 func_id; +}; + +#endif /* __LINUX_DEVICE_ID_ARM_SMCCC_H */ diff --git a/include/linux/mod_devicetable.h b/include/linux/mod_devicetable.h index a397213bedac..318c4b5c3451 100644 --- a/include/linux/mod_devicetable.h +++ b/include/linux/mod_devicetable.h @@ -16,6 +16,7 @@ #include "device-id/amba.h" #include "device-id/ap.h" #include "device-id/apr.h" +#include "device-id/arm_smccc.h" #include "device-id/auxiliary.h" #include "device-id/bcma.h" #include "device-id/ccw.h" diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 16a04202888b..e6ad4f2a475c 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -116,7 +116,7 @@ int rcutree_offline_cpu(unsigned int cpu); void rcutree_migrate_callbacks(int cpu); -/* Called from hotplug and also arm64 early secondary boot failure */ +/* Called from hotplug */ void rcutree_report_cpu_dead(void); #endif /* __LINUX_RCUTREE_H */ diff --git a/include/linux/suspend.h b/include/linux/suspend.h index b02876f1ae38..a3815027773c 100644 --- a/include/linux/suspend.h +++ b/include/linux/suspend.h @@ -401,6 +401,7 @@ int hibernate_quiet_exec(int (*func)(void *data), void *data); int hibernate_resume_nonboot_cpu_disable(void); int arch_hibernation_header_save(void *addr, unsigned int max_size); int arch_hibernation_header_restore(void *addr); +bool arch_hibernation_available(void); #else /* CONFIG_HIBERNATION */ static inline void register_nosave_region(unsigned long b, unsigned long e) {} diff --git a/kernel/events/core.c b/kernel/events/core.c index c4217c996f5a..2b38e9a0c834 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -10527,6 +10527,7 @@ void perf_event_itrace_started(struct perf_event *event) { WRITE_ONCE(event->attach_state, event->attach_state | PERF_ATTACH_ITRACE); } +EXPORT_SYMBOL_GPL(perf_event_itrace_started); static void perf_log_itrace_start(struct perf_event *event) { diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c index c13f68ab7f6e..e365e440cd52 100644 --- a/kernel/power/hibernate.c +++ b/kernel/power/hibernate.c @@ -106,11 +106,17 @@ bool hibernation_in_progress(void) return !atomic_read(&hibernate_atomic); } +__weak bool arch_hibernation_available(void) +{ + return true; +} + bool hibernation_available(void) { return nohibernate == 0 && !security_locked_down(LOCKDOWN_HIBERNATION) && - !secretmem_active() && !cxl_mem_active(); + !secretmem_active() && !cxl_mem_active() && + arch_hibernation_available(); } /** diff --git a/scripts/mod/devicetable-offsets.c b/scripts/mod/devicetable-offsets.c index b4178c42d08f..1492e9fdc7cb 100644 --- a/scripts/mod/devicetable-offsets.c +++ b/scripts/mod/devicetable-offsets.c @@ -254,6 +254,9 @@ int main(void) DEVID(auxiliary_device_id); DEVID_FIELD(auxiliary_device_id, name); + DEVID(arm_smccc_device_id); + DEVID_FIELD(arm_smccc_device_id, func_id); + DEVID(ssam_device_id); DEVID_FIELD(ssam_device_id, match_flags); DEVID_FIELD(ssam_device_id, domain); diff --git a/scripts/mod/file2alias.c b/scripts/mod/file2alias.c index 8d36c74dec2d..980d7da456bb 100644 --- a/scripts/mod/file2alias.c +++ b/scripts/mod/file2alias.c @@ -1349,6 +1349,13 @@ static void do_auxiliary_entry(struct module *mod, void *symval) module_alias_printf(mod, false, AUXILIARY_MODULE_PREFIX "%s", *name); } +static void do_arm_smccc_entry(struct module *mod, void *symval) +{ + DEF_FIELD(symval, arm_smccc_device_id, func_id); + + module_alias_printf(mod, false, ARM_SMCCC_MODULE_PREFIX "f%08X", func_id); +} + /* * Looks like: ssam:dNcNtNiNfN * @@ -1519,6 +1526,7 @@ static const struct devtable devtable[] = { {"mhi", SIZE_mhi_device_id, do_mhi_entry}, {"mhi_ep", SIZE_mhi_device_id, do_mhi_ep_entry}, {"auxiliary", SIZE_auxiliary_device_id, do_auxiliary_entry}, + {"arm_smccc", SIZE_arm_smccc_device_id, do_arm_smccc_entry}, {"ssam", SIZE_ssam_device_id, do_ssam_entry}, {"dfl", SIZE_dfl_device_id, do_dfl_entry}, {"ishtp", SIZE_ishtp_device_id, do_ishtp_entry}, diff --git a/tools/testing/selftests/arm64/fp/fp-ptrace.c b/tools/testing/selftests/arm64/fp/fp-ptrace.c index b435837c8c0e..9b184d78f4fd 100644 --- a/tools/testing/selftests/arm64/fp/fp-ptrace.c +++ b/tools/testing/selftests/arm64/fp/fp-ptrace.c @@ -115,23 +115,6 @@ static void handle_alarm(int sig, siginfo_t *info, void *context) got_alarm = true; } -#ifdef CONFIG_CPU_BIG_ENDIAN -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - u64 a = swab64(x); - u64 b = swab64(x >> 64); - - return ((__uint128_t)a << 64) | b; -} -#else -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - return x; -} -#endif - -#define arm64_le128_to_cpu(x) arm64_cpu_to_le128(x) - static bool sve_supported(void) { return getauxval(AT_HWCAP) & HWCAP_SVE; @@ -932,7 +915,7 @@ static void fpsimd_to_sve(__uint128_t *v, char *z, int vl) for (i = 0; i < __SVE_NUM_ZREGS; i++) { p = (__uint128_t *)&z[__SVE_ZREG_OFFSET(vq, i)]; - *p = arm64_cpu_to_le128(v[i]); + *p = v[i]; } } diff --git a/tools/testing/selftests/arm64/gcs/gcs-locking.c b/tools/testing/selftests/arm64/gcs/gcs-locking.c index 1e6abb136ffd..80a1317f6097 100644 --- a/tools/testing/selftests/arm64/gcs/gcs-locking.c +++ b/tools/testing/selftests/arm64/gcs/gcs-locking.c @@ -77,6 +77,21 @@ FIXTURE_VARIANT_ADD(valid_modes, enable_write_push) PR_SHADOW_STACK_PUSH, }; +FIXTURE_VARIANT_ADD(valid_modes, write) +{ + .mode = PR_SHADOW_STACK_WRITE, +}; + +FIXTURE_VARIANT_ADD(valid_modes, push) +{ + .mode = PR_SHADOW_STACK_PUSH, +}; + +FIXTURE_VARIANT_ADD(valid_modes, write_push) +{ + .mode = PR_SHADOW_STACK_WRITE | PR_SHADOW_STACK_PUSH, +}; + FIXTURE_SETUP(valid_modes) { } @@ -115,7 +130,7 @@ TEST_F(valid_modes, enable_lock_disable) ASSERT_EQ(ret, 0); ret = my_syscall2(__NR_prctl, PR_SET_SHADOW_STACK_STATUS, 0); - ASSERT_EQ(ret, -EBUSY); + ASSERT_EQ(ret, -EPERM); _exit(0); } @@ -131,7 +146,7 @@ TEST_F(valid_modes, lock_enable) ret = my_syscall2(__NR_prctl, PR_SET_SHADOW_STACK_STATUS, variant->mode); - ASSERT_EQ(ret, -EBUSY); + ASSERT_EQ(ret, -EPERM); ret = prctl(PR_GET_SHADOW_STACK_STATUS, &mode, 0, 0, 0); ASSERT_EQ(ret, 0); diff --git a/tools/testing/selftests/arm64/mte/check_buffer_fill.c b/tools/testing/selftests/arm64/mte/check_buffer_fill.c index 039b1d7d8566..54e1dcaf52ac 100644 --- a/tools/testing/selftests/arm64/mte/check_buffer_fill.c +++ b/tools/testing/selftests/arm64/mte/check_buffer_fill.c @@ -41,7 +41,7 @@ static int check_buffer_by_byte(int mem_type, int mode) mte_initialize_current_context(mode, (uintptr_t)ptr, sizes[i]); /* Set some value in tagged memory */ for (j = 0; j < sizes[i]; j++) - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); mte_wait_after_trig(); err = cur_mte_cxt.fault_valid; /* Check the buffer whether it is filled. */ @@ -82,7 +82,7 @@ static int check_buffer_underflow_by_byte(int mem_type, int mode, /* Set some value in tagged memory and make the buffer underflow */ for (j = sizes[i] - 1; (j >= -underflow_range) && (!cur_mte_cxt.fault_valid); j--) { - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); last_index = j; } mte_wait_after_trig(); @@ -180,7 +180,7 @@ static int check_buffer_overflow_by_byte(int mem_type, int mode, /* Set some value in tagged memory and make the buffer underflow */ for (j = 0, last_index = 0 ; (j < (sizes[i] + overflow_range)) && (cur_mte_cxt.fault_valid == false); j++) { - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); last_index = j; } mte_wait_after_trig(); @@ -308,8 +308,8 @@ static int check_buffer_by_block_iterate(int mem_type, int mode, size_t size) result = KSFT_PASS; mte_initialize_current_context(mode, (uintptr_t)dst, size); /* Set some value in memory and copy*/ - memset((void *)src, (int)'1', size); - memcpy((void *)dst, (void *)src, size); + memset_safe((void *)src, (int)'1', size); + memcpy_safe((void *)dst, (void *)src, size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) { result = KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_child_memory.c b/tools/testing/selftests/arm64/mte/check_child_memory.c index e6a8acca2a94..5c62359e5682 100644 --- a/tools/testing/selftests/arm64/mte/check_child_memory.c +++ b/tools/testing/selftests/arm64/mte/check_child_memory.c @@ -41,7 +41,7 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } else if (child == 0) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); /* Do copy on write */ - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) { fault = 1; @@ -56,14 +56,14 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } } mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memset(ptr - UNDERFLOW, '2', UNDERFLOW); + memset_safe(ptr - UNDERFLOW, '2', UNDERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false) { fault = 1; goto check_child_tag_inheritance_err; } mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memset(ptr + size, '3', OVERFLOW); + memset_safe(ptr + size, '3', OVERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false) { fault = 1; diff --git a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c index 23e4a7a9950c..85f1574b88be 100644 --- a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c +++ b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c @@ -106,7 +106,7 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } else if (child == 0) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); /* Do copy on write */ - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) { fault = 1; @@ -135,7 +135,7 @@ check_child_tag_inheritance_err: static int check_mte_memory(char *ptr, int size, int mode, int tag_check) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_mmap_options.c b/tools/testing/selftests/arm64/mte/check_mmap_options.c index 492f2cd41f43..0f4d43fe80f2 100644 --- a/tools/testing/selftests/arm64/mte/check_mmap_options.c +++ b/tools/testing/selftests/arm64/mte/check_mmap_options.c @@ -72,13 +72,13 @@ static int check_mte_memory(char *ptr, int size, int mode, ptr = mte_insert_atag(ptr); mte_initialize_current_context(mode, (uintptr_t)ptr, size); - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memset(ptr - UNDERFLOW, '2', UNDERFLOW); + memset_safe(ptr - UNDERFLOW, '2', UNDERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false && tag_check == TAG_CHECK_ON) return KSFT_FAIL; @@ -86,7 +86,7 @@ static int check_mte_memory(char *ptr, int size, int mode, return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memset(ptr + size, '3', OVERFLOW); + memset_safe(ptr + size, '3', OVERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false && tag_check == TAG_CHECK_ON) return KSFT_FAIL; @@ -95,13 +95,13 @@ static int check_mte_memory(char *ptr, int size, int mode, if (tag_op == TAG_OP_STONLY) { mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memcpy(buf, ptr - UNDERFLOW, MT_GRANULE_SIZE); + memcpy_safe(buf, ptr - UNDERFLOW, MT_GRANULE_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memcpy(buf, ptr + size, MT_GRANULE_SIZE); + memcpy_safe(buf, ptr + size, MT_GRANULE_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c index 6b4fa6705d7c..bc39be231ae2 100644 --- a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c +++ b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c @@ -23,7 +23,7 @@ static int verify_mte_pointer_validity(char *ptr, int mode) { mte_initialize_current_context(mode, (uintptr_t)ptr, BUFFER_SIZE); /* Check the validity of the tagged pointer */ - memset(ptr, '1', BUFFER_SIZE); + memset_safe(ptr, '1', BUFFER_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) { ksft_print_msg("Unexpected fault recorded for %p-%p in mode %x\n", @@ -159,7 +159,7 @@ static int check_none_included_tags(int mem_type, int mode) } mte_initialize_current_context(mode, (uintptr_t)ptr, BUFFER_SIZE); /* Check the write validity of the untagged pointer */ - memset(ptr, '1', BUFFER_SIZE); + memset_safe(ptr, '1', BUFFER_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) break; diff --git a/tools/testing/selftests/arm64/mte/mte_common_util.h b/tools/testing/selftests/arm64/mte/mte_common_util.h index 250d671329a5..547ec5659f78 100644 --- a/tools/testing/selftests/arm64/mte/mte_common_util.h +++ b/tools/testing/selftests/arm64/mte/mte_common_util.h @@ -64,6 +64,10 @@ void mte_restore_setup(void); int mte_switch_mode(int mte_option, unsigned long incl_mask, bool stonly); void mte_initialize_current_context(int mode, uintptr_t ptr, ssize_t range); +/* Safe memory access functions */ +void *memset_safe(void *s, int c, size_t n); +void *memcpy_safe(void *dest, const void *src, size_t n); + /* Common utility functions */ int create_temp_file(void); diff --git a/tools/testing/selftests/arm64/mte/mte_helper.S b/tools/testing/selftests/arm64/mte/mte_helper.S index a55dbbc56ed1..eb62abfa1eb4 100644 --- a/tools/testing/selftests/arm64/mte/mte_helper.S +++ b/tools/testing/selftests/arm64/mte/mte_helper.S @@ -128,3 +128,47 @@ ENTRY(mte_get_pstate_tco) ubfx x0, x0, #MT_PSTATE_TCO_SHIFT, #1 ret ENDPROC(mte_get_pstate_tco) + +/* + * memset_safe: Fill memory with a constant byte + * Input: + * x0 - destination pointer + * x1 - constant byte + * x2 - number of bytes to fill + * Return: + * x0 - original destination pointer + */ +ENTRY(memset_safe) + cbz x2, 2f + add x4, x2, x0 + mov x3, x0 +1: + strb w1, [x3] + add x3, x3, #0x1 + cmp x3, x4 + b.ne 1b +2: + ret +ENDPROC(memset_safe) + +/* + * memcpy_safe: Copy memory area + * Input: + * x0 - destination pointer + * x1 - source pointer + * x2 - number of bytes to copy + * Return: + * x0 - destination pointer + */ +ENTRY(memcpy_safe) + cbz x2, 2f + mov x3, #0x0 +1: + ldrb w4, [x1, x3] + strb w4, [x0, x3] + add x3, x3, #0x1 + cmp x3, x2 + b.ne 1b +2: + ret +ENDPROC(memcpy_safe) diff --git a/tools/testing/selftests/arm64/signal/Makefile b/tools/testing/selftests/arm64/signal/Makefile index 1381039fb36f..c4df4be41636 100644 --- a/tools/testing/selftests/arm64/signal/Makefile +++ b/tools/testing/selftests/arm64/signal/Makefile @@ -17,8 +17,8 @@ TEST_GEN_PROGS := $(notdir $(PROGS)) # the toplevel makefile include ../../lib.mk -$(TEST_GEN_PROGS): $(PROGS) - cp $(PROGS) $(OUTPUT)/ +$(TEST_GEN_PROGS): $(OUTPUT)/%: testcases/% + cp $< $@ # Common test-unit targets to build common-layout test-cases executables # Needs secondary expansion to properly include the testcase c-file in pre-reqs diff --git a/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c b/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c index e1ccf8f85a70..bcef280582ed 100644 --- a/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c +++ b/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c @@ -2,7 +2,7 @@ /* * Copyright (C) 2021 ARM Limited * - * Attempt to change the SVE vector length in a signal hander, this is not + * Attempt to change the SVE vector length in a signal handler, this is not * supported and is expected to segfault. */ diff --git a/tools/testing/selftests/rseq/rseq-arm64.h b/tools/testing/selftests/rseq/rseq-arm64.h index 21e1626a7235..f9fecf941bf2 100644 --- a/tools/testing/selftests/rseq/rseq-arm64.h +++ b/tools/testing/selftests/rseq/rseq-arm64.h @@ -12,12 +12,7 @@ * matches code endianness. */ #define RSEQ_SIG_CODE 0xd428bc00 /* BRK #0x45E0. */ - -#ifdef __AARCH64EB__ -#define RSEQ_SIG_DATA 0x00bc28d4 /* BRK #0x45E0. */ -#else #define RSEQ_SIG_DATA RSEQ_SIG_CODE -#endif #define RSEQ_SIG RSEQ_SIG_DATA |