arm64: cmpxchg: LL/SC: Avoid redundant extension

The __ll_sc__cmpxchg_case_##name##sz() template always takes 'old' as an
unsigned long. When 'sz' is less than 32 bits, the 'old' is explicitly
truncated so that old[31:sz-1] is zero, which is necessary so that the
W-register EOR+CBNZ sequence doesn't fail spuriously.

For 32-bit types specifically, no extension is necessary, but as 'old'
is 64 bits, the caller must extend 'old' to 64 bits. This leads to a
redundant instruction (typically a MOV) to extend the value, as can be
seen from disassembly of the test case below.

Avoid this by always taking 'old' as a 'u##sz' type, and explicitly
zero-extend this to a 64-bit or 32-bit type matching the X or W register
used by the assembly. To make this work, the table of cases now
explicitly lists 'x' for the cases where an X register is used.

For 32-bit types this removes the redundant instruction. For 64-bit
types there is no truncation, and hence no change. For {16,8}-bit types,
the existing zero-extension to 64-bit is equivalent to the new
zero-extension to 32-bit, and compilers happen to use the same
instructions for this.

I've placed the logic for zero extension into a new <asm/xwreg.h> header
as it will be used by other logic in subsequent patches.

Test case:

| u32 outline_cmpxchg_u32(u32 *p, u32 o, u32 n)
| {
|         return cmpxchg(p, o, n);
| }

Before this patch:

| <outline_cmpxchg_u32>:
|        b       1f
|        casal   w1, w2, [x0]
|        mov     w0, w1
|        ret
| 1:     mov     w3, w1
|        prfm    pstl1strm, [x0]
| 2:     ldxr    w1, [x0]
|        eor     w4, w1, w3
|        cbnz    w4, 3f
|        stlxr   w4, w2, [x0]
|        cbnz    w4, 2b
|        dmb     ish
| 3:     mov     w0, w1
|        ret

After this patch:

| <outline_cmpxchg_u32>:
|        b       1f
|        casal   w1, w2, [x0]
|        mov     w0, w1
|        ret
| 1:     prfm    pstl1strm, [x0]
| 2:     ldxr    w3, [x0]
|        eor     w4, w3, w1
|        cbnz    w4, 3f
|        stlxr   w4, w2, [x0]
|        cbnz    w4, 2b
|        dmb     ish
| 3:     mov     w0, w3
|        ret

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Tested-by: Muhammad Usama Anjum <usama.anjum@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

0 files changed, 0 insertions, 0 deletions