linux-toradex.git/arch/s390/include/asm/futex.h, branch v6.16

s390/mm: Reimplement lazy ASCE handling

2025-04-14T09:23:21+00:00

Reduce system call overhead time (round trip time for invoking a
non-existent system call) by 25%.

With the removal of set_fs() [1] lazy control register handling was removed
in order to keep kernel entry and exit simple. However this made system
calls slower.

With the conversion to generic entry [2] and numerous follow up changes
which simplified the entry code significantly, adding support for lazy asce
handling doesn't add much complexity to the entry code anymore.

In particular this means:

- On kernel entry the primary asce is not modified and contains the user
  asce

- Kernel accesses which require secondary-space mode (for example futex
  operations) are surrounded by enable_sacf_uaccess() and
  disable_sacf_uaccess() calls. enable_sacf_uaccess() sets the primary asce
  to kernel asce so that the sacf instruction can be used to switch to
  secondary-space mode. The primary asce is changed back to user asce with
  disable_sacf_uaccess().

The state of the control register which contains the primary asce is
reflected with a new TIF_ASCE_PRIMARY bit. This is required on context
switch so that the correct asce is restored for the scheduled in process.

In result address spaces are now setup like this:

CPU running in               | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE
-----------------------------|-----------|-----------|-----------
user space                   |  user     |  user     |  kernel
kernel (no sacf)             |  user     |  user     |  kernel
kernel (during sacf uaccess) |  kernel   |  user     |  kernel
kernel (kvm guest execution) |  guest    |  user     |  kernel

In result cr1 control register content is not changed except for:
- futex system calls
- legacy s390 PCI system calls
- the kvm specific cmpxchg_user_key() uaccess helper

This leads to faster system call execution.

[1] 87d598634521 ("s390/mm: remove set_fs / rework address space handling")
[2] 56e62a737028 ("s390: convert to generic entry")

Reviewed-by: Alexander Gordeev 
Signed-off-by: Heiko Carstens

s390/futex: Avoid KMSAN instrumention for user pointers

2025-01-26T16:24:08+00:00

Similar to commit eb6efdfeaeca ("s390/uaccess: add KMSAN support to
put_user() and get_user()") disable KMSAN instrumention for futex inline
assemblies, which contain dereferenced user pointers. With KMSAN
instrumentation this would lead to accesses of shadows for user pointers,
which should not happen.

Handle the futex operations like they copy a value (old) from user
space to kernel space.

Acked-by: Ilya Leoshkevich 
Signed-off-by: Heiko Carstens 
Signed-off-by: Alexander Gordeev

s390/futex: Cleanup futex_atomic_cmpxchg_inatomic()

2025-01-26T16:24:07+00:00

Cleanup the futex_atomic_cmpxchg_inatomic() inline assembly to make it
a bit more readable.

Acked-by: Ilya Leoshkevich 
Signed-off-by: Heiko Carstens 
Signed-off-by: Alexander Gordeev

s390/futex: Generate futex atomic op functions

2025-01-26T16:24:07+00:00

Cleanup the futex atomic op inline assembly and generate a function for
each futex atomic op. This makes the code hopefully a bit more readable.

Acked-by: Ilya Leoshkevich 
Signed-off-by: Heiko Carstens 
Signed-off-by: Alexander Gordeev

s390/futex: Fix FUTEX_OP_ANDN implementation

2025-01-15T16:12:31+00:00

The futex operation FUTEX_OP_ANDN is supposed to implement

*(int *)UADDR2 &= ~OPARG;

The s390 implementation just implements an AND instead of ANDN.
Add the missing bitwise not operation to oparg to fix this.

This is broken since nearly 19 years, so it looks like user space is
not making use of this operation.

Fixes: 3363fbdd6fb4 ("[PATCH] s390: futex atomic operations")
Cc: stable@vger.kernel.org
Signed-off-by: Heiko Carstens 
Acked-by: Alexander Gordeev 
Signed-off-by: Alexander Gordeev

s390/futex: add missing EX_TABLE entry to __futex_atomic_op()

2022-10-26T12:47:31+00:00

For some exception types the instruction address points behind the
instruction that caused the exception. Take that into account and add
the missing exception table entry.

Cc: 
Reviewed-by: Vasily Gorbik 
Signed-off-by: Heiko Carstens 
Signed-off-by: Vasily Gorbik

s390/extable: move EX_TABLE define to asm-extable.h

2022-03-07T23:33:00+00:00

Follow arm64 and riscv and move the EX_TABLE define to asm-extable.h
which is a lot less generic than the current linkage.h.

Also make sure that all files which contain EX_TABLE usages actually
include the new header file. This should make sure that the files
always compile and there won't be any random compile breakage due to
other header file dependencies.

Reviewed-by: Alexander Gordeev 
Signed-off-by: Heiko Carstens 
Signed-off-by: Vasily Gorbik

s390/mm: remove set_fs / rework address space handling

2020-11-23T11:01:12+00:00

Remove set_fs support from s390. With doing this rework address space
handling and simplify it. As a result address spaces are now setup
like this:

CPU running in              | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE
----------------------------|-----------|-----------|-----------
user space                  |  user     |  user     |  kernel
kernel, normal execution    |  kernel   |  user     |  kernel
kernel, kvm guest execution |  gmap     |  user     |  kernel

To achieve this the getcpu vdso syscall is removed in order to avoid
secondary address mode and a separate vdso address space in for user
space. The getcpu vdso syscall will be implemented differently with a
subsequent patch.

The kernel accesses user space always via secondary address space.
This happens in different ways:
- with mvcos in home space mode and directly read/write to secondary
  address space
- with mvcs/mvcp in primary space mode and copy from primary space to
  secondary space or vice versa
- with e.g. cs in secondary space mode and access secondary space

Switching translation modes happens with sacf before and after
instructions which access user space, like before.

Lazy handling of control register reloading is removed in the hope to
make everything simpler, but at the cost of making kernel entry and
exit a bit slower. That is: on kernel entry the primary asce is always
changed to contain the kernel asce, and on kernel exit the primary
asce is changed again so it contains the user asce.

In kernel mode there is only one exception to the primary asce: when
kvm guests are executed the primary asce contains the gmap asce (which
describes the guest address space). The primary asce is reset to
kernel asce whenever kvm guest execution is interrupted, so that this
doesn't has to be taken into account for any user space accesses.

Reviewed-by: Sven Schnelle 
Signed-off-by: Heiko Carstens

[parisc, s390, sparc64] no need for access_ok() in futex handling

2020-03-28T03:58:52+00:00

access_ok() is always true on those

Signed-off-by: Al Viro

futex: arch_futex_atomic_op_inuser() calling conventions change

2020-03-28T03:58:51+00:00

Move access_ok() in and pagefault_enable()/pagefault_disable() out.
Mechanical conversion only - some instances don't really need
a separate access_ok() at all (e.g. the ones only using
get_user()/put_user(), or architectures where access_ok()
is always true); we'll deal with that in followups.

Signed-off-by: Al Viro