linux-toradex.git/kernel/locking/qspinlock_paravirt.h, branch v7.1-rc7

locking/pvqspinlock: Convert fields of 'enum vcpu_state' to uppercase

2024-10-18T04:21:16+00:00

Convert the fields of 'enum vcpu_state' to uppercase for better
readability. No functional changes intended.

Acked-by: Waiman Long 
Signed-off-by: Qiuxu Zhuo 
Signed-off-by: Boqun Feng 
Link: https://lore.kernel.org/r/20240809014802.15320-1-qiuxu.zhuo@intel.com

locking/pvqspinlock: Correct the type of "old" variable in pv_kick_node()

2024-07-29T10:16:21+00:00

"enum vcpu_state" is not compatible with "u8" type for all targets,
resulting in:

error: initialization of 'u8 *' {aka 'unsigned char *'} from incompatible pointer type 'enum vcpu_state *'

for LoongArch. Correct the type of "old" variable to "u8".

Fixes: fea0e1820b51 ("locking/pvqspinlock: Use try_cmpxchg() in qspinlock_paravirt.h")
Closes: https://lore.kernel.org/lkml/20240719024010.3296488-1-maobibo@loongson.cn/
Reported-by: Bibo Mao 
Signed-off-by: Uros Bizjak 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Waiman Long 
Link: https://lore.kernel.org/r/20240721164552.50175-1-ubizjak@gmail.com

locking/pvqspinlock: Use try_cmpxchg() in qspinlock_paravirt.h

2024-04-12T09:42:39+00:00

Use try_cmpxchg(*ptr, &old, new) instead of
cmpxchg(*ptr, old, new) == old in qspinlock_paravirt.h
x86 CMPXCHG instruction returns success in ZF flag, so
this change saves a compare after cmpxchg.

No functional change intended.

Signed-off-by: Uros Bizjak 
Signed-off-by: Ingo Molnar 
Reviewed-by: Waiman Long 
Cc: Linus Torvalds 
Link: https://lore.kernel.org/r/20240411192317.25432-2-ubizjak@gmail.com

locking/pvqspinlock: Use try_cmpxchg_acquire() in trylock_clear_pending()

2024-04-12T09:40:51+00:00

Replace this pattern in trylock_clear_pending():

    cmpxchg_acquire(*ptr, old, new) == old

... with the simpler and faster:

    try_cmpxchg_acquire(*ptr, &old, new)

The x86 CMPXCHG instruction returns success in the ZF flag, so this change
saves a compare after the CMPXCHG.

Also change the return type of the function to bool and streamline
the control flow in the _Q_PENDING_BITS == 8 variant a bit.

No functional change intended.

Signed-off-by: Uros Bizjak 
Signed-off-by: Ingo Molnar 
Reviewed-by: Waiman Long 
Reviewed-by: Linus Torvalds 
Link: https://lore.kernel.org/r/20240325140943.815051-1-ubizjak@gmail.com

locking/qspinlock: Always evaluate lockevent* non-event parameter once

2024-03-21T19:45:17+00:00

The 'inc' parameter of lockevent_add() and the cond parameter of
lockevent_cond_inc() are only evaluated when CONFIG_LOCK_EVENT_COUNTS
is on. That can cause problem if those parameters are expressions
with side effect like a "++". Fix this by evaluating those non-event
parameters once even if CONFIG_LOCK_EVENT_COUNTS is off. This will also
eliminate the need of the __maybe_unused attribute to the wait_early
local variable in pv_wait_node().

Suggested-by: Ingo Molnar 
Signed-off-by: Waiman Long 
Signed-off-by: Ingo Molnar 
Reviewed-by: Boqun Feng 
Link: https://lore.kernel.org/r/20240319005004.1692705-1-longman@redhat.com

locking/qspinlock: Fix 'wait_early' set but not used warning

2024-02-28T12:08:37+00:00

When CONFIG_LOCK_EVENT_COUNTS is off, the wait_early variable will be
set but not used. This is expected. Recent compilers will not generate
wait_early code in this case.

Add the __maybe_unused attribute to wait_early for suppressing this
W=1 warning.

Reported-by: kernel test robot 
Signed-off-by: Waiman Long 
Signed-off-by: Ingo Molnar 
Reviewed-by: Boqun Feng 
Cc: Linus Torvalds 
Link: https://lore.kernel.org/r/20240222150540.79981-2-longman@redhat.com

Closes: https://lore.kernel.org/oe-kbuild-all/202312260422.f4pK3f9m-lkp@intel.com/

x86/qspinlock-paravirt: Fix missing-prototype warning

2023-08-03T15:15:05+00:00

__pv_queued_spin_unlock_slowpath() is defined in a header file as
a global function, and designed to be called from inline asm, but
there is no prototype visible in the definition:

  kernel/locking/qspinlock_paravirt.h:493:1: error: no previous \
    prototype for '__pv_queued_spin_unlock_slowpath' [-Werror=missing-prototypes]

Add this to the x86 header that contains the inline asm calling it,
and ensure this gets included before the definition, rather than
after it.

Signed-off-by: Arnd Bergmann 
Signed-off-by: Borislav Petkov (AMD) 
Link: https://lore.kernel.org/r/20230803082619.1369127-8-arnd@kernel.org

locking: Add __lockfunc to slow path functions

2022-08-19T17:47:51+00:00

So that we can skip the functions in the perf lock contention and other
places like /proc/PID/wchan.

Signed-off-by: Namhyung Kim 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Waiman Long 
Link: https://lore.kernel.org/r/20220810220346.1919485-1-namhyung@kernel.org

Revert "locking/pvqspinlock: Don't wait if vCPU is preempted"

2019-09-25T08:22:37+00:00

This patch reverts commit 75437bb304b20 (locking/pvqspinlock: Don't
wait if vCPU is preempted).  A large performance regression was caused
by this commit.  on over-subscription scenarios.

The test was run on a Xeon Skylake box, 2 sockets, 40 cores, 80 threads,
with three VMs of 80 vCPUs each.  The score of ebizzy -M is reduced from
13000-14000 records/s to 1700-1800 records/s:

          Host                Guest                score

vanilla w/o kvm optimizations     upstream    1700-1800 records/s
vanilla w/o kvm optimizations     revert      13000-14000 records/s
vanilla w/ kvm optimizations      upstream    4500-5000 records/s
vanilla w/ kvm optimizations      revert      14000-15500 records/s

Exit from aggressive wait-early mechanism can result in premature yield
and extra scheduling latency.

Actually, only 6% of wait_early events are caused by vcpu_is_preempted()
being true.  However, when one vCPU voluntarily releases its vCPU, all
the subsequently waiters in the queue will do the same and the cascading
effect leads to bad performance.

kvm optimizations:
[1] commit d73eb57b80b (KVM: Boost vCPUs that are delivering interrupts)
[2] commit 266e85a5ec9 (KVM: X86: Boost queue head vCPU to mitigate lock waiter preemption)

Tested-by: loobinliu@tencent.com
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: Waiman Long 
Cc: Paolo Bonzini 
Cc: Radim Krčmář 
Cc: loobinliu@tencent.com
Cc: stable@vger.kernel.org
Fixes: 75437bb304b20 (locking/pvqspinlock: Don't wait if vCPU is preempted)
Signed-off-by: Wanpeng Li 
Signed-off-by: Paolo Bonzini

locking/qspinlock_stat: Introduce generic lockevent_*() counting APIs

2019-04-10T08:56:03+00:00

The percpu event counts used by qspinlock code can be useful for
other locking code as well. So a new set of lockevent_* counting APIs
is introduced with the lock event names extracted out into the new
lock_events_list.h header file for easier addition in the future.

The existing qstat_inc() calls are replaced by either lockevent_inc() or
lockevent_cond_inc() calls.

The qstat_hop() call is renamed to lockevent_pv_hop(). The "reset_counters"
debugfs file is also renamed to ".reset_counts".

Signed-off-by: Waiman Long 
Acked-by: Peter Zijlstra 
Acked-by: Davidlohr Bueso 
Cc: Andrew Morton 
Cc: Arnd Bergmann 
Cc: Borislav Petkov 
Cc: Davidlohr Bueso 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Tim Chen 
Cc: Will Deacon 
Link: http://lkml.kernel.org/r/20190404174320.22416-8-longman@redhat.com
Signed-off-by: Ingo Molnar