linux-toradex.git/kernel/locking/rwsem.c, branch v5.13-rc3

locking: Fix typos in comments

2021-03-22T01:45:52+00:00

Fix ~16 single-word typos in locking code comments.

Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Paul E. McKenney 
Cc: Will Deacon 
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar

locking/rwsem: Fix comment typo

2021-03-17T08:34:39+00:00

s/folowing/following/

Signed-off-by: Bhaskar Chowdhury 
Signed-off-by: Ingo Molnar 
Acked-by: Randy Dunlap 
Link: https://lore.kernel.org/r/20210317041806.4096156-1-unixbhaskar@gmail.com

kernel: delete repeated words in comments

2021-02-26T17:41:03+00:00

Drop repeated words in kernel/events/.
{if, the, that, with, time}

Drop repeated words in kernel/locking/.
{it, no, the}

Drop repeated words in kernel/sched/.
{in, not}

Link: https://lkml.kernel.org/r/20210127023412.26292-1-rdunlap@infradead.org
Signed-off-by: Randy Dunlap 
Acked-by: Will Deacon 	[kernel/locking/]
Cc: Peter Zijlstra 
Cc: Ingo Molnar 
Cc: Arnaldo Carvalho de Melo 
Cc: Will Deacon 
Cc: Mathieu Desnoyers 
Cc: "Paul E. McKenney" 
Cc: Juri Lelli 
Cc: Vincent Guittot 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

locking/rwsem: Remove reader optimistic spinning

2020-12-09T16:08:48+00:00

Reader optimistic spinning is helpful when the reader critical section
is short and there aren't that many readers around. It also improves
the chance that a reader can get the lock as writer optimistic spinning
disproportionally favors writers much more than readers.

Since commit d3681e269fff ("locking/rwsem: Wake up almost all readers
in wait queue"), all the waiting readers are woken up so that they can
all get the read lock and run in parallel. When the number of contending
readers is large, allowing reader optimistic spinning will likely cause
reader fragmentation where multiple smaller groups of readers can get
the read lock in a sequential manner separated by writers. That reduces
reader parallelism.

One possible way to address that drawback is to limit the number of
readers (preferably one) that can do optimistic spinning. These readers
act as representatives of all the waiting readers in the wait queue as
they will wake up all those waiting readers once they get the lock.

Alternatively, as reader optimistic lock stealing has already enhanced
fairness to readers, it may be easier to just remove reader optimistic
spinning and simplifying the optimistic spinning code as a result.

Performance measurements (locking throughput kops/s) using a locking
microbenchmark with 50/50 reader/writer distribution and turbo-boost
disabled was done on a 2-socket Cascade Lake system (48-core 96-thread)
to see the impacts of these changes:

  1) Vanilla     - 5.10-rc3 kernel
  2) Before      - 5.10-rc3 kernel with previous patches in this series
  2) limit-rspin - 5.10-rc3 kernel with limited reader spinning patch
  3) no-rspin    - 5.10-rc3 kernel with reader spinning disabled

  # of threads  CS Load   Vanilla  Before   limit-rspin   no-rspin
  ------------  -------   -------  ------   -----------   --------
       2            1      5,185    5,662      5,214       5,077
       4            1      5,107    4,983      5,188       4,760
       8            1      4,782    4,564      4,720       4,628
      16            1      4,680    4,053      4,567       3,402
      32            1      4,299    1,115      1,118       1,098
      64            1      3,218      983      1,001         957
      96            1      1,938      944        957         930

       2           20      2,008    2,128      2,264       1,665
       4           20      1,390    1,033      1,046       1,101
       8           20      1,472    1,155      1,098       1,213
      16           20      1,332    1,077      1,089       1,122
      32           20        967      914        917         980
      64           20        787      874        891         858
      96           20        730      836        847         844

       2          100        372      356        360         355
       4          100        492      425        434         392
       8          100        533      537        529         538
      16          100        548      572        568         598
      32          100        499      520        527         537
      64          100        466      517        526         512
      96          100        406      497        506         509

The column "CS Load" represents the number of pause instructions issued
in the locking critical section. A CS load of 1 is extremely short and
is not likey in real situations. A load of 20 (moderate) and 100 (long)
are more realistic.

It can be seen that the previous patches in this series have reduced
performance in general except in highly contended cases with moderate
or long critical sections that performance improves a bit. This change
is mostly caused by the "Prevent potential lock starvation" patch that
reduce reader optimistic spinning and hence reduce reader fragmentation.

The patch that further limit reader optimistic spinning doesn't seem to
have too much impact on overall performance as shown in the benchmark
data.

The patch that disables reader optimistic spinning shows reduced
performance at lightly loaded cases, but comparable or slightly better
performance on with heavier contention.

This patch just removes reader optimistic spinning for now. As readers
are not going to do optimistic spinning anymore, we don't need to
consider if the OSQ is empty or not when doing lock stealing.

Signed-off-by: Waiman Long 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Davidlohr Bueso 
Link: https://lkml.kernel.org/r/20201121041416.12285-6-longman@redhat.com

locking/rwsem: Enable reader optimistic lock stealing

2020-12-09T16:08:48+00:00

If the optimistic spinning queue is empty and the rwsem does not have
the handoff or write-lock bits set, it is actually not necessary to
call rwsem_optimistic_spin() to spin on it. Instead, it can steal the
lock directly as its reader bias is in the count already.  If it is
the first reader in this state, it will try to wake up other readers
in the wait queue.

With this patch applied, the following were the lock event counts
after rebooting a 2-socket system and a "make -j96" kernel rebuild.

  rwsem_opt_rlock=4437
  rwsem_rlock=29
  rwsem_rlock_steal=19

So lock stealing represents about 0.4% of all the read locks acquired
in the slow path.

Signed-off-by: Waiman Long 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Davidlohr Bueso 
Link: https://lkml.kernel.org/r/20201121041416.12285-4-longman@redhat.com

locking/rwsem: Prevent potential lock starvation

2020-12-09T16:08:48+00:00

The lock handoff bit is added in commit 4f23dbc1e657 ("locking/rwsem:
Implement lock handoff to prevent lock starvation") to avoid lock
starvation. However, allowing readers to do optimistic spinning does
introduce an unlikely scenario where lock starvation can happen.

The lock handoff bit may only be set when a waiter is being woken up.
In the case of reader unlock, wakeup happens only when the reader count
reaches 0. If there is a continuous stream of incoming readers acquiring
read lock via optimistic spinning, it is possible that the reader count
may never reach 0 and so the handoff bit will never be asserted.

One way to prevent this scenario from happening is to disallow optimistic
spinning if the rwsem is currently owned by readers. If the previous
or current owner is a writer, optimistic spinning will be allowed.

If the previous owner is a reader but the reader count has reached 0
before, a wakeup should have been issued. So the handoff mechanism
will be kicked in to prevent lock starvation. As a result, it should
be OK to do optimistic spinning in this case.

This patch may have some impact on reader performance as it reduces
reader optimistic spinning especially if the lock critical sections
are short the number of contending readers are small.

Signed-off-by: Waiman Long 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Davidlohr Bueso 
Link: https://lkml.kernel.org/r/20201121041416.12285-3-longman@redhat.com

locking/rwsem: Pass the current atomic count to rwsem_down_read_slowpath()

2020-12-09T16:08:47+00:00

The atomic count value right after reader count increment can be useful
to determine the rwsem state at trylock time. So the count value is
passed down to rwsem_down_read_slowpath() to be used when appropriate.

Signed-off-by: Waiman Long 
Signed-off-by: Peter Zijlstra (Intel) 
Reviewed-by: Davidlohr Bueso 
Link: https://lkml.kernel.org/r/20201121041416.12285-2-longman@redhat.com

locking/rwsem: Fold __down_{read,write}*()

2020-12-09T16:08:47+00:00

There's a lot needless duplication in __down_{read,write}*(), cure
that with a helper.

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20201207090243.GE3040@hirez.programming.kicks-ass.net

locking/rwsem: Introduce rwsem_write_trylock()

2020-12-09T16:08:47+00:00

One copy of this logic is better than three.

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20201207090243.GE3040@hirez.programming.kicks-ass.net

locking/rwsem: Better collate rwsem_read_trylock()

2020-12-09T16:08:47+00:00

All users of rwsem_read_trylock() do rwsem_set_reader_owned(sem) on
success, move it into rwsem_read_trylock() proper.

Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20201207090243.GE3040@hirez.programming.kicks-ass.net