linux-toradex.git/kernel/locking/percpu-rwsem.c, branch v5.17-rc4

locking/percpu-rwsem: Use this_cpu_{inc,dec}() for read_count

2020-09-16T14:26:56+00:00

The __this_cpu*() accessors are (in general) IRQ-unsafe which, given
that percpu-rwsem is a blocking primitive, should be just fine.

However, file_end_write() is used from IRQ context and will cause
load-store issues on architectures where the per-cpu accessors are not
natively irq-safe.

Fix it by using the IRQ-safe this_cpu_*() for operations on
read_count. This will generate more expensive code on a number of
platforms, which might cause a performance regression for some of the
other percpu-rwsem users.

If any such is reported, we can consider alternative solutions.

Fixes: 70fe2f48152e ("aio: fix freeze protection of aio writes")
Signed-off-by: Hou Tao 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Acked-by: Oleg Nesterov 
Link: https://lkml.kernel.org/r/20200915140750.137881-1-houtao1@huawei.com

locking/percpu-rwsem: Fix a task_struct refcount

2020-04-08T10:05:06+00:00

The following commit:

  7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")

introduced task_struct memory leaks due to messing up the task_struct
refcount.

At the beginning of percpu_rwsem_wake_function(), it calls get_task_struct(),
but if the trylock failed, it will remain in the waitqueue. However, it
will run percpu_rwsem_wake_function() again with get_task_struct() to
increase the refcount but then only call put_task_struct() once the trylock
succeeded.

Fix it by adjusting percpu_rwsem_wake_function() a bit to guard against
when percpu_rwsem_wait() observing !private, terminating the wait and
doing a quick exit() while percpu_rwsem_wake_function() then doing
wake_up_process(p) as a use-after-free.

Fixes: 7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem")
Suggested-by: Peter Zijlstra 
Signed-off-by: Qian Cai 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Link: https://lkml.kernel.org/r/20200330213002.2374-1-cai@lca.pw

rcuwait: Add @state argument to rcuwait_wait_event()

2020-03-21T15:00:22+00:00

Extend rcuwait_wait_event() with a state variable so that it is not
restricted to UNINTERRUPTIBLE waits.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Thomas Gleixner 
Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lkml.kernel.org/r/20200321113241.824030968@linutronix.de

locking/percpu-rwsem: Add might_sleep() for writer locking

2020-02-11T12:11:02+00:00

We are missing this annotation in percpu_down_write(). Correct
this.

Signed-off-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Link: https://lkml.kernel.org/r/20200108013305.7732-1-dave@stgolabs.net

locking/percpu-rwsem: Fold __percpu_up_read()

2020-02-11T12:10:58+00:00

Now that __percpu_up_read() is only ever used from percpu_up_read()
merge them, it's a small function.

Signed-off-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Link: https://lkml.kernel.org/r/20200131151540.212415454@infradead.org

locking/percpu-rwsem: Remove the embedded rwsem

2020-02-11T12:10:56+00:00

The filesystem freezer uses percpu-rwsem in a way that is effectively
write_non_owner() and achieves this with a few horrible hacks that
rely on the rwsem (!percpu) implementation.

When PREEMPT_RT replaces the rwsem implementation with a PI aware
variant this comes apart.

Remove the embedded rwsem and implement it using a waitqueue and an
atomic_t.

 - make readers_block an atomic, and use it, with the waitqueue
   for a blocking test-and-set write-side.

 - have the read-side wait for the 'lock' state to clear.

Have the waiters use FIFO queueing and mark them (reader/writer) with
a new WQ_FLAG. Use a custom wake_function to wake either a single
writer or all readers until a writer.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Davidlohr Bueso 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Tested-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20200204092403.GB14879@hirez.programming.kicks-ass.net

locking/percpu-rwsem: Extract __percpu_down_read_trylock()

2020-02-11T12:10:55+00:00

In preparation for removing the embedded rwsem and building a custom
lock, extract the read-trylock primitive.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Davidlohr Bueso 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Tested-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20200131151540.098485539@infradead.org

locking/percpu-rwsem: Move __this_cpu_inc() into the slowpath

2020-02-11T12:10:54+00:00

As preparation to rework __percpu_down_read() move the
__this_cpu_inc() into it.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Davidlohr Bueso 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Tested-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20200131151540.041600199@infradead.org

locking/percpu-rwsem: Convert to bool

2020-02-11T12:10:54+00:00

Use bool where possible.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Davidlohr Bueso 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Tested-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20200131151539.984626569@infradead.org

locking/percpu-rwsem, lockdep: Make percpu-rwsem use its own lockdep_map

2020-02-11T12:10:53+00:00

As preparation for replacing the embedded rwsem, give percpu-rwsem its
own lockdep_map.

Signed-off-by: Peter Zijlstra (Intel) 
Signed-off-by: Ingo Molnar 
Reviewed-by: Davidlohr Bueso 
Acked-by: Will Deacon 
Acked-by: Waiman Long 
Tested-by: Juri Lelli 
Link: https://lkml.kernel.org/r/20200131151539.927625541@infradead.org