linux-toradex.git/kernel/locking/mutex.c, branch v3.17-rc6

arch, locking: Ciao arch_mutex_cpu_relax()

2014-07-17T10:32:47+00:00

The arch_mutex_cpu_relax() function, introduced by 34b133f, is
hacky and ugly. It was added a few years ago to address the fact
that common cpu_relax() calls include yielding on s390, and thus
impact the optimistic spinning functionality of mutexes. Nowadays
we use this function well beyond mutexes: rwsem, qrwlock, mcs and
lockref. Since the macro that defines the call is in the mutex header,
any users must include mutex.h and the naming is misleading as well.

This patch (i) renames the call to cpu_relax_lowlatency  ("relax, but
only if you can do it with very low latency") and (ii) defines it in
each arch's asm/processor.h local header, just like for regular cpu_relax
functions. On all archs, except s390, cpu_relax_lowlatency is simply cpu_relax,
and thus we can take it out of mutex.h. While this can seem redundant,
I believe it is a good choice as it allows us to move out arch specific
logic from generic locking primitives and enables future(?) archs to
transparently define it, similarly to System Z.

Signed-off-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra 
Cc: Andrew Morton 
Cc: Anton Blanchard 
Cc: Aurelien Jacquiot 
Cc: Benjamin Herrenschmidt 
Cc: Bharat Bhushan 
Cc: Catalin Marinas 
Cc: Chen Liqin 
Cc: Chris Metcalf 
Cc: Christian Borntraeger 
Cc: Chris Zankel 
Cc: David Howells 
Cc: David S. Miller 
Cc: Deepthi Dharwar 
Cc: Dominik Dingel 
Cc: Fenghua Yu 
Cc: Geert Uytterhoeven 
Cc: Guan Xuetao 
Cc: Haavard Skinnemoen 
Cc: Hans-Christian Egtvedt 
Cc: Heiko Carstens 
Cc: Helge Deller 
Cc: Hirokazu Takata 
Cc: Ivan Kokshaysky 
Cc: James E.J. Bottomley 
Cc: James Hogan 
Cc: Jason Wang 
Cc: Jesper Nilsson 
Cc: Joe Perches 
Cc: Jonas Bonn 
Cc: Joseph Myers 
Cc: Kees Cook 
Cc: Koichi Yasutake 
Cc: Lennox Wu 
Cc: Linus Torvalds 
Cc: Mark Salter 
Cc: Martin Schwidefsky 
Cc: Matt Turner 
Cc: Max Filippov 
Cc: Michael Neuling 
Cc: Michal Simek 
Cc: Mikael Starvik 
Cc: Nicolas Pitre 
Cc: Paolo Bonzini 
Cc: Paul Burton 
Cc: Paul E. McKenney 
Cc: Paul Gortmaker 
Cc: Paul Mackerras 
Cc: Qais Yousef 
Cc: Qiaowei Ren 
Cc: Rafael Wysocki 
Cc: Ralf Baechle 
Cc: Richard Henderson 
Cc: Richard Kuo 
Cc: Russell King 
Cc: Steven Miao 
Cc: Steven Rostedt 
Cc: Stratos Karafotis 
Cc: Tim Chen 
Cc: Tony Luck 
Cc: Vasily Kulikov 
Cc: Vineet Gupta 
Cc: Vineet Gupta 
Cc: Waiman Long 
Cc: Will Deacon 
Cc: Wolfram Sang 
Cc: adi-buildroot-devel@lists.sourceforge.net
Cc: linux390@de.ibm.com
Cc: linux-alpha@vger.kernel.org
Cc: linux-am33-list@redhat.com
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-c6x-dev@linux-c6x.org
Cc: linux-cris-kernel@axis.com
Cc: linux-hexagon@vger.kernel.org
Cc: linux-ia64@vger.kernel.org
Cc: linux@lists.openrisc.net
Cc: linux-m32r-ja@ml.linux-m32r.org
Cc: linux-m32r@ml.linux-m32r.org
Cc: linux-m68k@lists.linux-m68k.org
Cc: linux-metag@vger.kernel.org
Cc: linux-mips@linux-mips.org
Cc: linux-parisc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-s390@vger.kernel.org
Cc: linux-sh@vger.kernel.org
Cc: linux-xtensa@linux-xtensa.org
Cc: sparclinux@vger.kernel.org
Link: http://lkml.kernel.org/r/1404079773.2619.4.camel@buesod1.americas.hpqcorp.net
Signed-off-by: Ingo Molnar

Merge branch 'locking/urgent' into locking/core, before applying larger changes and to refresh the branch with fixes

2014-07-17T09:45:29+00:00

Signed-off-by: Ingo Molnar

locking/spinlocks/mcs: Introduce and use init macro and function for osq locks

2014-07-16T11:28:05+00:00

Currently, we initialize the osq lock by directly setting the lock's values. It
would be preferable if we use an init macro to do the initialization like we do
with other locks.

This patch introduces and uses a macro and function for initializing the osq lock.

Signed-off-by: Jason Low 
Signed-off-by: Peter Zijlstra 
Cc: Scott Norton 
Cc: "Paul E. McKenney" 
Cc: Dave Chinner 
Cc: Waiman Long 
Cc: Davidlohr Bueso 
Cc: Rik van Riel 
Cc: Andrew Morton 
Cc: "H. Peter Anvin" 
Cc: Steven Rostedt 
Cc: Tim Chen 
Cc: Konrad Rzeszutek Wilk 
Cc: Aswin Chandramouleeswaran 
Cc: Linus Torvalds 
Cc: Chris Mason 
Cc: Josef Bacik 
Link: http://lkml.kernel.org/r/1405358872-3732-4-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

locking/spinlocks/mcs: Convert osq lock to atomic_t to reduce overhead

2014-07-16T11:28:04+00:00

The cancellable MCS spinlock is currently used to queue threads that are
doing optimistic spinning. It uses per-cpu nodes, where a thread obtaining
the lock would access and queue the local node corresponding to the CPU that
it's running on. Currently, the cancellable MCS lock is implemented by using
pointers to these nodes.

In this patch, instead of operating on pointers to the per-cpu nodes, we
store the CPU numbers in which the per-cpu nodes correspond to in atomic_t.
A similar concept is used with the qspinlock.

By operating on the CPU # of the nodes using atomic_t instead of pointers
to those nodes, this can reduce the overhead of the cancellable MCS spinlock
by 32 bits (on 64 bit systems).

Signed-off-by: Jason Low 
Signed-off-by: Peter Zijlstra 
Cc: Scott Norton 
Cc: "Paul E. McKenney" 
Cc: Dave Chinner 
Cc: Waiman Long 
Cc: Davidlohr Bueso 
Cc: Rik van Riel 
Cc: Andrew Morton 
Cc: "H. Peter Anvin" 
Cc: Steven Rostedt 
Cc: Tim Chen 
Cc: Konrad Rzeszutek Wilk 
Cc: Aswin Chandramouleeswaran 
Cc: Linus Torvalds 
Cc: Chris Mason 
Cc: Heiko Carstens 
Cc: Josef Bacik 
Link: http://lkml.kernel.org/r/1405358872-3732-3-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

locking/mutexes: Optimize mutex trylock slowpath

2014-07-05T09:25:42+00:00

The mutex_trylock() function calls into __mutex_trylock_fastpath() when
trying to obtain the mutex. On 32 bit x86, in the !__HAVE_ARCH_CMPXCHG
case, __mutex_trylock_fastpath() calls directly into __mutex_trylock_slowpath()
regardless of whether or not the mutex is locked.

In __mutex_trylock_slowpath(), we then acquire the wait_lock spinlock, xchg()
lock->count with -1, then set lock->count back to 0 if there are no waiters,
and return true if the prev lock count was 1.

However, if the mutex is already locked, then there isn't much point
in attempting all of the above expensive operations. In this patch, we only
attempt the above trylock operations if the mutex is unlocked.

Signed-off-by: Jason Low 
Reviewed-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra 
Cc: akpm@linux-foundation.org
Cc: tim.c.chen@linux.intel.com
Cc: paulmck@linux.vnet.ibm.com
Cc: rostedt@goodmis.org
Cc: Waiman.Long@hp.com
Cc: scott.norton@hp.com
Cc: aswin@hp.com
Cc: Linus Torvalds 
Link: http://lkml.kernel.org/r/1402511843-4721-5-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

locking/mutexes: Try to acquire mutex only if it is unlocked

2014-07-05T09:25:42+00:00

Upon entering the slowpath in __mutex_lock_common(), we try once more to
acquire the mutex. We only try to acquire if (lock->count >= 0). However,
what we actually want here is to try to acquire if the mutex is unlocked
(lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering
the slowpath if it is unlocked, rather than if the lock count is non-negative.
This helps further reduce unnecessary atomic xchg() operations.

Furthermore, this patch uses !mutex_is_locked(lock) to do the initial
checks for if the lock is free rather than directly calling atomic_read()
on the lock->count, in order to improve readability.

Signed-off-by: Jason Low 
Acked-by: Waiman Long 
Signed-off-by: Peter Zijlstra 
Cc: akpm@linux-foundation.org
Cc: tim.c.chen@linux.intel.com
Cc: paulmck@linux.vnet.ibm.com
Cc: rostedt@goodmis.org
Cc: davidlohr@hp.com
Cc: scott.norton@hp.com
Cc: aswin@hp.com
Cc: Linus Torvalds 
Link: http://lkml.kernel.org/r/1402511843-4721-4-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

locking/mutexes: Delete the MUTEX_SHOW_NO_WAITER macro

2014-07-05T09:25:41+00:00

MUTEX_SHOW_NO_WAITER() is a macro which checks for if there are
"no waiters" on a mutex by checking if the lock count is non-negative.
Based on feedback from the discussion in the earlier version of this
patchset, the macro is not very readable.

Furthermore, checking lock->count isn't always the correct way to
determine if there are "no waiters" on a mutex. For example, a negative
count on a mutex really only means that there "potentially" are
waiters. Likewise, there can be waiters on the mutex even if the count is
non-negative. Thus, "MUTEX_SHOW_NO_WAITER" doesn't always do what the name
of the macro suggests.

So this patch deletes the MUTEX_SHOW_NO_WAITERS() macro, directly
use atomic_read() instead of the macro, and adds comments which
elaborate on how the extra atomic_read() checks can help reduce
unnecessary xchg() operations.

Signed-off-by: Jason Low 
Acked-by: Waiman Long 
Signed-off-by: Peter Zijlstra 
Cc: akpm@linux-foundation.org
Cc: tim.c.chen@linux.intel.com
Cc: paulmck@linux.vnet.ibm.com
Cc: rostedt@goodmis.org
Cc: davidlohr@hp.com
Cc: scott.norton@hp.com
Cc: aswin@hp.com
Cc: Linus Torvalds 
Link: http://lkml.kernel.org/r/1402511843-4721-3-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

locking/mutexes: Correct documentation on mutex optimistic spinning

2014-07-05T09:25:41+00:00

The mutex optimistic spinning documentation states that we spin for
acquisition when we find that there are no pending waiters. However,
in actuality, whether or not there are waiters for the mutex doesn't
determine if we will spin for it.

This patch removes that statement and also adds a comment which
mentions that we spin for the mutex while we don't need to reschedule.

Signed-off-by: Jason Low 
Acked-by: Davidlohr Bueso 
Signed-off-by: Peter Zijlstra 
Cc: akpm@linux-foundation.org
Cc: tim.c.chen@linux.intel.com
Cc: paulmck@linux.vnet.ibm.com
Cc: rostedt@goodmis.org
Cc: Waiman.Long@hp.com
Cc: scott.norton@hp.com
Cc: aswin@hp.com
Cc: Linus Torvalds 
Link: http://lkml.kernel.org/r/1402511843-4721-2-git-send-email-jason.low2@hp.com
Signed-off-by: Ingo Molnar

Merge branch 'x86-asmlinkage-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2014-03-31T21:13:25+00:00

Pull x86 LTO changes from Peter Anvin:
 "More infrastructure work in preparation for link-time optimization
  (LTO).  Most of these changes is to make sure symbols accessed from
  assembly code are properly marked as visible so the linker doesn't
  remove them.

  My understanding is that the changes to support LTO are still not
  upstream in binutils, but are on the way there.  This patchset should
  conclude the x86-specific changes, and remaining patches to actually
  enable LTO will be fed through the Kbuild tree (other than keeping up
  with changes to the x86 code base, of course), although not
  necessarily in this merge window"

* 'x86-asmlinkage-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (25 commits)
  Kbuild, lto: Handle basic LTO in modpost
  Kbuild, lto: Disable LTO for asm-offsets.c
  Kbuild, lto: Add a gcc-ld script to let run gcc as ld
  Kbuild, lto: add ld-version and ld-ifversion macros
  Kbuild, lto: Drop .number postfixes in modpost
  Kbuild, lto, workaround: Don't warn for initcall_reference in modpost
  lto: Disable LTO for sys_ni
  lto: Handle LTO common symbols in module loader
  lto, workaround: Add workaround for initcall reordering
  lto: Make asmlinkage __visible
  x86, lto: Disable LTO for the x86 VDSO
  initconst, x86: Fix initconst mistake in ts5500 code
  initconst: Fix initconst mistake in dcdbas
  asmlinkage: Make trace_hardirqs_on/off_caller visible
  asmlinkage, x86: Fix 32bit memcpy for LTO
  asmlinkage Make __stack_chk_failed and memcmp visible
  asmlinkage: Mark rwsem functions that can be called from assembler asmlinkage
  asmlinkage: Make main_extable_sort_needed visible
  asmlinkage, mutex: Mark __visible
  asmlinkage: Make trace_hardirq visible
  ...

locking/mutex: Fix debug checks

2014-03-12T12:49:47+00:00

OK, so commit:

  1d8fe7dc8078 ("locking/mutexes: Unlock the mutex without the wait_lock")

generates this boot warning when CONFIG_DEBUG_MUTEXES=y:

  WARNING: CPU: 0 PID: 139 at /usr/src/linux-2.6/kernel/locking/mutex-debug.c:82 debug_mutex_unlock+0x155/0x180() DEBUG_LOCKS_WARN_ON(lock->owner != current)

And that makes sense, because as soon as we release the lock a
new owner can come in...

One would think that !__mutex_slowpath_needs_to_unlock()
implementations suffer the same, but for DEBUG we fall back to
mutex-null.h which has an unconditional 1 for that.

The mutex debug code requires the mutex to be unlocked after
doing the debug checks, otherwise it can find inconsistent
state.

Reported-by: Ingo Molnar 
Signed-off-by: Peter Zijlstra 
Cc: jason.low2@hp.com
Link: http://lkml.kernel.org/r/20140312122442.GB27965@twins.programming.kicks-ass.net
Signed-off-by: Ingo Molnar