linux-toradex.git/include/linux/atomic.h, branch v5.1-rc1

locking/atomics: Switch to generated fallbacks

2018-11-01T10:00:46+00:00

As a step to ensuring the atomic* APIs are consistent, switch to fallbacks
generated by gen-atomic-fallback.sh.

These are checked in rather than generated with Kbuild, since:

* This allows inspection of the atomics with git grep and ctags on a
  pristine tree, which Linus strongly prefers being able to do.

* The fallbacks are not affected by machine details or configuration
  options, so it is not necessary to regenerate them to take these into
  account.

* These are included by files required *very* early in the build process
  (e.g. for generating bounds.h), and we'd rather not complicate the
  top-level Kbuild file with dependencies.

The new fallback header should be equivalent to the old fallbacks in
, but:

* It is formatted a little differently due to scripting ensuring things
  are more regular than they used to be.

* Fallbacks are now expanded in-place as static inline functions rather
  than macros.

* The prototypes for fallbacks are arragned consistently with the return
  type on a separate line to try to keep to a sensible line length.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Signed-off-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Cc: linux-arm-kernel@lists.infradead.org
Cc: catalin.marinas@arm.com
Cc: linuxdrivers@attotech.com
Cc: dvyukov@google.com
Cc: Boqun Feng 
Cc: arnd@arndb.de
Cc: aryabinin@virtuozzo.com
Cc: glider@google.com
Link: http://lkml.kernel.org/r/20180904104830.2975-3-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

locking/atomics: Rework ordering barriers

2018-07-25T09:53:59+00:00

Currently architectures can override __atomic_op_*() to define the barriers
used before/after a relaxed atomic when used to build acquire/release/fence
variants.

This has the unfortunate property of requiring the architecture to define the
full wrapper for the atomics, rather than just the barriers they care about,
and gets in the way of generating atomics which can be easily read.

Instead, this patch has architectures define an optional set of barriers:

* __atomic_acquire_fence()
* __atomic_release_fence()
* __atomic_pre_full_fence()
* __atomic_post_full_fence()

... which  uses to build the wrappers.

It would be nice if we could undef these, along with the __atomic_op_*()
wrappers, but that would break the cmpxchg() wrappers, which are written
in preprocessor. Undefs would have been nice, but alas.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Will Deacon 
Cc: Andrea Parri 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: andy.shevchenko@gmail.com
Cc: arnd@arndb.de
Cc: aryabinin@virtuozzo.com
Cc: catalin.marinas@arm.com
Cc: dvyukov@google.com
Cc: glider@google.com
Cc: linux-arm-kernel@lists.infradead.org
Cc: peter@hurleysoftware.com
Link: http://lkml.kernel.org/r/20180716113017.3909-7-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Clean up '*_andnot()' ifdeffery

2018-06-21T12:25:24+00:00

The ifdeffery for atomic*_{fetch_,}andnot() is unlike that for all the
other atomics. If atomic*_andnot() is not defined, the corresponding
atomic*_fetch_andnot() is assumed to not be defined.

Additionally, the fallbacks for the various ordering cases are written
much later in atomic.h as static inlines.

This isn't problematic today, but gets in the way of scripting the
generation of atomics. To prepare for scripting, this patch:

* Switches to separate ifdefs for atomic*_andnot() and
  atomic*_fetch_andnot(), updating implementations as appropriate.

* Moves the fallbacks into the standards ifdefs, as macro expansions
  rather than static inlines.

* Removes trivial andnot implementations from architectures, where these
  are superseded by core code.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-19-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make conditional inc/dec ops optional

2018-06-21T12:25:24+00:00

The conditional inc/dec ops differ for atomic_t and atomic64_t:

- atomic_inc_unless_positive() is optional for atomic_t, and doesn't exist for atomic64_t.
- atomic_dec_unless_negative() is optional for atomic_t, and doesn't exist for atomic64_t.
- atomic_dec_if_positive is optional for atomic_t, and is mandatory for atomic64_t.

Let's make these consistently optional for both. At the same time, let's
clean up the existing fallbacks to use atomic_try_cmpxchg().

The instrumented atomics are updated accordingly.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-18-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make unconditional inc/dec ops optional

2018-06-21T12:25:24+00:00

Many of the inc/dec ops are mandatory, but for most architectures inc/dec are
simply trivial wrappers around their corresponding add/sub ops.

Let's make all the inc/dec ops optional, so that we can get rid of these
boilerplate wrappers.

The instrumented atomics are updated accordingly.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Palmer Dabbelt 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make test ops optional

2018-06-21T12:25:24+00:00

Some of the atomics return the result of a test applied after the atomic
operation, and almost all architectures implement these as trivial
wrappers around the underlying atomic. Specifically:

 * _inc_and_test(v)    is (_inc_return(v)    == 0)
 * _dec_and_test(v)    is (_dec_return(v)    == 0)
 * _sub_and_test(i, v) is (_sub_return(i, v) == 0)
 * _add_negative(i, v) is (_add_return(i, v)  < 0)

Rather than have these definitions duplicated in all architectures, with
minor inconsistencies in formatting and documentation, let's make these
operations optional, with default fallbacks as above. Implementations
must now provide a preprocessor symbol.

The instrumented atomics are updated accordingly.

Both x86 and m68k have custom implementations, which are left as-is,
given preprocessor symbols to avoid being overridden.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Geert Uytterhoeven 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Palmer Dabbelt 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make atomic64_fetch_add_unless() optional

2018-06-21T12:25:24+00:00

Architectures with atomic64_fetch_add_unless() provide a preprocessor
symbol if they do so, and all other architectures have trivial C
implementations of atomic64_add_unless() which are near-identical.

Let's unify the trivial definitions of atomic64_fetch_add_unless() in
, so that we always have both
atomic64_fetch_add_unless() and atomic64_add_unless() with less
boilerplate code.

This means that atomic64_add_unless() is always implemented in core
code, and the instrumented atomics are updated accordingly.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-15-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics: Prepare for atomic64_fetch_add_unless()

2018-06-21T12:22:34+00:00

Currently all architectures must implement atomic_fetch_add_unless(),
with common code providing atomic_add_unless(). Architectures must also
implement atomic64_add_unless() directly, with no corresponding
atomic64_fetch_add_unless().

This divergence is unfortunate, and means that the APIs for atomic_t,
atomic64_t, and atomic_long_t differ.

In preparation for unifying things, with architectures providing
atomic64_fetch_add_unless, this patch adds a generic
atomic64_add_unless() which will use atomic64_fetch_add_unless(). The
instrumented atomics are updated to take this case into account.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Cc: Albert Ou 
Cc: Arnd Bergmann 
Cc: Benjamin Herrenschmidt 
Cc: Boqun Feng 
Cc: Ivan Kokshaysky 
Cc: Linus Torvalds 
Cc: Matt Turner 
Cc: Michael Ellerman 
Cc: Palmer Dabbelt 
Cc: Paul Mackerras 
Cc: Richard Henderson 
Cc: Russell King 
Cc: Thomas Gleixner 
Cc: Vineet Gupta 
Link: https://lore.kernel.org/lkml/20180621121321.4761-8-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make atomic_fetch_add_unless() optional

2018-06-21T12:22:33+00:00

Several architectures these have a near-identical implementation based
on atomic_read() and atomic_cmpxchg() which we can instead define in
, so let's do so, using something close to the existing
x86 implementation with try_cmpxchg().

Where an architecture provides its own atomic_fetch_add_unless(), it
must define a preprocessor symbol for it. The instrumented atomics are
updated accordingly.

Note that arch/arc's existing atomic_fetch_add_unless() had redundant
barriers, as these are already present in its atomic_cmpxchg()
implementation.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Geert Uytterhoeven 
Reviewed-by: Will Deacon 
Acked-by: Geert Uytterhoeven 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Palmer Dabbelt 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Cc: Vineet Gupta 
Link: https://lore.kernel.org/lkml/20180621121321.4761-7-mark.rutland@arm.com
Signed-off-by: Ingo Molnar

atomics/treewide: Make atomic64_inc_not_zero() optional

2018-06-21T12:22:33+00:00

We define a trivial fallback for atomic_inc_not_zero(), but don't do
the same for atomic64_inc_not_zero(), leading most architectures to
define the same boilerplate.

Let's add a fallback in , and remove the redundant
implementations. Note that atomic64_add_unless() is always defined in
, and promotes its arguments to the requisite types, so
we need not do this explicitly.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland 
Reviewed-by: Will Deacon 
Acked-by: Peter Zijlstra (Intel) 
Acked-by: Palmer Dabbelt 
Cc: Boqun Feng 
Cc: Linus Torvalds 
Cc: Thomas Gleixner 
Link: https://lore.kernel.org/lkml/20180621121321.4761-6-mark.rutland@arm.com
Signed-off-by: Ingo Molnar