linux-toradex.git/lib/percpu_counter.c, branch v4.4.93

percpu_counter: batch size aware __percpu_counter_compare()

2015-05-28T21:39:34+00:00

XFS uses non-stanard batch sizes for avoiding frequent global
counter updates on it's allocated inode counters, as they increment
or decrement in batches of 64 inodes. Hence the standard percpu
counter batch of 32 means that the counter is effectively a global
counter. Currently Xfs uses a batch size of 128 so that it doesn't
take the global lock on every single modification.

However, Xfs also needs to compare accurately against zero, which
means we need to use percpu_counter_compare(), and that has a
hard-coded batch size of 32, and hence will spuriously fail to
detect when it is supposed to use precise comparisons and hence
the accounting goes wrong.

Add __percpu_counter_compare() to take a custom batch size so we can
use it sanely in XFS and factor percpu_counter_compare() to use it.

Signed-off-by: Dave Chinner 
Acked-by: Tejun Heo 
Signed-off-by: Dave Chinner

percpu_counter: add @gfp to percpu_counter_init()

2014-09-08T00:51:29+00:00

Percpu allocator now supports allocation mask.  Add @gfp to
percpu_counter_init() so that !GFP_KERNEL allocation masks can be used
with percpu_counters too.

We could have left percpu_counter_init() alone and added
percpu_counter_init_gfp(); however, the number of users isn't that
high and introducing _gfp variants to all percpu data structures would
be quite ugly, so let's just do the conversion.  This is the one with
the most users.  Other percpu data structures are a lot easier to
convert.

This patch doesn't make any functional difference.

Signed-off-by: Tejun Heo 
Acked-by: Jan Kara 
Acked-by: "David S. Miller" 
Cc: x86@kernel.org
Cc: Jens Axboe 
Cc: "Theodore Ts'o" 
Cc: Alexander Viro 
Cc: Andrew Morton

percpu_counter: make percpu_counters_lock irq-safe

2014-09-08T00:51:29+00:00

percpu_counter is scheduled to grow @gfp support to allow atomic
initialization.  This patch makes percpu_counters_lock irq-safe so
that it can be safely used from atomic contexts.

Signed-off-by: Tejun Heo

lib/percpu_counter.c: fix bad percpu counter state during suspend

2014-04-08T23:48:51+00:00

I got a bug report yesterday from Laszlo Ersek in which he states that
his kvm instance fails to suspend.  Laszlo bisected it down to this
commit 1cf7e9c68fe8 ("virtio_blk: blk-mq support") where virtio-blk is
converted to use the blk-mq infrastructure.

After digging a bit, it became clear that the issue was with the queue
drain.  blk-mq tracks queue usage in a percpu counter, which is
incremented on request alloc and decremented when the request is freed.
The initial hunt was for an inconsistency in blk-mq, but everything
seemed fine.  In fact, the counter only returned crazy values when
suspend was in progress.

When a CPU is unplugged, the percpu counters merges that CPU state with
the general state.  blk-mq takes care to register a hotcpu notifier with
the appropriate priority, so we know it runs after the percpu counter
notifier.  However, the percpu counter notifier only merges the state
when the CPU is fully gone.  This leaves a state transition where the
CPU going away is no longer in the online mask, yet it still holds
private values.  This means that in this state, percpu_counter_sum()
returns invalid results, and the suspend then hangs waiting for
abs(dead-cpu-value) requests to complete which of course will never
happen.

Fix this by clearing the state earlier, so we never have a case where
the CPU isn't in online mask but still holds private state.  This bug
has been there since forever, I guess we don't have a lot of users where
percpu counters needs to be reliable during the suspend cycle.

Signed-off-by: Jens Axboe 
Reported-by: Laszlo Ersek 
Tested-by: Laszlo Ersek 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

percpu_counter: unbreak __percpu_counter_add()

2014-01-17T00:32:24+00:00

Commit 74e72f894d56 ("lib/percpu_counter.c: fix __percpu_counter_add()")
looked very plausible, but its arithmetic was badly wrong: obvious once
you see the fix, but maddening to get there from the weird tmpfs ENOSPCs

Signed-off-by: Hugh Dickins 
Cc: Ming Lei 
Cc: Paul Gortmaker 
Cc: Shaohua Li 
Cc: Jens Axboe 
Cc: Fan Du 
Cc: Andrew Morton 
Signed-off-by: Linus Torvalds

lib/percpu_counter.c: fix __percpu_counter_add()

2014-01-15T07:19:42+00:00

__percpu_counter_add() may be called in softirq/hardirq handler (such
as, blk_mq_queue_exit() is typically called in hardirq/softirq handler),
so we need to call this_cpu_add()(irq safe helper) to update percpu
counter, otherwise counts may be lost.

This fixes the problem that 'rmmod null_blk' hangs in blk_cleanup_queue()
because of miscounting of request_queue->mq_usage_counter.

This patch is the v1 of previous one of "lib/percpu_counter.c:
disable local irq when updating percpu couter", and takes Andrew's
approach which may be more efficient for ARCHs(x86, s390) that
have optimized this_cpu_add().

Signed-off-by: Ming Lei 
Cc: Paul Gortmaker 
Cc: Shaohua Li 
Cc: Jens Axboe 
Cc: Fan Du 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

percpu_counter: make APIs irq safe

2013-10-25T10:55:59+00:00

In my usage, sometimes the percpu APIs are called with irq locked,
sometimes not. lockdep complains there is potential deadlock. Let's
always use percpucounter lock in irq safe way. There should be no
performance penality, as all those are slow code path.

Cc: Andrew Morton 
Signed-off-by: Shaohua Li 
Signed-off-by: Jens Axboe

kernel: delete __cpuinit usage from all core kernel files

2013-07-14T23:36:59+00:00

The __cpuinit type of throwaway sections might have made sense
some time ago when RAM was more constrained, but now the savings
do not offset the cost and complications.  For example, the fix in
commit 5e427ec2d0 ("x86: Fix bit corruption at CPU resume time")
is a good example of the nasty type of bugs that can be created
with improper use of the various __init prefixes.

After a discussion on LKML[1] it was decided that cpuinit should go
the way of devinit and be phased out.  Once all the users are gone,
we can then finally remove the macros themselves from linux/init.h.

This removes all the uses of the __cpuinit macros from C files in
the core kernel directories (kernel, init, lib, mm, and include)
that don't really have a specific maintainer.

[1] https://lkml.org/lkml/2013/5/20/589

Signed-off-by: Paul Gortmaker

lib/percpu_counter.c: __this_cpu_write() doesn't need to be protected by spinlock

2013-07-03T23:07:43+00:00

__this_cpu_write doesn't need to be protected by spinlock, AS we are doing
per cpu write with preempt disabled.  And another reason to remove
__this_cpu_write outside of spinlock: __percpu_counter_sum is not an
accurate counter.

Signed-off-by: Fan Du 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

switch the protection of percpu_counter list to spinlock

2012-07-31T05:28:31+00:00

... making percpu_counter_destroy() non-blocking

Signed-off-by: Al Viro