linux-toradex.git/lib/debugobjects.c, branch v5.12-rc6

debugobjects: Free per CPU pool after CPU unplug

2020-10-01T14:13:54+00:00

If a CPU is offlined the debug objects per CPU pool is not cleaned up. If
the CPU is never onlined again then the objects in the pool are wasted.

Add a CPU hotplug callback which is invoked after the CPU is dead to free
the pool.

[ tglx: Massaged changelog and added comment about remote access safety ]

Signed-off-by: Zqiang 
Signed-off-by: Thomas Gleixner 
Cc: Waiman Long 
Link: https://lore.kernel.org/r/20200908062709.11441-1-qiang.zhang@windriver.com

debugobjects: Allow debug_obj_descr to be const

2020-09-24T19:56:24+00:00

The debugobject core could be slightly harder to corrupt if the
debug_obj_descr would be a pointer to const memory.

Depending on the architecture, const data structures are placed into
read-only memory and thus are harder to corrupt or hijack.

This descriptor is used to fix up stuff like timers and workqueues when
core kernel data structures are busted, so moving the descriptors to
read-only memory will make debugobjects more resilient to something going
wrong and then corrupting the function pointers inside struct
debug_obj_descr.

Signed-off-by: Stephen Boyd 
Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20200815004027.2046113-2-swboyd@chromium.org

debugobjects: Convert to DEFINE_SHOW_ATTRIBUTE

2020-07-17T21:25:46+00:00

Use DEFINE_SHOW_ATTRIBUTE macro to simplify the code.

[ tglx: Distangled it from the mess in -next ]

Signed-off-by: Qinglang Miao 
Signed-off-by: Thomas Gleixner 
Cc: hch@lst.de
Link: https://lkml.kernel.org/r/20200716084747.8034-1-miaoqinglang@huawei.com

debugobjects: Fix various data races

2020-01-17T14:45:01+00:00

The counters obj_pool_free, and obj_nr_tofree, and the flag obj_freeing are
read locklessly outside the pool_lock critical sections. If read with plain
accesses, this would result in data races.

This is addressed as follows:

 * reads outside critical sections become READ_ONCE()s (pairing with
   WRITE_ONCE()s added);

 * writes become WRITE_ONCE()s (pairing with READ_ONCE()s added); since
   writes happen inside critical sections, only the write and not the read
   of RMWs needs to be atomic, thus WRITE_ONCE(var, var +/- X) is
   sufficient.

The data races were reported by KCSAN:

  BUG: KCSAN: data-race in __free_object / fill_pool

  write to 0xffffffff8beb04f8 of 4 bytes by interrupt on cpu 1:
   __free_object+0x1ee/0x8e0 lib/debugobjects.c:404
   __debug_check_no_obj_freed+0x199/0x330 lib/debugobjects.c:969
   debug_check_no_obj_freed+0x3c/0x44 lib/debugobjects.c:994
   slab_free_hook mm/slub.c:1422 [inline]

  read to 0xffffffff8beb04f8 of 4 bytes by task 1 on cpu 2:
   fill_pool+0x3d/0x520 lib/debugobjects.c:135
   __debug_object_init+0x3c/0x810 lib/debugobjects.c:536
   debug_object_init lib/debugobjects.c:591 [inline]
   debug_object_activate+0x228/0x320 lib/debugobjects.c:677
   debug_rcu_head_queue kernel/rcu/rcu.h:176 [inline]

  BUG: KCSAN: data-race in __debug_object_init / fill_pool

  read to 0xffffffff8beb04f8 of 4 bytes by task 10 on cpu 6:
   fill_pool+0x3d/0x520 lib/debugobjects.c:135
   __debug_object_init+0x3c/0x810 lib/debugobjects.c:536
   debug_object_init_on_stack+0x39/0x50 lib/debugobjects.c:606
   init_timer_on_stack_key kernel/time/timer.c:742 [inline]

  write to 0xffffffff8beb04f8 of 4 bytes by task 1 on cpu 3:
   alloc_object lib/debugobjects.c:258 [inline]
   __debug_object_init+0x717/0x810 lib/debugobjects.c:544
   debug_object_init lib/debugobjects.c:591 [inline]
   debug_object_activate+0x228/0x320 lib/debugobjects.c:677
   debug_rcu_head_queue kernel/rcu/rcu.h:176 [inline]

  BUG: KCSAN: data-race in free_obj_work / free_object

  read to 0xffffffff9140c190 of 4 bytes by task 10 on cpu 6:
   free_object+0x4b/0xd0 lib/debugobjects.c:426
   debug_object_free+0x190/0x210 lib/debugobjects.c:824
   destroy_timer_on_stack kernel/time/timer.c:749 [inline]

  write to 0xffffffff9140c190 of 4 bytes by task 93 on cpu 1:
   free_obj_work+0x24f/0x480 lib/debugobjects.c:313
   process_one_work+0x454/0x8d0 kernel/workqueue.c:2264
   worker_thread+0x9a/0x780 kernel/workqueue.c:2410

Reported-by: Qian Cai 
Signed-off-by: Marco Elver 
Signed-off-by: Thomas Gleixner 
Link: https://lore.kernel.org/r/20200116185529.11026-1-elver@google.com

debugobjects: Move printk out of db->lock critical sections

2019-06-14T12:51:16+00:00

The db->lock is a raw spinlock and so the lock hold time is supposed
to be short. This will not be the case when printk() is being involved
in some of the critical sections. In order to avoid the long hold time,
in case some messages need to be printed, the debug_object_is_on_stack()
and debug_print_object() calls are now moved out of those critical
sections.

Signed-off-by: Waiman Long 
Signed-off-by: Thomas Gleixner 
Cc: Andrew Morton 
Cc: Yang Shi 
Cc: "Joel Fernandes (Google)" 
Cc: Qian Cai 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190520141450.7575-6-longman@redhat.com

debugobjects: Less aggressive freeing of excess debug objects

2019-06-14T12:51:15+00:00

After a system bootup and 3 parallel kernel builds, a partial output
of the debug objects stats file was:

pool_free     :5101
pool_pcp_free :4181
pool_min_free :220
pool_used     :104172
pool_max_used :171920
on_free_list  :0
objs_allocated:39268280
objs_freed    :39160031

More than 39 millions debug objects had since been allocated and then
freed. The pool_max_used, however, was only about 172k. So this is a
lot of extra overhead in freeing and allocating objects from slabs. It
may also causes the slabs to be more fragmented and harder to reclaim.

Make the freeing of excess debug objects less aggressive by freeing them at
a maximum frequency of 10Hz and about 1k objects at each round of freeing.

With that change applied, the partial output of the debug objects stats
file after similar actions became:

pool_free     :5901
pool_pcp_free :3742
pool_min_free :1022
pool_used     :104805
pool_max_used :168081
on_free_list  :0
objs_allocated:5796864
objs_freed    :5687182

Signed-off-by: Waiman Long 
Signed-off-by: Thomas Gleixner 
Cc: Andrew Morton 
Cc: Yang Shi 
Cc: "Joel Fernandes (Google)" 
Cc: Qian Cai 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190520141450.7575-5-longman@redhat.com

debugobjects: Reduce number of pool_lock acquisitions in fill_pool()

2019-06-14T12:51:15+00:00

In fill_pool(), the pool_lock is acquired and then released once per debug
object. If many objects are to be filled, the constant lock and unlock
operations are extra overhead.

To reduce the overhead, batch them up and do an allocation of 4 objects per
lock/unlock sequence.

Signed-off-by: Waiman Long 
Signed-off-by: Thomas Gleixner 
Cc: Andrew Morton 
Cc: Yang Shi 
Cc: "Joel Fernandes (Google)" 
Cc: Qian Cai 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190520141450.7575-4-longman@redhat.com

debugobjects: Percpu pool lookahead freeing/allocation

2019-06-14T12:51:14+00:00

Most workloads will allocate a bunch of memory objects, work on them
and then freeing all or most of them. So just having a percpu free pool
may not reduce the pool_lock contention significantly if large number
of objects are being used.

To help those situations, we are now doing lookahead allocation and
freeing of the debug objects into and out of the percpu free pool. This
will hopefully reduce the number of times the pool_lock needs to be
taken and hence its contention level.

Signed-off-by: Waiman Long 
Signed-off-by: Thomas Gleixner 
Cc: Andrew Morton 
Cc: Yang Shi 
Cc: "Joel Fernandes (Google)" 
Cc: Qian Cai 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190520141450.7575-3-longman@redhat.com

debugobjects: Add percpu free pools

2019-06-14T12:51:14+00:00

When a multi-threaded workload does a lot of small memory object
allocations and deallocations, it may cause the allocation and freeing of
many debug objects. This will make the global pool_lock a bottleneck in the
performance of the workload.  Since interrupts are disabled when acquiring
the pool_lock, it may even cause hard lockups to happen.

To reduce contention of the global pool_lock, add a percpu debug object
free pool that can be used to buffer some of the debug object allocation
and freeing requests without acquiring the pool_lock.  Each CPU will now
have a percpu free pool that can hold up to a maximum of 64 debug
objects. Allocation and freeing requests will go to the percpu free pool
first. If that fails, the pool_lock will be taken and the global free pool
will be used.

The presence or absence of obj_cache is used as a marker to see if the
percpu cache should be used.

Signed-off-by: Waiman Long 
Signed-off-by: Thomas Gleixner 
Cc: Andrew Morton 
Cc: Yang Shi 
Cc: "Joel Fernandes (Google)" 
Cc: Qian Cai 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190520141450.7575-2-longman@redhat.com

debugobjects: No need to check return value of debugfs_create()

2019-06-14T12:51:14+00:00

When calling debugfs functions, there is no need to ever check the
return value.  The function can work or not, but the code logic should
never do something different based on this.

Signed-off-by: Greg Kroah-Hartman 
Signed-off-by: Thomas Gleixner 
Cc: Qian Cai 
Cc: Andrew Morton 
Cc: Waiman Long 
Cc: "Joel Fernandes (Google)" 
Cc: Zhong Jiang 
Link: https://lkml.kernel.org/r/20190612153513.GA21082@kroah.com