linux-toradex.git/include/linux/vmalloc.h, branch v5.12-rc5

Merge branch 'for-mingo-rcu' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu

2021-02-12T11:56:55+00:00

Pull RCU updates from Paul E. McKenney:

- Documentation updates.

- Miscellaneous fixes.

- kfree_rcu() updates: Addition of mem_dump_obj() to provide allocator return
  addresses to more easily locate bugs.  This has a couple of RCU-related commits,
  but is mostly MM.  Was pulled in with akpm's agreement.

- Per-callback-batch tracking of numbers of callbacks,
  which enables better debugging information and smarter
  reactions to large numbers of callbacks.

- The first round of changes to allow CPUs to be runtime switched from and to
  callback-offloaded state.

- CONFIG_PREEMPT_RT-related changes.

- RCU CPU stall warning updates.
- Addition of polling grace-period APIs for SRCU.

- Torture-test and torture-test scripting updates, including a "torture everything"
  script that runs rcutorture, locktorture, scftorture, rcuscale, and refscale.
  Plus does an allmodconfig build.

Signed-off-by: Ingo Molnar

mm/vmalloc: separate put pages and flush VM flags

2021-02-05T19:03:47+00:00

When VM_MAP_PUT_PAGES was added, it was defined with the same value as
VM_FLUSH_RESET_PERMS.  This doesn't seem like it will cause any big
functional problems other than some excess flushing for VM_MAP_PUT_PAGES
allocations.

Redefine VM_MAP_PUT_PAGES to have its own value.  Also, rearrange things
so flags are less likely to be missed in the future.

Link: https://lkml.kernel.org/r/20210122233706.9304-1-rick.p.edgecombe@intel.com
Fixes: b944afc9d64d ("mm: add a VM_MAP_PUT_PAGES flag for vmap")
Signed-off-by: Rick Edgecombe 
Suggested-by: Matthew Wilcox 
Cc: Miaohe Lin 
Cc: Christoph Hellwig 
Cc: Daniel Axtens 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: Make mem_dump_obj() handle vmalloc() memory

2021-01-22T23:24:04+00:00

This commit adds vmalloc() support to mem_dump_obj().  Note that the
vmalloc_dump_obj() function combines the checking and dumping, in
contrast with the split between kmem_valid_obj() and kmem_dump_obj().
The reason for the difference is that the checking in the vmalloc()
case involves acquiring a global lock, and redundant acquisitions of
global locks should be avoided, even on not-so-fast paths.

Note that this change causes on-stack variables to be reported as
vmalloc() storage from kernel_clone() or similar, depending on the degree
of inlining that your compiler does.  This is likely more helpful than
the earlier "non-paged (local) memory".

Cc: Andrew Morton 
Cc: Joonsoo Kim 
Cc: 
Reported-by: Andrii Nakryiko 
Acked-by: Vlastimil Babka 
Tested-by: Naresh Kamboju 
Signed-off-by: Paul E. McKenney

mm/vmalloc: rework the drain logic

2020-12-15T20:13:41+00:00

A current "lazy drain" model suffers from at least two issues.

First one is related to the unsorted list of vmap areas, thus in order to
identify the [min:max] range of areas to be drained, it requires a full
list scan.  What is a time consuming if the list is too long.

Second one and as a next step is about merging all fragments with a free
space.  What is also a time consuming because it has to iterate over
entire list which holds outstanding lazy areas.

See below the "preemptirqsoff" tracer that illustrates a high latency.  It
is ~24676us.  Our workloads like audio and video are effected by such long
latency:


  tracer: preemptirqsoff

  preemptirqsoff latency trace v1.1.5 on 4.9.186-perf+
  --------------------------------------------------------------------
  latency: 24676 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 P:8)
     -----------------
     | task: crtc_commit:112-261 (uid:0 nice:0 policy:1 rt_prio:16)
     -----------------
   => started at: __purge_vmap_area_lazy
   => ended at:   __purge_vmap_area_lazy

                   _------=> CPU#
                  / _-----=> irqs-off
                 | / _----=> need-resched
                 || / _---=> hardirq/softirq
                 ||| / _--=> preempt-depth
                 |||| /     delay
   cmd     pid   ||||| time  |   caller
      \   /      |||||  \    |   /
crtc_com-261     1...1    1us*: _raw_spin_lock <-__purge_vmap_area_lazy
[...]
crtc_com-261     1...1 24675us : _raw_spin_unlock <-__purge_vmap_area_lazy
crtc_com-261     1...1 24677us : trace_preempt_on <-__purge_vmap_area_lazy
crtc_com-261     1...1 24683us : 
 => free_vmap_area_noflush
 => remove_vm_area
 => __vunmap
 => vfree
 => drm_property_free_blob
 => drm_mode_object_unreference
 => drm_property_unreference_blob
 => __drm_atomic_helper_crtc_destroy_state
 => sde_crtc_destroy_state
 => drm_atomic_state_default_clear
 => drm_atomic_state_clear
 => drm_atomic_state_free
 => complete_commit
 => _msm_drm_commit_work_cb
 => kthread_worker_fn
 => kthread
 => ret_from_fork


To address those two issues we can redesign a purging of the outstanding
lazy areas.  Instead of queuing vmap areas to the list, we replace it by
the separate rb-tree.  In hat case an area is located in the tree/list in
ascending order.  It will give us below advantages:

a) Outstanding vmap areas are merged creating bigger coalesced blocks,
   thus it becomes less fragmented.

b) It is possible to calculate a flush range [min:max] without scanning
   all elements.  It is O(1) access time or complexity;

c) The final merge of areas with the rb-tree that represents a free
   space is faster because of (a).  As a result the lock contention is
   also reduced.

Link: https://lkml.kernel.org/r/20201116220033.1837-2-urezki@gmail.com
Signed-off-by: Uladzislau Rezki (Sony) 
Cc: Hillf Danton 
Cc: Michal Hocko 
Cc: Matthew Wilcox 
Cc: Oleksiy Avramchenko 
Cc: Steven Rostedt 
Cc: Minchan Kim 
Cc: huang ying 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: remove alloc_vm_area

2020-10-18T16:27:10+00:00

All users are gone now.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Andrew Morton 
Cc: Boris Ostrovsky 
Cc: Chris Wilson 
Cc: Jani Nikula 
Cc: Joonas Lahtinen 
Cc: Juergen Gross 
Cc: Matthew Auld 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Minchan Kim 
Cc: Nitin Gupta 
Cc: Peter Zijlstra 
Cc: Rodrigo Vivi 
Cc: Stefano Stabellini 
Cc: Tvrtko Ursulin 
Cc: Uladzislau Rezki (Sony) 
Link: https://lkml.kernel.org/r/20201002122204.1534411-12-hch@lst.de
Signed-off-by: Linus Torvalds

mm: add a vmap_pfn function

2020-10-18T16:27:10+00:00

Add a proper helper to remap PFNs into kernel virtual space so that
drivers don't have to abuse alloc_vm_area and open coded PTE manipulation
for it.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Andrew Morton 
Cc: Boris Ostrovsky 
Cc: Chris Wilson 
Cc: Jani Nikula 
Cc: Joonas Lahtinen 
Cc: Juergen Gross 
Cc: Matthew Auld 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Minchan Kim 
Cc: Nitin Gupta 
Cc: Peter Zijlstra 
Cc: Rodrigo Vivi 
Cc: Stefano Stabellini 
Cc: Tvrtko Ursulin 
Cc: Uladzislau Rezki (Sony) 
Link: https://lkml.kernel.org/r/20201002122204.1534411-4-hch@lst.de
Signed-off-by: Linus Torvalds

mm: add a VM_MAP_PUT_PAGES flag for vmap

2020-10-18T16:27:10+00:00

Add a flag so that vmap takes ownership of the passed in page array.  When
vfree is called on such an allocation it will put one reference on each
page, and free the page array itself.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Andrew Morton 
Cc: Boris Ostrovsky 
Cc: Chris Wilson 
Cc: Jani Nikula 
Cc: Joonas Lahtinen 
Cc: Juergen Gross 
Cc: Matthew Auld 
Cc: "Matthew Wilcox (Oracle)" 
Cc: Minchan Kim 
Cc: Nitin Gupta 
Cc: Peter Zijlstra 
Cc: Rodrigo Vivi 
Cc: Stefano Stabellini 
Cc: Tvrtko Ursulin 
Cc: Uladzislau Rezki (Sony) 
Link: https://lkml.kernel.org/r/20201002122204.1534411-3-hch@lst.de
Signed-off-by: Linus Torvalds

mm: remove vmalloc_exec

2020-06-26T07:27:38+00:00

Merge vmalloc_exec into its only caller.  Note that for !CONFIG_MMU
__vmalloc_node_range maps to __vmalloc, which directly clears the
__GFP_HIGHMEM added by the vmalloc_exec stub anyway.

Link: http://lkml.kernel.org/r/20200618064307.32739-4-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: David Hildenbrand 
Acked-by: Peter Zijlstra (Intel) 
Cc: Catalin Marinas 
Cc: Dexuan Cui 
Cc: Jessica Yu 
Cc: Vitaly Kuznetsov 
Cc: Wei Liu 
Cc: Will Deacon 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: remove vmalloc_sync_(un)mappings()

2020-06-02T17:59:12+00:00

These functions are not needed anymore because the vmalloc and ioremap
mappings are now synchronized when they are created or torn down.

Remove all callers and function definitions.

Signed-off-by: Joerg Roedel 
Signed-off-by: Andrew Morton 
Tested-by: Steven Rostedt (VMware) 
Acked-by: Andy Lutomirski 
Acked-by: Peter Zijlstra (Intel) 
Cc: Arnd Bergmann 
Cc: Christoph Hellwig 
Cc: Dave Hansen 
Cc: "H . Peter Anvin" 
Cc: Ingo Molnar 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: "Rafael J. Wysocki" 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Link: http://lkml.kernel.org/r/20200515140023.25469-7-joro@8bytes.org
Signed-off-by: Linus Torvalds

mm/vmalloc: track which page-table levels were modified

2020-06-02T17:59:11+00:00

Track at which levels in the page-table entries were modified by
vmap/vunmap.

After the page-table has been modified, use that information do decide
whether the new arch_sync_kernel_mappings() needs to be called.

[akpm@linux-foundation.org: map_kernel_range_noflush() needs the arch_sync_kernel_mappings() call]
Signed-off-by: Joerg Roedel 
Signed-off-by: Andrew Morton 
Acked-by: Andy Lutomirski 
Acked-by: Peter Zijlstra (Intel) 
Cc: Arnd Bergmann 
Cc: Christoph Hellwig 
Cc: Dave Hansen 
Cc: "H . Peter Anvin" 
Cc: Ingo Molnar 
Cc: Matthew Wilcox (Oracle) 
Cc: Michal Hocko 
Cc: "Rafael J. Wysocki" 
Cc: Steven Rostedt (VMware) 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Link: http://lkml.kernel.org/r/20200515140023.25469-3-joro@8bytes.org
Signed-off-by: Linus Torvalds