linux-toradex.git/mm/hmm.c, branch v5.4-rc1

pagewalk: separate function pointers from iterator data

2019-09-07T07:28:04+00:00

The mm_walk structure currently mixed data and code.  Split out the
operations vectors into a new mm_walk_ops structure, and while we are
changing the API also declare the mm_walk structure inside the
walk_page_range and walk_page_vma functions.

Based on patch from Linus Torvalds.

Link: https://lore.kernel.org/r/20190828141955.22210-3-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Thomas Hellstrom 
Reviewed-by: Steven Price 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm: split out a new pagewalk.h header from mm.h

2019-09-07T07:28:04+00:00

Add a new header for the two handful of users of the walk_page_range /
walk_page_vma interface instead of polluting all users of mm.h with it.

Link: https://lore.kernel.org/r/20190828141955.22210-2-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Thomas Hellstrom 
Reviewed-by: Steven Price 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm/hmm: hmm_range_fault() infinite loop

2019-08-27T22:27:07+00:00

Normally, callers to handle_mm_fault() are supposed to check the
vma->vm_flags first. hmm_range_fault() checks for VM_READ but doesn't
check for VM_WRITE if the caller requests a page to be faulted in with
write permission (via the hmm_range.pfns[] value).  If the vma is write
protected, this can result in an infinite loop:

  hmm_range_fault()
    walk_page_range()
      ...
      hmm_vma_walk_hole()
        hmm_vma_walk_hole_()
          hmm_vma_do_fault()
            handle_mm_fault(FAULT_FLAG_WRITE)
            /* returns VM_FAULT_WRITE */
          /* returns -EBUSY */
        /* returns -EBUSY */
      /* returns -EBUSY */
    /* loops on -EBUSY and range->valid */

Prevent this by checking for vma->vm_flags & VM_WRITE before calling
handle_mm_fault().

Link: https://lore.kernel.org/r/20190823221753.2514-3-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell 
Reviewed-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm/hmm: hmm_range_fault() NULL pointer bug

2019-08-27T22:27:07+00:00

Although hmm_range_fault() calls find_vma() to make sure that a vma exists
before calling walk_page_range(), hmm_vma_walk_hole() can still be called
with walk->vma == NULL if the start and end address are not contained
within the vma range.

 hmm_range_fault() /* calls find_vma() but no range check */
  walk_page_range() /* calls find_vma(), sets walk->vma = NULL */
   __walk_page_range()
    walk_pgd_range()
     walk_p4d_range()
      walk_pud_range()
       hmm_vma_walk_hole()
        hmm_vma_walk_hole_()
         hmm_vma_do_fault()
          handle_mm_fault(vma=0)

Link: https://lore.kernel.org/r/20190823221753.2514-2-rcampbell@nvidia.com
Signed-off-by: Ralph Campbell 
Reviewed-by: Christoph Hellwig 
Signed-off-by: Jason Gunthorpe

mm/hmm: fix hmm_range_fault()'s handling of swapped out pages

2019-08-23T13:22:20+00:00

hmm_range_fault() may return NULL pages because some of the pfns are equal
to HMM_PFN_NONE. This happens randomly under memory pressure. The reason
is during the swapped out page pte path, hmm_vma_handle_pte() doesn't
update the fault variable from cpu_flags, so it failed to call
hmm_vam_do_fault() to swap the page in.

The fix is to call hmm_pte_need_fault() to update fault variable.

Fixes: 74eee180b935 ("mm/hmm/mirror: device page fault handler")
Link: https://lore.kernel.org/r/20190815205227.7949-1-Philip.Yang@amd.com
Signed-off-by: Philip Yang 
Reviewed-by: "Jérôme Glisse" 
Signed-off-by: Jason Gunthorpe

hmm: use mmu_notifier_get/put for 'struct hmm'

2019-08-20T12:35:02+00:00

This is a significant simplification, it eliminates all the remaining
'hmm' stuff in mm_struct, eliminates krefing along the critical notifier
paths, and takes away all the ugly locking and abuse of page_table_lock.

mmu_notifier_get() provides the single struct hmm per struct mm which
eliminates mm->hmm.

It also directly guarantees that no mmu_notifier op callback is callable
while concurrent free is possible, this eliminates all the krefs inside
the mmu_notifier callbacks.

The remaining krefs in the range code were overly cautious, drivers are
already not permitted to free the mirror while a range exists.

Link: https://lore.kernel.org/r/20190806231548.25242-6-jgg@ziepe.ca
Reviewed-by: Christoph Hellwig 
Reviewed-by: Ralph Campbell 
Tested-by: Ralph Campbell 
Signed-off-by: Jason Gunthorpe

mm/hmm: cleanup the hmm_vma_walk_hugetlb_entry stub

2019-08-07T17:58:06+00:00

Stub out the whole function and assign NULL to the .hugetlb_entry method
if CONFIG_HUGETLB_PAGE is not set, as the method won't ever be called in
that case.

Link: https://lore.kernel.org/r/20190806160554.14046-13-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm/hmm: cleanup the hmm_vma_handle_pmd stub

2019-08-07T17:58:06+00:00

Stub out the whole function when CONFIG_TRANSPARENT_HUGEPAGE is not set to
make the function easier to read.

Link: https://lore.kernel.org/r/20190806160554.14046-12-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm/hmm: only define hmm_vma_walk_pud if needed

2019-08-07T17:58:06+00:00

We only need the special pud_entry walker if PUD-sized hugepages and pte
mappings are supported, else the common pagewalk code will take care of
the iteration.  Not implementing this callback reduced the amount of code
compiled for non-x86 platforms, and also fixes compile failures with other
architectures when helpers like pud_pfn are not implemented.

Link: https://lore.kernel.org/r/20190806160554.14046-11-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe

mm/hmm: don't abuse pte_index() in hmm_vma_handle_pmd

2019-08-07T17:58:06+00:00

pte_index is an internal arch helper in various architectures, without
consistent semantics.  Open code that calculation of a PMD index based on
the virtual address instead.

Link: https://lore.kernel.org/r/20190806160554.14046-10-hch@lst.de
Signed-off-by: Christoph Hellwig 
Reviewed-by: Jason Gunthorpe 
Signed-off-by: Jason Gunthorpe