linux-toradex.git/mm/mlock.c, branch v5.12

mm/mlock: stop counting mlocked pages when none vma is found

2021-02-26T17:41:01+00:00

There will be no vma satisfies addr < vm_end when find_vma() returns NULL.
Thus it's meaningless to traverse the vma list below because we can't
find any vma to count mlocked pages.  Stop counting mlocked pages in this
case to save some vma list traversal cycles.

Link: https://lkml.kernel.org/r/20210204110705.17586-1-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin 
Reviewed-by: David Hildenbrand 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/swap.c: don't pass "enum lru_list" to del_page_from_lru_list()

2021-02-24T21:38:33+00:00

The parameter is redundant in the sense that it can be potentially
extracted from the "struct page" parameter by page_lru(). We need to
make sure that existing PageActive() or PageUnevictable() remains
until the function returns. A few places don't conform, and simple
reordering fixes them.

This patch may have left page_off_lru() seemingly odd, and we'll take
care of it in the next patch.

Link: https://lore.kernel.org/linux-mm/20201207220949.830352-6-yuzhao@google.com/
Link: https://lkml.kernel.org/r/20210122220600.906146-6-yuzhao@google.com
Signed-off-by: Yu Zhao 
Cc: Alex Shi 
Cc: Hugh Dickins 
Cc: Johannes Weiner 
Cc: Matthew Wilcox 
Cc: Michal Hocko 
Cc: Roman Gushchin 
Cc: Vladimir Davydov 
Cc: Vlastimil Babka 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/lru: introduce relock_page_lruvec()

2020-12-15T22:48:04+00:00

Add relock_page_lruvec() to replace repeated same code, no functional
change.

When testing for relock we can avoid the need for RCU locking if we simply
compare the page pgdat and memcg pointers versus those that the lruvec is
holding.  By doing this we can avoid the extra pointer walks and accesses
of the memory cgroup.

In addition we can avoid the checks entirely if lruvec is currently NULL.

[alex.shi@linux.alibaba.com: use page_memcg()]
  Link: https://lkml.kernel.org/r/66d8e79d-7ec6-bfbc-1c82-bf32db3ae5b7@linux.alibaba.com

Link: https://lkml.kernel.org/r/1604566549-62481-19-git-send-email-alex.shi@linux.alibaba.com
Signed-off-by: Alexander Duyck 
Signed-off-by: Alex Shi 
Acked-by: Hugh Dickins 
Acked-by: Johannes Weiner 
Acked-by: Vlastimil Babka 
Cc: Thomas Gleixner 
Cc: Andrey Ryabinin 
Cc: Matthew Wilcox 
Cc: Mel Gorman 
Cc: Konstantin Khlebnikov 
Cc: Tejun Heo 
Cc: Andrea Arcangeli 
Cc: "Chen, Rong A" 
Cc: Daniel Jordan 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joonsoo Kim 
Cc: Kirill A. Shutemov 
Cc: Kirill A. Shutemov 
Cc: Michal Hocko 
Cc: Michal Hocko 
Cc: Mika Penttilä 
Cc: Minchan Kim 
Cc: Shakeel Butt 
Cc: Vladimir Davydov 
Cc: Wei Yang 
Cc: Yang Shi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/lru: replace pgdat lru_lock with lruvec lock

2020-12-15T22:48:04+00:00

This patch moves per node lru_lock into lruvec, thus bring a lru_lock for
each of memcg per node.  So on a large machine, each of memcg don't have
to suffer from per node pgdat->lru_lock competition.  They could go fast
with their self lru_lock.

After move memcg charge before lru inserting, page isolation could
serialize page's memcg, then per memcg lruvec lock is stable and could
replace per node lru lock.

In isolate_migratepages_block(), compact_unlock_should_abort and
lock_page_lruvec_irqsave are open coded to work with compact_control.
Also add a debug func in locking which may give some clues if there are
sth out of hands.

Daniel Jordan's testing show 62% improvement on modified readtwice case on
his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu@ca-dmjordan1.us.oracle.com/

Hugh Dickins helped on the patch polish, thanks!

[alex.shi@linux.alibaba.com: fix comment typo]
  Link: https://lkml.kernel.org/r/5b085715-292a-4b43-50b3-d73dc90d1de5@linux.alibaba.com
[alex.shi@linux.alibaba.com: use page_memcg()]
  Link: https://lkml.kernel.org/r/5a4c2b72-7ee8-2478-fc0e-85eb83aafec4@linux.alibaba.com

Link: https://lkml.kernel.org/r/1604566549-62481-18-git-send-email-alex.shi@linux.alibaba.com
Signed-off-by: Alex Shi 
Acked-by: Hugh Dickins 
Acked-by: Johannes Weiner 
Cc: Rong Chen 
Cc: Michal Hocko 
Cc: Vladimir Davydov 
Cc: Yang Shi 
Cc: Matthew Wilcox 
Cc: Konstantin Khlebnikov 
Cc: Daniel Jordan 
Cc: Alexander Duyck 
Cc: Andrea Arcangeli 
Cc: Andrey Ryabinin 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joonsoo Kim 
Cc: Kirill A. Shutemov 
Cc: Kirill A. Shutemov 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Mika Penttilä 
Cc: Minchan Kim 
Cc: Shakeel Butt 
Cc: Tejun Heo 
Cc: Thomas Gleixner 
Cc: Vlastimil Babka 
Cc: Wei Yang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/lru: introduce TestClearPageLRU()

2020-12-15T22:48:04+00:00

Currently lru_lock still guards both lru list and page's lru bit, that's
ok.  but if we want to use specific lruvec lock on the page, we need to
pin down the page's lruvec/memcg during locking.  Just taking lruvec lock
first may be undermined by the page's memcg charge/migration.  To fix this
problem, we will clear the lru bit out of locking and use it as pin down
action to block the page isolation in memcg changing.

So now a standard steps of page isolation is following:
	1, get_page(); 	       #pin the page avoid to be free
	2, TestClearPageLRU(); #block other isolation like memcg change
	3, spin_lock on lru_lock; #serialize lru list access
	4, delete page from lru list;

This patch start with the first part: TestClearPageLRU, which combines
PageLRU check and ClearPageLRU into a macro func TestClearPageLRU.  This
function will be used as page isolation precondition to prevent other
isolations some where else.  Then there are may !PageLRU page on lru list,
need to remove BUG() checking accordingly.

There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
   temporary moment(isolating), the page may have no lru bit when
   it's on lru list.  but the page still must be on lru list when the
   lru bit set.
2, have to remove lru bit before delete it from lru list.

As Andrew Morton mentioned this change would dirty cacheline for a page
which isn't on the LRU.  But the loss would be acceptable in Rong Chen
 report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/

Link: https://lkml.kernel.org/r/1604566549-62481-15-git-send-email-alex.shi@linux.alibaba.com
Suggested-by: Johannes Weiner 
Signed-off-by: Alex Shi 
Acked-by: Hugh Dickins 
Acked-by: Johannes Weiner 
Acked-by: Vlastimil Babka 
Cc: Michal Hocko 
Cc: Vladimir Davydov 
Cc: Alexander Duyck 
Cc: Andrea Arcangeli 
Cc: Andrey Ryabinin 
Cc: Daniel Jordan 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joonsoo Kim 
Cc: Kirill A. Shutemov 
Cc: Kirill A. Shutemov 
Cc: Konstantin Khlebnikov 
Cc: Matthew Wilcox (Oracle) 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Mika Penttilä 
Cc: Minchan Kim 
Cc: Shakeel Butt 
Cc: Tejun Heo 
Cc: Thomas Gleixner 
Cc: Wei Yang 
Cc: Yang Shi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/mlock: remove __munlock_isolate_lru_page()

2020-12-15T22:48:04+00:00

__munlock_isolate_lru_page() only has one caller, remove it to clean up
and simplify code.

Link: https://lkml.kernel.org/r/1604566549-62481-14-git-send-email-alex.shi@linux.alibaba.com
Signed-off-by: Alex Shi 
Acked-by: Hugh Dickins 
Acked-by: Johannes Weiner 
Acked-by: Vlastimil Babka 
Cc: Kirill A. Shutemov 
Cc: Alexander Duyck 
Cc: Andrea Arcangeli 
Cc: Andrey Ryabinin 
Cc: "Chen, Rong A" 
Cc: Daniel Jordan 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joonsoo Kim 
Cc: Kirill A. Shutemov 
Cc: Konstantin Khlebnikov 
Cc: Matthew Wilcox (Oracle) 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Michal Hocko 
Cc: Mika Penttilä 
Cc: Minchan Kim 
Cc: Shakeel Butt 
Cc: Tejun Heo 
Cc: Thomas Gleixner 
Cc: Vladimir Davydov 
Cc: Wei Yang 
Cc: Yang Shi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/mlock: remove lru_lock on TestClearPageMlocked

2020-12-15T22:48:04+00:00

In the func munlock_vma_page, comments mentained lru_lock needed for
serialization with split_huge_pages.  But the page must be PageLocked as
well as pages in split_huge_page series funcs.  Thus the PageLocked is
enough to serialize both funcs.

Further more, Hugh Dickins pointed: before splitting in
split_huge_page_to_list, the page was unmap_page() to remove pmd/ptes
which protect the page from munlock.  Thus, no needs to guard
__split_huge_page_tail for mlock clean, just keep the lru_lock there for
isolation purpose.

LKP found a preempt issue on __mod_zone_page_state which need change to
mod_zone_page_state.  Thanks!

Link: https://lkml.kernel.org/r/1604566549-62481-13-git-send-email-alex.shi@linux.alibaba.com
Signed-off-by: Alex Shi 
Acked-by: Hugh Dickins 
Acked-by: Johannes Weiner 
Acked-by: Vlastimil Babka 
Cc: Kirill A. Shutemov 
Cc: Alexander Duyck 
Cc: Andrea Arcangeli 
Cc: Andrey Ryabinin 
Cc: "Chen, Rong A" 
Cc: Daniel Jordan 
Cc: "Huang, Ying" 
Cc: Jann Horn 
Cc: Joonsoo Kim 
Cc: Kirill A. Shutemov 
Cc: Konstantin Khlebnikov 
Cc: Matthew Wilcox (Oracle) 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Michal Hocko 
Cc: Mika Penttilä 
Cc: Minchan Kim 
Cc: Shakeel Butt 
Cc: Tejun Heo 
Cc: Thomas Gleixner 
Cc: Vladimir Davydov 
Cc: Wei Yang 
Cc: Yang Shi 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mlock: fix unevictable_pgs event counts on THP

2020-09-19T20:13:38+00:00

5.8 commit 5d91f31faf8e ("mm: swap: fix vmstats for huge page") has
established that vm_events should count every subpage of a THP, including
unevictable_pgs_culled and unevictable_pgs_rescued; but
lru_cache_add_inactive_or_unevictable() was not doing so for
unevictable_pgs_mlocked, and mm/mlock.c was not doing so for
unevictable_pgs mlocked, munlocked, cleared and stranded.

Fix them; but THPs don't go the pagevec way in mlock.c, so no fixes needed
on that path.

Fixes: 5d91f31faf8e ("mm: swap: fix vmstats for huge page")
Signed-off-by: Hugh Dickins 
Signed-off-by: Andrew Morton 
Reviewed-by: Shakeel Butt 
Acked-by: Yang Shi 
Cc: Alex Shi 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Mike Kravetz 
Cc: Qian Cai 
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008301408230.5954@eggly.anvils
Signed-off-by: Linus Torvalds

mm: replace hpage_nr_pages with thp_nr_pages

2020-08-15T02:56:56+00:00

The thp prefix is more frequently used than hpage and we should be
consistent between the various functions.

[akpm@linux-foundation.org: fix mm/migrate.c]

Signed-off-by: Matthew Wilcox (Oracle) 
Signed-off-by: Andrew Morton 
Reviewed-by: William Kucharski 
Reviewed-by: Zi Yan 
Cc: Mike Kravetz 
Cc: David Hildenbrand 
Cc: "Kirill A. Shutemov" 
Link: http://lkml.kernel.org/r/20200629151959.15779-6-willy@infradead.org
Signed-off-by: Linus Torvalds

mmap locking API: convert mmap_sem comments

2020-06-09T16:39:14+00:00

Convert comments that reference mmap_sem to reference mmap_lock instead.

[akpm@linux-foundation.org: fix up linux-next leftovers]
[akpm@linux-foundation.org: s/lockaphore/lock/, per Vlastimil]
[akpm@linux-foundation.org: more linux-next fixups, per Michel]

Signed-off-by: Michel Lespinasse 
Signed-off-by: Andrew Morton 
Reviewed-by: Vlastimil Babka 
Reviewed-by: Daniel Jordan 
Cc: Davidlohr Bueso 
Cc: David Rientjes 
Cc: Hugh Dickins 
Cc: Jason Gunthorpe 
Cc: Jerome Glisse 
Cc: John Hubbard 
Cc: Laurent Dufour 
Cc: Liam Howlett 
Cc: Matthew Wilcox 
Cc: Peter Zijlstra 
Cc: Ying Han 
Link: http://lkml.kernel.org/r/20200520052908.204642-13-walken@google.com
Signed-off-by: Linus Torvalds