linux-toradex.git/include/linux/page-flags.h, branch v4.5-rc2

mm: rework mapcount accounting to enable 4k mapping of THPs

2016-01-16T01:56:32+00:00

We're going to allow mapping of individual 4k pages of THP compound.  It
means we need to track mapcount on per small page basis.

Straight-forward approach is to use ->_mapcount in all subpages to track
how many time this subpage is mapped with PMDs or PTEs combined.  But
this is rather expensive: mapping or unmapping of a THP page with PMD
would require HPAGE_PMD_NR atomic operations instead of single we have
now.

The idea is to store separately how many times the page was mapped as
whole -- compound_mapcount.  This frees up ->_mapcount in subpages to
track PTE mapcount.

We use the same approach as with compound page destructor and compound
order to store compound_mapcount: use space in first tail page,
->mapping this time.

Any time we map/unmap whole compound page (THP or hugetlb) -- we
increment/decrement compound_mapcount.  When we map part of compound
page with PTE we operate on ->_mapcount of the subpage.

page_mapcount() counts both: PTE and PMD mappings of the page.

Basically, we have mapcount for a subpage spread over two counters.  It
makes tricky to detect when last mapcount for a page goes away.

We introduced PageDoubleMap() for this.  When we split THP PMD for the
first time and there's other PMD mapping left we offset up ->_mapcount
in all subpages by one and set PG_double_map on the compound page.
These additional references go away with last compound_mapcount.

This approach provides a way to detect when last mapcount goes away on
per small page basis without introducing new overhead for most common
cases.

[akpm@linux-foundation.org: fix typo in comment]
[mhocko@suse.com: ignore partial THP when moving task]
Signed-off-by: Kirill A. Shutemov 
Tested-by: Aneesh Kumar K.V 
Acked-by: Jerome Marchand 
Cc: Sasha Levin 
Cc: Aneesh Kumar K.V 
Cc: Jerome Marchand 
Cc: Vlastimil Babka 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: Johannes Weiner 
Cc: Christoph Lameter 
Cc: David Rientjes 
Signed-off-by: Michal Hocko 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm, thp: remove compound_lock()

2016-01-16T01:56:32+00:00

We are going to use migration entries to stabilize page counts.  It
means we don't need compound_lock() for that.

Signed-off-by: Kirill A. Shutemov 
Tested-by: Sasha Levin 
Tested-by: Aneesh Kumar K.V 
Acked-by: Vlastimil Babka 
Acked-by: Jerome Marchand 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Christoph Lameter 
Cc: David Rientjes 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: drop __TestClearPage*() helpers

2016-01-16T01:56:32+00:00

Nobody uses them.

Signed-off-by: Kirill A. Shutemov 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: look at head page if the flag is encoded in page->mapping

2016-01-16T01:56:32+00:00

PageAnon() and PageKsm() look at lower bits of page->mapping to check if
the page is Anon or KSM.  page->mapping can be overloaded in tail pages.

Let's always look at head page to avoid false-positives.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_uptodate behavior on compound pages

2016-01-16T01:56:32+00:00

We use PG_uptodate on head pages on transparent huge page.  Let's use
PF_NO_TAIL.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_uncached behavior on compound pages

2016-01-16T01:56:32+00:00

So far, only IA64 uses PG_uncached and only on non-compound pages.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_mlocked behavior on compound pages

2016-01-16T01:56:32+00:00

Transparent huge pages can be mlocked -- whole compund page at once.
Something went wrong if we're trying to mlock() tail page.  Let's use
PF_NO_TAIL.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_swapcache behavior on compound pages

2016-01-16T01:56:32+00:00

Swap cannot handle compound pages so far.  Transparent huge pages are
split on the way to swap.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_swapbacked behavior on compound pages

2016-01-16T01:56:32+00:00

PG_swapbacked is used for transparent huge pages.  For head pages only.
Let's use PF_NO_TAIL policy.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page-flags: define PG_reserved behavior on compound pages

2016-01-16T01:56:32+00:00

As far as I can see there's no users of PG_reserved on compound pages.
Let's use PF_NO_COMPOUND here.

Signed-off-by: Kirill A. Shutemov 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Cc: Dave Hansen 
Cc: Mel Gorman 
Cc: Rik van Riel 
Cc: Vlastimil Babka 
Cc: Christoph Lameter 
Cc: Naoya Horiguchi 
Cc: Steve Capper 
Cc: "Aneesh Kumar K.V" 
Cc: Johannes Weiner 
Cc: Michal Hocko 
Cc: Jerome Marchand 
Cc: Jérôme Glisse 
Cc: Hillf Danton 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds