linux-toradex.git/mm/page_isolation.c, branch v3.18.1

mm/page_alloc: restrict max order of merging on isolated pageblock

2014-11-14T00:17:05+00:00

Current pageblock isolation logic could isolate each pageblock
individually.  This causes freepage accounting problem if freepage with
pageblock order on isolate pageblock is merged with other freepage on
normal pageblock.  We can prevent merging by restricting max order of
merging to pageblock order if freepage is on isolate pageblock.

A side-effect of this change is that there could be non-merged buddy
freepage even if finishing pageblock isolation, because undoing
pageblock isolation is just to move freepage from isolate buddy list to
normal buddy list rather than to consider merging.  So, the patch also
makes undoing pageblock isolation consider freepage merge.  When
un-isolation, freepage with more than pageblock order and it's buddy are
checked.  If they are on normal pageblock, instead of just moving, we
isolate the freepage and free it in order to get merged.

Signed-off-by: Joonsoo Kim 
Acked-by: Vlastimil Babka 
Cc: "Kirill A. Shutemov" 
Cc: Mel Gorman 
Cc: Johannes Weiner 
Cc: Minchan Kim 
Cc: Yasuaki Ishimatsu 
Cc: Zhang Yanfei 
Cc: Tang Chen 
Cc: Naoya Horiguchi 
Cc: Bartlomiej Zolnierkiewicz 
Cc: Wen Congyang 
Cc: Marek Szyprowski 
Cc: Michal Nazarewicz 
Cc: Laura Abbott 
Cc: Heesub Shin 
Cc: "Aneesh Kumar K.V" 
Cc: Ritesh Harjani 
Cc: Gioh Kim 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/page_alloc: fix incorrect isolation behavior by rechecking migratetype

2014-11-14T00:17:05+00:00

Before describing bugs itself, I first explain definition of freepage.

 1. pages on buddy list are counted as freepage.
 2. pages on isolate migratetype buddy list are *not* counted as freepage.
 3. pages on cma buddy list are counted as CMA freepage, too.

Now, I describe problems and related patch.

Patch 1: There is race conditions on getting pageblock migratetype that
it results in misplacement of freepages on buddy list, incorrect
freepage count and un-availability of freepage.

Patch 2: Freepages on pcp list could have stale cached information to
determine migratetype of buddy list to go.  This causes misplacement of
freepages on buddy list and incorrect freepage count.

Patch 4: Merging between freepages on different migratetype of
pageblocks will cause freepages accouting problem.  This patch fixes it.

Without patchset [3], above problem doesn't happens on my CMA allocation
test, because CMA reserved pages aren't used at all.  So there is no
chance for above race.

With patchset [3], I did simple CMA allocation test and get below
result:

 - Virtual machine, 4 cpus, 1024 MB memory, 256 MB CMA reservation
 - run kernel build (make -j16) on background
 - 30 times CMA allocation(8MB * 30 = 240MB) attempts in 5 sec interval
 - Result: more than 5000 freepage count are missed

With patchset [3] and this patchset, I found that no freepage count are
missed so that I conclude that problems are solved.

On my simple memory offlining test, these problems also occur on that
environment, too.

This patch (of 4):

There are two paths to reach core free function of buddy allocator,
__free_one_page(), one is free_one_page()->__free_one_page() and the
other is free_hot_cold_page()->free_pcppages_bulk()->__free_one_page().
Each paths has race condition causing serious problems.  At first, this
patch is focused on first type of freepath.  And then, following patch
will solve the problem in second type of freepath.

In the first type of freepath, we got migratetype of freeing page
without holding the zone lock, so it could be racy.  There are two cases
of this race.

 1. pages are added to isolate buddy list after restoring orignal
    migratetype

    CPU1                                   CPU2

    get migratetype => return MIGRATE_ISOLATE
    call free_one_page() with MIGRATE_ISOLATE

                                grab the zone lock
                                unisolate pageblock
                                release the zone lock

    grab the zone lock
    call __free_one_page() with MIGRATE_ISOLATE
    freepage go into isolate buddy list,
    although pageblock is already unisolated

This may cause two problems.  One is that we can't use this page anymore
until next isolation attempt of this pageblock, because freepage is on
isolate buddy list.  The other is that freepage accouting could be wrong
due to merging between different buddy list.  Freepages on isolate buddy
list aren't counted as freepage, but ones on normal buddy list are
counted as freepage.  If merge happens, buddy freepage on normal buddy
list is inevitably moved to isolate buddy list without any consideration
of freepage accouting so it could be incorrect.

 2. pages are added to normal buddy list while pageblock is isolated.
    It is similar with above case.

This also may cause two problems.  One is that we can't keep these
freepages from being allocated.  Although this pageblock is isolated,
freepage would be added to normal buddy list so that it could be
allocated without any restriction.  And the other problem is same as
case 1, that it, incorrect freepage accouting.

This race condition would be prevented by checking migratetype again
with holding the zone lock.  Because it is somewhat heavy operation and
it isn't needed in common case, we want to avoid rechecking as much as
possible.  So this patch introduce new variable, nr_isolate_pageblock in
struct zone to check if there is isolated pageblock.  With this, we can
avoid to re-check migratetype in common case and do it only if there is
isolated pageblock or migratetype is MIGRATE_ISOLATE.  This solve above
mentioned problems.

Changes from v3:
Add one more check in free_one_page() that checks whether migratetype is
MIGRATE_ISOLATE or not. Without this, abovementioned case 1 could happens.

Signed-off-by: Joonsoo Kim 
Acked-by: Minchan Kim 
Acked-by: Michal Nazarewicz 
Acked-by: Vlastimil Babka 
Cc: "Kirill A. Shutemov" 
Cc: Mel Gorman 
Cc: Johannes Weiner 
Cc: Yasuaki Ishimatsu 
Cc: Zhang Yanfei 
Cc: Tang Chen 
Cc: Naoya Horiguchi 
Cc: Bartlomiej Zolnierkiewicz 
Cc: Wen Congyang 
Cc: Marek Szyprowski 
Cc: Laura Abbott 
Cc: Heesub Shin 
Cc: "Aneesh Kumar K.V" 
Cc: Ritesh Harjani 
Cc: Gioh Kim 
Cc: 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: memory-hotplug: enable memory hotplug to handle hugepage

2013-09-11T22:57:48+00:00

Until now we can't offline memory blocks which contain hugepages because a
hugepage is considered as an unmovable page.  But now with this patch
series, a hugepage has become movable, so by using hugepage migration we
can offline such memory blocks.

What's different from other users of hugepage migration is that we need to
decompose all the hugepages inside the target memory block into free buddy
pages after hugepage migration, because otherwise free hugepages remaining
in the memory block intervene the memory offlining.  For this reason we
introduce new functions dissolve_free_huge_page() and
dissolve_free_huge_pages().

Other than that, what this patch does is straightforwardly to add hugepage
migration code, that is, adding hugepage code to the functions which scan
over pfn and collect hugepages to be migrated, and adding a hugepage
allocation function to alloc_migrate_target().

As for larger hugepages (1GB for x86_64), it's not easy to do hotremove
over them because it's larger than memory block.  So we now simply leave
it to fail as it is.

[yongjun_wei@trendmicro.com.cn: remove duplicated include]
Signed-off-by: Naoya Horiguchi 
Acked-by: Andi Kleen 
Cc: Hillf Danton 
Cc: Wanpeng Li 
Cc: Mel Gorman 
Cc: Hugh Dickins 
Cc: KOSAKI Motohiro 
Cc: Michal Hocko 
Cc: Rik van Riel 
Cc: "Aneesh Kumar K.V" 
Signed-off-by: Wei Yongjun 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

page_isolation: Fix a comment typo in test_pages_isolated()

2013-08-20T11:03:41+00:00

pageblock_nr_page should be pageblock_nr_pages, and fist is
a typo of first.

Signed-off-by: Tang Chen 
Signed-off-by: Jiri Kosina

mm: fix zone_watermark_ok_safe() accounting of isolated pages

2013-01-05T00:11:46+00:00

Commit 702d1a6e0766 ("memory-hotplug: fix kswapd looping forever
problem") added an isolated pageblocks counter (nr_pageblock_isolate in
struct zone) and used it to adjust free pages counter in
zone_watermark_ok_safe() to prevent kswapd looping forever problem.

Then later, commit 2139cbe627b8 ("cma: fix counting of isolated pages")
fixed accounting of isolated pages in global free pages counter.  It
made the previous zone_watermark_ok_safe() fix unnecessary and
potentially harmful (cause now isolated pages may be accounted twice
making free pages counter incorrect).

This patch removes the special isolated pageblocks counter altogether
which fixes zone_watermark_ok_safe() free pages check.

Reported-by: Tomasz Stanislawski 
Signed-off-by: Bartlomiej Zolnierkiewicz 
Signed-off-by: Kyungmin Park 
Cc: Minchan Kim 
Cc: KOSAKI Motohiro 
Cc: Aaditya Kumar 
Cc: KAMEZAWA Hiroyuki 
Cc: Mel Gorman 
Cc: Michal Hocko 
Cc: Marek Szyprowski 
Cc: Michal Nazarewicz 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

memory-hotplug: skip HWPoisoned page when offlining pages

2012-12-12T01:22:22+00:00

hwpoisoned may be set when we offline a page by the sysfs interface
/sys/devices/system/memory/soft_offline_page or
/sys/devices/system/memory/hard_offline_page. If we don't clear
this flag when onlining pages, this page can't be freed, and will
not in free list. So we can't offline these pages again. So we
should skip such page when offlining pages.

Signed-off-by: Wen Congyang 
Cc: David Rientjes 
Cc: Jiang Liu 
Cc: Len Brown 
Cc: Benjamin Herrenschmidt 
Cc: Paul Mackerras 
Cc: Christoph Lameter 
Cc: Minchan Kim 
Cc: KOSAKI Motohiro 
Cc: Yasuaki Ishimatsu 
Cc: Andi Kleen 
Cc: Mel Gorman 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm/page_alloc: refactor out __alloc_contig_migrate_alloc()

2012-10-09T07:22:52+00:00

__alloc_contig_migrate_alloc() can be used by memory-hotplug so refactor
it out (move + rename as a common name) into page_isolation.c.

[akpm@linux-foundation.org: checkpatch fixes]
Signed-off-by: Minchan Kim 
Cc: Kamezawa Hiroyuki 
Reviewed-by: Yasuaki Ishimatsu 
Acked-by: Michal Nazarewicz 
Cc: Marek Szyprowski 
Cc: Wen Congyang 
Acked-by: David Rientjes 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

memory-hotplug: fix pages missed by race rather than failing

2012-10-09T07:22:46+00:00

If race between allocation and isolation in memory-hotplug offline
happens, some pages could be in MIGRATE_MOVABLE of free_list although the
pageblock's migratetype of the page is MIGRATE_ISOLATE.

The race could be detected by get_freepage_migratetype in
__test_page_isolated_in_pageblock.  If it is detected, now EBUSY gets
bubbled all the way up and the hotplug operations fails.

But better idea is instead of returning and failing memory-hotremove, move
the free page to the correct list at the time it is detected.  It could
enhance memory-hotremove operation success ratio although the race is
really rare.

Suggested by Mel Gorman.

[akpm@linux-foundation.org: small cleanup]
Signed-off-by: Minchan Kim 
Cc: KAMEZAWA Hiroyuki 
Reviewed-by: Yasuaki Ishimatsu 
Acked-by: Mel Gorman 
Cc: Xishi Qiu 
Cc: Wen Congyang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

memory-hotplug: bug fix race between isolation and allocation

2012-10-09T07:22:46+00:00

Like below, memory-hotplug makes race between page-isolation
and page-allocation so it can hit BUG_ON in __offline_isolated_pages.

	CPU A					CPU B

start_isolate_page_range
set_migratetype_isolate
spin_lock_irqsave(zone->lock)

				free_hot_cold_page(Page A)
				/* without zone->lock */
				migratetype = get_pageblock_migratetype(Page A);
				/*
				 * Page could be moved into MIGRATE_MOVABLE
				 * of per_cpu_pages
				 */
				list_add_tail(&page->lru, &pcp->lists[migratetype]);

set_pageblock_isolate
move_freepages_block
drain_all_pages

				/* Page A could be in MIGRATE_MOVABLE of free_list. */

check_pages_isolated
__test_page_isolated_in_pageblock
/*
 * We can't catch freed page which
 * is free_list[MIGRATE_MOVABLE]
 */
if (PageBuddy(page A))
	pfn += 1 << page_order(page A);

				/* So, Page A could be allocated */

__offline_isolated_pages
/*
 * BUG_ON hit or offline page
 * which is used by someone
 */
BUG_ON(!PageBuddy(page A));

This patch checks page's migratetype in freelist in
__test_page_isolated_in_pageblock.  So now
__test_page_isolated_in_pageblock can check the page caused by above race
and can fail of memory offlining.

Signed-off-by: Minchan Kim 
Acked-by: KAMEZAWA Hiroyuki 
Reviewed-by: Yasuaki Ishimatsu 
Acked-by: Mel Gorman 
Cc: Xishi Qiu 
Cc: Wen Congyang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: page_alloc: use get_freepage_migratetype() instead of page_private()

2012-10-09T07:22:45+00:00

The page allocator uses set_page_private and page_private for handling
migratetype when it frees page.  Let's replace them with [set|get]
_freepage_migratetype to make it more clear.

Signed-off-by: Minchan Kim 
Acked-by: KAMEZAWA Hiroyuki 
Reviewed-by: Yasuaki Ishimatsu 
Acked-by: Mel Gorman 
Cc: Xishi Qiu 
Cc: Wen Congyang 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds