linux-toradex.git/mm/mmap.c, branch v2.6.34.7

mm: make the vma list be doubly linked

2010-08-26T23:43:37+00:00

commit 297c5eee372478fc32fec5fe8eed711eedb13f3d upstream.

It's a really simple list, and several of the users want to go backwards
in it to find the previous vma.  So rather than have to look up the
previous entry with 'find_vma_prev()' or something similar, just make it
doubly linked instead.

Tested-by: Ian Campbell 
Signed-off-by: Linus Torvalds 
Signed-off-by: Greg Kroah-Hartman

mmap: check ->vm_ops before dereferencing

2010-04-27T15:26:51+00:00

Check whether the VMA has a vm_ops before calling close, just
like we check vm_ops before calling open a few dozen lines
higher up in the function.

Signed-off-by: Rik van Riel 
Reported-by: Dan Carpenter 
Signed-off-by: Linus Torvalds

vma_adjust: fix the copying of anon_vma chains

2010-04-13T00:54:11+00:00

When we move the boundaries between two vma's due to things like
mprotect, we need to make sure that the anon_vma of the pages that got
moved from one vma to another gets properly copied around.  And that was
not always the case, in this rather hard-to-follow code sequence.

Clarify the code, and fix it so that it copies the anon_vma from the
right source.

Reviewed-by: Rik van Riel 
Acked-by: Johannes Weiner 
Tested-by: Borislav Petkov  [ "Yeah, not so much this one either" ]
Signed-off-by: Linus Torvalds

Simplify and comment on anon_vma re-use for anon_vma_prepare()

2010-04-13T00:53:59+00:00

This changes the anon_vma reuse case to require that we only reuse
simple anon_vma's - ie the case when the vma only has a single anon_vma
associated with it.

This means that a reuse of an anon_vma from an adjacent vma will always
guarantee that both vma's are associated not only with the same
anon_vma, they will also have the same anon_vma chain (of just a single
entry in this case).

And since anon_vma re-use was the only case where the same anon_vma
might be associated with different chains of anon_vma's, we now have the
case that every vma that shares the same anon_vma will always also have
the same chain.  That makes it much easier to think about merging vma's
that share the same anon_vma's: you can always just drop the other
anon_vma chain in anon_vma_merge() since you know that they are always
identical.

This also splits up the function to validate the anon_vma re-use, and
adds a lot of commentary about the possible races.

Reviewed-by: Rik van Riel 
Acked-by: Johannes Weiner 
Tested-by: Borislav Petkov  [ "That didn't fix it" ]
Signed-off-by: Linus Torvalds

Add generic sys_old_mmap()

2010-03-12T23:52:32+00:00

Add a generic implementation of the old mmap() syscall, which expects its
argument in a memory block and switch all architectures over to use it.

Signed-off-by: Christoph Hellwig 
Cc: Ralf Baechle 
Cc: Benjamin Herrenschmidt 
Cc: Paul Mundt 
Cc: Jeff Dike 
Cc: Hirokazu Takata 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Reviewed-by: H. Peter Anvin 
Cc: Al Viro 
Cc: Arnd Bergmann 
Cc: Heiko Carstens 
Cc: Martin Schwidefsky 
Cc: "Luck, Tony" 
Cc: James Morris 
Cc: Andreas Schwab 
Acked-by: Jesper Nilsson 
Acked-by: Russell King 
Acked-by: Greg Ungerer 
Acked-by: David Howells 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: remove VM_LOCK_RMAP code

2010-03-06T19:26:26+00:00

When a VMA is in an inconsistent state during setup or teardown, the worst
that can happen is that the rmap code will not be able to find the page.

The mapping is in the process of being torn down (PTEs just got
invalidated by munmap), or set up (no PTEs have been instantiated yet).

It is also impossible for the rmap code to follow a pointer to an already
freed VMA, because the rmap code holds the anon_vma->lock, which the VMA
teardown code needs to take before the VMA is removed from the anon_vma
chain.

Hence, we should not need the VM_LOCK_RMAP locking at all.

Signed-off-by: Rik van Riel 
Cc: Nick Piggin 
Cc: KOSAKI Motohiro 
Cc: Larry Woodman 
Cc: Lee Schermerhorn 
Cc: Andrea Arcangeli 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: change anon_vma linking to fix multi-process server scalability issue

2010-03-06T19:26:26+00:00

The old anon_vma code can lead to scalability issues with heavily forking
workloads.  Specifically, each anon_vma will be shared between the parent
process and all its child processes.

In a workload with 1000 child processes and a VMA with 1000 anonymous
pages per process that get COWed, this leads to a system with a million
anonymous pages in the same anon_vma, each of which is mapped in just one
of the 1000 processes.  However, the current rmap code needs to walk them
all, leading to O(N) scanning complexity for each page.

This can result in systems where one CPU is walking the page tables of
1000 processes in page_referenced_one, while all other CPUs are stuck on
the anon_vma lock.  This leads to catastrophic failure for a benchmark
like AIM7, where the total number of processes can reach in the tens of
thousands.  Real workloads are still a factor 10 less process intensive
than AIM7, but they are catching up.

This patch changes the way anon_vmas and VMAs are linked, which allows us
to associate multiple anon_vmas with a VMA.  At fork time, each child
process gets its own anon_vmas, in which its COWed pages will be
instantiated.  The parents' anon_vma is also linked to the VMA, because
non-COWed pages could be present in any of the children.

This reduces rmap scanning complexity to O(1) for the pages of the 1000
child processes, with O(N) complexity for at most 1/N pages in the system.
 This reduces the average scanning cost in heavily forking workloads from
O(N) to 2.

The only real complexity in this patch stems from the fact that linking a
VMA to anon_vmas now involves memory allocations.  This means vma_adjust
can fail, if it needs to attach a VMA to anon_vma structures.  This in
turn means error handling needs to be added to the calling functions.

A second source of complexity is that, because there can be multiple
anon_vmas, the anon_vma linking in vma_adjust can no longer be done under
"the" anon_vma lock.  To prevent the rmap code from walking up an
incomplete VMA, this patch introduces the VM_LOCK_RMAP VMA flag.  This bit
flag uses the same slot as the NOMMU VM_MAPPED_COPY, with an ifdef in mm.h
to make sure it is impossible to compile a kernel that needs both symbolic
values for the same bitflag.

Some test results:

Without the anon_vma changes, when AIM7 hits around 9.7k users (on a test
box with 16GB RAM and not quite enough IO), the system ends up running
>99% in system time, with every CPU on the same anon_vma lock in the
pageout code.

With these changes, AIM7 hits the cross-over point around 29.7k users.
This happens with ~99% IO wait time, there never seems to be any spike in
system time.  The anon_vma lock contention appears to be resolved.

[akpm@linux-foundation.org: cleanups]
Signed-off-by: Rik van Riel 
Cc: KOSAKI Motohiro 
Cc: Larry Woodman 
Cc: Lee Schermerhorn 
Cc: Minchan Kim 
Cc: Andrea Arcangeli 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: use rlimit helpers

2010-03-06T19:26:24+00:00

Make sure compiler won't do weird things with limits.  E.g.  fetching them
twice may return 2 different values after writable limits are implemented.

I.e.  either use rlimit helpers added in
3e10e716abf3c71bdb5d86b8f507f9e72236c9cd ("resource: add helpers for
fetching rlimits") or ACCESS_ONCE if not applicable.

Signed-off-by: Jiri Slaby 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: mlock_vma_pages_range() only return success or failure

2010-03-06T19:26:24+00:00

Currently, mlock_vma_pages_range() only return len or 0.  then current
error handling of mmap_region() is meaningless complex.

This patch makes simplify and makes consist with brk() code.

Signed-off-by: KOSAKI Motohiro 
Cc: Nick Piggin 
Cc: Lee Schermerhorn 
Cc: Rik van Riel 
Cc: KAMEZAWA Hiroyuki 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

mm: mlock_vma_pages_range() never return negative value

2010-03-06T19:26:24+00:00

Currently, mlock_vma_pages_range() never return negative value.  Then, we
can remove some worthless error check.

Signed-off-by: KOSAKI Motohiro 
Cc: Nick Piggin 
Cc: Lee Schermerhorn 
Cc: Rik van Riel 
Cc: KAMEZAWA Hiroyuki 
Cc: Hugh Dickins 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds