diff options
| author | David Hildenbrand (Arm) <david@kernel.org> | 2026-11-13 11:18:23 +0100 |
|---|---|---|
| committer | David Hildenbrand (Arm) <david@kernel.org> | 2026-11-14 16:36:05 +0100 |
| commit | 25f5c92e1ac06496679ea229e5c825aa88968efd (patch) | |
| tree | eafdb35910f2917d010f02d36b3cc4b0706d4e25 /include/linux | |
| parent | 7183af7f01c839ecef87840a90c0d8c5843f37d7 (diff) | |
| parent | 2d9bcd380585abc436f055fbb618b34f98882004 (diff) | |
Merge patch series "mm: limit pagecache folio refcount"
David Hildenbrand (Arm) <david@kernel.org> says:
Users can create a lot of folio references to pagecache folios, making
it possible to overflow the refcount of large folios.
While we knew that one could in theory overflow a folio's refcount, we
mostly assumed that bad things would happen and it wouldn't go
unnoticed.
Not so sure if that is really always the case, and leaving that problem
unsolved is irresponsible, so let's limit pagecache folio refcounts.
This is mostly relevant for large folios / thps in the page cache, where
a single VMA can hold multiple folio references, allowing for many folio
references. But with sufficient (memory+process) resources, it is also
possible with small folios, just not that relevant in practice today.
With anon folios (including THPs), we should not be able to trigger folio
refcount overflows, as one would need so many processes that it
becomes impossible in practice. It was likely a problem for anon folios as
well before we had a dedicated pincount.
Let's limit the refcount before it turns negative, giving admin processes
some extra reserves (to recover a problematic system). We still leave some
headroom to allow for some slight limit overshooting, to make the
implementation easier.
But to make that work reliably, we have to make anybody that can take
an unbounded number of folio references obey these limits.
Patch #1 handles mapping references from page tables. We also have to take
care of mmu_gather, to not hold too many references when zapping (whereby
new page faults could remap pages concurrently), which we do in patch #3.
Patch #4 fixes iommufd cleanup when adding folio pins fails, which these
limits make easier to trigger.
Splicing pagecache folios to a pipe (patch #2) also needs care. After
feedback from Linus, these users really should be using try_get_page(),
where we now also implement the new limit check. We also teach GUP in
patch #5 to stay below the limit, to avoid negative refcounts that cause
trouble.
Doing some digging after getting a hint from Andrew, there are various
paths in net that can also take an unbounded number of page refs and are
problematic. Let's use try_get_page() in all affected paths.
Net paths that take page refs that I don't touch because the references
are either bounded or the paths are not relevant in practice:
- skb_split(): can add one reference per split, but each split
allocates a new skb that is charged to the socket, and the references
go away once the data was acked. Reworking that to be able to fail
requires more plumbing.
- IP-TFS reassembly: needs one new inner skb per additional reference.
- io_uring zerocopy send (managed frag references): each zerocopy piece
is charged at least 4 KiB of TCP memory, so not relevant in practice.
- kTLS software splicing: only transfers references from the pipe, so
not a problem.
- AF_XDP zerocopy TX, nfsd splice reads, sockmap / sk_msg, espintcp:
bounded, or require privileges / special setups, so not a problem in
practice.
- Driver-specific paths (e.g., kTLS device offload): require special
hardware, didn't look into these.
- page_pool and net_iov frags: not pagecache pages, so not a problem.
For page table mappings, this is only a temporary solution for
upstream+stable, until we have something better, which will be harder to
backport:
- Letting the mapcount of large folios be a long
- Holding a single refcount if the folio is mapped, across all mappings
- Reverting the limits introduced in this series
After that, most of patch #1 will be reverted, everything else will
stay.
Tested on x86-64 by lowering the folio refcount limits such that limits
can easily be exceeded, to then test fault (SIGBUS), fork(), splice and GUP
behavior briefly when we run into refcount limits on a shmem THP.
The net patches were tested similarly with AF_PACKET, nft dup, NFQUEUE,
software GSO segmentation, TCP receive coalescing with dropped ACKs and
socket splicing.
* patches from https://patch.msgid.link/20261112-refcount_overflow_stable-v4-0-ad3deef0bd23@kernel.org:
mm/gup: limit pagecache folio refcount when grabbing references
iommufd: fix cleanup when adding memfd folio pins fails
mm/mmu_gather: flush pending references early before the folio limit
mm: limit pagecache folio refcount when splicing into pipes
mm: limit pagecache folio refcount when mapping pages
The -net patches part of the series will be dealt with separately.
Link: https://patch.msgid.link/20261112-refcount_overflow_stable-v4-0-ad3deef0bd23@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Diffstat (limited to 'include/linux')
| -rw-r--r-- | include/linux/mm.h | 89 |
1 files changed, 75 insertions, 14 deletions
diff --git a/include/linux/mm.h b/include/linux/mm.h index 6af3b7008f1a..daf07b2d208f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2249,9 +2249,54 @@ vm_fault_t finish_fault(struct vm_fault *vmf); * back into memory. */ -/* 127: arbitrary random number, small enough to assemble well */ -#define folio_ref_zero_or_close_to_overflow(folio) \ - ((unsigned int) folio_ref_count(folio) + 127u <= 127u) +/* Leave separate reserves for administrators and concurrent reference gains. */ +#define FOLIO_REF_COUNT_LIMIT (INT_MAX - (1U << 26)) +#define FOLIO_REF_COUNT_ADMIN_LIMIT (INT_MAX - (1U << 25)) +#define FOLIO_REF_COUNT_FLUSH_LIMIT (INT_MAX - (1U << 27)) + +bool folio_ref_count_over_limit_slowpath(const struct folio *folio, + unsigned int ref_count); + +/** + * __folio_ref_count_over_limit - Check whether to refuse more folio references. + * @folio: The folio. + * @ref_count: The sampled folio reference count. + * + * Check before adding page table mappings or other significant refcount users + * to pagecache folios (!hugetlb), either before acquiring references or + * afterwards with rollback if the operation is refused. + * + * Also required when pinning folios without a dedicated pincount that use + * GUP_PIN_COUNTING_BIAS. + * + * Checking other folios does no harm. + * + * This check is an approximation; we leave some headroom such that concurrent + * increments, batches, temporary references, or other bounded refcount + * consumers are not a problem. + * + * Return: True if the caller should refuse the operation, false otherwise. + */ +static inline bool __folio_ref_count_over_limit(const struct folio *folio, + unsigned int ref_count) +{ + if (likely(ref_count < FOLIO_REF_COUNT_LIMIT)) + return false; + return folio_ref_count_over_limit_slowpath(folio, ref_count); +} + +/** + * folio_ref_count_over_limit - Check whether to refuse more folio references. + * @folio: The folio. + * + * See __folio_ref_count_over_limit(). + * + * Return: True if the caller should refuse the operation, false otherwise. + */ +static inline bool folio_ref_count_over_limit(const struct folio *folio) +{ + return __folio_ref_count_over_limit(folio, folio_ref_count(folio)); +} /** * folio_get - Increment the reference count on a folio. @@ -2263,7 +2308,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf); */ static inline void folio_get(struct folio *folio) { - VM_BUG_ON_FOLIO(folio_ref_zero_or_close_to_overflow(folio), folio); + VM_WARN_ON_FOLIO(folio_ref_count(folio) <= 0, folio); folio_ref_inc(folio); } @@ -2277,15 +2322,35 @@ static inline void get_page(struct page *page) folio_get(folio); } -static inline __must_check bool try_get_page(struct page *page) +/** + * folio_get_checked - Try to acquire a reference to a live folio. + * @folio: The folio. + * + * The caller must already hold a reference to @folio. Warn and fail on a + * non-positive refcount, and refuse additional references at the limit. + * + * Use folio_try_get() for speculative reference acquisition where the + * refcount can legitimately be zero. + * + * Return: True if the refcount was incremented, false otherwise. + */ +static inline __must_check bool folio_get_checked(struct folio *folio) { - page = compound_head(page); - if (WARN_ON_ONCE(page_ref_count(page) <= 0)) + const int ref_count = folio_ref_count(folio); + + if (WARN_ON_ONCE(ref_count <= 0)) + return false; + if (unlikely(__folio_ref_count_over_limit(folio, ref_count))) return false; - page_ref_inc(page); + folio_ref_inc(folio); return true; } +static inline __must_check bool try_get_page(struct page *page) +{ + return folio_get_checked(page_folio(page)); +} + /** * folio_put - Decrement the reference count on a folio. * @folio: The folio. @@ -2925,12 +2990,8 @@ static inline bool folio_maybe_dma_pinned(struct folio *folio) return atomic_read(&folio->_pincount) > 0; /* - * folio_ref_count() is signed. If that refcount overflows, then - * folio_ref_count() returns a negative value, and callers will avoid - * further incrementing the refcount. - * - * Here, for that overflow case, use the sign bit to count a little - * bit higher via unsigned math, and thus still get an accurate result. + * Use unsigned math so even an unexpected negative refcount is + * treated as possibly pinned. */ return ((unsigned int)folio_ref_count(folio)) >= GUP_PIN_COUNTING_BIAS; |
