summaryrefslogtreecommitdiff
path: root/include/linux
diff options
context:
space:
mode:
authorDavid Hildenbrand (Arm) <david@kernel.org>2026-11-13 11:18:23 +0100
committerDavid Hildenbrand (Arm) <david@kernel.org>2026-11-14 16:36:05 +0100
commit25f5c92e1ac06496679ea229e5c825aa88968efd (patch)
treeeafdb35910f2917d010f02d36b3cc4b0706d4e25 /include/linux
parent7183af7f01c839ecef87840a90c0d8c5843f37d7 (diff)
parent2d9bcd380585abc436f055fbb618b34f98882004 (diff)
Merge patch series "mm: limit pagecache folio refcount"
David Hildenbrand (Arm) <david@kernel.org> says: Users can create a lot of folio references to pagecache folios, making it possible to overflow the refcount of large folios. While we knew that one could in theory overflow a folio's refcount, we mostly assumed that bad things would happen and it wouldn't go unnoticed. Not so sure if that is really always the case, and leaving that problem unsolved is irresponsible, so let's limit pagecache folio refcounts. This is mostly relevant for large folios / thps in the page cache, where a single VMA can hold multiple folio references, allowing for many folio references. But with sufficient (memory+process) resources, it is also possible with small folios, just not that relevant in practice today. With anon folios (including THPs), we should not be able to trigger folio refcount overflows, as one would need so many processes that it becomes impossible in practice. It was likely a problem for anon folios as well before we had a dedicated pincount. Let's limit the refcount before it turns negative, giving admin processes some extra reserves (to recover a problematic system). We still leave some headroom to allow for some slight limit overshooting, to make the implementation easier. But to make that work reliably, we have to make anybody that can take an unbounded number of folio references obey these limits. Patch #1 handles mapping references from page tables. We also have to take care of mmu_gather, to not hold too many references when zapping (whereby new page faults could remap pages concurrently), which we do in patch #3. Patch #4 fixes iommufd cleanup when adding folio pins fails, which these limits make easier to trigger. Splicing pagecache folios to a pipe (patch #2) also needs care. After feedback from Linus, these users really should be using try_get_page(), where we now also implement the new limit check. We also teach GUP in patch #5 to stay below the limit, to avoid negative refcounts that cause trouble. Doing some digging after getting a hint from Andrew, there are various paths in net that can also take an unbounded number of page refs and are problematic. Let's use try_get_page() in all affected paths. Net paths that take page refs that I don't touch because the references are either bounded or the paths are not relevant in practice: - skb_split(): can add one reference per split, but each split allocates a new skb that is charged to the socket, and the references go away once the data was acked. Reworking that to be able to fail requires more plumbing. - IP-TFS reassembly: needs one new inner skb per additional reference. - io_uring zerocopy send (managed frag references): each zerocopy piece is charged at least 4 KiB of TCP memory, so not relevant in practice. - kTLS software splicing: only transfers references from the pipe, so not a problem. - AF_XDP zerocopy TX, nfsd splice reads, sockmap / sk_msg, espintcp: bounded, or require privileges / special setups, so not a problem in practice. - Driver-specific paths (e.g., kTLS device offload): require special hardware, didn't look into these. - page_pool and net_iov frags: not pagecache pages, so not a problem. For page table mappings, this is only a temporary solution for upstream+stable, until we have something better, which will be harder to backport: - Letting the mapcount of large folios be a long - Holding a single refcount if the folio is mapped, across all mappings - Reverting the limits introduced in this series After that, most of patch #1 will be reverted, everything else will stay. Tested on x86-64 by lowering the folio refcount limits such that limits can easily be exceeded, to then test fault (SIGBUS), fork(), splice and GUP behavior briefly when we run into refcount limits on a shmem THP. The net patches were tested similarly with AF_PACKET, nft dup, NFQUEUE, software GSO segmentation, TCP receive coalescing with dropped ACKs and socket splicing. * patches from https://patch.msgid.link/20261112-refcount_overflow_stable-v4-0-ad3deef0bd23@kernel.org: mm/gup: limit pagecache folio refcount when grabbing references iommufd: fix cleanup when adding memfd folio pins fails mm/mmu_gather: flush pending references early before the folio limit mm: limit pagecache folio refcount when splicing into pipes mm: limit pagecache folio refcount when mapping pages The -net patches part of the series will be dealt with separately. Link: https://patch.msgid.link/20261112-refcount_overflow_stable-v4-0-ad3deef0bd23@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Diffstat (limited to 'include/linux')
-rw-r--r--include/linux/mm.h89
1 files changed, 75 insertions, 14 deletions
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6af3b7008f1a..daf07b2d208f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2249,9 +2249,54 @@ vm_fault_t finish_fault(struct vm_fault *vmf);
* back into memory.
*/
-/* 127: arbitrary random number, small enough to assemble well */
-#define folio_ref_zero_or_close_to_overflow(folio) \
- ((unsigned int) folio_ref_count(folio) + 127u <= 127u)
+/* Leave separate reserves for administrators and concurrent reference gains. */
+#define FOLIO_REF_COUNT_LIMIT (INT_MAX - (1U << 26))
+#define FOLIO_REF_COUNT_ADMIN_LIMIT (INT_MAX - (1U << 25))
+#define FOLIO_REF_COUNT_FLUSH_LIMIT (INT_MAX - (1U << 27))
+
+bool folio_ref_count_over_limit_slowpath(const struct folio *folio,
+ unsigned int ref_count);
+
+/**
+ * __folio_ref_count_over_limit - Check whether to refuse more folio references.
+ * @folio: The folio.
+ * @ref_count: The sampled folio reference count.
+ *
+ * Check before adding page table mappings or other significant refcount users
+ * to pagecache folios (!hugetlb), either before acquiring references or
+ * afterwards with rollback if the operation is refused.
+ *
+ * Also required when pinning folios without a dedicated pincount that use
+ * GUP_PIN_COUNTING_BIAS.
+ *
+ * Checking other folios does no harm.
+ *
+ * This check is an approximation; we leave some headroom such that concurrent
+ * increments, batches, temporary references, or other bounded refcount
+ * consumers are not a problem.
+ *
+ * Return: True if the caller should refuse the operation, false otherwise.
+ */
+static inline bool __folio_ref_count_over_limit(const struct folio *folio,
+ unsigned int ref_count)
+{
+ if (likely(ref_count < FOLIO_REF_COUNT_LIMIT))
+ return false;
+ return folio_ref_count_over_limit_slowpath(folio, ref_count);
+}
+
+/**
+ * folio_ref_count_over_limit - Check whether to refuse more folio references.
+ * @folio: The folio.
+ *
+ * See __folio_ref_count_over_limit().
+ *
+ * Return: True if the caller should refuse the operation, false otherwise.
+ */
+static inline bool folio_ref_count_over_limit(const struct folio *folio)
+{
+ return __folio_ref_count_over_limit(folio, folio_ref_count(folio));
+}
/**
* folio_get - Increment the reference count on a folio.
@@ -2263,7 +2308,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf);
*/
static inline void folio_get(struct folio *folio)
{
- VM_BUG_ON_FOLIO(folio_ref_zero_or_close_to_overflow(folio), folio);
+ VM_WARN_ON_FOLIO(folio_ref_count(folio) <= 0, folio);
folio_ref_inc(folio);
}
@@ -2277,15 +2322,35 @@ static inline void get_page(struct page *page)
folio_get(folio);
}
-static inline __must_check bool try_get_page(struct page *page)
+/**
+ * folio_get_checked - Try to acquire a reference to a live folio.
+ * @folio: The folio.
+ *
+ * The caller must already hold a reference to @folio. Warn and fail on a
+ * non-positive refcount, and refuse additional references at the limit.
+ *
+ * Use folio_try_get() for speculative reference acquisition where the
+ * refcount can legitimately be zero.
+ *
+ * Return: True if the refcount was incremented, false otherwise.
+ */
+static inline __must_check bool folio_get_checked(struct folio *folio)
{
- page = compound_head(page);
- if (WARN_ON_ONCE(page_ref_count(page) <= 0))
+ const int ref_count = folio_ref_count(folio);
+
+ if (WARN_ON_ONCE(ref_count <= 0))
+ return false;
+ if (unlikely(__folio_ref_count_over_limit(folio, ref_count)))
return false;
- page_ref_inc(page);
+ folio_ref_inc(folio);
return true;
}
+static inline __must_check bool try_get_page(struct page *page)
+{
+ return folio_get_checked(page_folio(page));
+}
+
/**
* folio_put - Decrement the reference count on a folio.
* @folio: The folio.
@@ -2925,12 +2990,8 @@ static inline bool folio_maybe_dma_pinned(struct folio *folio)
return atomic_read(&folio->_pincount) > 0;
/*
- * folio_ref_count() is signed. If that refcount overflows, then
- * folio_ref_count() returns a negative value, and callers will avoid
- * further incrementing the refcount.
- *
- * Here, for that overflow case, use the sign bit to count a little
- * bit higher via unsigned math, and thus still get an accurate result.
+ * Use unsigned math so even an unexpected negative refcount is
+ * treated as possibly pinned.
*/
return ((unsigned int)folio_ref_count(folio)) >=
GUP_PIN_COUNTING_BIAS;