memcg: trim the per-cpu charge stock instead of draining it

Joy reported that an application generating a request/response traffic
pattern spends 44.6% to 57.0% of CPU in the memcg charge/uncharge path for
a range of message sizes, against 0.27% to 0.71% outside that range. 
Running from the root memcg, where socket memory accounting is skipped,
recovers the performance.

Tracing the charge path showed that the application generates a pattern
where the write syscall charges one page and the read syscall uncharges
two pages on the same CPU.  This hits a corner case in the memcg percpu
stock code that thrashes the stock continuously.

In the memcg percpu stock code, MEMCG_CHARGE_BATCH (64) is both the high
watermark and the emptying target, i.e.  on a request to charge one page
the kernel charges MEMCG_CHARGE_BATCH pages and caches (MEMCG_CHARGE_BATCH
- 1) of them in the percpu stock.  The following uncharge of 2 pages takes
the cached count to (MEMCG_CHARGE_BATCH + 1), and refill_stock() then
empties the cache completely.  With such a pattern the percpu stock
becomes completely ineffective.

Instead of a single boundary point for charges, use the technique the page
allocator uses for its own percpu caches, which keeps the watermark and
the emptying target apart: nr_pcp_free() frees between batch and high -
batch pages, leaving at least pcp->batch on the list.  Add a high
watermark MEMCG_STOCK_HIGH and, once the cached count goes over it, return
only the pages above MEMCG_STOCK_LOW.  The watermarks are
MEMCG_CHARGE_BATCH apart, so a page_counter update still covers a full
batch.

For now, keep MEMCG_STOCK_HIGH same as MEMCG_CHARGE_BATCH and in future we
will reevaluate if it makes sense to increase it.

Link: https://lore.kernel.org/20260820012010.2016086-1-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Joy Chaoyue Xiong <cxiong@meta.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Jakub Kacinski <kuba@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>

0 files changed, 0 insertions, 0 deletions