diff options
| author | Christian Brauner <brauner@kernel.org> | 2026-10-30 13:38:22 +0100 |
|---|---|---|
| committer | Christian Brauner <brauner@kernel.org> | 2026-11-11 11:59:27 +0100 |
| commit | 20a6a9ec19e522ae85f87337886b3fb47efce5f3 (patch) | |
| tree | 43117ce2c7287e5f3bb1cf324b481b103a688d1b /include/linux/stackprotector.h | |
| parent | a57ad4d3abd9735129e05f7990ba21406ccd9dab (diff) | |
dcache: let d_invalidate() wait for a d_set_mounted() in flight
d_invalidate() and d_set_mounted() are supposed to exclude each other.
Either the mount fails with -ENOENT because an ancestor of the
mountpoint was unhashed or d_invalidate() finds the mountpoint and
detaches the mount. d_invalidate() unhashes the dentry first and then
searches the subtree with d_walk() which starts out lockless and only
looks at the seqcount of rename_lock.
Since d_set_mounted() switched to read_seqlock_excl() nothing makes that
walk wait or retry:
CPU0: mount on A/M CPU1: d_invalidate(A)
d_set_mounted(M)
read_seqlock_excl(&rename_lock)
A is hashed, go on
__d_drop(A)
shrink_dcache_parent(A)
d_walk(A, find_submount)
read_seqbegin(&rename_lock)
M is not a mountpoint
need_seqretry() == false
return
M->d_flags |= DCACHE_MOUNTED
return 0
mount gets attached to M
The mount ends up below an unhashed dentry. It can't be reached anymore
and pins the subtree until its parent mount or the mount namespace goes
away. The window grows with the depth of the mountpoint. With a FUSE
directory 3000 levels deep a bind mount racing with
FUSE_NOTIFY_INVAL_ENTRY for the parent of the mountpoint survived in 151
of 200 attempts.
Before 5e317d23f6c6 the seqcount bump made the walk wait while
d_set_mounted() ran and retry under the lock when one had run in the
meantime. That was the right thing to wait for on the wrong counter:
every rename_lock reader paid for it, d_lookup(), d_path(), the ".."
step of the RCU walk.
So give d_set_mounted() a seqlock of its own. It is the only writer and
d_invalidate() is the only reader. After unhashing the dentry
d_invalidate() runs an empty read_seqbegin()/read_seqretry() section on
it. That spins while the count is odd, i.e. while a d_set_mounted() is
between its ancestor check and its flag, and goes round once more if
one started in the meantime. Nobody else retries anything. The ordering
comes from the locks both already take:
- If that d_set_mounted() saw the dentry hashed its check ran under the
dentry's d_lock before __d_drop() took it. So the count is odd by the
time d_invalidate() reads it and d_invalidate() waits.
- Once the count is even again the flag has been set under the
mountpoint's d_lock and the walk takes that lock to look at it.
A d_set_mounted() that starts after the unhash fails its ancestor check
and never touches the flag. So the only flag the walk can pick up
belongs to the one mount that saw the dentry hashed and is still in
flight, and there is at most one of those because namespace_sem
serializes mounts. Either it gets attached and detach_mounts() takes it
off after waiting for namespace_sem or it fails later on and drops the
flag again. In that case the walk goes round once more and finds
nothing. The same goes for the mountpoint that attach_recursive_mnt()
preallocates on the root of the tree it attaches. IOW, d_invalidate()
does at most one extra round, which the removed-name trap (cf. [1])
relies on.
Note that on PREEMPT_RT read_seqbegin() takes the lock instead of
spinning on a preempted d_set_mounted().
The cost is a seqlock around the ancestor walk for every new mountpoint
and two loads of a cacheline that only d_set_mounted() writes for every
d_invalidate() of a positive dentry. Walking the ancestors a second time
after setting the flag would close the race as well but doubles the
cost of d_set_mounted() with the depth of the mountpoint: about 1us at
64 levels and 75us at 4096 levels.
[1]: https://patch.msgid.link/20261028-work-mount-trap-b4-v1-0-f20d9d0730d6@kernel.org
Fixes: 5e317d23f6c6 ("d_set_mounted(): we don't need to bump seqcount component of rename_lock")
Reported-by: Josef Bacik <josef@toxicpanda.com>
Cc: stable@vger.kernel.org # v6.17+
Link: https://patch.msgid.link/20261030-work-dcache-ai-v1-1-cd6b86e0ccc7@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Diffstat (limited to 'include/linux/stackprotector.h')
0 files changed, 0 insertions, 0 deletions
