| Age | Commit message (Collapse) | Author |
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
- Fix a range of use-after-frees during rcu pathwalk caused by non-rcu
frees.
Whatever ->permission(), ->d_revalidate(), ->get_link(), ->d_hash()
and ->d_compare() read in rcu mode has to be freed after a grace
period. umount(2) hides most of it because namespace_unlock() waits
for one, but a chdir() out of a lazily unmounted directory or the
close() of the last file on it drops the last reference without.
* ext4_get_link() reads the sbi and the extent status counters for a
symlink stored in a data block. Both are freed after a grace
period now.
* The unicode map of casefolded ext4, f2fs and tmpfs was freed under
generic_ci_d_hash() and generic_ci_d_compare().
* ntfs3 with nocase freed its sbi and mount options under its dentry
hash and compare. A remount freed the old options right away too.
* eventfs freed an eventfs_inode after an SRCU grace period only
while tracefs_d_revalidate() reads it under rcu_read_lock().
* afs and 9p are only saved by the grace period their private bdi
gets in bdi_unregister(). They stop depending on it.
* create_ipc_ns() leaked the mqueue mount and its superblock on a
late failure. A long-term mount needs kern_unmount().
- Don't let idle threads pin the initramfs.
The idle threads of the secondary cpus are forked from PID 1 with a
private copy of its fs_struct, aren't on the task list so
pivot_root() never moves them and never exit. So the initramfs stayed
pinned forever on every machine with more than one cpu, 70 to 150 MB
in the reports. They share init_fs now like init_task does.
- A mount that copy_tree() leaves out can't take over a lock.
copy_tree() drops mounts of mount namespace files from the copies it
makes for unshare(CLONE_NEWNS) and propagation without looking at
MNT_LOCKED. So root in a user namespace could bind a child's
/proc/<pid>/ns/mnt beneath a locked cover, let it take the lock over,
unshare(CLONE_NEWNS) and read the file. copy_tree() refuses with
EPERM now and the move beneath a locked mount is refused up front.
- Revert "namei: don't reveal overmounted entries in refwalk" from the
rc7 fixes. The bug class is bigger than that patch.
- namei: lookup_fast_for_open() checked ->d_inode while
traverse_mounts() checks the type in d_flags, which is set second, so
a refwalk racing the instantiation could fail an O_CREAT open with
ENOENT. Use d_is_negative() in both places.
- Drop the smp_mb()s in mntput_no_expire_slowpath() and do_umount()
that mnt_get_count()'s own barrier made redundant.
- fsnotify: a mark added while fsnotify_destroy_marks() walks the
connector could be missed and the connector detached, so an fdinfo
read called igrab(NULL). New marks are refused during teardown.
- pipe: a failed second fasync enable on an O_RDWR FIFO removed the
existing reader entry and cleared FASYNC while the writer entry
stayed, a use-after-free in send_sigio() later.
- watch_queue: post_one_notification() could see nr_notes before the
note pool behind it and dereference a NULL array. An unprivileged
user can trigger that with a key watch. Publish nr_notes last.
- pid_namespace: zap_pid_ns_processes() waits without bound for a
tracer outside the namespace to reap its zombies and neither wait was
freezable, so every suspend attempt failed.
- exit: is_child_subreaper and has_child_subreaper shared a word with
other bitfields written without a common lock, so racing prctl()s
lost a bit. They are separate bools now.
- autofs: a notification write that failed on a pipe the daemon had
already replaced put the superblock into catatonic mode again and
closed the replacement. Only do that for the current pipe.
- ovl: reject a lowerdir ending with an unescaped backslash with EINVAL
instead of a confusing ENOENT later.
- ocfs2/dlm: take dlm_reco_state_lock when the debugfs state file walks
the recovery participant list.
- samples/pidfd: fix the userprogs typo that kept pidfd-metadata from
being built.
* tag 'vfs-7.3-rc8.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (24 commits)
Revert "namei: don't reveal overmounted entries in refwalk"
selftests/filesystems: test that a mntns file can't take over a lock
fs: let idle threads share init_fs
namespace: a mount that copy_tree() leaves out can't take over a lock
namei: test for a negative dentry the way traverse_mounts() does
fs: pass flags to copy_fs()
ovl: reject lowerdir ending with unescaped backslash
fsnotify: prevent mark additions during connector teardown
pid_namespace: make zap_pid_ns_processes() freezable
fs: drop the smp_mb()s that mnt_get_count() made redundant
autofs: ignore notification errors from a replaced pipe
ntfs3: free replaced mount options after a grace period
ntfs3: free the sbi after a grace period
unicode: free the map after a grace period
ipc: drop the mqueue mount with kern_unmount() on setup failure
eventfs: free the eventfs_inode after an rcu grace period too
9p: free the session after a grace period
watch_queue: Fix note pool publication race in watch_queue_set_size()
afs: free the super info after a grace period
pipe: preserve existing fasync readers on writer setup failure
...
|
|
Manual merge fixup in audit_tree.c:kill_rules() to work around a conflict
with a UAF fix that was merged for v7.3-rc6.
* audit/dev:
audit: separate file and process capability storage
audit: Fix filter rule accounting after automatic removal
audit: use copied skb length in kauditd_send_multicast_skb()
audit: add missing unlikely hint to audit_dummy_context() wrappers
audit: free proctitle in context so it can be re-set by fork on exec_binprm
|
|
* audit/topic-7.3-audit_fsopen:
audit: add FSOPEN record to log filesystem name
|
|
The menu governor was the last consumer of the per-CPU I/O-wait count
that needed its magnitude, apart from nr_iowait() summing it globally.
Commit 38f83090f515 ("cpuidle: menu: Remove iowait influence") removed
that use. All remaining per-CPU consumers only test for zero/nonzero.
Replace nr_iowait_cpu() with bool cpu_has_iowait() and update the generic
and s390 idle accounting callers. Have nr_iowait() read the counters
directly so that the global total and existing accounting are preserved.
Replace the stale menu comment with a description of the predicate.
Also convert the remaining open-coded idle accounting checks to use the
predicate, as suggested by Prateek.
The conversion was done with coccinelle:
@ predicate depends on patch @
expression cpu;
@@
- nr_iowait_cpu(cpu) > 0
+ cpu_has_iowait(cpu)
@ callers depends on patch @
expression cpu;
@@
- nr_iowait_cpu(cpu)
+ cpu_has_iowait(cpu)
Signed-off-by: Christian Loehle <christian.loehle@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20261016073341.575472-1-christian.loehle@arm.com
|
|
This reverts commit 819224e506bc7c2d61ec6a58ec6e876b505abce9.
The cfs_rq->curr == se case can actually happen with CFS_BANDWIDTH,
so the removed code is needed. Document this.
Fixes: 819224e506bc ("sched/fair: Remove dead code on enqueue_task_fair()")
Reported-by: Christian Loehle <christian.loehle@arm.com>
Signed-off-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/lkml/fd3f9d83-cb5b-496b-b7c1-b40f31cf795e@arm.com/
Link: https://patch.msgid.link/20261015082919.4344-1-kayracizmeci@gmail.com
|
|
update_sg_lb_stats() sums utilization over the CPUs in both
sched_group_span(group) and env->cpus, but takes capacity from
group->sgc->capacity for the whole group. record_sg_llc_stats()
publishes both.
Rejecting an all-affinity-pinned source runqueue removes that CPU from
env->cpus and retries group selection. The retry can publish the
partial utilization with the full capacity.
can_migrate_llc() then treats a busy preferred LLC as having spare
capacity. More tasks can aggregate into it, migrations out can be
blocked, and two busy LLCs may not fall back to generic balancing.
This increases CPU contention and task latency. Found by inspection.
Skip publication on affinity retries using LBF_ALL_PINNED, retaining
the first-pass snapshots and avoiding repeated cpumask checks. On the
first pass, publish either shared LLC statistic only when env->cpus
covers the whole group. Load balancing still uses the affinity-filtered
statistics.
The same window exists during CPU hotplug. env->cpus drops a CPU at
set_cpu_active(false), but the group span and sgc->capacity still
include it until cpuset_cpu_inactive() rebuilds the domains. Keeping
the previous snapshot avoids the mismatched publish. After the rebuild,
the next periodic balance records both statistics again.
Fixes: f025ef275388 ("sched/cache: Record per LLC utilization to guide cache aware scheduling decisions")
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Jemmy Wong <jemmywong512@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20261013084338.55100-1-jemmywong512@gmail.com
|
|
On idle exit the dyntick-idle accounting accounts the time up to now,
then the tick is restarted on its old period. The first tick accounts a
whole TICK_NSEC to whatever runs, although the part of that period
before the idle exit has just been accounted as idle time, or with
IRQ_TIME_ACCOUNTING as IRQ time. That is up to a full tick per idle
exit, and /proc/stat reports more idle time than wall time.
Record how much of the current tick period dyntick-idle has accounted,
and leave it out of the tick that ends the period.
Fixes: cf6444c3e1bb7 ("tick/sched: Unify idle cputime accounting")
Signed-off-by: Stian Halseth <stian@itx.no>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://lore.kernel.org/all/20261004142724.3896396-1-stian@itx.no/
Link: https://patch.msgid.link/20261007142950.547604-1-stian@itx.no
|
|
In the Meta fleet, we found PSI trigger destruction waiting for an RCU
grace period while holding kernfs node_mutex, blocking other users [1].
File close and node drain hold this mutex while calling the release
callback. Cgroup removal and disabling cgroup.pressure also hold
cgroup_mutex during the drain.
The wait was added by commit 0e94682b73bf ("psi: introduce psi monitor")
to protect worker and trigger lookups. Commit 461daba06bdc
("psi: eliminate kthread_worker from psi trigger scheduling mechanism")
replaced the worker queue with a group timer. The scheduler now only
checks whether rtpoll_task is NULL; it uses neither the task nor the
trigger. Commit a06247c6804f ("psi: Fix uaf issue when psi trigger is
destroyed while being polled") removed trigger replacement and polling's
RCU lookup.
Trigger lists are protected by mutexes. Proc poll/select entries are
removed before their file references are dropped. eventpoll_release()
removes epoll entries before the file release callback.
Commit aff037078eca ("sched/psi: use kernfs polling functions for PSI
trigger polling") moved cgroup polling to a kernfs waitqueue whose
lifetime follows the file.
A late timer firing only wakes the group waitqueue. The system group is
static, and commit 5457025fa8ca ("sched/psi: Shut down rtpoll_timer in
psi_cgroup_free()") shuts down the cgroup timer before freeing the group.
Cgroup reclamation already waits for scheduler readers.
Remove synchronize_rcu() from psi_trigger_destroy() and use
rcu_access_pointer() for the worker NULL check. Keep kthread_stop()
outside the trigger mutex.
Tested trigger churn, notifications and cgroup removal in an 8-CPU VM
with KASAN, lockdep, and full and lazy preemption. No warnings were found.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://github.com/bpftrace/user-tools/tree/master/runnablelockmonitor [1]
Link: https://patch.msgid.link/20261010191500.3895736-1-shakeel.butt@linux.dev
|
|
Since commit bd2756811766 ("perf: Rewrite core context handling"),
perf_rotate_context() only updates the context time before scheduling
out the flexible events. It no longer updates the cgroup time, which
ctx_sched_out() used to do.
Cgroup events take their times from the per-cpu cgroup clock, so
between two cgroup switches the rotations see a frozen clock. The
events rotated out lose the running time they accumulated, and the next
update, usually a read, credits the whole interval to the events that
happen to be on the PMU at that moment.
As a result, the time_running of multiplexed cgroup events is wrong,
and so are their scaled counts, as long as no other cgroup runs on the
CPU, e.g. a busy job alone on its CPUs. Counting cycles over one second
of a busy loop in a cgroup gives from half to several times the max
frequency of the CPU.
Update the cgroup time before rotating the CPU context, as was done
before the rewrite.
Fixes: bd2756811766 ("perf: Rewrite core context handling")
Closes: https://lore.kernel.org/r/PR3PR07MB69223331EED5BEBE150375B7849C2@PR3PR07MB6922.eurprd07.prod.outlook.com/
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Antoine Huret <antoine.huret@bull.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261015-perf-cgroup-rotation-time-v1-1-93e177dfee5d@bull.com
|
|
__perf_pmu_remove() walks the group leaders of a CPU and PMU with
perf_event_groups_for_cpu_pmu(), removing each one as it goes.
perf_remove_from_context() takes the leader out of the groups rbtree,
after which rb_next() on that node returns NULL, prematurely ending the
walk. Other groups stay behind in the source context.
On CPU hotplug, they are then torn down with the dying CPU. They keep
running until teardown, while the PMU's interrupt and migrated events
are on a new CPU, which drivers rely on never happening.
Walk ctx->event_list instead and pick out the PMU's events by their
pmu_ctx.
Fixes: bd2756811766 ("perf: Rewrite core context handling")
Signed-off-by: Haris Okanovic <harisokn@amazon.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20261013185603.1347265-2-harisokn@amazon.com
|
|
The locked_vm field in struct user_struct was originally introduced by
commit 789f90fcf6b0 ("perf_counter: per user mlock gift") to provide an
mlock-like budget for perf measured against mm->pinned_vm (since 2011).
perf grants each user a 'gift' of perf_event_mlock_kb * online CPUs of
buffer pages which is charged to user->locked_vm without any RLIMIT_MEMLOCK
check.
Only pages beyond the gift are checked against RLIMIT_MEMLOCK, per-mm, via
mm->pinned_vm, which makes it different from the standard mlock() check,
which is per-mm and made against RLIMIT_MEMLOCK.
However, since this was introduced, a number of other components have
utilised this field where a shared resource needed to be limited against
RLIMIT_MEMLOCK.
The other users are currently MSG_ZEROCOPY, io_uring, AF_XDP, iommufd,
s390 KVM zPCI and most recently, secretmem in commit 97d34aa65c29
("mm/secretmem: properly account locked pages").
This field is shared between all of these, but because the gift is not
checked against RLIMIT_MEMLOCK, user->locked_vm alone can exceed it - 16
CPUs * 516 KiB is already more than the default 8 MiB - after which every
other user of the field fails unconditionally.
This was reported by a user who saw secretmem failing while running a
simultaneous perf record.
Resolve the issue by simply giving perf its own field separate from the
rest.
BPF used user->locked_vm up until commit 80ee81e0403c ("bpf: Eliminate
rlimit-based memory accounting infra for bpf maps") which landed in 5.11.
Thus also eliminate the now-defunct ifdef for CONFIG_BPF_SYSCALL for
user->locked_vm.
This issue exists between all of the components which use user->locked_vm
and perf, however secretmem is the first user who has caused a report over
many years and has been backported, so target the fix at that.
Fixes: 97d34aa65c29 ("mm/secretmem: properly account locked pages")
Closes: https://lore.kernel.org/20261002175651.811343-1-ameer.hamza@truenas.com
Reported-by: Ameer Hamza <ameer.hamza@truenas.com>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261003-perf-locked-vm-fix-v1-1-d214dcaf0d76@kernel.org
|
|
place_entity() inflates an entity's virtual lag with
lag * (load + weight) / load
Commit 4823725d9d1d ("sched/fair: Increase weight bits for avg_vruntime")
replaced the scaled-down weights used in this calculation with
avg_vruntime_weight(), removing the arithmetic headroom provided by
scale_load_down().
The multiplication can overflow s64 before the division even when the
final quotient is representable. A task's hierarchical effective weight
can be as small as 2 after avg_vruntime_weight() scaling under group
scheduling, while entity_lag() bounds |vlag| by
calc_delta_fair(cfs_rq_max_slice(cfs_rq) + TICK_NSEC, se)
For example, with HZ=1000, NICE_0_LOAD=1048576, and the minimum scaled
hierarchical weight scale_load(2)=2048, this bound is 891289600. With 114
maximum-weight entities, the runqueue load is 10361604096. The
intermediate product is:
891289600 * (10361604096 + 2)
= 9235189971864780800
> S64_MAX
The signed overflow corrupts the entity's placement.
Rewrite
lag * (load + weight) / load
as
lag + lag * weight / load
The two expressions are equivalent, but the latter avoids multiplying the
virtual lag by the total runqueue weight. entity_lag() bounds vlag against
the entity's effective weight, so the remaining lag * weight product stays
bounded. rescale_entity() preserves this scale across h_load changes and
already relies on the same product when rescaling vlag.
Keep the original zero-load warning semantics by multiplying lag by weight
after replacing the divisor with one.
Fixes: 4823725d9d1d ("sched/fair: Increase weight bits for avg_vruntime")
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260928124754.1692077-1-sh_def@163.com
|
|
A bpf_task_work context can return to STANDBY before its scheduling
irq_work handler returns. Both callback completion and task_work_add()
failure can publish STANDBY while the irq_work is still BUSY. A new
round can then reinitialize and queue the same irq_work on another CPU
while the old handler is still running.
One concrete interleaving is:
1. CPU 0 successfully queues ctx->work with task_work_add() and pauses
before its SCHEDULING-to-SCHEDULED cmpxchg.
2. The target task on CPU 1 completes the callback, resets ctx->task
and publishes STANDBY.
3. CPU 2 starts a new round, changes STANDBY to PENDING and queues the
same irq_work. The new handler changes PENDING to SCHEDULING, but
task_work_add() returns -ESRCH because its target task has exited.
The handler has not yet reset the context or rolled back the state.
4. CPU 0 resumes and changes the new round's SCHEDULING to SCHEDULED,
even though no task work was queued for that round.
5. The new handler resets ctx->task, but its SCHEDULING-to-STANDBY
cmpxchg fails. The context remains SCHEDULED with a NULL task, and
subsequent attempts to acquire this context fail with -EBUSY.
6. Map-value deletion observes SCHEDULED and queues asynchronous
cancellation, which passes the NULL ctx->task to task_work_cancel().
Overlapping invocations also share one BUSY bit. An invocation can clear
that bit while the other invocation is still running.
Initialize the scheduling irq_work when the context is created and stop
reinitializing it for each round. After claiming the context with the
STANDBY-to-PENDING cmpxchg, check BUSY and reject reuse with -EBUSY if a
previous invocation is still in progress. Checking after the cmpxchg
orders the BUSY read after the STANDBY publication.
No irq_work has been queued for the attempted round at this point, so
the rejection can roll PENDING back to STANDBY. Use cmpxchg for that
rollback to preserve FREED if a concurrent deletion has published it.
Once BUSY clears, scheduling can be retried.
Fixes: 38aa7003e369 ("bpf: task work scheduling kfuncs")
Signed-off-by: Yun Lu <luyun@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://patch.msgid.link/20261012061915.231254-4-luyun_611@163.com
|
|
bpf_task_work_irq() publishes ctx->work with task_work_add() before
changing the context from SCHEDULING to SCHEDULED. The target task can
run the callback on another CPU before the handler reaches its final
state transition.
The handler then reloads ctx->task for cancellation if its final cmpxchg
observes FREED. By that point the callback or context destruction may
already have cleared ctx->task, causing task_work_cancel() to dereference
a NULL task pointer.
One concrete interleaving is:
1. CPU 0 changes PENDING to SCHEDULING and successfully queues
ctx->work with task_work_add(), but has not attempted the final
state cmpxchg.
2. The target task on CPU 1 runs the callback. It changes SCHEDULING
to RUNNING, executes the BPF subprogram, resets ctx->task,
publishes STANDBY and drops the callback's context reference.
3. CPU 2 deletes the map value. It changes STANDBY to FREED and drops
the map's context reference without queuing asynchronous
cancellation.
4. CPU 0 resumes. Its SCHEDULING-to-SCHEDULED cmpxchg observes FREED,
and the cancellation attempt passes NULL to task_work_cancel().
If deletion publishes FREED before the callback claims RUNNING, the
callback takes its FREED exit and drops its context reference. Context
destruction can then clear ctx->task before CPU 0 attempts cancellation.
A controlled reproducer with a delay before the final cmpxchg triggered
the following fault on a v7.3-rc3 based x86-64 KVM guest:
BUG: kernel NULL pointer dereference, address: 0000000000000890
RIP: 0010:task_work_cancel+0xd/0xa0
RDI: 0000000000000000
CR2: 0000000000000890
Call Trace:
<IRQ>
bpf_task_work_irq+0x95/0x100
irq_work_run_list+0x4f/0x90
irq_work_run+0x18/0x50
</IRQ>
0x890 is the task_struct::task_works offset in that build, read through
the NULL task argument.
Capture ctx->task before task_work_add() publishes the callback and use
that pointer for the handler's cancellation attempt. The round still
holds its task reference when the pointer is captured. Releasing that
reference uses bpf_task_release(), which calls put_task_struct_rcu_user(),
so the handler's existing RCU read-side section keeps the captured task
alive even if ctx->task is subsequently cleared.
The asynchronous cancellation path reads ctx->task directly, which is
safe there: queuing it requires deletion to observe SCHEDULED, and that
xchg wins against the callback's transition to RUNNING, so the callback
takes its FREED exit without resetting ctx->task, while the map
reference transferred to the cancellation handler prevents context
destruction until cancellation completes.
The handler-tail and asynchronous cancellation attempts are mutually
exclusive: asynchronous cancellation is queued only if deletion observes
SCHEDULED after the scheduling handler's final cmpxchg has succeeded.
If that cmpxchg instead observes FREED, the scheduling handler attempts
cancellation with the captured pointer.
Fixes: 38aa7003e369 ("bpf: task work scheduling kfuncs")
Reported-by: Jackie Liu <liuyun01@kylinos.cn>
Suggested-by: Mykyta Yatsenko <mykyta.yatsenko5@gmail.com>
Signed-off-by: Yun Lu <luyun@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261012061915.231254-2-luyun_611@163.com
|
|
For standard and PCPU (non-LRU) hash maps with small key sizes (less
than or equal to the word size), comparing keys requires only a single
instruction. Storing a cached 32-bit hash value to shortcut full key
comparisons provides no performance advantage for small keys, and
consumes memory for every element.
This memory can be saved by removing hash from struct htab_elem and
placing it directly before struct htab_elem only for the new
htab_elem_hashed type which is used only when keys are larger than the
word size or for LRU maps. This reduces elem_size by 8 bytes for small
keys while keeping key and hash at constant compile-time offsets from
struct htab_elem across all map types and key sizes.
All element variants requiring hashes get an anonymous
htab_elem_hashed embedding, ensuring that the -8 byte hash offset is
guaranteed for all element types by composition.
Elements can be recycled without a RCU grace period. Before this commit,
alloc_htab_elem() overwrote the key before initializing the value/pptr
and wrote the hash last, so during value assignment on a recycled
element, the new key was paired with the old hash preventing lockless
readers from matching either the old key or the new key except when
there was also a hash collision.
When hashes are omitted for small keys, only the key field guards
lookups. Writing the new key before value/pptr initialization would
allow a concurrent lookup of the new key to match an uninitialized value
or dereference a stale or freed pptr. So move the new key assignment to
the end of alloc_htab_elem() where the hash assignment was done. Like
the hash assignment it replaces, it does not enforce value store
ordering before key stores on weakly ordered architectures for lockless
readers that were already traversing l_new via a stale pointer before
hlist_nulls_add_head_rcu().
Before this commit htab_mem_dtor() was invoked with the start of the
allocation, which is no longer always the address of struct htab_elem
now that elem_offset can be nonzero for non-preallocated, non-per-CPU
maps. So the key_size field of struct htab_btf_record has been replaced
with a value_offset computed at map creation and passed through
bpf_ma_set_dtor(). That reduces the destructor to a call to
bpf_obj_free_fields() at a fixed offset, which is exactly what
rhtab_mem_dtor() did, so rhtab_mem_dtor() has been removed and replaced
with the new implementation of htab_mem_dtor() for BPF_MAP_TYPE_RHASH.
Together with the previous patch, this reduces the minimum standard and
preallocated hash map element size from 64 bytes down to 32 bytes, and
non-preallocated per-CPU element size from 64 bytes down to 40 bytes.
Signed-off-by: T.J. Mercier <tjmercier@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261008104108.993791-3-tjmercier@google.com
|
|
The htab_elem struct is used as the per-element type for all BPF hash
map types and includes bpf_lru_node in a union with a ptr_to_pptr
pointer. For standard (non-LRU, non-PCPU) hash maps, the 24 byte union
allocated for every element is entirely unused. For non-preallocated
PCPU maps, ptr_to_pptr only requires 8 bytes, leaving 16 bytes of unused
overhead in the union. For preallocated PCPU maps ptr_to_pptr is unused
since elements are freed to the PCPU freelist.
Eliminate this per-element memory overhead by splitting htab_elem into
dedicated structures for each map type:
- struct htab_elem: Minimal structure for standard hash maps and
preallocated PCPU maps (saves 24 bytes per element).
- struct htab_elem_pcpu: Structure for non-preallocated PCPU maps
containing ptr_to_pptr (saves 16 bytes per element).
- struct htab_elem_lru: Retains struct bpf_lru_node for LRU maps.
Place lru_node and ptr_to_pptr before struct htab_elem in
htab_elem_lru and htab_elem_pcpu respectively, and track the offset of
htab_elem from the start of the element allocation in htab->elem_offset.
This keeps key and value at constant compile-time offsets from struct
htab_elem across all hash map types, avoiding dynamic key offset
calculations on lookups.
Use sizeof(struct htab_elem_lru) for the element size rollover check in
htab_map_alloc_check() since it is now the largest element variant.
Signed-off-by: T.J. Mercier <tjmercier@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://patch.msgid.link/20261008104108.993791-2-tjmercier@google.com
|
|
The function profiler (trace_stat/function<cpu>) reports Hit, Time,
Avg and s^2 per function. Add a "max" field recording the maximum
single-call duration, which is valuable for real-time tuning and for
investigating occasional latency spikes.
Like Time/Avg, it honors the graph-time and sleep-time options, so it
reflects entry-to-return wall-clock latency.
Link: https://patch.msgid.link/20260924030108.2673654-1-yun.zhou@windriver.com
Signed-off-by: Yun Zhou <yun.zhou@windriver.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A histogram sorts its rows by the key named in sort=, and for a key over a
signed event field it sorts them signed: create_tracing_map_fields() picks
the comparison function with
else
cmp_fn = tracing_map_cmp_num(field->size,
field->is_signed);
hist_trigger_print_key() does not look at is_signed at all. Every numeric
key is printed with %llu, so a sorted histogram over a signed field comes
out looking unsorted. syscalls/sys_exit_lseek carries "long ret", which
SYSCALL_FIELD() marks signed and which is negative on every error:
# echo 'hist:keys=ret:sort=ret' > events/syscalls/sys_exit_lseek/trigger
# echo 1 > events/syscalls/sys_exit_lseek/enable
# (five lseek(2) calls returning -ESPIPE, -EINVAL, -EBADF, 0 and 100)
# cat events/syscalls/sys_exit_lseek/hist
{ ret: 18446744073709551587 } hitcount: 1
{ ret: 18446744073709551594 } hitcount: 1
{ ret: 18446744073709551607 } hitcount: 1
{ ret: 0 } hitcount: 1
{ ret: 100 } hitcount: 1
The rows are in ascending order and the numbers are not, because the three
leading ones are -29, -22 and -9 rendered as their two's complement.
Take the rendering from the same place the sort order comes from: the event
field when the key has one, and the hist_field itself when it does not. The
same histogram then reads
{ ret: -29 } hitcount: 1
{ ret: -22 } hitcount: 1
{ ret: -9 } hitcount: 1
{ ret: 0 } hitcount: 1
{ ret: 100 } hitcount: 1
A ".buckets" key stays unsigned here, and not because it is right.
hist_field_bucket() divides as unsigned, so the group holding S64_MAX also
holds S64_MIN, reported at
https://lore.kernel.org/all/20260914054614.82A9B1F000FF@smtp.kernel.org/
Printing that group signed would name a range that does not contain
S64_MIN, so the grouping has to be fixed with it, in a patch of its own.
Link: https://patch.msgid.link/20260917045918.370993-1-donggeunyoo.kernel@gmail.com
Fixes: 7ef224d1d0e3 ("tracing: Add 'hist' event trigger command")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
DEFINE_HIST_FIELD_FN() reads a trace event field and widens it to the u64
the histogram uses as a key or a value:
type *addr = (type *)(event + hist_field->field->offset);
return (u64)(unsigned long)*addr;
unsigned long is 32 bits on a 32-bit kernel, so the intermediate cast
discards the upper half of a 64-bit field, and widens a signed field by
zero extension from 32 bits rather than sign extension from its own
width. Five of the eight instantiations are wrong there: s64, u64, s32,
s16 and s8.
The cast came in with commit 79e577cbce4c ("tracing: Support string type
key properly"), which also added hist_field_dynstring() and
hist_field_pstring(). Those convert a char * to u64 and do warn without
an unsigned long in between on 32-bit. DEFINE_HIST_FIELD_FN() converts an
integer and never a pointer, so it gains nothing from the cast and only
loses information.
Drop the intermediate cast and let the usual arithmetic conversions widen
*addr according to its own type. unsigned long is 64 bits on LP64, so the
two forms are identical there and 64-bit kernels are unaffected.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260917015532.103081-1-donggeunyoo.kernel@gmail.com
Fixes: 79e577cbce4c ("tracing: Support string type key properly")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260914054614.82A9B1F000FF@smtp.kernel.org/
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A CPU coming online for the first time gets its per CPU buffer allocated
by trace_rb_cpu_prepare(), which then adds it to buffer->cpumask. If that
happens while ring_buffer_subbuf_order_set() is running, the CPU shows up
after the allocation loop and the install loop walks it with an empty
new_pages list, corrupting the buffer.
Take cpus_read_lock() as ring_buffer_resize() does.
Cc: stable@vger.kernel.org
Fixes: f9b94daa542a ("ring-buffer: Set new size of the ring buffer sub page")
Link: https://patch.msgid.link/20260916192628.274136-1-devnexen@gmail.com
Signed-off-by: David Carlier <devnexen@gmail.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
for_each_buffer_cpu() already iterates over buffer->cpumask. Remove the
unnecessary cpumask_test_cpu() found in ring_buffer_subbuf_order_set().
Link: https://patch.msgid.link/20260916152813.105082-3-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
ring_buffer_subbuf_order_set() checks for resize_disabled twice under
the same buffer->mutex hold. Moreover, this check duplicates the logic
in ring_buffer_resize(). Create a common helper rb_resize_disabled() to
factor out this code.
Link: https://patch.msgid.link/20260916152813.105082-2-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The current funcgraph-retval implementation suffers from two accuracy
issues:
1. Void-returning functions still print a return value, creating
misleading noise in the trace output.
2. For functions returning narrower types (e.g., char, short), the
displayed value can be incorrect because high bits of the register
may contain undefined data.
This patch addresses both problems by leveraging BTF to obtain the exact
return type of each traced kernel function. The key changes are:
1. Void function filtering: Functions with void return type no longer
display any return value in the trace output, eliminating unnecessary
clutter.
2. Type-aware value formatting: The return value is now properly truncated
to match the actual width of the return type before being displayed.
Additionally, the value is formatted according to its type for better
human readability.
Here is an output comparison:
Before:
# perf ftrace -G vfs_read --graph-opts retval
...
1) | touch_atime() {
1) | atime_needs_update() {
1) 0.069 us | make_vfsuid(); /* ret=0x0 */
1) 0.067 us | make_vfsgid(); /* ret=0x0 */
1) | current_time() {
1) 0.197 us | ktime_get_coarse_real_ts64_mg(); /* ret=0x187f886aec3ed6f5 */
1) 0.352 us | } /* current_time ret=0x69380753 */
1) 0.792 us | } /* atime_needs_update ret=0x0 */
1) 0.937 us | } /* touch_atime ret=0x0 */
After:
# perf ftrace -G vfs_read --graph-opts retval
...
2) | touch_atime() {
2) | atime_needs_update() {
2) 0.070 us | make_vfsuid(); /* ret=0x0 */
2) 0.070 us | make_vfsgid(); /* ret=0x0 */
2) | current_time() {
2) 0.162 us | ktime_get_coarse_real_ts64_mg();
2) 0.312 us | } /* current_time ret=0x69380649(trunc) */
2) 0.753 us | } /* atime_needs_update ret=false */
2) 0.899 us | } /* touch_atime */
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
Link: https://patch.msgid.link/20260916133218.1282590-4-dolinux.peng@gmail.com
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The trace_btf.c file provides BTF helper functions used by the ftrace
subsystem. Make the compilation of trace_btf.c solely depend on
CONFIG_DEBUG_INFO_BTF, allowing features like funcgraph-retval to also
utilize these helpers.
Additionally, the redundant dependency on CONFIG_PROBE_EVENTS_BTF_ARGS
is removed, as CONFIG_DEBUG_INFO_BTF already depends on
CONFIG_BPF_SYSCALL.
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Xiaoqin Zhang <zhangxiaoqin@xiaomi.com>
Link: https://patch.msgid.link/20260916133218.1282590-3-dolinux.peng@gmail.com
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The function graph trace formatter can be called from ftrace_dump_one()
with interrupts disabled. When ftrace_dump_on_oops is enabled, the trace
panic notifier can reach ftrace_dump() from the panic path, including the
nmi_panic() path used by the hard lockup detector.
btf_find_func_proto() calls bpf_find_btf_id(), which may initialize vmlinux
BTF under a mutex. Skip the BTF lookup when running in an atomic context
or with interrupts disabled, and fall back to the existing return value
formatting.
Link: https://patch.msgid.link/20260916133218.1282590-2-dolinux.peng@gmail.com
Fixes: ebeed8d4a555 ("tracing/probes: Move finding func-proto API and getting func-param API to trace_btf")
Signed-off-by: pengdonglin <pengdonglin@xiaomi.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A key with the ".buckets=N" modifier is stored as the start of its bucket,
and hist_trigger_print_key() prints the range it covers as start to
start + N - 1. That addition overflows once the start is within N of
U64_MAX:
# echo 'hist:keys=offset.buckets=10' >> \
events/syscalls/sys_enter_lseek/trigger
# ./seeker 5 -1 # lseek(0, 5) and lseek(0, -1)
# cat events/syscalls/sys_enter_lseek/hist
...
{ offset: ~ 18446744073709551610-3 } hitcount: 1
{ offset: ~ 0-9 } hitcount: 1
...
18446744073709551610 + 10 - 1 does not fit in 64 bits, so the range ends
below where it starts.
The bucket does end at U64_MAX: it holds every value hist_field_bucket()
maps onto its start, and there is no value above U64_MAX. Clamp the printed
end there. Printed honestly the group is shorter than the size asked for,
which histogram.rst does not mention, so say it there.
Link: https://patch.msgid.link/20260914132346.2234731-1-donggeunyoo.kernel@gmail.com
Fixes: de9a48a360b7 ("tracing: Add linear buckets to histogram logic")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
An action that names a field on another event needs that field as a
variable, so create_field_var_hist() registers a second hist trigger on
that event to provide it, and records it in
target_hist_data->field_var_hists[].
If a later part of the same command fails, create_actions() returns an
error and event_hist_trigger_parse() jumps to out_free. The write fails
with -EINVAL and sched_switch gets no trigger, but:
# echo 'hist:keys=pid:ts0=common_timestamp.usecs' >> \
events/sched/sched_waking/trigger
# echo 'my_synth u64 lat; int a; int b' >> synthetic_events
# echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0:\
onmatch(sched.sched_waking).my_synth($wakeup_lat,prio,nosuchfld)'\
>> events/sched/sched_switch/trigger
# cat events/sched/sched_waking/trigger
hist:keys=pid:vals=hitcount:ts0=common_timestamp.usecs:sort=hitcount:size=2048:clock=global [active]
hist:keys=pid:vals=hitcount:synthetic_prio=prio:sort=hitcount:size=2048 [active]
The second histogram is the one created for 'prio'. It is still active and
still counting.
out_free reaches destroy_field_var_hists(), which frees the
field_var_hists[] entries, but never unregister_field_var_hists(), which is
what removes the triggers and runs only from event_hist_trigger_free(). The
entry is freed and the trigger is left with nothing tracking it.
Unregister the field variable histograms in out_free as well, and record
each one as soon as its trigger is registered rather than after the lookup
that follows it, so no exit can leave a registered trigger untracked. The
entry then belongs to field_var_hists[] and is released by
destroy_field_var_hists(), so freeing it on that exit would be a double
free.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260914104036.1768529-1-donggeunyoo.kernel@gmail.com
Fixes: 02205a6752f2 ("tracing: Add support for 'field variables'")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260913204302.DB2551F000FF@smtp.kernel.org/
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
get_hist_field_flags() returns "stacktrace" for any field carrying
HIST_FIELD_FL_STACKTRACE, including the common_stacktrace pseudo-field,
which has no ftrace_event_field behind it. parse_field() no longer takes
the modifier there:
if (stack_modifier &&
(!field || field->filter_type != FILTER_STACKTRACE)) {
hist_err(tr, HIST_ERR_BAD_FIELD_MODIFIER, errpos(field_str));
so expr_field_str() renders "common_stacktrace.stacktrace", a spelling
that cannot be parsed back.
Report the modifier only when there is a field to report it for.
hist_field_print(), the other caller of get_hist_field_flags(), excludes
HIST_FIELD_FL_STACKTRACE before it calls, so this is confined to
expr_field_str(), whose only key renderer arrives in the next patch.
Link: https://patch.msgid.link/20260914053443.981201-4-donggeunyoo.kernel@gmail.com
Fixes: a5e70ba87ca8 ("tracing: Fix memory corruption from the histogram stacktrace modifier")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
expr_field_str() renders a HIST_FIELD_FL_BUCKET field as "pid.buckets",
dropping the count, and parse_field() rejects that spelling.
get_hist_field_flags() returns the bare string "buckets" and keeps the
count in hist_field->buckets. hist_field_print() appends it,
expr_field_str() never did. Nothing reaches it today, since .buckets is
refused on a value and no expression can carry one, but the next patch
renders histogram keys with expr_field_str(), where a .buckets key is
legal.
Append the count, as hist_field_print() does.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260914053443.981201-3-donggeunyoo.kernel@gmail.com
Fixes: de9a48a360b7 ("tracing: Add linear buckets to histogram logic")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
A key's bucket size reads back negative once it exceeds LONG_MAX:
# echo 'hist:keys=next_pid.buckets=18446744073709551615' > trigger
# cat trigger
hist:keys=next_pid.buckets=-1:vals=hitcount:...
parse_field() takes the size with kstrtoul() and rejects only zero, so the
whole unsigned long range is accepted and stored, but hist_field_print()
renders it with %ld.
Print it with %lu.
Link: https://patch.msgid.link/20260914053443.981201-2-donggeunyoo.kernel@gmail.com
Fixes: de9a48a360b7 ("tracing: Add linear buckets to histogram logic")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The codebase scatters hardcoded minimum page limits across multiple
places. Introduce RING_BUFFER_MIN_NR_PAGES and rb_calculate_nr_pages()
to formalize the requirement of two pages in a ring buffer.
Link: https://patch.msgid.link/20260911193937.602202-4-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Since commit 8a5f63637890 ("ring-buffer: Fix subbuf resize race with
ring buffer readers"), "capacity" refers to the memory available for
events, while "entries" refers to the number of events in a ring buffer.
As a consequence, rename trace_array::entries to capacity to align with
this convention.
While at it, rename also tracing_total_entries_fops. It is the aggragate
of all ring buffer capacities.
Link: https://patch.msgid.link/20260908090937.838544-4-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Since commit 8a5f63637890 ("ring-buffer: Fix subbuf resize race with
ring buffer readers"), "capacity" refers to the memory available for
events, while "entries" refers to the number of events in a ring buffer.
As a consequence, rename trace_array::entries to capacity to align with
this convention and avoid any confusion.
Rename also the helper functions trace_set_buffer_entries and
update_buffer_entries.
Link: https://patch.msgid.link/20260908090937.838544-3-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Since commit 8a5f63637890 ("ring-buffer: Fix subbuf resize race with
ring buffer readers"), "capacity" refers to the memory available for
events (excluding subbuffer headers) while "size" refers to the raw
allocation. ring_buffer_size() actually calculates this event capacity.
Rename it to ring_buffer_capacity() to align with this convention.
Link: https://patch.msgid.link/20260908090937.838544-2-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Make it possible to specify a trace event filter from the
kernel command line.
Example:
trace_event=syscalls:sys_enter_clock_nanosleep:filter:which_clock==0
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-7-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Make it possible to directly enable a filter when enabling an event.
For now this is unused, an upcoming patch will wire it up.
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-6-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The parsing logic is a bit complicated and about to become more so.
Add a unit test to avoid regressions.
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-5-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
This new function can be tested with a KUnit test to prevent regressions
when adding new fields to the format.
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-4-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
ftrace_set_clr_event() modifies its input buffer during parsing.
Before returning to the caller these modifications are undone.
This is restoration is error prone and was forgotten before.
Upcoming changes will also make it more complicated.
Copy the input argument and modify that copy and drop the
restoration logic.
Also mark the input argument as const.
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-3-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The function is also declared in include/linux/trace_events.h which is
even included from kernel/trace/trace.h.
Having two declarations creates churn when changing the signature.
Remove the unnecessary, duplicate declaration.
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-2-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When both the func_stack_trace option is enabled first, the func-args
option can not be enabled. Same if the func-args option is enabled first,
the func_stack_trace option can not be enabled.
Allow the two to work together. The stack trace is already slow and
testing the current tracer flags should be in the noise of overhead.
Link: https://patch.msgid.link/20260825191036.11b19036@gandalf.local.home
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
If a user were to use a glob expression with the "within" filter, it
currently just fails with an errno return but does not update the
error_log to tell the user why it failed. Update it to use the parsing
error log to tell the user why it failed.
# echo 'within ~ "*_read"' > events/kmem/kmem_cache_alloc/filter
-bash: echo: write error: Invalid argument
# cat error_log
[ 4676.606481] event filter parse error: error: 'within' filter cannot take glob expressions
Command: within ~ "*_read"
^
Link: https://patch.msgid.link/20260821233050.488131029@kernel.org
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Low-level kernel functions are called from many different paths.
When debugging, it is often useful to filter trace events to only
those occurring within a specific call chain.
Add a "within" filter predicate that tests whether a given function
appears in the current call stack at event time. The function name
is resolved to its address range via kallsyms during filter setup;
at runtime, stack_trace_save() captures the call stack and compares
each return address against the stored range.
Example:
echo 'within == "vfs_read"' > events/sched/sched_switch/filter
Only "==" and "!=" operators are supported. The filter depends on
CONFIG_STACKTRACE.
Link: https://patch.msgid.link/20260821233050.335986786@kernel.org
Link: https://patch.msgid.link/20260608145556.94931-1-chenjun102@huawei.com
Signed-off-by: Chen Jun <chenjun102@huawei.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
irq_seq_show() must be idempotent since it may be called multiple times
to render the same record, e.g. when seq_file's buffer is exceeded. It
emits the CPU header behind the one-shot print_header flag, which the
first pass clears, so the header is lost with the discarded buffer.
This can be seen on large systems, such as 8g.48xl (192 CPUs) and
8i.96xl (384 CPUs) AWS instance types:
$ head -1 /proc/interrupts
9: 0 0 ... 0 GICv3 25 Level vgic
Replace print_header with the position of the first record and include
the header whenever that record is emitted.
Fixes: 171cc0d9eed1 ("genirq/proc: Speed up /proc/interrupts iteration")
Signed-off-by: Haris Okanovic <harisokn@amazon.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Michael Kelley <mhklinux@outlook.com>
Reviewed-by: Radu Rendec <radu@rendec.net>
Link: https://patch.msgid.link/20260911-dev-harisokn-proc-interrupts-fix-b4-v1-1-f9ff208efcfa@amazon.com
|
|
Correct the spelling of "descriptor" and "preceded" in the kernel-doc for
msi_insert_desc() and msi_next_desc().
Signed-off-by: Hemanth Selam <hemanth.selam@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Radu Rendec <radu@rendec.net>
Link: https://patch.msgid.link/20261009064807.70345-1-hemanth.selam@gmail.com
|
|
irq_thread() registers irq_thread_dtor() as exit task_work using a
stack-allocated callback_head. On normal interrupt thread exit, this work
gets cancelled and never runs.
On x86, an abnormal interrupt thread exit (e.g. handler oops) discards the
stack, including the callback_head, while that callback_head is still
pending. By the time exit_task_work() runs, the stack memory formerly
backing callback_head is already clobbered.
irq_thread() <-- on_exit_work allocated
** IRQ handler faults **
oops_end()
rewind_stack_and_make_dead() <-- on_exit_work freed
make_task_dead()
do_exit(SIGSEGV)
...
exit_shm() <-- on_exit_work clobbered
...
exit_task_work() <-- stack use-after-free
exit_shm()+0x30 <-- wild jump to return addr
Thus, a benign threaded IRQ oops (e.g. a NULL pointer deref) is
turned into a wild jump. This is caught by kernel IBT:
** first fault **
Oops: general protection fault, kernel NULL pointer dereference 0x3c0: 0000 [#1] SMP NOPTI
...
** second fault **
Missing ENDBR: exit_shm+0x30/0x1c0
kernel BUG at arch/x86/kernel/cet.c:133!
...
? task_work_run+0x5c/0x90
? do_exit+0x209/0x460
? rewind_stack_and_make_dead+0x16/0x20
...
note: irq/165-AudioDS[1427] exited with irqs disabled
Fixing recursive fault but reboot is needed!
Fix this by storing the callback_head in struct irqaction. The irqaction
is only freed once irq_thread_dtor() finishes.
Fixes: 2deb4be28077 ("x86/dumpstack: When OOPSing, rewind the stack before do_exit()")
Signed-off-by: Richard Patel <ripatel@wii.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261009145842.1616117-3-ripatel@wii.dev
|
|
When an interrupt thread exits abnormally via do_exit() (e.g. the interrupt
handler oopses) without returning from irq_thread(), a concurrent call to
free_irq() could free the irqaction before the irq_thread_dtor() exit task
work runs.
irq thread (oopsed) free_irq()
------------------- ----------
do_exit()
exit_mm()
complete_vfork_done() -------> kthread_stop() returns
exit_files() ... kfree(action)
exit_task_work()
irq_thread_dtor()
action->... <- use-after-free
Track the pending exit work with the IRQTF_EXIT_WORK thread flag, and have
__free_irq() wait for the bit to clear before kfree().
irq_thread_dtor() only runs on irq_thread abnormal exit. If that dtor also
faults, free_irq() will hang indefinitely, but double IRQ thread faults
require a reboot anyways.
[ tglx: Add missing include ]
Fixes: ed3e694d78cc ("move exit_task_work() past exit_files() et.al.")
Signed-off-by: Richard Patel <ripatel@wii.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261009145842.1616117-2-ripatel@wii.dev
|
|
predicate_parse() uses the first operator stack entry before processing
any opening parentheses. A push then increments top before writing.
The existing check still permits that write when top already points to
the last allocated entry, which can access beyond the operator stack.
Reject the push when top reaches the last allocated entry.
Cc: stable@vger.kernel.org
Fixes: 39f7c41c908b ("tracing/filters: Enable filtering a cpumask field by another cpumask")
Link: https://patch.msgid.link/atELO9L0I9kg5CpC@Taeyangs-MacBook-Pro.local
Signed-off-by: Taeyang Lee <0wn@theori.io>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
predicate_parse() adds two to nr_preds to reserve the terminal TRUE
and FALSE entries. It also uses that increased value as the limit for
parsed predicates. If the initial count is smaller than the number
actually parsed, the program can fill the allocation and subsequent
accesses can go past its end.
Keep nr_preds as the predicate limit and use a separate value for the
allocation size. This leaves space for both terminal entries and a
zeroed entry when parsing stops at the limit.
Cc: stable@vger.kernel.org
Fixes: 39f7c41c908b ("tracing/filters: Enable filtering a cpumask field by another cpumask")
Link: https://patch.msgid.link/atELCd1XjKVhrkFL@Taeyangs-MacBook-Pro.local
Signed-off-by: Taeyang Lee <0wn@theori.io>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
tracing_set_tracer() assigns the flags of the newly selected tracer
with:
tr->current_trace_flags = t->flags ? : t->tracer->flags;
If the new tracer defines neither static flags nor default flags, like
hwlat, irqsoff or wakeup, add_tracer() has no per instance flags copy
for it, and both operands are NULL, so tr->current_trace_flags ends up
being NULL as well.
The ring buffer is not cleared when switching tracers: it is reset
inside tracer_init(), after current_trace_flags was already assigned.
If the buffer still holds function_graph entries of the previous
tracer, printing one of them dereferences the NULL pointer. This was
hit by cat trace during an ftrace stress test:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
RIP: 0010:print_graph_function_event+0xe/0x20
Call Trace:
s_show+0x4d/0x110
seq_read_iter+0x2d4/0x490
seq_read+0x130/0x170
vfs_read+0xb5/0x370
ksys_read+0x6d/0xe0
do_syscall_64+0xbc/0x450
The snapshot buffer is not reset at all, so this can be reproduced
without a race:
echo function_graph > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/snapshot
echo hwlat > /sys/kernel/tracing/current_tracer
cat /sys/kernel/tracing/snapshot
Rather than teaching every user of tr->current_trace_flags about this
corner case, make sure it is never NULL: when the newly selected
tracer has no flags at all, tracing_set_tracer() falls back to a
static empty no_tracer_flags. run_tracer_selftest() can hit the same
case, so the same fallback is added there too. The existing users
already handle opts == NULL, so these tracers keep behaving as before.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261015130658.221198-1-hao.ge@linux.dev
Fixes: e29aa918a928 ("tracing: Have function graph tracer define options per instance")
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|