linux-toradex.git/tools/tracing/rtla/Makefile, branch master

rtla/tests: Run Test::Harness in verbose mode

2026-01-07T14:57:16+00:00

Add -v flag to prove command to also print the names of tests that
succeeded, not only those that failed, to allow easier debugging of the
test suite.

Also, drop printing the option and value to stdout in
check_with_osnoise_options, which was a debugging print that was
accidentally left in the final commit, and which would be otherwise now
visible in make check output, as stdout is no longer suppressed.

Suggested-by: Crystal Wood 
Reviewed-by: Wander Lairson Costa 
Link: https://lore.kernel.org/r/20251126144205.331954-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar

rtla/tests: Test BPF action program

2026-01-07T14:57:16+00:00

Add a test that implements a BPF program writing to a test map, which
is attached to RTLA via --bpf-action to be executed on theshold
overflow.

A combination of --on-threshold shell with bpftool (which is always
present if BPF support is enabled) is used to check whether the BPF
program has executed successfully.

Suggested-by: Crystal Wood 
Link: https://lore.kernel.org/r/20251126144205.331954-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar

rtla/timerlat: Add example for BPF action program

2026-01-07T14:57:16+00:00

Add an example BPF action program that prints the measured latency to
the tracefs buffer via bpf_printk().

A new Makefile target, "examples", is added to build the example. In
addition, "sample/" subfolder is renamed to "example".

If BPF skeleton support is unavailable or disabled, a warning will be
displayed when building the BPF action program example.

Link: https://lore.kernel.org/r/20251126144205.331954-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar

Merge tag 'trace-tools-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

2025-03-28T00:03:01+00:00

Pull tracing tooling updates from Steven Rostedt:

 - Allow RTLA to collect data via BPF

   The current implementation of rtla uses libtracefs and libtraceevent
   to pull sample events generated by the timerlat tracer from the trace
   buffer. rtla then processes the sample by updating the histogram and
   summary (current, maximum, minimum, and sum values) as well as checks
   if tracing has been stopped due to threshold overflow.

   In use cases where a large number of samples is being generated, that
   is, with measurements running on many CPUs and with a low interval,
   this sample processing design causes a significant CPU load on the
   rtla side. Furthermore, with >100 CPUs and 100us interval, rtla was
   reported as not being able to keep up with the samples and dropping
   most of them, leading to it being unusable.

   Change the way the timerlat trace processes samples by attaching a
   BPF program to the trace event using the BPF skeleton feature of
   bpftool. Unlike the current implementation, the BPF implementation
   does not check whether tracing is stopped (in BPF mode, tracing is
   always off to improve performance), but waits for a write to a BPF
   ringbuffer instead. This allows rtla to exit immediately when a
   threshold is violated, without waiting for the next iteration of the
   while loop.

   If the requirements for the BPF implementation are not met, either at
   build time or at run time, the current implementation is used as
   fallback. Which implementation is being used can be seen when running
   rtla timerlat with "-D" option. rtla can be forced to run in non-BPF
   mode by setting the RTLA_NO_BPF option to 1, for debugging purposes.

 - Fix LD_FLAGS from being dropped in build

 - Refactor code to remove duplication of save_trace_to_file

 - Always set options and do not rely on default settings

   Do not rely on the default kernel settings of the tracers when
   starting. They could have been changed by the user which gives
   inconsistent results. Always set the options that rtla expects.

 - Add creation of ctags and TAGS for traversing code

* tag 'trace-tools-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
  rtla: Add the ability to create ctags and etags
  rtla/tests: Test setting default options
  rtla/tests: Reset osnoise options before check
  rtla: Always set all tracer options
  rtla/osnoise: Set OSNOISE_WORKLOAD to true
  rtla: Unify apply_config between top and hist
  rtla/osnoise: Unify params struct
  rtla: Fix segfault in save_trace_to_file call
  tools/build: Use SYSTEM_BPFTOOL for system bpftool
  rtla: Refactor save_trace_to_file
  tools/rv: Keep user LDFLAGS in build
  rtla/timerlat: Test BPF mode
  rtla/timerlat_top: Use BPF to collect samples
  rtla/timerlat_top: Move divisor to update
  rtla/timerlat_hist: Use BPF to collect samples
  rtla/timerlat: Add BPF skeleton to collect samples
  rtla: Add optional dependency on BPF tooling
  tools/build: Add bpftool-skeletons feature test
  rtla/timerlat: Unify params struct

tools/build: Use SYSTEM_BPFTOOL for system bpftool

2025-03-26T14:31:31+00:00

The feature test for system bpftool uses BPFTOOL as the variable to set
its path, defaulting to just "bpftool" if not set by the user.

This conflicts with selftests and a few other utilities, which expect
BPFTOOL to be set to the in-tree bpftool path by default. For example,
bpftool selftests fail to build:

$ make -C tools/testing/selftests/bpf/
make: Entering directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf'

make: *** No rule to make target 'bpftool', needed by '/home/tglozar/dev/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h'.  Stop.
make: Leaving directory '/home/tglozar/dev/linux/tools/testing/selftests/bpf'

Fix the problem by renaming the variable used for system bpftool from
BPFTOOL to SYSTEM_BPFTOOL, so that the new usage does not conflict with
the existing one of BPFTOOL.

Cc: John Kacur 
Cc: Luis Goncalves 
Link: https://lore.kernel.org/20250326004018.248357-1-tglozar@redhat.com
Fixes: 8a635c3856dd ("tools/build: Add bpftool-skeletons feature test")
Closes: https://lore.kernel.org/linux-kernel/5df6968a-2e5f-468e-b457-fc201535dd4c@linux.ibm.com/
Reported-by: Venkat Rao Bagalkote 
Tested-by: Venkat Rao Bagalkote 
Suggested-by: Quentin Monnet 
Acked-by: Quentin Monnet 
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add BPF skeleton to collect samples

2025-03-04T17:35:18+00:00

Add BPF program that attaches to the osnoise:timerlat_sample tracepoint
and collects both the summary and the histogram (if requested) into BPF
maps (one map of each kind per context).

The program is designed to be used for both timerlat-top and
timerlat-hist. If using with timerlat-top, the "entries" parameter is
set to zero, which prevents the BPF program from recording histogram
entries. In that case, the maps for histograms do not have to be
created, as the BPF verifier will identify the code using them as
unreachable.

An IRQ or thread latency threshold might be supplied to stop recording
if hit, similar to the timerlat tracer threshold, which stops ftrace
tracing if hit. A BPF ringbuffer is used to signal threshold overflow to
userspace. In aa-only mode, this is the only function of the BPF
program.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Gabriele Monaco 
Cc: Clark Williams 
Link: https://lore.kernel.org/20250218145859.27762-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla: Add optional dependency on BPF tooling

2025-03-04T17:35:17+00:00

If tooling required for building BPF CO-RE skeletons is present (that
is, libbpf, clang with BPF CO-RE support, and bpftool), turn on
HAVE_BPF_SKEL flag.

Those requirements are similar to what perf requires, with the
difference of using system libbpf and bpftool instead of in-tree
versions.

rtla can be forcefully built without BPF skeleton support by setting
BUILD_BPF_SKEL=0 manually; in that case, a warning is displayed.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Gabriele Monaco 
Cc: Clark Williams 
Link: https://lore.kernel.org/20250218145859.27762-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

Merge tag 'perf-tools-fixes-for-v6.14-2-2025-02-25' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools

2025-02-25T21:32:32+00:00

Pull perf tools fixes from Arnaldo Carvalho de Melo:

 - Fix tools/ quiet build Makefile infrastructure that was broken when
   working on tools/perf/ without testing on other tools/ living
   utilities.

* tag 'perf-tools-fixes-for-v6.14-2-2025-02-25' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools:
  tools: Remove redundant quiet setup
  tools: Unify top-level quiet infrastructure

tools: Remove redundant quiet setup

2025-02-18T19:27:43+00:00

Q is exported from Makefile.include so it is not necessary to manually
set it.

Reviewed-by: Jiri Olsa 
Signed-off-by: Charlie Jenkins 
Acked-by: Andrii Nakryiko 
Acked-by: Quentin Monnet 
Cc: Adrian Hunter 
Cc: Alexander Shishkin 
Cc: Alexei Starovoitov 
Cc: Benjamin Tissoires 
Cc: Daniel Borkmann 
Cc: Daniel Lezcano 
Cc: Eduard Zingerman 
Cc: Hao Luo 
Cc: Ian Rogers 
Cc: Ingo Molnar 
Cc: Jiri Kosina 
Cc: John Fastabend 
Cc: Josh Poimboeuf 
Cc: KP Singh 
Cc: Lukasz Luba 
Cc: Mark Rutland 
Cc: Martin KaFai Lau 
Cc: Mykola Lysenko 
Cc: Namhyung Kim 
Cc: Peter Zijlstra 
Cc: Rafael J. Wysocki 
Cc: Shuah Khan 
Cc: Song Liu 
Cc: Stanislav Fomichev 
Cc: Steven Rostedt (VMware) 
Cc: Yonghong Song 
Cc: Zhang Rui 
Link: https://lore.kernel.org/r/20250213-quiet_tools-v3-2-07de4482a581@rivosinc.com
Signed-off-by: Arnaldo Carvalho de Melo

tools/rtla: Add basic test suite

2025-01-23T17:21:45+00:00

Implement a simple TAP-based test engine in bash and a few basic tests
using it, to be used to check for bugs and regressions.

A new "check" target is added to the rtla Makefile that runs the test suite
using the "prove" command implemented by Test::Harness.

The only test format currently supported is running rtla with defined
command arguments per test, checking its exit code. In case the exit
code is non-zero, the output of rtla is displayed, together with the
exit code.

The test cases are adopted from rtla tests in the Continuous Kernel
Integration (CKI) project [1] with the authors' approval.

[1] https://gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/blob/main/rt-tests/us/rtla/

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Chang Yin 
Cc: Qiao Zhao 
Link: https://lore.kernel.org/20250120135630.802111-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)