linux-toradex.git/tools/tracing, branch v6.17-rc2

rtla/tests: Test timerlat -P option using actions

2025-07-28T14:22:39+00:00

The -P option is used to set priority of osnoise and timerlat threads.

Extend the test for -P with --on-threshold calling a script that looks
for running timerlat threads and checks if their priority is set
correctly.

As --on-threshold is only supported by timerlat at the moment, this is
only implemented there so far.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Link: https://lore.kernel.org/20250725133817.59237-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/tests: Add grep checks for base test cases

2025-07-28T14:22:38+00:00

Checking for patterns in rtla output with grep was added to test rtla
actions. Add grep checks also for base tests where applicable.

Also fix trace event histogram trigger check to use the correct syntax
for the command-line option so that the test passes with the grep check.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Link: https://lore.kernel.org/20250725133817.59237-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/tests: Limit duration to maximum of 10s

2025-07-25T20:43:57+00:00

Many of the original rtla tests included durations of 1 minute and 30
seconds. Experience has shown this is unnecessary, since 10 seconds as
waiting time for samples to appear.

Change duration of all rtla tests to at most 10 seconds. This speeds up
testing significantly.

Before:
$ make check
All tests successful.
Files=3, Tests=54, 536 wallclock secs
( 0.03 usr  0.00 sys + 20.31 cusr 22.02 csys = 42.36 CPU)
Result: PASS

After:
$ make check
...
All tests successful.
Files=3, Tests=54, 196 wallclock secs
( 0.03 usr  0.01 sys + 20.28 cusr 20.68 csys = 41.00 CPU)
Result: PASS

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-9-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/tests: Add tests for actions

2025-07-25T20:43:57+00:00

Add a bunch of tests covering most of both --on-threshold and --on-end.
Parts sensitive to implementation of hist/top are tested for both.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-8-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/tests: Check rtla output with grep

2025-07-25T20:43:57+00:00

Add argument to the check command in the test suite that takes a regular
expression that the output of rtla command is checked against. This
allows testing for specific information in rtla output in addition
to checking the return value.

Two minor improvements are included: running rtla with "eval" so that
arguments with spaces can be passed to it via shell quotations, and
the stdout of pushd and popd is suppressed to clean up the test output.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add action on end feature

2025-07-25T20:43:57+00:00

Implement actions on end next to actions on threshold. A new option,
--on-end is added, parallel to --on-threshold. Instead of being
executed whenever a latency threshold is reached, it is executed at the
end of the measurement.

For example:

$ rtla timerlat hist -d 5s --on-end trace

will save the trace output at the end.

All actions supported by --on-threshold are also supported by --on-end,
except for continue, which does nothing with --on-end.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add continue action

2025-07-25T20:43:57+00:00

Introduce option to resume tracing after a latency threshold overflow.
The option is implemented as an action named "continue".

Example:
$ rtla timerlat top -q -T 200 -d 1s --on-threshold \
exec,command="echo Threshold" --on-threshold continue
Threshold
Threshold
Threshold
                                     Timer Latency
...

The feature is supported for both hist and top. After the continue
action is executed, processing of the list of actions is stopped and
tracing is resumed.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat_bpf: Allow resuming tracing

2025-07-25T20:43:56+00:00

Currently, rtla-timerlat BPF program uses a global variable stored in a
.bss section to store whether tracing has been stopped.

Move the information to a separate map, so that it is easily writable
from userspace, and add a function that clears the value, resuming
tracing after it has been stopped.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Add action on threshold feature

2025-07-25T20:43:56+00:00

Extend the functionality provided by the -t/--trace option, which
triggers saving the contents of a tracefs buffer after tracing is
stopped, to support implementing arbitrary actions.

A new option, --on-threshold, is added, taking an argument
that further specifies the action. Actions added in this patch are:

- trace[,file=]: Saves tracefs buffer, optionally taking a
filename.
- signal,num=,pid=: Sends signal to process. "parent" might
be specified instead of number to send signal to parent process.
- shell,command=: Execute shell command.

Multiple actions may be specified and will be executed in order,
including multiple actions of the same type. Trace output requested via
-t and -a now adds a trace action to the end of the list.

If an action fails, the following actions are not executed. For
example, this command:

$ rtla timerlat -T 20 --on-threshold trace \
--on-threshold shell,command="grep ipi_send timerlat_trace.txt" \
--on-threshold signal,num=2,pid=parent

will send signal 2 (SIGINT) to parent process, but only if saved trace
contains the text "ipi_send".

This way, the feature can be used for flexible reactions on latency
spikes, and allows combining rtla with other tooling like perf.

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)

rtla/timerlat: Introduce enum timerlat_tracing_mode

2025-07-25T20:43:56+00:00

After the introduction of BPF-based sample collection, rtla-timerlat
effectively runs in one of three modes:

- Pure BPF mode, with tracefs only being used to set up the timerlat
tracer. Sample processing and stop on threshold are handled by BPF.

- tracefs mode. BPF is unsupported or kernel is lacking the necessary
trace event (osnoise:timerlat_sample). Stop on theshold is handled by
timerlat tracer stopping tracing in all instances.

- BPF/tracefs mixed mode - BPF is used for sample collection for top or
histogram, tracefs is used for trace output and/or auto-analysis. Stop
on threshold is handled both through BPF program, which stops sample
collection for top/histogram and wakes up rtla, and by timerlat
tracer, which stops tracing for trace output/auto-analysis instances.

Add enum timerlat_tracing_mode, with three values:

- TRACING_MODE_BPF
- TRACING_MODE_TRACEFS
- TRACING_MODE_MIXED

Those represent the modes described above. A field of this type is added
to struct timerlat_params, named "mode", replacing the no_bpf variable.
params->mode is set in timerlat_{top,hist}_parse_args to
TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output
and/or auto-analysis is requested. timerlat_{top,hist}_main then checks
if BPF is not unavailable or disabled, in that case, it sets
params->mode to TRACING_MODE_TRACEFS.

A condition is added to timerlat_apply_config that skips setting
timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those
are unnecessary, since they only turn off tracing, which is already
turned off in that case, since BPF is used to collect samples).

Cc: John Kacur 
Cc: Luis Goncalves 
Cc: Arnaldo Carvalho de Melo 
Cc: Chang Yin 
Cc: Costa Shulyupin 
Cc: Crystal Wood 
Cc: Gabriele Monaco 
Link: https://lore.kernel.org/20250626123405.1496931-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar 
Signed-off-by: Steven Rostedt (Google)