linux-toradex.git/kernel/trace/trace_functions_graph.c, branch v6.12-rc4

tracing/fgraph: Have fgraph handle previous boot function addresses

2024-08-15T12:35:48+00:00

Update the function graph code to modify the function addresses for a
previous boot buffer so that it matches the current kallsyms (note this
does not handle module addresses, yet).

After a reboot, instead of seeing:

 # trace-cmd show -B boot_mapped | tail -n30
       swapper/0-1       [000] d..2.    56.286470:  0)   0.481 us    |                    0xffffffff925da5c4();
       swapper/0-1       [000] d....    56.286471:  0)   4.065 us    |                  }
       swapper/0-1       [000] d....    56.286471:  0)   4.920 us    |                }
       swapper/0-1       [000] d..1.    56.286472:  0)               |                0xffffffff92536254() {
       swapper/0-1       [000] d..1.    56.286472:  0) + 28.974 us   |                  0xffffffff92534e30();
       swapper/0-1       [000] d....    56.286516:  0) + 43.881 us   |                }
       swapper/0-1       [000] d..1.    56.286517:  0)               |                0xffffffff925136c4() {
       swapper/0-1       [000] d..1.    56.286518:  0)               |                  0xffffffff92514a14() {
       swapper/0-1       [000] d..1.    56.286518:  0)   6.003 us    |                    0xffffffff92514200();
       swapper/0-1       [000] d....    56.286529:  0) + 11.510 us   |                  }
       swapper/0-1       [000] d....    56.286529:  0) + 12.895 us   |                }
       swapper/0-1       [000] d....    56.286530:  0) ! 382.884 us  |              }
       swapper/0-1       [000] d..1.    56.286530:  0)               |              0xffffffff92536444() {
       swapper/0-1       [000] d..1.    56.286531:  0)               |                0xffffffff92536254() {
       swapper/0-1       [000] d..1.    56.286531:  0) + 26.335 us   |                  0xffffffff92534e30();
       swapper/0-1       [000] d....    56.286560:  0) + 29.511 us   |                }
       swapper/0-1       [000] d....    56.286561:  0) + 30.452 us   |              }
       swapper/0-1       [000] d..1.    56.286562:  0)               |              0xffffffff9253c014() {
       swapper/0-1       [000] d..1.    56.286562:  0)               |                0xffffffff9253bed4() {
       swapper/0-1       [000] d..1.    56.286563:  0) + 13.465 us   |                  0xffffffff92536684();
       swapper/0-1       [000] d....    56.286577:  0) + 14.651 us   |                }
       swapper/0-1       [000] d....    56.286577:  0) + 15.821 us   |              }
       swapper/0-1       [000] d..1.    56.286578:  0)   0.667 us    |              0xffffffff92547074();
       swapper/0-1       [000] d..1.    56.286579:  0)   0.453 us    |              0xffffffff924f35c4();
       swapper/0-1       [000] d....    56.286580:  0) # 3906.348 us |            }
       swapper/0-1       [000] d..1.    56.286581:  0)               |            0xffffffff92531a14() {
       swapper/0-1       [000] d..1.    56.286581:  0)   0.518 us    |              0xffffffff92505cb4();
       swapper/0-1       [000] d..1.    56.286595:  0)               |              0xffffffff92db83c4() {
       swapper/0-1       [000] d..1.    56.286596:  0)               |                0xffffffff92dec2e4() {
       swapper/0-1       [000] d..1.    56.286597:  0)               |                  0xffffffff92db5304() {

It now shows:

 # trace-cmd show -B boot_mapped | tail -n30
       swapper/0-1       [000] d..2.   363.079099:  0)   0.483 us    |                    preempt_count_sub();
       swapper/0-1       [000] d....   363.079100:  0)   4.112 us    |                  }
       swapper/0-1       [000] d....   363.079101:  0)   4.979 us    |                }
       swapper/0-1       [000] d..1.   363.079101:  0)               |                disable_local_APIC() {
       swapper/0-1       [000] d..1.   363.079102:  0) + 29.153 us   |                  clear_local_APIC.part.0();
       swapper/0-1       [000] d....   363.079148:  0) + 46.517 us   |                }
       swapper/0-1       [000] d..1.   363.079149:  0)               |                mcheck_cpu_clear() {
       swapper/0-1       [000] d..1.   363.079149:  0)               |                  mce_intel_feature_clear() {
       swapper/0-1       [000] d..1.   363.079150:  0)   5.871 us    |                    lmce_supported();
       swapper/0-1       [000] d....   363.079161:  0) + 11.340 us   |                  }
       swapper/0-1       [000] d....   363.079161:  0) + 12.638 us   |                }
       swapper/0-1       [000] d....   363.079162:  0) ! 383.518 us  |              }
       swapper/0-1       [000] d..1.   363.079162:  0)               |              lapic_shutdown() {
       swapper/0-1       [000] d..1.   363.079163:  0)               |                disable_local_APIC() {
       swapper/0-1       [000] d..1.   363.079163:  0) + 26.144 us   |                  clear_local_APIC.part.0();
       swapper/0-1       [000] d....   363.079192:  0) + 29.424 us   |                }
       swapper/0-1       [000] d....   363.079192:  0) + 30.376 us   |              }
       swapper/0-1       [000] d..1.   363.079193:  0)               |              restore_boot_irq_mode() {
       swapper/0-1       [000] d..1.   363.079194:  0)               |                native_restore_boot_irq_mode() {
       swapper/0-1       [000] d..1.   363.079194:  0) + 13.863 us   |                  disconnect_bsp_APIC();
       swapper/0-1       [000] d....   363.079209:  0) + 14.933 us   |                }
       swapper/0-1       [000] d....   363.079209:  0) + 16.009 us   |              }
       swapper/0-1       [000] d..1.   363.079210:  0)   0.694 us    |              hpet_disable();
       swapper/0-1       [000] d..1.   363.079211:  0)   0.511 us    |              iommu_shutdown_noop();
       swapper/0-1       [000] d....   363.079212:  0) # 3980.260 us |            }
       swapper/0-1       [000] d..1.   363.079212:  0)               |            native_machine_emergency_restart() {
       swapper/0-1       [000] d..1.   363.079213:  0)   0.495 us    |              tboot_shutdown();
       swapper/0-1       [000] d..1.   363.079230:  0)               |              acpi_reboot() {
       swapper/0-1       [000] d..1.   363.079231:  0)               |                acpi_reset() {
       swapper/0-1       [000] d..1.   363.079232:  0)               |                  acpi_os_write_port() {

Cc: Masami Hiramatsu 
Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Linus Torvalds 
Cc: Ross Zwisler 
Cc: Vincent Donnefort 
Link: https://lore.kernel.org/20240813171257.478901820@goodmis.org
Signed-off-by: Steven Rostedt (Google)

function_graph: Move graph notrace bit to shadow stack global var

2024-06-04T14:37:44+00:00

The use of the task->trace_recursion for the logic used for the function
graph no-trace was a bit of an abuse of that variable. Now that there
exists global vars that are per stack for registered graph traces, use
that instead.

Link: https://lore.kernel.org/linux-trace-kernel/171509107907.162236.6564679266777519065.stgit@devnote2
Link: https://lore.kernel.org/linux-trace-kernel/20240603190823.796709456@goodmis.org

Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Alexei Starovoitov 
Cc: Florent Revest 
Cc: Martin KaFai Lau 
Cc: bpf 
Cc: Sven Schnelle 
Cc: Alexei Starovoitov 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Daniel Borkmann 
Cc: Alan Maguire 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Guo Ren 
Reviewed-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (VMware) 
Signed-off-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (Google)

function_graph: Move set_graph_function tests to shadow stack global var

2024-06-04T14:37:35+00:00

The use of the task->trace_recursion for the logic used for the
set_graph_function was a bit of an abuse of that variable. Now that there
exists global vars that are per stack for registered graph traces, use that
instead.

Link: https://lore.kernel.org/linux-trace-kernel/171509105520.162236.10339831553995971290.stgit@devnote2
Link: https://lore.kernel.org/linux-trace-kernel/20240603190823.472955399@goodmis.org

Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Alexei Starovoitov 
Cc: Florent Revest 
Cc: Martin KaFai Lau 
Cc: bpf 
Cc: Sven Schnelle 
Cc: Alexei Starovoitov 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Daniel Borkmann 
Cc: Alan Maguire 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Guo Ren 
Reviewed-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (VMware) 
Signed-off-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (Google)

function_graph: Have the instances use their own ftrace_ops for filtering

2024-06-04T14:36:52+00:00

Allow for instances to have their own ftrace_ops part of the fgraph_ops
that makes the funtion_graph tracer filter on the set_ftrace_filter file
of the instance and not the top instance.

This uses the new ftrace_startup_subops(), by using graph_ops as the
"manager ops" that defines the callback function and adds the functions
defined by the filters of the ops for each trace instance. The callback
defined by the manager ops will call the registered fgraph ops that were
added to the fgraph_array.

Co-developed with Masami Hiramatsu:
Link: https://lore.kernel.org/linux-trace-kernel/171509102088.162236.15758883237657317789.stgit@devnote2
Link: https://lore.kernel.org/linux-trace-kernel/20240603190822.832946261@goodmis.org

Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Alexei Starovoitov 
Cc: Florent Revest 
Cc: Martin KaFai Lau 
Cc: bpf 
Cc: Sven Schnelle 
Cc: Alexei Starovoitov 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Daniel Borkmann 
Cc: Alan Maguire 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Guo Ren 
Reviewed-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (VMware) 
Signed-off-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (Google)

ftrace: Allow function_graph tracer to be enabled in instances

2024-06-04T14:36:28+00:00

Now that function graph tracing can handle more than one user, allow it to
be enabled in the ftrace instances. Note, the filtering of the functions is
still joined by the top level set_ftrace_filter and friends, as well as the
graph and nograph files.

Co-developed with Masami Hiramatsu:
Link: https://lore.kernel.org/linux-trace-kernel/171509099743.162236.1699959255446248163.stgit@devnote2
Link: https://lore.kernel.org/linux-trace-kernel/20240603190822.190630762@goodmis.org

Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Alexei Starovoitov 
Cc: Florent Revest 
Cc: Martin KaFai Lau 
Cc: bpf 
Cc: Sven Schnelle 
Cc: Alexei Starovoitov 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Daniel Borkmann 
Cc: Alan Maguire 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Guo Ren 
Reviewed-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (VMware) 
Signed-off-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (Google)

ftrace/function_graph: Pass fgraph_ops to function graph callbacks

2024-06-04T14:36:22+00:00

Pass the fgraph_ops structure to the function graph callbacks. This will
allow callbacks to add a descriptor to a fgraph_ops private field that wil
be added in the future and use it for the callbacks. This will be useful
when more than one callback can be registered to the function graph tracer.

Co-developed with Masami Hiramatsu:
Link: https://lore.kernel.org/linux-trace-kernel/171509098588.162236.4787930115997357578.stgit@devnote2
Link: https://lore.kernel.org/linux-trace-kernel/20240603190822.035147698@goodmis.org

Cc: Mark Rutland 
Cc: Mathieu Desnoyers 
Cc: Andrew Morton 
Cc: Alexei Starovoitov 
Cc: Florent Revest 
Cc: Martin KaFai Lau 
Cc: bpf 
Cc: Sven Schnelle 
Cc: Alexei Starovoitov 
Cc: Jiri Olsa 
Cc: Arnaldo Carvalho de Melo 
Cc: Daniel Borkmann 
Cc: Alan Maguire 
Cc: Peter Zijlstra 
Cc: Thomas Gleixner 
Cc: Guo Ren 
Reviewed-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (VMware) 
Signed-off-by: Masami Hiramatsu (Google) 
Signed-off-by: Steven Rostedt (Google)

function_graph: Support recording and printing the return value of function

2023-06-20T22:38:37+00:00

Analyzing system call failures with the function_graph tracer can be a
time-consuming process, particularly when locating the kernel function
that first returns an error in the trace logs. This change aims to
simplify the process by recording the function return value to the
'retval' member of 'ftrace_graph_ret' and printing it when outputting
the trace log.

We have introduced new trace options: funcgraph-retval and
funcgraph-retval-hex. The former controls whether to display the return
value, while the latter controls the display format.

Please note that even if a function's return type is void, a return
value will still be printed. You can simply ignore it.

This patch only establishes the fundamental infrastructure. Subsequent
patches will make this feature available on some commonly used processor
architectures.

Here is an example:

I attempted to attach the demo process to a cpu cgroup, but it failed:

echo `pidof demo` > /sys/fs/cgroup/cpu/test/tasks
-bash: echo: write error: Invalid argument

The strace logs indicate that the write system call returned -EINVAL(-22):
...
write(1, "273\n", 4)                    = -1 EINVAL (Invalid argument)
...

To capture trace logs during a write system call, use the following
commands:

cd /sys/kernel/debug/tracing/
echo 0 > tracing_on
echo > trace
echo *sys_write > set_graph_function
echo *spin* > set_graph_notrace
echo *rcu* >> set_graph_notrace
echo *alloc* >> set_graph_notrace
echo preempt* >> set_graph_notrace
echo kfree* >> set_graph_notrace
echo $$ > set_ftrace_pid
echo function_graph > current_tracer
echo 1 > options/funcgraph-retval
echo 0 > options/funcgraph-retval-hex
echo 1 > tracing_on
echo `pidof demo` > /sys/fs/cgroup/cpu/test/tasks
echo 0 > tracing_on
cat trace > ~/trace.log

To locate the root cause, search for error code -22 directly in the file
trace.log and identify the first function that returned -22. Once you
have identified this function, examine its code to determine the root
cause.

For example, in the trace log below, cpu_cgroup_can_attach
returned -22 first, so we can focus our analysis on this function to
identify the root cause.

...

 1)          | cgroup_migrate() {
 1) 0.651 us |   cgroup_migrate_add_task(); /* = 0xffff93fcfd346c00 */
 1)          |   cgroup_migrate_execute() {
 1)          |     cpu_cgroup_can_attach() {
 1)          |       cgroup_taskset_first() {
 1) 0.732 us |         cgroup_taskset_next(); /* = 0xffff93fc8fb20000 */
 1) 1.232 us |       } /* cgroup_taskset_first = 0xffff93fc8fb20000 */
 1) 0.380 us |       sched_rt_can_attach(); /* = 0x0 */
 1) 2.335 us |     } /* cpu_cgroup_can_attach = -22 */
 1) 4.369 us |   } /* cgroup_migrate_execute = -22 */
 1) 7.143 us | } /* cgroup_migrate = -22 */

...

Link: https://lkml.kernel.org/r/1fc502712c981e0e6742185ba242992170ac9da8.1680954589.git.pengdonglin@sangfor.com.cn

Tested-by: Florian Kauer 
Acked-by: Masami Hiramatsu (Google) 
Signed-off-by: Donglin Peng 
Signed-off-by: Steven Rostedt (Google)

tracing: in_irq() cleanup

2021-10-13T22:19:41+00:00

Replace the obsolete and ambiguos macro in_irq() with new
macro in_hardirq().

Link: https://lkml.kernel.org/r/20210930000342.6016-1-changbin.du@gmail.com

Reviewed-by: Petr Mladek 
Signed-off-by: Changbin Du 
Signed-off-by: Steven Rostedt (VMware)

tracing: Disable "other" permission bits in the tracefs files

2021-10-08T22:08:43+00:00

When building the files in the tracefs file system, do not by default set
any permissions for OTH (other). This will make it easier for admins who
want to define a group for accessing tracefs and not having to first
disable all the permission bits for "other" in the file system.

As tracing can leak sensitive information, it should never by default
allowing all users access. An admin can still set the permission bits for
others to have access, which may be useful for creating a honeypot and
seeing who takes advantage of it and roots the machine.

Link: https://lkml.kernel.org/r/20210818153038.864149276@goodmis.org

Signed-off-by: Steven Rostedt (VMware)

tracing: Fix various typos in comments

2021-03-23T18:08:18+00:00

Fix ~59 single-word typos in the tracing code comments, and fix
the grammar in a handful of places.

Link: https://lore.kernel.org/r/20210322224546.GA1981273@gmail.com
Link: https://lkml.kernel.org/r/20210323174935.GA4176821@gmail.com

Reviewed-by: Randy Dunlap 
Signed-off-by: Ingo Molnar 
Signed-off-by: Steven Rostedt (VMware)