linux-toradex.git/arch/powerpc/kernel/watchdog.c, branch v6.7

nmi_backtrace: allow excluding an arbitrary CPU

2023-08-18T17:19:00+00:00

The APIs that allow backtracing across CPUs have always had a way to
exclude the current CPU.  This convenience means callers didn't need to
find a place to allocate a CPU mask just to handle the common case.

Let's extend the API to take a CPU ID to exclude instead of just a
boolean.  This isn't any more complex for the API to handle and allows the
hardlockup detector to exclude a different CPU (the one it already did a
trace for) without needing to find space for a CPU mask.

Arguably, this new API also encourages safer behavior.  Specifically if
the caller wants to avoid tracing the current CPU (maybe because they
already traced the current CPU) this makes it more obvious to the caller
that they need to make sure that the current CPU ID can't change.

[akpm@linux-foundation.org: fix trigger_allbutcpu_cpu_backtrace() stub]
Link: https://lkml.kernel.org/r/20230804065935.v4.1.Ia35521b91fc781368945161d7b28538f9996c182@changeid
Signed-off-by: Douglas Anderson 
Acked-by: Michal Hocko 
Cc: kernel test robot 
Cc: Lecopzer Chen 
Cc: Petr Mladek 
Cc: Pingfan Liu 
Signed-off-by: Andrew Morton

watchdog/hardlockup: rename some "NMI watchdog" constants/function

2023-06-10T00:44:20+00:00

Do a search and replace of:
- NMI_WATCHDOG_ENABLED => WATCHDOG_HARDLOCKUP_ENABLED
- SOFT_WATCHDOG_ENABLED => WATCHDOG_SOFTOCKUP_ENABLED
- watchdog_nmi_ => watchdog_hardlockup_
- nmi_watchdog_available => watchdog_hardlockup_available
- nmi_watchdog_user_enabled => watchdog_hardlockup_user_enabled
- soft_watchdog_user_enabled => watchdog_softlockup_user_enabled
- NMI_WATCHDOG_DEFAULT => WATCHDOG_HARDLOCKUP_DEFAULT

Then update a few comments near where names were changed.

This is specifically to make it less confusing when we want to introduce
the buddy hardlockup detector, which isn't using NMIs.  As part of this,
we sanitized a few names for consistency.

[trix@redhat.com: make variables static]
  Link: https://lkml.kernel.org/r/20230525162822.1.I0fb41d138d158c9230573eaa37dc56afa2fb14ee@changeid
Link: https://lkml.kernel.org/r/20230519101840.v5.12.I91f7277bab4bf8c0cb238732ed92e7ce7bbd71a6@changeid
Signed-off-by: Douglas Anderson 
Signed-off-by: Tom Rix 
Reviewed-by: Tom Rix 
Cc: Andi Kleen 
Cc: Catalin Marinas 
Cc: Chen-Yu Tsai 
Cc: Christophe Leroy 
Cc: Colin Cross 
Cc: Daniel Thompson 
Cc: "David S. Miller" 
Cc: Guenter Roeck 
Cc: Ian Rogers 
Cc: Lecopzer Chen 
Cc: Marc Zyngier 
Cc: Mark Rutland 
Cc: Masayoshi Mizuma 
Cc: Matthias Kaehlcke 
Cc: Michael Ellerman 
Cc: Nicholas Piggin 
Cc: Petr Mladek 
Cc: Pingfan Liu 
Cc: Randy Dunlap 
Cc: "Ravi V. Shankar" 
Cc: Ricardo Neri 
Cc: Stephane Eranian 
Cc: Stephen Boyd 
Cc: Sumit Garg 
Cc: Tzung-Bi Shih 
Cc: Will Deacon 
Signed-off-by: Andrew Morton

powerpc/watchdog: introduce a NMI watchdog's factor

2022-07-27T11:36:02+00:00

Introduce a factor which would apply to the NMI watchdog timeout.

This factor is a percentage added to the watchdog_tresh value. The value is
set under the watchdog_mutex protection and lockup_detector_reconfigure()
is called to recompute wd_panic_timeout_tb.

Once the factor is set, it remains until it is set back to 0, which means
no impact.

Signed-off-by: Laurent Dufour 
Reviewed-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20220713154729.80789-4-ldufour@linux.ibm.com

powerpc: Fix all occurences of duplicate words

2022-07-25T02:05:15+00:00

Since commit 87c78b612f4f ("powerpc: Fix all occurences of "the the"")
fixed "the the", there's now a steady stream of patches fixing other
duplicate words.

Just fix them all at once, to save the overhead of dealing with
individual patches for each case.

This leaves a few cases of "that that", which in some contexts is
correct.

Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20220718095158.326606-1-mpe@ellerman.id.au

powerpc: fix typos in comments

2022-05-05T12:12:44+00:00

Various spelling mistakes in comments.
Detected with the help of Coccinelle.

Signed-off-by: Julia Lawall 
Reviewed-by: Joel Stanley 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20220430185654.5855-1-Julia.Lawall@inria.fr

powerpc/watchdog: help remote CPUs to flush NMI printk output

2021-11-29T12:08:43+00:00

The printk layer at the moment does not seem to have a good way to force
flush printk messages that are created in NMI context, except in the
panic path.

NMI-context printk messages normally get to the console with irq_work,
but that won't help if the CPU is stuck with irqs disabled, as can be
the case for hard lockup watchdog messages.

The watchdog currently flushes the printk buffers after detecting a
lockup on remote CPUs, but they may not have processed their NMI IPI
yet by that stage, or they may have self-detected a lockup in which
case they won't go via this NMI IPI path.

Improve the situation by having NMI-context mark a flag if it called
printk, and have watchdog timer interrupts check if that flag was set
and try to flush if it was. Latency is not a big problem because we
were already stuck for a while, just need to try to make sure the
messages eventually make it out.

Depends-on: 5d5e4522a7f4 ("printk: restore flushing of NMI buffers on remote CPUs after NMI backtraces")
Signed-off-by: Nicholas Piggin 
Reviewed-by: Laurent Dufour 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211119113146.752759-6-npiggin@gmail.com

powerpc/watchdog: Fix wd_smp_last_reset_tb reporting

2021-11-25T10:48:39+00:00

wd_smp_last_reset_tb now gets reset by watchdog_smp_panic() as part of
marking CPUs stuck and removing them from the pending mask before it
begins any printing. This causes last reset times reported to be off.

Fix this by reading it into a local variable before it gets reset.

Fixes: 76521c4b0291 ("powerpc/watchdog: Avoid holding wd_smp_lock over printk and smp_send_nmi_ipi")
Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211125103346.1188958-1-npiggin@gmail.com

powerpc/watchdog: read TB close to where it is used

2021-11-25T00:25:34+00:00

When taking watchdog actions, printing messages, comparing and
re-setting wd_smp_last_reset_tb, etc., read TB close to the point of use
and under wd_smp_lock or printing lock (if applicable).

This should keep timebase mostly monotonic with kernel log messages, and
could prevent (in theory) a laggy CPU updating wd_smp_last_reset_tb to
something a long way in the past, and causing other CPUs to appear to be
stuck.

These additional TB reads are all slowpath (lockup has been detected),
so performance does not matter.

Signed-off-by: Nicholas Piggin 
Reviewed-by: Laurent Dufour 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211110025056.2084347-5-npiggin@gmail.com

powerpc/watchdog: Avoid holding wd_smp_lock over printk and smp_send_nmi_ipi

2021-11-25T00:25:33+00:00

There is a deadlock with the console_owner lock and the wd_smp_lock:

CPU x takes the console_owner lock
CPU y takes a watchdog timer interrupt and takes __wd_smp_lock
CPU x takes a soft-NMI interrupt, detects deadlock, spins on __wd_smp_lock
CPU y detects deadlock, tries to print something and spins on console_owner
-> deadlock

Change the watchdog locking scheme so wd_smp_lock protects the watchdog
internal data, but "reporting" (printing, issuing NMI IPIs, taking any
action outside of watchdog) uses a non-waiting exclusion. If a CPU detects
a problem but can not take the reporting lock, it just returns because
something else is already reporting. It will try again at some point.

Typically hard lockup watchdog report usefulness is not impacted due to
failure to spewing a large enough amount of data in as short a time as
possible, but by messages getting garbled.

Laurent debugged this and found the deadlock, and this patch is based on
his general approach to avoid expensive operations while holding the lock.
With the addition of the reporting exclusion.

Signed-off-by: Laurent Dufour 
[np: rework to add reporting exclusion update changelog]
Signed-off-by: Nicholas Piggin 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211110025056.2084347-4-npiggin@gmail.com

powerpc/watchdog: tighten non-atomic read-modify-write access

2021-11-25T00:25:33+00:00

Most updates to wd_smp_cpus_pending are under lock except the watchdog
interrupt bit clear.

This can race with non-atomic RMW updates to the mask under lock, which
can happen in two instances:

Firstly, if another CPU detects this one is stuck, removes it from the
mask, mask becomes empty and is re-filled with non-atomic stores. This
is okay because it would re-fill the mask with this CPU's bit clear
anyway (because this CPU is now stuck), so it doesn't matter that the
bit clear update got "lost". Add a comment for this.

Secondly, if another CPU detects a different CPU is stuck and removes it
from the pending mask with a non-atomic store to bytes which also
include the bit of this CPU. This case can result in the bit clear being
lost and the end result being the bit is set. This should be so rare it
hardly matters, but to make things simpler to reason about just avoid
the non-atomic access for that case.

Signed-off-by: Nicholas Piggin 
Reviewed-by: Laurent Dufour 
Signed-off-by: Michael Ellerman 
Link: https://lore.kernel.org/r/20211110025056.2084347-3-npiggin@gmail.com