linux-toradex.git/arch/x86/kernel/cpu/mcheck/mce_intel.c, branch v3.10.76

x86/mce: Rework cmci_rediscover() to play well with CPU hotplug

2013-04-02T21:04:01+00:00

Dave Jones reports that offlining a CPU leads to this trace:

numa_remove_cpu cpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offline
BUG: using smp_processor_id() in preemptible [00000000] code:
cpu-offline.sh/10591
caller is cmci_rediscover+0x6a/0xe0
Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2
Call Trace:
 [] debug_smp_processor_id+0xdd/0x100
 [] cmci_rediscover+0x6a/0xe0
 [] mce_cpu_callback+0x19d/0x1ae
 [] notifier_call_chain+0x66/0x150
 [] __raw_notifier_call_chain+0xe/0x10
 [] cpu_notify+0x23/0x50
 [] cpu_notify_nofail+0xe/0x20
 [] _cpu_down+0x302/0x350
 [] cpu_down+0x36/0x50
 [] store_online+0x8d/0xd0
 [] dev_attr_store+0x18/0x30
 [] sysfs_write_file+0xdb/0x150
 [] vfs_write+0xa2/0x170
 [] sys_write+0x4c/0xa0
 [] system_call_fastpath+0x16/0x1b

However, a look at cmci_rediscover shows that it can be simplified quite
a bit, apart from solving the above issue. It invokes functions that
take spin locks with interrupts disabled, and hence it can run in atomic
context. Also, it is run in the CPU_POST_DEAD phase, so the dying CPU
is already dead and out of the cpu_online_mask. So take these points into
account and simplify the code, and thereby also fix the above issue.

Reported-by: Dave Jones 
Signed-off-by: Srivatsa S. Bhat 
Signed-off-by: Tony Luck

Merge branch 'x86-ras-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2012-12-14T17:59:59+00:00

Pull x86 RAS update from Ingo Molnar:
 "Rework all config variables used throughout the MCA code and collect
  them together into a mca_config struct.  This keeps them tightly and
  neatly packed together instead of spilled all over the place.

  Then, convert those which are used as booleans into real booleans and
  save some space.  These bits are exposed via
     /sys/devices/system/machinecheck/machinecheck*/"

* 'x86-ras-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86, MCA: Finish mca_config conversion
  x86, MCA: Convert the next three variables batch
  x86, MCA: Convert rip_msr, mce_bootlog, monarch_timeout
  x86, MCA: Convert dont_log_ce, banks and tolerant
  drivers/base: Add a DEVICE_BOOL_ATTR macro

x86/mce: Do not change worker's running cpu in cmci_rediscover().

2012-10-30T21:38:12+00:00

cmci_rediscover() used set_cpus_allowed_ptr() to change the current process's
running cpu, and migrate itself to the dest cpu. But worker processes are not
allowed to be migrated. If current is a worker, the worker will be migrated to
another cpu, but the corresponding  worker_pool is still on the original cpu.

In this case, the following BUG_ON in try_to_wake_up_local() will be triggered:
BUG_ON(rq != this_rq());

This will cause the kernel panic. The call trace is like the following:

[ 6155.451107] ------------[ cut here ]------------
[ 6155.452019] kernel BUG at kernel/sched/core.c:1654!
......
[ 6155.452019] RIP: 0010:[]  [] try_to_wake_up_local+0x115/0x130
......
[ 6155.452019] Call Trace:
[ 6155.452019]  [] __schedule+0x764/0x880
[ 6155.452019]  [] schedule+0x29/0x70
[ 6155.452019]  [] schedule_timeout+0x235/0x2d0
[ 6155.452019]  [] ? mark_held_locks+0x8d/0x140
[ 6155.452019]  [] ? __lock_release+0x133/0x1a0
[ 6155.452019]  [] ? _raw_spin_unlock_irq+0x30/0x50
[ 6155.452019]  [] ? trace_hardirqs_on_caller+0x105/0x190
[ 6155.452019]  [] wait_for_common+0x12b/0x180
[ 6155.452019]  [] ? try_to_wake_up+0x2f0/0x2f0
[ 6155.452019]  [] wait_for_completion+0x1d/0x20
[ 6155.452019]  [] stop_one_cpu+0x8a/0xc0
[ 6155.452019]  [] ? __migrate_task+0x1a0/0x1a0
[ 6155.452019]  [] ? complete+0x28/0x60
[ 6155.452019]  [] set_cpus_allowed_ptr+0x128/0x130
[ 6155.452019]  [] cmci_rediscover+0xf5/0x140
[ 6155.452019]  [] mce_cpu_callback+0x18d/0x19d
[ 6155.452019]  [] notifier_call_chain+0x67/0x150
[ 6155.452019]  [] __raw_notifier_call_chain+0xe/0x10
[ 6155.452019]  [] __cpu_notify+0x20/0x40
[ 6155.452019]  [] cpu_notify_nofail+0x15/0x30
[ 6155.452019]  [] _cpu_down+0x262/0x2e0
[ 6155.452019]  [] cpu_down+0x36/0x50
[ 6155.452019]  [] acpi_processor_remove+0x50/0x11e
[ 6155.452019]  [] acpi_device_remove+0x90/0xb2
[ 6155.452019]  [] __device_release_driver+0x7c/0xf0
[ 6155.452019]  [] device_release_driver+0x2f/0x50
[ 6155.452019]  [] acpi_bus_remove+0x32/0x6d
[ 6155.452019]  [] acpi_bus_trim+0x87/0xee
[ 6155.452019]  [] acpi_bus_hot_remove_device+0x88/0x16b
[ 6155.452019]  [] acpi_os_execute_deferred+0x27/0x34
[ 6155.452019]  [] process_one_work+0x219/0x680
[ 6155.452019]  [] ? process_one_work+0x1b8/0x680
[ 6155.452019]  [] ? acpi_os_wait_events_complete+0x23/0x23
[ 6155.452019]  [] worker_thread+0x12e/0x320
[ 6155.452019]  [] ? manage_workers+0x110/0x110
[ 6155.452019]  [] kthread+0xc6/0xd0
[ 6155.452019]  [] kernel_thread_helper+0x4/0x10
[ 6155.452019]  [] ? retint_restore_args+0x13/0x13
[ 6155.452019]  [] ? __init_kthread_worker+0x70/0x70
[ 6155.452019]  [] ? gs_change+0x13/0x13

This patch removes the set_cpus_allowed_ptr() call, and put the cmci rediscover
jobs onto all the other cpus using system_wq. This could bring some delay for
the jobs.

Signed-off-by: Tang Chen 
Signed-off-by: Miao Xie 
Signed-off-by: Tony Luck

x86, MCA: Finish mca_config conversion

2012-10-26T12:37:58+00:00

mce_ser, mce_bios_cmci_threshold and mce_disabled are the last three
bools which need conversion. Move them to the mca_config struct and
adjust usage sites accordingly.

Signed-off-by: Borislav Petkov 
Acked-by: Tony Luck

x86, MCA: Convert the next three variables batch

2012-10-26T12:37:57+00:00

Move them into the mca_config struct and adjust code touching them
accordingly.

Signed-off-by: Borislav Petkov 
Acked-by: Tony Luck

x86/mce: Provide boot argument to honour bios-set CMCI threshold

2012-09-27T17:08:00+00:00

The ACPI spec doesn't provide for a way for the bios to pass down
recommended thresholds to the OS on a _per-bank_ basis. This patch adds
a new boot option, which if passed, tells Linux to use CMCI thresholds
set by the bios.

As fail-safe, we initialize threshold to 1 if some banks have not been
initialized by the bios and warn the user.

Signed-off-by: Naveen N. Rao 
Signed-off-by: Tony Luck

x86/mce: Add CMCI poll mode

2012-08-09T18:44:51+00:00

On Intel systems corrected machine check interrupts (CMCI) may be sent to
multiple logical processors; possibly to all processors on the affected
socket (SDM Volume 3B "15.5.1 CMCI Local APIC Interface").  This means
that a persistent error (such as a stuck bit in ECC memory) may cause
a storm of interrupts that greatly hinders or prevents forward progress
(probably on many processors).

To solve this we keep track of the rate at which each processor sees
CMCI. If we exceed a threshold, we disable CMCI delivery and switch to
polling the machine check banks. If the storm subsides (none of the
affected processors see any more errors for a complete poll interval) we
re-enable CMCI.

[Tony: Added console messages when storm begins/ends and increased storm
threshold from 5 to 15 so we have a few more logged entries before we
disable interrupts and start dropping reports]

Signed-off-by: Chen Gong 
Signed-off-by: Thomas Gleixner 
Tested-by: Chen Gong 
Signed-off-by: Tony Luck

x86/mce: Make cmci_discover() quiet

2012-08-09T17:59:21+00:00

cmci_discover() works out which machine check banks support CMCI, and
which of those are shared by multiple logical processors. It uses this
information to ensure that exactly one cpu is designated the owner of
each bank so that when interrupts are broadcast to multiple cpus, only one
of them will look in a shared bank to log the error and clear the bank.

At boot time cmci_discover() performs this task silently. But during
certain cpu hotplug operations it prints out a set of summary lines
like this:

CPU 35 MCA banks CMCI:0 CMCI:1 CMCI:3 CMCI:5 CMCI:6 CMCI:7 CMCI:8 CMCI:9 CMCI:10 CMCI:11
CPU 1 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 39 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 38 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 32 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 37 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 36 MCA banks CMCI:0 CMCI:1 CMCI:3
CPU 34 MCA banks CMCI:0 CMCI:1 CMCI:3

The value of these messages seems very low. A user might painstakingly
cross-check against the data sheet for a processor to ensure that all
CMCI supported banks are correctly reported, but this seems improbable.
If users really wanted to do this, we should print the information at
boot time too.

Remove the messages.

Signed-off-by: Tony Luck

locking, x86: mce: Annotate cmci_discover_lock as raw

2011-09-13T09:12:09+00:00

The cmci_discover_lock can be taken in atomic context (cpu bring
up sequence) and therefore cannot be preempted on -rt.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar

x86: Replace uses of current_cpu_data with this_cpu ops

2010-12-30T11:22:03+00:00

Replace all uses of current_cpu_data with this_cpu operations on the
per cpu structure cpu_info.  The scala accesses are replaced with the
matching this_cpu ops which results in smaller and more efficient
code.

In the long run, it might be a good idea to remove cpu_data() macro
too and use per_cpu macro directly.

tj: updated description

Cc: Yinghai Lu 
Cc: Ingo Molnar 
Acked-by: H. Peter Anvin 
Acked-by: Tejun Heo 
Signed-off-by: Christoph Lameter 
Signed-off-by: Tejun Heo