1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
|
Documentation for kdump - the kexec-based crash dumping solution
================================================================
DESIGN
======
Kdump uses kexec to reboot to a second kernel whenever a dump needs to be taken.
This second kernel is booted with very little memory. The first kernel reserves
the section of memory that the second kernel uses. This ensures that on-going
DMA from the first kernel does not corrupt the second kernel.
All the necessary information about Core image is encoded in ELF format and
stored in reserved area of memory before crash. Physical address of start of
ELF header is passed to new kernel through command line parameter elfcorehdr=.
On i386, the first 640 KB of physical memory is needed to boot, irrespective
of where the kernel loads. Hence, this region is backed up by kexec just before
rebooting into the new kernel.
In the second kernel, "old memory" can be accessed in two ways.
- The first one is through a /dev/oldmem device interface. A capture utility
can read the device file and write out the memory in raw format. This is raw
dump of memory and analysis/capture tool should be intelligent enough to
determine where to look for the right information. ELF headers (elfcorehdr=)
can become handy here.
- The second interface is through /proc/vmcore. This exports the dump as an ELF
format file which can be written out using any file copy command
(cp, scp, etc). Further, gdb can be used to perform limited debugging on
the dump file. This method ensures methods ensure that there is correct
ordering of the dump pages (corresponding to the first 640 KB that has been
relocated).
SETUP
=====
1) Download http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz
and apply http://lse.sourceforge.net/kdump/patches/kexec-tools-1.101-kdump.patch
and after that build the source.
2) Download and build the appropriate (2.6.13-rc1 onwards) vanilla kernel.
Two kernels need to be built in order to get this feature working.
A) First kernel:
a) Enable "kexec system call" feature (in Processor type and features).
CONFIG_KEXEC=y
b) This kernel's physical load address should be the default value of
0x100000 (0x100000, 1 MB) (in Processor type and features).
CONFIG_PHYSICAL_START=0x100000
c) Enable "sysfs file system support" (in Pseudo filesystems).
CONFIG_SYSFS=y
d) Boot into first kernel with the command line parameter "crashkernel=Y@X".
Use appropriate values for X and Y. Y denotes how much memory to reserve
for the second kernel, and X denotes at what physical address the reserved
memory section starts. For example: "crashkernel=64M@16M".
B) Second kernel:
a) Enable "kernel crash dumps" feature (in Processor type and features).
CONFIG_CRASH_DUMP=y
b) Specify a suitable value for "Physical address where the kernel is
loaded" (in Processor type and features). Typically this value
should be same as X (See option d) above, e.g., 16 MB or 0x1000000.
CONFIG_PHYSICAL_START=0x1000000
c) Enable "/proc/vmcore support" (Optional, in Pseudo filesystems).
CONFIG_PROC_VMCORE=y
d) Disable SMP support and build a UP kernel (Until it is fixed).
CONFIG_SMP=n
e) Enable "Local APIC support on uniprocessors".
CONFIG_X86_UP_APIC=y
f) Enable "IO-APIC support on uniprocessors"
CONFIG_X86_UP_IOAPIC=y
Note: i) Options a) and b) depend upon "Configure standard kernel features
(for small systems)" (under General setup).
ii) Option a) also depends on CONFIG_HIGHMEM (under Processor
type and features).
iii) Both option a) and b) are under "Processor type and features".
3) Boot into the first kernel. You are now ready to try out kexec-based crash
dumps.
4) Load the second kernel to be booted using:
kexec -p <second-kernel> --args-linux --elf32-core-headers
--append="root=<root-dev> init 1 irqpoll"
Note: i) <second-kernel> has to be a vmlinux image. bzImage will not work,
as of now.
ii) By default ELF headers are stored in ELF64 format. Option
--elf32-core-headers forces generation of ELF32 headers. gdb can
not open ELF64 headers on 32 bit systems. So creating ELF32
headers can come handy for users who have got non-PAE systems and
hence have memory less than 4GB.
iii) Specify "irqpoll" as command line parameter. This reduces driver
initialization failures in second kernel due to shared interrupts.
iv) <root-dev> needs to be specified in a format corresponding to
the root device name in the output of mount command.
v) If you have built the drivers required to mount root file
system as modules in <second-kernel>, then, specify
--initrd=<initrd-for-second-kernel>.
5) System reboots into the second kernel when a panic occurs. A module can be
written to force the panic or "ALT-SysRq-c" can be used initiate a crash
dump for testing purposes.
6) Write out the dump file using
cp /proc/vmcore <dump-file>
Dump memory can also be accessed as a /dev/oldmem device for a linear/raw
view. To create the device, type:
mknod /dev/oldmem c 1 12
Use "dd" with suitable options for count, bs and skip to access specific
portions of the dump.
Entire memory: dd if=/dev/oldmem of=oldmem.001
ANALYSIS
========
Limited analysis can be done using gdb on the dump file copied out of
/proc/vmcore. Use vmlinux built with -g and run
gdb vmlinux <dump-file>
Stack trace for the task on processor 0, register display, memory display
work fine.
Note: gdb cannot analyse core files generated in ELF64 format for i386.
TODO
====
1) Provide a kernel pages filtering mechanism so that core file size is not
insane on systems having huge memory banks.
2) Modify "crash" tool to make it recognize this dump.
CONTACT
=======
Vivek Goyal (vgoyal@in.ibm.com)
Maneesh Soni (maneesh@in.ibm.com)
|
