How to use crash utility to perform some pre-analysis on vmcore – in RHEL5/RHEL6
Whenever a system with a valid kdump configuration crashes, it will to create a vmcore core file either locally or remotely based on the KDUMP location paramets in kdump.conf file. And we need to upload these vmcore files to Red Hat support staff for further analysis of actual root cause of kernel crash.
Sometime, the advanced linux administrators uses the crash utility to analyse the vmcore prior to uploading it to Red Hat Support site. In this post we will discuss about some of the crash commands that helps us to perform pre-analysis of vmcore.
The Crash Command Set
Each crash command generally falls into one of the following categories:
The remainder of this section breaks the command set into categories, and gives a short description of each command in that category. However, for complete details and examples, recall that the crash utility has a self-contained help page for each command; to view the full help page, click on the command name next to its description below.
crash version: 4.0-8.11 gdb version: 6.1 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash>
Symbolic Display of Kernel Text or Data
The following commands typically take full advantage of the power of gdb to display kernel data structures symbolically.
|struct||Displays a formatted kernel data structure type located at a given address, or at an address referred to by a symbol; if no address is specified, the structure definition is displayed. The output can be narrowed down to a singular member of the structure, or to display the offset of every member from the beginning of the structure. A count may be appended to display an array of structures. Its usage is so common that two short-cuts exist such that the user need not enter the “struct” command name:
|union||Same as struct command, but used for kernel data types defined as unions instead of structures..|
|*||“Pointer-to” command which can be used in lieu of entering struct or union; the gdbmodule first determines whether the argument is a structure or a union, and then calls the appropriate function.|
|p||Displays the contents of a kernel variable; the arguments are passed on to gdb‘s printcommand for proper formatting. Two builtin aliases, px and pd, set the numerical output radix to hexadecimal or decimal for the print operation, temporarily overriding the current default.|
|whatis||Displays all available symbol table information concerning a data type or a data symbol.|
|sym||Translates a kernel symbol name to its kernel virtual address and section, or a kernel virtual address to its symbol name and section. It can also be used to dump the complete list of kernel symbols, or to query the symbol list for all symbols containing a given sub-string.|
|dis||Disassembles the text of complete kernel function, or from a specified address for a given number of instructions, or from the beginning of a function up to a specified address.|
The majority of crash commands come from the following set of “kernel-aware” commands, which delve into various kernel subsystems on a system-wide or per-task basis. The task-specific commands are context-sensitive, meaning that they act upon the current context unless a PID or task address is specified as an argument.
|bt||Arguably the most useful crash command, bt displays a task’s kernel stack back-trace, including full exception frame dumps. It is context-sensitive, although the -a option will display the stack traces of the active task on each CPU. This command is often used within the foreach wrapper command in order to display the back traces of all tasks with one command.|
|dev||Displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data.|
|files||This context-sensitive command displays the task’s current root directory and working directories, and then for each open file descriptor, shows:
Another option acts upon a specified dentry address, showing:
It can be called from the foreach wrapper command.
|fuser||Displays a list of tasks that reference a specified filename or inode address as the current root or working directory, an open file descriptor, or which mmap the file.|
|irq||Display data concerning interrupt request numbers and bottom-half handling.|
|kmem||This command has numerous options that delve into the state of several kernel memory subsystems:
Also, given an address, this command searches the symbol table, the slab subsystem, the free list, the page_hash_table, the vmlist, and the mem_map array, displaying where it was found.
|log||Dumps the kernel message buffer chronologically, accounting for any wrap-around.|
|mach||Displays machine and/or processor specific data.|
|mod||Displays the list of currently-loaded kernel modules. More importantly, it loads the debug data from the module object files if they are available, allowing symbolic debugging capability of kernel modules.|
|mount||For each mounted filesystem, or for just a specified filesystem, displays:
Options exist to dump a list of a specified filesystem’s open files or dirty inodes. Filesystems may be specified by vfsmount, super_block, or inode addresses, or by device name or mount point names.
|net||Displays various network-related data:
It can be called from the foreach wrapper command.
|ps||Useful process status command, in typical Linux ps command type output, containing:
Also has an option to show a task’s parental hierarchy back to the init process, and another to show all children of a task.
|pte||This command translates the contents of a PTE into its physical page address and page bit settings, or if it references a swap location, the swap device and offset.|
|runq||Displays list of tasks on the run queue.|
|sig||A context-sensitive command which displays a task’s signal information, including:
Other options list the signal number/names combination for a processor type, and translate the contents of a sigset_t into the signal names whose bits are set. It can be called from the foreach wrapper command.
|swap||For each configured swap device, this command displays the same data that is shown by the Linux command swapon -s.|
|sys||Re-displays the same system-related data that is seen during crash initialization:
Other options display information concerning the system call table, and one allows theroot user to panic a live system.
|task||This context-sensitive command displays a task’s complete task_struct contents, or one or more members of the structure. This command is often used within the foreachwrapper command in order to display task_struct data for all tasks with one command.|
|timer||Displays the timer queue entries in chronological order, listing the target function names, the current value of jiffies, and the expiration time of each entry.|
|vm||This powerful, context-sensitive command displays a wealth of information concerning a task’s virtual memory data, including:
Other options translate the flags of a vm_area_struct, or display the full contents of a task’s mm_struct or of each vm_area_struct. It can be called from the foreach wrapper command.
|vtop||This context-sensitive command translates a user or kernel virtual address to its physical address. Also displayed are:
It can be called from the foreach wrapper command.
|waitq||Lists the tasks linked on a specified kernel wait queue.|
The following commands are a set of useful helper commands serving various purposes, some simple, others quite powerful.
|ascii||Translates a numerical value into its ASCII components; with no arguments, displays an ASCII chart.|
|btop||Translates a byte value (physical address) to its page number.|
|eval||A simple calculator, evaluates an expression and displays the result in hexadecimal, decimal, octal and binary, and optionally showing the bit numbers set in the result.|
|list||Dumps the entries of a linked list of structures. It can handle lists of structures that are singly-linked with simple “next” pointers, or those with embedded list_head structures. The output may be constrained to simply display the address of each structure in the list, or if directed, also dump each complete structure, or just one member of each structure. The gathered list entries are hashed, so a corrupted list that loops back upon itself will be recognized.|
|ptob||translates a page frame number to its byte value (physical address).|
|ptov||Translates a physical address into a kernel virtual address by adding the appropriatePAGE_OFFSET value.|
|search||Searches a range of user or kernel memory space for given value, with an optional “don’t care” bit-mask argument.|
|rd||Displays a specified amount of user virtual, kernel virtual, or physical memory in several formats, such as 8, 16, 32 or 64 bit values, hexadecimal or decimal, symbolically, and with ASCII translations. When reading user virtual addresses, the command is context-sensitive.|
|wr||Modifies the contents of memory on a live system. Write permission on /dev/mem is required; this command should obviously be used with great care. The write operation is constrained to one 8, 16, 32 or 64 bit location.|
Session Control Commands
The following commands typcally aid in the efficient running of a crash session.
|alias||Creates a single-word alias for a command string. Several aliases are built into crash; user-defined aliases may also be defined in a .crashrc file, or during a crash session by entering it on the command line or reading it from an input file.|
|exit||Shuts down the crash session (same as q).|
|extend||Extend the crash command set by dynamically loading a shared object library containing one or more user-written commands.|
|foreach||Quite often it is helpful, or even necessary, to run the same crash context-sensitive command on a number of tasks by just entering one command. This wrapper command sets off the execution of a given crash command on each of a defined set of tasks, temporarily changing the current context to that of the targeted task before running the command. The set of tasks that are issued the given command can be defined by:
The identifiers above may be mixed if it makes sense, such as using a combination of PIDs, task addresses, and command names. The context-sensitive commands that can be issued to the selected tasks are: A header containing the PID, task address, CPU and command name will be pre-pended before the command output for each selected task.
|gdb||This command passes its arguments directly to gdb for processing. This is typically not necessary, but where ambiguities between crash and gdb command names exist, this will force the command to be executed by gdb.|
|repeat||This wrapper command repeats a crash command indefinitely, optionally delaying a given number of seconds between each command execution. Obviously this command is only useful when running on a live system.|
|set||This primary purpose for this command is to set the crash context to a new task, or to display the current context. It can also be used to view or change one of a set of internal crash variables that modify program behavior, such as the default output radix or scrolling behavior. It can be called from the foreach wrapper command for viewing the context data of each task.|
|q||Shuts down the crash session (same as exit).|
sys – system data
crash> sys KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore CPUS: 1 DATE: Mon Apr 30 21:38:40 2007 UPTIME: 00:04:04 LOAD AVERAGE: 0.36, 0.23, 0.08 TASKS: 36 NODENAME: localhost.localdomain RELEASE: 2.6.9-22.EL VERSION: #1 Mon Sep 19 18:20:28 EDT 2005 MACHINE: i686 (1862 Mhz) MEMORY: 1 GB PANIC: "Oops: 0002 [#1]" (check log for details)
bt – backtrace
crash> bt PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash" #0 [f7191e04] start_disk_dump at f89d7bb3 #1 [f7191e18] die at c010682e #2 [f7191e48] do_page_fault at c011ab00 [...] #9 [f7191fc0] system_call at c030f918 EAX: 00000004 EBX: 00000001 ECX: b7de7000 EDX: 00000002 DS: 007b ESI: 00000002 ES: 007b EDI: b7de7000 SS: 007b ESP: bfe01650 EBP: bfe01670 CS: 0073 EIP: 003297a2 ERR: 00000004 EFLAGS: 00000246
log – dump system message buffer
crash> log [...] SysRq : Crashing the kernel by request Unable to handle kernel NULL pointer dereference at virtual address 00000000 printing eip: c0233fa7 *pde = 3e9f3067 Oops: 0002 [#1] Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod CPU: 0 EIP: 0060: Not tainted VLI EFLAGS: 00010246 (2.6.9-22.EL) EIP is at sysrq_handle_crash+0x0/0x8 eax: 00000063 ebx: c0370db4 ecx: 00000000 edx: 00000000 esi: 00000063 edi: 00000000 ebp: 00000000 esp: f7191f60 ds: 007b es: 007b ss: 0068 Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0) Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002 f7191fac c01a8a13 c0362740 c0168205 f7191fac b7de7000 f7b6adc0 fffffff7 b7de7000 f7191000 c01682cf f7191fac 00000000 00000000 00000000 00000001 00000002 Call Trace:  __handle_sysrq+0x58/0xc6  write_sysrq_trigger+0x23/0x29  vfs_write+0xb6/0xe2  sys_write+0x3c/0x62  syscall_call+0x7/0xb Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff 05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89
ps – display process status information
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c0358be0 RU 0.0 0 0 [swapper]
1 0 0 f7e01770 IN 0.1 1680 684 init
2380 1 0 f7ac2800 IN 0.0 1604 504 mingetty
2769 2371 0 f7ac3970 IN 0.2 5740 1636 bash
2852 1 0 f7b1a880 IN 0.2 4240 2012 sshd
2855 2852 0 f7b66680 IN 0.3 8316 2756 sshd
> 2857 2855 0 f7b677f0 RU 0.2 6260 1628 bash
Sometimes it is useful to know which process belongs to which parent or vice versa.
-p to show the child and parent processes.
crash> ps -p 2857 PID: 0 TASK: c0358be0 CPU: 0 COMMAND: "swapper" PID: 1 TASK: f7e01770 CPU: 0 COMMAND: "init" PID: 2852 TASK: f7b1a880 CPU: 0 COMMAND: "sshd" PID: 2855 TASK: f7b66680 CPU: 0 COMMAND: "sshd" PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash"
files – open files
crash> files PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash" ROOT: / CWD: /root FD FILE DENTRY INODE TYPE PATH 0 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 1 f7b6adc0 f7190130 f7b9ca4c REG /proc/sysrq-trigger 2 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 10 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 255 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 crash> files 2852 PID: 2852 TASK: f7b1a880 CPU: 0 COMMAND: "sshd" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 1 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 2 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 3 f7b69600 f7bf5280 f7aadafc SOCK socket:/
dev – device data
crash> help dev [...] If no argument is entered, this command dumps the contents of the chrdevs and blkdevs arrays. crash> dev CHRDEV NAME OPERATIONS 1 mem (none) 4 /dev/vc/0 (none) 4 tty (none) [...] BLKDEV NAME OPERATIONS 1 ramdisk c0376d08 2 fd (unknown) 8 sd f880e070