How to use crash utility to perform some pre-analysis on vmcore – in RHEL5/RHEL6

Whenever a system with a valid kdump configuration crashes,  it will to create a  vmcore core file either locally or remotely based on the KDUMP location paramets in kdump.conf file. And we need to upload these vmcore files to Red Hat support staff for further analysis of actual root cause of kernel crash.  

Sometime, the advanced linux administrators uses the  crash utility  to analyse the vmcore prior to uploading it to Red Hat Support site.  In this post we will discuss about some of the crash commands that helps us to perform pre-analysis of vmcore.


The Crash Command Set

Each crash command generally falls into one of the following categories:

The remainder of this section breaks the command set into categories, and gives a short description of each command in that category. However, for complete details and examples, recall that the crash utility has a self-contained help page for each command; to view the full help page, click on the command name next to its description below.


crash> help
* files mod runq union
alias foreach mount search vm
ascii fuser net set vtop
bt gdb p sig waitq
btop help ps struct whatis
dev irq pte swap wr
dis kmem ptob sym q
eval list ptov sys
exit log rd task
extend mach repeat timer

crash version: 4.0-8.11   gdb version: 6.1
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".




Symbolic Display of Kernel Text or Data

The following commands typically take full advantage of the power of gdb to display kernel data structures symbolically.

Command   Description
struct   Displays a formatted kernel data structure type located at a given address, or at an address referred to by a symbol; if no address is specified, the structure definition is displayed. The output can be narrowed down to a singular member of the structure, or to display the offset of every member from the beginning of the structure. A count may be appended to display an array of structures. Its usage is so common that two short-cuts exist such that the user need not enter the “struct” command name:

  1. The “pointer-to” * command below can be substituted.
  2. If a structure name is entered as the first token on a command line, the “struct” command is actually not necessary.
union   Same as struct command, but used for kernel data types defined as unions instead of structures..
*   “Pointer-to” command which can be used in lieu of entering struct or union; the gdbmodule first determines whether the argument is a structure or a union, and then calls the appropriate function.
p   Displays the contents of a kernel variable; the arguments are passed on to gdb‘s printcommand for proper formatting. Two builtin aliases, px and pd, set the numerical output radix to hexadecimal or decimal for the print operation, temporarily overriding the current default.
whatis   Displays all available symbol table information concerning a data type or a data symbol.
sym   Translates a kernel symbol name to its kernel virtual address and section, or a kernel virtual address to its symbol name and section. It can also be used to dump the complete list of kernel symbols, or to query the symbol list for all symbols containing a given sub-string.
dis   Disassembles the text of complete kernel function, or from a specified address for a given number of instructions, or from the beginning of a function up to a specified address.

System State

The majority of crash commands come from the following set of “kernel-aware” commands, which delve into various kernel subsystems on a system-wide or per-task basis. The task-specific commands are context-sensitive, meaning that they act upon the current context unless a PID or task address is specified as an argument.

Command   Description
bt   Arguably the most useful crash command, bt displays a task’s kernel stack back-trace, including full exception frame dumps. It is context-sensitive, although the -a option will display the stack traces of the active task on each CPU. This command is often used within the foreach wrapper command in order to display the back traces of all tasks with one command.
dev   Displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data.
files   This context-sensitive command displays the task’s current root directory and working directories, and then for each open file descriptor, shows:

  • its file struct address
  • its dentry struct address
  • its inode struct address
  • the file type
  • the file’s full pathname

Another option acts upon a specified dentry address, showing:

  • its inode struct address
  • its superblock struct address
  • the file type
  • the file’s full pathname

It can be called from the foreach wrapper command.

fuser   Displays a list of tasks that reference a specified filename or inode address as the current root or working directory, an open file descriptor, or which mmap the file.
irq   Display data concerning interrupt request numbers and bottom-half handling.
kmem   This command has numerous options that delve into the state of several kernel memory subsystems:

  • general memory usage, similar in scope to /proc/meminfo
  • kmalloc slab memory allocator, including an option that lists each slab object and its state, verifying the slab chain
  • display and verification of free page lists
  • vmalloc memory allocator vmlist contents
  • display and verification of the page cache
  • the mem_map page list
  • display NUMA data, if applicable

Also, given an address, this command searches the symbol table, the slab subsystem, the free list, the page_hash_table, the vmlist, and the mem_map array, displaying where it was found.

log   Dumps the kernel message buffer chronologically, accounting for any wrap-around.
mach   Displays machine and/or processor specific data.
mod   Displays the list of currently-loaded kernel modules. More importantly, it loads the debug data from the module object files if they are available, allowing symbolic debugging capability of kernel modules.
mount   For each mounted filesystem, or for just a specified filesystem, displays:

  • its vfsmount struct address
  • its super_block struct address
  • its type
  • its device name
  • its mount point

Options exist to dump a list of a specified filesystem’s open files or dirty inodes. Filesystems may be specified by vfsmount, super_block, or inode addresses, or by device name or mount point names.

net   Displays various network-related data:

  • displays each configured network device’s net_device address, its name, and IP address
  • displays the ARP cache
  • context-sensitive display of information concerning the open sockets of a task
  • translates an IP address expressed as a decimal or hexadecimal value into a standard numbers-and-dots notation

It can be called from the foreach wrapper command.

ps   Useful process status command, in typical Linux ps command type output, containing:

  • PID number
  • PPID number
  • CPU number
  • task address
  • process state
  • percent of physical memory consumed
  • virtual address size
  • resident set size
  • command name

Also has an option to show a task’s parental hierarchy back to the init process, and another to show all children of a task.

pte   This command translates the contents of a PTE into its physical page address and page bit settings, or if it references a swap location, the swap device and offset.
runq   Displays list of tasks on the run queue.
sig   A context-sensitive command which displays a task’s signal information, including:

  • whether an unblocked signal is pending
  • the pending and blocked signals
  • the handler data for each signal
  • queued signals, if any

Other options list the signal number/names combination for a processor type, and translate the contents of a sigset_t into the signal names whose bits are set. It can be called from the foreach wrapper command.

swap   For each configured swap device, this command displays the same data that is shown by the Linux command swapon -s.
sys   Re-displays the same system-related data that is seen during crash initialization:

  • the kernel object filename
  • the dumpfile name
  • the number of CPUS
  • the date
  • system uptime
  • system load average
  • the number of tasks
  • the nodename
  • the kernel release and version data
  • the processor type and speed
  • the amount of memory
  • the panic string

Other options display information concerning the system call table, and one allows theroot user to panic a live system.

task   This context-sensitive command displays a task’s complete task_struct contents, or one or more members of the structure. This command is often used within the foreachwrapper command in order to display task_struct data for all tasks with one command.
timer   Displays the timer queue entries in chronological order, listing the target function names, the current value of jiffies, and the expiration time of each entry.
vm   This powerful, context-sensitive command displays a wealth of information concerning a task’s virtual memory data, including:

  • its mm_struct address
  • its page directory address
  • its resident set size
  • its total virtual memory size
  • each vm_area_struct address, along with its start and ending virtual address, flags, and source file if applicable.
  • optionally, every virtual page referenced by a vm_area_struct can be translated into its physical address, or if not resident, its file and offset.

Other options translate the flags of a vm_area_struct, or display the full contents of a task’s mm_struct or of each vm_area_struct. It can be called from the foreach wrapper command.

vtop   This context-sensitive command translates a user or kernel virtual address to its physical address. Also displayed are:

  • the full PTE translation from page directory through to the page table
  • the vm_area_struct data for user virtual addresses
  • the mem_map page data associated with the physical page
  • the swap location or file location if a user virtual page is not currently mapped

It can be called from the foreach wrapper command.

waitq   Lists the tasks linked on a specified kernel wait queue.

Utility Functions

The following commands are a set of useful helper commands serving various purposes, some simple, others quite powerful.

Command   Description
ascii   Translates a numerical value into its ASCII components; with no arguments, displays an ASCII chart.
btop   Translates a byte value (physical address) to its page number.
eval   A simple calculator, evaluates an expression and displays the result in hexadecimal, decimal, octal and binary, and optionally showing the bit numbers set in the result.
list   Dumps the entries of a linked list of structures. It can handle lists of structures that are singly-linked with simple “next” pointers, or those with embedded list_head structures. The output may be constrained to simply display the address of each structure in the list, or if directed, also dump each complete structure, or just one member of each structure. The gathered list entries are hashed, so a corrupted list that loops back upon itself will be recognized.
ptob   translates a page frame number to its byte value (physical address).
ptov   Translates a physical address into a kernel virtual address by adding the appropriatePAGE_OFFSET value.
search   Searches a range of user or kernel memory space for given value, with an optional “don’t care” bit-mask argument.
rd   Displays a specified amount of user virtual, kernel virtual, or physical memory in several formats, such as 8, 16, 32 or 64 bit values, hexadecimal or decimal, symbolically, and with ASCII translations. When reading user virtual addresses, the command is context-sensitive.
wr   Modifies the contents of memory on a live system. Write permission on /dev/mem is required; this command should obviously be used with great care. The write operation is constrained to one 8, 16, 32 or 64 bit location.

Session Control Commands

The following commands typcally aid in the efficient running of a crash session.

Command   Description
alias   Creates a single-word alias for a command string. Several aliases are built into crash; user-defined aliases may also be defined in a .crashrc file, or during a crash session by entering it on the command line or reading it from an input file.
exit   Shuts down the crash session (same as q).
extend   Extend the crash command set by dynamically loading a shared object library containing one or more user-written commands.
foreach   Quite often it is helpful, or even necessary, to run the same crash context-sensitive command on a number of tasks by just entering one command. This wrapper command sets off the execution of a given crash command on each of a defined set of tasks, temporarily changing the current context to that of the targeted task before running the command. The set of tasks that are issued the given command can be defined by:

  • one or more PID numbers
  • one or more task numbers
  • one or more command name
  • all user tasks
  • all kernel tasks
  • the active task on each CPU

The identifiers above may be mixed if it makes sense, such as using a combination of PIDs, task addresses, and command names. The context-sensitive commands that can be issued to the selected tasks are: A header containing the PID, task address, CPU and command name will be pre-pended before the command output for each selected task.

gdb   This command passes its arguments directly to gdb for processing. This is typically not necessary, but where ambiguities between crash and gdb command names exist, this will force the command to be executed by gdb.
repeat   This wrapper command repeats a crash command indefinitely, optionally delaying a given number of seconds between each command execution. Obviously this command is only useful when running on a live system.
set   This primary purpose for this command is to set the crash context to a new task, or to display the current context. It can also be used to view or change one of a set of internal crash variables that modify program behavior, such as the default output radix or scrolling behavior. It can be called from the foreach wrapper command for viewing the context data of each task.
q   Shuts down the crash session (same as exit).



Example Commands:

sys – system data

crash> sys
       KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux
     DUMPFILE: /home/eteo/crash/
         CPUS: 1
         DATE: Mon Apr 30 21:38:40 2007
       UPTIME: 00:04:04
 LOAD AVERAGE:  0.36, 0.23, 0.08
        TASKS: 36
     NODENAME: localhost.localdomain
      RELEASE: 2.6.9-22.EL
      VERSION: #1 Mon Sep 19 18:20:28 EDT 2005
      MACHINE: i686 (1862 Mhz)
       MEMORY: 1 GB
        PANIC: "Oops: 0002 [#1]" (check log for details)

bt – backtrace

crash> bt
PID: 2857   TASK: f7b677f0 CPU: 0    COMMAND: "bash"
 #0 [f7191e04] start_disk_dump at f89d7bb3
 #1 [f7191e18] die at c010682e
 #2 [f7191e48] do_page_fault at c011ab00
 #9 [f7191fc0] system_call at c030f918
    EAX: 00000004 EBX: 00000001 ECX: b7de7000 EDX: 00000002
    DS: 007b       ESI: 00000002 ES: 007b        EDI: b7de7000
    SS: 007b       ESP: bfe01650 EBP: bfe01670
    CS: 0073       EIP: 003297a2 ERR: 00000004 EFLAGS: 00000246

log – dump system message buffer

crash> log
SysRq : Crashing the kernel by request
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
*pde = 3e9f3067
Oops: 0002 [#1]
Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod
CPU:    0
EIP:    0060:[]    Not tainted VLI
EFLAGS: 00010246    (2.6.9-22.EL)
EIP is at sysrq_handle_crash+0x0/0x8
eax: 00000063   ebx: c0370db4    ecx: 00000000  edx: 00000000
esi: 00000063   edi: 00000000    ebp: 00000000  esp: f7191f60
ds: 007b   es: 007b    ss: 0068
Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0)
Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002
       f7191fac c01a8a13 c0362740 c0168205 f7191fac b7de7000 f7b6adc0
       fffffff7 b7de7000 f7191000 c01682cf f7191fac 00000000 00000000
       00000000 00000001 00000002
Call Trace:
 [] __handle_sysrq+0x58/0xc6
 [] write_sysrq_trigger+0x23/0x29
 [] vfs_write+0xb6/0xe2
 [] sys_write+0x3c/0x62
 [] syscall_call+0x7/0xb
Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff  05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89 

ps – display process status information

crash> ps
0 0 0 c0358be0 RU 0.0 0 0 [swapper]
1 0 0 f7e01770 IN 0.1 1680 684 init
2380 1 0 f7ac2800 IN 0.0 1604 504 mingetty
2769 2371 0 f7ac3970 IN 0.2 5740 1636 bash
2852 1 0 f7b1a880 IN 0.2 4240 2012 sshd
2855 2852 0 f7b66680 IN 0.3 8316 2756 sshd
> 2857 2855 0 f7b677f0 RU 0.2 6260 1628 bash

Sometimes it is useful to know which process belongs to which parent or vice versa. ps has -c and -p to show the child and parent processes.

crash> ps -p 2857
PID: 0       TASK: c0358be0 CPU: 0    COMMAND: "swapper"
 PID: 1       TASK: f7e01770 CPU: 0    COMMAND: "init"
  PID: 2852    TASK: f7b1a880 CPU: 0    COMMAND: "sshd"
   PID: 2855    TASK: f7b66680 CPU: 0    COMMAND: "sshd"
    PID: 2857    TASK: f7b677f0 CPU: 0    COMMAND: "bash" 

files – open files

crash> files
PID: 2857    TASK: f7b677f0 CPU: 0   COMMAND: "bash"
ROOT: /     CWD: /root
  0 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0
  1 f7b6adc0 f7190130 f7b9ca4c     REG   /proc/sysrq-trigger
  2 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0
 10 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0
255 f7a6e7c0 f7790198 f7b0fdcc     CHR   /dev/pts/0
crash> files 2852
PID: 2852    TASK: f7b1a880 CPU: 0   COMMAND: "sshd"
ROOT: /     CWD: /
  0 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null
  1 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null
  2 f7b336c0 f78001d8 f7cb1ba4     CHR   /dev/null
  3 f7b69600 f7bf5280 f7aadafc     SOCK  socket:/[6277]


dev – device data

crash> help dev
 If no argument is entered, this command dumps the contents of the
 chrdevs and blkdevs arrays.
crash> dev
 1      mem              (none)
 4      /dev/vc/0        (none)
 4      tty              (none)
 1      ramdisk        c0376d08 
 2      fd            (unknown)
 8      sd             f880e070  






I have started ( aka in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' You can connect me at -

What is in your mind, about this post ? Leave a Reply

  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us