Did your Solaris OS suddenly crashed, hang and rebooted by it self for no reason? After initial checking no amber light found on or HW faulty on the server ? its time to check the crash dump log file.

If your system properly installed and configured, the moment system crash, it will save all data on the memory to specific file so call crash dump log file. Check your dump log file location with “dumpadm” command. Usually, its located at: /var/crash/<server_hostname>/, file name: vmdump.0

What we need to do is to read and analyze this most important file. Unfortunately, this dump log file can’t be read manually, its required special package: SUNWscat – Oracle Solaris Crash Analysis Tool.

Download and install SUNWscat – Solaris Crash Analyzer Tool

The latest version Oracle Solaris Crash Analysis Tool is version 5.5, this patches are available on MOS, and can be found by searching on the patchIDs 21099218 (Combined package supporting SPARC and X86/X64) and 21099215 (platform specific packages).

Go to MOS: https://support.oracle.com and login, Click on tab entitled Patches and Updates at top, Enter 21099218 and 21099215 for patch numbers.

————

[Install the package:

root@solaris10 # ls
p21099218_55000_Generic.zip
root@solaris10 # unzip p21099218_55000_Generic.zip
Archive: p21099218_55000_Generic.zip
inflating: Readme.txt
inflating: SUNWscat5.5-GA-combined.pkg.gz
root@solaris10 #
root@solaris10 # gunzip SUNWscat5.5-GA-combined.pkg.gz
root@solaris10 #
root@solaris10 # ls
Readme.txt SUNWscat5.5-GA-combined.pkg p21099218_55000_Generic.zip
root@solaris10 # pkgadd -d SUNWscat5.5-GA-combined.pkg

The following packages are available:
1 SUNWscat Oracle Solaris Crash Analysis Tool (5.5 GA)
(any) 5.5

Select package(s) you wish to process (or ‘all’ to process
all packages). (default: all) [?,??,q]: all

Processing package instance <SUNWscat> from </export/home/SCAT/SUNWscat5.5-GA-combined.pkg>
….
….
Installation of <SUNWscat> was successful.

[decompress the crash dump log file:
root@solaris10 # savecore -vf vmdump.0
savecore: System dump time: Mon Aug 10 05:43:52 2015

savecore: saving system crash dump in /opt/crash/solaris10/{unix,vmcore}.0
Constructing namelist /opt/crash/solaris10/unix.0
Constructing corefile /opt/crash/solaris10/vmcore.0
1:15 100% done: 341483 of 341483 pages saved
48444 (14%) zero pages were not written
1:16 dump decompress is done
root@solaris10 #

[Let start analyze the log:
root@solaris10 # cd /opt/crash/solaris10/
root@solaris10 # /opt/SUNWscat/bin/scat vmcore.0

Oracle Solaris Crash Analysis Tool
Version 5.5 for Oracle Solaris 10 64-bit UltraSPARC

Copyright (c) 1989, 2015, Oracle and/or its affiliates. All rights reserved.

Please note: Do not submit any health, payment card or other sensitive
production data that requires protections greater than those specified in
the Oracle GCS Security Practices. Information on how to remove data from
your submission is available at:
https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1227943.1

For support, please use the Oracle Solaris kernel community at
https://community.oracle.com/community/support/oracle_sun_technologies/
Select “Subspaces” and then “Oracle Solaris Performance, Panics,
Hangs, and Dtrace”.
Further information may be found at https://blogs.oracle.com/SolarisCAT/

opening unix.0 vmcore.0 …dumphdr…symtab…maps…done
loading crashdump data: modules…CTF…globals…done

crash file: /opt/crash/solaris10/vmcore.0
user: Super-User (root:0)
release: 5.10 (64-bit)
version: Generic_144488-06
machine: sun4v
node name: XXXX
domain: default.solaris10.com
hw_provider: Sun_Microsystems
system type: SUNW,Netra-T5440 (UltraSPARC-T2+)
hostid: XXXXXXXX
…

…
disks…done

[run ‘analyze’:

CAT(vmcore.0/10V)> analyze

crash file: /opt/crash/solaris10/vmcore.0
user: Super-User (root:0)
release: 5.10 (64-bit)
version: Generic_144488-06
machine: sun4v
node name: XXX
domain: default.server.com
hw_provider: Sun_Microsystems
system type: SUNW,Netra-T5440 (UltraSPARC-T2+)
hostid: xxxxxxxx
dump_conflags: 0x10000 (DUMP_KERNEL) on /dev/dsk/c1t0d0s1(8G)
time of crash: Mon Aug 10 05:43:01 WIT 2015
age of system: 37 days 19 hours 54 minutes 4 seconds
panic CPU: 96 (96 CPUs, 15.7G memory, 2 nodes)
panic string: xt_sync: timeout
==== panic thread: 0x300c2653200 ==== CPU: 96 ====
==== panic user (LWP_SYS) thread: 0x300c2653200 PID: 22900 on CPU: 96 ====
cmd: /bin/sh /opt/scripts/xxxxxxx Called from script ‘/opt/scrips’    >>>>THE ROOT CAUSE OF CRASH/HUNG
t_procp: 0x60024f870b0
p_as: 0x6003a631c18 size: 1769472 RSS: 1482752

….
— switch to user thread’s user stack —

——————-

‘analyze’ is just one of initial investigation command, type help for other commands:

CAT(vmcore.0/10V)> help
CAT(vmcore.0/10V)>