Today, I got an error like this:
SUNW-MSG-ID: FMD-8000-2K, TYPE: Defect, VER: 1, SEVERITY: Minor EVENT-TIME: Tue Aug 4 14:52:43 WIT 2009 PLATFORM: SUNW,SPARC-Enterprise, CSN: BEF09142B8, HOSTNAME: server SOURCE: fmd-self-diagnosis, REV: 1.0 EVENT-ID: d232928b-dd11-e149-b515-82ff5df189f8 DESC: A Solaris Fault Manager component has experienced an error that required the module to be disabled. Refer to http://sun.com/msg/FMD-8000-2K for more information. AUTO-RESPONSE: The module has been disabled. Events destined for the module will be saved for manual diagnosis. IMPACT: Automated diagnosis and response for subsequent events associated with this module will not occur. REC-ACTION: Use fmdump -v -u <EVENT-ID> to locate the module. Use fmadm reset <module> to reset the module. root@server #
Aug 4 14:37:13 server ufs: NOTICE: alloc: /var: file system full
it seem “/var” directory full. OK, lets check disk space.
root@server # df -h Filesystem size used avail capacity Mounted on /dev/md/dsk/d10 20G 7.0G 12G 37% / /devices 0K 0K 0K 0% /devices ctfs 0K 0K 0K 0% /system/contract proc 0K 0K 0K 0% /proc mnttab 0K 0K 0K 0% /etc/mnttab swap 88G 1.6M 88G 1% /etc/svc/volatile objfs 0K 0K 0K 0% /system/object sharefs 0K 0K 0K 0% /etc/dfs/sharetab fd 0K 0K 0K 0% /dev/fd /dev/md/dsk/d30 20G 20G 0K 100% /var swap 88G 32K 88G 1% /tmp swap 88G 72K 88G 1% /var/run /dev/md/dsk/d40 12G 12M 12G 1% /oracle /dev/md/dsk/d50 36G 37M 36G 1% /internaldisk1
now, lets check whats going on with “/var” directory.
root@server # cd /var/adm/ root@server # ls -ltr total 4242 -rw-rw-rw- 1 root bin 0 Aug 25 2008 spellhist -rw------- 1 uucp bin 0 Aug 25 2008 aculog drwxr-xr-x 2 adm adm 512 Jun 25 18:15 exacct drwxr-xr-x 2 adm adm 512 Jun 25 18:15 log drwxr-xr-x 2 root sys 512 Jun 25 18:15 streams drwxr-xr-x 2 root sys 512 Jun 25 18:19 pool drwxrwxr-x 5 adm adm 512 Jun 25 18:26 acct drwxrwxr-x 2 adm sys 512 Jun 25 18:26 sa drwxr-xr-x 2 root sys 512 Jun 25 18:57 sm.bin -rw-r--r-- 1 root root 0 Jun 25 19:25 vold.log -rw-r--r-- 1 root root 34472 Jun 25 19:27 messages.3 -rw-r--r-- 1 root root 73131 Jul 2 13:40 messages.2 -rw-r--r-- 1 root root 249 Jul 24 18:35 messages.1 -rw-r--r-- 1 root root 1906434 Jul 31 20:15 messages.0 -r--r--r-- 1 root root 28 Aug 4 15:18 lastlog -rw-r--r-- 1 root bin 2232 Aug 4 15:18 utmpx -rw-r--r-- 1 adm adm 51708 Aug 4 15:18 wtmpx -rw-r--r-- 1 root root 81457 Aug 4 15:19 messages
wtmpx,utmpx,messages are normal in size.
Check crash dump and core files:
root@server # dumpadm Dump content: kernel pages Dump device: /dev/md/dsk/d30 (dedicated) Savecore directory: /var/crash/server Savecore enabled: yes root@server # root@server # cd /var/crash/ root@server # ls server root@server # ls -ltr total 2 drwx------ 2 root root 512 Jun 25 19:25 server root@server # cd server/ root@server # ls root@server # ls -ltr total 0
OK, no crash dump or core dump on /var.
finally, after checking each directory on “/var” one by one, found directory “/var/fm” with 18GB space.
root@server # du -sh /var/fm/fmd 18G fmd
root@server # pwd /var/fm/fmd root@server # ls ckpt core.fmd.1506 core.fmd.1726 core.fmd.1948 core.fmd.2168 core.fmd.1284 core.fmd.1508 core.fmd.1730 core.fmd.1950 core.fmd.2170 core.fmd.1287 core.fmd.1510 core.fmd.1732 core.fmd.1952 core.fmd.2172 core.fmd.1290 core.fmd.1512 core.fmd.1734 core.fmd.1954 core.fmd.2174 core.fmd.1293 core.fmd.1514 core.fmd.1736 core.fmd.1956 core.fmd.2176 core.fmd.1295 core.fmd.1516 core.fmd.1738 core.fmd.1958 core.fmd.2178 core.fmd.1297 core.fmd.1518 core.fmd.1740 core.fmd.1960 core.fmd.2180 core.fmd.1299 core.fmd.1520 core.fmd.1742 core.fmd.1962 core.fmd.2182 core.fmd.1302 core.fmd.1522 core.fmd.1744 core.fmd.1964 core.fmd.2184 core.fmd.1304 core.fmd.1526 core.fmd.1746 core.fmd.1966 core.fmd.2186 core.fmd.1307 core.fmd.1528 core.fmd.1748 core.fmd.1968 core.fmd.2188 core.fmd.1310 core.fmd.1530 core.fmd.1750 core.fmd.1970 core.fmd.2190 core.fmd.1312 core.fmd.1532 core.fmd.1752 core.fmd.1972 core.fmd.2192 core.fmd.1314 core.fmd.1534 core.fmd.1754 core.fmd.1974 core.fmd.2194 core.fmd.1316 core.fmd.1536 core.fmd.1756 core.fmd.1976 core.fmd.2196
Found alot of “core.fmd.xxx” files on “/var/fm/fmd”.
did you know that, this error is related with Solaris Fault Management, generated by fmd services on Solaris.
what is fmd services?
- fmd is a daemon that runs in the background on each Solaris system. fmd receives telemetry information relating to problems detected by the system software, diagnoses these problems, and initiates proactive self-healing activities such as disabling faulty components. When appropriate, the fault manager also sends a message to the syslogd(1M) service to notify an administrator that a problem has been detected. The message directs administrators to a knowledge article on Sun’s web site, http://www.sun.com/msg/, which explains more about the problem impact and appropriate responses.
Each problem diagnosed by the fault manager is assigned a Universal Unique Identifier (UUID). The UUID uniquely identifes this particular problem across any set of systems. The fmdump(1M) utility can be used to view the list of problems diagnosed by the fault manager, along with their UUIDs and knowledge article message identifiers. The fmadm(1M) utility can be used to view the resources on the system believed to be faulty. The fmstat(1M) utility can be used to report statistics kept by the fault manager. The fault manager is started automatically when Solaris boots, so it is not necessary to use the fmd command directly. Sun’s web site explains more about what capabilities are currently available for the fault manager on Solaris.
The Solaris Fault Management Facility is designed to be integrated into the Service Management Facility to provide a self-healing capability to Solaris 10 systems. The fmd daemon is responsible for monitoring several aspects of system health.
my “/var/fm/fmd” is full because today I’m working on IPMP configuration and testing and there are a problem with my network connection to Cisco Switch. so thats way, Solaris create alot of “core.fmd” files.
you can run “mdump -v -u <EVENT-ID>” to locate the module.
Use fmadm reset <module> to reset the module
# fmdump -v -u 815bf413-9de6-4667-e118-93dc3bc33e71
for temporary solution, I disable the “fmd” services and remove all core.fmd files from “/var/fm/fmd” directory.
root@server # svcs fmd STATE STIME FMRI online Jul_29 svc:/system/fmd:default root@server # svcadm disable fmd root@server # svcs fmd STATE STIME FMRI disabled 16:48:03 svc:/system/fmd:default
root@server # rm core.fmd.* I'll re enable again the fmd services when I finished with my IPMP configuration..