Solaris Troubleshooting : Filesystem Full Errors in solaris

The following errors could be seen when a filesystem is full:

On the system console:

<date> <hostname> vmunix <filesystem> : file system full

WARNING: <filesystem>: file system full

NOTICE: alloc: <filesystem> file system full no space left on device

vmunix: file system full

For example:

WARNING: /: filesystem full

There are several reasons why a filesystem gets full. An important thing to consider  is how you set up your filesystems during installation,;you need to take care how much space is used for each filesystem and think ahead.

With forward thinking it is less likely that your filesystems will get full, but will  not prevent a filesystem getting full. This document will show the most common reasons why a filesystem may become full and how to handle them.

Troubleshooting:
 There are many ways of finding what’s filling up a filesystem, which can sometimes be a difficult process. One problem is that a filesystem can be filled up by one or few very large files (which is generally easy to find) or by thousands of smaller files (which can be difficult to find and pinpoint the cause). Solaris 10 Operating System for x86 Platforms

First you need to figure out which files are filling up your filesystem. A very useful way to list the size of files in a filesystem is with the du command.The following example lists files from largest to smallest on the root filesystem:

$ du -akd / | sort -nr | more

   or

$ du -akd / | sort -nr > /tmp/du.out

The latter will give you a file you can review at your convenience.The -d option of the du command keeps du from crossing partition boundaries.

The  -a  option tells du to report file sizes (without this option du just reports the amount of space used in each directory. The  -k  option means that du will report in terms of kilobytes rather than 512-byte blocks. On Solaris[TM] 9 or later replace  k  with  h  if you prefer  human-readable  output, that is output in terms of kilobytes, megabytes or gigabytes depending on the number reported. The -nr option of sort puts the files in reverse numerical order.

Of course, this can be used on filesystems other than root, just substitute the required path for  /  in the “du” command. The command  du -skd /  summarizes the amount of kilobytes used for a filesystem, in the given case for the root filesystem. One common problem with df showing more usage than du is existing data or files in directories that are used as mount points. Unmount any mounted filesystems and check the mount point directories for files.  Remove the files, or move them if you think you need them, and mount the filesystems again.

For the /tmp filesystem, you will have to boot the system into single-user mode to access the /tmp directory without having swap mounted over it.

For /var and /usr, you will have to boot the system from cdrom, mount the root filesystem, and then check the /var and /usr directories under the mounted root filesystem. These should normally be empty when /var or /usr is not mounted.

Another good way to search for files is to use the command ‘/usr/bin/find’

Here is the syntax to find files that are greater in size than 1 MB:
# cd <to desired directory>
# find . -size +1000000c -exec ls -l {} +

The  -mount  option to the find command can be used to restrict the search to the filesystem containing the directory specified. To find files generated by NFS and remove them if they are older than seven days:
# find / -name .nfs* -mtime +7 -exec rm -f {} + -o -fstype nfs -prune

To search for core files starting at the root directory and delete them:

# find / -name core -exec rm {} +

To search for core files that have not been accessed in seven days and display them to the screen:
# find / -name core -atime +7 -print

To identify all files owned by a particular user and send a long listing of these files to the superuser:
# find / -user <username> -ls |  mailx -s “users files” root@hostname

Look for files that have not been modified in 90 days in the /export/home directory:
# find /export/home -mtime +90 -print

To find files that are greater than 400 blocks (512-byte blocks) and display the matching path names:
# find /export/home -size +400 -print

Check /var/saf for a _log file and under /var/saf/tcp and /var/saf/zsmon for log files; they can be truncated with:
# cat /dev/null > filename

Do an ls -Ll of the /dev directory to make sure that all entries are character devices or block devices, not files or use this find:

# find /devices /dev -type f ! -size 0 -ls

**Specifically check the /dev/rmt directory, all entries should be links, not large files.

Also it could be that you are running out of inodes and are getting the message  file system full .  In this case recreate a partition with more inodes.  The basic steps are:

• Remove unneeded files.
• Backup the partition.
• recreate using newfs -i nbpi /dev/<rfsname> where nbpi is chosen smaller then the  default for the disk size, and rfsname is the raw filesystem; e.g. /dev/rdsk/cNtNdNsN.  See man newfs(1M) for more information.
• Restore information back to the partition.

In the course of normal system operation, the root and usr filesystems (or directories) are mostly static (do not grow over time).  /var however, does grow over time (because it contains log files, package database, print and mail spoolers, etc.).  The name  var  is in fact an abbreviation for  varying  or  variable  as the  /var  filesystem is intended for files which vary in size and content over time (see the filesystem(5) manual page for more details about this).  It is good system administration practice to monitor log files to make sure they don’t get too large.

If a filesystem suddenly fills up, that could have been caused by installing a new piece of software into a wrong directory.Check any lost+found directory on any filesystem that is full.

Another approach would be to list files by their modification date (if the date of when the filesystem filled up is known).

# ls -lRt / | more will list all the files and sort them by the modification dates.

Under very rare circumstances a UFS filesystem might appear full, while you still have data blocks and i-nodes available