Solaris 10 - Filesystem full and No space left on device error

Filesystem full and No space left on device error

Lots of comtrolM job failed. Audit and bart were enabled. it filled the /var filesystem. At the same time, it also dumped log on control log location.
Removed the audit log which was 7GB and other logs and free up 15GB of space but /tool was 100%.

Informed app team, but they had problem deleting. I tried to remove the file, same error No space left error.

used dd command to null out the file but it fills up quickly.
Tried to remove snapshot, could not remove it.

Later realize that there was issue on ControlM. Requested app team to bounce the ControlM and clean up some space.


# df -h
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/pBE_01222018   73G   7.4G     0K   100%    /
serptrnint-tool/tools     29G    29G     0K   100%    /tools

# ls -l
-rw-r--r--   1 ctrlmag  dba        15914 Mar 19 05:38 AS_20945.log
-rw-r--r--   1 ctrlmag  dba        31828 Mar 19 05:39 AS_20958.log
-rw-r--r--   1 ctrlmag  dba        15924 Mar 19 05:39 AS_20960.log
# ls -lh /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
-rwxrwxrwx   1 ctrlmag  dba         148M Sep  5  2017 /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN

# rm /tools/ctrlmag/ctm/proclog/AS_16935.log
rm: AS_16935.log not removed: No space left on device
# rm AS_16942.log
rm: AS_16942.log not removed: No space left on device
# rm -rf AS_20945.log
rm: cannot remove `./AS_20945.log': No space left on device

# ls -li AS_20945.log
    745 -rw-r--r--   1 ctrlmag  dba        15914 Mar 19 05:38 AS_20945.log
# find . -inum 745 -exec rm -i {} \;
rm: remove ./AS_20945.log (yes/no)? y
rm: cannot remove `./AS_20945.log': No space left on device


# du -sh /tools/ctrlmag/ctm/proclog
  16G   /tools/ctrlmag/ctm/proclog

# ls -lh /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
-rwxrwxrwx   1 ctrlmag  dba         148M Sep  5  2017 /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
# dd if=/dev/null of=/tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
0+0 records in
0+0 records out
# dd if=/dev/null of=/tools/ctrlmag/PAKAI.9.0.00.300_Solar
# ls -lh /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
-rwxrwxrwx   1 ctrlmag  dba            0 Mar 21 10:00 /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN
# df -h .
Filesystem             size   used  avail capacity  Mounted on
serptrnint-tool/tools  29G    29G     0K   100%    /tools
# rm /tools/ctrlmag/PAKAI.9.0.00.300_Solaris_INSTALL.BIN

# ls -l | wc -l
   38250

# rm AS_20945.log
# ls -li AS_20945.log
AS_20945.log: No such file or directory


Remove old files that are older then 30 days.

Locate files based on the time the files were last changed, accessed, or modified using the -ctime, -atime, or -mtime respectively.  So, first find all the files in a directory that are older than 30 days old.

# cd /myapp/location
# find . -mtime +30 -print

Review the output and remove the one that you don't need.

or you can pipe the find command output to xargs. Use -r option for xargs in case there is no output. The -r flag in xargs tells the command to skip execution when there is no data.

What the command below does is a recursive search under /myapp/location. Once it finds the matching entries, it deletes them. The arguement -print0 helps to find files on the location specify on which file name may have space.

# find /myapp/location -type f -mtime +30 -print0 | xargs -o rm -rf