Solaris Troubleshooting : Memory usage checking and troubleshooting

First you’ll need to check  how much Memory is used in Kernel and how much is used in User Memory. This is important to decide, what troubleshooting steps are required.
A nice mdb dcmd ( this command can take several minutes to complete, and you have to be extremely careful to run this command ) Solaris 9 Operating System or greater only !

# echo “::memstat” | mdb -k
Page Summary      Pages                 MB          %Tot
———— —————- —————-  —-
Kernel              151528              1183            10%
Anon                185037              1445            12%
Exec and libs      17092                133              1%
Page cache       281433              2198            19%
Free (cachelist) 633893              4952            42%
Free (freelist)    235470              1839            16%
Total               1504453            11753

User memory usage :  print out processes using most USER – memory

% prstat -s size # sorted by userland virtual memory consumption
% prstat -s rss # sorted by userland phyiscal memory consumption

% prstat -s rss
PID  USERNAME SIZE     RSS  STATE  PRI NICE TIME  CPU  PROCESS/NLWP
552     alfredm       341M    156M   sleep    59     0    0:52:31  9.4%        Xsun/1
760     alfredm       148M   131M    sleep    49     0    4:58:00  8.4%        mozilla-bin/9
1803   alfredm         87M     77M    sleep    49     0    0:00:12  0.0%        dtpad/1
682     alfredm         23M     15M    sleep    49     0    0:04:37  0.1%        xchat/1
9         root          9872K  8520K    sleep    59     0    0:00:22  0.0%        svc.configd/16
679     alfredm         11M  8184K    sleep   59     0    0:01:26  0.1%        dtwm/5
667     alfredm         11M  7144K    sleep   49     0    0:00:52  0.0%        dtsession/3
1861   alfredm     8688K   6256K   sleep    49    0    0:00:00   0.0%       dtcm/1
373     root              12M  6168K    sleep   59     0    0:00:04   0.0%       fmd/14
537     root          9296K   6056K    sleep   59    0    0:00:04   0.0%       snmpd/1
7         root          8648K   5912K    sleep   59    0    0:00:08   0.0%       svc.startd/12

User Memory Usage : shows Shared Memory and Semaphores:

% ipcs -a
IPC status from
T  ID      KEY        MODE       OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID  STIME   RTIME   CTIME
Message Queues:
q  0   0x55460272 -Rrw-rw—-   root        root          root             root            0             0      4194304   1390   18941  14:12:20 14:12:21  10:23:32
q  1   0x41460272 –rw-rw—-    root        root          root             root            0             0      4194304   5914     1390    8:03:34   8:03:34  10:23:39
q  2   0x4b460272 –rw-rw—-    root        root          root             root            0             0      4194304         0           0   no-entry  no-entry 10:23:39
T  ID      KEY        MODE      OWNER GROUP CREATOR CGROUP NATTCH  SEGSZ      CPID  LPID    ATIME   DTIME  CTIME
Shared Memory:
m  0  0x50000b3f  –rw-r–r–     root       root         root            root                    1                     4   738      738  18:50:36 18:50:36 18:50:36
m  1  0x52574801 –rw-rw—-   root       oracle      root            oracle                35   1693450240 2049 26495  10:30:00 10:30:00 18:51:13
m  2  0x52574802 –rw-rw—-   root       oracle      root            oracle                35   1258291200 2049 26495  10:30:00 10:30:00 18:51:16
m  3  0x52594801 –rw-rw—-   root       oracle      root            oracle                12     241172480 2098 14328    7:58:33   7:58:33 18:51:27
m  4  0x52594802 –rw-rw—-   root       oracle      root            oracle                12       78643200 2098 14329    7:58:32   7:58:33 18:51:27
m  5  0x52584801 –rw-rw—-   root       oracle      root            oracle                13     125829120 2125 27492    1:36:12   1:36:12 18:51:34
m  6  0x52584802 –rw-rw—-   root       oracle      root            oracle                13     268435456 2125 27487    1:36:10   1:36:11 18:51:34
m  7  0x525a4801 –rw-rw—-   root       oracle      root            oracle                15      912261120 2160 27472   1:36:09   1:36:09 18:51:40
m  8  0x525a4802 –rw-rw—-   root       oracle      root            oracle                15      268435456 2160 27467   1:36:08   1:36:09 18:51:42
m 8201 0x4d2      –rw-rw-rw-   root       root         root            root                     0             32008  1528 1543   10:26:03 10:26:04 10:25:53
T ID KEY   MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME  CTIME
Semaphores:
s  0   0x1  –ra-ra-ra-  root        root           root          root               1      16:17:35 18:50:33
s  1     0   –ra-ra—-   root       oracle         root         oracle           36     10:33:28 18:51:17
s  2     0   –ra-ra—-   root       oracle         root         oracle           13     10:33:28 18:51:27
s  3     0   –ra-ra—-   root       oracle         root         oracle           14     10:33:28 18:51:34
s  4     0   –ra-ra—-   root       oracle         root         oracle           16     10:33:27 18:51:42
s  5 0x4d2 –ra-ra-ra- root       root            root         root                1      no-entry 10:25:53
s  6 0x4d3 –ra-ra-ra- root       root            root         root                1      no-entry 10:25:53

User Memory Usage : lists User Memory usage of all processes ( except PID 0,2,3 )

# pmap -x /proc/* > /var/tmp/pmap-x
short list of total usage of these processes

% egrep “[0-9]:|^total” /var/tmp/pmap-x
1:   /sbin/init
total Kb 2336 2080  128 –
1006:  rlogin cores4
total Kb 2216 1696    80 –
1007:  rlogin cores4
total Kb 2216 1696  104 –
115:  /usr/sbin/nscd
total Kb 4208 3784 1704 –
— snip —

====================================================================

User Memory Usage : check the usage of /tmp

% df -kl /tmp
Filesystem kbytes        used        avail capacity  Mounted on
swap        1355552    2072 1353480        1%      /tmp

print the biggest 10 files and dirs in /tmp

% du -akd /tmp | sort -n | tail -10 8
/tmp/hsperfdata_root 8
/tmp/jnlpgenerator-8 8
/tmp/jnlpgenerator-8-1 8
/tmp/jnlpgenerator-8-2 8
/tmp/nscopy.tmp 8
/tmp/nsmail.eml 8
/tmp/speckeysd.lock 16
/tmp/tt 48
/tmp/lucreate.man 800
/tmp/dtdbcache_:0 1088
/tmp

User Memory Usage : Overall Memory usage on system


% vmstat -p 3
memory               page          executable  anonymous  filesystem
swap       free      re  mf  fr de sr epi epo epf  api apo apf  fpi fpo fpf
1468048 508176 1 48 0  0   0   1   0    0     0    0    0     2  0    0
1355376 459272 0   9 0  0   0   0   0    0     0    0    0     0  0    0
1355376 459272 0   2 0  0   0   0   0    0     0    0    0     0  0    0

scanrate should be 0 ( or near zero )

User Memory Usage : Swap usage

% swap -l
swapfile                  dev    swaplo blocks      free
/dev/dsk/c0t0d0s1 32,25        16 1946032 1946032

% swap -s
total: 399400k bytes allocated + 18152k reserved = 417552k used, 1355480k available

Check common kernel statistics

print out all kernel statistics in a parse’able format
% kstat -p > /var/tmp/kstat-p

kernel memory statistics:
% kstat -p -c kmem_cache
% kstat -p -m vmem
% kstat -p -c vmem

alternatively to kstat you can get kernel memory usage with kmastat  prints kmastat buffers

# echo “::kmastat” | mdb -k > /var/tmp/kmastat
% more /var/tmp/kmastat
cache                             buf     buf     buf     memory       alloc  alloc
name                             size    in use  total    in use     succeed    fail
————————- —— —— —— ——— ——— —–
kmem_magazine_1           16      470    508      8192         470     0
kmem_magazine_3           32      970  1016    32768       1164     0
kmem_magazine_7           64    1690  1778   114688      1715     0

Look for the highest numbers in column “memory in use” and for any numbers higher than ‘0’ in column “alloc fail”

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

18 Responses

  1. seema says:

    Hello Ramdev I have swap issues application are running java
    # Reduce memory load on the system
    # Increase physical memory or swap space
    # Check if swap backing store is full i have run swap -l swap -s checked df -k /tmp but how will i know or say apps team this is reason … when i run prstat or vmstat how can i explain from output to apps team or how can i support …

  2. Yogesh Raheja says:

    prstat -a will give the apps consuming high cpu and mem.

  3. seema says:

    thanks @ yogesh ….. Can you provide me link or oracle recommended doc for LDOM creation i have T5220 … i have done my research but when i look into come documents they give me commands … i would like to know how physical resources are partition for logical domain …. while on my reasrch any issue will update ..

  4. Yogesh Raheja says:

    @Seema, I configured Ldoms 3 years back. Since then till now I havent got a chance again to play with Ldoms. But let me check if I have a config. doc with me. Else I will try my level best to search and provide you with the same.

  5. Michael Michael says:

    good one 

  6. Michael Michael says:

    ps -e -o vsz=MEMSIZE -o args=COMMAND

    • Ramdev Ramdev says:

      @Micheal – this is a good command, and i would like you to explain the purpose of this command on your own words, to our other folks.

  7. Andy says:

    Hi,

    I read your steps and, found it useful. Some of the commands you provided do not mention what to look for and, what do they mean. For example “ipcs -a”. can you explain in plain english what it is showing (I know it is shared memory. how can I use it to my advantage).

  8. Ramdev Ramdev says:

    Hi Andy, What I look there normally is SEGSZ  ( i.e. shared segment size ) and the owner and group of the segment. 

  9. sameer says:

    I found good information here. My issue is resolved. Thanks for sharing this information.

  10. Brad says:

    Thanks for above info. Is there a quicker way to get the total/cumulative private physical memory used by processes? I see such a number can be figured out by a combination of above commands, but ones like pmap -x /proc/* and mdb -k take a long time to complete. 

  11. Ramdev Ramdev says:

    HI Brad, sorry … I couldn’t think of any other quickest way of doing this.

  12. swamy says:

    swap -l showing
    swapfile dev swaplo blocks free
    /dev/dsk/c0t0d0s1 32,25 16 1946032 1946032

    here free means
    is it 1946032 this much is free ?
    i didn’t under stood

  13. Ramdev Ramdev says:

    Swamy, Yes. that actually indicates your system not yet using the swap file system.

  14. Madhukiran Attivilli says:

    which command, when run from local zone, provides physical memory of the system, as seen from global zone?

    • Madhukiran Attivilli says:

      In other words, how to determine whether a local zone has capped-memory configured, by running a command or application program from local zone itself, and not from global zone?

  1. September 16, 2015

    […] Read – Memory usage checking and troubleshooting […]

Leave a Reply to seema Cancel reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us