Solaris Troubleshooting : Monitoring Kernel Memory Utilisation :
Other Learning Articles that you may like to read
Free Courses We Offer
Paid Training Courses we Offer
Memory usage checking and troubleshooting
First you’ll need to check how much Memory is used in Kernel and how much is used in User Memory. This is important to decide, what troubleshooting steps are required.
# echo “::memstat” | mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 151528 1183 10%
Anon 185037 1445 12%
Exec and libs 17092 133 1%
Page cache 281433 2198 19%
Free (cachelist) 633893 4952 42%
Free (freelist) 235470 1839 16%Total 1504453 11753
User memory usage : print out processes using most USER – memory
% prstat -s size # sorted by userland virtual memory consumption
% prstat -s rss # sorted by userland phyiscal memory consumption% prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
552 alfredm 341M 156M sleep 59 0 0:52:31 9.4% Xsun/1
760 alfredm 148M 131M sleep 49 0 4:58:00 8.4% mozilla-bin/9
1803 alfredm 87M 77M sleep 49 0 0:00:12 0.0% dtpad/1
682 alfredm 23M 15M sleep 49 0 0:04:37 0.1% xchat/1
9 root 9872K 8520K sleep 59 0 0:00:22 0.0% svc.configd/16
679 alfredm 11M 8184K sleep 59 0 0:01:26 0.1% dtwm/5
667 alfredm 11M 7144K sleep 49 0 0:00:52 0.0% dtsession/3
1861 alfredm 8688K 6256K sleep 49 0 0:00:00 0.0% dtcm/1
373 root 12M 6168K sleep 59 0 0:00:04 0.0% fmd/14
537 root 9296K 6056K sleep 59 0 0:00:04 0.0% snmpd/1
7 root 8648K 5912K sleep 59 0 0:00:08 0.0% svc.startd/12
User Memory Usage : shows Shared Memory and Semaphores:
% ipcs -a
IPC status from
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME
Message Queues:
q 0 0x55460272 -Rrw-rw—- root root root root 0 0 4194304 1390 18941 14:12:20 14:12:21 10:23:32
q 1 0x41460272 –rw-rw—- root root root root 0 0 4194304 5914 1390 8:03:34 8:03:34 10:23:39
q 2 0x4b460272 –rw-rw—- root root root root 0 0 4194304 0 0 no-entry no-entry 10:23:39
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x50000b3f –rw-r–r– root root root root 1 4 738 738 18:50:36 18:50:36 18:50:36
m 1 0x52574801 –rw-rw—- root oracle root oracle 35 1693450240 2049 26495 10:30:00 10:30:00 18:51:13
m 2 0x52574802 –rw-rw—- root oracle root oracle 35 1258291200 2049 26495 10:30:00 10:30:00 18:51:16
m 3 0x52594801 –rw-rw—- root oracle root oracle 12 241172480 2098 14328 7:58:33 7:58:33 18:51:27
m 4 0x52594802 –rw-rw—- root oracle root oracle 12 78643200 2098 14329 7:58:32 7:58:33 18:51:27
m 5 0x52584801 –rw-rw—- root oracle root oracle 13 125829120 2125 27492 1:36:12 1:36:12 18:51:34
m 6 0x52584802 –rw-rw—- root oracle root oracle 13 268435456 2125 27487 1:36:10 1:36:11 18:51:34
m 7 0x525a4801 –rw-rw—- root oracle root oracle 15 912261120 2160 27472 1:36:09 1:36:09 18:51:40
m 8 0x525a4802 –rw-rw—- root oracle root oracle 15 268435456 2160 27467 1:36:08 1:36:09 18:51:42
m 8201 0x4d2 –rw-rw-rw- root root root root 0 32008 1528 1543 10:26:03 10:26:04 10:25:53
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
Semaphores:
s 0 0x1 –ra-ra-ra- root root root root 1 16:17:35 18:50:33
s 1 0 –ra-ra—- root oracle root oracle 36 10:33:28 18:51:17
s 2 0 –ra-ra—- root oracle root oracle 13 10:33:28 18:51:27
s 3 0 –ra-ra—- root oracle root oracle 14 10:33:28 18:51:34
s 4 0 –ra-ra—- root oracle root oracle 16 10:33:27 18:51:42
s 5 0x4d2 –ra-ra-ra- root root root root 1 no-entry 10:25:53
s 6 0x4d3 –ra-ra-ra- root root root root 1 no-entry 10:25:53
User Memory Usage : lists User Memory usage of all processes ( except PID 0,2,3 )
# pmap -x /proc/* > /var/tmp/pmap-x
short list of total usage of these processes% egrep “[0-9]:|^total” /var/tmp/pmap-x
1: /sbin/init
total Kb 2336 2080 128 –
1006: rlogin cores4
total Kb 2216 1696 80 –
1007: rlogin cores4
total Kb 2216 1696 104 –
115: /usr/sbin/nscd
total Kb 4208 3784 1704 –
— snip —
User Memory Usage : check the usage of /tmp
% df -kl /tmp
Filesystem kbytes used avail capacity Mounted on
swap 1355552 2072 1353480 1% /tmpprint the biggest 10 files and dirs in /tmp
% du -akd /tmp | sort -n | tail -10 8
/tmp/hsperfdata_root 8
/tmp/jnlpgenerator-8 8
/tmp/jnlpgenerator-8-1 8
/tmp/jnlpgenerator-8-2 8
/tmp/nscopy.tmp 8
/tmp/nsmail.eml 8
/tmp/speckeysd.lock 16
/tmp/tt 48
/tmp/lucreate.man 800
/tmp/dtdbcache_:0 1088
/tmp
User Memory Usage : Overall Memory usage on system
% vmstat -p 3
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
1468048 508176 1 48 0 0 0 1 0 0 0 0 0 2 0 0
1355376 459272 0 9 0 0 0 0 0 0 0 0 0 0 0 0
1355376 459272 0 2 0 0 0 0 0 0 0 0 0 0 0 0scanrate should be 0 ( or near zero )
User Memory Usage : Swap usage
% swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t0d0s1 32,25 16 1946032 1946032% swap -s
total: 399400k bytes allocated + 18152k reserved = 417552k used, 1355480k available
Common kernel statistics : print out all kernel statistics in a parse’able format
% kstat -p > /var/tmp/kstat-p
kernel memory statistics:
% kstat -p -c kmem_cache
% kstat -p -m vmem
% kstat -p -c vmem
alternatively to kstat you can get kernel memory usage with kmastat prints kmastat buffers
# echo “::kmastat” | mdb -k > /var/tmp/kmastat
% more /var/tmp/kmastat
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
————————- —— —— —— ——— ——— —–
kmem_magazine_1 16 470 508 8192 470 0
kmem_magazine_3 32 970 1016 32768 1164 0
kmem_magazine_7 64 1690 1778 114688 1715 0
Look for the highest numbers in column “memory in use” and for any numbers higher than ‘0’ in column “alloc fail”
Hi Ram,
I used read your posts. I need some help to find out memory usage culprit.
I have 32GB physical memory and running on 2 local zones. one zone is using 11GB memory as heap size and second one idle. They is no limit set to zones. from global zone i can see only 4gb available free but only using 11GB. I want to find out where remaining memory is used.
echo ::memstat | mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 3015457 11779 36%
ZFS File Data 1242712 4854 15%
Anon 2880578 11252 34%
Exec and libs 30363 118 0%
Page cache 4875 19 0%
Free (cachelist) 30003 117 0%
Free (freelist) 1182401 4618 14%
Total 8386389 32759
Physical 8177494 31943
TOP output:
last pid: 16396; load avg: 1.62, 1.54, 1.54; up 6+07:28:50 15:12:19
157 processes: 155 sleeping, 2 on cpu
CPU states: 83.7% idle, 12.8% user, 3.5% kernel, 0.0% iowait, 0.0% swap
Memory: 32G phys mem, 3800M free mem, 2048M total swap, 2048M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
8666 7612 718 53 2 11G 10G cpu/0 539:39 14.30% java
16214 7612 1 59 0 2684K 1628K sleep 0:04 0.10% vmstat
2190 noaccess 18 59 0 151M 117M sleep 19:12 0.06% java
2151 noaccess 18 59 0 151M 117M sleep 18:40 0.05% java
15962 noaccess 18 59 0 219M 168M sleep 0:16 0.04% java
830 root 1 59 0 21M 18M sleep 11:31 0.03% Xorg
9885 root 12 59 0 37M 23M sleep 6:29 0.03% agtrep
9661 root 4 59 0 25M 11M sleep 3:57 0.03% opcacta
9533 root 29 59 0 21M 13M sleep 7:04 0.03% ovcd
9869 root 8 59 0 42M 26M sleep 13:09 0.02% opcmona
1519 root 1 59 0 19M 13M sleep 6:36 0.02% dtgreet
prstat -Z
430 root 2488K 1520K sleep 59 0 0:00:00 0.0% ttymon/1
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
1 60 12G 11G 33% 12:23:08 21% rws-prd-app-01
0 65 347M 326M 1.0% 1:21:59 0.1% global
2 36 229M 203M 0.6% 0:26:14 0.0% rws-prd-app-02
Can you please suggest what i missed and how to find out the culprint?
Hi Ramesh,
============================
echo ::memstat | mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 3015457 11779 36% <--- Kernel memory ZFS File Data 1242712 4854 15% <-- ZFS ARC ============================== i could see 11GB of memory eaten by kernel, you need to further check. what exactly taking more memory in kernel ( echo "::kmastat"|mdb -k ). And 4.5 GB was used by ZFS file data i.e the amount of memory currently allocated in all memory caches associated with ARC file data. check actual ZZFS ARC usage with " kstat -n arcstats -s size" to understand more about ARC issues you can read this nice article - http://dtrace.org/blogs/brendan/2012/01/09/activity-of-the-zfs-arc/