Solaris Performance: Identify memory usage by a specific process

Understanding how much memory individual processes are using, and what types of memory are in use is both a simple and difficult question to answer.

Simple because there are a number of tools that can report the memory usage per process, but difficult to explain, as processes can share memory in a number of ways;

This post purposed to tell the user a little about process memory usage, and explain how they can determine how much of what types of memory are is use by a given process.

Steps to Follow
The Simple Method to observe memory usage by process (Solaris 8 and later)
prstat for realtime monitoring,

prstat -s rss prstat -s size

Depending on the memory type the user is interested in. rss is a close estimate of the physical memory in use, whereas size is a close estimate of total virtual memory in use by the process (Physical + Swap).

For point in time viewing below command will show the highest memory users, sorted by RSS (Physical memory in use).

# ps -efly | sort +7n

S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD

S root 2 0 0 0 SY 0 0 Jul 18 0:00 pageout

S root 3 0 0 0 SY 0 0 Jul 18 318:02 fsflush

T root 0 0 0 0 SY 0 0 Jul 18 3:17 sched

R root 233 1 0 40 20 632 1060 Jul 18 0:10 /usr/lib/utmpd

S root 400 398 0 40 20 696 2084 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot

S root 401 398 0 40 20 696 2084 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot

S root 3112 193 0 50 20 768 5696 19:40:39 console 0:00 sort +7n

S root 228 7 0 40 20 788 1072 Jul 18 console 0:00 -sh

O root 3111 193 0 40 20 808 1260 19:40:39 console 0:00 ps -efly

S root 118 1 0 40 20 844 1272 Jul 18 0:00 /usr/lib/power/powerd

S root 219 7 0 40 20 908 1696 Jul 18 0:01 /usr/lib/saf/sac -t 300

S root 234 1 0 40 20 944 2236 Jul 18 0:00 /usr/sbin/cron

S root 1 0 0 40 20 1064 2004 Jul 18 0:10 /sbin/init

S root 236 219 0 40 20 1072 1800 Jul 18 0:01 /usr/lib/saf/ttymon

S root 184 1 0 40 20 1136 2212 Jul 18 0:00 /usr/sbin/keyserv

S root 355 1 0 40 20 1224 3332 Jul 18 0:00 /usr/lib/ssh/sshd

S root 457 1 0 40 20 1252 2036 Jul 18 0:02 /usr/lib/snmp/snmpdx -y -c /etc/snm

S root 104 1 0 40 20 1300 2136 Jul 18 0:00 /usr/lib/sysevent/syseventd

S daemon 178 1 0 40 20 1356 2512 Jul 18 0:03 /usr/sbin/rpcbind

S root 398 1 0 40 20 1388 2088 Jul 18 0:00 /usr/sadm/lib/smc/bin/smcboot

 

This is great for viewing things from a systemic perspective. The more detailed per process view

 What if we want to see more detail on a single process?

It depends on the OS and what you mean by memory:

 

See example output below:

$ /usr/proc/bin/pmap 1701

1701: /bin/ksh

00010000 184K read/exec /usr/bin/ksh

0004C000 8K read/write/exec /usr/bin/ksh

0004E000 40K read/write/exec [ heap ]

EF600000 592K read/exec /usr/lib/libc.so.1

EF6A2000 24K read/write/exec /usr/lib/libc.so.1

EF6A8000 8K read/write/exec [ anon ]

EF6C0000 16K read/exec /usr/platform/sun4u/lib/libc_psr.so.1

EF6D0000 16K read/exec /usr/lib/libmp.so.2

EF6E2000 8K read/write/exec /usr/lib/libmp.so.2

EF700000 448K read/exec /usr/lib/libnsl.so.1

EF77E000 32K read/write/exec /usr/lib/libnsl.so.1

EF786000 24K read/write/exec [ anon ]

EF790000 32K read/exec /usr/lib/libsocket.so.1

EF7A6000 8K read/write/exec /usr/lib/libsocket.so.1

EF7A8000 8K read/write/exec [ anon ]

EF7B0000 8K read/exec/shared /usr/lib/libdl.so.1

EF7C0000 112K read/exec /usr/lib/ld.so.1

EF7EA000 16K read/write/exec /usr/lib/ld.so.1

EFFFC000 16K read/write/exec [ stack ] total 1600K

 

In the above example, the first column is the address of the segment (Within that processes address space, NOT a systemwide virtual address), the second is the size of the mapping at that address, the third is the permissions and the fourth tells where the segment comes from (library, binary, etc.).

 

An even more exciting view of the processes memory usage is the pmap -x output.

# pmap -x $$

529: ksh -o vi

Address Kbytes Resident Shared Private Permissions Mapped File

00010000 200 200 200 – read/exec ksh

00052000 8 8 – 8 read/write/exec ksh

00054000 32 32 24 8 read/write/exec [ heap ]

FF180000 688 632 632 – read/exec libc.so.1

FF23C000 32 32 32 – read/write/exec libc.so.1

FF270000 16 16 16 – read/exec libc_psr.so.1

FF280000 568 536 536 – read/exec libnsl.so.1

FF31E000 40 40 40 – read/write/exec libnsl.so.1

FF328000 24 16 16 – read/write/exec libnsl.so.1

FF340000 16 16 16 – read/exec libmp.so.2

FF354000 8 8 8 – read/write/exec libmp.so.2

FF370000 8 8 – 8 read/write/exec [ anon ]

FF380000 40 40 40 – read/exec libsocket.so.1

FF39A000 8 8 8 – read/write/exec libsocket.so.1

FF3A0000 8 8 8 – read/exec libdl.so.1

FF3B0000 160 160 160 – read/exec ld.so.1

FF3E6000 16 16 – 16 read/write/exec ld.so.1

FFBEC000 16 16 8 8 read/write/exec [ stack ]

——– —— —— —— ——

total Kb 1888 1792 1744 48

 

This output includes a great deal of detail, including the size of the mapped segment, how much if it is RSS (Real memory), how much of that is shared, the sizes and permissions of the segment. It is the output above that can be ‘compared’ over time to see if a given process is growing, or ‘leaking’ memory.

 

Shared memory

 

Though the above output is excellent in terms of seeing how much of what type of memory is mapped per process, it’s still not the complete answer, as EVERY processes that maps the same shared resource will include that segment in it’s RSS.

 

This means that if, for example, 5 processes mapped the same 5GB segment, it might appear to the casual observer that there is actually 25GB of memory in use by those 5 processes, whereas in reality it’s the same 5GB mapped into 5 different address spaces!

 

Shared memory segments, and the number of processes attached can be viewed using

ipcs -ma

 

Which will return something that should resemble this:

$ ipcs -ma

IPC status from Server1 as of Monday April 30 16:17:47 SGT 2011 T

ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Shared Memory:

m 553648156 0x2116 –rw-rw—- ms157374 staff bobby staff 0 64528 8470 8470 11:21:35 11:41:05 11:21:35

 

The most interesting column from the above output  is SEGSZ, which is the size of the segment in bytes.

Shared memory makes things very hard to directly attribute memory usage to individual processes, as every process should indeed report that it is using that memory, however, which process should it be ‘attributed’ to? It is possible in many cases that the creator of the segment no longer exists.

The summary is that for a ‘quick and dirty’ view, prstat or ps are great alternatives. If you really need to understand which processes are using what memory, it will require a combination of pmap -x, ipcs and some reasonable Application knowledge to understand which processes would be attached to given shared memory segments.

An example case follows on a system with 4GB of memory, running Oracle

This can be observed in the startup messages from the database:

SQL> ORACLE instance started.

Total System Global Area 1291845632 bytes

Fixed Size 1978400 bytes

Variable Size 335548384 bytes

Database Buffers 939524096 bytes

Redo Buffers 14794752 bytes Database mounted. Database opened.

 

The SGA is located in shared memory so that multiple processes can attach to, map and use that segment in their own address space.

We can see this shared memory in the output of ipcs -mbop
# ipcs -mbpo
IPC status from Server1  as of Tue Oct 2 08:33:43 SGT2011
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:

m 1 0x8f430250 –rw-r—– oracle oper 35 1291853824 14195 14274

 

Note that the shared memory segment size is 1291853824 bytes, approximately 1.1GB.

 

Below command shows the multiple oracle processes that are attached to this segment.

# prstat -s rss

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

14203 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11

14207 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11

14205 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11

14209 oracle 1408M 1352M sleep 47 4 0:00:00 0.0% oracle/11

14221 oracle 1403M 1347M sleep 47 4 0:00:01 0.0% oracle/1

14211 oracle 1413M 1347M sleep 47 4 0:00:00 0.0% oracle/11

14213 oracle 1402M 1347M sleep 47 4 0:00:00 0.0% oracle/11

14219 oracle 1402M 1346M sleep 47 4 0:00:00 0.0% oracle/1

14197 oracle 1402M 1345M sleep 47 4 0:00:00 0.0% oracle/1

14272 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1

14227 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1

14215 oracle 1400M 1345M sleep 47 4 0:00:01 0.0% oracle/1

14217 oracle 1400M 1345M sleep 47 4 0:00:00 0.0% oracle/1

14225 oracle 1401M 1345M sleep 47 4 0:00:00 0.0% oracle/1

14223 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1

14265 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1

14274 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1

14199 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1

14201 oracle 1400M 1344M sleep 47 4 0:00:00 0.0% oracle/1

709 noaccess 293M 150M sleep 59 0 0:03:28 0.0% java/55

9791 root 208M 75M sleep 59 0 0:00:19 0.0% iscsitgtd/11

Total: 67 processes, 331 lwps, load averages: 0.03, 0.04, 0.02

 

Recall that this system has only 4GB of memory:

 $ prtdiag

System Configuration: Sun Microsystems sun4v SPARC Enterprise T2000

Memory size: 3968 Megabytes

<…>

# pmap -x 14203 | grep -v lib

14203: ora_dbw0_orcl

Address Kbytes RSS Anon Locked Mode Mapped File

0000000100000000 100128 99400 – – r-x– oracle

00000001062C6000 808 808 320 – rwx– oracle

0000000106390000 448 448 448 – rwx– [ heap ]

0000000106400000 8192 8192 8192 – rwx– [ heap ]

0000000380000000 1261576 1261576 – 1261576 rwxsR [ ism shmid=0x1 ]

<…>

 

Note above that the ism (Intimate Shared Memory) shmid (Shared Memory Identifier) is 0x1.

 This can be directly matched to the ipcs output seen previously, mapping the shmid to the ID column in the ipcs -mbop output.

 Thus, the processes can be directly associated with the shared segment, and memory usage better understood. If desired, the size of the shared memory segment can be subtracted from the processes total memory usage, to provide an idea of the individual processes memory usage, not including the shared memory segment(s).

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

2 Responses

  1. September 16, 2015

    […] Read – Identify memory usage by a specific process […]

  2. September 22, 2015

    […] Read – Identify memory usage by a specific process […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us