Solaris 10 : Configuring Interprocess communication Parameters

Many system administrators, particularly those managing large database servers, have had difficulty tuning shared memory and semaphore parameters in /etc/system. Often, these difficulties are experienced before starting an application instance. The Solaris 10 Operating System addresses this issue by integrating the System V IPC tunables into a powerful and flexible framework for easy administration.


Solaris 10 raises the system defaults for these parameters so that most applications can run out of the box. Solaris 10 also obsoletes many of the older IPC tunables.

The Solaris 10 Operating System extends the resource control (rctl) facility, introduced in the Solaris 9 Operating System, to incorporate the IPC tunables. These new IPC rctls can now be modified within the Solaris Resource Management framework using the standard resource management commands and features. They can be administered on a per-process or per-project basis to provide fine grained control.

The Solaris 10 Operating System introduces the following new resource controls to replace the old /etc/system tunables:

                              Old             Old       New         New
Resource control          tunable         default   Max value   default
----------------------    -------------   -------   ---------   ----------
process.max-msg-qbytes    msginfo_msgmnb  4096      ULONG_MAX   65536
process.max-msg-messages  msginfo_msgtql  40        UINT_MAX    8192
process.max-sem-ops       seminfo_semopm  10        INT_MAX     512
process.max-sem-nsems     seminfo_semmsl  25        SHRT_MAX    512
project.max-shm-memory    shminfo_shmmax  0x800000  UINT64_MAX  1/4 physmem
project.max-shm-ids       shminfo_shmmni  100       2**24       128
project.max-msg-ids       msginfo_msgmni  50        2**24       128
project.max-sem-ids       seminfo_semmni  10        2**24       128

As the names suggest, these rctls are attributes of either processes or projects.

The following tunables are now obsolete in the Solaris 10 Operating System:

    Shared Memory        Semaphores             Message-Queue
--------------       --------------         ---------------
shminfo_shmseg       seminfo_semmns         msginfo_msgmax
shminfo_shmmin       seminfo_semvmx         msginfo_msgssz
shminfo_shmmax*      seminfo_semmnu         msginfo_msgmni*
shminfo_shmmni*      seminfo_semaem         msginfo_msgtql*
seminfo_semume         msginfo_msgmnb*
seminfo_semusz         msginfo_msgmap
seminfo_semmap         msginfo_msgseg
seminfo_semmsl*
seminfo_semopm*
seminfo_semmni*

values ending in * are technically obsolete, but if they are present in the /etc/system file then at boot the kernel will translate the values into global resource controls.

Configuring InterProcess Communication parameters

The /etc/system file is no longer the preferred way to set IPC tunables.  The Resource Manager framework provides several methods to administer resource controls:

1. Persistent settings using the project database and the project file.  The last field of a project entry in the /etc/project file is a semi-colon separated list of “attributes” of a project, which includes rctls. Attributes are specified in a “name=value” syntax, and with rctls the “value” field is a comma-separated 3-tuple called an “action clause”. An action clause consists of:

  • a privilege level that dictates who can manipulate the rctls
  • the threshold value of the rctl Threshold values in /etc/project must always be in the “base” unit for the rctl. For example, the base unit for project.max-shm-memory is a byte.
  • the action to be taken when the threshold is exceeded The default deny action denies any resource requests greater than the threshold.

Specifying a limit in the /etc/project file extends that limit to all processes belonging to the project. Here is an example of an /etc/project file containing IPC settings for both the oracle_oltp and oracle_dss projects:

oracle_oltp:100:Oracle OLTP:oracle:: 
project.max-shm-memory=(privileged,51539607552,deny);
process-max-sem-nsems=(privileged,300,deny)
oracle_dss:101:Oracle DSS:oracle:: 
project.max-shm-memory=(privileged,17179869184,deny)

This sets a limit of 48GB per shared memory segment and 300 semaphores for all processes in the oracle_oltp project, and a 16GB shared memory segment limit for all processes in the oracle_dss project.

Note: lines in the /etc/project file must be one line only. The previous examples have been broken into multiple lines for readability purposes only.

The recommended method for modifying the /etc/project file is to use the “proj*” commands, such as projadd for creating a project and projmod  for modifying a project.

2. Resource controls can be changed dynamically using prctl , assuming the user has the correct privileges.

The syntax of prctl  can, at first, seem complex. Some common usages are:

    # prctl -i process <pid>
    to list all resource controls for process <pid>
    # prctl -i project <project>
    to list all resource controls for project <project>
    # prctl -n <rctl> -i process <pid>
    lists only the resource control named <rctl> for process <pid>
    # prctl -n <rctl> -r -v <value> -i process <pid>
    replaces (-r) the named rctl setting with the value <value> for process <pid>

Unlike the /etc/project file, prctl allows the use of “scale factors” to simplify resource control management. Values specified with the -v switch can be “human readable” values such as 48GB instead of the 51539607552 bytes required in the project database. For example, assuming the preceding /etc/project file we can check the values for the Shared Memory setting for the oracle_dss project:

% prctl -n project.max-shm-memory -i project oracle_dss
project: 101: oracle_dss
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
privileged      16.0GB      -   deny                                 -
system          16.0EB    max   deny                                 -

Should we need to temporarily increase the setting to 24GB:

% prctl -n project.max-shm-memory -r -v 24GB -i project oracle_dss
% prctl -n project.max-shm-memory -i project oracle_dss
project: 101: oracle_dss
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
privileged      24.0GB      -   deny                                 -
system          16.0EB    max   deny

Resource controls can be expressed as in units of size (bytes), time (seconds), or as a count (integer). See the resource_controls(5) manpage for more information about available resource controls, their units and possible string modifiers which can be used in the prctl, projadd, and projmod commands.

3. Programmatically using the system calls getrctl(2), setrctl(2) and libc functions such as rctlblk_set_value. Applications can use these system calls and library functions to check and set their own resource controls. These methods are beyond the scope of this document.

# projadd -c “Oracle” ‘user.oracle’

Once the project is created, assign resource controls corresponding to the Solaris 9 values:

set semsys:seminfo_semmni=100

This value is less than the new default for project.max-sem-ids: 128. As such, we could artificially lower the value to 100, or simply ignore it. We chose the latter.

set semsys:seminfo_semmsl=256

Again, this value is less than the new default for process.max-sem-nsems: 512. In this case, however, we wish to limit Oracle to only 256 semaphores per process:

    # projmod -s -K "process.max-sem-nsems=(privileged,256,deny)" 'user.oracle'
set shmsys:shminfo_shmmax=4294967295

This system has 8GB of memory so this value (4GB, in bytes) is larger than new default for project.max-shm-memory (1/4 physmem, or 2GB on this system) and so an rctl must be created:

# projmod -s -K "project.max-shm-memory=(privileged,4GB,deny)" 'user.oracle'
set shmsys:shminfo_shmmni=256

Again, this value is larger than the new default for project.max-shm-ids so a rctl needs to be created:

# projmod -s -K “project.max-shm-ids=(privileged,256,deny)” ‘user.oracle’

There are no more lines from /etc/system that pertain to IPC, so we remove the old lines:

# /bin/cp /etc/system /etc/system.solaris9
# /bin/egrep -v "semsys:|shmsys:|msgsys:" /etc/system > /etc/system.solaris10
# /bin/mv /etc/system.solaris10 /etc/system

and our final /etc/project file:

# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
user.oracle:100:Oracle:::process.max-sem-nsems=(privileged,256,deny);project.max-shm-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Note, however, that these project rctls will not affect any processes run by a currently logged in userid ‘oracle’. We can check that any new oracle logins will receive the correct rctls by running:

# su oracle -c ‘sh -c “prctl -n process.max-sem-nsems -i process $$”‘

process: 3652: csh -c prctl -n process.max-sem-nsems -i process $$
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
process.max-sem-nsems
privileged        256       -   deny                                 -
system          32.8K     max   deny                                 -

Which verifies that the new limit of 256 semaphores is present for the oracle user. We could then verify the other limits are also correct individually as above, or run:

   # su oracle -c 'sh -c "prctl -i process $$"'

to check all the rctls simultaneously.

Conclusion

Most applications should receive sufficient System V IPC resources out of the  box with the new Solaris 10 Operating System defaults. Administrators also have powerful new mechanisms for constraining IPC resources among applications on large systems or tuning them for optimum performance.

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

1 Response

  1. September 16, 2015

    […] Read – Configuring Inter process communication Parameters […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us