General Procedure for Kernel Patching in Solaris.

 

 

General Procedure for Kernel Patching in Solaris:

We are getting multiple requests for Solaris Kernel Patching procedure from many of your Gurkul Followers. Here we go with all the steps for all of them.

I am trying to present the simple patching procedure (when our disks are under Solaris Volume Manager Control, SVM). No zones, No clusters considered in this example. Basic procedure for Zones/Clusters will also remain the same but some additional outputs needs to be taken in that case.

 

1.) Download the bundle patch (Recommended Solaris Cluster Patch) from www.support.oracle.com (It need your login credentials for Oracle site).

This cluster patch will be in zip form, copy it on your server and unzip it where you have sufficient space.

2.) Take necessary backups of the server. (make one directory under /var/tmp and put the outputs in the server also).

Take backup of uname -a –>(will give your current Kernel Patch Level), df -k , metastat -p, metadb -i, netastat -nr, ifconfig -a, prtconf, prtdiag -v, metastat -ac (sol. 10), metastat -t, eeprom, echo | format etc.

[better make one script for youself and run that before proceeding with any activity on any server  –> I used to follow this as my best practice :)]

Make a habbit of Running Sun Explorer before proceeding with any of the activity in Solaris.

3.) Identify the rootdisk and rootmirror disk with metastat -p or df -k or metastat -ac (-ac only work in Solaris 10).

4.) Dont forget to take the backups of /etc/vfstab and /etc/system.

Note: Take outputs before detaching the mirror else you would have to take the backups seperately for your detached disk.

5.) Break the mirror i.e isolate one disk (usually rootmirror as best practise).

Firstly Delete the statedatabase replica for your rootmirror disk.

metadb -i

metadb -d /dev/rdsk/<slicename>

Proceed with the mirror detach for your root-mirror disk.

metastat -p

metadeatach <mirror> <submirror for second disk>

metaclear <submirror>

Repeat the same for all submirrors for rootmirror disk.

Check the outputs by metastat -ac, metastat -p, metastat -t and metadb -i

6.) Mount your detached disk (rootmirror) on mountpoint /mnt and edit /mnt/etc/vfstab & /mnt/etc/system.

mount /dev/dsk/<rootmirror disk with slice 0) /mnt

vi /mnt/etc/vfstab

change all md (SVM) entries with native devices eg:

/dev/md/dsk/d0 —> /dev/dsk/<rootmirror disk with corresponding slice>

&

/dev/md/rdsk/d0 —> /dev/rdsk/<rootmirror disk with corresponding slice>

eg: This is only for one slice, repeat same for all slices for your rootmirror disk.

yogesh-test#grep -i /dev/md/dsk/d0 /mnt/etc/vfstab
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no logging

yogesh-test#vi /mnt/etc/vfstab
/dev/dsk/c0t1d0s0 /dev/rdsk/c0t1d0s0 / ufs 1 no logging

—————————————————————-

grep -i md /mnt/etc/system (to check SVM related entries in system file which we need to comment out in order to bring mirrordisk out of SVM).

yogesh-test#grep -i md /mnt/etc/system
* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,0,blk ——————-> (comment it out with star)
* End MDD root info (do not edit)

After Changes the system file should look like as given below for rootmirror disk.

yogesh-test#grep -i md /mnt/etc/system
* Begin MDD root info (do not edit)
* rootdev:/pseudo/md@0:0,0,blk
* End MDD root info (do not edit)

7.) Un-mount your root-mirror disk.

umount /mnt

NOTE: 1.) Please keep in mind that all changes are applicable to rootmirror disk. Donot do any changes on other disk (i.e root-disk).
2.) It is recommended to run fsck on you rootmirror disk (slice 0, rootFS slice) so that it wont give any errors while booting.
3.) Before proceeding with the patching make sure that /var FS have enough free space as all logs will be written there. else you will get errors while patching.

8.) Now your Rootdisk is under SVM control and Rootmirror disk is under native raw Solaris disk. By doing this step we are making sure that in case if any thing goes against our action plan then we will be able to recover our server from other disk.

9.) We have to test the booting of our server from both the disks to make sure that the server will come up with both of the disks without any issues.

# init 0

At OK prompt

# devalias
# boot <rootmirror disk> ———-> (if no alias is there create it (using nvalias aliasname <diskpath>) as we have alread taken the backup of echo | format)

Once the server booted from rootmirror disk, do the post checks like df -k, ifconfig etc..

Now boot the server from rootdisk as well.

#init 0
#devalias
#boot <rootdisk>

Once booted from rootdisk, do the post check like df -k, ifconfig etc..

10.) We are sure at this point of time that our server is coming up from both the disks.

Here we go with the patching now, boot your server in single your mode with your root-disk.

# init 0
# devalias
# boot rootdisk -s
# mount your slice where the unzipped bundle patch file is placed
# Go-though the read-me file as well.
# and run the installpatchset script with passcode to start bundle patching

###########EXAMPLE: Bundle Patch Installation##############

# Run the installpatchset script.

# cd 10_Recommended

# ./installpatchset –s10patchset ———–(passcode required for script execution, which you can find from Cluster Patch ReadMe file))

Setup …..
Recommended OS Patchset Solaris 10 SPARC (2011.06.22)

The patch set will complete installation in this session. No intermediate
reboots are required.

Application of patches started : 2011.06.23 17:48:26

Applying 120900-04 ( 1 of 344) … skipped
Applying 121133-02 ( 2 of 344) … skipped
Applying 119254-81 ( 3 of 344) … skipped
.
.
.
Applying 146861-01 (342 of 344) … skipped
Applying 147182-01 (343 of 344) … success
Applying 147227-01 (344 of 344) … success

######################################################################################################

11.) It will take some hours (depending upon your previous kernel patch level). Once the patching gets completed, go with the reconfig reboot. (reboot — -r or init 0 and boot -r).

12.) Once the server will come up do the post-checks, compare the outputs which you have taken earlier to be pretty much sure about the success.

Check the latest kernel patch level with “uname -a” and you are done with the kernel patching :)

Note: If your server is having database & application.
1.) Before proceeding with the patching, make sure your application & databases are down.
2.) After successfully completion ask your database and application team to test/check there applications.
3.) Do not re-mirror the rootmirror disk immediately, wait for 5 to 7 days to notice any issues with the OS or application. If all is good then remirror the disks for redundancy.
4.) Shutdown —> Applications first –> DB second –> then proceed with OS activity.
Startup —> OS first –> DB second –> Application at last.
5.) There will be some difference in the procedure if zones are there (need to take backup of .xml files etc etc..) or the server is in cluster (have to take some serious precaution while doing patching on cluster nodes either VCS or Sun Cluster)

Special Note: 1.) Always change the value of auto-boot to false using eeprom command when ever you are procceding with any upgrade to avaoid your system to boot automatically with other disk/media. After completion of the activity change its value to True.

yogesh-test# eeprom | grep -i auto*
auto-boot?=true

yogesh-test# eeprom auto-boot?=false

yogesh-test# eeprom | grep -i auto*
auto-boot?=false

2.) I will come up with the roolback plan (If any activity failed and you have your one disk preserved) when I will post on SVM (Solaris Volume Manager). Because in roll-back you need to boot from the rootmirror disk and create all the SVM mirrors back on rootmirror disk from start and then remirror rootdisk back with rootmirror disk.

Hope this will help all Unix Champs who are waiting for this from many days. :)