Hands on Lab – Replacing Failed Disks from ZFS Pools ( RaidZ2 / RaidZ3 ) – Part2
In my earlier hands on lab post ( Hands on Lab – Replacing Failed Disks from ZFS Pools ( Simple / Mirrored / RaidZ )). I have explained the disk failure scenarios for ZPOOL mirror and RaidZ volumes, and this post is just continuation to that post. And in this post I will showing the multiple disk failure scenarios for the RAIDZ2 and RAIDZ3, and the way to recover it.
Testing RaidZ2 pool with two disk failures
Actual working Disk Configuration
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0
Creating RaidZ2 Pool
root@gurkulunix3:~# zpool create -f rz2pool raidz2 c3t2d0 c3t3d0 c3t4d0
root@gurkulunix3:~# zpool status rz2pool
pool: rz2pool
state: ONLINE
scan: none requested
config:NAME STATE READ WRITE CKSUM
rz2pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0root@gurkulunix3:~# zfs create rz2pool/testfs
root@gurkulunix3:/downloads# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs<<== for total of 6 gb ( 3 x 2gb ) we got only 1.9G available for usage
After Simulating the Dual Disk Failure Manually
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0Specify disk (enter its number): Specify disk (enter its number):
<< == We lost 2 disks from the original configuration
root@gurkulunix3:~# zpool status -xv
pool: rz2pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open
<=== Pool is still available after two disk failure
After replacing the two failed disks
Disk Configuration
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0 <<== new disk
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0 <<== new disk
/pci@0,0/pci8086,2829@d/disk@4,0root@gurkulunix3:~# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
sata6/0::dsk/c3t0d0 disk connected configured ok
sata6/1::dsk/c3t1d0 cd/dvd connected configured ok
sata6/2::dsk/c3t2d0 disk connected configured ok
sata6/3::dsk/c3t3d0 disk connected configured ok
sata6/4::dsk/c3t4d0 disk connected configured ok
Zpool Status Showing the Volume as Degraded, and the volume is still available for usage
root@gurkulunix3:~# zpool status -xv
pool: rz2pool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scan: resilvered 16K in 0h0m with 0 errors on Sun Sep 16 12:58:35 2012
config:NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 FAULTED 0 0 0 corrupted data
c3t4d0 FAULTED 0 0 0 corrupted data
Replace the Zpool failed disks once the physical disks replaced in the same location
root@gurkulunix3:~# zpool replace rz2pool c3t3d0
root@gurkulunix3:~# zpool status -xv
pool: rz2pool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 13:13:32 2012
config:NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 FAULTED 0 0 0 corrupted dataerrors: No known data errors
root@gurkulunix3:~# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs
root@gurkulunix3:~# zpool replace rz2pool c3t4d0
root@gurkulunix3:~# zpool status -xv
all pools are healthy
root@gurkulunix3:~# zpool status rz2pool
pool: rz2pool
state: ONLINE
scan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 13:14:52 2012
config:NAME STATE READ WRITE CKSUM
rz2pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs
root@gurkulunix3:~#
Testing RaidZ3 Pool with total three disk failure
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0
4. c3t5d0
/pci@0,0/pci8086,2829@d/disk@5,0
5. c3t6d0
/pci@0,0/pci8086,2829@d/disk@6,0
Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool create -f rz3pool raidz3 c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t6d0
root@gurkulunix3:~# zpool status rz3pool
pool: rz3pool
state: ONLINE
scan: none requested
config:NAME STATE READ WRITE CKSUM
rz3pool ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# zfs create rz3pool/testfs
After copying some sample data
root@gurkulunix3:/downloads# df -h /rz3pool/testfs/
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 30% /rz3pool/testfs
After manual Simulation of 3 disk Failure
root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
root@gurkulunix3:~# df -h /rz3pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 31% /rz3pool/testfsroot@gurkulunix3:~# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
sata6/0::dsk/c3t0d0 disk connected configured ok
sata6/1::dsk/c3t1d0 cd/dvd connected configured ok
sata6/2::dsk/c3t2d0 disk connected configured ok
sata6/3::dsk/c3t3d0 disk connected configured ok
sata6/4 sata-port empty unconfigured ok
sata6/5 sata-port empty unconfigured ok
sata6/6 sata-port empty unconfigured ok
sata6/7 sata-port empty unconfigured ok
root@gurkulunix3:~# echo|format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
Specify disk (enter its number): Specify disk (enter its number):<< == Missing disks c3t4d0 , c3t5d0 , c3t6d0
root@gurkulunix3:~#
After replacing three Failed Physical disks in same location
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0
4. c3t5d0
/pci@0,0/pci8086,2829@d/disk@5,0
5. c3t6d0
/pci@0,0/pci8086,2829@d/disk@6,0
Specify disk (enter its number): Specify disk (enter its number):root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
root@gurkulunix3:~# zpool replace rz3pool c3t6d0
root@gurkulunix3:~# zpool replace rz3pool c3t4d0
root@gurkulunix3:~# zpool replace rz3pool c3t5d0
Once Disks were replaced Zpool ReSilvering will start
root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Sep 16 15:44:22 2012
1.35G scanned out of 2.95G at 3.87M/s, 0h7m to go
822M resilvered, 45.66% done
config:NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
replacing-2 DEGRADED 0 0 0
c3t6d0/old FAULTED 0 0 0 corrupted data
c3t6d0 ONLINE 0 0 0 (resilvering)
replacing-3 DEGRADED 0 0 0
c3t4d0/old FAULTED 0 0 0 corrupted data
c3t4d0 ONLINE 0 0 0 (resilvering)
replacing-4 DEGRADED 0 0 0
c3t5d0/old FAULTED 0 0 0 corrupted data
c3t5d0 ONLINE 0 0 0 (resilvering)errors: No known data errors
Status of rz3pool, once the resilvering completed
root@gurkulunix3:~# zpool status rz3pool
pool: rz3pool
state: ONLINE
scan: resilvered 1.77G in 0h12m with 0 errors on Sun Sep 16 15:56:35 2012
config:NAME STATE READ WRITE CKSUM
rz3pool ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# df -h /rz3pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 31% /rz3pool/testfs
root@gurkulunix3:~#
This Concludes the Disk Failure Scenarios of for Advanced Redundant volumes of ZFS. W can avoid most of these above shown volume failures by configuring proper hot spare disks . I will be showing you the hot spare configuration and the way ZFS deal with disk failures du
ring spare disk availability, in my next post
Hi,
Good Day,
Could you please explain step by step to create zones on Solaris 11.
I am able to create zones on solars11 but unable to create network portion on zones.
Thanks in Advance.
Satya.
Very soon we will start the articles for solaris 11.
Hello there, You’ve done an excellent job. I will definitely digg it and individually suggest to my friends. I am sure they will be benefited from this web site.
Thank for your support.
Thanks Karren, for  your feedback..