Hands on Lab – Replacing Failed Disks from ZFS Pools ( RaidZ2 / RaidZ3 ) – Part2

 

 

In my earlier hands on lab post ( Hands on Lab – Replacing Failed Disks from ZFS Pools ( Simple / Mirrored / RaidZ )). I have explained the disk failure scenarios for ZPOOL  mirror and RaidZ volumes, and this post is just continuation to that post. And in this post I will showing the multiple disk failure scenarios for the RAIDZ2 and RAIDZ3, and the way to recover it.  

 

 

Testing RaidZ2 pool with two disk failures 

 Actual working Disk Configuration

root@gurkulunix3:~# echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0

Creating RaidZ2  Pool

root@gurkulunix3:~# zpool create -f rz2pool raidz2 c3t2d0 c3t3d0 c3t4d0

root@gurkulunix3:~# zpool status rz2pool

pool: rz2pool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rz2pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0

root@gurkulunix3:~# zfs create rz2pool/testfs

root@gurkulunix3:/downloads# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs

          <<== for total of 6 gb ( 3 x 2gb ) we got only 1.9G available for usage

 

After Simulating the Dual Disk Failure Manually

 

root@gurkulunix3:~# echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0

Specify disk (enter its number): Specify disk (enter its number):

             <<  == We lost 2 disks from the original configuration

root@gurkulunix3:~# zpool status -xv

pool: rz2pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open

<=== Pool is still available after two disk failure

After replacing the two failed disks

 

Disk Configuration

root@gurkulunix3:~# echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0 <<== new disk
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0 <<== new disk
/pci@0,0/pci8086,2829@d/disk@4,0

root@gurkulunix3:~# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
sata6/0::dsk/c3t0d0 disk connected configured ok
sata6/1::dsk/c3t1d0 cd/dvd connected configured ok
sata6/2::dsk/c3t2d0 disk connected configured ok
sata6/3::dsk/c3t3d0 disk connected configured ok
sata6/4::dsk/c3t4d0 disk connected configured ok

Zpool Status Showing the Volume as Degraded, and the volume is still available for usage

root@gurkulunix3:~# zpool status -xv
pool: rz2pool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scan: resilvered 16K in 0h0m with 0 errors on Sun Sep 16 12:58:35 2012
config:

NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 FAULTED 0 0 0 corrupted data
c3t4d0 FAULTED 0 0 0 corrupted data

 

Replace the Zpool failed disks once the physical disks replaced in the same location

root@gurkulunix3:~# zpool replace rz2pool c3t3d0
root@gurkulunix3:~# zpool status -xv
pool: rz2pool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-4J
scan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 13:13:32 2012
config:

NAME STATE READ WRITE CKSUM
rz2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 FAULTED 0 0 0 corrupted data

errors: No known data errors
root@gurkulunix3:~# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs
root@gurkulunix3:~# zpool replace rz2pool c3t4d0
root@gurkulunix3:~# zpool status -xv
all pools are healthy
root@gurkulunix3:~# zpool status rz2pool
pool: rz2pool
state: ONLINE
scan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 13:14:52 2012
config:

NAME STATE READ WRITE CKSUM
rz2pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0

errors: No known data errors
root@gurkulunix3:~# df -h /rz2pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz2pool/testfs 1.9G 209M 1.7G 11% /rz2pool/testfs
root@gurkulunix3:~#

 

Testing RaidZ3 Pool with total three disk failure

root@gurkulunix3:~# echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0
4. c3t5d0
/pci@0,0/pci8086,2829@d/disk@5,0
5. c3t6d0
/pci@0,0/pci8086,2829@d/disk@6,0
Specify disk (enter its number): Specify disk (enter its number):

 

root@gurkulunix3:~# zpool create -f rz3pool raidz3 c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t6d0

root@gurkulunix3:~# zpool status rz3pool

pool: rz3pool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rz3pool ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0

errors: No known data errors

root@gurkulunix3:~# zfs create rz3pool/testfs

After copying some sample data

root@gurkulunix3:/downloads# df -h /rz3pool/testfs/
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 30% /rz3pool/testfs

After manual Simulation of 3 disk Failure

root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot open

errors: No known data errors

root@gurkulunix3:~# df -h /rz3pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 31% /rz3pool/testfs

root@gurkulunix3:~# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
sata6/0::dsk/c3t0d0 disk connected configured ok
sata6/1::dsk/c3t1d0 cd/dvd connected configured ok
sata6/2::dsk/c3t2d0 disk connected configured ok
sata6/3::dsk/c3t3d0 disk connected configured ok
sata6/4 sata-port empty unconfigured ok
sata6/5 sata-port empty unconfigured ok
sata6/6 sata-port empty unconfigured ok
sata6/7 sata-port empty unconfigured ok

 root@gurkulunix3:~# echo|format

Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
Specify disk (enter its number): Specify disk (enter its number):

                      << == Missing disks c3t4d0 , c3t5d0 , c3t6d0 
root@gurkulunix3:~#

After replacing three Failed Physical disks in same location

root@gurkulunix3:~# echo|format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@0,0/pci8086,2829@d/disk@0,0
1. c3t2d0
/pci@0,0/pci8086,2829@d/disk@2,0
2. c3t3d0
/pci@0,0/pci8086,2829@d/disk@3,0
3. c3t4d0
/pci@0,0/pci8086,2829@d/disk@4,0
4. c3t5d0
/pci@0,0/pci8086,2829@d/disk@5,0
5. c3t6d0
/pci@0,0/pci8086,2829@d/disk@6,0
Specify disk (enter its number): Specify disk (enter its number):

root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 UNAVAIL 0 0 0 cannot open
c3t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot open

errors: No known data errors

root@gurkulunix3:~# zpool replace rz3pool c3t6d0
root@gurkulunix3:~# zpool replace rz3pool c3t4d0
root@gurkulunix3:~# zpool replace rz3pool c3t5d0

Once Disks were replaced Zpool ReSilvering will start

root@gurkulunix3:~# zpool status -xv
pool: rz3pool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Sep 16 15:44:22 2012
1.35G scanned out of 2.95G at 3.87M/s, 0h7m to go
822M resilvered, 45.66% done
config:

NAME STATE READ WRITE CKSUM
rz3pool DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
replacing-2 DEGRADED 0 0 0
c3t6d0/old FAULTED 0 0 0 corrupted data
c3t6d0 ONLINE 0 0 0 (resilvering)
replacing-3 DEGRADED 0 0 0
c3t4d0/old FAULTED 0 0 0 corrupted data
c3t4d0 ONLINE 0 0 0 (resilvering)
replacing-4 DEGRADED 0 0 0
c3t5d0/old FAULTED 0 0 0 corrupted data
c3t5d0 ONLINE 0 0 0 (resilvering)

errors: No known data errors

 

Status of rz3pool, once the resilvering completed

root@gurkulunix3:~# zpool status rz3pool
pool: rz3pool
state: ONLINE
scan: resilvered 1.77G in 0h12m with 0 errors on Sun Sep 16 15:56:35 2012
config:

NAME STATE READ WRITE CKSUM
rz3pool ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c3t6d0 ONLINE 0 0 0
c3t4d0 ONLINE 0 0 0
c3t5d0 ONLINE 0 0 0

errors: No known data errors

root@gurkulunix3:~# df -h /rz3pool/testfs
Filesystem Size Used Available Capacity Mounted on
rz3pool/testfs 3.9G 1.2G 2.7G 31% /rz3pool/testfs
root@gurkulunix3:~#

This Concludes the Disk Failure Scenarios of for Advanced Redundant volumes of ZFS.  W can avoid most of these above shown volume failures by configuring proper hot spare disks . I will be showing you the hot spare configuration and the way ZFS deal with disk failures du
ring spare disk availability, in my next post