Hands on Lab – Replacing Failed Disks from ZFS Pools ( Simple / Mirrored / RaidZ )
In our earlier post ( To get Started with ZFS ) , yogesh discussed about various ZFS pool and file system Operations. In this post I will be demonstrating the redundancy capability for different ZFS pools and also the recovery procedure from the disk failure scenarios. I have performed this lab on Solaris 11 , these instructions are same for Solaris 10 though.
Quick Recap about ZFS Pools
1. Simple and Striped Pool ( Equivalent to Raid-0 and Data is Non redundant)
2. Mirrored Pool ( Equivalent to Raid-1)
3. Raidz pool ( Equivalent to Single Parity Raid 5 – Can with stand upto single disk failure)
4. Raidz-2 pool ( Equivalent to Dual Parity Raid 5 – Can withstand upto two disk failures)
5. Raidz-3 pool ( Equivalent to Triple Partity Raid 5 – Can with stand upto thre disk Failures)
2. Mirrored Pool ( Equivalent to Raid-1)
3. Raidz pool ( Equivalent to Single Parity Raid 5 – Can with stand upto single disk failure)
4. Raidz-2 pool ( Equivalent to Dual Parity Raid 5 – Can withstand upto two disk failures)
5. Raidz-3 pool ( Equivalent to Triple Partity Raid 5 – Can with stand upto thre disk Failures)
RAIDZ Configuration Requirements and Recommendations
A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.
- Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
- Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
- Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
- (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
- The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups
A general consideration is whether your goal is to maximum disk space or maximum performance.
- A RAIDZ configuration maximizes disk space and generally performs well when data is written and read in large chunks (128K or more).
- A RAIDZ-2 configuration offers better data availability, and performs similarly to RAIDZ. RAIDZ-2 has significantly better mean time to data loss (MTTDL) than either RAIDZ or 2-way mirrors.
- A RAIDZ-3 configuration maximizes disk space and offers excellent availability because it can withstand 3 disk failures.
- A mirrored configuration consumes more disk space but generally performs better with small random reads.
Disk Failure Scenario for Simple/Striped ZFS Non Redundant Pool
Disk Configuration:
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,03. c3t5d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@5,04. c3t6d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@6,0
Creating Simple ZFS Storage Pool
root@gurkulunix3:/dev/chassis# zpool create poolnr c3t2d0 c3t3d0‘poolnr’ successfully created, but with no redundancy; failure of onedevice will cause loss of the pool
root@gurkulunix3:/dev/chassis# zpool listNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTpoolnr 3.97G 92.5K 3.97G 0% 1.00x ONLINE –rpool 63.5G 5.21G 58.3G 8% 1.00x ONLINE –
Creating Sample Filesystem for new pool
root@gurkulunix3:/dev/chassis# zfs create poolnr/testfs
root@gurkulunix3:/downloads# zpool status poolnrpool: poolnrstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMpoolnr ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0errors: No known data errors
After Manual Simulation of the Disk ( c3t2d0) failure:
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <drive type unknown>/pci@0,0/pci8086,2829@d/disk@2,02. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,03. c3t5d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@5,04. c3t6d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@6,0
root@gurkulunix3:~# zpool status poolnrpool: poolnrstate: UNAVAILstatus: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool tocontinue functioning.action: Destroy and re-create the pool from a backup source. Manually marking the devicerepaired using ‘zpool clear’ may allow some data to be recovered.scan: none requestedconfig:NAME STATE READ WRITE CKSUMpoolnr UNAVAIL 0 0 0 insufficient replicasc3t2d0 FAULTED 1 0 0 too many errorsc3t6d0 ONLINE 0 0 0
From The Above Scenario it has been observed that Simple ZFS pool cannot withstand for any disk failures.
Disk Failure Scenario for Mirror Pool
Initial Disk Configuration
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
Create Mirror Pool
root@gurkulunix3:~# zpool create mpool mirror c3t4d0 c3t7d0root@gurkulunix3:~# zfs create mpool/mtestfs
>>> Copy Some Sample data to new file system
root@gurkulunix3:~# df -h|grep /mpool/mtestfsmpool 2.0G 32K 2.0G 1% /mpoolmpool/mtestfs 2.0G 31K 2.0G 1% /mpool/mtestfs
root@gurkulunix3:~# zpool status mpoolpool: mpoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMmpool ONLINE 0 0 0mirror-0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
After Manually simulating the Disk Failure
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,0Specify disk (enter its number): Specify disk (enter its number):
<== we lost the disk c3t7d0
Checking pool Status after Disk Failure
root@gurkulunix3:~# zpool status mpoolpool: mpoolstate: DEGRADEDstatus: One or more devices could not be opened. Sufficient replicas exist forthe pool to continue functioning in a degraded state.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-2Qscan: none requestedconfig:NAME STATE READ WRITE CKSUMmpool DEGRADED 0 0 0mirror-0 DEGRADED 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
After physically Replacing the Failed disk ( placing new disk in same location)
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 1022 alt 2 hd 128 sec 32>/pci@0,0/pci8086,2829@d/disk@7,0 << New Disk
>>> Label new disk with SMI Label ( A requirement to attach to ZFS pool)
root@gurkulunix3:~# format -L vtoc -d c3t7d0Searching for disks…doneselecting c3t7d0[disk formatted]c3t7d0 is labeled with VTOC successfully.
Replace the Failed Disk Component from the ZFS pool
root@gurkulunix3:~# zpool replace mpool c3t7d0root@gurkulunix3:~# zpool status -x mpoolpool ‘mpool’ is healthyroot@gurkulunix3:~# zpool status mpoolpool: mpoolstate: ONLINEscan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 10:41:21 2012config:NAME STATE READ WRITE CKSUMmpool ONLINE 0 0 0mirror-0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0 <<< Disk Onlineerrors: No known data errorsroot@gurkulunix3:~#
Single and Double Disk Failure Scenarios for ZFS Raid-Z Pool
Disk Configuration Available for new Raid-Z pool Creation
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
Creating New RaidZ Pool
root@gurkulunix3:~# zpool create rzpool raidz c3t2d0 c3t3d0 c3t4d0 c3t7d0invalid vdev specificationuse ‘-f’ to override the following errors:/dev/dsk/c3t2d0s0 is part of exported or potentially active ZFS pool poolnr. Please see zpool(1M).
==> Here we had a issue with one of the disk we selected for the pool, and the reason is the disk already used by some zpool earlier. But now the old zpool no longer available, and we want to reuse the disk for the new zpool.
==> We can solve the problem by two ways 1. Use -f option to override the configuration 2. Reinitialize the partition table for the disk ( Solaris X86 only).
==> In this example I have reinitialized the whole disk as solaris partition with below command
root@gurkulunix3:~# fdisk -B /dev/rdsk/c3t3d0p0
root@gurkulunix3:~# zpool create rzpool raidz c3t2d0 c3t3d0 c3t4d0 c3t7d0root@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
Create File system and Copy some test data to rzpool/r5testfs
root@gurkulunix3:~# zfs create rzpool/r5testfsroot@gurkulunix3:/downloads# df -h|grep testrzpool/r5testfs 5.8G 575M 5.3G 10% /rzpool/r5testfsroot@gurkulunix3:/downloads# cd /rzpool/r5testfs/root@gurkulunix3:/rzpool/r5testfs# ls -ltotal 1176598-rw-r–r– 1 root root 602057762 Sep 16 11:09 OLE6-U2-VM-Template.ziproot@gurkulunix3:/rzpool/r5testfs#
After Manual Simulation of the Disk failure ( i.e. c3t7d0)
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,0 <<== c3t7d0 missingSpecify disk (enter its number): Specify disk (enter its number):
Checking the zpool Status – it is in Degraded State
root@gurkulunix3:~# zpool status -x rzpoolpool: rzpoolstate: DEGRADEDstatus: One or more devices could not be opened. Sufficient replicas exist forthe pool to continue functioning in a degraded state.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-2Qscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
Checking if the File system is Still Accessible
root@gurkulunix3:~# df -h |grep testfsrzpool/r5testfs 5.8G 575M 5.3G 10% /rzpool/r5testfsroot@gurkulunix3:~# cd /rzpool/r5testfsroot@gurkulunix3:/rzpool/r5testfs# ls -ltotal 1176598-rw-r–r– 1 root root 602057762 Sep 16 11:09 OLE6-U2-VM-Template.ziproot@gurkulunix3:/rzpool/r5testfs#
After replacing the failed disk with new disk, in the same location
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool status -xpool: rzpoolstate: DEGRADEDstatus: One or more devices could not be used because the label is missing orinvalid. Sufficient replicas exist for the pool to continuefunctioning in a degraded state.action: Replace the device using ‘zpool replace’.see: http://www.sun.com/msg/ZFS-8000-4Jscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 FAULTED 0 0 0 corrupted data <<== this State Changed to Faulted just because the zpool could see the new disk but with no/corrupted dataerrors: No known data errors
Replacing the Failed Disk Component in the Zpool
root@gurkulunix3:~# zpool replace rzpool c3t7d0invalid vdev specificationuse ‘-f’ to override the following errors:/dev/dsk/c3t7d0s0 is part of exported or potentially active ZFS pool mpool. Please see zpool(1M).root@gurkulunix3:~# zpool replace -f rzpool c3t7d0 <<== using -f option to override above messageroot@gurkulunix3:~# zpool status -xall pools are healthyroot@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: resilvered 192M in 0h1m with 0 errors on Sun Sep 16 11:50:49 2012config:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
Two Disk Failures Scenario for RaidZ pool – And it Fails
Zpool Status Before Disk Failure
root@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: resilvered 192M in 0h1m with 0 errors on Sun Sep 16 11:50:49 2012config:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0
Disk Configuration After Simulating double disk failure
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,0 <== C3t4d0 & c3t7d0 missingSpecify disk (enter its number): Specify disk (enter its number):
Zpool Status after the Double Disk Failure
root@gurkulunix3:~# zpool status -xpool: rzpoolstate: UNAVAILstatus: One or more devices could not be opened. There are insufficientreplicas for the pool to continue functioning.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-3Cscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool UNAVAIL 0 0 0 insufficient replicasraidz1-0 UNAVAIL 0 0 0 insufficient replicasc3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 UNAVAIL 0 0 0 cannot openc3t7d0 UNAVAIL 0 0 0 cannot open
Conclusion: /rzpool/r5testfs filesystem not available for usage and the Zpool cannot be recovered from the current state
It’s too long here to post the RaidZ2 and RaidZ3 disk failure scenarios, I will be posting them as a separate post.
Excellent article RAM
All very interesting. However if you had set up your pools with spares your failure scenarios would be obselete.Â
Hi Dennis, very true. Having a hot spare is part of preventive configuration.
Hi Ram,
Good document however i feel that you should also unconfigure the device using cfgadm -c xxxxx.So that it will not be controlled by kernel.
Hi Gyan, I agree.