Hands on Lab – Replacing Failed Disks from ZFS Pools ( Simple / Mirrored / RaidZ )

2. Mirrored Pool ( Equivalent to Raid-1)
3. Raidz pool ( Equivalent to Single Parity Raid 5 – Can with stand upto single disk failure)
4. Raidz-2 pool ( Equivalent to Dual Parity Raid 5 – Can withstand upto two disk failures)
5. Raidz-3 pool ( Equivalent to Triple Partity Raid 5 – Can with stand upto thre disk Failures)
A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.
- Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
- Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
- Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
- (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
- The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups
- A RAIDZ configuration maximizes disk space and generally performs well when data is written and read in large chunks (128K or more).
- A RAIDZ-2 configuration offers better data availability, and performs similarly to RAIDZ. RAIDZ-2 has significantly better mean time to data loss (MTTDL) than either RAIDZ or 2-way mirrors.
- A RAIDZ-3 configuration maximizes disk space and offers excellent availability because it can withstand 3 disk failures.
- A mirrored configuration consumes more disk space but generally performs better with small random reads.
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,03. c3t5d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@5,04. c3t6d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@6,0
root@gurkulunix3:/dev/chassis# zpool create poolnr c3t2d0 c3t3d0‘poolnr’ successfully created, but with no redundancy; failure of onedevice will cause loss of the pool
root@gurkulunix3:/dev/chassis# zpool listNAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOTpoolnr 3.97G 92.5K 3.97G 0% 1.00x ONLINE –rpool 63.5G 5.21G 58.3G 8% 1.00x ONLINE –
root@gurkulunix3:/dev/chassis# zfs create poolnr/testfs
root@gurkulunix3:/downloads# zpool status poolnrpool: poolnrstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMpoolnr ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <drive type unknown>/pci@0,0/pci8086,2829@d/disk@2,02. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,03. c3t5d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@5,04. c3t6d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@6,0
root@gurkulunix3:~# zpool status poolnrpool: poolnrstate: UNAVAILstatus: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool tocontinue functioning.action: Destroy and re-create the pool from a backup source. Manually marking the devicerepaired using ‘zpool clear’ may allow some data to be recovered.scan: none requestedconfig:NAME STATE READ WRITE CKSUMpoolnr UNAVAIL 0 0 0 insufficient replicasc3t2d0 FAULTED 1 0 0 too many errorsc3t6d0 ONLINE 0 0 0
root@gurkulunix3:~# echo|format
Searching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool create mpool mirror c3t4d0 c3t7d0root@gurkulunix3:~# zfs create mpool/mtestfs
root@gurkulunix3:~# df -h|grep /mpool/mtestfsmpool 2.0G 32K 2.0G 1% /mpoolmpool/mtestfs 2.0G 31K 2.0G 1% /mpool/mtestfs
root@gurkulunix3:~# zpool status mpoolpool: mpoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMmpool ONLINE 0 0 0mirror-0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,0Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool status mpoolpool: mpoolstate: DEGRADEDstatus: One or more devices could not be opened. Sufficient replicas exist forthe pool to continue functioning in a degraded state.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-2Qscan: none requestedconfig:NAME STATE READ WRITE CKSUMmpool DEGRADED 0 0 0mirror-0 DEGRADED 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 1022 alt 2 hd 128 sec 32>/pci@0,0/pci8086,2829@d/disk@7,0 << New Disk
root@gurkulunix3:~# format -L vtoc -d c3t7d0Searching for disks…doneselecting c3t7d0[disk formatted]c3t7d0 is labeled with VTOC successfully.
root@gurkulunix3:~# zpool replace mpool c3t7d0root@gurkulunix3:~# zpool status -x mpoolpool ‘mpool’ is healthyroot@gurkulunix3:~# zpool status mpoolpool: mpoolstate: ONLINEscan: resilvered 210M in 0h0m with 0 errors on Sun Sep 16 10:41:21 2012config:NAME STATE READ WRITE CKSUMmpool ONLINE 0 0 0mirror-0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0 <<< Disk Onlineerrors: No known data errorsroot@gurkulunix3:~#
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool create rzpool raidz c3t2d0 c3t3d0 c3t4d0 c3t7d0invalid vdev specificationuse ‘-f’ to override the following errors:/dev/dsk/c3t2d0s0 is part of exported or potentially active ZFS pool poolnr. Please see zpool(1M).
root@gurkulunix3:~# fdisk -B /dev/rdsk/c3t3d0p0
root@gurkulunix3:~# zpool create rzpool raidz c3t2d0 c3t3d0 c3t4d0 c3t7d0root@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# zfs create rzpool/r5testfsroot@gurkulunix3:/downloads# df -h|grep testrzpool/r5testfs 5.8G 575M 5.3G 10% /rzpool/r5testfsroot@gurkulunix3:/downloads# cd /rzpool/r5testfs/root@gurkulunix3:/rzpool/r5testfs# ls -ltotal 1176598-rw-r–r– 1 root root 602057762 Sep 16 11:09 OLE6-U2-VM-Template.ziproot@gurkulunix3:/rzpool/r5testfs#
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,0 <<== c3t7d0 missingSpecify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool status -x rzpoolpool: rzpoolstate: DEGRADEDstatus: One or more devices could not be opened. Sufficient replicas exist forthe pool to continue functioning in a degraded state.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-2Qscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 UNAVAIL 0 0 0 cannot openerrors: No known data errors
root@gurkulunix3:~# df -h |grep testfsrzpool/r5testfs 5.8G 575M 5.3G 10% /rzpool/r5testfsroot@gurkulunix3:~# cd /rzpool/r5testfsroot@gurkulunix3:/rzpool/r5testfs# ls -ltotal 1176598-rw-r–r– 1 root root 602057762 Sep 16 11:09 OLE6-U2-VM-Template.ziproot@gurkulunix3:/rzpool/r5testfs#
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,03. c3t4d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@4,04. c3t7d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@7,0Specify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool status -xpool: rzpoolstate: DEGRADEDstatus: One or more devices could not be used because the label is missing orinvalid. Sufficient replicas exist for the pool to continuefunctioning in a degraded state.action: Replace the device using ‘zpool replace’.see: http://www.sun.com/msg/ZFS-8000-4Jscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool DEGRADED 0 0 0raidz1-0 DEGRADED 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 FAULTED 0 0 0 corrupted data <<== this State Changed to Faulted just because the zpool could see the new disk but with no/corrupted dataerrors: No known data errors
root@gurkulunix3:~# zpool replace rzpool c3t7d0invalid vdev specificationuse ‘-f’ to override the following errors:/dev/dsk/c3t7d0s0 is part of exported or potentially active ZFS pool mpool. Please see zpool(1M).root@gurkulunix3:~# zpool replace -f rzpool c3t7d0 <<== using -f option to override above messageroot@gurkulunix3:~# zpool status -xall pools are healthyroot@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: resilvered 192M in 0h1m with 0 errors on Sun Sep 16 11:50:49 2012config:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0errors: No known data errors
root@gurkulunix3:~# zpool status rzpoolpool: rzpoolstate: ONLINEscan: resilvered 192M in 0h1m with 0 errors on Sun Sep 16 11:50:49 2012config:NAME STATE READ WRITE CKSUMrzpool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0c3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 ONLINE 0 0 0c3t7d0 ONLINE 0 0 0
root@gurkulunix3:~# echo|formatSearching for disks…doneAVAILABLE DISK SELECTIONS:0. c3t0d0 <SUN ZFS 7120 HARDDISK-1.0 cyl 8351 alt 2 hd 255 sec 63>/pci@0,0/pci8086,2829@d/disk@0,01. c3t2d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@2,02. c3t3d0 <SUN ZFS 7120 HARDDISK-1.0-2.00GB>/pci@0,0/pci8086,2829@d/disk@3,0 <== C3t4d0 & c3t7d0 missingSpecify disk (enter its number): Specify disk (enter its number):
root@gurkulunix3:~# zpool status -xpool: rzpoolstate: UNAVAILstatus: One or more devices could not be opened. There are insufficientreplicas for the pool to continue functioning.action: Attach the missing device and online it using ‘zpool online’.see: http://www.sun.com/msg/ZFS-8000-3Cscan: none requestedconfig:NAME STATE READ WRITE CKSUMrzpool UNAVAIL 0 0 0 insufficient replicasraidz1-0 UNAVAIL 0 0 0 insufficient replicasc3t2d0 ONLINE 0 0 0c3t3d0 ONLINE 0 0 0c3t4d0 UNAVAIL 0 0 0 cannot openc3t7d0 UNAVAIL 0 0 0 cannot open
Excellent article RAM
All very interesting. However if you had set up your pools with spares your failure scenarios would be obselete.Â
Hi Dennis, very true. Having a hot spare is part of preventive configuration.
Hi Ram,
Good document however i feel that you should also unconfigure the device using cfgadm -c xxxxx.So that it will not be controlled by kernel.
Hi Gyan, I agree.