Hard drive has gone bad in DRBD - Revision history

Digimer: /* Migrate the VMs off of the Effected Node */

2011-07-11T15:03:44Z

Migrate the VMs off of the Effected Node

← Older revision		Revision as of 15:03, 11 July 2011
Line 333:		Line 333:

	= Migrate the VMs off of the Effected Node =		= Migrate the VMs off of the Effected Node =

			{{note\|1=Re-write this to use <span class="code">clusvcadm</span>.}}

	From either node, preferably the good node, ssh into in with X-Forwarding enabled and then start '''convirt'''. In our example, '''Node1''' is effected, so we will connect to '''Node2'''.		From either node, preferably the good node, ssh into in with X-Forwarding enabled and then start '''convirt'''. In our example, '''Node1''' is effected, so we will connect to '''Node2'''.

Digimer: /* Boot as Single User */

2010-09-10T19:04:34Z

Boot as Single User

@@ Line 689: / Line 689: @@
 Interrupt the Grub boot screen by pressing '''<esc>''' at the appropriate time. With the default kernel selected, press '''e''' to edit it. Append the word "''' single'''" to the end of the line (note the leading space).
-... '''''This tutorial jumped to the [[Lost a Node; How To Replace#Does the Node Boot]] section. I will continue writing this tomorrow.'''''
+=== Recovering the RAID Arrays Manually ===
-If you postponed configuring the replacement drive, do so now.
+In this case, the node failed to boot. Under the rescue DVD, I was able to rebuild the arrays manually. This hasn't solved the boot problem yet, but I'll get back to that tomorrow.
-[[#Prepare the Replacement Drive]]
+= RAID Rebuild in Rescue Mode =
 {{footer}}

Digimer: Created page with '{{howto_header}} So you've lost or are losing a hard drive in one of the cluster nodes. Steps needed: # Identify the failed drive. This example will use '''/dev/sda''' on '''No…'

2010-09-10T18:50:09Z

Created page with '{{howto_header}} So you've lost or are losing a hard drive in one of the cluster nodes. Steps needed: # Identify the failed drive. This example will use '''/dev/sda''' on '''No…'

Páàjì titun

{{howto_header}}

So you've lost or are losing a hard drive in one of the cluster nodes.

Steps needed:
# Identify the failed drive. This example will use '''/dev/sda''' on '''Node1'''.
# Migrate the hosted VMs to the ok node. This document will migrate from '''Node1''' '''to''' '''Node2'''.
# Break the RAID 1 mirror by removing the defective drive from the effected MD device. Here we will remove '''/dev/sda''' from the '''/dev/md0''' device.
# Power off the defective server, physically replace the effected drive and power the repaired server back on.
# Add the replaced '''/dev/sda''' into '''/dev/md0''' and begin the RAID 1 rebuild procedure.
# Migrate the effected virtual servers back onto the effected server.

= Identifying the Failed Drive =

== SMART Control ==

If it's not clear, check the drives' states using '''smartctl'''. For each questionable drive, run:

<source lang="bash">
smartctl -a /dev/sda
</source>

Replace '''sda''' with the drive you want to examine. You should see output like:

=== Good Drive ===

'''Good Drive''' (/dev/sdb):
<source lang="text">
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model: ST31500341AS
Serial Number: 9VS1XL54
Firmware Version: CC1H
User Capacity: 1,500,301,910,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Feb 3 12:35:54 2010 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 617) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 113 099 006 Pre-fail Always - 55840141
3 Spin_Up_Time 0x0003 100 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 101
5 Reallocated_Sector_Ct 0x0033 097 097 036 Pre-fail Always - 151
7 Seek_Error_Rate 0x000f 079 060 030 Pre-fail Always - 89466942
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3584
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 102
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Unknown_Attribute 0x0032 100 093 000 Old_age Always - 71
189 High_Fly_Writes 0x003a 066 066 000 Old_age Always - 34
190 Airflow_Temperature_Cel 0x0022 069 052 045 Old_age Always - 31 (Lifetime Min/Max 23/48)
194 Temperature_Celsius 0x0022 031 048 000 Old_age Always - 31 (0 20 0 0)
195 Hardware_ECC_Recovered 0x001a 032 024 000 Old_age Always - 55840141
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 51737176051199
241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 1591700793
242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 95914747

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
</source>

=== Bad Drive Output ===

'''Bad Drive''' (/dev/sda):
<source lang="text">
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model: ST31500341AS
Serial Number: 9VS1Q4Q3
Firmware Version: CC1H
User Capacity: 1,500,301,910,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Feb 3 12:37:38 2010 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 609) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 229324280
3 Spin_Up_Time 0x0003 100 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 95
5 Reallocated_Sector_Ct 0x0033 096 096 036 Pre-fail Always - 191
7 Seek_Error_Rate 0x000f 065 058 030 Pre-fail Always - 30092885574
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3541
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 95
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 076 076 000 Old_age Always - 24
188 Unknown_Attribute 0x0032 100 097 000 Old_age Always - 4295032890
189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 241
190 Airflow_Temperature_Cel 0x0022 067 054 045 Old_age Always - 33 (Lifetime Min/Max 23/46)
194 Temperature_Celsius 0x0022 033 046 000 Old_age Always - 33 (0 21 0 0)
195 Hardware_ECC_Recovered 0x001a 026 026 000 Old_age Always - 229324280
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 5
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 5
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 6
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 229084965637588
241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2647572671
242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 3166798893

SMART Error Log Version: 1
ATA Error Count: 24 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 24 occurred at disk power-on lifetime: 1785 hours (74 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 ff ff ff ef 00 19d+13:50:04.321 READ DMA EXT
27 00 00 00 00 00 e0 00 19d+13:50:04.291 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 02 19d+13:50:04.283 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 02 19d+13:50:04.238 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 e0 00 19d+13:50:04.115 READ NATIVE MAX ADDRESS EXT

Error 23 occurred at disk power-on lifetime: 1785 hours (74 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 ff ff ff ef 00 19d+13:50:01.385 READ DMA EXT
27 00 00 00 00 00 e0 00 19d+13:50:01.355 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 02 19d+13:50:01.347 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 02 19d+13:50:01.325 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 e0 00 19d+13:50:01.275 READ NATIVE MAX ADDRESS EXT

Error 22 occurred at disk power-on lifetime: 1785 hours (74 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 ff ff ff ef 00 19d+13:49:58.449 READ DMA EXT
27 00 00 00 00 00 e0 00 19d+13:49:58.419 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 02 19d+13:49:58.411 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 02 19d+13:49:58.366 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 e0 00 19d+13:49:58.247 READ NATIVE MAX ADDRESS EXT

Error 21 occurred at disk power-on lifetime: 1785 hours (74 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 ff ff ff ef 00 19d+13:49:55.488 READ DMA EXT
27 00 00 00 00 00 e0 00 19d+13:49:55.459 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 02 19d+13:49:55.451 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 02 19d+13:49:55.428 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 e0 00 19d+13:49:55.379 READ NATIVE MAX ADDRESS EXT

Error 20 occurred at disk power-on lifetime: 1785 hours (74 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 00 ff ff ff ef 00 19d+13:49:52.540 READ DMA EXT
27 00 00 00 00 00 e0 00 19d+13:49:52.511 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 02 19d+13:49:52.488 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 02 19d+13:49:52.463 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 e0 00 19d+13:49:52.420 READ NATIVE MAX ADDRESS EXT

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
</source>

You will notice in the above example that the drive's SMART status is OK, but that it has generated errors. These errors were sufficient to cause poor enough performance to trigger a fence against the effected server.

== Locating the Physical Drive ==

Once you know the logical block device path, '''/dev/sda''' here, you will need to locate it's physical position in the server. To do this, reference the docs for the effected server. This mapping should have been recorded when the node was built.

If it wasn't, first go kick the admin in the shins. Next, you will need to guess which is which. We can make an educated guess though because the above output includes the Serial Number ('''9VS1Q4Q3''' above). In fact, reference the Serial Number anyway, in case the OS changed things up at some point.

* '''Node2'''
* '''Node1'''

Using the serial number and the docs in '''Node1''', we know that the effected drive is in '''Tray 1'''.

= Migrate the VMs off of the Effected Node =

From either node, preferably the good node, ssh into in with X-Forwarding enabled and then start '''convirt'''. In our example, '''Node1''' is effected, so we will connect to '''Node2'''.

'''From your PC''':
<source lang="bash">
ssh root@an-node02.alteeve.com -X
</source>

'''Once on Node2''':
<source lang="bash">
convirt &
</source>

== Convirt ==

With '''convirt''' running, connect to each node by clicking on their names user the '''Servers''' item. This will prompt you to enter the Node's '''root''' password.

Then, for each VM on the effected node, do the following:

# Click to highlight the VM.
# Click on Migrate.
# Select the good node as the destination. This is '''Node2''' in this example.
# Confirm the live migration.
## '''Note''': The migration could take some time, so be sure to warn [[John]] or whomever might be using the effected VM prior to initiating the migration. No processes will need to be stopped, but to the user, the VM will appear to "freeze" for the duration of the migration.

== xm ==

If you do not want to use '''convirt''', you can use the '''xm''' command line tool to perform the migration procedure. The syntax is:

'''UNTESTED'''!
<source lang="bash">
#xm migrate [domain_id] [host] -l
xm migrate sql01 an-node02 -l
</source>

= Break the RAID Arrays =

We will tell the array that the drive '''/dev/sda''' is no longer usable.

== Confirming Your Partition Structure ==

Continuing our example, we will need to replace '''/dev/sda''' which has four partitions:
'''/dev/sda1'''; In ''/dev/md0'' - 250MB '/boot' partition
'''/dev/sda2'''; In ''/dev/md2'' - 2000MB '<swap>' partition
'''/dev/sda3'''; In ''/dev/md1'' - 20000MB '/' partition
'''/dev/sda5'''; In ''/dev/md3'' - 1+MB '<LVM>' partition

Confirm the above configuration by checking '''/proc/mdadm''':
<source lang="bash">
cat /proc/mdstat
</source>

Which should show:
<source lang="text">
md0 : active raid1 sdb1[1] sda1[0]
256896 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
2048192 blocks [2/2] [UU]

md3 : active raid1 sdb5[1] sda5[0]
1442347712 blocks [2/2] [UU]

md2 : active raid1 sdb2[1] sda2[0]
20482752 blocks [2/2] [UU]

unused devices: <none>
</source>

Depending on how your drive has failed, you may see one or more entries with: '''[_U]'''. If this is the case, the corresponding partition may be absent or, if there, will look like: '''sda2[2](F)'''.

== Failing The mdX Devices ==

Given these four partitions, we will need to run the following commands to remove their four partitions from their four respective '''/dev/mdX''' devices. Adapt this to your needs:
<source lang="bash">
mdadm --fail /dev/md0 /dev/sda1
mdadm --fail /dev/md1 /dev/sda3
mdadm --fail /dev/md2 /dev/sda2
mdadm --fail /dev/md3 /dev/sda5
</source>

Confirm that all the arrays are now broken by again running:
<source lang="bash">
cat /proc/mdstat
</source>

Which should show:
<source lang="text">
md0 : active raid1 sdb1[1] sda1[0](F)
256896 blocks [2/1] [_U]

md1 : active raid1 sdb3[1] sda3[0](F)
2048192 blocks [2/1] [_U]

md3 : active raid1 sdb5[1]
1442347712 blocks [2/1] [_U]

md2 : active raid1 sdb2[1] sda2[0](F)
20482752 blocks [2/1] [_U]

unused devices: <none>
</source>

In the above example, '''sda1''', '''sda2''' and '''sda3''' where failed my the '''mdadm --fail''' call while '''sda5''' has failed in such a way that it is not visible at all.

= Replace The Defective Drive =

With the knowledge of the defective drive's serial number and port in hand, power off the server.

'''DO NOT POWER IT BACK ON WHILE CONNECTED TO THE NETWORK!!!'''.

Under no circumstances do we want the cluster the re-assemble until '''''after''''' the defective drive has been replaced, re-added to the array and confirmed good!

== Prepare the Replacement Drive ==

I prefer to pre-partition the replacement drive on a separate workstation, but this can be done safely in the server itself once it's been installed. If you wish to delay partitioning until then, skip to the next step and then return to here once you reach: [[#Power the Node Back in SINGLE USER MODE]] below.

=== Ensure the New Drive is Blank ===

In my case, the replacement drive comes up on my workstation as '''/dev/sdb'''. If yours is different, simple replace '''sdb''' with your drive letter.

First, wipe the drive by writing 10000 blocks to the drive using '''dd''' from '''/dev/zero'''. Confirm the drive is where we expect it:

<source lang="bash">
fdisk -l
</source>
<source lang="text">
Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x13662e6d

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1216 9767488+ 7 HPFS/NTFS
/dev/sda2 1217 1340 996030 82 Linux swap / Solaris
/dev/sda3 1341 9729 67384642+ 83 Linux

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f0012

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 32 257008+ fd Linux raid autodetect
/dev/sdb2 33 287 2048287+ fd Linux raid autodetect
/dev/sdb3 288 182401 1462830705 fd Linux raid autodetect
</source>
In the case above, the replacement drive had three partitions on it. New drives will usually be blank. Also, I know that the '''/dev/sdb''' is the right drive by looking at their capacities. I could further confirm this using '''smartctl -a /dev/sdb''' if I had any doubt.

Now blank the drive:

<source lang="bash">
dd if=/dev/zero of=/dev/sdb count=10000
</source>
<source lang="text">
10000+0 records in
10000+0 records out
5120000 bytes (5.1 MB) copied, 1.40715 s, 3.6 MB/s
</source>

Now confirm that the drive is clear by re-running '''fdisk -l''':

<source lang="bash">
fdisk -l
</source>
<source lang="text">
Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x13662e6d

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1216 9767488+ 7 HPFS/NTFS
/dev/sda2 1217 1340 996030 82 Linux swap / Solaris
/dev/sda3 1341 9729 67384642+ 83 Linux

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
</source>

Perfect!

=== Create a Duplicate Partition Structure ===

Now we need to create the new partitions in such a way to identically mimic the old drive.

To do this, run '''fdisk''' against an okay drive and take node of the start and end cylinders for each good partition. These will be our guide when re-creating the partition scheme on the replacement drive.

Here is the output from a good drive on the surviving node:

<source lang="bash">
fdisk -l /dev/sda
</source>
<source lang="text">
Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 32 257008+ fd Linux raid autodetect
/dev/sda2 33 2582 20482875 fd Linux raid autodetect
/dev/sda3 2583 2837 2048287+ fd Linux raid autodetect
/dev/sda4 2838 182401 1442347830 5 Extended
/dev/sda5 2838 182401 1442347798+ fd Linux raid autodetect
</source>

The '''Start''' and '''End''' columns have the values we will need to set for the corresponding partitions.

Below is a fairly large dump from my terminal using '''fdisk'''. I prefer this method over graphical tools as I can be very precise this way. I'll assume here that you are familiar with the '''fdisk''' shell. If not, [http://tldp.org/HOWTO/Partition/fdisk_partitioning.html GO LEARN] before proceeding!

Start the '''fdisk''' shell:
<source lang="bash">
fdisk /dev/sdb
</source>

And build the partitions:
<source lang="text">
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x2d841b1b.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

The number of cylinders for this disk is set to 182401.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-182401, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-182401, default 182401): 32

Command (m for help): a
Partition number (1-4): 1

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (33-182401, default 33):
Using default value 33
Last cylinder, +cylinders or +size{K,M,G} (33-182401, default 182401): 2582

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (2583-182401, default 2583):
Using default value 2583
Last cylinder, +cylinders or +size{K,M,G} (2583-182401, default 182401): 2837

Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (2838-182401, default 2838):
Using default value 2838
Last cylinder, +cylinders or +size{K,M,G} (2838-182401, default 182401):
Using default value 182401

Command (m for help): n
First cylinder (2838-182401, default 2838):
Using default value 2838
Last cylinder, +cylinders or +size{K,M,G} (2838-182401, default 182401):
Using default value 182401

Command (m for help): t
Partition number (1-5): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 3
Hex code (type L to list codes): fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): fd
Changed system type of partition 5 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
</source>

What was done above, in short, was;
# Create the first partition as primary ending on cyl. 32
# Set the first partition to be bootable
# Created the remaining partitions as primary, primary, extended, primary.
# Changed the '''t'''ype of all partitions to '''fd''', '''''Linux raid autodetect'''''.

Confirm that the replacement drive now matches what the remaining good drive is partitioned as:
<source lang="bash">
fdisk -l /dev/sdb
</source>
<source lang="text">
/dev/sdb1 * 1 32 257008+ fd Linux raid autodetect
/dev/sdb2 33 2582 20482875 fd Linux raid autodetect
/dev/sdb3 2583 2837 2048287+ fd Linux raid autodetect
/dev/sdb4 2838 182401 1442347830 5 Extended
/dev/sdb5 2838 182401 1442347798+ fd Linux raid autodetect
</source>

Perfect! Now we can install it in place of the defective drive.

== Power Off and Unrack ==

<source lang="bash">
poweroff
</source>

Unrack the server and move it to a work area.

== Replace the Drive ==

Remove the drive you suspect to be the failed one. Confirm it is the right one by comparing the Serial Number reported by the '''smartctl -a /dev/sda''' call from step 1 (switch '''sda''' for your drive, of course). Once you have confirmed that the proper drive is in hand, remove it from it's carrier and set it aside to process for RMA later. Install the replacement drive and '''''UPDATE THE DOCS'''''!

== Power the Node Back in SINGLE USER MODE ==

With the server on your work-bench and '''not''' connected to any network, power on the server.

=== If You Don't Get the Grub Screen ===

If you replaced the first drive ('''sda'''), then there is a good chance the node will not boot but instead appear to hang with a black screen. This happens because the replacement drive is flagged bootable but has no data. To get around this, select the Boot Device BBS prompt. On most systems, including '''Node2''' and '''Node1''', this is done by pressing '''<F8>''' during the POST. Once you get the Boot Device list, select the second hard drive to boot from and the proceed with the next step.

=== Boot as Single User ===

Interrupt the Grub boot screen by pressing '''<esc>''' at the appropriate time. With the default kernel selected, press '''e''' to edit it. Append the word "''' single'''" to the end of the line (note the leading space).

... '''''This tutorial jumped to the [[Lost a Node; How To Replace#Does the Node Boot]] section. I will continue writing this tomorrow.'''''

If you postponed configuring the replacement drive, do so now.

[[#Prepare the Replacement Drive]]

{{footer}}