Linux: Rebuild RAID Just after Disk Failure
We start by listing the partition tables present on the server to find the new tricky disk
Type fdisk -l to windows 7 ultimate product key verify creation of partition on the second disk
[root@mail ~]# fdisk -lDisk /dev/hda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start Finish Blocks Id System
/dev/hda1 1 13 104391 fd Linux raid autodetect
/dev/hda2 14 6387 51199155 fd Linux raid autodetect
/dev/hda3 6388 6769 3068415 fd Linux raid autodetect
/dev/hda4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/hdc does not contain a valid partition table
Disk /dev/md1: 52.four GB, 52427816960 bytes
2 heads, 4 sectors/track, 12799760 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 does not include a valid partition table
Disk /dev/md3: 444.4 GB, 444428255232 bytes
2 heads, 4 sectors/track, 108502992 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md3 does not include a valid partition table
Disk /dev/md2: 3141 MB, 3141926912 bytes
2 heads, four sectors/track, 767072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md2 does not contain a valid partition table
Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, four sectors/track, 26080 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md0 does not include a valid partition table
Disk /dev/sda: 1000.two GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start off Finish Blocks Id System
/dev/sda1 1 121601 976760001 83 Linux
[root@mail ~]#
The
above output shows that there are two hard disks (hda and hdc).
Partitions are visible on hda nevertheless hdc will not contain a valid
partition table. Actually this is the replacement for the faulty disk.
To view the health on the Computer software RAID sort cat /proc/mdstat
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hda1[1]
104320 blocks [2/1] [_U]md2: active raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: active raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: active raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
[root@mail ~]#
Note
that [2/1] shows that you'll find two disks inside the array on the
other hand only 1 is active. The symbols [_U] mean that a single of your
two disks is missing from the array and have to be reconfigured.
Since
the new disk is /dev/hdc we want produce the partitions essential for
the RAID. Use fdisk to make the boot partition on the new disk.
[root@mail ~]# fdisk /dev/hdc
Device includes neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Adjustments will remain in memory only,
until you decide to write them. Following that, not surprisingly, the previous
content will not be recoverable.The number of cylinders for this disk is set to 60801.
There is absolutely nothing incorrect with that, but this can be larger than 1024,
and could in certain setups bring about issues with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will likely be corrected by w(rite)
Command (m for assist): p
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Begin Finish Blocks Id System
Command (m for help): n
Command action
e extended
p major partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-60801, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-60801, default 60801): +100M
Command (m for enable): p
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start Finish Blocks Id System
/dev/hdc1 1 13 104391 83 Linux
Command (m for enable): t
Selected partition 1
Hex code (type L to list codes): l
0 Empty 1e Hidden W95 FAT1 80 Old Minix be Solaris boot
1 FAT12 24 NEC DOS 81 Minix / old Lin bf Solaris
2 XENIX root 39 Strategy 9 82 Linux swap / So c1 DRDOS/sec (FAT-
3 XENIX usr 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT-
4 FAT16 <32M 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT-
5 Extended 41 PPC PReP Boot 85 Linux extended c7 Syrinx
6 FAT16 42 SFS 86 NTFS volume set da Non-FS data
7 HPFS/NTFS 4d QNX4.x 87 NTFS volume set db CP/M / CTOS /.
8 AIX 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility
9 AIX bootable 4f QNX4.x 3rd part 8e Linux LVM df BootIt
a OS/2 Boot Manag 50 OnTrack DM 93 Amoeba e1 DOS access
b W95 FAT32 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O
c W95 FAT32 (LBA) 52 CP/M 9f BSD/OS e4 SpeedStor
e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs
f W95 Ext'd (LBA) 54 OnTrackDM6 a5 FreeBSD ee EFI GPT
10 OPUS 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/
11 Hidden FAT12 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b
12 Compaq diagnost 5c Priam Edisk a8 Darwin UFS f1 SpeedStor
14 Hidden FAT16 <3 61 SpeedStor a9 NetBSD f4 SpeedStor
16 Hidden FAT16 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary
17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fd Linux raid auto
18 AST SmartSleep 65 Novell Netware b8 BSDI swap fe LANstep
1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid ff BBT
1c Hidden W95 FAT3 75 PC/IX
Hex code (type L to list codes): fd
Changed method sort of partition 1 to fd (Linux raid autodetect)
Command (m for support): p
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Get started End Blocks Id System
/dev/hdc1 1 13 104391 fd Linux raid autodetect
Command (m for enable):w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@mail ~]#
[root@mail ~]# fdisk -l
Disk /dev/hda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start Finish Blocks Id System
/dev/hda1 1 13 104391 fd Linux raid autodetect
/dev/hda2 14 6387 51199155 fd Linux raid autodetect
/dev/hda3 6388 6769 3068415 fd Linux raid autodetect
/dev/hda4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Commence Finish Blocks Id System
/dev/hdc1 1 13 104391 fd Linux raid autodetect
Disk /dev/md1: 52.4 GB, 52427816960 bytes
2 heads, 4 sectors/track, 12799760 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md1 does not include a valid partition table
Disk /dev/md3: 444.four GB, 444428255232 bytes
2 heads, four sectors/track, 108502992 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md3 does not contain a valid partition table
Disk /dev/md2: 3141 MB, 3141926912 bytes
2 heads, four sectors/track, 767072 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md2 doesn't include a valid partition table
Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, four sectors/track, 26080 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md0 does not include a valid partition table
Disk /dev/sda: 1000.two GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Begin End Blocks Id System
/dev/sda1 1 121601 976760001 83 Linux
[root@mail ~]#
Software
RAID requires disks in the array to be partitioned identically. You can
try to do this manually calculating the exact number of cylinders
expected for each partition to match their counterparts on the other
disk. That is highly discouraged. Instead we use the sfdisk command.
[root@mail ~]# sfdisk -d /dev/hda
# partition table of /dev/hda
unit: sectors/dev/hda1: start= 63, 208782, Id=fd
/dev/hda2: start= 208845, Id=fd
/dev/hda3: start=102607155, 6136830, Id=fd
/dev/hda4: start=108743985, Id=fd
[root@mail ~]#
The output above displays the partition sizes on the main disk
In order to make identical partitions on the secondary disk we type the following
[root@mail ~]# sfdisk -d /dev/hda | sfdisk /dev/hdc
Checking that no-one is using this disk right now...
OKDisk /dev/hdc: 60801 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Get started Finish #cyls #blocks Id System
/dev/hdc1 0+ 12 13- 104391 fd Linux raid autodetect
/dev/hdc2 0 - 0 0 0 Empty
/dev/hdc3 0 - 0 0 0 Empty
/dev/hdc4 0 - 0 0 0 Empty
New situation:
Units = sectors of 512 bytes, counting from 0
Device Boot Commence End #sectors Id System
/dev/hdc1 63 208844 208782 fd Linux raid autodetect
/dev/hdc2 208845 102607154 102398310 fd Linux raid autodetect
/dev/hdc3 102607155 108743984 6136830 fd Linux raid autodetect
/dev/hdc4 108743985 976768064 868024080 fd Linux raid autodetect
Warning: no key partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table
Re-reading the partition table...
If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the initially 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@mail ~]#
Typing fdisk -l once again displays that the partitions on both disks are in truth identical
[root@mail ~]# fdisk -lDisk /dev/hda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Get started End Blocks Id System
/dev/hda1 1 13 104391 fd Linux raid autodetect
/dev/hda2 14 6387 51199155 fd Linux raid autodetect
/dev/hda3 6388 6769 3068415 fd Linux raid autodetect
/dev/hda4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Begin Finish Blocks Id System
/dev/hdc1 1 13 104391 fd Linux raid autodetect
/dev/hdc2 14 6387 51199155 fd Linux raid autodetect
/dev/hdc3 6388 6769 3068415 fd Linux raid autodetect
/dev/hdc4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/md1: 52.four GB, 52427816960 bytes
2 heads, 4 sectors/track, 12799760 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 does not include a valid partition table
Disk /dev/md3: 444.4 GB, 444428255232 bytes
2 heads, four sectors/track, 108502992 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md3 doesn't contain a valid partition table
Disk /dev/md2: 3141 MB, 3141926912 bytes
2 heads, 4 sectors/track, 767072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md2 doesn't include a valid partition table
Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, four sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md0 does not include a valid partition table
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start out Finish Blocks Id System
/dev/sda1 1 121601 976760001 83 Linux
[root@mail ~]#
Now that the partitions are identical we can re-sync the information across the RAID configuration
Type mdadm -a /dev/md0 /dev/hdc1 to add /dev/hdc1 partition to the /dev/md0 RAID partition.
[root@mail ~]# mdadm -a /dev/md0 /dev/hdc1
mdadm: added /dev/hdc1
[root@mail ~]#
Type cat /proc/mdstat to view the re-syncing process
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[2] hda1[1]
104320 blocks [2/1] [_U]
[=>... ] recovery = 7.8% (8512/104320) finish=0.7min speed=2128K/secmd2: active raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: active raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: active raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[2] hda1[1]
104320 blocks [2/1] [_U]
[=======>... ] recovery = 37.2% (39808/104320) finish=0.5min speed=1809K/sec
md2: active raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: active raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: active raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
The
below shows how the output should be when the re-syncing process has
ended. Note the "[UU]". This confirms that both disks are active.
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[0] hda1[1]
104320 blocks [2/2] [UU]md2: active raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: active raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: active raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
[root@mail ~]#
The same process has to be performed on the other partitions
[root@mail ~]# mdadm -a /dev/md1 /dev/hdc2
mdadm: added /dev/hdc2
[root@mail ~]# mdadm -a /dev/md2 /dev/hdc3
mdadm: added /dev/hdc3
[root@mail ~]# mdadm -a /dev/md3 /dev/hdc4
mdadm: added /dev/hdc4
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[0] hda1[1]
104320 blocks [2/2] [UU]md2: active raid1 hdc3[2] hda3[1]
3068288 blocks [2/1] [_U]
resync=DELAYED
md3: active raid1 hdc4[2] hda4[1]
434011968 blocks [2/1] [_U]
resync=DELAYED
md1: active raid1 hdc2[2] hda2[1]
51199040 blocks [2/1] [_U]
[>... ] recovery = 0.0% (37376/51199040) finish=501.5min speed=1698K/sec
unused devices:
[root@mail ~]#
Finally we need to have to install GRUB on the second disk's MBR.
[root@mail ~]# grub
GNU GRUB version 0.97 (640K lower / 3072K upper memory)[ Minimal BASH-like line editing is supported. For the initial word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
grub> find /grub/grub.conf
(hd0,0)
(hd1,0)
grub> root (hd1,0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd1)"... 15 sectors are embedded.
succeeded
Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
grub> quit
Filesystem
windows 7 ultimate upgrade key sort is ext2fs, partition kind 0xfd
grub> setup (hd1)