Linux Software Raid

Howto create, manage, maintain, grow, and finally destroy your linux software raid.

Create new raid

Create the partition tables on one disk (here sda) and copy the disk layout to the second one (here sdb)
(assuming you have two identical SATA-Disks: sda and sdb)

$ sfdisk -d /dev/sda | sfdisk /dev/sdb

Create a new software raid 1

mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/sda5 /dev/sdb5

If you want to create serveral raid devices at once try this:

for i in 6 7 8; do mdadm --create --verbose /dev/md$i --level=1 --raid-devices=2 /dev/sda$i /dev/sdb$i; done

Adjust sync rate

First, check the current values (K/sec):

cat /proc/sys/dev/raid/speed_limit_max
cat /proc/sys/dev/raid/speed_limit_min

Obviously we want to increase speed_limit_min

echo 50000 > /proc/sys/dev/raid/speed_limit_min

This would boost the rate to ~50 MB/sec

Saving your setup to config file

Once you set up your array it would be helpful (but not necessary) to create a /etc/mdadm/mdadm.conf

mdadm --detail --scan > /etc/mdadm/mdadm.conf

you will get something like this:

ARRAY /dev/md1 level=raid1 num-devices=2 UUID=727232b4:09441cd2:2c40d2be:4d07af58
ARRAY /dev/md5 level=raid1 num-devices=2 UUID=aa0be2f1:35f327ba:8231611c:30c1cd8d
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=9f9978f4:d7475f57:c42e1ed2:45b4665c
ARRAY /dev/md7 level=raid1 num-devices=2 UUID=1fd86d6c:e32f550b:870fec6e:7f3724f5
ARRAY /dev/md8 level=raid1 num-devices=2 UUID=a90171e7:98e53b47:0a4aa1b7:3561e627

Every device used within your array has to be specified like this:

DEVICE /dev/sda* /dev/sdb*

Work with your raid

Stop an unmounted raid

mdadm --stop /dev/md2

Show details:

mdadm --detail /dev/md0


        Version : 00.90.03
  Creation Time : Tue Nov 18 08:41:45 2008
     Raid Level : raid1
     Array Size : 128384 (125.40 MiB 131.47 MB)
    Device Size : 128384 (125.40 MiB 131.47 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Nov 18 14:56:49 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 4b2dce7e:0343afb2:26449295:6e62cb31
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

Status in /proc/mdstat
You can see the progress of a (re)sync of you raid with the following command:

cat /proc/mdstat
watch -n 3 cat /proc/mdstat


Personalities : [raid1]
md1 : active raid1 sda3[0] sdb3[1]
      158754240 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      128384 blocks [2/2] [UU]

unused devices: <none>

Replace dead harddrive and resync your raid

Assumption: one of your drives died (raid 1), you replaced the drive while your server was powerd off. Now your server is running in degraded mode and the new drive is empty and waiting to be integrated into your raid:

Copy the partition table layout to the new drive (be sure, to specify the correct drives or you will end up in desaster!)

  • /dev/sda - good drive with your data
  • /dev/sdb - new drive, completely empty

Let's have a look at the raid arry:

 mdadm --detail --scan
ARRAY /dev/md1 level=raid1 num-devices=2
ARRAY /dev/md5 level=raid1 num-devices=2
ARRAY /dev/md6 level=raid1 num-devices=2
ARRAY /dev/md7 level=raid1 num-devices=2
ARRAY /dev/md8 level=raid1 num-devices=2

OK, /dev/sdb is missing. We copy the partition table from sda to sdb:

sfdisk -d /dev/sda | sfdisk /dev/sdb

And now we tell the raid array to sync the new partitions of sdb into the array:

for i in 1 5 6 7 8; do mdadm --add --verbose /dev/md$i /dev/sda$i /dev/sdb$i; done

Now the raid will sync the data back to sdb, where it belongs.

Repair your raid

If one of your raid sets run out of sync you can try the following.

As you can see, there is only one mirror online. sdb2 has been kicked of the array.

md0 : active raid1 sda2[0] 
      96320 blocks [2/2] [U_]

Well, what shall we do?

Ensure sdb2 has really been kicked off the array:

mdadm /dev/md0 –-fail /dev/sdb2 -–remove /dev/sdb2

Then re-add sdb2 to the array:

mdadm /dev/md0 –-add /dev/sdb2

This will start the initialisation of the raid. If there is no hardware defect we should soon be in business again.

Grow your raid

To grow your raid by adding an additional harddrive, see

Don't do this without a backup of your data!

The steps in short (growing a 3 disk raid5 to a 4 disk raid5):

# grow the raid
mdadm --add /dev/md0 /dev/sdd3
mdadm --grow /dev/md0 --raid-devices=4
# modify /etc/mdadm.conf to reflect the new layout (num-devices=4)
vim /etc/mdadm.conf
# if you use LVM on top of raid, grow the physical volume:
pvdisplay # to see the situation before
pvresize /dev/md0
pvdisplay # did we grow?
# grow the lv
lvextend -L +50G -n your_lv volume_group
# finally grow filesystem
# check the filesystem first
fsck /dev/mapper/volume_group-your_lv
# grow ext3 filesystem:
resize2fs /dev/mapper/volume_group-your_lv

Raid 5

Create a RAID5 with 4 disks (sda1 through sdd1)

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 --chunk=128 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

(Re)activate moved raid array

I moved an existing raid array of 4 disks to another host. After booting the array wasn't present. This is what I did:

mdadm --examine --scan >> /etc/mdadm/mdadm.conf
# check for privious found arrays which are not present any more
vim /etc/mdadm/mdadm.conf
# activate raid
mdadm --assemble --scan
# scan for LVM
# activate lv sitting on md device:
lvchange -a y /dev/vg02/dta0
# done, lv can be mounted ...

Destroy a raid array

First check how the array is configured:

 mdadm --detail --scan
ARRAY /dev/md/0 metadata=1.2 name=sysresccd:0 UUID=7959e2c7:4681b50c:dd0c19e1:1195129c
root@bart:~# mdadm --detail /dev/md0
        Version : 1.2
  Creation Time : Wed May 16 15:47:28 2012
     Raid Level : raid5
     Array Size : 5860536576 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Wed Dec 12 08:07:13 2012
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           Name : sysresccd:0
           UUID : 7959e2c7:4681b50c:dd0c19e1:1195129c
         Events : 270

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       49        2      active sync   /dev/sdd1
       4       8       65        3      active sync   /dev/sde1

Stop the array:

mdadm --stop /dev/md0

Remove superblock from every device in the array:

mdadm --zero-superblock /dev/sd{a,b,e,d}1

.. and the array is gone …

Boot / Grub Issues With Raid1

One of my servers with 4 harddrives and a software raid 1 (two spares) for the boot partition drove me crazy when I tried to upgrade the kernel: it just did not update - I was stuck with my old kernel no matter what I tried:

  • update-grub
  • grub-install –-recheck /dev/sdX
  • downgrade/reinstall linux-image-3.2.0-4-amd64
  • install linux-image-3.12-0.bpo.1-amd64 from backports

It must have to do something with my raid1 and these 4 partitions - and then I found the problem:

        Version : 0.90
  Creation Time : Mon Oct 25 10:24:01 2010
     Raid Level : raid1
     Array Size : 104320 (101.89 MiB 106.82 MB)
  Used Dev Size : 104320 (101.89 MiB 106.82 MB)
   Raid Devices : 2
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Feb 21 13:10:10 2014
          State : clean
 Active Devices : 2
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 2

           UUID : 0c97ebac:b3935eac:add8e9f6:7cc60436
         Events : 0.896

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       17        1      active sync   /dev/sdb1

       2       8        1        -      spare   /dev/sda1
       3       8       49        -      spare   /dev/sdd1

The problem is in fact that my mirror consists of the 2nd and 3rd harddrive (sdb and sdc) for the boot partition while the other two are just spares. You wanna guess which of my drives was first in BIOS? Yes, sda of course! Don't know why and when sda was kicked out of the active drive list, but after changing the boot order in the BIOS, my box immediately booted the kernel it should - so it is really important to have the BIOS boot order match your raid array layout!

linux/software-raid.txt · Last modified: 2014-02 by tb
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0 ipv6 ready