Software RAID 5 in Ubuntu/Debian with mdadm9 min read

Notes:

  1. I am performing these examples in Virtualbox, so the hard drive sizes will be much smaller than what you’ll have in reality, but this will serve as a good demonstration of how to perform the actions.
  2. RAID-5 requires a minimum of 3 drives, and all should be the same size. It provides the ability for one drive to fail without any data loss. Here’s a quick way to calculate how much space you’ll have when you’re complete.

Usable space = (number of drives – 1) * size of smallest drive

Required software

To create, format and resize the array we will need to install only one package, mdadm. To get started, let’s first switch to the root user, and then install mdadm, ssh, and parted. Personally, I like to upgrade to a newer version of mdadm, but first I’ll show you the instructions for a standard install.

Initial setup

Before we jump into creating the actual RAID array, I would suggest you put a partition on each drive that you plan to use in your array. This is not a requirement with mdadm, but I like to have the disks show up in fdisk as partitioned. In the past I would have shown you how to create the partitions with fdisk, but fdisk doesn’t support partitions greater than 2TB, so that rules out many modern hard drives.

Instead, I’ll show you how to created the partitions with parted using GPT labels. But first, let’s view a list of our available hard drives and partitions.

This will output, for each drive you have, something along the lines of:

This shows (4) 1GB drives with no partition, and /dev/sda that is where the operating system is installed. The last 4 drives should be safe to use in our array. Next, let’s actually partition those (4) disks.

After being started with the above switch, parted will align all the partitions created in such a way that every partition on the drive will start from a cylinder that is divisible by 4096. As a result, it will be properly aligned as the align-check command shows above.

If you are using GPT, then you can use sgdisk to clone the partition table from /dev/sdb to the other three drives. This has the other benefit of having a backup of your disk partition table.

Creating the Array
Now that our disks are partitioned correctly, it’s time to start building an mdadm RAID5 array. To create the array, we use the mdadm create flag. We also need to specify what RAID level we want, as well as how many devices and what they are. The following command will use 3 of our newly partitioned disks.

The verbose flag tells it to output extra information. In the above command I am creating a RAID-5 array at /dev/md0, using 3 partitions. The number of partitions you are using, and their names may be different, so do not just copy and paste all of the command above without verifying your setup first. Note that the partition name is something like /dev/sdb1, whereas the drive name is something like /dev/sdb; the 1 refers to the partition number on the disk.

RAID6

If you wanted to build a RAID6 array, it’s equally as easy. For this example, I’ll throw in a couple new example drives to make our array bigger.

While the array is being built you can view its status in the file /proc/mdstat. Here the watch command comes in handy:

This will output the contents of the file to the screen, refreshing every 2 seconds (by default). While the array is being built it will show how much of the “recovery” has been done, and an estimated time remaining. This is what it looks like when it’s building.

When it’s completed syncing, it should look like this. This process can take many hours depending on how big the array you’re assembling is.

Now that we have set up the array, we need to edit the mdadm configuration file so that it knows how to assemble the array when the system reboots.

Next, update the initramfs, so that the OS has access to your new mdadm array at boot time.

You can view your new array like this.

It should look something like this.

You can see that the chunk size is 512K and the metadata version is 1.2.

Verify that your email is working like this. You’ll need to setup an [email server](http://zackreed.me/articles/39-send-system-email-with-gmail-and-ssmtp) first. This will ensure that you get alerts if something goes wrong on your system.

Creating and mounting the filesystem
Now that the array is built we need to format it. What filesystem you choose is up to you, but I would recommend ext4, unless your array will be bigger than 16TB as that’s current the maximum size supported by e2fsprogs. I will show you the quick way to apply a filesystem to your array first. If you want to optimize your filesystem performance, follow the next example…

Unoptimized

This will take a while, especially if your array is large. If you want to optimize your filesystem performance on top of mdadm, you’ll need to do a little work, or use this [calculator](http://uclibc.org/~aldot/mkfs_stride.html). Here’s how you do it.

Optimized
1. chunk size = 512kB (see chunk size advise above)
2. block size = 4kB (recommended for large files, and most of time)
3. stride = chunk / block in this example 512kB / 4k = 128
4. stripe-width = stride * ( (n disks in raid5) – 1 ) this example: 128 * ( (3) – 1 ) = 256
So, your optimized mkfs command would look like this.

Note: As a caveat for using ext4 with volumes > 16TB, you’ll need to use a newer version of e2fsprogs if you want to create a filesystem that will support 16TB+. Ubuntu 12.04 comes with e2fsprogs version 1.42, and this version supports creating a 64bit filesystem (will support > 16TB) like this.

If you chose to use ext2/3/4 you should also be aware of reserved space. By default ext2/3/4 will reserve 5% of the drives space, which only root is able to write to. This is done so a user cannot fill the drive and prevent critical daemons writing to it, but 5% of a large RAID array which isn’t going to be written to by critical daemons anyway, is a lot of wasted space. I chose to set the reserved space to 0%, using tune2fs:

Next we should add the array to the fstab, so that it will automatically be mounted when the system boots up. This can be done by editing the file /etc/fstab.

Your fstab should already contain a few entries (if it doesn’t something is wrong!). At the bottom add a line similar to the following:

*Press crtl+x, and then y to save and exit the program.*

I chose to mount my array on /storage, but you may well wish to mount it somewhere else. As I said earlier I chose to use ext4, but here you will need to enter whatever filesystem you chose earlier. If the folder you chose doesn’t exist you will need to create it like this.

Now, mount the array.

This will mount anything mentioned in the fstab that isn’t currently mounted. Hopefully, your array is now available on /storage. Check your available space like this.

It should look something like this…

You should now have a working RAID-5 array. Next, I would strongly suggest you read my other articles to setup email for monitoring, SMART information monitoring, spinning down disks, setting up a UPS battery backup, and other raid array actions. Being able to cope with one drive failing is no use if you don’t notice when it fails and let a second fail before replacing the failed drive.

Zack

Zack

I love learning new things and trying out the latest technology.

You may also like...

2 Responses

  1. gnilsen says:

    Hi Zack, nice guide.

    When I paste these lines I get errors:

    I am thinking these are html translation errors and the > are supposed to be >

    Can you instead post what the mdadm.conf file is supposed to look like after these bash commands are done?

    Regards, George

    • Zack Zack says:

      Hello George. There were a couple HTML errors. I have corrected these. I don’t use mdadm anymore, so I would need to re-create this to post a mdadm.conf file. This should work as is now 🙂

Leave a Reply