Ok, so I've been a bit slack while on leave and let Scotty
down by not posting up my speedometer. I swear I will, however, today is more
about server administration than an electronic project.
The reason I am writing about this is that it took me a
while and a lot of reading to get everything I needed to grow, repair, destroy
and create (in that order oddly enough) a RAID5 software array with and overlaying
Logical Volume.
A while back I purchased a HP micro server and with the assistance
of a mate got it up and running with Ubuntu 12.04 LTS. The OS runs of a USB key inside
the server, while the RAID5 array sits in the bay drives. I needed the help
originally as I had no real idea about LVM ‘s (or for that matter any real
working knowledge of a RAID) – so to begin with I had four 2TB hard disks,
which gave me about 5.4TB of usable space on the array.
As with all things, I 'had' to tinker with my Micro Server.
The plan (as suggested by said friend), install two more 2TB hard disks in the
optical bay slot of the server and increase the RAID5. After all, the whole point
of the LVM is to allow the administrator to easily increase/modify disk space.
So I purchased two new hard drives, a power cable splitter,
an esata to sata cable as well as a straight sata cable (all 3Gb/s). Finally, I
purchased a Nexus twin hard drive mount. The nexus is an awesome little bit of kit that
mounts two 3.5” hard drives on top of each other and pushes out vibration
dampeners to the side so that the drives mount into the 5.25” optical bay.
So now the fun part – I will actually quote direct commands
here as this was all done in CLI, thus I can state these all work in Ubuntu
12.04 LTS, having said that if you've stumbled upon this, read the whole bloody
thing - for the sake of your own sanity. I will headline each actual task as
best I can in the order they're listed above.
Growing the Array
This should be simple – and it is. It must be done a
disk at a time but overall it’s very easy. I do not have hot-plug enabled (I
have no idea why I don’t, I just decided I didn't want it, so didn't enable it
in my BIOS). If you do have hot-plug then no need to power down for you.
So I had to power down the server, physically connect one of the new
disks and power the server back up. All going well there should be a lovely new hard drive
in the system. You should be able to spot your drive
allocation by using:
sudo cat /proc/diskstats
If you don’t know, you should also find out the allocation
of your existing array:
sudo cat /proc/mdstat
So the result for mdstat is that my array iss md0 (I knew
this already, but for the sake of someone reading who doesn't know that's how it's done). Diskstats
told me that sde was my new disk.
First things first, let’s partition the new disk:
sudo fdisk /dev/sde
n – for a new partition
p – for a primary partition (followed by default boundaries)
t – to change partition type
1 – to select partition to change (you should only have one
partition in this example)
fd – as the partition type (Linux RAID auto detect)
p – to view what your about to write to the disk matches the
above
w – to write it to the disk!
Congratulations you've now partitioned the disk. Let’s add
it to the existing array:
sudo mdadm --add /dev/md0 /dev/sde1
So now your array begins to grow. It can be a long process,
an array the size of mine took around 24 – 30 hours (I can’t tell you exactly
as the speed fluctuates and I didn't hang around waiting for it), later I’ll
explain a couple of little tricks to speed this process up, some are common,
some are not.
Now this is where my problems began, I didn't really read the
output of ‘p’ in fdisk and hadn't partitioned the disk properly. This wasn't a
major deal, I just now had sda1, sdb1, sdc1, sdd1 and sde. My OCD kicked in here – I want them all to be partitioned
the same blah blah blah (it honestly doesn't matter, it really is just internal aesthetics but oh well).
Removing a Device from the Array
Now I’m not actually shrinking my array here, I just wanted
to remove my disk and re-partition it before re-adding it to the array. Should be
simple right?
It’s slow, it’s painful and here is where it all came crashing
down. In saying that, the commands below are correct, it was just the idiot telling the computer what to do that should of double checked his own work.
First I failed the disk out and removed it:
sudo mdadm --manage /dev/md0 --fail /dev/sde
sudo mdadm --manage /dev/md0 --remove /dev/sde
Now at this stage I rebooted, re-partitioned my disk (as previously done) and
re-added it:
sudo mdadm --manage /dev/md0 --add /dev/sde1
This was my undoing as the system re-allocated the disk
names on reboot and I didn't check – I (somehow) partitioned an existing,
active RAID disk, destroying the superblock in the process. So now I have a
RAID5 with two failed disks. What do we know about RAID5 kids – two failed disks
means an unrecoverable array - $#&*&!
After some googling and some luck (and through the lack of detail
here you can probably tell I’m not exactly sure how), I managed to get the RAID
to commence recovering. Recovery takes a lot longer than creating or growing,
it’s a lot more work for the drives, as in this case, the missing disk was
effectively from the middle of the array, needless to say the heads were
spinning feverishly.
The original estimate on the recovery was 4.2 days (7000
odd minutes) and it came pretty close to this. When it finished I was disappointed to see it didn't save
everything, the damage was done and I had lost some data. Thankfully I keep a
back-up of my data, the most recent being a month earlier, by combining what I
could save from the array and comparing it to my backup I found I was missing
around 18GB, out of 2.1TB of data.
This I can live with as it is not mission critical
stuff and easily replaced. So now what to do, the array is repaired but clearly not in
a great state – I WANT A FRESH START!
Destroying a MDADM Array and LV
First thing first, this is how to destroy a RAID5 array, so
be sure you want to do it – there is no coming back from this once it’s done!
In my case, my array is mounted on boot through fstab so
I had to disable it in fstab and reboot (I find it easier this way as there
aren't remnants hiding in a system process stopping the array from being shutdown). So rebooting my server without fstab mounting the array and
binding my network share to it meant that I could deactivate everything.
First we need to deactivate the Logical Volume and Volume Group:
sudo lvdisplay – get the name of the volume group (Array in this instance).
sudo lvchange –a n Array – this deactivates the Volume Group
sudo lvremove Array – removes Volume Group
An example of lvdisplay from a previous setup:
sudo vgdisplay – get the name of the volume group (RAID5 in
this instance).
sudo vgchange –a n RAID5 – this deactivates the Volume Group
sudo vgremove RAID5 – removes Volume Group
So now we need to stop the array:
sudo mdadm --stop /dev/md0
Zero the superblocks:
sudo mdadm --zero-superblock /dev/sda1
(repeat for all
disks, one at a time sda1 – sdx)
Remove the array:
sudo mdadm --remove /dev/md0
Finally, I did this to make sure the partition table was
gone too:
sudo shred –n 1 /dev/sda (repeat for all disks, one at a time sda – sdx)
It should be noted I’d let the above run for five minutes or
so and then cancel it (Ctrl – C), we only need to destroy the data on the front
of the disk so we can create our new array without mdadm complaining (it’ll
destroy the rest of the data anyway, also shredding even a single pass on a 2TB
disk will take a while and there’s no real need unless you want to be painfully security conscious).
Creating a RAID5 Software Array and LV
So now it’s time to make our new array (thank Christ!), this
is not as slow as recovering or even growing an existing array. All up creating
the (now) 10TB array (9.3TB usable), took around six hours. Creating the LVM
over the top is less than fifteen minutes.
So first, let’s partition our disks (properly this time –
take it from me, check your work):
sudo fdisk /dev/sda
n – for a new partition
p – for a primary partition (followed by default boundaries)
t – to change partition type
1 – to select partition to change (you should only have one partition in this example)
fd – as the partition type (Linux RAID auto detect)
p – to view what your about to write to the disk matches the above
w – to write it to the disk!
Complete the above for all disks to be used in the array, in
my case it was sda through sdf.
Now let’s create our new array:
sudo mdadm --create /dev/md0 --level=5 –n 6 /dev/sda1
/dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
This is the long part as now the system will start creating the
array. Level as referenced above refers to the RAID type, 'n' is the number of devices (six hard disks). A spare drive can also be added to assist in automating things when there's a failure, though this does not interest me.
Fast forward six hours and my array is complete (at this point I was
also quite drunk after six cans of Guinness). Let’s update the mdadm configuration
file and propagate our new array through the system before we do anything (kernel
included):
sudo mdadm --detail –-scan
We need the UUID of our mdadm array (don’t confuse this with
the UUID obtained later) and a few other details that were just spat out at you
(see below for what you need). Right now though we need to edit the mdadm.conf file (it’s located at /etc/mdadm/mdadm.conf).
Below ‘#definitions of existing arrays’, add the following from the above
output (delete any existing entry here if you removed your old array too):
ARRAY /dev/md0 metadata=x.x name=xxxx:x
UUID=xxx:xxx:xxx:xxx
Where the x’s are the data spat out from the above command
(you’re probably wise to copy and paste the output TBH).
Propagate the changes through the system:
sudo update-initframs -u
The above is especially important if you've removed a
previous array and are replacing it. Anyway time to get our LVM on.
Let’s create the physical volume:
sudo pvcreate /dev/md0
Now the Volume Group:
sudo vgcreate RAID5 /dev/md0
RAID5 is the name I chose for my Volume Group, I kept it the
same as the one my friend helped me make originally so it was easier for me to
follow.
To create the Logical Volume we need to know the total size
of our Volume Group:
sudo vgdisplay RAID5 | grep “Total PE”
The result in my case was ‘2384496’. Now let’s make the LV:
sudo lvcreate –l 2384496 –m Array RAID5 /dev/md0
Where ‘Array’ is the name of my LV and ‘RAID5’ the name of
my previously created Volume Group.
Finally, we need a usable file system, in my case ext4:
sudo lvdisplay – get the full path of our LV
sudo mkfs.ext4 /dev/RAID5/Array
A short time later the process will complete and you’ll have
a software RAID5 with a ext4 Logical Volume.
I added mine back to fstab so it would auto mount, first you
need the UUID of the array:
sudo blkid
Here we’re looking for the UUID of our new array
/dev/mapper/RAI5-Array in my case (note the VG and LV name use here, that’s how
you’ll know you've got the right one).
Finally, in fstab itself add a line similar to the below
(you’ll have to be root to edit fstab BTW):
UUID=xxx-xxx-xxx-xxx-xxx /my/mount-point ext4 defaults 1 1
Where the UUID is the UUID from blkid, /my/mount-point is
where you are going to mount your new LVM (an example might be /opt/myarray),
ext4 is the file system in use and the rest you don’t need to worry about
unless you’re an advanced user (in which case you’re probably not reading this). The spacing between the mount point, ext4, default values etc should be a TAB spacing.
Reboot and you’re done.
Speeding up Growth, Recovery and Creation
If you've kept reading to this point good for you! Here’s some tips for getting better disk speeds when dealing with growing,
recovering or creating an array. Some of these can be altered while the process
is going, however things like bit mapping cannot and must be done beforehand. There’s some pros and cons and I’ll touch on them a little where applicable.
To give some ballpark figures from my system (these will be
different based on the machine used):
Growing the array, 15 – 20 MB/s
Recovering the array, 5 – 7 MB/S peaking briefly at times to
20MB/s
Creating the array, 100MB/s (this dropped to 60MB/s at the
end)
To monitor the systems progress during any of the above:
cat /proc/mdstat
or
sudo mdadm --detail /dev/md0
From here on in I did have some issues with ‘sudo’. From
what I've been told it sounds like a re-direct issue, I couldn't execute these
commands (permission denied error). Logging in as root however fixed this (sudo
su). Just something to keep in mind.
Minimum and maximum disk speeds (can be changed during a
running process):
sudo echo 20000 > /proc/sys/dev/raid/speed_limit_min
sudo echo 200000 > /proc/sys/dev/raid/speed_limit_max
This sets the minimum speed to 20MB/s and the maximum to
200MB/s. From experience this doesn't mean the system will hold these speeds if
it can’t, so adjust the according to what you’re seeing from cat /proc/mdstat
(use some common sense basically).
Stripe buffer caching (can be changed during a running
process):
This is a bit like reading ahead, it accepts values from 16
to 32768 (32MB). There were a lot of warnings about this when I was reading
about it, basically setting the value too high had caused swap storms and the
like - too much memory (and/or virtual memory) had been used for the process is the short version.
I took into account the age of the posts (they were reasonably old) and monitored the usage closely. I set the stripe buffer to 32MB and noticed an increase around the fifteen percent mark in RAM usage. I
have 8GB of RAM and a 3.7GB swap partition, at this point I decided I was safe to leave it there. The default value is 256, obviously the reference to
md0 below relates directly to the fact md0 is my array:
sudo echo 32768 > /sys/block/md0/md/stripe_cache_size
Bit-mapping (can only be enabled prior to the process
commencing):
Now I have to admit I have no experience with this one, but
if it prevents you sitting there for 4.2 days then it’s worth it. Bit-mapping reportedly improves recovery times after crashing as well as growth times, it does however
gradually degrade your arrays performance if you do not disable it at the
completion of the process (it should also be noted this is only relevant for certain RAID configurations of which RAID5 is one).
Enabled:
sudo mdadm --grow --bitmap=internal /dev/md0
Disabled:
sudo mdadm --grow --bitmap=none /dev/md0
Conclusion
Well that’s it folks. My system is now back up and running
as it was before I broke it (ya!) and I hope to never have to do this again.
Hopefully this saves someone from having as many browser windows open as I did
when trying to learn all this stuff.
Here’s a picture of my hacked up server with its two 3.5”
drives secreted in the optical bay slot – enjoy!
