Hot resize Multipath Disk – Linux
This post is for the users of the great dm-multipath system in Linux, who encounter a major availability problem when attempting a resize of mpath devices (and their partitions), and find themselves scheduling a reboot.
This documented is based on a document created by IBM called "Hot Resize Multipath Storage Volume on Linux with SVC", and its contents are good for any other storage. However - it does not cover the procedure required in case of a partition on the mpath device (for example - mpath1p1 device).
I will demonstrate with only two paths, but, with understanding this process, it can be well used for any amount of paths for a device.
I do not explain how to reduce a LUN size, but the apt viewer will be able to generate a method out of this document. I, for myself, try to avoid as much as I can from shrinking LUNs. I prefer backup, LUN recreation, and then restore. In many case - it's just faster.
So - back to our topic - first - increase the size of your LUN on the storage.
Now, you need to collect the paths used for your mpath device. Check this example:
mpath1 (360a980005033644b424a6276516c4251) dm-2 NETAPP,LUN
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=4][active]
\_ 2:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:0:0 sdb 8:16 [active][ready]
The devices marked in bold are the ones we will need to change. Lets get their current size:
blockdev --getsz /dev/sdb
419430400
Keep this number somewhere safe. We can (and should!) assume that sdc has the same values, otherwise, this is not the same exact path.
Collect this info for the partition as well. It will be smaller by a tiny bit:
blockdev --getsz /dev/sdb1
419424892
Keep this number as well.
Now we need to reread the current (storage-based) size parameters of the devices. We will run
blockdev --rereadpt /dev/sdb
blockdev --rereadpt /dev/sdc
Now, our size will be slightly different:
blockdev --getsz /dev/sdb
734003200
Of course, the partition size will not change. We will deal with it later. Keep the updated values as well. Of course, the multipath still holds the disks with their original size values, so running 'multipath -ll' will not reveal any size change. Not yet.
We now need to create editable dmsetup map. Use the current command to create two files: cur and org containing this map:
dmsetup table mpath1 | tee org cur
0 419424892 multipath 1 queue_if_no_path 0 2 1 round-robin 0 1 1 8:32 128 round-robin 0 1 1 8:16 128
Important part - explaining some of these values. The map shows the device's size in blocks - 419424892. It shows some parameters, it shows path groups info (0 2 1), and both sub devices - sdc being 8:32 and sdb being 8:16. Try it with 'ls -la /dev/sdb' to see the minor and major. At this point, if you are not familiar with majors and minors, I would recommend you do some reading about it. Not mandatory, but will make your life here safer.
We need to delete one of the paths, so we can refresh it. I have decided to remove sdb first:
multipathd -k"del path sdb"
Now, running the multipath command, we will get:
mpath1 (360a980005033644b424a6276516c4251) dm-2 NETAPP,LUN
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=4][active]
\_ 2:0:0:0 sdc 8:32 [active][ready]
Only one path. Good. We will need to edit the 'cur' file created earlier to reflect the new settings we are to introduce:
0 419424892 multipath 1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:32 128
The only group left was the one containing 'sdc' (8:32), and since one group down, the bold number was changed from 2 to 1 (as there is only a single path group now!)
We need to reload multipath with these settings:
dmsetup suspend mpath1; dmsetup reload mpath1 cur; dmsetup resume mpath1
The correct response for this line is 'ok'. We pause mpath1, reload and then resume it. It is best to be in a single line, as this process freezes IO for a short period of time on the device, and we prefer it to be as short as possible.
Now, as /dev/sdb is not a part of the multipath managed devices, we can modify it. I usually use 'fdisk' - deleting the old partition, and recreating it in the new size, but you must make sure, if your device requires LUN alignment, that you recreated the partition from the same start point. I will dedicate a post some time to LUN alignment, but not at this particular time. Just a hint - if you're not sure, run fdisk in expert mode and get a printout of your partition table (fdisk /dev/sdb and then x and then p). If your partition starts at 128 or 64, it is aligned. If not (usually for large LUNs - at 63), you are not, and you should either be worried about it, but not now, or should not care at all.
Back to our task.
We need to grab the size of the newly created partition, for later use. Write it down somewhere.
blockdev --getsz /dev/sdb1
733993657
Following the partition recreation, we need to introduce the device to the multipath daemon. We do this by:
multipathd -k"add path sdb"
followed by immediately removing the remaining device:
multipathd -k"del path sdc"
We need to have our 'cur' file updated, so we can release the device to our uses. This time, we update both the size section with the new size, and the new, remaining path. Now, the file looks like this:
0 734003200 multipath 1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:16 128
As mentioned before - the large number in bold is the new size of the block device. The amount of failure groups is one (1), also in bold, and the device name is 'sdb' which is 8:16. Save this modified file, and run:
dmsetup suspend mpath1; dmsetup reload mpath1 cur; dmsetup resume mpath1
Running the command 'multipath -ll' you will get the real size of the device.
mpath1 (360a980005033644b424a6276516c4251) dm-2 NETAPP,LUN
[size=350G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 1:0:0:0 sdb 8:16 [active][ready]
We will need to reread the partition layout of /dev/sdc. The quickest way is by running:
partprobe
This should do it. We can now add it back in:
multipathd -k"add path sdc"
and then run
multipath
(which should result in all the available paths, and the correct size).
Our last task is to update the partition size. The partition, normally, is called mpath1p1, so we need to read its parameters. Lets keep it in a file:
dmsetup table mpath1p1 | tee partorg partcur
We should now edit the newly created file 'partcur' with the new size. You should not change anything else. Originally, it looked like this:
0 419424892 linear 253:2 128
and it was modified to look like this:
0 733993657 linear 253:2 128
Notice that the size (in bold) is the one obtained from /dev/sdb1 (!!!) and not /dev/sdb.
We need to reload the device. Again - attempt to do it in a single line, or else you will freeze IO for a long time (which might cause things to crush):
dmsetup suspend mpath1p1; dmsetup reload mpath1p1 partcur; dmsetup resume mpath1p1
Do not mistaked mpath1 with mpath1p1.
Our device is updated, our paths are online. We are happy. All left to do is to online resize the file system. With ext3, this is done like this:
resize2fs /dev/mapper/mpath1p1
The mount will increase in size online, and all left for us is to wait for it to complete, and then go home.
I hope this helps. It helped me.
Create Huge disk partitions – Above 2TB...
HP-UX – allowed shells, and connecting FC Multipath to NetApp...
Aquiring and exporting external disk software RAID and LVM...
Related posts brought to you by Yet Another Related Posts Plugin.
Tags: dm-multipath, hot resize, Linux, lun, multipath, netapp, online resize, storage
August 23rd, 2011 at 12:51 pm
Now I’m really amazed. Most people will tel you this is impossible.
Mind if I refer to this in my next LVM training?
Florian
August 23rd, 2011 at 9:58 pm
You have my full support. This knowledge is meant to be shared.
I would be happy, though, to get a link to my site. Sometimes I write something intelligible
Ez
October 25th, 2011 at 10:11 pm
what about this procedure below?
http://zenhat.org/2011/04/02/how-to-extend-lun-without-reboot-redhat/
October 26th, 2011 at 2:11 am
Quite similar, however, requires a umount/mount. My procedure does not.
Ez
February 26th, 2012 at 6:16 pm
blockdev –rereadpt /dev/sde
BLKRRPART: Device or resource busy
February 29th, 2012 at 1:37 am
Too little info. Are you sure the device is available at that time?
May 29th, 2012 at 10:08 am
No offense but you make this look like it is really hard. You can fix this way easier.
echo 1 > /sys/block/sdc/device/rescan (where sdc is replaced by the device names for both paths, retrievable from multipath -ll)
multipathd -k
At the prompt(path names dependent on what you got from multipath -ll):
del path sdc
add path sdc
del path sdd
add path sdd
Now look at fdisk -l
You should see the new size.
Remove the partition and recreate it with the same startpoint and the last block as endpoint.
partprobe
resize2fs /dev/mpath/…
That should do the trick. The thing above is really dangerous to do.
Another option is using LVM and adding another partition in the new space instead of resizing the existing one.
Hope this helps someone
June 6th, 2012 at 7:04 am
I will be very glad to find out that you are correct. Simpler is better!
Next time I face resize issues, I will test your method.
CHeers!
Ez
June 25th, 2012 at 1:54 pm
worked again for me – (even with 4 paths!!)
just a small issue – could you re-edit the “blockdev –getsz /dev/sdb” into “blockdev –getsz /dev/sdb” so that cut&paste into linux will work?
10x again..
June 25th, 2012 at 10:43 pm
You don’t know how complicated it is to make WordPress not convert double minus symbol to a single dash. You own me one!
Ez
August 3rd, 2012 at 11:31 pm
Good G !! but I have more easyer again :
scan for all device path
echo 1 > /sys/block/sdm/device/rescan
and
multipath /dev/mpath/mpath1
multipathd -k’resize map mpath1′
Let’s go to lvextend
August 3rd, 2012 at 11:46 pm
sorry I have forgotten the
pvresize /dev/dm-28
before
lvextend -L +1G /dev/vgname/lvname
resize2fs /dev/vgname/lvname
August 10th, 2012 at 1:49 am
Does it work with partitions?
Ez
August 10th, 2012 at 1:49 am
I’m asking because I haven’t had the chance to try it just yet.
Ez