btrfs - time to add a new drive!

[oshunluvr]

Ok, I got a nice cash bonus from my boss. So I immediately spent it on a second Samsung 840 Pro 256GB SSD (as we all would ). Since my current S840P has been my main boot drive and is nearing fullness, I thought a nice 2-disk RAID would be the ticket.

In the olden days, I might have been using mdadm and ext4. Then I would be tediously backing up my drive, making a new RAID partition, restoring same data, crossing fingers, etc.

But not now. Not with btrfs! Please keep in mind - all the stuff being done below is on my boot drive while running! No USB or gparted discs required!

Here's my drive currently:

[COLOR=#222222][FONT=Verdana]Label:'ssd840pro'  uuid: 8f0c1661-4e84-4512-b875-23bcfd5be1d8
Total devices 1 FS bytes used 176.37GB
[/FONT][/COLOR][COLOR=#222222][FONT=Verdana]devid   1 size 222.47GB used 222.47GB path /dev/sdi2[/FONT][/COLOR]

It has a swap partition on it that I've never used, thus the less-than-full 256GB partition size. So I want to do two things: Add the new drive into the old one as a RAID device and get rid of the virgin swap partition. Rather than revert to the old school methods outlined above, I'm going whole-hog (no, not SnowHog that's something completely different! ) new school!

Plugging in the new drive and booting up, I did these simple commands:

sudo mount /dev/sdi2 /mnt/s840p
sudo btrfs device add /dev/sdj /mnt/s840p

Now my filesystem looks like:

[FONT=Verdana]Label:'ssd840pro'  uuid: 8f0c1661-4e84-4512-b875-23bcfd5be1d8[/FONT]
[COLOR=#222222][FONT=Verdana]Total devices 2 FS bytes used 176.37GB[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]devid   1 size 222.47GB used 222.47GB path /dev/sdi2[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]devid   2 size 238.47GB used 0.00 path /dev/sdj[/FONT][/COLOR]

Ok, Now on to using that new space. A quick blkid shows that the old partition and the new drive are sharing the same UUID and label:

[FONT=Verdana]root@office:~#blkid -o list[/FONT][COLOR=#222222][FONT=Verdana]
device                fs_type  label      mount point              UUID[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]------------------------------------------------------------------------------------------------------------[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sda              btrfs    data1      (in use)                 ff5136d4-33e1-418e-80ca-d616b37bc70f[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdf              btrfs    data1      (in use)                 ff5136d4-33e1-418e-80ca-d616b37bc70f[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdg              btrfs    data1      (in use)                 ff5136d4-33e1-418e-80ca-d616b37bc70f[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdh              btrfs    red        (in use)                 dce8ff64-d316-40df-9425-961963d8c42d[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdj              btrfs    ssd840pro  (in use)                  8f0c1661-4e84-4512-b875-23bcfd5be1d8[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdi1             swap                <swap>                   efa9e3d5-0537-4411-844c-0a6f48fd3848[/FONT][/COLOR]
[COLOR=#222222][FONT=Verdana]/dev/sdi2             btrfs    ssd840pro  (in use)                  8f0c1661-4e84-4512-b875-23bcfd5be1d8[/FONT][/COLOR]

Note the "in Use" designation! Next, I turned off swap and removed the swap line from fstab. Then, I wanted to move the data to the new drive so I could remove the swap partition for the old drive and use the entire device.

sudo btrfs device delete /dev/sdi2 /mnt/s840p

does the trick (this took quite awhile - several hours). Now I have single drive filesystem.
Now my filesystem looks like:

Label: 'ssd840pro'  uuid: 8f0c1661-4e84-4512-b875-23bcfd5be1d8
Total devices 1 FS bytes used 177.80GB
devid 2 size 238.47GB used 180.03GB path /dev/sdj

Time to remove the old, smaller filesystem to clean the old drive off.

sudo mkfs.btrfs -f /dev/sdi

The "force" option ( -f ) was needed because btrfs detected the DOS partition table that existed on the drive.

Here we go:

Label: none  uuid: 18dfb82f-3ed7-4fc8-99e0-4aaf83aff031
        Total devices 1 FS bytes used 28.00KB
        devid    1 size 238.47GB used 2.04GB path /dev/sdi

Now we add this one to the other:

sudo btrfs device add /dev/sdi /mnt/s840p

Now we see:

Label: 'ssd840pro'  uuid: 8f0c1661-4e84-4512-b875-23bcfd5be1d8
        Total devices 2 FS bytes used 177.80GB
        devid    2 size 238.47GB used 180.03GB path /dev/sdj
        devid    3 size 238.47GB used 0.00 path /dev/sdi

OK, time to make it a RAID0 device:

sudo btrfs balance start -dconvert=raid0 /mnt/s840p

In an hour or so, I will have a RAID0 boot drive.

[GreyGeek]

Awesome post! I have a question. You wrote:

/dev/sdi1 swap <swap> efa9e3d5-0537-4411-844c-0a6f48fd3848
/dev/sdi2 btrfs ssd840pro (in use) 8f0c1661-4e84-4512-b875-23bcfd5be1d8

Note the "in Use" designation! Next, I turned off swap and removed the swap line from fstab. Then, I wanted to move the data to the new drive so I could remove the swap partition for the old drive and use the entire device.

sudo btrfs device delete /dev/sdi2 /mnt/s840p

I know that doing "sudo swap off" and removing the swap line from fstab would eliminate it from the system, even though /dev/sdi1 would still be in the disk map.
However, I wasn't aware that the "btrfs device delete <device> <path>" command would also move data off of /dev/sdi2 (/mnt/s840p) onto "data1" subvolume during the process of deletion.

The btrfs executable is on /sbin and is owned by root, so the use of sudo is necessary.

:~$ whereis btrfs
btrfs: /sbin/btrfs /usr/include/btrfs /usr/share/man/man8/btrfs.8.gz
:~$ vdir /sbin/btrfs
-rwxr-xr-x 1 root root 353424 Jan  1 05:09 /sbin/btrfs
:~$

To make sure I understand what you wrote, you are saying that s840p is your boot drive, /boot. Is it the same drive that contains the root filesystem, where /sbin resides?
If so then that is remarkable! It is like sawing of the branch you are setting on while cutting down a tree, but not falling to the ground when you do it.

The reason I ask is because I have been contemplating what I'd do if I found it necessary to reverse a bad update, one that borked my accelerated video, or killed by wireless.
I have apt-btrfs-snapshot installed. When the updater runs it first creates a snapshot of the system before installing the packages.

:~$ [FONT=courier new][B]sudo btrfs subvolume list /[/B][/FONT]
ID 257 gen 9602 top level 5 path @
ID 258 gen 9602 top level 5 path @home
ID 385 gen 9568 top level 257 path .snapshots
ID 386 gen 9568 top level 5 path @home/.snapshots
ID 387 gen 9012 top level 257 path .snapshots/1/snapshot
ID 388 gen 9012 top level 5 path @home/.snapshots/1/snapshot
ID 389 gen 9012 top level 5 path @home/.snapshots/2/snapshot
...
ID 409 gen 8997 top level 5 path @home/jerry/data
I[B]D 412 gen 9012 top level 5 path @apt-snapshot-2014-01-26_20:15:56[/B]
ID 415 gen 8997 top level 5 path @home/jerry/data_snapshot
......
ID 537 gen 9485 top level 257 path .snapshots/24/snapshot
ID 538 gen 9485 top level 5 path @home/.snapshots/25/snapshot
[B]ID 539 gen 9523 top level 5 path @apt-snapshot-2014-01-27_08:52:23[/B]
ID 540 gen 9566 top level 257 path .snapshots/25/snapshot
ID 541 gen 9566 top level 5 path @home/.snapshots/26/snapshot

I can access all of the snapshots setting in the /.snapshot directory and see the files they contain, and modify or change them if I want. BUT, the @ap-snapshots are not visible or accessible unless I mount my running sd4 partition on /mnt (also on my running subvolume). I have done that to delete some of them, as I wrote in another post. BUT, if I were to do something, as root, like:

mount -o /mnt/sda4 /mnt
mv /mnt/@ /mnt/@old
mv /mnt/@apt-snapshot /mnt/@
btrfs subvolume delete /mnt/@old
umount /mnt

I would not lose my access to the mv and btrfs commands after I moved /mnt/@ to /mnt/@old No need to reboot, or change grub?

[oshunluvr]

I believe to elevate a snapshot, you can't be running from it at the time. I haven't tried this one, so I can't say for sure. It would be a good experiment. I have booted to a snapshot directly, but not followed through with removing the source afterward. Grub is another matter. AFAIK, as long as the subvolume names and kernels are the same as what's in grub.cfg, you will boot. It would be interesting to make a snapshot, boot to it manually, then run update-grub to see how it handles it.

As far as the "device delete", yes the delete moves the files and metadata to the other device. Multi-device btrfs filesystems work in several different ways. Starting with a single device:
If you add a new device without specifying a raid level, you get JBOD. Deleting a device from a JBOD moves the data off the deleted device.
If you add a new device and specify raid1, you get two drives duplicated. You cannot delete a device from a 2 disk raid1 unless you first add a 3rd disk.
If you add a new device and specify raid0, you get striped data but duplicated metadata. If you delete a device from a raid0, you get a single device again.
You can even do raid10 if you have 4 devices. It's stupidly easy - just specify raid10 when you create or add the devices required.

Changing from one to the other raid level takes time - the data has to move and re-balance. You can follow the progress via dmesg. Since btrfs is still in development, you don't get much verbosity from the cli. You just get a message when it's done.

One thing I stupidly forgot in my above experiment - Since I used the entire drive rather than a partition for my raid0, I wiped my grub space (the first megabyte). I was able to move my data on and off the drive OK, bit now I can't boot it! No biggie: I'm re-doing my boot device with 13.10 Ubuntu so I can get back to booting. I have 6 drives on this machine, so plenty of boot locations are available.

[GreyGeek]

....As far as the "device delete", yes the delete moves the files and metadata to the other device. ...

So, when it moves the files does it recreate the same hierarchy on the other device, preserving the directories, subvolumes, attributes, etc? Or does it just pile them all into / ?

[oshunluvr]

It is totally transparent to the user space. All structures remain intact and you can still use the file system while the balance/delete/add/balance functions are occurring. I did notice that my system was busy - things were happening a bit slower than usual - otherwise no effect to operations. Totally an on-line operation. It's really amazing.

[dibl]

Nice writeup -- thanks.

When you were ready to reconfigure the original SSD, I think an alternative approach that would have given an equally satisfactory result would have been to use Gparted (or fdisk) and delete the partition table, reverting it to a raw device. btrfs is happy to make a filesystem on a raw device with no partition table, or to add such a device to an existing btrfs filesystem.

[GreyGeek]

Rather than risk my Tahr installation I decided to create a Tahr guest OS in VirtualBox.

I read somewhere that one can't boot if btrfs is on /boot. So, I created a 32GB fixed vmdk and during the install I manually created a 62Mb EXT2 /boot (sda1) and the root, /, as btrfs using manual configuration. Things appeared to go well until "Post installation trigger initrmfs tools" threw an "Error removing initrmfs, subprocess installation post installation script returned error, exit status 1"

I repeated the install using EXT4 for /boot but got the same error. So, I made both /boot and / as EXT4 partitions. (EDIT) I got the same error. (/EDIT) Then I redid the install and created a single / partition using EXT4. It booted OK. I then downloaded btrfs-tools, snapper, brtfs-convert and apt-btrfs-snapshot. That's when I ran into the problem of how to run brtfs-convert while running on the / partition. I don't know of a way to boot a vmdk disk and run apps on it.

[dibl]

I read somewhere that one can't boot if btrfs is on /boot.

AFAIK that is not generally true -- see for example the Arch wiki. It is true on my system only because my PCI bus SSD is not recognized by the BIOS as a bootable device, but that problem is the same for my ext4 OS filesystem.

Your VMDK virtual disk is only useful for installing an OS -- I'm not aware of anything else you can do with it. So you need an installer/Live CD/USB stick that can address the empty VM and do things like partition it or make a filesystem or install an OS.

[oshunluvr]

Re. /boot and btrfs: It was true, but no longer. There is no problem booting to a btrfs partition or subvolume with grub.

I am however, in the midst of fixing my last mess-up. Grub2 uses the first MB or so of your disk for it's stuff. This is not a problem as fdisk and the like now start at sector 2048 so that area is available. However, if you elect to use the entire disk rather than a partition for a btrfs filesystem, it starts at sector 0 - effectively preventing the installation of grub to that disk.

Fortunately, you can start the btrfs fs further down the disk by using --alloc-offset switch. I'm researching now what the proper amount of offset would be.

[GreyGeek]

OK. I'm going to retry creating a Kubuntu 14.04 guest OS and create just one partition on the drive, "/". Will btrfs create @ and @home (/ and /home)? Let you know how it goes.

I'm finding that my btrfs is very stable, quick and for most purposes transparent. I wish I could say the same for the btrfs documentation. But, as an old programmer, I understand how hard it is to document a rapidly moving target. That stuff about btrfs not being able to boot by itself was in a document written last April. I'm limiting my googleing of btrfs information to the last month and mainly to ubuntu info.

[oshunluvr]

Yeah, docs are a bit light at the moment. Of course, since it's v0.20rc1 we can't expect too much.

OK. I'm going to retry creating a Kubuntu 14.04 guest OS and create just one partition on the drive, "/". Will btrfs create @ and @home (/ and /home)? Let you know how it goes.

It should, but with 14.04 who knows what they've done to Ubiquity.