Announcement

Collapse
No announcement yet.

HOWTO: Using btrfs subvolumes for home data files

Collapse
This topic is closed.
X
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    HOWTO: Using btrfs subvolumes for home data files

    I am in the process of changing over to 12.04 from 11.04. I usually preserve my previous working install and do a full new install along side so I don't leave myself computer-less if something goes wrong. Since I have multiple installs on the computer, I tend to keep /home within each install so settings and the like are kept unique. This has the potiential to leave things like documents and downloads scattered all over the place. To prevent this scattering and to reduce the required space for each install (having six separate /homes can use a lot of hard drive space!) I have a large single partition and keep Documents, Pictures, Music, Videos, and Downloads folders on it. Then after each new install, I mount the data partition, remove the default cooresponding folders from my /home directory, and create symlinks to the folders on the data partition.

    This time I decided to try something different. Since I am using btrfs anyway I decided to get more into the filesystem's abilities and use subvolumes for these files. The primary functional difference is rather than a symlink, I use mount points defined in fstab for each of the four folders.

    Why do it this way? Several reasons;

    Easier and more efficent backups: btrfs has subvolume "snapshot" capabilities that you can't do with just subdirectries. Also, by having the file types in different subvolumes I can prioritize backups.

    To learn how to do it:
    This usually is enough reason for me.

    Visually consistant: The folders appear as "normal" folders in my home rather than links.

    Flexability of the filesystem: btrfs allows on-the-fly (while mounted) expansion and contraction. If you run out of room - simply add in another partition or drive.

    HOWTO:
    I will skip how to create the initial btrfs filesystem. The link above explains that in enough detail.

    Example terminal commands assuming your btrfs partition is /dev/sda5:

    Mount the root directory (not a subvolume) somewhere and navigate to it;
    Code:
    sudo mkdir /mnt/data
    sudo mount /dev/sda5 /mnt/data
    cd /mnt/data
    Create a subvolume and take ownership of it;
    Code:
    sudo btrfs subvolume create @mydocuments
    sudo chown myuser:myuser @mydocuments
    Move your existing files to the new subvolume;
    Code:
    mv ~/Documents/* /mnt/data/@mydocuments
    Edit /etc/fstab using your preferred method and add the mount points:
    Code:
    /dev/sda5 /home/myuser/Documents  btrfs  space_cache,compress,subvol=@mydocuments 0       0
    You can use the rootvolume UUID instead of the device name if you prefer.

    Mount the new subvolume in the correct place and you're done:
    Code:
    sudo mount /home/myuser/Documents
    One interesting (and not benefitial) difference is when moving files across subdirectories on a single partition, the file "pointers" are moved but not the actual data. When using subvolumes, the file is actually moved from one subvolume to the other, even if they exist on the same partition. Obviously, this takes quite a bit longer.

    Another small downside is you can't share directories this way. A symlink is still the best way to go for that.

    I'll likely have more to add as I play with this new setup...

    Please Read Me

    #2
    Interesting - the flexibility (dynamic resizing, etc) of btrfs seems like a major advantage here.

    The primary functional difference is rather than a symlink, I use mount points defined in fstab for each of the four folders.

    ...

    Visually consistant: The folders appear as "normal" folders in my home rather than links.
    Technically I think you could do this with multiple ext4 partitions as well - but of course it gets to be quite a hassle having lots of logical partitions at the disk level.

    One interesting (and not benefitial) difference is when moving files across subdirectories on a single partition, the file "pointers" are moved but not the actual data. When using subvolumes, the file is actually moved from one subvolume to the other, even if they exist on the same partition. Obviously, this takes quite a bit longer.
    That's the same as it would be with multple ext4 partitions. From Linux's pov, moving files within a filesystem is just a case of making a new folder entry pointing to the same inode; but inodes are specific to a filesystem and to move to a different one, it requires a physical copy. Although it looks like btrfs subvolumes don't use different extents within the partition (from a v brief reading) - so conceivably a file could be moved to another subvolume without a physical copy/move of the file data - it might require a smarter Linux driver to support this.

    Another small downside is you can't share directories this way. A symlink is still the best way to go for that.
    Do you mean sharing directories between different installed OSes? Or between users?
    Last edited by SecretCode; Mar 04, 2012, 02:46 AM.
    I'd rather be locked out than locked in.

    Comment


      #3
      Originally posted by SecretCode View Post
      Do you mean sharing directories between different installed OSes? Or between users?
      I was thinking between users. You could have different mount points for each user to the same directory, but a link to a single mounted directory makes more sense to me. For example, the default skel user has a "Public" folder intended for Samba file-sharing. In my case, I have 4 people that sometimes use my computer. I can replace that folder with a link to @Public subvolume and give it world access. A somewhat safer course would be to have group ownership of the public areas and add that group to all your users. In actual practice, I have a home server so that acts as the "Public" storage/file exchange.

      I decided to do this post to introduce btrfs to more people. Dibl and I have been using it for a couple of years now and we've both found it to be quite stable. I suspect once the file system check tool is fully functional, it will be the default fs for many distros. They are also going to change the way subvolumes are handled likely sometime this year, so they'll be a format change then (which will be fun!) but that should solve a lot of larger issues and make the fs easier to use.

      In a desktop/server environment, btrfs really shines. Without taking the filesystem off-line you can add and remove drives and partitions at will and RAID happens almost automatically and is configurable to your needs. In my previous server setup, mdadm RAID was used, but when a drive needed removal it was a real chore. With btrfs, it's a breeze.

      Re. Inodes: From a post by Ian Kent, a btrfs dev.:

      1) You cannot hardlink between subvolumes. This is because subvolumes have their own inode numbers and such, think of them as separate mounts in this case, you cannot hardlink between two mounts because the link needs to point to the same on disk inode, which is impossible between two different filesystems. The same is true for subvolumes, they have their own trees with their own inodes and inode numbers, so it's impossible to hardlink between them.

      Another way to use subvolumes: rather than mounting each of them, you can "CD" into them like any other partition. In the case of a Public subvolume, this might be a better course than mounting.

      Please Read Me

      Comment


        #4
        My btrfs filesystem on a pair of WD1002FAEX 6GB/s drives is about 16 months old now, and I see no issues or signs of problems. I chose the drives to take advantage of the Marvell 9128 controller on my Asus P6X58D-E motherboard. I used the defaults for a multi-drive btrfs installation, resulting in mirrored metadata and striped data. My pile of data amounts to about 768GB, out of a total filesystem capacity of 1.8TB, today:

        Code:
        root@imerabox:/home/don# btrfs fi df /mnt/DATA
        Data, RAID0: total=820.00GB, used=768.36GB
        Data: total=8.00MB, used=0.00
        System, RAID1: total=8.00MB, used=72.00KB
        System: total=4.00MB, used=0.00
        Metadata, RAID1: total=12.62GB, used=1.55GB
        Metadata: total=8.00MB, used=0.00
        I'm sold on btrfs as a "easy" way to set up large multi-drive filesystems -- and cost-efficient, when compared to buying a serious hardware RAID controller.
        Last edited by dibl; Mar 04, 2012, 04:29 PM.

        Comment


          #5
          Oshun and Dibl... can you guys comment on any improvements to the speed of Btrfs? Last time I looked at it, about a year ago, it took almost an hour for a 300 MB apt-get upgrade to churn -- and on an SSD, no less. Same thing on EXT4 finished in 4 minutes.

          Comment


            #6
            I can't speak for the apt-get portion of your time. I created a roughly 400MB file (dd bs=4096 count=102400 all zeros) and copied it between two ext4 partions and two btrfs partitions:

            ext4-to-ext4
            real 0m3.297s
            user 0m0.008s
            sys 0m0.596s

            btrfs-to-btrfs
            real 0m0.504s
            user 0m0.004s
            sys 0m0.364s

            ext4-to-btrfs
            real 0m0.452s
            user 0m0.000s
            sys 0m0.404s

            btrfs-to-ext4
            real 0m3.139s
            user 0m0.012s
            sys 0m0.624s

            This is way less than a comprehensive test. Small file performance and really large file performance would likely vary. Also, these are non-RAID ex4 partitions and a three disk btrfs array using a few "advanced" mounting options. btrfs: space_cache,compress ext4: nobh,data=writeback,noatime.

            I posted some more extensive testing way back a like a year ago or so and compared the performance to ext4 in a 4-disk mdadm RAID to a 4-disk btrfs RAID array. The ext4 RAID performed better in some areas and the btrfs array better in others,. However, that was before space_cache and compress were available. At the time I felt the ease of setup and use of the btrfs made it the clear winner, not to mention the other features (like snapshots and subvolumes).

            Remember though, I'm using it in a server and performance desktop both with multiple drives and was using mdadm RAID everywhere until I switched to btrfs. There are also some other considerations to btrfs that haven't been worked out yet. For example, in a RAID array, btrfs doesn't handle large differences in partition size well - it tends to treat the larger partition as though it were the same size as the smaller one. Simple work around is to partition to nearly equal sizes and expand the number of partitions in an array. For example, my server has 2x2TB and 2x1TB drives. I had reserved 200GB on the 1TB drives before I added the 2TB drives to the mix. Since I had a functioning btrfs array of 2x800GB, I created 2 800GB partitions on each 2TB drive and added all 4 of them to the existing array giving me 6x800GB (4.8TB) on that array and used the remaining 2x500GB on the 2TB drives for a second array. Obviously, having partitions on the same physical device in an array doesn't add any performance, but it doesn't appear to hurt any either. Next year when I re-do the server I'll make the main array 4x1TB (1 per drive) and the secondary 2x750GB (unless I acquire a couple more larger drives )

            Please Read Me

            Comment


              #7
              Those Btrfs writes in your test are much faster than the EXT4 writes -- but since your Btrfs mounts include caching and compression, it makes sense that a file of all zeros would show such difference in performance. How would the numbers look if you created a 400 MB file from /dev/urandom instead?

              Comment


                #8
                For my buddy, Steve...

                ext4-to-ext4

                real 0m3.627s
                user 0m0.004s
                sys 0m0.564s

                ext4-to-btrfs
                real 0m6.955s
                user 0m0.008s
                sys 0m0.404s

                btrfs-to-btrfs
                real 0m0.524s
                user 0m0.004s
                sys 0m0.452s

                brtfs-to-ext4
                real 0m3.178s
                user 0m0.004s
                sys 0m0.548s

                As I said before - the 3-disk btrfs array will clobber the ext4 single partition every time. A better test would be a 3-disk mdadm RAID0 with ext4. Unfortunatly, I just installed 12.04 beta 1 so I'm busy migrating my setup.

                For the record the ext4 partitions are on different disks. The 2 btrfs arrays are on the same set of disks. The times were retrived using a simple time cp file target/file command.

                Please Read Me

                Comment


                  #9
                  Originally posted by SteveRiley View Post
                  Oshun and Dibl... can you guys comment on any improvements to the speed of Btrfs? Last time I looked at it, about a year ago, it took almost an hour for a 300 MB apt-get upgrade to churn -- and on an SSD, no less. Same thing on EXT4 finished in 4 minutes.
                  There's no reason btrfs shouldn't work fine for the OS partition, but I really wouldn't recommend it, as I don't think it would offer any advantage over ext4, especially on a SSD. But I don't know why an update or a dist-upgrade should take any longer on btrfs.

                  Btrfs is really targeted to large, and possibly dynamic, data storage requirements. While I haven't had any reason to use the dynamic storage features that it offers, it's obvious to me why it should be very attractive to the IT guys at the server farm.

                  One caveat -- btrfs absolutely sucks for hosting a Virtual Machine (which is really a single very large file). It's a known issue and I would suppose that situation will be improved as the development continues.
                  Last edited by dibl; Mar 05, 2012, 07:38 AM.

                  Comment


                    #10
                    Originally posted by dibl View Post
                    One caveat -- btrfs absolutely sucks for hosting a Virtual Machine (which is really a single very large file).
                    Why is that? I mean why should it be any worse than ext3/4? (I can see why advantages might be lost.)

                    And would it have any implications for things like database tools that keep masses of data in one file with lots of random seeks and writes? (Like Microsoft Access. I'm not actually sure if there are real-world cases of this in common Linux apps.)
                    I'd rather be locked out than locked in.

                    Comment


                      #11
                      Originally posted by SecretCode View Post
                      Why is that? I mean why should it be any worse than ext3/4? (I can see why advantages might be lost.)

                      And would it have any implications for things like database tools that keep masses of data in one file with lots of random seeks and writes? (Like Microsoft Access. I'm not actually sure if there are real-world cases of this in common Linux apps.)
                      I really have not studied it much -- I wasn't aware of the issue, and tried doing something on a backup VM that I had parked on the btrfs partition, and was surprised to see how slow it ran. A quick google search turned up this and so that's how I learned about the issue, and that's about all I know about it.

                      But is your MS Access toolbox 10GB or 20GB or more in size? That's the scale of a VM hard drive.

                      Comment


                        #12
                        I have no Ms Access files at all! What kind of person do you take me for?

                        I'm just remembering stuff that people used to talk about.

                        I think huge databases are the only things that compare with VMs in size (even video files aren't), but such large dbs would often run on dedicated partitions and use multiple files (and NOT use Ms Access!) - and some of the database engines have their own raw file systems do they not?

                        eta: I like the language in that bug report ... "I will now spend some time getting this to suck less" etc
                        I'd rather be locked out than locked in.

                        Comment


                          #13
                          Originally posted by SecretCode View Post
                          I have no Ms Access files at all! What kind of person do you take me for?
                          Whew! Well, I did work a bit with MS Access many years ago, but I could not imagine a 20GB Access file.

                          I would be surprised if a conventional RDBMS had a big problem running its data tables on a btrfs filesystem -- they surely had that kind of application in mind when they designed it in the first place. But my stuff is just videos, music, images, docs, etc., so I don't really know. I think the problem must be related to the unique way a VM opens the virtual hard disk and uses it.

                          eta: I like the language in that bug report ... "I will now spend some time getting this to suck less" etc
                          Yeah, they seem to have a good attitude about it.

                          Comment


                            #14
                            I'm using a btrfs array for my os (11.04 and 12.04) and btrfs array for my home that includes my VirtualBox stuff and I haven't noticed any issues, speed or otherwise.

                            Please Read Me

                            Comment


                              #15
                              Thanks, Oshun, for running the second test. One thing that caught my eye was the difference between the ext4-to-btrfs runs. Here are your numbers:

                              Originally posted by oshunluvr View Post
                              File of zeros:

                              ext4-to-btrfs
                              real 0m0.452s
                              user 0m0.000s
                              sys 0m0.404s
                              Originally posted by oshunluvr View Post
                              File of random bytes:

                              ext4-to-btrfs
                              real 0m6.955s
                              user 0m0.008s
                              sys 0m0.404s
                              Less than half a second to almost seven seconds...if I'm interpreting your numbers correctly, it would appear that Btrfs with data that doesn't lend itself to compression is still performing not too well...?

                              Originally posted by dibl View Post
                              But I don't know why an update or a dist-upgrade should take any longer on btrfs.
                              It was over a year ago, using Arch. I could have had a bad build or something. The symptom was incessant disk thrashing, very similar to what's described in that Redhat bug post you found.

                              Comment

                              Working...
                              X