Announcement

Collapse
No announcement yet.

Ahhh, the joys of btrfs. True backup and fault protection in one. An article...

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Ahhh, the joys of btrfs. True backup and fault protection in one. An article...

    Another +1 for btrfs in my book.

    The current implementation includes "send-receive" functionality that allows you to send an entire subvolume from one disk to another - even over a network - or to a file. This is an amazing tool and real time saver.

    In the process of re-building my server (new hardware and re-configuring the drives), I used an older, small SSD. I immediately began having boot and space available problems on this drive (it wasn't full). Seeing the writing on the wall, before my last reboot, I did the send-receive from the SSD to one of the hard drives. I had both the Ubuntu server 14.04 and Kubuntu 15.10 installed to this drive (same btrfs partition, different subvolumes). I then prepared both of these new, received, subvolumes for booting: Which means I mounted them, edited the grub.cfg's and fstab's to change the UUIDs to the destination partition's UUID. Finally, I installed grub to the hard drive.

    All of the above took a few minutes (the send-receive not included - that takes some time). The nice thing is I now had a complete bootable copy of two installs and I didn't have to go through all the hoops of other methods to replicate an install. Well, as luck would have it, the very next reboot the SSD failed totally. Not even detected in the BIOS. The destination hard drive happened to already be set as the next boot device, so as I watched the reboot I wasn't even aware that I was booting to the replacement subvolume, right up until mounting swap failed - the swap partition was on the SSD.

    In order to do a send-receive, the subvolume must be read-only. So the first step is to make a read-only snapshot of the subvolume. This is required because as you might imagine, making a copy of a subvolume that is in use could result in corruption. A read-only snapshot can't be edited so it is presumed safe to send. Then you need only to have a destination to send it to. Since you're sending a read-only subvolume, you also are receiving a read-only subvolume so once the receiving is done, if you want to put it in use you must convert it to read-write. If you're just making a backup, you don't need to do that right away, but it's simple enough to do.

    Here's a quick summary of the terminal commands required (as root) to accomplish the task I did above. I had three subvolumes on the SSD so I repeated this three times:

    ### Make a read-only snapshot of the subvolume named @Ubuntu_Server_14_04 ###
    btrfs subvolume snapshot -r @Ubuntu_Server_14_04 @Ubuntu_Server_14_04_ro

    ### Send the read-only snapshot to a btrfs file system mounted at /mnt/backup ###
    btrfs send @Ubuntu_Server_14_04_ro | btrfs receive /mnt/backup

    ### When complete, make a read-write snapshot of the received subvolume ###
    btrfs subvolume snapshot /mnt/backup/@Ubuntu_Server_14_04_ro /mnt/backup/@Ubuntu_Server_14_04

    ### Delete the read-only snapshots for clean-up ###
    btrfs subvolume delete @Ubuntu_Server_14_04_ro
    btrfs subvolume delete /mnt/backup/@Ubuntu_Server_14_04_ro


    and that's it. I just made a fully usable copy of my Ubuntu server install. Now I only have to edit grub and fstab to reflect the new UUID's and it's bootable. Since we're using the terminal to do this, another convenient feature of btrfs is you need only use the first two letters of most of the commands to accomplish a task. So this

    btrfs subvolume snapshot -r @Ubuntu_Server_14_04 @Ubuntu_Server_14_04_ro

    can be this:
    btrfs su sn -r @Ubuntu_Server_14_04 @Ubuntu_Server_14_04_ro

    to save a few keystrokes. BTW, if you wanted further protection against accidental deletions or potential software update snafu's, just keep the snapshot. Then you can roll back an install or recover a deleted file with ease.
    Last edited by oshunluvr; Sep 19, 2015, 09:47 AM.

    Please Read Me

    #2
    Excellent info, oshunluvr!

    I believe you are KFN's official btrfs master!
    "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
    – John F. Kennedy, February 26, 1962.

    Comment


      #3
      I guess I should have responded to your other e-mail sooner. I was going to say it sounded as though it was failing. It seems it did! Good you did what you did and even had BTRFS as your FS. If I didn't have to dual boot I'd like to try BTRFS but oh well!

      Comment


        #4
        Moonrise: Obviously I recommend btrfs! If for no other reason, you can protect your install and your home by simply taking a snapshot once in a while.

        GreyGeek: I think it's time for a few really well done automated ways for btrfs to do data protection. I haven't tried snapper since your last go at it, but this winter maybe I'll have the time to get to work on something like that. An automated backup script would be easy enough as a cron job for sure.

        One of the other things I did when I reconfigured my server was to separate all the data stored on it into subvolumes (11 in total). To my other computers, it's transparent because I mount all the subvolumes in their own folder under a single folder named "exports." Then I export them all to the various services I use (NFS, samba, dlna, Plex, etc.) as a group. I did this so I could send-receive the subvolumes individually. My server currently has a 6TB drive (with about 3TB of data) and 3x2TB ones. The 6TB drive is new so I'm planning on making a full backup of each subvolume onto the three other drives, then using send-receive in a cron to make automated incremental backups of them. For the server install subvolume, maybe a daily snapshot and a weekly backup or something like that - along with a weekly auto-update so I don't have to interact with it directly at all. I'll just have it send me a reboot notification or something like that. Still trying to decide what's the best way to go forward with this plan...

        Please Read Me

        Comment


          #5
          Originally posted by oshunluvr View Post
          GreyGeek: I think it's time for a few really well done automated ways for btrfs to do data protection. I haven't tried snapper since your last go at it, but this winter maybe I'll have the time to get to work on something like that. An automated backup script would be easy enough as a cron job for sure.
          I started to expand my snapper script, which I had previously posted online here in another msg, but when my wife fell and broke her hip on July 18th I haven't had the time since then. I may pick it back up later this winter.

          Your use of btrfs subvolumes and send/recieve is awesome.
          "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
          – John F. Kennedy, February 26, 1962.

          Comment


            #6
            I found another project recently - btrbk - which is a script that automates backups using send-receive. Looks very interesting. Automation is the way to go for a server. I just need to wrap my head around what the best methodology is and then apply it.

            Please Read Me

            Comment


              #7
              OK, btrfs masters, I think there's more than one now with V & GG coming in close, but the guy that loves the ocean is the instigator of this, I should know better than to ask a question in an area I haven't paid attention to, but here goes.

              GRUB? Or, for that matter, booting using rEFInd? How's that work? Work OK? (or, UEFI, setting up the EFI System Partition?) Easy to set up dual-booting with BIOS+MBR? with UEFI+GPT? Easy to re-install GRUB? IOW, anything different using btrfs vs ext4? Anyone look at this yet?

              One thing I can think of is, a practical matter, solving many problems--and helping people solve their problems here--is hard enough in this business without having new people switch to a FS most of us aren't familiar with.
              An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

              Comment


                #8
                Interesting. I downloaded the tar file and will play with it and compare it to Snapper's original intents. Snapper running automatically with the default configs ate up too much disk space too quickly. In reading btrbk's man pages it looks like it, too, could consume a lot of disk space unless the data was sent to an SDD or two. To back up my @root and @home I'd need at least 500MB of SSD. Perhaps two 256MB sticks as a single pool.
                "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                – John F. Kennedy, February 26, 1962.

                Comment


                  #9
                  Originally posted by Qqmike View Post
                  OK, btrfs masters, I think there's more than one now with V & GG coming in close, but the guy that loves the ocean is the instigator of this, I should know better than to ask a question in an area I haven't paid attention to, but here goes.
                  Uhhh, jack of all trades, master of none, but I'm in...

                  Originally posted by Qqmike View Post
                  GRUB? Or, for that matter, booting using rEFInd? How's that work? Work OK? (or, UEFI, setting up the EFI System Partition?) Easy to set up dual-booting with BIOS+MBR? with UEFI+GPT? Easy to re-install GRUB? IOW, anything different using btrfs vs ext4? Anyone look at this yet?
                  Haven't needed UEFI so far - don't boot windows so don't care. GRUB; well i'm a special case. I multi-boot to the same btrfs filesystem as well as a few other partitions (of which I've written about extensively) so I've managed a separate GRUB partition for years now. The first time I removed an install only to find out I had also inadvertently removed my ability to boot gave me the idea to seek a more permanent solution. I can say that btrfs compatibility has been part of GRUB for quite some time. The last time I bothered to look, you still couldn't boot directly to a multi-disk btrfs filesystem so in that case you would need a separate boot partition (just like RAID used to). Other than that - no issues with GRUB - it works as expected without special requirements.

                  Originally posted by Qqmike View Post
                  One thing I can think of is, a practical matter, solving many problems--and helping people solve their problems here--is hard enough in this business without having new people switch to a FS most of us aren't familiar with.
                  Hmmm, food for thought. However a couple things come to mind: When a problem is presented does one ignore or even hide a solution based on the presumption of another's skill level? Or does one offer what might be a challenging option and let the subject determine their own fate? I opt for the latter. Furthermore, does one withhold progress in the name of familiarity or does one boldly strive forward into the unknown? Again - I opt for the latter. Honestly, btrfs is much easier to deal with in many ways than legacy FSs are (even the author of ext4 calls it a stop-gap to the next better FS) and you can even convert ext4 to btrfs and back again if you choose to.

                  I often post on the benefits of btrfs and have been using it continuously for more than 3 years. Dibl - another KFN member (recently less present due to switching to Siduction as his primary OS) has been doing the same for as long as me and Jerry (GreyGeek) - never fearful - has jumped in over the last couple of years and has tried and tested many features.

                  I also always try to remember to offer caveats and post my failures as they occur. To me, Linux has been, is, and will always be an adventure. I will continue to advocate the experience (always with backups ) of the newest challenges while still trying to maintain a usable real-world system for everyday use. Current threads you and I are involved in bear this out: btrfs offers what other, longer in the tooth filesystems cannot - yet it is in the kernel (unlike ZFS) and is under very active development. It will not be long (IMO only of course) before btrfs is the Linux default FS for many distros.

                  I think a valid parallel is a topic you have devoted many hours to: UEFI. An obvious difference being the hardware is foisting this upon in many cases us where file systems are still a total choice. However, as distros and users adopt btrfs it will become more and more mainstream, and when it's clearly the solution (thinking snapshots and backups here) I take delight in exposing my fellow Linux adventurers to the possibilities just as you have enlightened many on the how-tos and whys of UEFI.

                  I look forward to your knowledgeable and adept participation in exposing others to the successes and pitfalls of your journey into btrfs. The gauntlet has been thrown

                  Please Read Me

                  Comment


                    #10
                    Originally posted by GreyGeek View Post
                    Interesting. I downloaded the tar file and will play with it and compare it to Snapper's original intents. Snapper running automatically with the default configs ate up too much disk space too quickly. In reading btrbk's man pages it looks like it, too, could consume a lot of disk space unless the data was sent to an SDD or two. To back up my @root and @home I'd need at least 500MB of SSD. Perhaps two 256MB sticks as a single pool.
                    Exactly the source of my "methodology" comment. I'm not willing to blindly offer up my systems to another's ideas of how things should work but the concept of total automation is where I'm headed. Please let me know what you discover

                    Please Read Me

                    Comment


                      #11
                      Yes, I know dibl rather well. He used to hang out here daily. We did a lot with live flash drives using GRUB Legacy (the old-fashioned way: building them manually). Now I do recall he went off with btrfs. dibl has always been on the frontier of things. Wish he'd come by more often these days.
                      An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

                      Comment


                        #12
                        I haven't found a distro which has made Btrfs its default FS but several are making it an option. However, as I noted in another thread, Kubuntu's deployment of just two subvolumes, @root and @home, is IMO superior to OpenSUSE's, for example, which makes subvolumes out of about every directory which starts from /. A directory cannot be converted to a subvolume but it can be added to a subvolume.
                        "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                        – John F. Kennedy, February 26, 1962.

                        Comment


                          #13
                          Originally posted by oshunluvr View Post
                          Honestly, btrfs is much easier to deal with in many ways than legacy FSs are (even the author of ext4 calls it a stop-gap to the next better FS) and you can even convert ext4 to btrfs and back again if you choose to.
                          and back to ? elaborate .

                          I know you can convert ext4 to btrfs ,,,,,,,,we did that to the 1TB ext4 formatted GPT storage drive on this box when I got it ,,,,,,,,but back again ,,,,I missed that ,,,,,,,,oops O ya,,,, IF you keep the @ext2-saved ,,,,,,,,I got rid of that to make it permanent,,,

                          and I will add that it worked perfectly ,,,,, and I see no need to even think of converting it back.

                          right now I am thinking of converting all the partitions in the 500GB ext4 formatted MBR OS disk to it,,,,,,,,,,?,,,,,,or not,,, I want 1 big btrfs disk not 4 btrfs partitions ,,,,,,more planing to do .

                          VINNY
                          i7 4core HT 8MB L3 2.9GHz
                          16GB RAM
                          Nvidia GTX 860M 4GB RAM 1152 cuda cores

                          Comment


                            #14
                            From the Oracle manual:
                            5.11 Converting an Ext2, Ext3, or Ext4 File System to a Btrfs File System



                            You can use the btrfs-convert utility to convert an ext2, ext3, or ext4 file system to btrfs. The utility preserves an image of the original file system in a snapshot named ext2_saved. This snapshot allows you to roll back the conversion, even if you have made changes to the btrfs file system.
                            If you convert the root file system to btrfs, you can use snapshots to roll back changes such as upgrades that you have made to the file system.
                            Note

                            You cannot convert a bootable partition, such as /boot, to a btrfs file system.

                            Please Read Me

                            Comment


                              #15
                              Originally posted by VickyWaran
                              I think you're conflatin[embedded URL deleted] SM[embedded URL deleted]RT reporting with on-disk CRC.
                              Wow, I can't remember when I've seen such an off-target nonsensical post. Troll? Robo-post?
                              Last edited by Snowhog; Oct 16, 2015, 10:33 AM.

                              Please Read Me

                              Comment

                              Working...
                              X