Announcement

Collapse
No announcement yet.

kubuntu 20.04 with BTRFS, full disk encryption and backup

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    kubuntu 20.04 with BTRFS, full disk encryption and backup

    Hi,
    I just wanted to share my experience of setting up my system with the kubuntu 20.04 LTS release and BTRFS, in case that is useful for anyone else.
    My first attempt, where I was just following the regular LUKS full disk encryption guide for ubuntu 18.04 failed completely - I think the partitions and grub were not set up correctly, because btrfs handles partitions/subvolumes a bit differently. In my case the step where you use the chroot environment to set up crypttab ended in a complete disaster with the old guide (i.e. chroot giving me errors all the time).

    Instead, I used this guide, which worked perfectly (including chroot):
    https://mutschler.eu/linux/install-guides/ubuntu-btrfs/
    I opted for a swap file and also installed Timeshift along with timeshift-autosnap-apt and grub-btrfs, as suggested, however, I do not include the @home subvolume in Timeshift.
    The reason is that I have a separate backup process using btrbk (which is also in the ubuntu repos).
    I think this is fairly important, as Timeshift only creates local snapshots on your disk. This is very fast, but if your disk fails or you loose your laptop, you are out of luck. Timeshift seems to be convenient to restore your system after you broke it or an upgrade failed, but it is not a backup solution!

    I hope someone will find this useful - feel free to ask questions or comment on my setup!

    Before I describe my backup solution, one word of advice: if you are not familiar with Btrfs and you are considering to switch (and there are reasons to do so, in particular related to easy backups), do some research first! There are some important conceptual differences to regular file systems like ext4, and the steps in the guide linked above will make a lot more sense, once you are aware of that.
    Read some articles and in particular, make sure you understand Copy-on-Write and subvolumes. For example, your system-root "/" is not the btrfs filesystem root, it is actually just a subvolume of your btrfs <root_fs>, which is conventionally called '@'.
    I found this post here on kubuntuforums quite useful, too:
    https://www.kubuntuforums.net/showth...-and-SNAPSHOTS

    The way I set up my backup using btrbk requires some preparation:
    First I created a larger partition with an encrypted LUKS volume on an external HDD, formatted that with btrfs, created a subfolder (not a subvolume) to hold my backups, and mounted it on my computer. I created the LUKS and did the formatting with the KDE Partition Manager (GUI). You could add a key file to mount this volume automatically when the HDD is plugged in, but I just use Dolphin and kwallet to unlock it when I log in; just make sure this is mounted when the backup-process runs.
    Then I mount my btrfs <root_fs> with the OS/root/@ and home/@home subvolumes at a mount point (say /mnt/btrfs_pool) and create a subfolder there to hold my snapshots.
    With that, you can run btrbk through a cronjob. btrbk requires a config file, which is quite straight forward to set up and should be run as a root cronjob.
    https://digint.ch/btrbk/doc/readme.html
    Personally, I also have a wrapper script for btrbk, which mounts the <root_fs> before btrbk is called, and unmounts it afterwards. Note, however, that in principle this does not appear to be necessary, because Timeshift seems to always have the <root_fs> of your system drive mounted at /run/timeshift/backup/.
    This creates automatic rotating/cycling backups of your system and home subvolumes on an external, encrypted disk. One of the nice things is that these backups are done incrementally (similar to rsync, but smarter and faster); for me a complete incremental backup from my system SSD to my external HDD takes less than a minute and I have them scheduled daily; the initial backup of course takes longer (~30min.).

    I hope someone will find this useful - feel free to ask questions or comment on my setup!

    #2
    And if you were wondering (like me), how you can reclaim any space on your drive with all these snapshots kicking around (since files only really get deleted if there is no snapshot referencing them), I found this post on reddit that basically outlines the process:
    https://www.reddit.com/r/synology/co...al_files_from/
    In a nutshell, you set all snapshots to read-write, use find to delete the file in all snapshots (or a loop or regex, if you know where it is), and set them back to read-only using the btrfs-progs.

    This thread on the btrbk GitHub also talks about this:
    https://github.com/digint/btrbk/issues/221
    The solution is essentially the same, but there is a warning that reads:
    Note that if you do this, you also confuse btrbk incremental feature: Your modified snapshots will still have the same UUID, but don't hold the deleted files anymore.
    I don't really understand what that means - can anyone explain? Does that mean the next incremental backup may fail? Anyone game to try?

    EDIT: I tried this and I am happy to report that it works - see post #7 below for details and caveats. There is one major caveat if you are running incremental external backups, but you can still make it work (see post below).
    Last edited by Chopstick; Mar 24, 2022, 08:14 PM.

    Comment


      #3
      Great that you're writing this up. Encryption is fairly new to BTRFS and people need to get the info. Thanks

      Please Read Me

      Comment


        #4
        Originally posted by Chopstick View Post
        And if you were wondering (like me), how you can reclaim any space on your drive with all these snapshots kicking around (since files only really get deleted if there is no snapshot referencing them), I found this post on reddit that basically outlines the process:
        https://www.reddit.com/r/synology/co...al_files_from/
        In a nutshell, you set all snapshots to read-write, use find to delete the file in all snapshots (or a loop or regex, if you know where it is), and set them back to read-only using the btrfs-progs.

        This thread on the btrbk GitHub also talks about this:
        https://github.com/digint/btrbk/issues/221
        The solution is essentially the same, but there is a warning that reads:

        I don't really understand what that means - can anyone explain? Does that mean the next incremental backup may fail? Anyone game to try?
        That's a Btrbk problem:
        https://github.com/digint/btrbk/blob...ed-uuid-is-set
        You probably restored a backup with send-receive, and made it read/write using btrfs property set. This is bad, as all snapshots and backups will inherit this identical "Received UUID", which results in all these subvolumes will be treated as "containing same data".

        To fix this, create a "proper" snapshot:
        # cd /mnt/btr_pool
        # mv mysubvolume mysubvolume.broken
        # btrfs subvolume snapshot mysubvolume.broken mysubvolume


        Now, mysubvolume should have an empty "Received UUID". Note that in order to have a clean environment, you also need to fix all subvolumes (snapshots as well as backups) that you created with the broken subvolume.

        Check if there are more broken subvolumes:
        # btrfs subvolume show mysubvolume.broken
        # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
        # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>


        Either delete them (they won't be used for incremental send-receive anyways), or clean them as follows:
        # btrfs subvolume snapshot listed_ro_subvol listed_ro_subvol.rw
        # btrfs subvolume delete listed_ro_subvol
        # btrfs subvolume snapshot -r listed_ro_subvol.rw listed_ro_subvol
        # btrfs subvolume delete listed_ro_subvol.rw


        Finally, don't forget to delete the broken source subvolume:
        # btrfs subvolume delete mysubvolume.broken

        You should now have a clean environment, and btrbk will not complain any more.
        Here is my / (@) subvolume info:
        Code:
        [FONT=monospace][COLOR=#000000]:[/COLOR][COLOR=#5454ff][B]~[/B][/COLOR][COLOR=#000000]$ sudo btrfs subvol show / [/COLOR]
        [sudo] password for jerry:  
        @ 
                Name:                   @ 
                UUID:                   e11be9cf-95ff-f640-834e-0ed05439b51b 
                Parent UUID:            d2001490-2064-2a4c-a870-1be84141a78a 
                [B]Received UUID:[/B]          - 
                Creation time:          2021-03-15 14:43:28 -0500 
                Subvolume ID:           1695 
                Generation:             969049 
                Gen at creation:        965705 
                Parent ID:              5 
                Top level ID:           5 
                Flags:                  - 
                Snapshot(s): 
                                        snapshots/@202103152204 
                                        snapshots/@202103162149 
                                        snapshots/@202103171903 
                Quota group:            0/1695 
                  Limit referenced:     - 
                  Limit exclusive:      - 
                  Usage referenced:     102.83GiB 
                  Usage exclusive:      541.24MiB[/FONT]
        My @'s "Reserved UUID" is empty. I didn't have to "repair it". I have rarely used " btrfs property set" and use "btrfs su snapshot ...", leaving out the "-r" switch in order to create the @ subvolume as rw.
        mv /mnt/@ /mnt/@OLD
        btrfs su snapshot /mnt/snapshots/@yyyymmdd /mnt/@
        btrfs subvol delete /mnt/@OLD

        Here's what my "Received UUID"'s look like:
        Code:
        [FONT=courier new][FONT=monospace][COLOR=#000000]:~# btrfs subvolume list -a -R /mnt/snapshots | grep "-"             [/COLOR]
        ID 1693 gen 963398 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path snapshots/@202103132115 [/COLOR]
        ID 1694 gen 965705 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path snapshots/@202103142159 [/COLOR]
        ID 1695 gen 969130 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path @ [/COLOR]
        ID 1696 gen 966534 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path snapshots/@202103152204 [/COLOR]
        ID 1697 gen 967268 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path snapshots/@202103162149 [/COLOR]
        ID 1724 gen 968231 top level 5 received_uuid [COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]      path snapshots/@202103171903[/COLOR]
        [/FONT][/FONT]

        So, my @ and all five of my snapshots have empty "Received UUID"'s.
        Last edited by GreyGeek; Mar 18, 2021, 03:34 PM.
        "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
        – John F. Kennedy, February 26, 1962.

        Comment


          #5
          Hm... I am not entirely sure if I understand...
          I did 'btrfs subvolume list -a -R' on my local and external snapshots, the local ones do not have a received_uuid, while the external ones (created by send-receive) have one... external here refers to an external backup HDD and local to my system drive (an SSD).
          Using the additional '-u' flag, I also verified that the received_uuid of the external snapshot is the uuid of the corresponding local snapshot - I guess that makes sense.

          I suppose the issue would arise when doing an incremental backup or restore, where I deleted something in the local snapshot, but not in the remote one, since the parent snapshots are assumed to be identical for both (external and local).

          But I guess deleting the same files in the external copy would make them identical again. In order to make incremental backups work again, I guess just deleting it from the last external snapshot should also be sufficient.

          At least my understanding is that the notion of "incremental" only exists for the purpose of send-receive, not for snapshots on disk (otherwise if would be like git diffs...), so the fact that the "diff" between existing snapshots changed should be irrelevant... and if you restore a backup from a full snapshot, this also does not matter, right? And this is the most likely scenario, if you have to resort to your external backup (if your drive still works, why use an external backup...).

          The received_uuid field basically makes sure we know the exact correspondence between local and external snapshots and we need to make sure that local and remote diffs will be the same for incremental send-receives, right?

          So, for incremental backups to work, just delete the deleted files on the last instance of your external backup (it will still be in the older snapshots) and move on? And maybe make a note not to do incremental send-receive on older snapshots... or does the UUID magically change if I make any changes? Then that would actually break incremental backups and you would have to do a new full send-receive ones and go from there.

          Sorry for the long stream of consciousness...

          Comment


            #6
            If one attempts to send a file from a remote backup to the system volume:

            btrfs send /backup/@yyyymmd2 | btrfs receive /mnt (or where ever)

            and it terminates with an th
            e "Aborted: "Received UUID" is set" error. Redoing send&receive fails again. Now, instead of using "btrfs property set" on /backup.@yyymmd2, which Btrbk says messes up "Receive UUID", it is easier to use, without the "-r" switch:

            btrfs subvol snapshot /backup/@yyyymmd2 /backup/tempvol

            without the "-r" switch, delete /backup/@yyyymmd2 and use

            mv /backup/tempvol /backup/@yyyymmd2


            That process eliminates the "Aborted: "Received UUID" is set" error when you attempt to redo the send&receive' after it failed the first time.

            Btrbk states:

            "Btrbk on the other hand relies on subvolume UUID's being universally unique, and uses them as hash keys for identifying and caching filesystem and subvolume trees, which leads to undefined behavior if multiple identical UUID's are processed."

            That's why it is a Btrbk problem. I've never encountered that error in 5+ years of using Btrfs, so I only know what I read about it.

            I checked the Received UUID's on my /backup drive:
            Code:
            [FONT=monospace][COLOR=#000000]:~# [B]btrfs subvolume list -a -R /backup | grep "-" [/B][/COLOR]
            ID 986 gen 2175 top level 5 [B]received_uuid[/B] d2001490[COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]2064[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]2a4c[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]a870[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]1be84141a78a path @202103142159 [/COLOR]
            ID 988 gen 2181 top level 5 received_uuid eb3bb582[COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]8eb8[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]354c[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]92a0[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]beee97449579 path @202103152204 [/COLOR]
            ID 991 gen 2187 top level 5 received_uuid 339d9140[COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]fc04[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]d64c[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]ac37[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]932265a407c8 path @202103162149 [/COLOR]
            ID 993 gen 2193 top level 5 received_uuid 69f88f80[COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]0c51[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]e244[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]8187[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]16328eba0bf3 path @202103171903 [/COLOR]
            ID 995 gen 2196 top level 5 [B]received_uuid[/B] ecb4b7eb[COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]3edb[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]e74f[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]bf1d[/COLOR][COLOR=#ff5454][B]-[/B][/COLOR][COLOR=#000000]dd9796b44514 path @202103182218[/COLOR][/FONT]
            The Received UUID's are populated and are unique. A link on the btrfs mailing list identifies when "Received UUID", and some other properties, are changed:
            Implement setting received uuid ioctl

            When using send/receive, the received subvolume gets some attributes set
            right before it's made read only: received_uuid, rtime, rtransid, stime
            and stransid. Because receive is user space code, it uses the ioctl to
            do so.

            * The received_uuid is set to the uuid of the subvolume that was sent.
            * rtime and rtransid is set by the kernel code to current time and
            generation of the fs.
            * stime and stransid are provided by us. The receive code puts the
            ctransid value of the sent subvolume into rtransid, and as it seems
            never sets the value of stime, so it's always zero.

            Having this ioctl available also means we can abuse it ourselves to play
            around with it.
            IF I list the uuid's of my snapshots:
            Code:
            [FONT=monospace][COLOR=#000000]:~# [B]btrfs subvol list -u /mnt [/B][/COLOR]
            ID 1694 gen 965705 top level 5 uuid d2001490-2064-2a4c-a870-1be84141a78a path snapshots/@202103142159 
            ID 1695 gen 970765 top level 5 uuid e11be9cf-95ff-f640-834e-0ed05439b51b path @ 
            ID 1696 gen 966534 top level 5 uuid eb3bb582-8eb8-354c-92a0-beee97449579 path snapshots/@202103152204 
            ID 1697 gen 967268 top level 5 uuid 339d9140-fc04-d64c-ac37-932265a407c8 path snapshots/@202103162149 
            ID 1724 gen 968231 top level 5 uuid 69f88f80-0c51-e244-8187-16328eba0bf3 path snapshots/@202103171903 
            ID 1725 gen 969768 top level 5 uuid ecb4b7eb-3edb-e74f-bf1d-dd9796b44514 path snapshots/@202103182218[/FONT]
            All of my snapshot uuid's are populated. You'll notice that the Received UUID's on my /backup snapshots have the same identifiers as the UUID's on my system /mnt/snapshots. Because the @202103182218 on /backup has a Received UUID (ecb4b7eb-3edb-e74f-bf1d-dd9796b44514) that matches the UUID of /mnt/snapshots/@202103182218 Btrbk expects both to have identical contents.

            I'm assuming that Btrbk uses send&receive in is innards to do the sending and receiving. I've never attempted to send a snapshot, say @202103182218, from /backup to /mnt/snapshots, where it could overwrite /mnt/snapshots/@202103182218, but that snapshot on /mnt has an empty Received UUID.

            btrfs send /backup/@2021103182218 | btrfs receive /mnt/snapshots
            or
            better yet:
            btrfs send /backup/@2021103182218 | btrfs receive /mnt/

            Then I could "mv /mnt/@ /mnt/@old" and

            btrfs su snapshot /mnt/@202103182218 /mnt/@

            which creates a rw snapshot labeled @.

            So, I don't understand how modifying /mnt/snapshots/@202103182218 by adding or removing files and/or folders would change the UUID, or if it would. But, Btrbk uses the UUID's as unique hashes. That's all I can say in my stream of consciousness.


            Last edited by GreyGeek; Mar 19, 2021, 08:14 PM.
            "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
            – John F. Kennedy, February 26, 1962.

            Comment


              #7
              By way of update, I did actually try deleting a file from my btrfs drive for good by deleting it from all of my snapshots, as described in this reddit post:
              https://www.reddit.com/r/synology/comments/d1id10/psa_you_actually_can_delete_individual_files_from
              (This reddit post is also linked and mentioned in my second post above and discussed here)

              I am happy to report that it works as described! However, I do seem to recall that I had to run a 'btrfs balance' operation, before I saw most of the space actually getting unallocated and usable.

              There is also one more major caveat to this, which was the main subject of discussion here: if left 'as is', doing this will screw up any incremental backups you may be running on external drives, since after deleting files from backup snapshots, your last local and your remote snapshot will not be identical anymore. As a consequence, sending and incremental diff to the external backup drive will fail.
              There is, however, a simple fix for this: you can simply delete the file(s) in question from the most recent remote backup (not all of them), and the incremental backup should work again (since the local and remote parents are identical again).
              At least for me, using 'btrbk_backup', this works.

              Comment


                #8
                Originally posted by Chopstick View Post
                ....

                I am happy to report that it works as described! However, I do seem to recall that I had to run a 'btrfs balance' operation, before I saw most of the space actually getting unallocated and usable.

                There is also one more major caveat to this, which was the main subject of discussion here: if left 'as is', doing this will screw up any incremental backups you may be running on external drives, since after deleting files from backup snapshots, your last local and your remote snapshot will not be identical anymore. As a consequence, sending and incremental diff to the external backup drive will fail.
                There is, however, a simple fix for this: you can simply delete the file(s) in question from the most recent remote backup (not all of them), and the incremental backup should work again (since the local and remote parents are identical again).
                At least for me, using 'btrbk_backup', this works.
                Excellent point, caveat and solution!

                "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                – John F. Kennedy, February 26, 1962.

                Comment

                Working...
                X