Incremental Backups?

[PhysicistSarah]

I am not sure I fully understand what you are after? Are you saying you only want to format a separate, external hdd as btrfs but not put the OS on it and boot from it? If that's the case things will be pretty straight forward, format the drive as btrfs, create a sub volume and start with your snapshots. Whether your home folder is encrypted or not doesn't really matter in this case.

If, however, you are considering to encrypt your / and /home and use btrfs there are a few things to consider.
First of all, btrfs does not yet work with swap files, so you will need a separate swap partition if you need swap space.
The biggest challenge for me was that the Kubuntu installer does not allow you to encrypt the OS partition and also pick btrfs. The moment you select encryption it will default to ext4 without an option to change the fs.
The only workaround I found was to use the Ubuntu installer and add Plasma manually, not really a clean option though.
I opened a bug but never heard back, so I assume this issue is still there.

But once you got it all set up it will work just fine. My setup is a laptop with 2 internal drives, both encrypted and both btrfs.

Edit: I am not using BackInTime but can confirm that with a btrfs snapshot all individual files are accessible without the need for any additional tools.

Nice. That sounds great. Thanks!

I'll have to look into setting up swap.

[Thomas00]

Hmm, after I format my external HDD as BTRFS I'll attempt the snapshot process.

I still have a few questions though.

My home folder is encrypted using the standard method (I have the key stored in my KeePass for safekeeping and keep a backup of the keepass archive off the disk), but is this a problem for btrfs snapshots? If I snapshot @home can I just drop it on a new install and have it be read by the system or are there additional steps for this?

Also, aside for snapshotting the entire system, are there any major differences between btrfs snapshots and regular BackInTime snapshots? I heard that individual files can't be restored from the backup (Which is a nice feature of BackInTime)

I am not sure I fully understand what you are after? Are you saying you only want to format a separate, external hdd as btrfs but not put the OS on it and boot from it? If that's the case things will be pretty straight forward, format the drive as btrfs, create a sub volume and start with your snapshots. Whether your home folder is encrypted or not doesn't really matter in this case.

If, however, you are considering to encrypt your / and /home and use btrfs there are a few things to consider.
First of all, btrfs does not yet work with swap files, so you will need a separate swap partition if you need swap space.
The biggest challenge for me was that the Kubuntu installer does not allow you to encrypt the OS partition and also pick btrfs. The moment you select encryption it will default to ext4 without an option to change the fs.
The only workaround I found was to use the Ubuntu installer and add Plasma manually, not really a clean option though.
I opened a bug but never heard back, so I assume this issue is still there.

But once you got it all set up it will work just fine. My setup is a laptop with 2 internal drives, both encrypted and both btrfs.

Edit: I am not using BackInTime but can confirm that with a btrfs snapshot all individual files are accessible without the need for any additional tools.

[oshunluvr]

I'm assuming you mean partition encryption as AFAIK that's the only way to encrypt btrfs. Might explain why you're having odd subvolume behavior. You might have mentioned that before.

I don't know exactly what you're asking. I've never used back-in-time but btrfs backups are a copy of a btrfs subvolume. There's no more or less access to your files in the backup than there are in the source subvolume. What I mean by that is, if you want "restore" /some_folders/some_file from your backup you mount the subvolume or the root file system and just locate the file. It'll still be in /some_folders/some_file. If you mount the root file system instead of the subvolume, it would be in some_subvolume/some_folders/some_file.

If you backup to a file, you have to restore the file to a btrfs file system before access it.

BTW, formatting your external HD to BTRFS doesn't effect the "snapshot" process on your root or home subvolumes. Snapshots must reside on the same root file system as the source subvolume. Once you take a snapshot, you may then send it to the external HD but then it would be a "backup" not a snapshot.

[PhysicistSarah]

Hmm, after I format my external HDD as BTRFS I'll attempt the snapshot process.

I still have a few questions though.

My home folder is encrypted using the standard method (I have the key stored in my KeePass for safekeeping and keep a backup of the keepass archive off the disk), but is this a problem for btrfs snapshots? If I snapshot @home can I just drop it on a new install and have it be read by the system or are there additional steps for this?

Also, aside for snapshotting the entire system, are there any major differences between btrfs snapshots and regular BackInTime snapshots? I heard that individual files can't be restored from the backup (Which is a nice feature of BackInTime)

[oshunluvr]

No disrespect taken! My understanding is that snapshots of changed data files - for the purpose of recovery - are saved on the filesystem where the data file resides. That's the "increment" I was speaking of. Sorry, my bad I should have used the snapshot terminology. Now if those can be directed to some other device, again learning takes place.

I'm still not a BTRFS fan it's a more complicated system administration tool than ext4. And if the answer to that is "you don't have to use the tools", then just another reason to not use the filesystem. I get it, BTRFS is probably very useful in complex enterprise systems, that's just not my situation, and RedHat doesn't seem too fond of it either, as there are better systems and methods available for their enterprise solutions.

It's a choice, Linux IS choice, and I make mine with every clean install of Kubuntu on my desktop unit or some other Linux on my laptop.

So therefore, no disrespect taken

Good, I didn't want to you think I was picking on you, but read on ( - kidding of course).

As far as snapshots vs. backups with regards to BTRFS - a snapshot is a file system moment frozen in time, not a copy. When you literally send it to another btrfs file system, it then becomes a backup (an actual copy). An incremental backup is comparing two snapshots of the same subvolume and sending only the differences to an already existing backup.

Snapshots are rather like a record of what a file system looked like when the snapshot was taken and consumes no space by itself - we're talking data space. Think of the file system like a library. Data are books and the metadata is a record of where the books are and what they contain - like the Card Catalog a library has. When you take a snapshot, you aren't adding any books to the shelves, you're creating a second Card Catalog. As you make changes to the source file system (the books), the source Card Catalog changes to keep up. However, the snapshot Card Catalog doesn't change - every book is still the same. If you delete a file (a book) from the source, the book remains on the shelf. Just the card is pulled from the source Catalog. But since the snapshot Catalog still has the card for that book, the book is still there on the shelf and you can still read it - no data space has been released. If you delete the book from the snapshot also - both cards are pulled from the Catalogs and the book is gone for good - and the space it consumed is finally free.

The backups work the same way. The Incremental backup is sent over with only the changes to the cards in the Catalog and any edits to or additions to the library. If you delete the first backup after the incremental backup was sent, the deleted books are gone.

Does all that make any sense? It's a bit to get your head around at first, but once you get it, it's easy-peasy.


Clearly though, we see this topic (EXT vs. BTRFS) differently. Sorry but EXT4 is vastly more complicated to manage well, more time consuming to backup, and snapshot capability is non-existent, and I can go on...

...oh look, I did LOL!

An easy example is a simple package upgrade. They do fail from time to time, leaving us without a usable system. With EXT4 you have to boot in, if you can. Try and figure out what is crashing the system and how to get rid of it, and cross you fingers that it works again. If you can't boot in at all, you find a bootable USB, hope your backup is still good, restore it, and hope it works. With BTRFS, I simply take a snapshot before the upgrade. If it fails but I can still boot, I simply rename my root subvolume to @bad, rename my snapshot to @, and reboot, done. Last time I had to do that, it took less than 15 seconds. If I can't boot back in, I boot to a live USB (or another install as I have several ) and rename the subvolumes, and reboot.

I can't see how EXT4 is easier to administer and I can't imagine a scenario where you can convince me that NOT having snapshots is in anyway better or easy to administer on any system - enterprise or not.

Besides that, just because subvolumes are in use doesn't mean you the user has to avail yourself of accessing them that way (although that's like not using gears 2,3,4, and 5 in your 5-speed car). As a user, you wouldn't even know they were there.

And lets talk about partitions and drives - how many do you have? How often do you have to redo them because of space concerns or changes in your needs? I have one BTRFS file system. It currently has 5 installs on it, but has been up to 8 - no partitioning required. My file system can be 1 partition, 2 partitions, 8 partitions, 1 drive, 4 drives, or any combination of them, function as RAID of almost any level - and I can add or subtract drives or partitions or change RAID levels without rebooting. I have moved entire installs from one drive to another while still using the system. Clearly when it comes to any of these topics - partitioning, multiple devices, RAID, or backups - EXT4 is far more difficult to administer.

My last upgrade included the addition of a new nVME drive. My installs were all on a two-drive single BTRFS file system. For obvious reasons I wanted to move to the much faster nVME drive. Did I backup and restore? Did I clonezilla image and restore? Did I reinstall? Did I boot to a live USB and make a full copy of my partition? None of the above. I simply installed the old drives into the new computer and booted to my KDEneon install there. Formatted the new nVME drive appropriately, sent a backup of all my existing installs and home(s) to the new drive, updated grub, and rebooted. I run from the nVME drive and kept the two older drives to use as backup storage. I never stopped using my system except to reboot to switch to the new drive.

I would really like to know what specifically is better about EXT over BTRFS (there's one example, but I'm not telling! ) except that it's 26 years old so it's officially an antique - LOL. Even the developer of EXT4 has publicly stated that it's a stop-gap comprise to keep EXT usable long enough for a better file system to become the standard. BTRFS tools are still being developed and improve with every kernel drop. EXT looks the same.

But clearly as you stated, Linux is all about choice and you are welcome to yours.

Obviously though, BTRFS is way better unless you don't feel a need to avail yourself of it's advantages. If you see no need for native backup capability, snapshots, using multiple devices, easy and on-the-fly RAID, reduced partition requirements, sharing and combining partitions and drive space, and more - by all means, stick to old faithful ...

...Jumping off my soap box now!

[oshunluvr]

The best explanation of incremental backup I've found is here:https://btrfs.wiki.kernel.org/index....emental_Backup

In a nut shell:

Take a read-only snapshot.
Send it as a full backup.
Don't delete it.
<time passes>
Take another read-only snapshot.
Send it incrementally (compare this snap to the previous snap).
Delete the first snaps, but save the second.
and so on...

Make sense?

As far as your target, yes a backup isn't a backup unless it's less prone to loss than the source. By definition it must be at least on another device, better if on another computer, better yet if they are not co-located. A backup on the same drive is only protection against accidental deletion of files - same as a snapshot in the case of btrfs. If that's all you're after, don't bother with a backup at all, just take snapshots.

Not sure what you mean by "mess around with the boot partition" - snapshots and backups won't "mess" with anything. I also don't get is what exactly you've done with your install. Where's your root mount? Why is so much mounted under /media as if Dolphin was the only way you mount stuff? Sorry, I'm not trying to be critical, I just don't get what you've created. I need some explanation. Also not sure of your "I think I need a file for this?" question. It seems to me you might have made things more complicated than necessary in your mind.

If one was to do a btrfs install of Kubuntu (or any other *buntu flavor) here's what that normally like;

You have an EFI partition (assuming one is using EFI - I usually don't).
You have a swap partition (because BTRFS doesn't handle dynamically created files like swap files yet).
You have a BTRFS partition.

Ubiquity (the installer) creates two subvolumes on the BTRFS partition selected for "/" - one named "@" for the system, one named "@home" for use as home (obviously). The names aren't important, that's just what the developers decided.
As Ubiquity creates the fstab file, is mounts "@" at / and "@home" at /home.

At this point, the user notices nothing different from any EXT4 install except that they got to use a single partition for both root and home, thus allowing all the free space to be equally accessible by the OS and for your personal data. So if you had a 200GB partition and install to it this way, assuming 12GB for the OS and 8GB in data files in your home, both the OS mount and the home mount would show 20GB used and 180GB free. From here, you can make backups to old-fashioned way with a utility or rsync if you like or any other method you like. However, if you're aware of BTRFS and it's benefits, there's a better way (or two).

First, you mount the root BTRFS file system. By this I mean the whole file system, not a specific subvolume. I use "/mnt/subvol" to mount my root file system because, well, it holds the subvolumes. Makes sense to me that way. If you look at fstab after installation you see your subvolumes mounted something like:

# / was on /dev/nvme0n1p3 during installation
UUID=8f0691df-074c-4b0b-ad38-d993d17fef67 /               btrfs   defaults,subvol=@ 0       1
# /home was on /dev/nvme0n1p3 during installation
UUID=8f0691df-074c-4b0b-ad38-d993d17fef67 /home btrfs defaults,subvol=@home 0 2
# swap was on /dev/nvme0n1p2 during installation
UUID=09b3c599-a311-4ad9-b2d0-4fd1d28364dc none swap sw 0 0

I then add my root file system mount like:

UUID=8f0691df-074c-4b0b-ad38-d993d17fef67 /mnt/subvol btrfs   defaults 0       0

Note that everything is exactly the same except for the "subvol=" entry. This is how BTRFS accesses subvolumes as mounts. Now that you've mounted the root file system, you can also access the subvolumes directly as you would any normal folder.

Ok, now that we can see and access the subvolumes, we're free to take snapshots. A snapshot of your "@" subvolume will allow you to roll-back your OS in the event of a bad update or some other catastrophic event. A snapshot of "@home" will preserve files in their present state in case you accidentally delete something or want to see what a document looked like before you edited it. Really easy to do this and what a feature! It will save you someday as it has me more times that I care to admit. Currently, I run a cronjob that takes a snapshot of all my subvolumes daily automatically - just in case.

Once you have snapshots, you can make backups with BTRFS. It's as simple as send and receive. The backups are whole and complete copies of the entire file system contents bit not the file system itself. This difference is important. A file system backup is an image. Images require restoration onto an equally sized partition and have other constraints. A BTRFS backup can be sent to any existing BTRFS file system - as long as it's big enough to hold all the files. Once complete, you can browse your backup and view or copy files just like you did on the source. If desired of needed, you can also make a BTRFS backup to a file - but then it requires expansion to a BTRFS file before accessing it.

Finally - Incremental backups. Like a snapshot, an incremental backup is the whole entire file system contents. I requires a previous snapshot to compare to, then it transmits only the changes to the source file system. You cannot do incremental backups to a file. Incremental backups only send the changes made to the file system in between snapshots, thus the transaction is completed much quicker. The requirement is to keep a previous snapshot of your source subvolume so the incremental operation has something to compare to.

Let me try and explain further:

@home = 20GB on Monday.
You take a snapshot and "send" it as a backup to another BTRFS file system (hopefully on another drive).
The send operation takes 40 minutes (not a real figure - just for this example).
You now have a single full backup copy of your files.
This week you add 2GB of files and delete 2GB of other files, so @home still equals 20GB
You take another snapshot on the following Monday.
Using the "Incremental" backup feature, you send this second snapshot to the same BTRFS backup file system.
This time only 4 minutes to complete the send operation because only the difference in data is sent (the 2GB of new files).
The 2GB of deleted files are also not in the second backup sent.
You "clean up" to recover drive space by deleting the first Monday snapshot at the source and at the backup location.
You now again have a single full backup copy of your files.

You need not delete the previous backups if you wish to keep a history of your files, or delete them periodically if you wish. It depends on your needs.

I know this may not make your questions much clearer, but some fundamental knowledge of how BTRFS stores data is required to have an understanding of how snapshots, backups, and incremental backups work. Please ask for clarity where you require it.

[jglen490]

No disrespect taken! My understanding is that snapshots of changed data files - for the purpose of recovery - are saved on the filesystem where the data file resides. That's the "increment" I was speaking of. Sorry, my bad I should have used the snapshot terminology. Now if those can be directed to some other device, again learning takes place.

I'm still not a BTRFS fan it's a more complicated system administration tool than ext4. And if the answer to that is "you don't have to use the tools", then just another reason to not use the filesystem. I get it, BTRFS is probably very useful in complex enterprise systems, that's just not my situation, and RedHat doesn't seem too fond of it either, as there are better systems and methods available for their enterprise solutions.

It's a choice, Linux IS choice, and I make mine with every clean install of Kubuntu on my desktop unit or some other Linux on my laptop.

So therefore, no disrespect taken

[oshunluvr]

Sarah, I'm busy at the moment but I'll come back here later today and answer.

jglen490: No disrespect, but you completely don't understand btrfs or how it works. Incremental backups are not "partial" backups. They are full backups that do not require re-backing up files already backed up. It's functionally no different than using rsync to make a copy of a set of files, then using rsync again to "update" the copy - something that is done every day and is a well respected and often used method of reducing the time required to make a backup. If the intention is to have multiple full backups as you would if you were backing up to multiple devices, then you would do just that - make multiple backups. Portability - or not - has nothing to do with btrfs or not, that's solely about the device. BTRFS is in the kernel just as EXT4 is, therefore is just as "transportable". In fact, since you can create btrfs backups natively - meaning without outside tools or utilities unlike EXT4 or almost any other filesystem - and you can backup to a filesystem OR to a file for transfer or storage, it seems to me that BTRFS is in foact more portable than other options.

[jglen490]

First I will say that I don't "do" BTRFS, so my knowledge while not zero is purely academic, and my opinions of BTRFS are just that.

What I would like to say is that, in my opinion, relying on a backup mechanism that backs up any portion of the data on the same drive as the data that backup is intended to protect, is not a good idea. This is not to say that there may be a method in BTRFS to direct the backups to some other location - I don't know. Just think this out carefully. In my opinion a viable, high availability backup and recovery method will depend on external storage, whether permanently mounted with the rest of the file systems, or not, or both. If you have a lot of data and some amount of that is very dynamic, at least part of your backup and recovery solution should be external and detachable.

My Kubuntu 18.04 desktop PC rides on an ext4 journaling filesystem. My data is not very extensive and is not particularly dynamic, with the exception of email. My backup solution is to backup only my /home directory structure. That backup is done periodically on a rotating set of three (at this point) USB connected external hard drives. I've chosen SATA 2.5" drives in enclosures which are connected only during the backup activity. I use rsync as my backup software. This will likely not work for you, but that's O.K., it's just an example of what can be done. The final piece of my backup solution is email. The Thunderbird client leaves emails on the server, so I can always recover from the server in the event of a disk death. I don't backup the OS.

I figure the time I don't spend on backing up everything else not in /home, is more than equal to the time it takes for a clean install plus reinstall of any other software.

So that's my story and I'm sticking to it

[PhysicistSarah]

Ok, so I've read the master posts and Grey's post (https://www.kubuntuforums.net/showth...-and-SNAPSHOTS) on subvoumes and incremental backups, but I'm still a bit unclear on how to perform an incremental backup.

I'm guessing that I would do:

btrfs subvolume snapshot -r @home (I think I need a file for this? Would this be ~?) /media/sarah/HomeBackup-03-01-2019
btrfs send -p /media/sarah/ELYSIUM/HomeBackup-01-01-2019    ~/HomeBackup-03-01-2019 | btrfs receive /media/sarah/ELYSIUM/BackupFolder

The only concern that I have when doing this is that it appears to require that the backup is on the same disk that created it. (Which is a problem in my case since my whole system backup would be 600GB and my hard drive is 1TB. I could be wrong about that, but what would be done in this case?

In following the first snapshots tutorial, my /dev/sda1 partition is the boot partition as revealed by lsblk:

[FONT=monospace][COLOR=#54FF54][B]sarah@ConvergentRefuge[/B][/COLOR]:[COLOR=#5454FF][B]~[/B][/COLOR]$ lsblk 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT 
sda      8:0    0 931.5G  0 disk  
├─sda1   8:1    0   512M  0 part /boot/efi 
├─sda2   8:2    0 775.6G  0 part /home 
├─sda3   8:3    0 138.9G  0 part /media/sarah/79F8D858044BC8BB 
└─sda4   8:4    0   819M  0 part  
sdb      8:16   0  22.4G  0 disk  
└─sdb1   8:17   0   3.8G  0 part  
sdc      8:32   0 111.8G  0 disk  
└─sdc1   8:33   0 111.8G  0 part /media/sarah/ELYSIUM 
sdd      8:48   1   7.5G  0 disk  
├─sdd1   8:49   1   1.8G  0 part /media/sarah/Kubuntu 18.04.1 LTS amd64 
└─sdd2   8:50   1   2.3M  0 part  
sde      8:64   1  14.9G  0 disk  
└─sde1   8:65   1  14.9G  0 part  
sdg      8:96   0 931.5G  0 disk  
└─sdg1   8:97   0 931.5G  0 part /media/sarah/SENTINEL 
sr0     11:0    1  1024M  0 rom  
[/FONT]

I'm not entirely sure that I should mount of mess around with the boot parition. Is that what is required to get the tree available for snapshot use?