Announcement

**vinnywright** · Oct 31, 2018, 06:13 PM

Humm are you doing this correctly .

I think you should be doing it , a bit more like this .

take a read only snapshot of lets say @home ,,, use btrfs send AND btrfs receive to transfer it .

see hear https://www.kubuntuforums.net/showth...l=1#post419738

VINNY

EDIT : oops you were over their already

**GreyGeek** · Oct 31, 2018, 08:54 PM

Your problem would be resolved if you could get a snapshot of @ and @home on your remote server. Then you could use the BTRFS incremental backup method to do periodic backups that wouldn't take much time at all.

First, I would focus on why your network is "disappearing". Is your server a LAN or WAN? Is it connected via Ethernet cable or wifi? IF it is wifi why is the connection dropping? Due to overload? What does modinfo give for your wifi driver that might give you some control over the work load?

**Thomas00** · Nov 01, 2018, 07:46 AM

Thanks for your responses guys, I had incremential backups on my list of todos anyway, and you are right the dropping network is a good reason to bring this forward. Need to do some more reading and testing first though. The way I understand it is that I will always need backup1 (= an exact copy of the subvolume) plus the most recent incremental backup. But like I said, I haven‘t fully wrapped my mind around this.
Also not sure yet, how I would do a restore with two backups in play.
Anyway, I will figure this out...

First, I would focus on why your network is "disappearing". Is your server a LAN or WAN? Is it connected via Ethernet cable or wifi? IF it is wifi why is the connection dropping? Due to overload? What does modinfo give for your wifi driver that might give you some control over the work load?

This only occurs in this particular backup situation, at least so far, situation, and only when on wifi. Means it‘s more of an annoyance than an issue and hence went to my todo list :-)
Probably should start a new thread...

Server is connected via Ethernet, the issue occurs only when the laptop is on wifi (so far), the situation afterwards is that all wifi networks are gone, the system tray network app is empty, like if it was switched off. But I haven't really had time to look into this. Not sure what to do with modinfo, it wants parameters but I don't know which?

And like I said, this network issue is not top of my list right now:

@hermes:~$ t ls | grep Kubuntu
17 @Kubuntu Distorted logon screen in 2 monitor scenario
23 @Kubuntu Fix Todo.txt Dropbox issue
16 @Kubuntu Plasma crash / blinking Cursor after changing from docked to undocked
22 @Kubuntu Plasmashell grabs one cpu when machine is left alone
21 @Kubuntu SSD disapears -> OS crashes
19 @Kubuntu System wont hibernate when plugged into power
20 @Kubuntu Update backups script to use incremental backup
18 @Kubuntu Wifi Network stack drops while large transfer is on

**GreyGeek** · Nov 01, 2018, 01:55 PM

Originally posted by Thomas00 View Post

... I had incremential backups on my list of todos anyway, ..... The way I understand it is that I will always need backup1 (= an exact copy of the subvolume) plus the most recent incremental backup. ...

Not quite ..

An incremental backup is not a "piece" of the subvolume being backed up. It becomes the entire subvolume.

Assume that yesterday you created a snapshot of @home: @home20181031. And, you also sent that entire snapshot to your backup drive, also formatted with Btrfs. So, both your <ROOT_FS>/mnt/snapshots and /backup/snapshots has @home20181931 That will be the last time you will need to send an entire snapshot to the backup archive.

Today you create a snapshot of @home: @home20181101. You are now ready to do an incremental backup of @home20181101, which is not on /backup/snapshots.

btrfs send -p /mnt/snapshots/@home20181031 /mnt/snapshots/@home20181101 | btrfs receive /backup/snapshots

What happens: "btrfs send -p" compares @home20181031 with @home20181101 and sends only the DIFFERENCES to the receive command.

Btrfs receive is informed that @home20181031 is being used as the base subvolume and it searches /backup/snapshots for that subvolume. IF it finds it then it immediately creates on /backup/snapshots a copy of @home20181031 but labels it as subvolume @home20181101. Then receive begins changing @home20181101 according the the contents of the differential stream until the copy of @home20181101 on /backups/snapshot is identical to @home20181101 on /mnt/snapshots.

Sending the entire subvolume might take 30 minutes or more depending on the size of the subvolume. Sending the incremental difference can take as little as a minute or less. Daily incremental snapshots would be very quick for most users. Weekly incremental backups will take longer, but not nearly as long as a full backup. How long it takes depends on how much change in your system was made between the snapshot on 20181031 and that made on 20181101. I do "when I feel like it" snapshots, and all my archival backups are incremental. Depending on what I'm doing (massive update, massive data deletion, adding or removing significant software like dev tools, etc) I may make two or three snapshots on a single day. Or, I may go two weeks without making snapshots if all I am doing is browsing and emailing.

Keep in mind that the suggested limit to the number of snapshots saved under /mnt/snapshots (or wherever you put them) is eight per subvolume. Also keep in mind that while the initial size of a snapshot is essential zero, as you make changes in your system a copy of the files as it exists, before the changes are made, is sent to the snapshot. Such changes gradually begin to fill the snapshot up. Older snapshots are much fuller than more recent ones. Not only that, if you have more than one snapshot of @home in /mnt/snapshots and make a change to file A, then every snapshot of @home will be updated with a copy of A. And, also keep in mind that when you snapshot your system it is recommended that you snapshot BOTH @ and @home at the same time, using the same label extension so that if a rollback becomes necessary you won't get a mismatch between different @ and @home snapshots. And, IF you delete file A, a copy of it still exists in one or more @home snapshots. IOW, snapshots do NOT shrink as you delete files from your system.

When/If you defragment, data is copied from the old extents into fewer new extents. The new extents are distinct from the old extents. All other copies of the files in existing snapshots still point to the old extents. Therefore, you have snapshot bloat. Your free disk space is shrinking. Using autodefrag in the mount lines of @ and @home do the same thing, on a file by file basis as changes occur. The net result is that older snapshots tend to enlarge and approach the size of the subvolume they were made from as time progresses. My approach to this is to keep a maximum of 5 snapshots each for @ and @home. Before I create a new snapshot pair I always delete the oldest snapshot pair first, then increment the new pair to /backup/snapshots.

IF, for some reason I want to run the btrfs balance command I first make sure my most recent snapshots are safely in archive. I can browse them with Dolphin to be sure. Then, I delete all my /mnt/snapshots before running the balance command. After the balance command is completed I do a reboot to make sure it can reboot. Then, I create new @ and @home snapshots and increment them to /backup/snapshots. IF I were not using autodefrag in fstab I would also delete all my snapshots before running the btrfs defragment command as well. Then reboot and make new @ and @home snapshots.

Keep in mind that while you are using any of these commands you can continue to use your system and it will continue to perform with little or no degradation.

**Thomas00** · Nov 01, 2018, 04:32 PM

Wow, quite a thorough response! Thanks for taking the time GreyGeek.

I think I understand the concept of incremental backups, have been doing them for I guess thirty years.
However, traditional incremental backups are chained, means each increment holds only the delta, the files which changed since the last increment was created.
For a full restore, all elements of the chain down the the very first backup are required.

I understand that Btrfs is different in that each differential backup is in fact a copy of the previous backup + the increment.
I assume this follows the same concept like a snapshot where identical files are stored only once but two snapshots may refer to it.
So each chunk of data is stored only once no matter if multiple backups refer to it.
This answers my question regarding a restore, itÂ‘s always the latest incremental backup only which is required for the restore. No need to juggle with multiple increments.
I can even delete the previous backup as in essence I would only delete links to files, at least for those which are shared in both backups.
I think I got it.

However, what I am not clear about is how this will work if the backup drive is on the network and not attached to the same machine? My server drives are btrfs but I canÂ‘t mount them on my laptop as btrfs but have to use Samba or NFS.
I might still be able to send the increment but the output will have to be a text file.
IÂ’d be very surprised if the incremental text file would be linked to the previous text file, there is no btrfs receive which could do the mingling. So either incremental is not possible over a network or I will end up with a chained backup where each text file holds the delta only and in the case of a restore needs to be applied one after another.
HavenÂ‘t tried this yet but can do so tomorrow...

As far as the number of snapshots is concerned, I am aware that the number needs to be kept low, I read your initial btrfs post thoroughly :-)
In fact I have started a script which will automatically create my backups and delete the oldest snapshots if more than 5 snapshots are found.
Well, I am actually creating 3 instances of my data:

Snapshot in /mnt/snapshots
Backup to a second internal hdd
Backup to a remote server

Guess I am a bit paranoid about my backups :-)

And yes, I always process both, @ and @home the same way, when creating, copying and deleting.
HavenÂ’t run a defrag yet but will keep your advice in mind when I do...

Edit:
As promised I did try differential with file output today:

sudo btrfs send -f /media/thomas/Windows10/differential.txt -p /mnt/snapshots/@_2018-10-31 /mnt/snapshots/@_2018-11-02

This did work and created a 5GiB text file for @home! Cool!
This also confirms that things work on a binary and not file level as otherwise the file should have been at least 80GiB which is the size of my Windows VM which I surely opened in the last two days.
Question remains though how this will play out in a restore scenario? I can see two options:
1. Either one has to manually restore each differential backup from oldest to current, similar to what a regular, chained incremental backup would do
2. Or the individual differential backups will overwrite the previous ones which means a full restore is not possible this way.

Decided to stop here and not do restore tests because I think for my use case having file level access to my backups is more important. Hence it's going to be rsync for snapshots to non btrfs targets.

**GreyGeek** · Nov 02, 2018, 02:31 PM

Originally posted by Thomas00 View Post

...

Edit:
As promised I did try differential with file output today:

sudo btrfs send -f /media/thomas/Windows10/differential.txt -p /mnt/snapshots/@_2018-10-31 /mnt/snapshots/@_2018-11-02

This did work and created a 5GiB text file for @home! Cool!

Decided to stop here and not do restore tests because I think for my use case having file level access to my backups is more important. Hence it's going to be rsync for snapshots to non btrfs targets.

For the most part I think you've got it!

I do have one question about your successful backup using the "-f" parameter, but your -p paramter looks fine. How did it work? How was "differential.txt" created, given that you don't show a source? I can't see it.

The use of the -f parameter in the Btrfs send command is as follows:

sudobtrfs send <source subvol> -f <destination ascii file>

You are lacking a source subvolume for the -f parameter. (BTW, isn't your snapshot of @ and not of @home?

In my guide I gave an example using the -f paramter:

IF your destination is not formatted with BTRFS then your send command should be:

sudo btrfs send /mnt/snapshots/@_basic_install -f /media/@_basic_install.txt

where /media is a mounted USB stick formatted with EXT4 or FAT32, or a network destination.

The "-f" option sends the subvolume to the destination as a stdout file, i.e., ASCII.

To convert it back to a subvolume one would use:

sudo btrfs receive -f /media/@_basic_install.txt /mnt/snapshots

The received subvolume will be set to read-only. Use the mv command to remove the ".txt" extension.

sudo mv /mnt/snapshots/@_basic_install.txt /mnt/snapshots/@_basic_install

To add the write attribute use
sudo btrfs property set -ts @_basic_install ro false

This source may be of use to you:
https://wiki.samba.org/index.php/Server-Side_Copy

**oshunluvr** · Nov 02, 2018, 04:01 PM

Ok, a lot of detail here but I have a couple general comments that may or may not have already been addressed:

#1 Backing up to a file (btrfs send with the -f option) is generally used to prevent having to backup over a network or to removable device, not to combine with a network mount or removable device. In other words there's no advantage to sending a backup to a file over a network to a mount. I would make the backup local, then copy to the backup location (rsync is fine but cp works too). The reason I say this is exactly what you're experiencing - an interrupted btrfs send operation is not a backup (it's trash) and not easily detected. If you copy a file, at least you have an error telling you the copy operation was interrupted. This is a known problem and will be addressed in the future. The one instance you would have to use a file is if the destination file system is not btrfs. Then a file backup is required, but I'd still use a backup-to-file THEN copy method if the destination was not local.

#2 I agree that if you're not using btrfs on your backup media, you might be better off just doing an old fashioned rsync backup. No real advantage to using send-to-a-file if you want to retain file level access to your backups.

#3 I don't know if incremental backups would even work when using the backup-to-a-file method or how you would go about restoring them successfully.

The backup-to-a-file option was really intended for use with removable devices. Incremental backup is an awesome and easy way to backup if you're using btrfs backup storage, but if you're not there no advantage to using send at all IMO.

**Thomas00** · Nov 02, 2018, 04:21 PM

I do have one question about your successful backup using the "-f" parameter, but your -p paramter looks fine. How did it work? How was "differential.txt" created, given that you don't show a source? I can't see it.

Good question, I guess the output of -p becomes the source?
My previous command was "btrfs send -v -f /media/clientbackup/$machine/@_$(date +%Y-%m-%d).txt /mnt/snapshots/@_$(date +%Y-%m-%d)"
Then I tried " btrfs send -f /media/thomas/Windows10/@home-differential.txt -p /mnt/snapshots/@home_2018-10-31 /mnt/snapshots/@home_$(date +%Y-%m-%d", I simply assumed that everything after -p becomes the source.

I tested with @ and @home, @ gave me 400MiB file and @home lead to 4.9GiB. I did use my dormant Win10 partition instead of the file server because I figured it doesn't matter if it's a network drive or NTFS, what matters is if the target is btrfs or not.

**Thomas00** · Nov 02, 2018, 04:29 PM

@oshunluvr Glad to read your post as this is exactly where I arrived at, just took me a while to get to this conclusion

#1 Is what I ended up with, first do a snapshot, second copy the snapshot to a separate drive local to the machine, third send the snapshot via rsync (or cp) to a remote location.

**oshunluvr** · Nov 02, 2018, 06:28 PM

Originally posted by Thomas00 View Post

@oshunluvr Glad to read your post as this is exactly where I arrived at, just took me a while to get to this conclusion

#1 Is what I ended up with, first do a snapshot, second copy the snapshot to a separate drive local to the machine, third send the snapshot via rsync (or cp) to a remote location.

Yup, that's the best for your use-case IMO.

Cool thing is - you can still use incremental snapshots to the local drive - thus shortening the time spent waiting for that to finish, then do a full backup-to-file from the incremental snapshot and copy that to the remote backup location. Just make the backup-to-file from the last incremental backup and you'll have a full and latest backup-to-file version on the external drive and a set of incremental backup snapshots on the local drive. Best of both worlds.

**GreyGeek** · Nov 02, 2018, 07:31 PM

Well, Thomas00, the Btrfs guru is back, and I suspect he is very familiar with using the -f formatting option of the send command, and especially using rsync, cp and the like.

Going with #1, your commands would most likely be similar to:

First, create read only snapshots of @ and @home:
btrfs su snapshot -r /mnt/@ /mnt/snapshots/@_$(date +%Y-%m-%d)
btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home_$(date +%Y-%m-%d)

Now, create txt files of the ro snapshots:
sudo btrfs send /mnt/snapshots/@20181102 -f /mnt/snapshots/@_20181102.txt
sudo btrfs send /mnt/snapshots/@home20181102 -f /mnt/snapshots/@home_20181102.txt

And from there you'd cp the txt files to the remote server.

IF you've deleted those txt files from your system and wanted to convert those text files back to subvolumes on your system then cp them back to /mnt/snapshots/ and use:

btrfs recieve -f /mnt/snapshots/@_20181102.txt /mnt/snapshots/@_20181102
etc for @home_20181102

and before mv'ing them to @ and @home change them to rw using
btrfs property set -ts @_20181102 ro false
btrfs property set -ts @home_20181102 ro false

As it turns out, using the "list" command of "btrfs property" shows that the only property you can get or set on subvolumes is the ro property. "-t" means type, and "s" means subvolume.

I guess that it goes without saying that txt files created with the "-f" parameter are not changed when the subvolume used to create them does change. And, I suspect, that even using the "receive -f" command to convert txt files back into subvolumes does not make them aware of the system subvolume from which they were derived. (That's an educated guess).

EDIT: Saw oshunluver's post about doing incremental backups to your own system drive, i.e., /mnt/snapshots (?).

I'm not sure how that would work. Say, using yesterday's snapshot as the parent:

btrfs send -p /mnt/snapshots/@_20181101 /mnt/snapshots/@_20181102 | btrfs receive /mnt/snapshots

wouldn't work because the -p command expects both the parent and the child snapshot to be present for the "send -p" command. The "receive" command on the remote server expects to see yesterday's snapshot and will create today's by adding the differential stream to yesterday's snapshot. If you take a snapshot of today to use as a comparison to yesterday's snapshot then there is no need to be incremental, since snapshots only take a fraction of a second.

The method would work IF you had second storage device mounted to /, say /backup, but then it would be the usual method for incremental backups, which I use all the time to back up snapshots to my 2nd and 3rd Internal drives, which are not part of my Btrfs root subvolume.

**Thomas00** · Nov 03, 2018, 02:24 AM

Well, let me share where I am at right now script wise.
Disclaimer, this is work in progress, I am testing and changing with every time I run the script and I am by no means a bash script programmer.

Maybe there's some inspiration this can provide and maybe someone can identify more elegant ways of doing things.

Print is a little function which formats my screen and log output the way I want it, nothing of importance for this subject
Deletesnapshots is another function which counts how many snapshots/backups exists and in case there are more than the 5 I set will delete the oldest one

First create the snapshots

Code:

StartScript=$(date +%S)
echo "#########################################"
print "Checking snapshot folder for old snapshots"
deletesnapshots /mnt/snapshots 1
echo "#########################################"
print "Creating snapshots"
snap="/mnt/snapshots/@_$(date +%Y-%m-%d)"
if [ ! -d $snap ]; then
 btrfs su snapshot -r /mnt/@ /mnt/snapshots/@_$(date +%Y-%m-%d)
 btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home_$(date +%Y-%m-%d)
 print "Snapshots created!"
else
 print "E: Snapshots do exist already, did not create new snapshots!" 
fi

Now I have copy #1 in my root drive / disk 1
Next is a differential backup to the internal drive 2, mounted on /media/data

Code:

 print "Check number of snapshots in $LocalBackupLocation"
 deletesnapshots $LocalBackupLocation

 snap="$LocalBackupLocation/@_$(date +%Y-%m-%d)"
 if [ ! -d $snap ]; then
     print "Send @"
     files=($LocalBackupLocation/@_*)
     ii=${#files[*]}
     btrfs send -p ${files[$ii-1]} /mnt/snapshots/@_$(date +%Y-%m-%d) | sudo btrfs receive $LocalBackupLocation      
     print "Send @home"
     files=($LocalBackupLocation/@home_*)
     ii=${#files[*]}
     btrfs send -p ${files[$ii-1]} /mnt/snapshots/@home_$(date +%Y-%m-%d) | sudo btrfs receive $LocalBackupLocation
     print "Completed local backup"
 else
     print "E: Backup does already exist, did not create new backup!" 
 fi

Now the remote backup to my file server

Code:

echo "#########################################"
print "Starting server backup..."

print "I: Wakeup server"
# Wake up in case Server is asleep
etherwake -D -i wlp2s0 bc:ae:c5:29:79:b7
print "Waiting for Homeserver...10 Seconds..."
sleep 10s

mount /media/clientbackup
go=$?
echo $go
#Drive already mounted?
if [ $go -eq 32 ]
then
mounted=$(mount | grep "clientbackup")
if [ "$mounted" ] 
then 
 go=0
 print "W: Drive already mounted..."
fi
fi

if [ $go -ne 0 ]
then
print "E: Mount error $?"
print "E: Backup aborted!"
exit 
fi

#print "Check number of snapshots in /homeserver/clientbackup"
#deletesnapshots /media/clientbackup/$machine

print "Send @"
rsync -avP --exclude=run --exclude=media --delete /mnt/snapshots/@_$(date +%Y-%m-%d)/   /media/clientbackup/Hermes/@
print "Send @home"
rsync -avP --exclude=Trash --exclude=lost+found --exclude=.cache --exclude=.mozilla --exclude=.trash--delete /mnt/snapshots/@home_$(date +%Y-%m-%d)/ /media/clientbackup/Hermes/@home

print "Completed server backup"
umount /media/clientbackup

The rsync backup could be done in two ways I guess:
1. Always sync into the same folder, /mnt/snapshots/@_$(date +%Y-%m-%d)/
2. Create separate backup copies, /mnt/snapshots/@_$(date +%Y-%m-%d)

The difference is in the "/" at the end.

Benefit of #1 is that it will be fast from second run onwards as only updates will be copied, downside is that I won't have older versions
Benefit of #2 is that I keep older versions but I will have to delete older folders and copy all data each and every time.

For now I am going with #1, but what I want to do at some point is to switch to #2 but not deleting the oldest backup but rather renaming it to the incoming one before I do the rsync. Expectation is that even in the oldest backup most files won't have changed and hence don't need to be copied again. So much for the theory :-)

Thomas

**oshunluvr** · Nov 03, 2018, 06:29 AM

Incremental snapshots require one previous snapshot to compare to and the same on on the backup location(s) You can delete earlier ones as necessary. Example:

source file system:
@
@-20181101
@-20181102
@-20181103

backup file system:
@-20181101
@-20181102
@-20181103

where @-20181101 is a full backup and the subsequent snapshots are done incrementally.

You could now delete @-20181101 and @-20181102 in both locations. Your next incremental would be @-20181104 and after it's complete @-20181103 could be deleted.

In my view, here's the real benefit of btrfs incremental backups over a non-btrfs (backup-to-file) operation: On the backup file system you can keep all the backup snapshots all long as you wish or as long as your backup space holds out. This provides a daily (or whatever interval you choose) "history" if you will, of your data transactions. Alternately, you could combine intervals and keep, let's say, the last seven daily snapshots, one each weekly snapshot for two months, one each monthly snapshot for six months. Then you'd have a string of 22 or so snapshots reaching back 6 months. And you'd still only need to keep one previous snapshot on the source file system.

Remember, each snapshot shares data space with the previous ones so each new incremental snapshot only contains the difference between it and it's previous snapshot(s). Your backup space requirements would depend on how much change you data underwent, but it would be less than you think I suspect. If you began with 200MB of data, you'd have to cycle through (add, edit, or delete) a full additional 200MB of data before your backup space would require only double the amount of original data. Normal users (IMO) rarely manipulate 100% of their data in a year. More than 80% of my data is simple storage (videos, pictures, etc.) that doesn't change very often at all. As an example, these snapshots and their sizes:

@-20181101 = 200MB
@-20181102 = 2MB
@-20181103 = 4MB
@-20181104 = 3MB

The total data space used is 209MB. Assuming each snapshot contains 1MB of new data and the remainder is deletions. If I delete @-20181101, my numbers now look like:

@-20181102 = 201MB
@-20181103 = 4MB
@-20181104 = 3MB

So I recovered 1MB of space - the amount I deleted between 1101 and 1102. To continue;

@-20181103 = 202MB
@-20181104 = 3MB

Now I've recovered another 3MB of space and @20181103 contains all the files from 1101 and 1102 except those I deleted in between. Stretching this out further, if I on average add 1MB of data per day and delete 1MB per day, that's only 2MB of transactions per day. So reaching 400MB of needed backup storage would require about 100 days of backups. That's a lot of backup history. Obviously, YMMV depending on the amount of daily transactions you incur.

On the other hand, backing up to a file would mean only 2 200MB backups to reach 400MB of required storage. If - and a big if as I've never tried or researched it - you can do incremental backup-to-file operations, can you imagine having to restore 100 days of incremental backups to reach today? Since each incremental backup is basically a "diff" file, you'd have to keep each one (again - not positive here as I don't even know if it possible to do this) and restore them in order to reach a full recovery. Yikes.

With btrfs backups, you'd need only keep as many backups as you wish to have history - and the latest one if you did incremental backups.

Announcement

btrfs send -f vs. rsync

[SOLVED] btrfs send -f vs. rsync

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment