Announcement

**Snowhog** · Mar 04, 2022, 04:54 PM

GreyGeek Okay, after changing the HDD IDs to reflect mine, and commenting out the mounting line for the external HDD (as already explained, I accept mounting when it is plugged in), I ran the script. The results:

paul@barley-cat:~$ sudo -i
root@barley-cat:~# cd /
root@barley-cat:/# /make_snapshot.sh
Mounting drives
mount: /mnt: special device /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a434156e does not exist.
Making today's snapshot
Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203041643'
Finding previous snapshot as parent
Attempting incremental backup
btrfs send -p /mnt/snapshots/@202203041643 /mnt/snapshots/@home202203032257 | btrfs receive /backup
At subvol /mnt/snapshots/@home202203032257
At snapshot @home202203032257
ERROR: parent subvol is not reachable from inside the root subvol
Delete subvolume (commit): '/mnt/snapshots/@202203032257'
ERROR: Not a Btrfs subvolume: Invalid argument
Snapshots completed, oldest snapshots deleted
umount: /backup: not mounted.
Drives unmounted and 3 syncs done
root@barley-cat:/#

The unmount: /backup: not mounted is my bad. I didn't change the eval 'umount /backup' line.

As to the to ERROR lines. Given my setup, what do I make of them?
The mount: /mnt: error is my bad. Didn't write out the UUID correctly.

**GreyGeek** · Mar 04, 2022, 05:02 PM

I always used the UUID and never thought about using the label. I have given the label "BACKUP" to my internal NVMe SSD and to an external USB HDD as well. They both cannot be mounted at the same time as "BACKUP". So, before I plug in my external USB SSD labeled "BACKUP" the listing is:

Code:

$ vdir /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 15 Mar 4 07:26 BACKUP -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Mar 4 07:26 DATA -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Mar 4 07:26 EFI -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 4 07:26 ROOTFS -> ../../sda3
lrwxrwxrwx 1 root root 10 Mar 4 07:26 SWAP -> ../../sda2

AFTER I plug in the external SSD the listing shows:

Code:

$ vdir /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 Mar 4 16:46 BACKUP -> ../../sdb1
lrwxrwxrwx 1 root root 15 Mar 4 16:46 DATA -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Mar 4 16:46 EFI -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 4 16:46 ROOTFS -> ../../sda3
lrwxrwxrwx 1 root root 10 Mar 4 16:46 SWAP -> ../../sda2
~$

The NVMe is hidden. If I were to use

Code:

$ sudo -i
# mount BACKUP /backup                      
mount: /backup: special device BACKUP does not exist.
# mount /dev/disk/by-label/BACKUP /backup
# vdir /backup
total 0
drwxr-xr-x 1 root root 334 Feb 18 20:55 @202202182046
drwxr-xr-x 1 root root 334 Feb 23 19:11 @202202231903
drwxr-xr-x 1 root root 334 Feb 24 20:57 @202202242057
drwxr-xr-x 1 root root 334 Feb 25 19:18 @202202251913
drwxr-xr-x 1 root root 334 Feb 26 20:41 @202202262040
drwxr-xr-x 1 root root 334 Feb 28 16:10 @202202281609
drwxr-xr-x 1 root root 334 Mar  1 18:15 @202203011815
drwxr-xr-x 1 root root 334 Mar  2 22:13 @202203022213

The external drive gets mounted.

Code:

/dev/sdb1 on /backup type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

So, I could mount the internal BACKUP or the external backup using

Code:

mount /dev/disk/by-label/BACKUP /backup

and which one gets mounted would be determined by plugging in the external SSD or not.
Nice!

PS: The UUID mistype plagued me until I restored to just using blkid in the Konsole and then C&P to put it onto the command line or into a script. Using the BACKUP label is going to change all of that. I can run the same script, first without the external SSD plugged in, and then with the external SSD plugged in.

**Snowhog** · Mar 06, 2022, 09:11 AM

GreyGeek I gave my external USB HDD a label (BACKUP) and made the necessary reference changes to the script.

I'm also unmounting the external USB HDD so I can then mount it at /backup

This is the script:

#!/bin/bash
# make_snapshot.sh

# Script to create backup snapshots to /mnt/snapshots and a differential backup to /backup
# To be run as root from /
#
# Before executing this script:
# Plug in external HDD. Select Mount in Disk & Devices pop-up
# External drive will be mounted to /media/paul/BACKUP
#
echo "Mounting drives"
# mount internal HDD to /mnt
eval "mount /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a43941568e /mnt"
# Unmount external HDD so it can be remounted to /backup
eval "umount /media/paul/BACKUP"
# mount external HDD to /backup
eval "mount /dev/disk/by-label/BACKUP /backup"

# capture the date and time
NOW=$(date +%Y%m%d%H%M)
echo "Making today's snapshot"
MKSNAP='btrfs su snapshot -r /mnt/@ /mnt/snapshots/@'$NOW
eval $MKSNAP
eval 'sync'
eval 'sync'

echo "Finding previous snapshot as parent "
PREVSNAP=""
list=($(ls /mnt/snapshots/))
PREVSNAP=${list[-2]}
NOW=${list[-1]}
echo "Attempting incremental backup"
if [[ -s "/mnt/snapshots/"$PREVSNAP ]];
then
MKINC='btrfs send -p /mnt/snapshots/'$PREVSNAP
MKINC=$MKINC' /mnt/snapshots/'$NOW
MKINC=$MKINC' | btrfs receive /backup'
echo $MKINC
eval $MKINC
eval 'sync'
eval 'sync'
eval "sync"
DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
eval $DELSNAP
eval 'sync'
eval 'sync'
eval 'sync'
list=''
list=($(ls /backup))
DELSNAP='btrfs subvol delete -C /backup'${list[0]}
eval $DELSNAP
eval 'sync'
eval 'sync'
eval 'sync'
echo "Snapshots completed, oldest snapshots deleted"
eval 'umount /backup'
eval 'umount /mnt'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
echo "Drives unmounted and 3 syncs done"
else
echo 'Incremental backup failed using '$PREVSNAP' and '$NOW
echo 'Drives still mounted, ready to clean up'
fi

I start by plugging in my external USB HDD and selecting Mount in the Disk & Devices pop-up. Then I open a konsole and issue:

sudo -i
cd /
/make_snapshot.sh

This is the output from this morning:

paul@barley-cat:/$ sudo -i
root@barley-cat:~# cd /
root@barley-cat:/# /make_snapshot.sh
Mounting drives
Making today's snapshot
Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203060645'
Finding previous snapshot as parent
Attempting incremental backup
btrfs send -p /mnt/snapshots/@202203060645 /mnt/snapshots/@home202203032257 | btrfs receive /backup
At subvol /mnt/snapshots/@home202203032257
At snapshot @home202203032257
ERROR: cannot find parent subvolume
Delete subvolume (commit): '/mnt/snapshots/@202203042120'
ERROR: Could not statfs: No such file or directory
Snapshots completed, oldest snapshots deleted
Drives unmounted and 3 syncs done

Why am I getting those two ERROR's?

**GreyGeek** · Mar 06, 2022, 12:53 PM

Don't do this part:

# Plug in external HDD. Select Mount in Disk & Devices pop-up
# External drive will be mounted to /media/paul/BACKUP
#

All you have to do is plug the external HDD in for it to be registered in /dev/disk/by-label as "BACKUP"

My script is for backing up just @, because I don't have a @home subvolume. I merged it into @/home and took out the line in fstab that mounted it. I wrote that script after I had merged the subvolumes.

The script you modified and ran tried to this:

btrfs send -p /mnt/snapshots/@202203060645 /mnt/snapshots/@home202203032257 | btrfs receive /backup

Notice that the parent is @202203060645 BUT the child is @home202203032257
That doesn't work because the list command:

list=($(ls /mnt/snapshots/))

doesn't not distinguish between @ and @home snapshots.

IF you are going to keep both @ and @home then I would create two separate subdirectories:
/mnt/snapshots/root
/mnt/snapshots/home
and store my @yyyymmddhhmm snapshots in the root subdir and @homeyyyymmddhhmm in the home subdir.
Then, add a section of code that picks out the root parent and root child, and ditto for the home parent and the home child.
That way the list command can properly identify the previous snapshot as parent.

**Snowhog** · Mar 06, 2022, 01:09 PM

Thank you for the explanation as to why "I" get those errors.

The laptop is sacrificial; nothing critical on it. It's for playing/experimenting on.

I already figured out the "Don't do this part". When I plug in my external USB HDD, the Disk & Devices pop-up appears, but I (now) just ignore it. The drive is powered and identified by the system, as you point out. So that area of the script now reads:

# Before executing this script:
# Plug in external HDD. Ignore Disk & Devices pop-up.
# External drive will be powered and identified by system but not yet mounted.
#
echo "Mounting drives"
# mount internal HDD to /mnt
eval "mount /dev/disk/by-uuid/67fab1ac-b53c-4e10-9faf-f9a43941568e /mnt"
# mount external HDD to /backup
eval "mount /dev/disk/by-label/BACKUP /backup"

Given my setup; separate @ and @home; how would the following part of the script be written so it all works?

# capture the date and time
NOW=$(date +%Y%m%d%H%M)
echo "Making today's snapshot"
MKSNAP='btrfs su snapshot -r /mnt/@ /mnt/snapshots/@'$NOW
eval $MKSNAP
eval 'sync'
eval 'sync'

echo "Finding previous snapshot as parent "
PREVSNAP=""
list=($(ls /mnt/snapshots/))
PREVSNAP=${list[-2]}
NOW=${list[-1]}
echo "Attempting incremental backup"
if [[ -s "/mnt/snapshots/"$PREVSNAP ]];
then
MKINC='btrfs send -p /mnt/snapshots/'$PREVSNAP
MKINC=$MKINC' /mnt/snapshots/'$NOW
MKINC=$MKINC' | btrfs receive /backup'
echo $MKINC
eval $MKINC
eval 'sync'
eval 'sync'
eval "sync"
DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
eval $DELSNAP
eval 'sync'
eval 'sync'
eval 'sync'
list=''
list=($(ls /backup))
DELSNAP='btrfs subvol delete -C /backup'${list[0]}
eval $DELSNAP
eval 'sync'
eval 'sync'
eval 'sync'
echo "Snapshots completed, oldest snapshots deleted"
eval 'umount /backup'
eval 'umount /mnt'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
echo "Drives unmounted and 3 syncs done"
else
echo 'Incremental backup failed using '$PREVSNAP' and '$NOW
echo 'Drives still mounted, ready to clean up'
fi

I know that's asking you to make the changes for me, and I hope you don't mind. It's that, or instruct me on how to merge (?) my @home into @ so my btrfs is setup like yours and the script would work as currently written.

**GreyGeek** · Mar 06, 2022, 03:55 PM

The script I wrote for snapshotting @ would have to be duplicated and in the second part @ is changed to @home, pointing to the their respective subdirectories under /mnt/snapshots/...

Here's the easy way.
https://www.kubuntuforums.net/forum/...-only-root-sub
My method was to sudo to root and mount /mnt. Then run mc and open @ /home in the left panel. In the right panel highlight jerry (@home/jerry) and use the copy command to copy jerry over to @/home. When it was done I edited /etc/fstab and put a # in front of the line that mounted @home to /home, and saved it. Then I unmounted everything and rebooted. Use your own account name, of course.

In that thread Oshunluver posted his method, which I think is better.

**tomcloyd** · Mar 06, 2022, 04:41 PM

Arrrgh...this thread is convincing me to stick with Timeshift for backups. When the new Kubuntu LTS version is out I well may install with btrfs - haven't decided. Timeshift can do btrfs backups.

**Snowhog** · Mar 07, 2022, 11:35 AM

GreyGeek I've been working on this (script) since last night, and have what I believe to be correct (for what I want "at this time"). I'd like your critical eye on it to see if you notice anything wrong.

This script doesn't deal with incremental snapshots; maybe later; and it doesn't (yet) deal with cleaning up (deleting) older snapshots. That will be next. Baby Steps. First things first.

#!/bin/bash
# snapshot.sh
#
# To be run as root from /
# Type: /snapshot.sh
# To get the time taken to complete, use: time
# Type: time /snapshot.sh
#
# Before executing this script:
# Plug in external HDD. Ignore Disk & Devices pop-up. # External drive will be powered and identified by system but not yet mounted.
#
echo "Mounting drives"
# mount internal HDD to /mnt
eval "mount /dev/disk/by-label/LAPTOP /mnt"
echo "Internal HDD mounted to /mnt"
# mount external HDD to /backup
eval "mount /dev/disk/by-label/BACKUP /backup"
echo "External USB HDD mounted to /backup"
echo ""
# capture current date and time
NOW=$(date +%Y%m%d%H%M)
echo "Timestamp is $NOW"
echo ""
# Create snapshots
echo "Making today's snapshot of @"
MKSNAP1="btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW"
eval $MKSNAP1
eval 'sync'
eval 'sync'
echo "@$NOW successfully created"
echo ""
echo "Making today's snapshot of @home"
MKSNAP2="btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home$NOW"
eval $MKSNAP2
eval 'sync'
eval 'sync'
echo "@home$NOW successfully created"
echo ""
# Send snapshots
SNDSNAP1="btrfs send /mnt/snapshots/@$NOW | btrfs receive /backup"
SNDSNAP2="btrfs send /mnt/snapshots/@home$NOW | btrfs receive /backup"
echo "Sending @$NOW and @home$NOW to external USB HDD"
eval $SNDSNAP1
eval 'sync'
eval 'sync'
eval $SNDSNAP2
eval 'sync'
eval 'sync'
echo "@$NOW and @home$NOW successfully sent to external USB HDD"
echo ""
# Show snapshots on External USB HDD
vdir /backup
echo ""
echo "Unmounting drives"
eval 'umount /backup'
eval 'umount /mnt'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
eval 'sync'
eval 'sleep 1'
echo ""
echo "Drives unmounted and 3 syncs done"

Added:
I ran this and it completed without any errors. Sweet!

**jlittle** · Mar 07, 2022, 02:59 PM

Why all the "evals"? Using them stops your editor from properly syntax colouring the script. evals can be treacherous.

I wonder why

Code:

MKSNAP1="btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW"
eval $MKSNAP1

and not just

Code:

btrfs su snapshot -r /mnt/@ /mnt/snapshots/@$NOW

All those syncs look excessive. Some of my external drives are active for 20 or 30 seconds after my backup script finishes, while several GB in cache are flushed to the drives, but IIRC unmounts stall till the flush is complete. I wonder if a btrfs filesystem sync /backup is a good idea.

**GreyGeek** · Mar 07, 2022, 03:02 PM

That looks OK to me as for a coding is concerned.

However, your code still puts both @yyyymmddhhmm and @homeyyyyddhhmm into the same directory. Five days of snapshots could look something like this:

.../snapshots/@yyyymmd-5hhmm
.../snapshots/@yyyymmd-4hhmm
.../snapshots/@yyyymmd-3hhmm
.../snapshots/@yyyymmd-2hhmm
.../snapshots/@yyyymmd-1hhmm
.../snapshots/@yyyymmdhhmm
.../snapshots/@homeyyyymmd-5hhmm
.../snapshots/@homeyyyymmd-4hhmm
.../snapshots/@homeyyyymmd-3hhmm
.../snapshots/@homeyyyymmd-2hhmm
.../snapshots/@homeyyyymmd-1hhmm
.../snapshots/@homeyyyymmdhhmm

OR, hopefully, they could like something like this

.../snapshots/@yyyymmd-5hhmm
.../snapshots/@homeyyyymmd-5hhmm
.../snapshots/@yyyymmd-4hhmm
.../snapshots/@homeyyyymmd-4hhmm
.../snapshots/@yyyymmd-3hhmm
.../snapshots/@homeyyyymmd-3hhmm
.../snapshots/@yyyymmd-2hhmm
.../snapshots/@homeyyyymmd-2hhmm
.../snapshots/@yyyymmd-1hhmm
.../snapshots/@homeyyyymmd-1hhmm
.../snapshots/@yyyymmdhhmm
.../snapshots/@homeyyyymmdhhmm

The list commands in my code which identifies the oldest snapshot in a folder are

list=''
list=($(ls /mnt/snapshots ))
DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
....
list=''
list=($(ls /backup))
DELSNAP='btrfs subvol delete -C /backup/'${list[0]}

Using the second listing the deletion code could be:

list=''
list=($(ls /mnt/snapshots ))
DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
list=''
list=($(ls /mnt/snapshots ))
DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
list=''
list=($(ls /mnt/snapshots ))
DELSNAP='btrfs subvol delete -C /backup/'${list[0]}
list=''
list=($(ls /mnt/snapshots ))
DELSNAP='btrfs subvol delete -C /backup/'${list[0]}

My list foo isn't good enough to remove the two oldest listings in the first set. Now you know why I recommend saving @ snapshots in .../snapshots/root/
and @home snapshots in
.../snapshots/home/

**Snowhog** · Mar 07, 2022, 03:05 PM

jlittle I'm just using what GreyGeek wrote as my starting point. As I said in my first post, btrfs is voodoo magic to me. I learn better when I have a working example. His script works for him, and my current rendition of it works for me. Can it be improved on? Most certainly! Can it be fine tuned to use the least amount of resources? Again, most certainly.

I'm all for having a 'lean, mean machine', so if you have suggestions on how I can improve my endeavor, please don't hesitate to suggest. (Examples most welcome!)

**Snowhog** · Mar 07, 2022, 03:29 PM

Unlike GreyGeeks use case, I don't have a need for such a detailed backup routine. The laptop isn't my production PC. What I'm really interested in is what I've accomplished so far, and that's creating full snapshots of @ and @home, and then sending them to an external storage device; my external USB HDD.

So, my current question is this: As I have successfully created and sent full snapshots to my external USB HDD, and I'm not (at this time anyway) going to be creating incremental snapshots, is there any reason (other than convenience and speed) for keeping the snapshots that are on my internal HDD?

**GreyGeek** · Mar 07, 2022, 03:29 PM

Here is where I got my list foo from. It involves the use of curly brackets and square brackets on bash variables filled with the ls command.
https://www.linux.com/topic/desktop/...y-braces-bash/

**Snowhog** · Mar 07, 2022, 07:07 PM

GreyGeek Why do you only send incremental snapshots to /backup?

**GreyGeek** · Mar 07, 2022, 09:25 PM

Because, typically, an incremental snapshot usually takes less than 20 seconds, while full snapshots can take 15-20 minutes. The result is the same. My internal backup is a 1Tb NVMe SSD. Its NVM Cntl chip has a maximum T of 70C and it begin throttling at 68.8C. The NVMe VNAND chip starts heating up slowly but continues heating up after the snapshot is done and can get as high as 68.8 as well. I use incremental snapshots to minimize the heating and because they take significantly less time. I have a heat sink on the NVMe but because of the restricted space it is only 2mm thick. I tried a 4mm thick heat sink and it kept the T at or below 52C, but it was so thick that I couldn't completely close the back cover.

BTW, comments do not show up on the "New Posts" or "Today's Posts" listings.

Announcement

Would like some assistance

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment