Would like some assistance

[GreyGeek]

Interesting to see how your mind works!

So, how long did your day 2 snapshots and increments take?

[Snowhog]

And here it is.

#!/bin/bash
clear
STARTCLOCK=$(date +%s)
# snapshot.sh
# Created on 2022-03-16
#
# Based on the script by GreyGeek
# and the grateful assistance
# of GreyGeek, claydoh and others
# without who's help I might not have
# gotten this done.
#
# Currently this script only does the following:
#
# The first time this script is run, no snapshots yet exist.
# Snapshots are made of the /@ and /@home sub-volumes on the
# internal HDD and sent to the external USB HDD. This is called
# initial bootstraping, and corresponds to a full backup.
# The internal HDD now has one /@ and /@home snapshot.
# The external USB HDD now has one FULL BACKUP of /@ and /@home.
# The script is finished at that point.
#
# On subsequent runs a new snapshot is made of the /@ and /@home
# on the internal HDD.
# The internal HDD now has two /@ and /@home snapshots.
# Incremental snapshots of the /@ and /@home sub-volumes on the
# internal HDD are created on the external USB HDD.
# The internal HDD now has two /@ and /@home snapshots.
# The external USB HDD now has one FULL BACKUP of /@ and /@home
# and one INCREMENTAL BACKUP of /@ and /@home.
# The previous /@ and /@home snapshot on the internal HDD are
# deleted.
# The internal HDD now has one /@ and /@home snapshot.
#
# Change Log:
# Syntax to use:
# YYYY-MM-DD
# What was done
#

# To be run as root from /
#
# Before executing this script:
# Plug in external HDD and ignore Disk & Devices pop-up.
# External drive will be powered and identified by system but not yet mounted.

# Check backup drive connection status

USBDRIVE=($(ls /dev/disk/by-label/|grep 'USBHDD'))
TEST="USBHDD"
if [[ $USBDRIVE == $TEST ]];
then
echo "USB HDD is connected and will be mounted"
read -p "Press any key to continue, Ctrl+C to Quit"
else
echo "USB HDD needs to be connected"
read -p "When connected, press any key to continue, Ctrl+C to Quit"
sleep 2
fi
clear

LAPTOP="mount /dev/disk/by-label/LAPTOP /mnt"
USBHDD="mount /dev/disk/by-label/USBHDD /backup"

# Mount Drives
## Internal Drive

echo "Mounting Internal HDD"
echo "Command is"
echo ""
echo " "$LAPTOP
echo ""
eval $LAPTOP
sleep 2

# When this script is run for the first time,
# and no snapshots exist,
# the two array variables following will be NULL
# otherwise they will contain at least one element each

SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT'))
SNAPLIST2=($(ls /mnt/snapshots/|grep 'HOME'))

echo ""
echo "Current content of /mnt"
echo ""
vdir /mnt
echo ""
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""

## External Drive

echo "Mounting USB HDD"
echo "Command is"
echo ""
echo " "$USBHDD
echo ""
eval $USBHDD
sleep 2
echo ""
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
read -p "Press any key to continue, Ctrl+C to Quit"

# Snapshot Routine

echo
if [[ ${#SNAPLIST1[@]} -le 0 ]];
then
clear
echo "Day 1 snapshot process"
echo ""
read -p "If this is not Day 1 press Ctrl+C NOW!"
clear

# Create Day 1 snapshots of @ and @home on /mnt/snapshots and send to /backup
# Variables

# Base Names

ROOTNAME=$(date +%F_%T)'ROOT'
HOMENAME=$(date +%F_%T)'HOME'

# Make Snapshots

MKSNAP1="btrfs subvolume snapshot -r /mnt/@ /mnt/snapshots/$ROOTNAME"
MKSNAP2="btrfs subvolume snapshot -r /mnt/@home /mnt/snapshots/$HOMENAME"

# Send Snapshots

SNDSNAP1="btrfs send /mnt/snapshots/$ROOTNAME | btrfs receive /backup"
SNDSNAP2="btrfs send /mnt/snapshots/$HOMENAME | btrfs receive /backup"

echo "This two step process will create a Full Backup of /@ and /@home sub-volumes"
echo ""
echo "Step 1 creates READ-ONLY snapshots and saves them in /mnt/snapshots"
echo "Step 2 creates and sends the Full Backup to /backup"
echo ""
read -p "Press any key to continue"
clear
echo "Step 1. Creating READ-ONLY snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKSNAP1
echo " "$MKSNAP2
echo ""
echo "Creating snapshots"
echo ""
eval $MKSNAP1
eval $MKSNAP2
echo ""
echo "Snapshots created"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Initial bootstrapping

echo "Step 2. Creating Full Backups"
echo ""
echo "Sending $ROOTNAME to /backup"
echo "Sending $HOMENAME to /backup"
echo ""
echo "Commands are"
echo ""
echo " "$SNDSNAP1
echo " "$SNDSNAP2
echo ""
echo "This will take a while"
echo ""
eval $SNDSNAP1
eval $SNDSNAP2
sync;sync;sync
echo ""
echo "Creation of Full Backups complete"
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
echo "Initial snapshot process completed"
echo ""

# Array Variables now contain one element each

else

# This is run two or higher, so one snapshot of /@ and /@home
# from previous run exist and will be identified as
# PREVIOUSSNAP1
# PREVIOUSSNAP2

# Array Variables

SNAPLIST1=($(ls /mnt/snapshots/|grep 'ROOT'))
SNAPLIST2=($(ls /mnt/snapshots/|grep 'HOME'))
#
# Each Array Variable now contain one or more elements

# Variables

ROOTNAME=$(date +%F_%T)'ROOT-Inc'
HOMENAME=$(date +%F_%T)'HOME-Inc'

# ONLY used if two or more of each Snapshot is being saved

OLDESTSNAP1=${SNAPLIST1[0]}
OLDESTSNAP2=${SNAPLIST2[0]}

PREVIOUSSNAP1=${SNAPLIST1[-1]}
PREVIOUSSNAP2=${SNAPLIST2[-1]}

CURRENTSNAP1=$ROOTNAME
CURRENTSNAP2=$HOMENAME

# Take Snapshots

MKSNAP1="btrfs subvolume snapshot -r /mnt/@ /mnt/snapshots/$ROOTNAME"
MKSNAP2="btrfs subvolume snapshot -r /mnt/@home /mnt/snapshots/$HOMENAME"

# Make Incrementals
## @

MKINC1='btrfs send -p /mnt/snapshots/'$PREVIOUSSNAP1
MKINC1=$MKINC1' /mnt/snapshots/'$CURRENTSNAP1
MKINC1=$MKINC1' | btrfs receive /backup'

## @home

MKINC2='btrfs send -p /mnt/snapshots/'$PREVIOUSSNAP2
MKINC2=$MKINC2' /mnt/snapshots/'$CURRENTSNAP2
MKINC2=$MKINC2' | btrfs receive /backup'

# Make Snapshots

clear
echo "Running Day 2+ process"
echo ""
read -p "If this is not Day 2+ press Ctrl+C now!"
echo "Otherwise press any key to continue"
clear

echo "This two step process will create an Incremental Backup of /@ and /@home sub-volumes"
echo ""
echo "Step 1 creates READ-ONLY snapshots and saves them in /mnt/snapshots"
echo "Step 2 creates and sends the Incremental Backup to /backup"
echo ""
read -p "Press any key to continue"
clear
echo "Step 1. Taking READ-ONLY snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKSNAP1
echo " "$MKSNAP2
echo ""
eval $MKSNAP1
eval $MKSNAP2
echo ""
echo "Snapshots created"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Make Incremental Snapshots

echo "Step 2. Creating Incremental snapshots"
echo ""
echo "Commands are"
echo ""
echo " "$MKINC1
echo " "$MKINC2
echo ""
eval $MKINC1
eval $MKINC2
eval 'sync;sync;sync'
echo ""
echo "Incremental snapshots received to /backup"
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
echo "Current content of /backup"
echo ""
vdir /backup
echo ""
read -p "Press any key to continue"
clear
fi

# Delete Oldest Snapshot

DELSNAP1='btrfs subvolume delete -C /mnt/snapshots/'$OLDESTSNAP1
DELSNAP2='btrfs subvolume delete -C /mnt/snapshots/'$OLDESTSNAP2

# Delete Oldest Snapshots

echo "There are now two snapshots of /@ and /@home on /mnt/snapshots;"
echo "the ones created during this run, and the ones created during the previous one."
echo "The previous snapshots will now be deleted from /mnt/snapshots."
echo ""
echo "Commands are"
echo ""
echo " "$DELSNAP1
echo " "$DELSNAP2
echo ""
echo "Deleting"
echo ""
echo " "$OLDESTSNAP1
echo " "$OLDESTSNAP2
echo ""
eval $DELSNAP1
eval $DELSNAP2
eval 'sync;sync;sync'
echo ""
echo "Snapshots have been deleted"
echo ""
echo "Current content of /mnt/snapshots"
echo ""
vdir /mnt/snapshots
echo ""
read -p "Press any key to continue"
clear

# Calculate and display how much time creating and sending snapshots took

STOPCLOCK=$(date +%s)
SECS=$((STOPCLOCK - STARTCLOCK))
DURATION=$(date --date "0 $SECS sec" +%Hh:%Mm:%Ss)

echo "Finished. Total process took: $DURATION"
echo ""
echo ""
echo "Drives remain mounted. Remember to umount drives and power-off USB HDD"

[Snowhog]

Geez! 61 preceding posts in this Topic. I'm trending!

It took me 16 days to get my script to work for my use case and in the way I wanted without any errors! That occurred about an hour ago. I'm so happy.

The script isn't actually a finished/polished product yet. Now that it works as designed, I can, at my leisure, re-examen it for the purpose of simplifying and improving the code. So maybe later this evening or sometime tomorrow I'll present it here for all to heap their accolades upon or to say WTF! I'm good either way.

But until then, I'm taking some time to mentally unwind.

[Snowhog]

Today was 'day 1' running my script. It worked flawlessly. Full Backups of @ and @home were created on my external USB HDD. Tomorrow will be 'day 2', creating Incremental Snapshots. Personally, I think they should referred to as Incremental Backups, but, however one pronounces 'salmon', it's still a fish. If tomorrows results are what I expect them to be; no errors; I'll post my script.

I've learned/relearned A LOT since I started this Topic 13 days ago.

[GreyGeek]

Showing /backup

total 0
drwxr-xr-x 1 root root 222 Mar 11 22:27 @202203112214
drwxr-xr-x 1 root root 250 Mar 12 14:27 @202203121415
drwxr-xr-x 1 root root 36 Mar 11 22:29 @home202203112214
drwxr-xr-x 1 root root 36 Mar 12 14:29 @home202203121415

Attempting incremental snapshot of @
Command is: btrfs send -p /mnt/snapshots/@202203112214 /mnt/snapshots/@202203121415 | btrfs receive /backup

At subvol /mnt/snapshots/@202203121415
At snapshot @202203121415
ERROR: creating snapshot @202203112214 -> @202203121415 failed: File exists

You cannot overwrite an existing snapshot using its same name, regardless of if their contents are the same or not.

Since you've already sent it to /backup the incremental backup cannot use the parent and create a new child because the child already exists on /backup and is read only.

So, why do an incremental backup if you've already sent the snapshot to /backup?

[Snowhog]

I don't understand why btrfs doesn't like the command (See the error in red below).

This is the only error the script is generating. If I can get help on it, I can apply that to the second send/receive (@home) and the script will be done.

root@barley-cat:/# /snapshot.sh
Snapshots will be named @202203121415 and @home202203121415

Mounting Internal HDD
Command is: mount /dev/disk/by-label/LAPTOP /mnt
Showing /mnt

total 0
drwxr-xr-x 1 root root 250 Mar 12 14:15 @
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home
drwxr-xr-x 1 root root 60 Mar 12 12:57 snapshots

Mounting External USB HDD
Command is: mount /dev/disk/by-label/USBHDD /backup
Showing /backup

total 0
drwxr-xr-x 1 root root 222 Mar 11 22:27 @202203112214
drwxr-xr-x 1 root root 36 Mar 11 22:29 @home202203112214

Making today's snapshot of @ on /mnt/snapshots

Command is: btrfs su snapshot -r /mnt/@ /mnt/snapshots/@202203121415

Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203121415'
@202203121415 successfully created

Sending @202203121415 to /backup

Command is: btrfs send /mnt/snapshots/@202203121415 | btrfs receive /backup

At subvol /mnt/snapshots/@202203121415
At subvol @202203121415
@202203121415 successfully sent to /backup

Making today's snapshot of @home on /mnt/snapshots

Command is: btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home202203121415

Create a readonly snapshot of '/mnt/@home' in '/mnt/snapshots/@home202203121415'
@home202203121415 successfully created

Sending @home202203121415 to /backup

Command is: btrfs send /mnt/snapshots/@home202203121415 | btrfs receive /backup

At subvol /mnt/snapshots/@home202203121415
At subvol @home202203121415
@home202203121415 successfully sent to /backup

Showing /mnt/snapshots

total 0
drwxr-xr-x 1 root root 222 Mar 11 22:10 @202203112214
drwxr-xr-x 1 root root 250 Mar 12 14:15 @202203121415
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203112214
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203121415

Showing /backup

total 0
drwxr-xr-x 1 root root 222 Mar 11 22:27 @202203112214
drwxr-xr-x 1 root root 250 Mar 12 14:27 @202203121415
drwxr-xr-x 1 root root 36 Mar 11 22:29 @home202203112214
drwxr-xr-x 1 root root 36 Mar 12 14:29 @home202203121415

Attempting incremental snapshot of @
Command is: btrfs send -p /mnt/snapshots/@202203112214 /mnt/snapshots/@202203121415 | btrfs receive /backup

At subvol /mnt/snapshots/@202203121415
At snapshot @202203121415
ERROR: creating snapshot @202203112214 -> @202203121415 failed: File exists

IF send/receive command FAILED, Press Ctrl+C now to Quit.

root@barley-cat:/# vdir /mnt/snapshots/
total 0
drwxr-xr-x 1 root root 222 Mar 11 22:10 @202203112214
drwxr-xr-x 1 root root 250 Mar 12 14:15 @202203121415
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203112214
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203121415
root@barley-cat:/# vdir /backup/
total 0
drwxr-xr-x 1 root root 222 Mar 11 22:27 @202203112214
drwxr-xr-x 1 root root 250 Mar 12 14:27 @202203121415
drwxr-xr-x 1 root root 36 Mar 11 22:29 @home202203112214
drwxr-xr-x 1 root root 36 Mar 12 14:29 @home202203121415
root@barley-cat:/#

[Snowhog]

I have 'one' remaining btrfs error that I need to resolve before I'll publish my script here. I'll explain the error a bit later; I've only just gotten up, and I haven't had my requisite volume of coffee yet!

[jlittle]

"Surgeons don't operate on themselves, so don't try to proof read your own code!"

I can't agree, but all those unnecessary "eval"s don't help.

[Snowhog]

I want to thank everyone who has commented in this Topic. I've learned A LOT over the last several days. I finally got the script finished. I'm having another set of eyes look it over, but the first part; the creation of both @ and @home snapshots and then sending them to /backup, works. I've tested it. Right now there are exactly one @ and @home snapshots on /mnt/snapshots and /backup.

The incremental section of my script is written to check that at least two snapshots of @ and @home are present on /mnt/snapshots and /backup. If there are, the incremental section runs. If there are not, then it says so and exits the script. That way, I can look into it.

I'll post my finished script later, after it's been reviewed by the other set of eyes and it's given the "Looks good to me. I don't see any glaring errors."

"Surgeons don't operate on themselves, so don't try to proof read your own code!"

[GreyGeek]

There is nothing on /backup.

That prompts this question: This script doesn't send the created FULL snapshot to /backup, and neither does yours. How are you getting your created FULL snapshots to /backup? I don't see that happening in your script.

Is that the crux of the issue with my script; there are no @ and @home snapshots on /backup?

That there are NO snapshots on /backup is why the reference to the parent snapshot fails.

Do this:

btrfs send /mnt/snapshots/@202203111046 | btrfs receive /backup

which will send the parent snapshot to /backup. It will take about 16 minutes.

Then run

btrfs send -p /mnt/snapshots/@202203111046 /mnt/snapshots/@202203111047 | btrfs receive /backup

which will work because the parent snapshot is now on /backup.

[Snowhog]

There is nothing on /backup.

That prompts this question: This script doesn't send the created FULL snapshot to /backup, and neither does yours. How are you getting your created FULL snapshots to /backup? I don't see that happening in your script.

Is that the crux of the issue with my script; there are no @ and @home snapshots on /backup?

[GreyGeek]

The parent snapshot, @202203111046, is already supposed to be on /backup because of a previous send. Is it?

[Snowhog]

Already found/fixed typos in my script, which were the cause of some of the errors.

I've echoed the script up to the point of the command to make the incremental snapshot of @ and have it received on /backup. That command is executed and then I Ctrl+C out of the script. If the @ incremental section isn't working, so the @home incremental section would fail as well.

root@barley-cat:/# /snapshotTESTE.sh
Mounting drives
mount: /mnt: /dev/sda1 already mounted on /.
Internal HDD mounted to /mnt

mount: /backup: /dev/sdb1 already mounted on /backup.
External USB HDD mounted to /backup

Timestamp: 202203111047

Making today's snapshot of @
Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203111047'
@202203111047 successfully created

Making today's snapshot of @home
Create a readonly snapshot of '/mnt/@home' in '/mnt/snapshots/@home202203111047'
@home202203111047 successfully created

Current content of /mnt/snapshots
total 0
drwxr-xr-x 1 root root 418 Mar 11 09:28 @202203110929
drwxr-xr-x 1 root root 418 Mar 11 09:41 @202203110941
drwxr-xr-x 1 root root 418 Mar 11 09:42 @202203110943
drwxr-xr-x 1 root root 418 Mar 11 10:09 @202203111010
drwxr-xr-x 1 root root 418 Mar 11 10:22 @202203111022
drwxr-xr-x 1 root root 418 Mar 11 10:30 @202203111030
drwxr-xr-x 1 root root 418 Mar 11 10:40 @202203111041
drwxr-xr-x 1 root root 418 Mar 11 10:45 @202203111046
drwxr-xr-x 1 root root 418 Mar 11 10:47 @202203111047
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203110929
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203110941
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203110943
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111010
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111022
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111030
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111041
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111046
drwxr-xr-x 1 root root 36 Mar 1 20:42 @home202203111047

Incremental snapshot sub-routine variables for root will be displayed. Press any key to continue.
Variable content for use in @ incremental snapshots

SNAPLIST is @202203110929 @202203110941 @202203110943 @202203111010 @202203111022 @202203111030 @202203111041 @202203111046 @202203111047
OLDESTSNAP is @202203110929
PREVIOUSSNAP is @202203111046
NOW is @202203111047

Press any key to continue.
Attempting incremental backup of @

PREVIOUSSNAP is: @202203111046
NOW is: @202203111047
MKINC is: btrfs send -p /mnt/snapshots/@202203111046 /mnt/snapshots/@202203111047 | btrfs receive /backup

At subvol /mnt/snapshots/@202203111047
At snapshot @202203111047
ERROR: cannot find parent subvolume

Current content of /backup
total 0
Press Ctrl+C now^C
root@barley-cat:/#

As far as what this is saying just before the execution of the command, the incremental snapshot should be received on /backup, but as the ERROR indicates, it isn't.
I just don't understand what is wrong. What am I missing or not seeing?

[GreyGeek]

You're right ... my bad.

[kubicle]

Strange indexing, isn't it?

What do you mean? Looks right to me, list[0] is the first variable of the array, list[1] is the second, list[-1] is the last and list[-2] is the second to last.

Didn't read the whole thread or looked at the script, so no comments on the issue itself.