New HD came and I reinstalled Neon

[GreyGeek]

using the 3-17-2017 download.

I chose the manual install to the HD so that I could create an 8GB swap partition so suspend would work without problems. And, of course, I used Btrfss as the filing system.

I always install using a cable, not the wifi. I do not allow updates to take place during the install and I always check the box to install proprietary codecs.

The install and boot up was uneventful. My HP P1606dn printer is always attached to a USB port so I installed HPLIP. When I fired up HPLIP it was as if my printer was already configured. It appeared in the Print Manager Window instantly. I clicked the "Install the required plugin" on the Actions tab and ran into the usual error in which HP demands the "root" password. When I enter my password the plugin installer always fails. I went to the FOSS HP plugin website and downloaded the hplip-3.16.3-plugin.bin and ran it. It installed the plugin but errored out. Oh well. Duplex works perfectly anyway.

The next thing was to install the nvidia driver. First I installed the intel-microcode. I had good luck with the 4.4 kernel and the nvidia-370 driver so I installed the 4.4 kernel image & headers and their dependencies and removed the 4.8 kernel. After the install and reboot the NVidia X-Settings dialog didn't have the part which showed the NVida performance settings or the temperature graph. So, I purged nvidia and reinstalled the 4.8 kernel stuff, used ucaresystem-core to clean things up and rebooted. Then I tried the nvidia 378.12 package. Joy! After the reboot I had the FULL NVidia X-Setting dialog GUI. and Minecraft increased from a paltry 25-50 fps to 150 fps +-50, with occurrences of up to 500 fps. Then I installed and tested Steam & Uinverse Sandbox^2. Uh oh. Momentary lag when lots of particles appeared following collisions, etc.... Going to the NVidia setting I noticed that the performance was set at "adaptive". The lag was NVidia switching to higher performance while under load. I set it to high performance permanently and the Universe ran as it should.

The next step was to install KMail. I missed it sorely while using Thunderbird for a few months. Importing 1,500+ msgs from gmail's mbox was a pain in Thunderbird. It was a breeze in KMail. So was importing my old KMail email from as far back as 10 years. KDE's PIM requires the KMail share the addressbook with other personal info apps, so it is not "part" of KMail and is installed separately as KAddressbook, which was uneventful. It has an import option and Thunderbird's ldif file imported nicely. After installing Kpgp I generated another 4096 byte key and added it to the cryptography tab of KMail's account setup.

I then installed lmsensors and thermald. From the widgets download dialog I downloaded Thermal Monitor, which I used to track three temperatures: CPU, HD and the GPU. For the GPU I used nvidia.smi. They are currently showing 41, 38 and 28 C. Nice.

I took a look at installing WINE but noticed it would install 135 files, and some included mesa and GL drivers. So, I version locked all my installed Nvidia packages first, then proceeded to create a Btrfs snapshot of @ and @home. Here are the steps I followed:

jerry@jerry-Aspire-V3-771:~$ sudo -i
root@jerry-Aspire-V3-771:~#
root@jerry-Aspire-V3-771:/# mkdir /mnt/btr
root@jerry-Aspire-V3-771:/# mount /dev/sda1 /mnt/btr
root@jerry-Aspire-V3-771:/# cd /mnt


root@jerry-Aspire-V3-771:/mnt# vdir
btr
root@jerry-Aspire-V3-771:/mnt# vdir btr/
total 0
drwxr-xr-x 1 root root 264 Mar 27 14:22 @
drwxr-xr-x 1 root root  30 Mar 24 14:46 @home


root@jerry-Aspire-V3-771:/mnt# mkdir btr/snapshots
root@jerry-Aspire-V3-771:/mnt# vdir btr/
total 0
drwxr-xr-x 1 root root 264 Mar 27 14:22 @
drwxr-xr-x 1 root root  30 Mar 24 14:46 @home
drwxr-xr-x 1 root root   0 Mar 27 18:09 snapshots


root@jerry-Aspire-V3-771:/mnt# btrfs subvolume snapshot -r btr/@ btr/snapshots/@bkup-2017-03-27
Create a readonly snapshot of 'btr/@' in 'btr/snapshots/@bkup-2017-03-27'
root@jerry-Aspire-V3-771:/mnt# btrfs subvolume snapshot -r btr/@home btr/snapshots/@home_bkup-2017-03-27
Create a readonly snapshot of 'btr/@home' in 'btr/snapshots/@home_bkup-2017-03-27'


root@jerry-Aspire-V3-771:/mnt# vdir btr/
total 0
drwxr-xr-x 1 root root 264 Mar 27 14:22 @
drwxr-xr-x 1 root root  30 Mar 24 14:46 @home
drwxr-xr-x 1 root root  74 Mar 27 18:12 snapshots


root@jerry-Aspire-V3-771:/mnt# vdir btr/snapshots/
total 0
drwxr-xr-x 1 root root 264 Mar 27 14:22 @bkup-2017-03-27
drwxr-xr-x 1 root root  30 Mar 24 14:46 @home_bkup-2017-03-27
root@jerry-Aspire-V3-771:/mnt# vdir btr/snapshots/@home_bkup-2017-03-27/
total 0
drwxr-xr-x 1 jerry jerry 26114 Mar 27 17:56 jerry  <---@home_bkup contains my 
home directory!


root@jerry-Aspire-V3-771:/mnt# cd ..
root@jerry-Aspire-V3-771:/# umount /mnt/btr 
root@jerry-Aspire-V3-771:/# exit
logout
jerry@jerry-Aspire-V3-771:~$

(Note that even though "home" is in the root (@) subvolume it is empty. The "real" /home is @home, which is why I showed you that it contains my account directory)

Now that I am covered in case of disaster I proceeded to install WINE. It, too, was uneventful and didn't harm any of my previous installs, especially nvidia.

I then installed the Parker-Hamilton IQANDesigner.exe file which gives me the PLC development tool I used to write tractor control software for an ag engineer. It runs beautifully, as if it were a native Linux application!

What if I need to restore from a disaster?

Notice that even though I have unmmounted /mnt/btr the Btrfs file system still knows that the backup snapshot directories are just below "rootfs", in which @ and @home also reside, and have access to them through "rootfs". Since they are read only nothing in them can be deleted, so they can't be used to replace @ and @home until you move @ to @_old and @home to @home_old (for example)

Create the new @ and @home from the read only snapshots WITHOUT using the "-r" parameter:


btrfs subvolume snapshot /mnt/btr/snapshots/@_bkup-2017-03-27 /mnt/btr/@_new
btrfs subvolume snapshot /mnt/btr/snapshots/@home_bkup-2017-03-27 /mnt/btr/@home_new


which makes a copy of the backup but WITHOUT the -r switch (read only), so that files can be added, changed or removed from it.


root@jerry-Aspire-V3-771:/mnt# mv /mnt/btr/@ /mnt/btr/@_old
root@jerry-Aspire-V3-771:/mnt# mv /mnt/btr/@home /mnt/btr/@home_old

root@jerry-Aspire-V3-771:/mnt# mv /mnt/btr/@_new /mnt/btr/@
root@jerry-Aspire-V3-771:/mnt# mv /mnt/btr/@home_new /mnt/btr/@home

Delete the @_old and @home_old and then exit from Konsole and reboot.
Since the UUID is still the same, you don't have to edit /etc/fstab to change them.

[oshunluvr]

Nice write-up Jerry. BTRFS really makes the transitions easy, doesn't it? I added a small SSD to my server for a boot device (I think you read about it here) and just sent the entire OS to the new drive. I didn't think about it at the time, but I bet you could create the partition on the new disk and force the UUID to match, then send the subvolume(s) to it, then either change the UUID of the source partition or just delete it, and just reboot.

Sure is a lot easier than it was just a couple years ago.

Have you attempted snapper again? If so, how's it working?

I am thinking we really need a configurable package install and rollback tool that automagically utilizes all the great features of btrfs and snapshots. I'm thinking: set a snapshot folder, whenever you begin an apt operation, trigger a snapshot, keep a configurable number of snaps, auto-remove them at a certain number. Rollback snaps occur separately at a configurable interval and are saved for a configurable time or quantity. Then every operation would be "protected" without any manual intervention. I guess the final piece would be a rollback tool to pick from the available list and reboot to it. I don't think it would be too hard to create.

[GreyGeek]

...

Have you attempted snapper again? If so, how's it working?

I am thinking we really need a configurable package install and rollback tool that automagically utilizes all the great features of btrfs and snapshots. I'm thinking: set a snapshot folder, whenever you begin an apt operation, trigger a snapshot, keep a configurable number of snaps, auto-remove them at a certain number. Rollback snaps occur separately at a configurable interval and are saved for a configurable time or quantity. Then every operation would be "protected" without any manual intervention. I guess the final piece would be a rollback tool to pick from the available list and reboot to it. I don't think it would be too hard to create.

Snapper is working great using my script, which creates only singleton snapshots. If I uninstall a package I don't want, or the install borks and leaves detritus around, I use ucaresystem-core to do the initial cleanup. Then I use snapper undochange to remove the changes the uninstall and ucare didn't get. I don't create read-only snapper snapshot because that's what I have the /mnt/btr/snnapshots/@_backup & @home_backup snapshots for. If I lose the ability to boot into my desktop then rather than putzing around trying to recover I rollback from the snapshots. Being read-only and inaccessible from normal operations the HD would have to die horribly to lose the system.

I downloaded Btrbk-Master.zip and have been studying its code to decide either to install it or write my own script based on it. I am also going to create a @Data subvolume on /mnt/Btr, mkdir /Data and then mount @Data to /Data in fstab and start storing my data and install pkgs there. Right now I'm looking for the Dolphin KDE5 service menu that converts new Chase Pdf files to a PDF format that Ocular can read. I had installed it before but forgot to save the URL or the pkg.

[oshunluvr]

Found this today: https://btrfs.wiki.kernel.org/index....emental_Backup

Might be some useful stuff there.

[GreyGeek]

Found this today: https://btrfs.wiki.kernel.org/index....emental_Backup

Might be some useful stuff there.

I saw that, and it left me with some questions. A regular snapshot contains all the blocks in the specified path. The incremental snapshot contains only blocks in that path with changes in them. The article called that snapshot HOMEBACKUP-new, as shown:

 btrfs subvolume snapshot -r /home /home/BACKUP-new
   sync
[COLOR=#000000][FONT=sans-serif]We can now send the difference between the old and new backup to the backup volume:[/FONT][/COLOR]
   btrfs send -p /home/BACKUP /home/BACKUP-new | btrfs receive /backup/home
[COLOR=#000000][FONT=sans-serif]Once this command completes, we should have these 4 subvolumes: /home/BACKUP, /home/BACKUP-new, /backup/home/BACKUP and /backup/home/BACKUP-new. We will now need to migrate the new backup as the old one, and do something for the old one. We could keep it around, maybe timestamped with the date of that backup, or just straight out delete it. Here, I am deleting it:[/FONT][/COLOR]
   btrfs subvolume delete /home/BACKUP
   mv /home/BACKUP-new /home/BACKUP
   btrfs subvolume delete /backup/home/BACKUP
   mv /backup/home/BACKUP-new /backup/home/BACKUP

Now, /home/BACKUP contains ALL the blocks and /home/BACKUP-new contains only blocks with changes. Where does the integration take place? The mv command won't do it. Does the "receive" command merge /home/BACKUP with /home/BACKUP-new. Otherwise, the mv command, mv /home/BACKUP-new /home/BACKUP leave the latter holding only changed blocks.

[oshunluvr]

My understanding is yes, "receive" does the integration on the receiving end. So in the above example, the target subvolume (backup/home) contains a current full backup. The two source subvolumes contain the original backup (/home/BACKUP) and the incremental backup (/home/BACKUP-new) and they are combined by sending the difference.

The whole second section with the renaming is about "elevating" the second source backup to the former position of the original source backup. The names he used make it a little confusing. The steps using using numbered snapshots and more normal subvolume names:

Initial backup:

btrfs subvolume snapshot -r @home BACKUP1
btrfs send BACKUP1 | btrfs receive /backup/home_backup

First incremental backup:

btrfs subvolume snapshot -r @home BACKUP2
btrfs send BACKUP1 -p BACKUP2 | btrfs receive /backups/@home_backup

At this point BACKUP2 is a full current snapshot and BACKUP1 is a snapshot of the previous state.

Second incremental backup:

btrfs subvolume snapshot -r @home BACKUP3
btrfs send BACKUP2 -p BACKUP3 | btrfs receive /backups/@home_backup

repeat ad nauseum. Each Incremental operation sends only the changes to the target subvolume.

Now you would have multiple snapshots:

BACKUP1 -> The original
BACKUP2 -> First incremental
BACKUP3 -> Second incremental
...etc...

and one full backup:

/backups/@home_backup

I think his idea was to delete BACKUP1 and replace it with BACKUP2, then in the next go-round delete BACKUP2 and replace it with BACKUP3, and so on In the scenario where you were using a script to do this, deleting the first snapshot and renaming the second snapshot to the name of the first would have the effect of keeping only one snapshot for backup send/receive purposes at a time, thus preventing an full drive condition, while allowing the script to run unattended - like as a cron job. One could, easily enough, write the script to keep the last three or four snapshots instead of just the last.

MY question is: Since the sent snapshot is read-only, the received snapshot is also read-only. Therefore, how does the second command work?
My assumption is the "receive" command over-rides the RO flag. I'm going to have to test this out.

[GreyGeek]

....

btrfs send BACKUP1 -p BACKUP2 | /backups/@home_backup

That was my assumption.

MY question is: Since the sent snapshot is read-only, the received snapshot is also read-only. Therefore, how does the second command work?
My assumption is the "receive" command over-rides the RO flag. I'm going to have to test this out.

Your assumption is the only logical conclusion. "Receive" may be more than one step, since for users the only way to convert an ro snapshot to an rw one is to snapshot it to another name without the -r flag.

[GreyGeek]

So, with our conversation fresh on my mind I decided to reformat my WD Paasport USB 320GB HD using Btrfs and KParted. No problems. I unmounted it.
I used "sudo -i" to log into root to use the commands without needing to use sudo in front of each.

Then I began the sequence of commands.
First, see what exists:

root@jerry-Aspire-V3-771:/# mnt /dev/sdb1 /backup
root@jerry-Aspire-V3-771:/# mnt /dev/sda1 /mnt/btr

root@jerry-Aspire-V3-771:/# vdir /mnt/btr
total 0
drwxr-xr-x 1 root root 276 Mar 29 11:16 @
drwxr-xr-x 1 root root 30 Mar 24 14:46 @home
drwxr-xr-x 1 root root 74 Mar 27 18:12 snapshots

root@jerry-Aspire-V3-771:/# vdir /mnt/btr/snapshots/
total 0
drwxr-xr-x 1 root root 264 Mar 27 14:22 @bkup-2017-03-27
drwxr-xr-x 1 root root 30 Mar 24 14:46 @home_bkup-2017-03-27
root@jerry-Aspire-V3-771:/#

root@jerry-Aspire-V3-771:/# vdir /backup
total 0
root@jerry-Aspire-V3-771:/# btrfs subvolume create /backup/backups
Create subvolume '/backup/backups'
root@jerry-Aspire-V3-771:/# vdir /backup/
total 0
drwxr-xr-x 1 root root 0 Mar 29 11:19 backups


Now, do the actual send & receive:

root@jerry-Aspire-V3-771:/# btrfs send -ve /mnt/btr/@ /mnt/btr/@home | btrfs receive /backup/backups
ERROR: subvolum /mnt/btr/@ is not read-only

Opps! Need to make a couple of ro snapshots. Good thing, because they'll include the installation of WINE and the IQAN packages, along with PlayOnLinux.


root@jerry-Aspire-V3-771:/# btrfs subvolume snapshot -r /mnt/btr/@home /mnt/btr/snapshots/@home_bkup-2017-03-29
Create a readonly snapshot of '/mnt/btr/@home' in '/mnt/btr/snapshots/@home_bkup-2017-03-29'
root@jerry-Aspire-V3-771:/# btrfs subvolume snapshot -r /mnt/btr/@ /mnt/btr/snapshots/@_bkup-2017-03-29
Create a readonly snapshot of '/mnt/btr/@' in '/mnt/btr/snapshots/@_bkup-2017-03-29'


NOW, do the send & receive:

root@jerry-Aspire-V3-771:/# btrfs send -ve /mnt/btr/snapshots/@_bkup-2017-03-29 /mnt/btr/snapshots/@home_bkup-2017-03-29 | btrfs receive /backup/backups
At subvol /mnt/btr/snapshots/@_bkup-2017-03-29
At subvol @_bkup-2017-03-29
BTRFS_IOC_SEND returned 0
joining genl thread
At subvol /mnt/btr/snapshots/@home_bkup-2017-03-29
ERROR: parent determination failed for 0
ERROR: unexpected EOF in stream.
root@jerry-Aspire-V3-771:/#

Not good.
I did some experimenting with the commands and found that even though my version of Btrfs is 4.4 the receive command would not honor the "-ve" switches.
So, I decided to do the send one subvol at a time:


root@jerry-Aspire-V3-771:/# btrfs send -v /mnt/btr/snapshots/@_bkup-2017-03-29 | btrfs receive /backup/backups
At subvol /mnt/btr/snapshots/@_bkup-2017-03-29
At subvol @_bkup-2017-03-29
BTRFS_IOC_SEND returned 0
joining genl thread
root@jerry-Aspire-V3-771:/# btrfs send -v /mnt/btr/snapshots/@home_bkup-2017-03-29 | btrfs receive /backup/backups
At subvol /mnt/btr/snapshots/@home_bkup-2017-03-29
At subvol @home_bkup-2017-03-29
BTRFS_IOC_SEND returned 0
joining genl thread
root@jerry-Aspire-V3-771:/#

Success!

About how much space was taken?

root@jerry-Aspire-V3-771:/# btrfs filesystem show /
Label: none uuid: 12980ae8-4117-4cc5-bbb8-8065e82af93d
Total devices 1 FS bytes used 82.40GiB
devid 1 size 691.19GiB used 84.02GiB path /dev/sda1


root@jerry-Aspire-V3-771:/# btrfs filesystem show /backup/
Label: 'BAXKUP' uuid: e7f0f5b8-9a4c-4698-9a5d-50643ce9d586
Total devices 1 FS bytes used 75.34GiB
devid 1 size 297.43GiB used 77.02GiB path /dev/sdb1


Time to unmount:

root@jerry-Aspire-V3-771:/# umount /backup
root@jerry-Aspire-V3-771:/#
root@jerry-Aspire-V3-771:/# umount /mnt/btr
root@jerry-Aspire-V3-771:/#


For newbs: the unmount command IS spelled umount. And, while my live system is on sda1 and I mounted that partition to /mnt/btr, when I umount it my live system is not touched. That's one of the beauties of Btrfs: you can work on it LIVE! (BTW, "umount /mnt" won't work because /mnt wasn't mounted, /mnt/btr was.)

Now, I have the very latest copy of @ and @home residing safely on my WD 320GB USB drive. Considering how quickly my other identical WD drive died, its questionable. I may be taking another look at that Samsung EVO Pro SSD.