some btrfs questions

[oldgeek]

I'm still in the kindergarten stage of learning btrfs, and although I've checked a number of different websites, I have a couple of questions that I think would be more clearly answered in this forum. First of all, I've decided to avoid snapper and just use the terminal to do my btrfs chores, thanks to a heads-up by GreyGeek. I don't mind using terminal commands and I don't want another learning curve to using Snapper. I currently am using /subvol as my btrfs folder for @ and @home and snapshots and have no swap files/partitions and no other OS except Kubuntu 18.04 (great distro, by the way). I have my /subvol mounted automatically as per oshunluvr's instructions in a previous post. I have taken two snapshots of both @ and @home under two different names at two different dates, so I know I'm doing that correctly. Now I want to use the backup feature that GreyGeek described so that I can roll back if my computer goes bonkers or if I want to change to Neon when it is based on 18.04. I am currently clearing my future backup external HD of the Aptik backup I did earlier, and I want to use this as my btrfs backup. What I need to know:
1) How do I format this HD into a btrfs-compatible HD;
2) What program do I use to do this (Kpartition or Gparted);
3) What happens when I create the backup--does the command copy the whole setup onto the hard disk?
4) How do I use the btrfs backup to restore (roll back?) my configuration/files to a new OS if the previous one was irreparably damaged?
5) What are the basic btrfs commands I need to know for daily use.

Sorry for the length of this missive but since there's no 'Btrfs guide for Dummies' available I want to make sure I'm giving enough information to be able to advance in my knowledge of how to use this new file system. Plus, as an ex-English teacher, I can be rather wordy at times.

[SpecialEd]

My problem --so far-- with Btrfs is there is no linear, up-to-date, instruction set. I'm not impressed with the wicki or other pages found on the web, most of them being several years old. There is a lot of good information here on the forum but to find it one has to do a search and click on each return and hope it covers what you are looking for. I'm a reasonably intelligent person but I just haven't found a source that works for me. Hell, I'd be happy with a BTRFS For Dummies book, right about now.

[GreyGeek]

...
What I need to know:
1) How do I format this HD into a btrfs-compatible HD;
2) What program do I use to do this (Kpartition or Gparted);
3) What happens when I create the backup--does the command copy the whole setup onto the hard disk?
4) How do I use the btrfs backup to restore (roll back?) my configuration/files to a new OS if the previous one was irreparably damaged?
5) What are the basic btrfs commands I need to know for daily use.
...

1) Use the KDE Partition Manager ("Partition Editor") to delete the existing partitions on the HD. Then create a new partition using the "raw" unformatted HD. Right click on the new partition and select Properties. On that GUI select either GPT or MSDOS, depending on your HD, and then use the drop down combo box to change the default EXT4 to Btrfs. Add a label if you wish. Click OK. Then click "Apply" in the menu of KDE Partition Manager.

2) I use KDE Partition Manger (Its actual name is in the repository is "partitionmanager").

3) When you create a snapshot of @, or @home, or what ever, the snapshot is created almost instantly. It contains some meta data but nothing else. As you continue to use your system and make changes, the files as they existed before they were changed are Copied On Write (COW) to the snapshot. IF nothing changes then nothing is copied to the snapshot. IF you were to "touch" each file then every file you touched would be copied to the snapshot. That's true for all the snapshots you make and store in the system Btrfs pool (<ROOT_FS>).

However, there are two kinds of snapshots: read-write and read-only. The Btrfs send & receive command can only be used with a read-only snapshot, hence the use of the "-r" switch when creating it. But, since a read-only snapshot cannot be changed, so it cannot be used to roll @ or @home back to a previous condition. (There is a property setting that allows one to change a snapshot from r to rw, or the other way around, but do not use it for reasons explained in another post) The solution is to take a snapshot of the ro snapshot but without using the "-r" switch, which creates a rw snapshot out of the ro snapshot.

4) Rolling back. For some reason you cannot boot into your existing @ or @home. Bad grub, update failure, etc... Or, having taken a snapshot of @ and @home before you tried a series of experiments, you decide you want to roll back to that version of @ and @home, which you've labeled @before and @homebefore.

a) Open a Konsole and use "sudo -i" to go to root.

b) You mount your live system (amazing, isn't it!) using its UUID
mount /dev/disk/bu-uuid/theactualuuid /mnt
(or what ever mount point you want) (Note: some operations involving /dev/sdXn can cause the drive letter to switch, so always use the uuid of the drive) In this case /mnt becomes the <ROOT_FS> of your system Btrfs pool. In your case probably /subvol.

You can easily locate the UUID of your HDs by doing
vdir /dev/disk/by-uuid
which will list them. Then, when you enter the mount command below, enter the first few characters of the desired UUID you just listed above and press the tab key. It will automatically fill in the rest for you.

c) Move @ and @home out of the way
mv @ @old
mv @home @homeold
Note: DO NOT delete @ (or @home), because they are still giving you access to the system directories despite being renamed. If you do delete @ all the system commands you are using will disappear and nothing will work on the CLI any more. You won't be able to use any Btrfs command, any mv command, vdir, ls, nothing.... Your only recourse will be to insert a LiveUSB and press the power button and use the LiveUSB to save your system.
Assume you have <ROOT_FS>/snapshots, which contain your @YYYYMNMDD and @homeYYYYMMDD snapshots.
btrfs su snapshot /mnt/snapshots/@homebefore /mnt/@home
syncbtrfs su snapshot /mnt/snapshots/@before /mnt/@
sync

You've now replaced @ with what was @before and @home with what was @homebefore.
Reboot. Fstab will mount @ and @home the way it always does, and you are now running @ as it was when you took the snapshot of @ and labeled it @before.

d) After you have booted back up you can mount the <ROOT_FS>, as shown above, and delete @old and @homeold.
btrfs subvol delete -c /mnt/@before
sync
btrfs subvol delete -c /mnt/@homebefore
sync
NOTE: Do not attempt to delete a snapshot using the rm or rmdir command. It won't work. Also, if you decide to delete /mnt/snapshots then first use btrfs subvol delete to remove all the snapshots it contains. Then use rmdir to delete the snapshots subdirectory.

5) Backing up.
Please remember that a snapshot is not a backup, since it resides under your <ROOT_FS> and if you were to lose that because of HD problems (lost superblocks, HD mechanical failure, etc) then you do not have access to your snapshots either.
a) mount your system pool as described above.

b) Mount your backup HD or USB
mount /dev/disk/by-uuid/theactualuuid /backup

c) Mount your live system (<ROOT_FS>)
mount /dev/disk/bu-uuid/theactualuuid /mnt

d) Make a snapshot of @ and @home, using the date to identify it.
btrfs su snapshot -r /mnt/@ /mnt/snapshots/@YYYYMMDD
sync
btrfs su snapshot -r /mnt/@home /mnt/snapshots/@homeYYYYMMDD
sync

e) Transfer the @YYYYMMDD and @homeYYYYMMDD ro snapshots to /backup
btrfs send /mnt/snapshots/@YYYYMMDD | btrfs receive /backup
sync
btrfs send /mnt/snapshots/@homeYYYYMMDD | btrfs receive /backup
sync


Notice that while taking snapshots is almost instantaneous sending a ro snapshot to a remote btrfs pool, like /backup, takes several minutes or more. That is because every file in the snapshot is transfered to the /backup pool. On my system it takes about 10 minutes to send & receive @YYYYMMDD and about 20-30 minutes to send & receive @homeYYYYMMDD because @homeYYYYMMDD has around 135GB in it while @YYYYMMDD has less than 30GB. However, the Konsole in which the send & receive is taking place can be minimized and the system can continue to be used while that process is taking place. I experience no obvious or any apparent slowdown in my use of my system.

A snapshot is a subvolume like any other, with given initial content. By default, snapshots are created read-write, unless the -r switch is used. File modifications in a rw snapshot do not affect the files in the original subvolume. That means that I can browse @homeYYYYMD and, if it is rw, delete a file from it. That file will remain in @home. IF I modify or delete a file in @home (in /home/youracct) that file will be copied into @homeYYYYMMDD as it was before the modification or deletion. It also means that I can copy a file into @homeYYYYMMDD (as long as it is rw) that is not in @home (in /home/youracct). If I then restore @home using @homeYYYYMMDD then @home will contain that file I copied into @homeYYYYMMDD.

Cleaning up.
Use a couple sync commands to make sure everything is done.
Then, umount /mnt and /backup (or what ever you mounted to do the snapshotting and backing up).
Use exit to leave root.
Use exit again to close the Konsole.

To make things easier I generally use bash command history to recall my previous mount and snapshot commands, modifying them to suit the date. One can also create alias variables that contain the commands, which allows each command sequence to be issued with a single alias command.
alias mountm='mount /dev/disk/by-uuid/uuuuiiiidddd /mnt'
etc...

[GreyGeek]

My problem --so far-- with Btrfs is there is no linear, up-to-date, instruction set. I'm not impressed with the wicki or other pages found on the web, most of them being several years old. There is a lot of good information here on the forum but to find it one has to do a search and click on each return and hope it covers what you are looking for. I'm a reasonably intelligent person but I just haven't found a source that works for me. Hell, I'd be happy with a BTRFS For Dummies book, right about now.

When you do your next search for Btrfs information set your browser to select the year or the month.

An excellent beginning place is the Kernel's btrfs wiki page:
https://btrfs.wiki.kernel.org/index....#Documentation

It time stamps its information.

Also, most webpages about Btrfs have a time stamp somewhere on the page. Anything over a year old I wouldn't use as being current. For example, you will see LOTS of pages warning about how Btrfs is "experimental" and not to use it in production. That warning was removed almost three years ago.

If you do "man btrfs" and scroll to the end of the man page you will see in the lower left corner a version number for btrfs. Mine shows v4.4 That means that when I look for information I prefer 4.4. If I see a more recent version I won't use it because I am not using that version.
The btrfs change logs is here:
https://btrfs.wiki.kernel.org/index.php/Changelog
It shows that version 4.4 was released in January of 2016 and the most recent version is 4.17. This corresponds to Kubuntu 16.04, which is what Neon is also based on. I haven't checked Bionic's version of Btrfs, but I'd wager it will be 4.15. Ditto for Neon based on 18.04, when it comes out.

IF you want to run v4.17 you can go to github and download it.
https://github.com/kdave/btrfs-progs

[oldgeek]

I've formatted the new btrfs backup HD, but am unable to mount it properly. The error message says '/backup mount point does not exist'. How do I make it exist? I was using step 5, by the way.

[GreyGeek]

If it is your backup HD, then you can mount it on any spare directory, say /mnt, or you can "sudo mkdir /backup" to create a mount point under /.

Then, as root, you can issue
mount /dev/disk/by-uuid/whatevertheuuidis /backup
After you have used it as the destination for btrfs send & receive, and have used "sync", then you can umount it: umount /backup

[oldgeek]

I did what you said and got the following error message:

root@steve-H55H-M:~# mount /dev/sdb1/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 /backup
mount: /backup: special device /dev/sdb1/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 does not exist (a path prefix is not a directory).

I did not create the new directory on the external HD, but in the internal HD--is that my problem? How do I do it on the external HD? As you can see, I'm not exactly a whiz using the terminal.

[GreyGeek]

I did what you said and got the following error message:

root@steve-H55H-M:~# mount /dev/sdb1/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 /backup
mount: /backup: special device /dev/sdb1/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 does not exist (a path prefix is not a directory).

I did not create the new directory on the external HD, but in the internal HD--is that my problem? How do I do it on the external HD? As you can see, I'm not exactly a whiz using the terminal.

Yes, that is your problem. You have to mount the external HD to a directory under "/" to be able to use it. That's why I recommended that you create the /backup directory. You are thinking like a Windows users in terms of C: or D:, etc..., where D: has no part of C:, and not as a Linux user. Don't worry, you'll get the hang of it. In Linux EVERYTHING is a file. Everything connects via a file. Printers, wifi, displays, sockets, etc.... The TOP of the file hierarchy is /, known as root. It has it's own account, /root, but there is generally nothing in it because most "root" commands are summoned using "sudo". Everything that connects to a Linux system must connect somewhere under /, or in the case of Btrfs, under <ROOT_FS>, i.e., ID=5, the top of the Btrfs pool.

As far as the actual mounting command, you didn't do what I wrote.
I wrote: open a Konsole and use "sudo -i" to become root. Then issue:

mount /dev/disk/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 /backup

or, from a Konsole issue:

sudo mount /dev/disk/by-uuid/4d4ceb48-355c-423b-ad3a-93469a1275d4 /backup


You replaced "disk" with sdb1. A logical mistake but incorrect. Literally, you are using the path to "4d4ceb48-355c-423b-ad3a-93469a1275d4", which is "/dev/disk/by-uuid/".

Hang in there! Rome wasn't built over night, but it burned down in one night, so don't set fire to anything!

[jlittle]

I've said this before here, but it might be worth repeating.

... The TOP of the file hierarchy is /, known as root...

I worked in directory tree structures for so long that I unconsciously assume that everything hangs down from the root. Drives have their own top level and "mounting" them meant hanging their trees from some suitable mount point, adding to the tree. Simple. (There are some mechanisms such as bind mounts that don't fit in that picture, but usually they're for specialised purposes.)

But with btrfs, subvolumes can be mounted independently, and if Ubuntu is booting from a btrfs drive, its root, /, is a subvolume, which is not the root of the drive (id=5, for some hoary reason). That btrfs root subvolume can be mounted in the usual hierarchy under the unix /. This means it's not a strict tree any more. You can cd round and round in circles if you want.

I recommend mounting the btrfs root, subvol=5, somewhere using /etc/fstab. I use /mnt/top, so /mnt/top/@ is the same place as /. It's easy to misplace subvolumes if they are created by something in its internals; APT does that. I would get confused about what was where, but going to the btrfs root now
I can work it out.

Regards, John Little

[GreyGeek]

Thanks for pointing that out, John. I was going to mention the Btrfs exception (because it is not an EXT*) but flat out forgot. Par for my course these days.

[oldgeek]

I did create the /backup directory, but it's in the root folder, as is /subvol with my snapshots and @ and @home. But the /backup appears on my internal HD, but not my external HD. I tried to create a /backup on the external HD, but I can't. So what do I do now in order to send the snapshots to the ext. HD as backup? It seems a little complicated at first, but I think I'm getting the hang of it--at least I've been abole to mount the HD and take the snapshots. By the way, I forgot th use the sync command while taking snapshots, but did it after both snapshots were taken. Is that all right?

[oldgeek]

Update: I realized I was still in Windows mode and so mounted my ext HD under /backup and proceeded with the send instructions for the @ snapshot I took yesterday and there are no error messages and the light on the HD keeps flashing and it's taking longer than a snapshot so I guess it's working all right. The @ backup is finished and so now I'm doing the @home backup,which will probably take a lot longer. I really appreciate your advice, GreyGeek. I'm not going to give up on this as it seems pretty simple once I practice the commands. Your help and others from this forum have been crucial. I am not a programmer and have never taken any computer courses but feel I have a little more knowledge than the average user plus some experience programming simple BASIC and Pascal programs (not so much with C) and Linux is very interesting for me. So is btrfs. I'll let you know how this turns out.

[oldgeek]

I now have a backup for my system. Thanks for all the help.