Beginner's guide to BTRFS

[GreyGeek]

@GreyGeek, you wrote the below in your first post here, but the link seems to take me to a conversation which is about having too many snapshots rather than related to balancing?



Also, would it make sense to update the section about trim to reflect the fact that a weekly trim might be already setup by the OS? And that people should run sudo systemctl status fstrim.timer before setting up a cron job? I guess we should avoid running trim twice.

Thanks for catching that. I've updated the #1 post to point to an explanation and added an additional link, and also added more discussion to fstrim.

[Thomas00]

Don't know what's wrong or right but monthly sounds good to me. For those who want to change the default systemctl job to monthly here's how it worked for me just now:

1. Run "sudo nano /etc/systemd/system/timers.target.wants/fstrim.timer" and change OnCalendar=weekly to OnCalendar=monthly
2. Reload daemons: sudo systemctl daemon-reload
3. Start trim: sudo systemctl start fstrim.timer
4. Check results: sudo systemctl status fstrim.timer

[oshunluvr]

IMO, running tirm weekly is excessive unless your drive is 80%+ full. I do mine monthly and even that is too often. 2-4 times a year is plenty for most users.

[Thomas00]

@GreyGeek, you wrote the below in your first post here, but the link seems to take me to a conversation which is about having too many snapshots rather than related to balancing?

Balance
reason to do balance: https://www.kubuntuforums.net/showth...ormance-Issues

Also, would it make sense to update the section about trim to reflect the fact that a weekly trim might be already setup by the OS? And that people should run sudo systemctl status fstrim.timer before setting up a cron job? I guess we should avoid running trim twice.

[Thomas00]

Continuing my little btrfs journey :-)
Have to say this was quite a learning experience

First of all, forget about my "is this a bug" talk above.
I am pretty sure it's by design, the desktop installer of Ubuntu/Kubuntu just does a few things automatically which a server guy will have to do manually.

Here's what I believe the Kubuntu installer does different to the Ubuntu server installation:
1. Create the btrfs filesystem followed by creating the @ and @home subvolumes. Like any new subvolume they will be empty.
2. Instead of copying the OS files to / they will be copied to the respective subvolumes. Aside from the two subvolumes FS_Root remains empty.
3. Updates grub to reflect booting from /@ instead of /

Pretty straightforward when done as part of the installation.

To achieve the same result manually a few more steps are required:

1. Create subvolume: btrfs subvolcume create /@
-> Now you got an empty @ subvolume under /

2. Change grub.cfg: nano /boot/grub/grub.cfg -> rootflags=subvol=@
-> For now let's only update the default menuentry 'Ubuntu' one. Once you sucesfully converted the machine you can do the others.
"linux /boot/vmlinuz-4.15.0-36-generic root=UUID=f6ca238e-c8b6-11e8-8c8d-bcaec52979b7 ro rootflags=subvol=@ maybe-ubiquity"

3. Grab the UUID of your / via blkid
4. Update /etc/fstab to reflect booting from /@ : nano /etc/fstab ...defaults, subvol=@ 0 0
5. Add FS_Root to /etc/fstab: UUID=<yourUUID> /mnt (You can do this later as well)

6. Copy the OS content of / into /@ : rsync -av --exclude @ --exclude proc --exclude sys / /@
-> I am excluding proc and sys because they produce errors and are dynamic anyway

7. Manually create the folders excluded from rsync: mkdir /@/sys, mkdir /@/proc
-> This is critical! If these folders are not there during boot you bricked your install. I know what I am talking about :-)
-> Now we have a exact copy of the OS in /@

8. Reboot
----- Drum roll ----
-> If the machine came up again your OS is running from /@ instead /
Run ll / and if you can't see @ you successfully switched your root to @
Check with mount | grep btrfs -> subvol needs to be =/@ not =/

root@homeserver:~# mount | grep btrfs
/dev/sdd2 on / type btrfs (rw,noatime,ssd,space_cache,autodefrag,subvolid=25 9,subvol=/@)
/dev/sdd2 on /mnt type btrfs (rw,relatime,ssd,space_cache,autodefrag,subvolid=5 ,subvol=/)
root@homeserver:~# ll /
total 32
drwxr-xr-x 1 root root 224 Oct 8 08:01 ./
drwxr-xr-x 1 root root 224 Oct 8 08:01 ../
drwxr-xr-x 1 root root 2430 Jul 25 23:04 bin/
drwxr-xr-x 1 root root 620 Oct 6 17:46 boot/

-> But the machine did not yet boot from /@ !!!

9. Update grub
update-grub
grub-install /dev/sdx
-> without this one your system will still try to boot from /boot instead of /@/boot

10. If you haven't done so already in step 5, mount the btrfs root to /mnt: mount /dev/sdx /mnt
-> Once everything is working add this mount to fstab, use the UUID instead of /dev/sdx

11. Delete the now obsolete OS folders in FS_ROOT, everything but /@ : rm -r /mnt/boot
-> Be careful with the rm command...just saying :-)
-> Start with "rm -r /mnt/boot" and reboot, if you got one of the previous steps wrong you will have to start all over, better to fail early I guess :-)
-> If the machine reboots, you can safely delete the other folders.

12. Update the remaining entries in grub.cfg similar to #2

Done!

This is what your folders should look like after you worked through all of this. I didn't bother creating a @home subvol for my server use case.

root@homeserver:~# ll /
total 32
drwxr-xr-x 1 root root 224 Oct 8 08:01 ./
drwxr-xr-x 1 root root 224 Oct 8 08:01 ../
drwxr-xr-x 1 root root 2430 Jul 25 23:04 bin/
drwxr-xr-x 1 root root 620 Oct 6 17:46 boot/
...
root@homeserver:~# ll /mnt
total 16
drwxr-xr-x 1 root root 2 Oct 8 08:14 ./
drwxr-xr-x 1 root root 224 Oct 8 08:01 ../
drwxr-xr-x 1 root root 224 Oct 8 08:01 '@'/
root@homeserver:~# mount | grep btrfs
/dev/sdd2 on / type btrfs (rw,noatime,ssd,space_cache,autodefrag,subvolid=25 9,subvol=/@)
/dev/sdd2 on /mnt type btrfs (rw,relatime,ssd,space_cache,autodefrag,subvolid=5 ,subvol=/)
root@homeserver:~#

Once you arrived here you can basically go back to the first post in this thread, create the /mnt/snapshots folder and your very first snapshot! Yay :-)
root@homeserver:/mnt# mkdir /mnt/snapshots
root@homeserver:/mnt# btrfs su snapshot -r /mnt/@ /mnt/snapshots/@clean

[Thomas00]

Yes, I figured that 5 is an important id, was reading up here: https://www.linux.com/learn/how-crea...s-linux-part-2
And I can see from my laptops setup that id 5 is my root

/dev/mapper/nvme0n1p3_crypt on / type btrfs (rw,noatime,ssd,space_cache,autodefrag,subvolid=25 7,subvol=/@)
/dev/mapper/nvme0n1p3_crypt on /mnt type btrfs (rw,relatime,ssd,space_cache,autodefrag,subvolid=5 ,subvol=/)
/dev/mapper/nvme0n1p3_crypt on /home type btrfs (rw,relatime,ssd,space_cache,autodefrag,subvolid=1 016,subvol=/@home)

But I am experimenting on a VM and took a snapshot so I can happily screw things up :-)
Just need to somehow create a @ which points to /, map this one as / in fstab and then I should be in business?

[GreyGeek]

http://manpages.ubuntu.com/manpages/...bvolume.8.html

A subvolume is a part of filesystem with its own independent file/directory hierarchy.
Subvolumes can share file extents. A snapshot is also subvolume, but with a given initial
content of the original subvolume.

Note
A subvolume in btrfs is not like an LVM logical volume, which is block-level snapshot
while btrfs subvolumes are file extent-based.

A subvolume looks like a normal directory, with some additional operations described
below. Subvolumes can be renamed or moved, nesting subvolumes is not restricted but has
some implications regarding snapshotting.

A subvolume in btrfs can be accessed in two ways:

· like any other directory that is accessible to the user

· like a separately mounted filesystem (options subvol or subvolid)

In the latter case the parent directory is not visible and accessible. This is similar to
a bind mount, and in fact the subvolume mount does exactly that.

A freshly created filesystem is also a subvolume, called top-level, internally has an id
5. This subvolume cannot be removed or replaced by another subvolume. This is also the
subvolume that will be mounted by default, unless the default subvolume has been changed
(see subcommand set-default). (BUT - Ubuntu has warned against doing that!)

A snapshot is a subvolume like any other, with given initial content. By default,
snapshots are created read-write. File modifications in a snapshot do not affect the files
in the original subvolume.

set-default [<subvolume>|<id> <path>]
Set the default subvolume for the (mounted) filesystem.

Set the default subvolume for the (mounted) filesystem at <path>. This will hide the
top-level subvolume (ie. the one mounted with subvol=/ or subvolid=5). Takes action on
next mount.

Bottom line: DO NOT use the set-default command to change the subvolid to any other value except 5.

[GreyGeek]

Never use
btrfs subvolume set-default
to change the root id to some value other than 5. IOW, don't use that command on Ubuntu based distros.

You can mount any Btrfs subvolume under your <ROOT_FS>, which is already running, but you are depending on @ and @home as your base of operations. They are mounted in fstab to create / and /home. But even though you think you may have a @ & @home operating system on /dev/sdd1 you won't get it by mounting 261.

[Thomas00]

I used the first one, https://www.ubuntu.com/download/server
But let me play around with the subvols a bit more, the more I think about this the more I agree that in a server environment it would make sense to let the admin decide what snapshots are required. Hence, by design not a bug.
Just found this "subvolid" mount option which may do what we want: mount -o subvolid=261 /dev/sdd1 /btrfs/

[GreyGeek]

Which Ubuntu server version are you using?
One from here:

https://www.ubuntu.com/download/server

Or, one from the alternate site?

If you require advanced networking and storage features such as; LVM, RAID, multipath, vlans, bonds, or re-using existing partitions, you will want to continue to use the alternate installer.

https://www.ubuntu.com/download/alte...rver-installer

I'd recommend the amd ISO from the alternate site:

http://cdimage.ubuntu.com/releases/1...rver-amd64.iso

[Thomas00]

This is not Kubuntu Bionic, it's Ubuntu server and no, no secret sauce. Bare metal from a USB stick.
But given that I can replicate this behaviour even in a Virtual Machine I was thinking if maybe the server install is just handling btrfs differently and by default is stopping short on creating the two default subvolumes?
Leaving it to the admin to manually create @ and @home and update fstab

I got my server VM running right now, trying to create those subvolumes manually, so far whatever I do looks slightly different to what I see on my laptop...

[GreyGeek]

I doubt that it is a "bug" in the installer. No one else installing Kubuntu Bionic using Btrfs is posting your "bug".

More than likely you are overlooking some aspect of how you use the partition manager, or encryption, or the installer itself. So, what extra sauce are you using before you attempt an install? Are you using some type of encryption software on your HD before you install? Are you still trying to put / on one partition and /home on another, while assigning Btrfs to both?

Can't you just do a normal install and then add encryption to the @ and @home later, in the fstab?

[Thomas00]

Two installs later which both ended up with the same result, no btrfs subvolumes, I think I found a hint where things go wrong.
This time I decided to turn on the detailed log when installing and as messages were flipping by I noticed a "..etc/fstab warning". Too quick to actually read it but enough to make me look for an installation log and there's the error again. Somehow the installer picks up a key type encrypted which then seems to make things fall over.

thomas@homeserver:~$ cat /var/log/installer/installer-journal.txt | grep fstab
Oct 05 14:50:42 ubuntu-server curtin_log.1919[2187]: {'WORKING_DIR': '/tmp/tmpgzcfbsxb/scratch', 'OUTPUT_FSTAB': '/tmp/tmpgzcfbsxb/state/fstab', 'OUTPUT_INTERFACES': ...
Oct 05 14:52:56 ubuntu-server curtin_log.1919[2187]: cryptsetup: WARNING: could not determine root device from /etc/fstab
Oct 05 14:53:12 ubuntu-server curtin_log.1919[2187]: This error is probably caused by an invalid /etc/fstab
Oct 05 14:53:15 ubuntu-server curtin_event.1919[2731]: start: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: writing etc/fstab
Oct 05 14:53:15 ubuntu-server curtin_event.1919[2731]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: SUCCESS: writing etc/fstab
Oct 05 14:53:15 ubuntu-server curtin_log.1919[2187]: start: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: writing etc/fstab
Oct 05 14:53:15 ubuntu-server curtin_log.1919[2187]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: SUCCESS: writing etc/fstab

Looked for more details on crypt:

thomas@homeserver:~$ cat /var/log/installer/installer-journal.txt | grep crypt
Oct 05 14:48:49 ubuntu-server kernel: Key type encrypted registered
Oct 05 14:48:49 ubuntu-server systemd[1]: Reached target Local Encrypted Volumes.
Oct 05 14:49:58 ubuntu-server ureadahead[1015]: ureadahead:/usr/lib/tmpfiles.d/cryptsetup.conf: Error retrieving chunk extents: Operation not supported
Oct 05 14:52:56 ubuntu-server curtin_log.1919[2187]: cryptsetup: WARNING: could not determine root device from /etc/fstab

No idea how cryptsetup gets into this, a leftover flag which survives formatting? But I don't think the boot drive was ever encrypted, I also removed my backup drive and tried to install again, just in case.

I then went through the same process again with ext4, same error but obviously the server is functioning as there is no subvolumes concept with ext4
Yet another install with all partitions deleted first, same error.

So I went ahead an created a VM and installed the server with btrfs...
Same cryptsetup error...so clearly this has got nothing to do with my hardware.
Guess what, same error! No sub-volumes and fstab not properly set up!

The cryptsetup warnning may be a false flag but I'd say this looks like a bug in the Ubuntu 18.04 server installer?

By the way, Oshunluvr, this seems to take a completely different direction, maybe you want to consider to move this conversation into a separate thread?

[Thomas00]

OBVIOUSLY, make a backup of any data you don't want to lose in case something goes sideways again.

I like that
There's a single BTRFS partition on the boot drive and I have a good backup, so I guess there is nothing to loose...I think I could even let it format and start really clean, the backup is recent (and is working) I won't miss a thing.
I documented all the installation steps I made on Wednesday to get the server back in shape again so this new attempt should finish a lot quicker. Think I just convinced myself to go through it again :-)

Oshunluvr and all the others on this thread, thanks for having patience, I am not trying to be difficult but rather want to understand what I do and why I do it. This case of a BTRFS file system without any sub-volumes may have been a good playground to learn things, but then again, I need this machine in production again.

[oshunluvr]

I do manual partitioning all the time and it works fine, just don't set and mount points except / and it should work fine.

BTW, if you already have a single BTRFS partition, there's no need to re-partition and format. Simply install again, this time point the installer at your existing BTRFS partition and do not check format - just set the mount point as /

It will install into the BTRFS partition without wiping your current install and you should be able to boot to both installs - your current and the new one. Then, you can transition from to new install - transfer settings or docs, whatever - and when ready just delete the files from the old install leaving @ and @home intact.

OBVIOUSLY, make a backup of any data you don't want to lose in case something goes sideways again.