Snapper v Btrbk v Manual

[GreyGeek]

:~$ vdir /etc/mtab
[COLOR=#ff0000][B]l[/B][/COLOR]rwxrwxrwx 1 root root 19 Jan  5  2020 /etc/mtab[COLOR=#ff0000][B] -> [/B][/COLOR]../proc/self/mounts

So much for my "keen powers of observation"...

Now you know why I "KISS" it with a flat snapshot structure.

[jlittle]

...snapper can also be used as an automated snapshotting tool (even if you don't use it's rollback/recovery tools, and do all that manually)...

When I adopted snapper the rollback/recovery tools were rudimentary, and documented well (as far as I haphazardly found) for single file recovery, so I have been doing "all that manually", as part, or as another dimension, of a menagerie of methods (vim timestamped saves, git, backups to internal hard drive, to external drives, uploads to a VPS, ...). But what sold me on snapper was recovering two files in a couple of months, because they had been automatically snapshotted.

I suppose I could easily cobble together a cron job or a systemd timer to make regular snapshots. But then I'd have to clean them up, keeping some of them with metadata about them, and write a hooks for APT and booting. A point I've made before, that's what snapper is, why reinvent it.

I should do a deeper dive into what snapper does now. Being installed as part of some distros means a lot of users have been testing it for me. Like Fedora 34 using btrfs as its default fs should result in improvements in robustness and performance.

[kubicle]

It pre-lists /etc/mtab, the mount table, and won't revert changes made to it. That doesn't make any sense to me.

/etc/mtab in a current linux system is a symlink to /proc/self/mounts that lists currently mounted filesystems (/proc is a special virtual filesystem that is generated at boot, and not a part of the physical / filesystem), even with a manual rebase /etc/mtab contents are regenerated at boot. I'm not sure why snapper specifically lists this for exclusion (possibly something to do with how it does the "online" rollbacks or possibly to accommodate "old style" mtab-files [which also don't survive reboots], because reverting list of currently mounted filesystems is not a good idea on a running system...it's not a problem if you rollback "externally", but snapper's rollback tools can also be used "online").

When you see something that doesn't seem to make sense, a good rule of thumb is that there is probably a reason for it being the way it is (in this case it's likely the combination of facts that you *never* need to revert mtab [nor would doing have any effect on a rebooted system, it would have a "new" mtab anyway], and that it is a bad idea to do so on a running system [because mtab is used, for example, when the system unmounts filesystems during shutdown, and not having mtab accurately describing mounted filesystems could cause some of them not being unmounted cleanly before power off]...with symlinked mtab that is not a problem, but some systems might be using traditional mtab files).

IF I want to roll back to snapshot 2 from snapshot 3 I want 2 in its entirety, not just pieces of it. IF there is something in 3 I want to keep, but roll back everything else, I'd just copy that something to the #2 snapshot and then do the rollback.

Sometimes it makes sense to exclude parts from reverting back like /var/log (to see what was wrong and/or keep the logs you have accumulated since the snapshot), or other changes you made to to the filesystem that you know are not a problem. You can get them back by copying, but if you do a lot of rollbacks it might be simpler/quicker to add these to exclusion. It's a nice feature to have, even if you don't use it (or need it).

That's what I like about my flat system. When I roll @ back to @yyyymmdd I know that I am getting @ just the way it was in every respect, when I made the @yyyymmdd snapshot. Restoring just a thing or two from @yyyymmdd to @ is easy to do as well. Just use mc to browse to @yyyymmdd in the left panel, @ in the right panel, and copy what I want from the left panel to the right panel. I know one can use snapper to rollback (undochange) specific files but I'm getting too old to wrap my brain around snapper.

You can do all that with snapper, too. Just by doing the rollbacks manually (and avoid it's automated rollbacks). I also have my own automated system in place, so I don't need snapper, but snapper can also be used as an automated snapshotting tool (even if you don't use it's rollback/recovery tools, and do all that manually). It's snapshots can be used just as well with manual recovery/copy whatever, if you keep in mind the subtle differences between nested/non-nested subvolumes. Like I said, nothing wrong in doing everything manually or with your own automation, but not everyone has the time, motivation or skills to reliably do that, and could use snapper fairly easily. It's not a bad tool (even for advanced users).

[GreyGeek]

I always do manual rollbacks.

Snapper won't roll back any files listed in /etc/snapper/filters/*.txt
It pre-lists /etc/mtab, the mount table, and won't revert changes made to it. That doesn't make any sense to me.
IF I want to roll back to snapshot 2 from snapshot 3 I want 2 in its entirety, not just pieces of it. IF there is something in 3 I want to keep, but roll back everything else, I'd just copy that something to the #2 snapshot and then do the rollback.

That's what I like about my flat system. When I roll @ back to @yyyymmdd I know that I am getting @ just the way it was in every respect, when I made the @yyyymmdd snapshot. Restoring just a thing or two from @yyyymmdd to @ is easy to do as well. Just use mc to browse to @yyyymmdd in the left panel, @ in the right panel, and copy what I want from the left panel to the right panel. I know one can use snapper to rollback (undochange) specific files but I'm getting too old to wrap my brain around snapper.

[kubicle]

That is an interesting comment in Reddit. The remark that all snapshots are read only is wrong, though. A snapshot is read only ONLY IF one includes the "-r" switch when creating it. Without the "-r" switch the snapshot is read-write.

You are right of course. He could have meant to say that snapper's snapshots are created read-only by default (so it says in the manual, already purged snapper, so can't check) when using btrfs, but even if he did, that's poorly worded.

The necessity to add an entry for /.snapshots/xx/snapshot in fstab to avoid the multiple mirrors syndrome encumbers and complicates things considerably, at least for my 80 year old brain which, admittedly, is not as agile as it used to be. That's why I like to "KISS" it and keep everything hanging off of the <ROOT_FS> and operate in the manual mode.

It doesn't say to add individual snapshots (/.snapshots/xx/snapshot) into fstab, just the @/.snapshots subvolume (remember that the individual snapshot subvolumes are nested, they appear automatically if the parent is mounted).

The "multiple mirrors" (if you are referring to the recursivity), would only happen if snapper would keep making snapshots when you boot into a snapshot (from grub...it doesn't, like jlittle mentioned...adding an fstab entry should make creating snapper snapshots possible, even when booted into a snapshot) ...or possibly after an automated snapper rollback (I don't know enough of snapper internals to say how it performs the roll-backs, never done one...never plan to do one ). If I would use snapper, I'd do any rollbacks manually (just to be sure I know what's happening).

[GreyGeek]

That is an interesting comment in Reddit. The remark that all snapshots are read only is wrong, though. A snapshot is read only ONLY IF one includes the "-r" switch when creating it. Without the "-r" switch the snapshot is read-write.

The necessity to add an entry for /.snapshots/xx/snapshot in fstab to avoid the multiple mirrors syndrome encumbers and complicates things considerably, at least for my 80 year old brain which, admittedly, is not as agile as it used to be. That's why I like to "KISS" it and keep everything hanging off of the <ROOT_FS> and operate in the manual mode.

[kubicle]

Nested snapshots have the... feature ... that they stay where they're nested. If, like me, one mucks about booting into various snapshots to try out stuff, like release upgrades, or to keep an old release for testing, snapper keeps running in only one of the boots. I keep forgetting this.

Would I have trouble if I move the .snapshots subvolumes to follow the booted subvolume?

Your guess is probably as good as mine

While I can navigate my way around btrfs subvolumes and snapshots, my first experience with running snapper was hours ago...so I'm not at all familiar with snapper's inner workings or configs.

You could try something like this (but I cannot guarantee it'll work as expected):
https://www.reddit.com/r/btrfs/comme...mountpoint_to/

[jlittle]

Nested snapshots have the... feature ... that they stay where they're nested. If, like me, one mucks about booting into various snapshots to try out stuff, like release upgrades, or to keep an old release for testing, snapper keeps running in only one of the boots. I keep forgetting this.

Would I have trouble if I move the .snapshots subvolumes to follow the booted subvolume?

Sent from my VFD 822 using Tapatalk

[kubicle]

Absolutely no harm done. One of the pros of a flat structure that it is generally easier to grasp a hold of (both conceptually and procedurally). Especially for people who have spent years working with earlier file systems and their partitioning schemes (flat structure shares some resemblance to that, while nested can feel somewhat alien...at least at first). Of course, snapper does it best to automate the restoration process so one could avoid the pitfalls (nested subvolume can be trickier to work with manually, even though it's possible to do the same things manually as with a flat structure).

• Nested subvolumes are not going to be part of snapshots created from their parent subvolume.

This is a necessary "restriction". If snapshots of a subvolume would include nested "child" subvolumes, you'd see the endless recursion of snapshot_of_@>>@>>snapshot_of_@>>@>>snapshot_of_@> >ad_infinitum in snapshots (that I think you referred to earlier in the thread). Because of this "restriction", that doesn't happen (note that the subvolume separation is also true with a flat structure subvolumes, if you take a snapshot of @, it does not include @home because they are separate subvolumes, just like nested subvolumes are).

This feature can also be a pro, for example, in the regard that by clever planning, you can easily exclude certain parts of the filesystem (that don't need to be snapshotted) from snapshots if you make those parts nested subvolumes (to save space,disk I/O etc.).

But like I said the pros and cons of flat, nested or mixed structure greatly depends on your needs and preferences (it's fairly difficult to give general recommendations that would fit all).

[GreyGeek]

I see my problem: I was confusing Nested subvolumes with Flat subvolumes.
https://btrfs.wiki.kernel.org/index.php/SysadminGuide

Flat (which is what I use)

Flat

Subvolumes are children of the top level subvolume (ID 5), typically directly below in the hierarchy or below some directories belonging to the top level subvolume, but especially not nested below other subvolumes,...

With one of the advantages being:

Everything in the volume that's not beneath a subvolume that has been mounted, is not accessible or even visible. This may be beneficial for security, especially when being used with snapshots, see below.

Nested (which I've never knew about, or if I ever did I'd forgotten about it)

Nested

Subvolumes are located anywhere in the file hierarchy, typically at their desired locations ..., especially below other subvolumes that are not the top-level subvolume

Its advantages and disadvantages are

This has several implications (advantages and disadvantages):

• Subvolumes don't need to be mounted manually (or via fstab) to their desired locations, they "appear automatically" at their respective locations.
• For each of these subvolumes the mount options of their mount point applies.
• Everything is visible. ...
• Nested subvolumes are not going to be part of snapshots created from their parent subvolume.

That last disadvantage is what I misinterpreted as meaning that subvolumes can't be nested. In a normal install two subvolumes are created under the <ROOT-FS> volume: @ and @home. They are mounted separately in fstab. If one navigates @ they will see the /home folder but will notice it is empty. The /home folder in snapshots of @ will be empty as well, requiring the user to snapshot both @ and @home to create a complete archive. That I've known from the beginning of my usage of Btrfs and assumed that snapshots can't be nested unless a normal directory is between them, such as /n/ in the snapshot situation.

With Snapper the root snapshot hierarchy under / is
/.snapshot/
which contains a normal folder with a numeric label n.
/.snapshot/n/
which contains
info.xml The meta data for the n'th snapper snapshot
snapshot The actual snapshot of @.

I don't have a @home subvolume.
Sorry for the mixup, Kubicle. (a.k.a. E.F. Hutton!)

[kubicle]

When I do "sudo btrfs subvol list /"

[FONT=monospace][B][COLOR=#000000]:[/COLOR][COLOR=#5454ff][B]~[/B][/COLOR][COLOR=#000000]$ sudo btrfs subvol list / [/COLOR][/B]
ID 1694 gen 965705 top level 5 path snapshots/@202103142159 
ID 1695 gen 969969 top level 5 path @ 
ID 1696 gen 966534 top level 5 path snapshots/@202103152204 
ID 1697 gen 967268 top level 5 path snapshots/@202103162149 
ID 1724 gen 968231 top level 5 path snapshots/@202103171903 
ID 1725 gen 969768 top level 5 path snapshots/@202103182218[/FONT]

All of my snapshots also have an ID, but that doesn't mean that they are under @. It means that they are under the <ROOT_FS>. My @ ID is 1695. The "Parent ID" of @ is 5, which is <ROOT_FS>. @ is a child of the Top Level ID 5, which is <ROOT_FS>, i.e., the base subvolume formatted onto the disk when Ubuntu was installed with Btrfs as the root fs, and onto which every thing else is written.

My snapshots do NOT have "@" prefixed in front of their paths.
Snapper snapshots show the path "/.snapshot/n/, which contains info.xml and snapshot, the actual suvolume.

Sorry for the delayed response to this one, I had to actually install snapper (and create a single snapshot with it) so I could show you the structure of snapper's nested structure:
(this is a btrfs partition mounted into /mnt after making a snaphot)
ID 257 gen 630120 top level 5 path @
ID 297 gen 630117 top level 257 path @/.snapshots
ID 298 gen 630116 top level 297 path @/.snapshots/1/snapshot

both the @/.snapshots (ID 297) and the @/.snapshots/1/snapshot (ID 298) are individual subvolumes (that can be mounted/worked with independently from @, like all subvolumes can be). But, since both are nested [297 is a child of 257=(@) and 298 is a child of 297] they get automatically mounted as /.snapshots and /.snapshots/1/snapshot if @ is mounted to /. This does *not* mean they are inside @ (they aren't), they are separate subvolumes (and can be worked with like separate subvolumes in all practical and theoretical purposes one would associate with non-nested subvolumes, they only appear to be inside @ if you mount @ (because they are nested subvolumes).

In case I haven't worded it strongly enough previously, it's utterly impossible to create snapshots (or any subvolumes) that are actually inside another subvolume with btrfs, it simply cannot be done. Not by snapper, not manually. All snaphots are individual subvolumes.

For other comments, see my previous post.

[kubicle]

I can access Snapper's root snapshots as root at /.snapshot without mounting the <ROOT_FS> (the HD or SSD) to /mnt. I can also access @home's snapshots at /home/.snapshot without mounting <ROOT_FS> As root I can cd to / and see /.snapshot.

Yes you can, because nested snapshots do get mounted automatically if you mount the "parent" subvolume (this is one of the benefits of having a nested structure, as I mentioned there are pros and cons to either structure). In case of snapper, when @ is mounted, the nested snapshot subvolumes get mounted to @/.snapshots/ automatically (the same goes for @home and it's nested snapshots, that are mounted to @home/.snapshots/). You don't need to mount them separately from @ (but you can if you need to or want to).

This is conceptually the same than what happens if you mount the <ROOT_FS>, it's nested "child" subvolumes ( for example @ and @home), appear automatically into the mountpoint, but they can (and usually are) also be mounted separately like mounting @ to / and @home to /home...but you can also mount @ to /mnt or /path/to/wherever...you can do the same with nested snapshot subvolumes independently of their "parent". This also explains why you can't "cd" to the parent from the directory you have mounted a nested subvolume in, as this "hides" the parent (if you mount @ at /, you can't "cd" from / to <ROOT_FS> [the parent of @], similar thing happens if you mount a nested snapshot somewhere)...this does not mean you can't do all the btrfs magic with the nested subvolume like replacing @ with a nested snapshot of @ (this process goes quite the same as with a non-nested snapshot, with mainly subtle differences))

.

IMO, snapper's snapshots are definitely INSIDE @, which is mounted as / in fstab

This means you haven't fully grasped the concept of nested subvolumes. They are not inside @, it only seems that way.
I can say I'm absolutely sure I'm correct (this is not an opinion), but you don't have to take my word for it:

Subvolumes can be nested and each subvolume (except the top-level subvolume) has a parent subvolume. Mounting a subvolume also makes any of its nested child subvolumes available at their respective location relative to the mount-point.

(nested) Subvolumes don't need to be mounted manually (or via fstab) to their desired locations, they "appear automatically" at their respective locations.

(quotes above from: https://btrfs.wiki.kernel.org/index.php/SysadminGuide)
This is a good read and explains most of the concepts quite well.

IMO, snapper snapshots ARE inside @, or one would have to mount <ROOT_FS> to see them.

This isn't correct, I'm not voicing my opinions, only what can actually be read in btrfs documentation.

I don't have any @ snapshots inside my / hierarchy so when @ doesn't mount, which has happened only 3 times in the last 5+ years, I go the LiveUSB route and restore @ from a snapshot that is obviously not under @. For problems not arising from a busted @ subvolume I mount my SSD to /mnt as <ROOT_FS> and do the manual commands necessary to roll @ back to a snapshot I created just before I created the problem or the updates failed. Snapshots take about a minute, including incremental backups to my remote medium, and a rollback takes about 2 minutes. A minute later, after the reboot, I am working on my desktop again.

I do not doubt that at all, nor am I suggesting that nested snapshots are somehow better than what you are doing (I quite prefer having "non-nested" snapshot structure myself, and I'm not alone in that). I'm just telling you that all that can be also be done with nested snapshots (like the ones used by snapper)...and trying to correct the misconceptions you're voicing about nested snapshots, like the idea that it is possible to have an issue with @ that is recoverable with a non-nested snapshot of @, but not recoverable with a nested snapshot of @...or that you need to be able to boot into @ or mount @ to access snapper snapshots.

[GreyGeek]

I can access Snapper's root snapshots as root at /.snapshot without mounting the <ROOT_FS> (the HD or SSD) to /mnt. I can also access @home's snapshots at /home/.snapshot without mounting <ROOT_FS> As root I can cd to / and see /.snapshot.

However, I cannot "cd .." upward to the level of <ROOT_FS>, hence <ROOT_FS>, @ and @home are not accessible to root without mounting <ROOT_FS>.

IMO, snapper's snapshots are definitely INSIDE @, which is mounted as / in fstab.

Btrfs gives every subvolume an ID, and a Btrfs command even shows what cannot be accessed even as root unless one mounts <ROOT_FS>

[FONT=monospace][COLOR=#000000]:[/COLOR][COLOR=#5454ff][B]~[/B][/COLOR][COLOR=#000000]$ [B]sudo btrfs subvol show / [/B][/COLOR]
@ 
        Name:                   @ 
        UUID:                   e11be9cf-95ff-f640-834e-0ed05439b51b 
        Parent UUID:            d2001490-2064-2a4c-a870-1be84141a78a 
        Received UUID:          - 
        Creation time:          2021-03-15 14:43:28 -0500 
        Subvolume ID:           1695 
        Generation:             969955 
        Gen at creation:        965705 
        [B]Parent ID:              5[/B] 
        Top level ID:           5 
        Flags:                  - 
        Snapshot(s): 
                                snapshots/@202103152204 
                                snapshots/@202103162149 
                                snapshots/@202103171903 
                                snapshots/@202103182218 
        Quota group:            0/1695 
          Limit referenced:     - 
          Limit exclusive:      - 
          Usage referenced:     102.32GiB 
          Usage exclusive:      241.61MiB[/FONT]

When I do "sudo btrfs subvol list /"

[FONT=monospace][B][COLOR=#000000]:[/COLOR][COLOR=#5454ff][B]~[/B][/COLOR][COLOR=#000000]$ sudo btrfs subvol list / [/COLOR][/B]
ID 1694 gen 965705 top level 5 path snapshots/@202103142159 
ID 1695 gen 969969 top level 5 path @ 
ID 1696 gen 966534 top level 5 path snapshots/@202103152204 
ID 1697 gen 967268 top level 5 path snapshots/@202103162149 
ID 1724 gen 968231 top level 5 path snapshots/@202103171903 
ID 1725 gen 969768 top level 5 path snapshots/@202103182218[/FONT]

All of my snapshots also have an ID, but that doesn't mean that they are under @. It means that they are under the <ROOT_FS>. My @ ID is 1695. The "Parent ID" of @ is 5, which is <ROOT_FS>. @ is a child of the Top Level ID 5, which is <ROOT_FS>, i.e., the base subvolume formatted onto the disk when Ubuntu was installed with Btrfs as the root fs, and onto which every thing else is written.

My snapshots do NOT have "@" prefixed in front of their paths.
Snapper snapshots show the path "/.snapshot/n/, which contains info.xml and snapshot, the actual suvolume.

IMO, snapper snapshots ARE inside @, or one would have to mount <ROOT_FS> to see them. It may arise that @ is not mountable but can still be browsed. If that happens then it may be possible to create a new @ from /.snapshot/n/snapshot and boot into it, which is a "rollback" method that Snapper does not do nor does it support, because one would have to boot into a LiveUSB, mount the HD to /mnt as root and do the work manually.

I don't have any @ snapshots inside my / hierarchy so when @ doesn't mount, which has happened only 3 times in the last 5+ years, I go the LiveUSB route and restore @ from a snapshot that is obviously not under @. For problems not arising from a busted @ subvolume I mount my SSD to /mnt as <ROOT_FS> and do the manual commands necessary to roll @ back to a snapshot I created just before I created the problem or the updates failed. Snapshots take about a minute, including incremental backups to my remote medium, and a rollback takes about 2 minutes. A minute later, after the reboot, I am working on my desktop again.

[kubicle]

then nothing inside of @ is accessible either.

But the snapper snapshots at @/.snapshots/ are not actually "inside" @ (it only looks that way on the surface, because they are nested), they are separate subvolumes that can be mounted separately/independently from @ (although, due to them being nested, they do also get automatically mounted if you do mount @...which can obviously confuse one to think that they are inside @). You do not need the ability to mount @ and/or browse it.

I could not browse @ after mounting my /dev/sda1 to /mnt, but I could access /mnt/snapshots/@yyyymmdd and create a new @ from it.

Then you could also have mounted your snapshot @yyyymmdd even if it had been nested within @ (like in @/path/to/whatever), and also replace @ with it. There exist no issues that you could have on @ that would only affect one type of snapshot and not the other (nested/non-nested).

The trick with the above is not to mount /dev/sda1 and try to browse @, but to mount the nested snapshot subvolume directly (if you wish to browse the contents of it), or you can simply mount /dev/sda1 and then use the btrfs command rundown to replace @ with the nested snapshot...in a mostly similar way you probably did with your non-nested snapshot).

Btrfs developers are reasonably smart people. If there were issues with nested subvolume becoming inaccesible if the parent is inaccessible while non-nested subvolumes would still function, they wouldn't have made nesting subvolumes possible (the snapper devs wouldn't have been using nested snapshots...and we wouldn't be having this discussion). There are file system issues that do make snapshots inaccessible or unusable, but these issues affect both types in a similar way.

[GreyGeek]

To do ""sudo mount -o subvolid=259 /dev/sdb1 /mnt" " you have to boot some platform, and one with Btrfs loaded or loadable. Hence the LiveUSB or whatever. My own experience is that if @ is corrupted and won't boot then nothing inside of @ is accessible either. I could not browse @ after mounting my /dev/sda1 to /mnt, but I could access /mnt/snapshots/@yyyymmdd and create a new @ from it. As you said in #4 "you can easily make a top-level @snaphots subvolume and mount that into @/.snaphots to store the snaphots outside of @ (but like above this doesn't help in case of @ corruption, they are just as worthless as the ones stored inside @)" That's what I've been saying.