BTRFS saves the day again...

[oshunluvr]

TL/DR: Use btrfs and subvolumes, make a snapshot and backup every day, and in minutes you can return to your previous state and you'll never lose more than a day's effort.

Full story:
My KDEneon install was still on kernel 6.8.0 and there have been a lot of changes in the newer kernels. quite a few re. btrfs and more are coming in the next couple releases as well. I hadn't upgraded until yesterday because I was using the amdpro drivers and they aren't compatible with the kernels above 6.8. A bit of reading revealed that much of the gains from using the pro drivers have been exceeded by the open source driver with the the newer kernel so yesterday morning I removed the amdpro drivers and installed the 6.14 kernel.

However, when I rebooted the 6,14 kernel hung and I ended up having to mash the reset button. This time I selected the 6.8 kernel and it booted fine. I started looking into what went wrong. I realized I had removed the dkms pro drivers and installed the new kernel in the same session so I thought maybe initramfs needed an additional rebuild. So I did that and was then able to successfully boot into 6.14.

When I began to use the system I started with my usual launching of my preferred web browser - Vivaldi - but something was terribly wrong with it. I received an error referring to my Vivaldi account and that it couldn't log in. This meant my links, passwords, history, etc., was gone. This was more than a little concerning and annoying so I set about trying to fix it. 30-40 fruitless minutes later just left me more perturbed.

Then it dawned on my that I have set my system up to take snapshots of my three subvolumes (root, home, and user cache). at 5:30am before I am usually awake. So I restored all three snapshots and rebooted and viola! I was back in business. I then repeated my amdpro driver removal, rebooted, installed 6.14 and rebooted again. I launched Vivaldi and it was normal as it had been the day before. All I did was waste 40 minutes trying to fix something that was easily recovered in less than a minute. I have no idea what caused the problem with Vivaldi but it doesn't matter since I literally lost nothing.

The snapshots and backups:
Rather than use a third party tool like btrfsbk or snapper or timeshift. I decided long ago to customize a snapshot and backup script for my own use. This was so I have total control over what happens. The early days of snapper and the others, they were notorious for filling up your drive with snapshots and I wanted to avoid this at all cost. Over the years, I have changed how my script functions to met my changing preferences. When I built my current computer, my new motherboard has 4 nvme slots and I took advantage of that immediately by installing 4x1tb nvme drives. I also had a 2tb SSD that was not being used (it had been a work device and I retired so it was in a drawer) so I installed that too. This gave me a tremendous amount of usable storage so I was able to expand my snapshot and backup procedures to take advantage of the available space.

My script runs this as a daily routine:
Every morning a snapshot is taken of all three subvolumes on my primary nvme drive.
The snapshots are retained for "rolling" 2 weeks, meaning when I have 14 daily snapshots the oldest is deleted.
The newest snapshots of root and home (not cache) are then incrementally sent as backups to a second nvme drive.
Once a week, on the backup nvme drive a snapshot is taken of the backup subvolumes.
The weekly backup snapshots are retained for 90 days, then deleted.

The end results:
The boot nvme drive has 2 weeks worth of snapshots available - a total of 45 snapshots - one per day of each of the 3 subvolume for two weeks, plus today's.
The backup nvme drive has today's root and home backups plus 90 days of Sunday snapshots for a total of 16 snapshots - 8 per subvolume.
I exclude my user cache from backups even though it has a daily snapshot.
I also moved my steam folder to a separate drive so it is not in the backup script. I occasionally make a backup of it but I don't really play much on steam so there's little need to automate a backup for it.
Also on the boot nvme drive are two other installs - Kubuntu 24.04 and a dedicated grub install.

All of this results in the boot nvme drive having 577gb used of 932gb available - 38% free space. The backup nvme drive has 608gb used and 35% available free space.

[jfabernathy]

Glad it worked out for you and saved the day. I've also been saved by BTRFS snapshots. Recently I've been testing which snapshot system provides the best support for me without doing all of it myself. I tested Snapper with btrfs-assistant and Timeshift.


I did the tests all on the newest Debian 13 Trixie KDE. It turns out that if you only need snapshots of / and /home, Timeshift worked the best for me. If things were totally screwed and I could not boot using grub, I could boot the KDE Install ISO of Debian 13 and install Timeshift and it would find my old snapshots on my SSD and allow me to restore them. I could also have grub-btrfs installed and boot old snapshots to see if my last snapshot was okay before restoring from it. I only have to use the "boot the install ISO method" if the system will not boot any snapshot from grub.

Snapper and btrfs-assistant are more flexible about which subvolumes are snapshotted, and btrfs-assistant is very powerful with it's restore capabilities. but if you can't boot at all, then booting an ISO and getting btrfs-assistant ready to do the restore is a lot of work. Maybe more than doing it your way.

In my case I don't need the backups of snapshots since I have no local data that isn't temporary. I keep my finished work on a Synology NAS and that NAS is backed up to IDrive every night.


I don't know how we did things before BTRFS snapshots. I still have flashbacks about dropping a large deck of IBM punch cards with my latest creation on it.

[jlittle]

Thanks for the write-up.

Once a day seems tame to me. My main install does hourly snapshots. I've met users online that have set 5 minute snapshots.

Of course, with frequent snapshots one better have something managing them, and exclude likes of browser caches from the frequently snapshotted subvolumes. I use snapper in a limited way, just to snapshot frequently and prune its snapshots.

Folks compare file systems on reliability, and don't rate btrfs well due to (IMO) historic problems. But, at least in my case, I'm the most likely cause of problems, and non-snapshotting filesystems force reliance on backups.

[oshunluvr]

I don't know how we did things before BTRFS snapshots. I still have flashbacks about dropping a large deck of IBM punch cards with my latest creation on it.

LOL, I had a love/hate with those old punch cards in high school. Carrying a card box to and from my locker was a real treat too. When I got to college, we had paper tape so impossible to mix up the card stack.

Before BTRFS, backups were a real chore. I lost data several time because I got lazy. Now, I just let it happen in the background.

[oshunluvr]

Thanks for the write-up.

Once a day seems tame to me. My main install does hourly snapshots. I've met users online that have set 5 minute snapshots.

Of course, with frequent snapshots one better have something managing them, and exclude likes of browser caches from the frequently snapshotted subvolumes. I use snapper in a limited way, just to snapshot frequently and prune its snapshots.

Folks compare file systems on reliability, and don't rate btrfs well due to (IMO) historic problems. But, at least in my case, I'm the most likely cause of problems, and non-snapshotting filesystems force reliance on backups.

Everyone has there own fault tolerance. I'm no longer working so the potential for serious disaster vs. just an inconvenience is rather low.

The thing I've found interesting is how little the amount of actual storage use went up when I increased the snapshots from one to two weeks. I probably actually have more in the way of snapshot/backups than I need, but the space is there so why not. I run a similar setup on my server as well.

[claydoh]

I had a setup keeping snapshots at boot (3), daily(5), weekly (3) and monthly (4), and one just after the fresh install and setup. No reason, I just decided on that pattern. Probably just to be able to get at more previous versions of files, and just because I could.

I think the original snapshot and the monthlies is where the space usage increased, since the differences between the more current ones and those older ones could be pretty strong.
Once I went to a more sane setup and ditched the original snapshot and monthlies, a lot of space was recovered. I can't recall the specific amount but it was pretty large. The original snapshot was almost a year old, and considering the rolling Plasma with constant changes.......

I haven't yet grokked fully how to send a snapshot elsewhere, but I have migrated my physical backups to btrfs subvolumes on different drives, so I can still have longer term snapshots of my documents if I want.


For me, I find it interesting that since planning for bad things and setting up snapshots and reasonable backup strategies, I have not had to recover or restore or roll back my OS or anything in at least two years, I think.
This doesn't count planned experiments, where I planned to roll back anyway.

[oshunluvr]

I haven't yet grokked fully how to send a snapshot elsewhere, but I have migrated my physical backups to btrfs subvolumes on different drives, so I can still have longer term snapshots of my documents if I want.
.

Well, the early advice was to not use btrfs send|receive over a network or USB connection. The recommended method was to btrfs send to a file, then copy the file somewhere else. To retrieve it, you would need to copy it back and then btrfs receive it.

My current situation involves only making backups to a different drive on the same PC. Since it's desktop pc, I'm unlikely to "lose" it. If I have a house fire that destroys it, I'll have bigger problem than lost data. I probably should find a internet storage service for the really important stuff like old family photos and important document scans. All that totals less than 90GB so I believe Amazon S3 would be free for me.