Multibooting GRUB from a btrfs subvolume.

[jlittle]

Further findings. Firstly, my use of grub-install to set the boot entry won't work with secure boot, because the grubx64.efi executable is signed and can't be changed. I've no need for secure boot, but some reader might.

Secondly, I came up with a way to stop grub-install from trampling on my set-up, when it's run by APT on a new version of the grub package. It is to create another EFI boot entry for a copy of the corrected grub executable:

cd /boot/efi/EFI/ubuntu
sudo cp grubx64.efi mygrubx64.efi
sudo efibootmgr -c -d /dev/sdc -l '\EFI\ubuntu\mygrubx64.efi' -L mine

This will stick if the "debconf" configuration variable grub2/update_nvram is set to false, as revealed by

sudo debconf-show grub-efi-amd64 | grep nvram             
* grub2/update_nvram: false

You're supposed to set this by running sudo dpkg-reconfigure grub-efi-amd64, but be aware that that will rerun grub-install and update-grub. You could live dangerously and edit /var/cache/debconf/config.dat directly.

[GreyGeek]

[emoji41]
Neat!

[jlittle]

I wanted to clean up my old snapshots, after doing several backups, and an update to grub messing up my set up, so I've revisited this idea. I've found that now the simple, straightforward approach now works. Note that I have subvol=/ mounted as /mnt/top, and the ESP mounted at /boot/efi.

sudo btrfs subvolume create /mnt/top/@grub
sudo grub-install --boot-directory=/mnt/top/@grub/boot --efi-directory=/boot/efi
sudoedit /mnt/top/@grub/boot/grub/grub.cfg

and make my grub.cfg as below. Super simple, and can be redone quickly. When APT installs a new version of grub, it tramples the EFI ubuntu entry, so I just need to notice and rerun the grub-install command.

# John's very minimal grub.cfg
#
set default="0"

insmod part_gpt
insmod btrfs
insmod all_video

menuentry 'Kubuntu Bionic' --class ubuntu --class gnu-linux --class gnu --class os {
set root='hd2,gpt2'
search --no-floppy --set=root --label "main"
linux /@_bionic/vmlinuz root=UUID=c908b7ec-dc1d-4594-85e4-f62a9eb4e44e ro rootflags=subvol=@_bionic
initrd /@_bionic/initrd.img
}
menuentry 'System setup' $menuentry_id_option 'uefi-firmware' {
fwsetup
}

If someone wanted to use this grub.cfg, they'd want to change:

• the labels
• the linux line to have the correct UUID and subvolume names (here @_bionic)
• the search lines to use the correct label, or use --fs-uuid with the correct UUID
• maybe the set root lines; if the search commands are successful the set root is not used, they're there as a fallback

Now, the grub.cfg I use isn't quite so minimal, it uses a picture, a bigger font, and some more entries, but the above is all that's needed.

[GreyGeek]

That's an even BETTER idea!

[oshunluvr]

update-grub works fine for whatever install you're running at the time regardless of subvolume name, it just doesn't parse additional subvolumes which is why I created the custom menu process. It works really well, but it requires manual upkeep.

Actually, it's os-prober that is the hang up. I know some bug reports have been filed and patching has been done with some distros - Fedora mainly - in an attempt to fix this, but I've not really explored the status of it because frankly I'm comfortable how things are working here.

I just had an interesting thought though: What if Ubiquity, along with allowing naming of subvolumes at install time, also put grub in a subvolume? Subsequent installs would detect the existence of the grub sobvolume and offer the choice to add to the existing menu rather than replacing it, thus supporting multi-booting out of the box Of course, once you installed a non-Ubiquity distro, you'd have to repair grub but that's the same as it is now. You'd probably need a grub manager of some sort. This probably wouldn't work unless more distros went to btrfs subvolumes as the default when using btrfs,

[GreyGeek]

....
The one serious change I'd like to see in Ubiquity and in fact, to all installers, is to be able to select a subvolume name at install time when choosing btrfs as the installation format. This one change would save 80-90% of the work of doing multiple installs.

I'm with you there!
I suspect that it would also require a re-write of update-grub that detects when Btrfs is installed and to see the subvolumes as bootable partitions that should be included in the menu. So, if you deleted a subvolume to make room for another install, the update-grub would see the new subvolume and include it in the menu, but not the deleted subvolume.

[oshunluvr]

Like all of Oshunluver's threads, this one is also great, and very informative. For my hardware and configuration, unfortunately, the methodology does not work as well.

Thanks, and yes, this thread details a very specific use-case.

It's useful only for booting multiple distros all installed to a single btrfs file system AND using a dedicated grub install to manage how they boot. The really neat thing is, if you are not now using a dedicated grub install but want to, and are currently using btrfs; you don't need to create a partition to use grub in this way.

I started using a dedicated grub partition almost a decade ago after the third time I wiped a distro from a partition that was "hosting" grub without first replacing grub . Back then, I wasn't yet using btrfs in the way I am now and it hadn't occurred to me to transition my dedicated grub partition to a dedicated grub subvolume.

Thanks again to jlittle and Qqmike for the suggestion and for contributing EFI info.

The one serious change I'd like to see in Ubiquity and in fact, to all installers, is to be able to select a subvolume name at install time when choosing btrfs as the installation format. This one change would save 80-90% of the work of doing multiple installs.

[oshunluvr]

I just noticed that in your custom grub you set
set default='KDE Neon'

BUT, there is no menu entry with that exact name, so it will probably default to the first menu entry:

menuentry 'KDE Neon- It works!'

and thus "it works!" by accident?

I thought I mentioned this eariler, but I put "It Works!" in the new btrfs grub partition custom grub menu so I could tell I was actually booting from it rather than one of the two grub partitions I had at the time. Once I saw the "It Works" in the menu I knew I was booting from the btrfs grub installation, I edited it out and deleted the former grub stand-alone partitions.

[oshunluvr]

At some point a grub-install is needed.

Yes, as I showed in post #3

[GreyGeek]

Bionic is supported for three years not five.

I love this thread though, I'm going to be installing Bionic and btrfs on a new drive when I get it shortly. So, this is all great and useful reading

It's usual for hardware maintenance updates end in three years, BUT, software maintenance updates end in five years.
https://www.ubuntu.com/info/release-end-of-life

Like all of Oshunluver's threads, this one is also great, and very informative. For my hardware and configuration, unfortunately, the methodology does not work as well. Specifically, even though changes to config.cfg are preserved through an update-grub command, the actual grub menu that is displayed reverts back to /@/boot/grub/... and has to be edited manually to cause a successful boot. Equally unfortunate, reverting grub.cfg back to /@/ doesn't work and my /@Neon1604 settings are preserved. Without my grub fu I'd be dead in the water right now.

I don't consider the event a total loss, however, because I was intending to update my hardware config in the near future and this experience hastened that move and hardened the direction I am choosing.

Bionic is, by all reports, a stellar release, and I have no clue as to when Neon will be switching to 18.04. I suspect they are waiting till after the first 18.04 bug repair release, so its release may be months away.

Ergo, I will probably being installing Bionic within the next week or so.

[Bings]

My system uses msdos
What I haven’t decided yet is to install Bionic for the next five years or to continue to roll with Neon.

Bionic is supported for three years not five.

I love this thread though, I'm going to be installing Bionic and btrfs on a new drive when I get it shortly. So, this is all great and useful reading

[jlittle]

Alas, on EFI the grub-install has problems:

$ sudo grub-install --root-directory /mnt/grub /dev/sdc
Installing for x86_64-efi platform.
grub-install: error: cannot find EFI directory.

I suspect I'll have to chroot into the @grub subvolume, or use --efi-directory and --target=x86_64-efi.

I found a way with EFI, following the spirit (IMO) of oshunluvr's method, with no new stuff.

My boot drive with btrfs is on /dev/sdc2, my only functional install is 18.04 bionic beaver installed in @_bionic and @home_bionic. I have the top level of /dev/sdc2 mounted in /mnt/top. And I have sudoedit set up.

1. Snapshot the bionic root to @grub: sudo btrfs subvolume snapshot /mnt/top/@_bionic /mnt/top/@grub.
2. Edit the fstab in the @grub subvolume to mount / to subvol=@grub with sudoedit /mnt/top/@grub/etc/fstab. Leave the /home mount as it is.
3. Add a grub entry in bionic to boot to @grub, by pasting the first "menuentry" block in /boot/grub/grub.cfg into /etc/grub.d/40_custom after the existing lines there, edit its name (to avoid confusion) and edit the linux and initrd lines to read
   

   linux	/@grub/vmlinuz root=UUID=c908b7ec-dc1d-4594-85e4-f62a9eb4e44e ro rootflags=subvol=@grub
   initrd	/@grub/initrd.img

   Note well that in two lines, there's three edits.
4. Run sudo update-grub. Now grub in the bionic install can boot to the cloned install.
5. Add the converse entry to /mnt/top/@grub/etc/grub.d/40_custom, with @_bionic replacing @grub in three places. Now grub in the cloned install can boot bionic, but we can't start it yet.
6. Reboot, and choose the entry named in step 3, and log in normally. Run findmnt to check that / is on subvol=@grub.
7. Run sudo update-grub to pick up the entry created in step 5.
8. Run sudo grub-install /dev/sdc. Now the EFI entry points at @grub.
9. Reboot, and choose the step 5 entry.

Now I can rejoin oshunluvr's instructions. I can hack away at /mnt/top/@grub/boot/grub/grub.cfg, like I did with my BIOS firmware computers knowing that an update-grub in any install won't change it. That grub.cfg can be much simpler, look the way I want it to, and with a configfile entry pick up the full grub menu in any install, just in case there's something needed there, like an older kernel.

Remaining to do:

• Clean up /mnt/top/@grub/boot/grub/grub.cfg. Remove the entries that boot into linux in @grub, as they'll shortly point nowhere.
• Prune /mnt/top/@grub back to just the boot directory.
• Remove the entry in step 3 above.

[GreyGeek]

My system uses msdos


I have no trouble using grub rescue or editing a menu option on the grub menu to set the proper path to “Linux” and “initrd” to allow bootup and login. My problem is that even though I change grub.cfg to point to @Neon1604 AND those changes survive update-grub, when I reboot grub menu always has “/@/boot...” instead of “/@Neon1604/boot...”.

I’ve used grub-install to switch between /dev/sda1 and /dev/sda, and have even made both bootable. Doesn’t matter, except that it shifted the failure point to “initrdms”.

I thought that perhaps my Btrfs was corrupted because of all the booting changes but a “btrfs check —repair /dev/sda1” showed that it was clean, no errors.

A copy of grub that is being used to display the grub menu is not being updated properly by update-grub (which I’ve also done by booting a LiveUSB of Bionic, mounting /dev/sda1 and chrooting into it).

No matter. I won’t lose anything because of archived snapshots and this will give me the incentive to make some hardware changes. I bought a new Toshiba 750GB HD a year ago and while exploring possible causes for this failure mode I discovered that the Toshiba has 16 Sectors that failed after only 580+ hours of on time. My two WD drives, with 5K hours each, still have 0 failures. I keep my archive drive spun down so I am going to make the Toshiba the archive drive.

What I haven’t decided yet is to install Bionic for the next five years or to continue to roll with Neon.

[Qqmike]

In my EFI system the location of /boot/grub is baked into /boot/efi/EFI/ubuntu/grubx64.efi as I reported above. I don't know the correct incantation to get grub-install to set another install's location into grubx64.efi.

jlittle, I'm not following this thread, nor do I know anything about btrfs. With that said then, you are correct that the location of the grub.cfg that the UEFI uses is hard coded into the efi GRUB executable. I don't know that you can easily hard code your own into the grubx64.efi. But, it may be possible to get what you want another way, using the --no-uefi-secure-boot option in the grub-install command. man grub-install mentions this option. I explain what I am able to explain in my how-to's, and I have tested it. Secure boot, of course, must be turned off. Fwiw, my scribbles are here:

https://www.kubuntuforums.net/showth...l=1#post412601

For example, scrolling down to

UEFI for Kubuntu--simplified. And ... some dual-booting tips for Kubuntu
https://www.kubuntuforums.net/showthread.php/43221-GRUB-2-A-Guide-for-Users/page13?p=373198&viewfull=1#post373198

and checking in SECTION 6 of that how-to:

Section 6: All about dual-booting, 5 ways to do it, how to fix it; creating separate EFI subdirectories for your (K)Ubuntu OSs (using --no-uefi-secure-boot option).; creating and using more than one ESP, changing labels on UEFI boot entries (useful for multiple "ubuntu" entries).

One more thing I noticed in this thread: When using EFI, grub-install defaults to the ESP that is being used by the OS you are booted into when you issued grub-install, and it goes to the default: /boot/efi. You can make it go to another mounted location using the option grub-install --efi-directory=DIR, where DIR is the mounted location of the ESP the OS was set up to use (so the default value of DIR is /boot/efi). With EFI, using the "root" directory option with grub-install, I don't think makes any sense (because the GRUB executables are placed in the ESP and are referenced that way by the EFI, the executables are not referenced in the root filesystem of the OS by the EFI firmware).

[jlittle]

At some point a grub-install is needed.

On BIOS systems the grub "core image" in the "MBR gap" has the location of the /boot/grub to use. That core image may be compressed, always if the gap is the usual 62 sectors, so a hex dump won't reveal it. "bootinfoscript" hasn't been updated for grub 2.02 so it doesn't give useful info.

In my EFI system the location of /boot/grub is baked into /boot/efi/EFI/ubuntu/grubx64.efi as I reported above. I don't know the correct incantation to get grub-install to set another install's location into grubx64.efi.

Now if your grub shell fu is strong, and you manage to boot into the install in @grub, then running grub-install from there will do the job I think. I think that's a cleverness in oshunluvr's method, exploiting the ease, speed, and minimal resource of btrfs snapshots to have a whole cloned OS to work with, then pruning it back to what's needed.