Multibooting GRUB from a btrfs subvolume.

[GreyGeek]

I attempted the first part of the multi-boot process:
Sudo -i to root.
Mount -t btrfs /dev/sda1 /mnt
Cloning @ and @home to @Neon1604 and @Neon1604home, and @ to @grub.
I edited @Neon1604/etc/fstab to change /@ to /@Neon1604 and @home to @Neon1604home.

I mv'd @ to @old and @home to @homeold.

I edited /mnt/@Neon1604/boot/grub/grub.cfg to change all lines which contain "...rootflags=subvol=@' to "...rootflags=subvol=@Neon1604".

At this point the instructions suddenly became vague. Upgrade-grub now? The instructions didn't say to update grub at this point. They only say that "Rather miraculously this setting will be preserved when I DO update-grub".

I updated grub. Checking the grub.cfg settings I found that all rootflags changes had reverted back to /@.

I booted into my Neon LiveUSB stick and mounted /dev/sda1 and edited grub.cfg. I changed them back to /@Neon1604.

I rebooted and was thrown into <grub rescue> with the error that "/@/....bold-16.pf2" could not be found. With my LiveUSB stick I fixed that and all other occurances of /@/, of which there are many.
I rebooted.
I was greeted with grub rescue again.

In grub rescue> I did:
set prefix=(hd0,1)/@Neon1604/boot/grub
set root=(hd0,1)
insmod normal
normal


At that moment I booted into my Neon installation.
I didn't have to do:
insmod linux
linux /@Neon1604/boot/vmlinuz-4.8.0-42-generic root=/dev/sda1
initrd /@neon1604/boot/initrd.img-4.8.0-42-generic
boot

Once my desktop appeared I opened a root terminal and issued "update-grub". It completed. Checking the /boot/grub/grub.cfg file I found that all changes of /@ to /@Neon1604 remained. Confidently, I rebooted.

I was again greeted with the grub rescue prompt. THIS TIME the error is
"error: file '/@/boot/grub/i386-pc/normal.mod' not found"

Searching through grub.cfg revealed no call to the i386-pc's folder. At this point I don't know where that call is coming from so I can't change it. A DDG search of the Internet revealed few links and none offered any help.

I redid the previous "set prefix", "set root", "insmod normal" followed by "normal" and was booted up to my desktop.

I know for sure that I am running in @Neon1604 and @Neon1604home because @ and @home no longer exist. They were replaced with @old and @homeold.

That's where I stand right now.

[GreyGeek]

I just noticed that in your custom grub you set
set default='KDE Neon'

BUT, there is no menu entry with that exact name, so it will probably default to the first menu entry:

menuentry 'KDE Neon- It works!'

and thus "it works!" by accident?

[oshunluvr]

Correct again, and certainly;

## Custom grub config
#
set default='KDE Neon'


if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi


export menuentry_id_option


if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi


function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function recordfail {
  set recordfail=0


}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}


insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
font="/@grub/boot/grub/fonts/unicode.pf2"


if loadfont $font ; then
  set gfxmode=3840x1600
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=3
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=3
  fi
fi


### BEGIN /etc/grub.d/05_debian_theme ###


set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
#set_background_image /@KDEneon/usr/share/wallpapers/neonbykde2.jpg";
insmod jpeg
if background_image background1.jpg; then
  true
else
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
fi


### END /etc/grub.d/05_debian_theme ###


### BEGIN Custom menu ###
menuentry 'KDE Neon' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@KDEneon/boot/grub/grub.cfg
}
menuentry 'KDE Neon - new' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@KDEneon_new/boot/grub/grub.cfg
}
menuentry 'Kubuntu 18.04' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Kubuntu_1804/boot/grub/grub.cfg
}
menuentry 'Kubuntu 16.04' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Kubuntu_16_04/boot/grub/grub.cfg
}
menuentry 'Ubuntu 16.04' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Ubuntu_16_04/boot/grub/grub.cfg
}
menuentry 'Manjaro free' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Manjaro_free/boot/grub/grub.cfg
}
menuentry 'Manjaro nonfree' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Manjaro_nonfree/boot/grub/grub.cfg
}
menuentry 'Memory test (memtest86+)' {
    insmod part_gpt
    insmod ext2
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    knetbsd /@grub/boot/memtest86+.elf
}
menuentry 'Memory test (memtest86+, serial console 115200)' {
    insmod part_gpt
    insmod ext2
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    linux16 /@grub/boot/memtest86+.bin console=ttyS0,115200n8
}
menuentry 'KDEneon ISO' {
    set isofile="/neon-useredition-20170608-2351-amd64.iso"
    loopback loop (hd0,2)$isofile
    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile noprompt noeject
    initrd (loop)/casper/initrd.lz
}
### END Custom menu###

[GreyGeek]

....
As to booting my new Kubuntu 18.04 install; When I installed Kubuntu 18.04 without a bootloader, update-grub did not run. Therefore there was no grub.cfg in the Kubuntu 18.04 to select from my custom grub.cfg and boot from. I had to manually boot it by entering the proper commands to boot it into grub. This was much less difficult than it sounds. Since all my distros reside on the same file system, they have the same UUID. All I had to do was select KDEneon in my first grub menu (the custom one), when the second grub menu (from the grub.cfg in KDEneon) appeared, I pressed the "E" key to enter the grub editor. Then I edited these two lines (the last in the menuentry):

     linux    /[B]@KDEneon[/B]/boot/vmlinuz-[B]4.15.0-15[/B]-generic root=UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 ro rootflags=subvol=[B]@KDEneon [/B] quiet splash $vt_handoff
     initrd    /[B]@KDEneon[/B]/boot/initrd.img-[B]4.15.0-15[/B]-generic

to look like this:

     linux    /[B]@Kubuntu_18.04[/B]/boot/vmlinuz-[B]4.15.0-20[/B]-generic root=UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 ro rootflags=subvol=[B]@Kubuntu_1804[/B]
     initrd    /[B]@Kubuntu_18.04[/B]/initrd.img-[B]4.15.0-20[/B]-generic

I only had to change "@KDEneon" to "@Kubuntu_1804" in three places and correct the kernel version number in two. I had forgotten to note the kernel version that was installed with Kubuntu 18.04 before I rebooted, but no big deal - while in the grub menu, pressing F2 brings up the grub console, where I could "ls" my way into "@Kubuntu_1804/boot" and see the kernel version there.

Once I had successfully booted into @Kubuntu 18.04, I ran update-grub and created a grub.cfg. Then I mounted @grub at /mnt/grub, edited /mnt/grub/boot/grub/grub.cfg (my custom grub.cfg) and added a menuentry for Kubuntu 18.04. Now it boots just like the other six distros.

So even though you edited the @KDEneon grub.cfg and replaced that term with @Kubuntu_1804, etc., the Neon grub wasn't actually changed and still pointed to Neon on subsequent boots. Doing update-grub fixed its grub.cfg.

Now, would you let me play Biden and post your entire custom grub.cfg?

[oshunluvr]

Alas, on EFI the grub-install has problems:

$ sudo grub-install --root-directory /mnt/grub /dev/sdc
Installing for x86_64-efi platform.
grub-install: error: cannot find EFI directory.

I suspect I'll have to chroot into the @grub subvolume, or use --efi-directory and --target=x86_64-efi. I want to get better backed up before I try that.

I have noticed that in the ESP, seen as /boot/efi/EFI/ubuntu there's a stub grub.cfg that the bionic installer put there, with only three lines that just loads /@/boot/grub.cfg; but it's not used, bionic is now in @_bionic. Instead, (,gpt2)/@_bionic/boot/grub is baked into the grubx64.efi executable. (I found it with strings grubx64.efi | grep @.)

Sorry, I can't help you with EFI. QQmike has a lot of experience and knowledge with EFI, maybe he can help. Hopefully you can get it figured out because moving grub to a subvolume is a brilliant idea! I used to maintain two grub partitions on separate drives to have a backup grub to boot too. Now I can make a backup of @grub just like I do with any other subvolume! Thanks for the idea

[oshunluvr]

So everything in @grub except the files in the root of /boot, and /boot and /sys (?) were deleted?

Everything except /boot/grub and it's contents and I left the three memtest kernels in /boot. EVERYTHING else is gone - no Linux kernels, no other folders.

[FONT=monospace][COLOR=#54FF54][B]stuart@office[/B][/COLOR][COLOR=#000000]:[/COLOR][COLOR=#5454FF][B]~[/B][/COLOR][COLOR=#000000]$ ll /subvol/@grub/boot/[/COLOR]
total 548
drwxr-xr-x 1 root root    112 Apr 27 14:43 [COLOR=#5454FF][B].[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root      8 Apr 27 14:42 [COLOR=#5454FF][B]..[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root    120 Apr 27 14:34 [COLOR=#5454FF][B]grub[/B][/COLOR][COLOR=#000000]/[/COLOR]
-rw-r--r-- 1 root root 182704 Jan 28  2016 memtest86+.bin
-rw-r--r-- 1 root root 184380 Jan 28  2016 memtest86+.elf
-rw-r--r-- 1 root root 184840 Jan 28  2016 memtest86+_multiboot.bin
[COLOR=#54FF54][B]stuart@office[/B][/COLOR][COLOR=#000000]:[/COLOR][COLOR=#5454FF][B]~[/B][/COLOR][COLOR=#000000]$ ll /subvol/@grub/boot/grub/[/COLOR]
total 2356
drwxr-xr-x 1 root root     120 Apr 27 14:34 [COLOR=#5454FF][B].[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root     112 Apr 27 14:43 [COLOR=#5454FF][B]..[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root      22 Apr 27 13:41 [COLOR=#5454FF][B]fonts[/B][/COLOR][COLOR=#000000]/[/COLOR]
-rw-r--r-- 1 root root     712 Apr 26 14:26 gfxblacklist.txt
-rw-r--r-- 1 root root    3844 Apr 27 14:34 grub.cfg
-rw-r--r-- 1 root root    1024 Apr 27 13:17 grubenv
drwxr-xr-x 1 root root    6228 Apr 27 13:47 [COLOR=#5454FF][B]i386-pc[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root      68 Apr 27 13:47 [COLOR=#5454FF][B]locale[/B][/COLOR][COLOR=#000000]/[/COLOR]
-rw-r--r-- 1 root root 2397557 Apr 26 14:25 unicode.pf2

[/FONT]

All those changes are made in the new subvolume. Got that.

Correct

Which subvolume is the "existing" grub.cfg in? The one you are currently running under I assume?

The @grub subvolume contains the custom grub.cfg with my menu entries that point at the other grub.cfgs. Each distro subvolume has it's own grub cfg.

To explain: When I boot, I initally see my custom grub menu that looks like this:

KDE Neon
KDE Neon - new
Kubuntu 18.04
Kubuntu 16.04
Ubuntu 16.04
Manjaro free
Manjaro nonfree
Memory test (memtest86+)
Memory test (memtest86+, serial console 115200)
KDEneon ISO'

Each of these entries except the last three point to another grub.cfg. So now since I've added Kubuntu 18.04, it's menuentry within this custom grub.cfg looks like:

menuentry 'Kubuntu 18.04' {
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root 8f0c1661-4e84-4512-b875-23bcfd5be1d8
    configfile /@Kubuntu_1804/boot/grub/grub.cfg
}

When I initially select Kubuntu 18.04, it doesn't bootup right away, instead a second grub menu appears - the one that's in Kubuntu 18.04 (note the last line in the above menuentry). If I hit "ESC" grub returns to the first (the custom) grub menu that's in @grub. Effectively, they are nested menus. Since I go through two grub menus each time I boot, I set the timeout to 3 seconds in the initial grub.cfg and 5 seconds on the other grub.cfgs. This gives me enough time to change my mind without delaying the boot process too much if don't do anything during bootup. I keep my current (daily driver) distro as the default in the first grub.cfg so if I power up and walk away, my regular distro is booted (KDEneon at this time).

Again: the advantage with this is the custom menu is (mostly) set-it-and-forget-it. I only have to edit it when I add or remove a distro. Each distro, since they all have their own grubs and grub.cfgs, maintain their own grub.cfgs just like they would in a stand-alone installation. Kernel changes are not an issue. Also, since grub is stand-alone, no one distro can muck it up.

A few weeks back when my KDEneon was having trouble booting and I need to roll-back to a snapshot, I just booted to Manjaro, did the roll-back, and rebooted. Done in less than a minute.

I wasn't aware that the grub-install command allowed --root-directory to be more than one device. That command is saving grub.cfg in both /mnt/grub and /dev/sda

Or, is it "--root-directory=/mnt/grub" and /dev/sda is the destination?

grub-install doesn't do anything with grub.cfg except tell the grub bootloader where it is. update-grub creates the grub.cfg in /boot/grub. grub-install points the bootloader at whatever grub.cfg you tell it to. Since I switched the location of my grub files from /dev/sda4 (my previous dedicated grub partition) to the new @grub subvolume in my btrfs file system, I had to tell the bootloader of the new location of the grub files. That's what grub-install does. In fact, I currently have four drives in my computer and for safety purposes I make them all bootable. So I ran grub-install four times, once for each drive. The "--root-directory" tells the bootloader where to go when booting to find it's files and grub.cfg. The bootloader is only on the first 446 bytes of a bootable drive.

As to booting my new Kubuntu 18.04 install; When I installed Kubuntu 18.04 without a bootloader, update-grub did not run. Therefore there was no grub.cfg in the Kubuntu 18.04 to select from my custom grub.cfg and boot from. I had to manually boot it by entering the proper commands to boot it into grub. This was much less difficult than it sounds. Since all my distros reside on the same file system, they have the same UUID. All I had to do was select KDEneon in my first grub menu (the custom one), when the second grub menu (from the grub.cfg in KDEneon) appeared, I pressed the "E" key to enter the grub editor. Then I edited these two lines (the last in the menuentry):

     linux    /[B]@KDEneon[/B]/boot/vmlinuz-[B]4.15.0-15[/B]-generic root=UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 ro rootflags=subvol=[B]@KDEneon [/B] quiet splash $vt_handoff
     initrd    /[B]@KDEneon[/B]/boot/initrd.img-[B]4.15.0-15[/B]-generic

to look like this:

     linux    /[B]@Kubuntu_18.04[/B]/boot/vmlinuz-[B]4.15.0-20[/B]-generic root=UUID=8f0c1661-4e84-4512-b875-23bcfd5be1d8 ro rootflags=subvol=[B]@Kubuntu_1804[/B]
     initrd    /[B]@Kubuntu_18.04[/B]/initrd.img-[B]4.15.0-20[/B]-generic

I only had to change "@KDEneon" to "@Kubuntu_1804" in three places and correct the kernel version number in two. I had forgotten to note the kernel version that was installed with Kubuntu 18.04 before I rebooted, but no big deal - while in the grub menu, pressing F2 brings up the grub console, where I could "ls" my way into "@Kubuntu_1804/boot" and see the kernel version there.

Once I had successfully booted into @Kubuntu 18.04, I ran update-grub and created a grub.cfg. Then I mounted @grub at /mnt/grub, edited /mnt/grub/boot/grub/grub.cfg (my custom grub.cfg) and added a menuentry for Kubuntu 18.04. Now it boots just like the other six distros.

[jlittle]

$ sudo grub-install --root-directory /mnt/grub /dev/sda

Alas, on EFI the grub-install has problems:

$ sudo grub-install --root-directory /mnt/grub /dev/sdc
Installing for x86_64-efi platform.
grub-install: error: cannot find EFI directory.

I suspect I'll have to chroot into the @grub subvolume, or use --efi-directory and --target=x86_64-efi. I want to get better backed up before I try that.

I have noticed that in the ESP, seen as /boot/efi/EFI/ubuntu there's a stub grub.cfg that the bionic installer put there, with only three lines that just loads /@/boot/grub.cfg; but it's not used, bionic is now in @_bionic. Instead, (,gpt2)/@_bionic/boot/grub is baked into the grubx64.efi executable. (I found it with strings grubx64.efi | grep @.)

[GreyGeek]

At this very moment:

[FONT=monospace][COLOR=#54FF54][B]stuart@office[/B][/COLOR][COLOR=#000000]:[/COLOR][COLOR=#5454FF][B]~[/B][/COLOR][COLOR=#000000]$ ll /subvol[/COLOR]
total 4
drwxr-xr-x 1 root root 720 Apr 27 14:41 [COLOR=#5454FF][B].[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]..[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root   8 Apr 27 14:42 [COLOR=#5454FF][B]@grub[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]@KDEneon[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]@KDEneon_backup[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 252 Apr 18 06:33 [COLOR=#5454FF][B]@KDEneon_daily_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 252 Apr 16 19:54 [COLOR=#5454FF][B]@KDEneon_daily_3[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 428 Apr 15 18:55 [COLOR=#5454FF][B]@KDEneon_daily_5[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 428 Apr 14 18:55 [COLOR=#5454FF][B]@KDEneon_daily_6[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 396 Apr 13 18:55 [COLOR=#5454FF][B]@KDEneon_daily_7[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Jan 22 16:02 [COLOR=#5454FF][B]@KDEneon_home[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Jan 22 16:02 [COLOR=#5454FF][B]@KDEneon_home_backup[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 24 09:03 [COLOR=#5454FF][B]@KDEneon_home_daily_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 24 07:34 [COLOR=#5454FF][B]@KDEneon_home_daily_2[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 18 06:33 [COLOR=#5454FF][B]@KDEneon_home_daily_3[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 17 07:01 [COLOR=#5454FF][B]@KDEneon_home_daily_4[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 16 19:54 [COLOR=#5454FF][B]@KDEneon_home_daily_5[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 196 Jun 13  2017 [COLOR=#5454FF][B]@KDEneon_new[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 262 Dec 13  2016 [COLOR=#5454FF][B]@Kubuntu_16_04[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 224 Apr 27 13:00 [COLOR=#5454FF][B]@Kubuntu_1804[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  32 Apr 26 14:24 [COLOR=#5454FF][B]@Kubuntu_1804_home[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 282 Jun 27  2017 [COLOR=#5454FF][B]@Manjaro_free[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 228 Feb 26 11:39 [COLOR=#5454FF][B]@Manjaro_nonfree[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 234 Apr 17 06:26 [COLOR=#5454FF][B]@Ubuntu_16_04[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 234 Apr 17 07:26 [COLOR=#5454FF][B]@Ubuntu_16_04_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
[/FONT]

Wow!

[GreyGeek]

Final step: deleting the unneeded @ and @home subvolumes and everything not needed in @grub.
.....
I also deleted everything in @grub except the contents of /boot/grub and the memtest kernels. I keep memtest on my dedicated grub so I don't need to install it anywhere else. With seven distros installed, if each of them install memtest too - that's a lot of memtest!

I now have a dedicated bootable grub in it's own subvolume.

So everything in @grub except the files in the root of /boot, and /boot and /sys (?) were deleted?

.....

Next step is to get the @Kubuntu subvolume bootable. To do this you need to edit the fstab file and the grub.cfg (there's nother way to fix the grub.cfg, but this way is easier IMO). I used kate and replace every instance of "@" with "@Kubuntu_1804" (I also snapshoted @home to @Kubuntu_1804_home so I had two home subvolumes at this point. If you do this, you'll need to include this edit in fstab.) To be perfectly - I did these edit in the @Kubuntu_1804 subvolume only.

All those changes are made in the new subvolume. Got that.

Now I need to boot into @Kubuntu_1804 to make sure it works. I edited my existing grub.cfg to allow me to boot the new install in the new @Kubuntu_1804 subvolume.

Which subvolume is the "existing" grub.cfg in? The one you are currently running under I assume?

Now I wanted to create a boot from a dedicated grub subvolume which I had already created as @grub. This was pretty easy. All I had to do was to mount /boot from @grub somewhere and run grub-install. My btrfs root filesystem resides on sdd3.

The commands:

sudo mkdir /mnt/grub
sudo mount -o subvol=@grub /dev/sdd3 /mnt/grub
sudo grub-install --root-directory /mnt/grub /dev/sda

and that's pretty much it. I did not need to edit fstab or any kernel statements in grub.cfg because this method of booting will not actually boot to @grub - it just uses it to direct the boot to a selected distro.

I wasn't aware that the grub-install command allowed --root-directory to be more than one device. That command is saving grub.cfg in both /mnt/grub and /dev/sda

Or, is it "--root-directory=/mnt/grub" and /dev/sda is the destination?

[oshunluvr]

At this very moment:

[FONT=monospace][COLOR=#54FF54][B]stuart@office[/B][/COLOR][COLOR=#000000]:[/COLOR][COLOR=#5454FF][B]~[/B][/COLOR][COLOR=#000000]$ ll /subvol[/COLOR]
total 4
drwxr-xr-x 1 root root 720 Apr 27 14:41 [COLOR=#5454FF][B].[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]..[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root   8 Apr 27 14:42 [COLOR=#5454FF][B]@grub[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]@KDEneon[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 286 Apr 25 07:37 [COLOR=#5454FF][B]@KDEneon_backup[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 252 Apr 18 06:33 [COLOR=#5454FF][B]@KDEneon_daily_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 252 Apr 16 19:54 [COLOR=#5454FF][B]@KDEneon_daily_3[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 428 Apr 15 18:55 [COLOR=#5454FF][B]@KDEneon_daily_5[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 428 Apr 14 18:55 [COLOR=#5454FF][B]@KDEneon_daily_6[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 396 Apr 13 18:55 [COLOR=#5454FF][B]@KDEneon_daily_7[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Jan 22 16:02 [COLOR=#5454FF][B]@KDEneon_home[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Jan 22 16:02 [COLOR=#5454FF][B]@KDEneon_home_backup[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 24 09:03 [COLOR=#5454FF][B]@KDEneon_home_daily_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 24 07:34 [COLOR=#5454FF][B]@KDEneon_home_daily_2[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 18 06:33 [COLOR=#5454FF][B]@KDEneon_home_daily_3[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 17 07:01 [COLOR=#5454FF][B]@KDEneon_home_daily_4[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  48 Apr 16 19:54 [COLOR=#5454FF][B]@KDEneon_home_daily_5[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 196 Jun 13  2017 [COLOR=#5454FF][B]@KDEneon_new[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 262 Dec 13  2016 [COLOR=#5454FF][B]@Kubuntu_16_04[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 224 Apr 27 13:00 [COLOR=#5454FF][B]@Kubuntu_1804[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root  32 Apr 26 14:24 [COLOR=#5454FF][B]@Kubuntu_1804_home[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 282 Jun 27  2017 [COLOR=#5454FF][B]@Manjaro_free[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 228 Feb 26 11:39 [COLOR=#5454FF][B]@Manjaro_nonfree[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 234 Apr 17 06:26 [COLOR=#5454FF][B]@Ubuntu_16_04[/B][/COLOR][COLOR=#000000]/[/COLOR]
drwxr-xr-x 1 root root 234 Apr 17 07:26 [COLOR=#5454FF][B]@Ubuntu_16_04_1[/B][/COLOR][COLOR=#000000]/[/COLOR]
[/FONT]

[GreyGeek]

So, what does a directory listing of your FS_ROOT look like?

[oshunluvr]

Partitions? Who needs stinkin' partitions? Now I even have my stand-alone grub in a subvolume thanks to jlittle's idea.

The use of subvolumes is simply brilliant; no more partitioning, easy backups and data transfers, roll-back capability, etc. I only need to partition for grub/efi and that's it.

Right now I'm re-arranging the data on my server. Once that's done, I'll be re-writing my backup automation since I'm moving away from RAID1 for the time being. This will leave my server with a 10TB drive for data - zero partitions, 2x6TB drives for backups - zero partitions, and a 60GB SSD that I boot from. The SSD holds grup. swap. and the installs. My only concern if the SSD fails, I would have to boot to a USB stick. I may just get one of those little thumb-nail sized USB drives and set it up as a backup boot device.

[GreyGeek]

Thanks for the confirmation!

Based on what I think I know about Btrfs installations I wanted to be sure I understood what you are doing.

I’ve never attempted to set up the multi-boot system you are running, but plan to definitely switch to it when I do a fresh install of KDE Neon UE based on 18.04.

That will include switching from device names (/dev/sd X) to UUIDs. I’m going to use my two newest WD 750GB HD’s as a two drive pool and the oldest HD as archive storage, which I’ll keep spun down till I use it to receive subvolumes.

When I used EXT4 I always created only the “/“ partition so that I wouldn’t have to guesstimate how much space to give to /home or /root or /use or what ever. Each file or directory took what ever space it needed from the available unused space. Your multi-boot Btrfs technique does for booting what using just / does for partitioning.

[oshunluvr]

For people doing this the first time, before they do the above steps they should FIRST, IF I understand what you are doing, rename their existing @ and @home to @SOMENAME and @homeSOMENAME, and edit the /etc/fstab to point to @SOMENAME and @homeSOMENAME, and then redo the grup-update command to refresh grub.cfg.

That way they'll end up with
FS_ROOT:
@
@home
@SOMENAME
@homeSOMENAME


Otherwise, won't they overwrite their existing @ with the new @ and their existing @home with the new @home when they chose "/ and uncheck the "Format" box? Yes? No?

Yes, absolutely. This post was supposed to be more about creating and using a grub subvolume as a stand-alone grub install but I complicated it a bit with adding a new distro install at the same time.

[GreyGeek]

OK, so I installed Kubuntu to my btrfs file system that already contains 6 other distros. You do this by selecting "Manual" drive configuration during installation, choosing an existing btrfs file system as "/", DO NOT check the "Format..." box, and install.

This leaves the new Kubuntu install in a subvolume called "@" and your new home in "@home". Since I will want to install other distros in the future, I have to change this. I also want to create a bootable grub subvolume from this installation.
....

For people doing this the first time, before they do the above steps they should FIRST, IF I understand what you are doing, rename their existing @ and @home to @SOMENAME and @homeSOMENAME, and edit the /etc/fstab to point to @SOMENAME and @homeSOMENAME, and then redo the grup-update command to refresh grub.cfg.

That way they'll end up with
FS_ROOT:
@
@home
@SOMENAME
@homeSOMENAME


Otherwise, won't they overwrite their existing @ with the new @ and their existing @home with the new @home when they chose "/ and uncheck the "Format" box? Yes? No?