Migrating From EXT4 to BTRFS?

[PhysicistSarah]

I'm interested in migrating my system from EXT4 to BTRFS for it's better features such as snapshots. I currently have these disk partitions:

I'm dual-booting with Windows 10. I'm planning on doing a full-reinstall and have all the programs/settings/files backed up with aptik.

I'm also interested in learning how to maintain a BTRFS system, as it seems that there is occasional work necessary to ensure proper screenshot setup/restoration capability, whereas I never really interacted directly with the EXT4 system.

[chimak111]

Instead of an image, you could post the output of
sudo fdisk -l
and
sudo parted -l.

[GreyGeek]

FYI: Here is a list of Btrfs gotchas:
https://btrfs.wiki.kernel.org/index.php/Gotchas

Conversion from ext4 may not be undoable

• In kernels 4.0+: the empty block groups are reclaimed automatically that can affect the following:
  • a converted filesystem may not be able to do a rollback because of the removed block groups

That page is from the Btrfs Wiki man pages:
https://btrfs.wiki.kernel.org/index.php/Main_Page

[GreyGeek]

It looks like you have 833Gb devoted to EXT4 on /dev/sda2
You cannot convert a mounted EXT4 system to Btrfs so you will have to create a LiveUSB of Kubuntu 18.04 and boot from it.
Once you have the Kubuntu desktop on the LiveUSB you can open a Konsole and use it. You are automatically root in a LiveUSB.

If the LiveUSB system has automatically mounted /dev/sda2 then locate the mount point and do:
sudo umount whateverthemountpointis
Then do:
fsck.ext4 -f /dev/sda2
btrfs-convert /dev/sda2

Mount the btrfs system:
mount -t btrfs /dev/sda2 /btrfs (you may have to make /btrfs: mkdir /btrfs)
/btrfs should contain @ and @home subvolumes.
Browse in /btrfs to /btrfs/@/etc/fstab and edit that file. Replace ext4 with btrfs. Save it.

Your EXT4 system will be stored in a subvolume labeled ext2_saved (or something similar). Mount it.
mount -t btrfs -o subvol=ext2_saved /dev/xxx /ext2_saved (ditto ext2_saved)

Loopback mount the image file:
mount -t ext4 -o loop,ro /ext2_saved/image /ext4 (Ditto ext4)

Compare the contents of the loopback image with the /btrfs system to make sure things got carried over.

Rolling back the conversion:
Rolling back requires you unmount the Btrfs filesystem


umount /ext4
umount /ext2_saved
umount /btrfs



Convert back into ext file system by typing
btrfs-convert -r /dev/sda2


Now you are ready to remount the filesystem ext4, if you want.


mount -t ext4 /dev/sda2 /ext4

Continuing on with the conversion:
While still running the LiveUSB you can do a defrag and balance of the UNMOUNTED /btrfs
umount /btrfs
btrfs fi defrag -r /dev/sda2
btrfs balance start /dev/sda2

If things went well and you liked what you saw you can delete the ext2_saved subvolume:
btrfs subvol delete ext2_saved


Reboot. Select your Linux menu entry. You should be up and running with Btrfs as your root file system.

OR ...
You can copy your important data to an external HD and then boot the LiveUSB.
From the desktop choose the install option from the desktop or the menu.
When you get to the part about selecting how to use the HD choose the manual partition method and select sda2 as the partition where you want to install. In the dropdown combo the EXT3 file system is default. Click on it and a dropdown will show the Btrfs, among others. Select Btrfs. For a partition name use "/", without the quotes. Continue on. When the installation is shut down, remove the LiveUSB and power up. Choose the Linux system from the boot menu and when the desktop appears go about importing all your saved data.

[oshunluvr]

IMO, The problem with conversion is your install will still be in the root filesystem rather than in subvolumes which removes most of the benefits of btrfs - which means you'll be moving the files eventually. For several reasons, I suggest you shrink the partition, make a new one and migrate the data to the new partition. The reasons are:

1. Less chance of damaging your current install.
2. Keeping your current install means you have a bootable backup install in case the new one breaks.
3. If you mess something up along the way or even just decide btrfs isn't for you, you need only reboot and you're back to your current install.
4. Later, if you decide to not use the EXT4 partition any longer your can allow your btrfs file system to absorb the space.
5. If you want to be able to easily migrate data to Windows from Linux, you can install EXT4 support in Windows and use the EXT4 partition in both Windows and Linux.

Here's what I would do: Create a bootable Gparted Live USB disk. Assuming about 50GB used on the EXT4 partition, boot to Gparted and shrink the EXT4 partition to 60GB or larger if you plan on using it with Windows also (maybe 100GB?). From there, either do a new install or copy your current install into a subvolume on the new btrfs filesystem. Honestly, unless you've done a lot of modifications, it would be quicker to do a new install. However, if you want to copy your current install instead, while still using Gparted Live;

1. Create a new BTRFS partition in the free space and mount it.
2. Create two new subvolumes in the btrfs subvolume - one for root, one for home.
3. Mount the EXT4 install and copy everything except the home folder into the root subvolume.
4. Copy everything in the home folder into the home subvolume.
5. Edit fstab in the copied root subvolume to use the new BTRFS file system.

The final bit will be getting the new install into grub so you can boot to it. Then you'll have two bootable Kubuntu installs - the EXT4 install as a backup and the BTRFS one to use and learn about btrfs.

[PhysicistSarah]

@oshunluvr @GreyGeek So how do BTRFS snapshots work? How much space do they typically take up? Is there any place that has a decent introduction to using BTRFS operations in *buntu systems to get me started?

[oshunluvr]

Snapshots initially take no data space. As the source subvolume changes, the snapshot grows because it retains the original source contents.

BTRFS is not a distro specific, it's implemented in the kernel thus any distro's info will apply to *buntu systems. There may be some add-on scripts or the like that may be distro specific, but there are a whole lot of those.

Arch generally and in this case specifically maintains the best BTRFS Wiki.

[TWPonKubuntu]

I'd like to expand on PhysicistSarah's question.

Assume that I take a BTRFS snapshot before installing a new version of something, for instance an upgrade to the next OS version. I'm not talking about a Full Version jump, but just a minor version increment. Can you give me an idea of how much space that snapshot will occupy?

This is a concern because I don't have terabyte drives available for my systems.

[GreyGeek]

I'd like to expand on PhysicistSarah's question.

Assume that I take a BTRFS snapshot before installing a new version of something, for instance an upgrade to the next OS version. I'm not talking about a Full Version jump, but just a minor version increment. Can you give me an idea of how much space that snapshot will occupy?

This is a concern because I don't have terabyte drives available for my systems.

The default Btrfs subvolumes are @ and @home, which are bound respectively in /etc/fstab to / and /home.
Because many updates and application installations affect both root and home I always take a pair of snapshots: @YYYYMMDD and @homeYYYYMMDD. If I roll back I roll back both for a given date.

However, if I create a subvolume called @data, for instance, and bind it in fstab to /home/username/data, then I can take a snapshot of just @data without dragging along the rest of /home/username.

How to do an incremental backup is explained in this HowTo. After an initial snapshot one uses the "-p" switch to allow btrfs to compare the intital snapshot with the current one when sending & receiving the increment to the storage volume.

Note this: If I made a backup snapshot pair three days ago and because of some misfortune I have to do a rollback to that backup snapshot, then all emails, installls, uninstalls, modifications, configuration changes, etc... will be lost.
IF I were still developing code I definitely would do a daily incremental snapshot. Yesterday I made a snapshot pair before I did the latest download update. This morning when I turned my computer on I could no longer connect via my wifi. Did the update change or kill something? I recreated my 5Ghz connection but no joy. So, I rolled back to yesterday's snapshot. The wifi still refuses to come up, but I now know it is not related to the software or the yesterday's update.

Unfortunately, Btrfs is no help in repairing wifi chips.

[TWPonKubuntu]

GG:

So how much of your hard disk space is being used for those snapshots?

[oshunluvr]

GG:

So how much of your hard disk space is being used for those snapshots?

The answer has been given: "Snapshots initially take no data space. As the source subvolume changes, the snapshot grows because it retains the original source contents."

There is no other way I can think of to say it. A snapshot contains no data of it's own until you amend the source subvolume. Then the data space occupied by the snapshot equals the amount of the amended data from the source:

Example:
Subvolume @A is 500MB in data size and occupies 500MB of your hard drive.
A snapshot is taken and named @B. The total data space occupied at that moment for both @A and @B is still 500MB
You delete 100MB of files from @A. The total data space occupied remains for both @A and @B at 500MB
You add 100MB of new files to @A. The total data space occupied for both @A and @B is now 600MB

[TWPonKubuntu]

oshunluver:

I think I understand what you mean, but surely there is some overhead involved in the snapshot itself.

Do you mean that the snapshot merely sets a flag on data records, hence taking no addition space for the file itself?

[oshunluvr]

oshunluver:

I think I understand what you mean, but surely there is some overhead involved in the snapshot itself.

Do you mean that the snapshot merely sets a flag on data records, hence taking no addition space for the file itself?

Precisely.

Here's a better breakdown:

[FONT=monospace][COLOR=#54FF54][B]stuart@office[/B][/COLOR][COLOR=#000000]:[/COLOR][COLOR=#5454FF][B]~[/B][/COLOR][COLOR=#000000]$ bt fi df /   [/COLOR]
Data, single: total=221.01GiB, used=163.20GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=5.01GiB, used=2.38GiB
GlobalReserve, single: total=350.03MiB, used=0.00B
[/FONT]

A snapshot may use some metadata and system space, but no data space until changes occur in the data. Now if you transmit (btrfs send|receive) a snapshot to a different btrfs filesystem, then it consumes an equal amount of data space as the original source subvolume.

[GreyGeek]

I have three 750 GB HD's inside my laptop.
sda1 contains my Neon installation.
sdb was formatted as a btrfs subvolume that is not part of my Neon pool. I use it for archival storage
sdc1 ditto.

root@jerry-Aspire-V3-771:~# clear
root@jerry-Aspire-V3-771:~# mount /dev/sda1 /mnt
root@jerry-Aspire-V3-771:~# vdir /mnt
total 0
drwxr-xr-x 1 root root 262 May 24 18:36 @
drwxr-xr-x 1 root root  10 May  3 23:06 @home
drwxr-xr-x 1 root root  10 May  3 23:06 @homeold
drwxr-xr-x 1 root root 262 May 24 18:36 @old
drwxr-xr-x 1 root root 192 May 25 22:26 snapshots

@old and @homeold are the original @ and @home when I booted up this morning.
I did "mv /mnt/@ /mnt/@old" and "mv /mnt/@home /mnt/@homeold" and followed those commands with
btrfs su snapshot /mnt/snapshots/@_20180525 /mnt/@
btrfs su snapshot /mnt/snapshots/@home_20180525 /mnt/@home
Then I closed the konsole and rebooted.

I make all my snapshots as read only ("-r"). In terms of size there is no difference between a rw snapshot and a ro snapshot, until changes are made in @ and @home.

I used those two snapshot commands because the @_20150525 and @home_20180525 snapshots were read only and could not be "mv"'d to @ and @home. IF I were to restore using snapshots on my archival drives (sdb and sdc1) I would have to use the "send & receive" commands because subvolumes cannot be moved outside the pool using mv.

root@jerry-Aspire-V3-771:~# [B]vdir /mnt/snapshots/[/B]
total 0
drwxr-xr-x 1 root root 206 May  4 09:11 @_20180507
drwxr-xr-x 1 root root 256 May  9 10:22 @_20180512
drwxr-xr-x 1 root root 262 May 14 22:39 @_20180522
drwxr-xr-x 1 root root 262 May 24 18:36 @_20180525
drwxr-xr-x 1 root root  10 May  3 23:06 @home_20180507
drwxr-xr-x 1 root root  10 May  3 23:06 @home_20180512
drwxr-xr-x 1 root root  10 May  3 23:06 @home_20180522
drwxr-xr-x 1 root root  10 May  3 23:06 @home_20180525


root@jerry-Aspire-V3-771:~# [B]btrfs fi df /mnt/snapshots/@_20180525[/B]
Data, [B]single[/B]: [B]total=112.01GiB, used=111.60GiB[/B]
System, [B]DUP[/B]: total=8.00MiB, used=16.00KiB
Metadata, [B]DUP[/B]: total=1.50GiB, used=989.50MiB
GlobalReserve, [B]single[/B]: total=157.53MiB, used=0.00B
root@jerry-Aspire-V3-771:~# [B]btrfs fi df /mnt/snapshots/@home_20180525[/B]
Data, [B]single[/B]: [B]total=112.01GiB, used=111.60GiB[/B]
System, [B]DUP[/B]: total=8.00MiB, used=16.00KiB
Metadata, [B]DUP[/B]: total=1.50GiB, used=989.50MiB
GlobalReserve, [B]single[/B]: total=157.53MiB, used=0.00B

Both @_20180525 and @home_20180525 have the same "total" and "used" because they are part of the same "ROOT_FS" btrfs pool. At the time I made them yesterday they contained essentially zero bytes, aside from soe metadata. The size you see is essentially the pool size data, because over night nothing had been added, deleted or changed. IF you compare sizes between two subsequent snapshots you can get an idea of how much has changed in your system.

root@jerry-Aspire-V3-771:~# [B]btrfs fi usage /mnt/snapshots/@_20180525 [/B]
Overall:
    Device size:                 698.64GiB
    Device allocated:            115.02GiB
    Device unallocated:          583.61GiB
    Device missing:                  0.00B
    Used:                        [B]113.53GiB[/B]
    Free (estimated):            584.02GiB      (min: 292.21GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              157.53MiB      (used: 0.00B)

Data,single: Size:112.01GiB, [B]Used:111.60GiB[/B]
   /dev/[B]sda1[/B]     112.01GiB

Metadata,DUP: Size:1.50GiB, [B]Used:989.50MiB[/B]
   /dev/[B]sda1[/B]       3.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sda1      16.00MiB

Unallocated:
   /dev/sda1    [B] 583.61GiB[/B]

root@jerry-Aspire-V3-771:~# [B]btrfs fi usage /mnt/snapshots/@home_20180525
Overall:[/B]
    Device size:                 698.64GiB
    Device allocated:            115.02GiB
    Device unallocated:          583.61GiB
    Device missing:                  0.00B
    Used:                        [B]113.53GiB[/B]
    Free (estimated):            584.02GiB      (min: 292.21GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              157.53MiB      (used: 0.00B)

Data,single: Size:112.01GiB, [B]Used:111.60GiB[/B]
   /dev/sda1     112.01GiB

Metadata,DUP: Size:1.50GiB, Used:989.50MiB
   /dev/sda1       3.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sda1      16.00MiB

Unallocated:
   /dev/sda1     583.61GiB


root@jerry-Aspire-V3-771:~# [B]btrfs fi show /[/B]
Label: none  uuid: 47a4794f-b168-4c61-9d0c-cc22f4880116
        Total devices 1 FS bytes [B]used 112.57GiB[/B]
        devid    1 size 698.64GiB [B]used 115.02GiB[/B] path /dev/sda1

Difference is 3Gb of Metadata

root@jerry-Aspire-V3-771:~#[B] btrfs su list -p .[/B]
ID 366 gen 9228 parent 5 top level 5 path snapshots/@_20180507
ID 367 gen 9230 parent 5 top level 5 path snapshots/@home_20180507
ID 374 gen 17474 parent 5 top level 5 path snapshots/@home_20180512
ID 375 gen 17476 parent 5 top level 5 path snapshots/@_20180512
ID 386 gen 35470 parent 5 top level 5 path snapshots/@_20180522
ID 387 gen 35473 parent 5 top level 5 path snapshots/@home_20180522
ID 391 gen 37836 parent 5 top level 5 path @old
ID 392 gen 37836 parent 5 top level 5 path @homeold
ID 397 gen 37833 parent 5 top level 5 path snapshots/@_20180525
ID 398 gen 37832 parent 5 top level 5 path snapshots/@home_20180525
ID 399 gen 37994 parent 5 top level 5 path @home
ID 400 gen 37988 parent 5 top level 5 path @

Here is the data on my sdc1 backup HD.

root@jerry-Aspire-V3-771:~# [B]mount -t btrfs /dev/sdc1 /backup/[/B]
root@jerry-Aspire-V3-771:~#[B] vdir /backup[/B]
total 9856
drwxr-xr-x 1 root  root      262 May 24 17:31 [B]@_20180522[/B]
-rw-rw-r-- 1 jerry jerry    2800 May  1 11:07 addressbook2018_05_01.csv
drwxrwxr-x 1 jerry jerry      56 Sep  7  2017 bin
-rw-rw-r-- 1 jerry jerry  241247 Apr 10 21:21 corn_worksheets.zip
drwxr-xr-x 1 jerry jerry    8126 Apr 16 15:55 Documents
drwxr----- 1 jerry jerry    5974 Apr 30 20:59 Downloads
drwxrwxr-x 1 jerry jerry      78 Dec 15 15:41 Etcher
drwxr-xr-x 1 jerry jerry    2452 Feb 10  2011 GoogleEarthfiles
drwxr-xr-x 1 jerry jerry    1744 Dec 17  2016 HolySpirit
drwxr-xr-x 1 root  root       10 May 24 18:08 [B]@home_20180522[/B]
drwxr-xr-x 1 jerry jerry     446 Mar 27  2017 IQAN\ Applications
drwxr-xr-x 1 root  root     4584 May  2 10:31 jerry
drwxr-xr-x 1 jerry jerry    1266 Feb 10  2011 jlkreps_net
drwxr-xr-x 1 jerry jerry      18 Nov  4  2016 kmail_emails
drwx------ 1 jerry jerry     622 Oct  2  2017 Mail
drwxrwxr-x 1 jerry jerry     356 Apr 13 11:41 MCserver112
drwxrwxr-x 1 jerry jerry     510 Mar 23 13:55 Minecraft
drwxrwxr-x 1 jerry jerry    1722 Sep 14  2017 Minecraft112
drwxr-xr-x 1 jerry jerry    1152 Jun 25  2017 Music
drwxr-xr-x 1 jerry jerry    3508 Feb 15  2004 MyWebPageBKUP
drwxr-xr-x 1 root  root      384 May  2 11:56[B] @Neon1604_20180502[/B]
drwxr-xr-x 1 root  root       52 May  2 12:38 [B]@Neon1604home_20180502[/B]
drwxr-xr-x 1 jerry jerry   14818 Apr 26 20:05 Pictures
drwxr-xr-x 1 jerry jerry     420 Apr 15 21:27 SageMath
-rw-rw-r-- 1 jerry jerry  236705 May  1 21:02 Screenshot_2018-05-01_21-01-58.png
-rw-rw-r-- 1 jerry jerry  236142 May  1 21:03 Screenshot_2018-05-01_21-03-21.png
drwxrwxr-x 1 jerry jerry     252 Apr 11  2017 SimpleScreenRecorder
drwxrwxr-x 1 jerry jerry     606 Jan  8 21:53 WebPlotDigitizer-linux-x64
-rw-r----- 1 jerry jerry 9369618 Sep 29  2010 wireless_kismet.pdf


root@jerry-Aspire-V3-771:~# [B]btrfs fi show /backup[/B]
Label: none  uuid: b3131abd-c58d-4f32-9270-41815b72b203
        Total devices 1 FS bytes [B]used 248.80GiB[/B]
        devid    1 [B]size 698.64GiB[/B] [B]used 252.02GiB[/B] path /dev/sdc1


root@jerry-Aspire-V3-771:~# [B]btrfs fi df /backup/@Neon1604_20180502[/B]
Data, single: total=248.01GiB, [B]used=247.33GiB[/B]
System, DUP: total=8.00MiB, used=48.00KiB
Metadata, DUP: total=2.00GiB, used=1.47GiB
GlobalReserve, single: total=351.14MiB, used=0.00B

root@jerry-Aspire-V3-771:~#[B] btrfs fi df /backup[/B]
Data, single: total=248.01GiB, [B]used=247.33GiB[/B]
System, DUP: total=8.00MiB, used=48.00KiB
Metadata, DUP: total=2.00GiB, used=1.47GiB
GlobalReserve, single: total=351.14MiB, used=0.00B

Note that when one does a send & receive to an external subvolume the snapshot is populated in the process, and only ro snapshots can be sent, which is why I make mine ro. Both a snapshot and an ro snapshot of the same subvolume are created nearly instantly, so no significant amount of data is being piled into the snapshot when it is created. However, when I, for example, send @YYYYMMDD to the /backup drive it takes about 10-15 minutes. To send @homeYYYYMMDD takes about 15-20 because there is more stuff in it.

[B]btrfs su snapshot -r /mnt/@  /mnt/snapshots/@20180525
btrfs send /mnt/snapshots/@20180522 | btrfs receive /backup[/B]
[B]sync[/B]

As you can see, with 10 snapshots and two subvolumes, @ and @home, I still have 583 GB unallocated on my primary drive, sda1.
My sdc1, with four snapshots shows 446 GB unallocated, at that drive, as you can see, holds a TON of stuff besides the four snapshots.

root@jerry-Aspire-V3-771:~# [B]btrfs fi usage /backup[/B]
Overall:
    Device size:                 698.64GiB
    Device allocated:            252.02GiB
    Device unallocated:          446.61GiB
    Device missing:                  0.00B
    Used:                        250.27GiB
    Free (estimated):            447.29GiB      (min: 223.99GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              351.14MiB      (used: 0.00B)

Data,single: Size:248.01GiB, Used:247.33GiB
   /dev/sdc1     248.01GiB

Metadata,DUP: Size:2.00GiB, Used:1.47GiB
   /dev/sdc1       4.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB
   /dev/sdc1      16.00MiB

Unallocated:
   /dev/sdc1     [B]446.61GiB[/B]

All together, I'd estimate that I have room for 10-15 more snapshots on sda1 and 5-10 on sdc1.

PS - I forgot about sdb1. I have about 690GB unused on it, so add 20-30 more snapshots to the total. Or, I may move the few GB that are on it to sdc and use it to play with ZFS.

[TWPonKubuntu]

oshunluver and GG:

Thank you both, I've now got a better understanding of snapshots.

I still need bigger drives, but that's for a next generation laptop.