Migrating From EXT4 to BTRFS?

[gnomek]

I understand but ... not all of it.

Let's say I have Kubuntu and as you recommend both root and home are on the same Btrfs partition (the same swimming pool) and I want to try another distribution let's say OpenSuse and get rid of Kubuntu but keep my user settings.

With ext4 and seperate root and home on two partitions it was obvious: just format root during install of OpenSuse and leave home partition unformatted and asign new home to old home.

What is the procedure on Btrfs disk?

[GreyGeek]

LuciFer experiment begun

mmm... I've had more failures with USB HD's (CDW's Passport drives) than with any USB stick I've used. In fact, since I began using USB sticks for storage I've never had one fail on me during the middle of a write. That said, I just bought a 256 GB USB 2.0 stick for $25 from Amazon. I plan to format it with Btrfs using compression to make it appear to be about 500GB as an experiment. The stick may be Chinese junk Either way I'll soon know.
....

I got the $25 USB with 256GB of space. It was formatted with exfat. And, as I expected, it is Chinese. It remains to be seen if it is Chinese junk or not.

I plugged it in and used the partition manager to destroy the old partition and create a new MSDOS partition, sdd1, as btrfs. I quit the partition manager and unmounted the USB stick and unplugged it.

Then I plugged it in. It was recognized by Dolphin. I opened a Konsole, "sudo -i" to root and did the following:

mount /dev/disk/by-uuid/fac47a34-ffd2-4850-85ab-e9034874983a /mnt

vdir /mnt
total 0

mkdir /mnt/snapshots

btrfs subvolume create /mnt/@data
Create subvolume '/mnt/@data'

sync
vdir /mnt
total 0
drwxr-xr-x 1 root root 0 Jun 20 15:27 @data
drwxr-xr-x 1 root root 0 Jun 20 15:24 snapshots

umount /mnt

mkdir /data

mount -t btrfs -o subvol=@data,defaults /dev/disk/by-uuid/fac47a34-ffd2-4850-85ab-e9034874983a /data

vdir /data
total 0

btrfs subvolume show /data
/data
Name: @data
UUID: 95720b06-c958-af44-be60-06da186c0240
Parent UUID: -
Received UUID: -
Creation time: 2018-06-20 15:27:30 -0500
Subvolume ID: 257
Generation: 14
Gen at creation: 11
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):

Well, now that @data is mounted at /data, can I mount data's <ROOT_FS>?
Sure can!

mount /dev/disk/by-uuid/fac47a34-ffd2-4850-85ab-e9034874983a /mnt

vdir /mnt
total 0
drwxr-xr-x 1 root root 0 Jun 20 15:27 @data
drwxr-xr-x 1 root root 0 Jun 20 15:24 snapshots

grep data /proc/mounts
/dev/sdd1 /data btrfs rw,relatime,space_cache,subvolid=257,subvol=/@data 0 0


So, @data is mounted at /data on my system HD, and the <ROOT_FS> is mounted at /mnt also on my system HD, showing @data as a subvolume and snapshots as a subdirectory.

Now I can make snapshots of @data and mv them to /mnt/snapshots without unmounting /data, thus keeping it in service while I do maintenance work.

What I haven't tested yet is making the 256GB USB stick appear to be about 500GB using compression as an option during creation. Partition manager gave me the option of using LUK to encrypt it, but not adding compression. This is done by adding "-o compress" or "-o compress-force" to the mount options:


mount -t btrfs -o subvol=@data,defaults,compress-force /dev/disk/by-uuid/fac47a34-ffd2-4850-85ab-e9034874983a /data

I am currently loading up the 256 USB stick (trade name LuciFer) with data from my home account after I mounted it with the compress-force (zlib) option.

[oshunluvr]

I understand but ... not all of it.

Let's say I have Kubuntu and as you recommend both root and home are on the same Btrfs partition (the same swimming pool) and I want to try another distribution let's say OpenSuse and get rid of Kubuntu but keep my user settings.

With ext4 and seperate root and home on two partitions it was obvious: just format root during install of OpenSuse and leave home partition unformatted and asign new home to old home.

What is the procedure on Btrfs disk?

https://www.kubuntuforums.net/showth...trfs+multiboot
https://www.kubuntuforums.net/showth...trfs+multiboot

[GreyGeek]

LuciFer experiment continued

In a couple of posts above I described my experiences putting a btrfs subvolume on a Chinese 256GB USB stick with a logo of "LuciFer", hoping it doesn't behave as its namesake and give me a devil of a time.

Last night I opened a Konsole and mounted LuciFer with the "-o compress-force" option. Then, using mc, I began copying selected directories in my home account to LuciFer. ASCII based files copy comparatively fast, up to 38MBps, *.so (shared object libraries) not so fast, around 5-10mbps, and mp3 and mp4 files copy slowly, at around 800KBps or less.

mc uses only two of my eight cores so the rest of my system is usable in what I'd call a normal manner.

Last night mc trucked along for 9 hours, copying 40GB of the 60GB I selected to copy, before I decided to abort the copy and check things out. It took 20 minutes for mc to actually return to the CLI and return a prompt. I immediately issued "sync" and it took another 30 minutes before sync gave me the prompt again. That was surprising.

I mounted LuciFer's <ROOT_FS> and saved a snapshot, @data20180621 into /mnt/snapshots. It took a fraction of a second.

Then I restarted mc and resumed copying the ~/Video subdirectory into LuciFer, skipping what had been previously copied. That was 53 minutes ago. It is now at 1.95GB out of 20GB, copying an mp4 file at about 350Kbps. Mc says it has about 8 and 1/2 hours to go. I am going to let it run in the background to finish, and then take another snapshot.

sudo btrfs fi show /data
Label: '256GB' uuid: fac47a34-ffd2-4850-85ab-e9034874983a
Total devices 1 FS bytes used 40.52GiB
devid 1 size 250.00GiB used 44.02GiB path /dev/sdd1

[FONT=courier new][B]sudo btrfs fi usage /data[/B][/FONT]
Overall:
    Device size:                 250.00GiB
    Device allocated:             44.02GiB
    Device unallocated:          205.97GiB
    Device missing:                  0.00B
    Used:                         40.75GiB
    Free (estimated):            207.61GiB      (min: 104.62GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               67.05MiB      (used: 0.00B)

Data,single: Size:42.01GiB, Used:40.37GiB
   /dev/sdd1      42.01GiB

Metadata,DUP: Size:1.00GiB, Used:194.17MiB
   /dev/sdd1       2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdd1      16.00MiB

Unallocated:
   /dev/sdd1     205.97GiB

My next milestone will be the 64GB mark. IF LuciFer can store beyond that then I'll know it can hold more than 64GB. Then the next milestone will be the 128GB mark. IF I can get beyond that then more than likely LuciFer is a 256GB device. So far, I can say that LuciFer does not appear to be Chinese trash. I checked it for heat and it is only slightly warm to the touch. My IR gun says it is 79.4F, only 7 degrees above ambient temperature of the granite desktop it is setting on. Anyway, I am confident enough in this USB stick that I ordered two more.

Impressions so far:
LuciFer is a USB 2.0 device, and its speed betrays it as such. That said, the maximum speed of a USB 2.0 stick is 53MBps. My fastest, according to mc, was 38MBps on ASCII text, WITH COMPRESSION ON. Honestly, not too shabby. When one considers that speeds on either 2.0 or 3.0 depends as much on the device maker, hence price, and that at most the practical increase in speed of a USB 3.0 over a USB 2.0 is around 3X, LuciFer is not doing too bad, IMO.

The effects of compression are difficult to determine. I am going to wait till all that I want to copy to LuciFer has been copied, then I'll compare the sizes of those files on my disk with the over all size shown on LuciFer.

[oshunluvr]

Let's say I have Kubuntu and as you recommend both root and home are on the same Btrfs partition (the same swimming pool) and I want to try another distribution let's say OpenSuse and get rid of Kubuntu but keep my user settings.

With ext4 and seperate root and home on two partitions it was obvious: just format root during install of OpenSuse and leave home partition unformatted and asign new home to old home.

Generally, you would rename or snapshot your current home subvolume, do the new installation, then mount your old home subvolume at /home.

Opensuse will create a dozen or so subvolumes and effectively "hide" them at the user level so it's a bit more difficult with that particular distro. Still doable, just a bit more work.

[GreyGeek]

gnomek, I saw your kernel log and your fstab file. You formatted sda1 and sda5 using Btrfs (to put @ and @home on separate partions ?) but you made them EXT4 systems in your fstab file. That won't work. To make matters worse, you used the tune2fs tool on them.

With Ubuntu based distros the usual procedure is to let the installer do its magic on a single partition. That Btrfs pool on that partition will contain both @ and @home. OpenSUSE doesn't do it that way, and their paradigm doesn't mix with Ubuntu's.

After you finish a normal Btrfs install on Kubuntu, you can mount the <ROOT_FS> to, say, /mnt, which will expose ID=5, the <ROOT_FS> level.
mount /dev/disk/by-uuid/uuuuiiiidddd /mnt
Doing "vdir /mnt" will show
/mnt/@
/mnt/@home

In /etc/fstab the installer binds / to @ and /home to @home.

You can then use "btrfs subvol create /mnt/@somesubvol.
Then, as root, create a directory under / called somesubvol (not under /mnt)

In the fstab bind /somesubvol to @somesubvol
etc.... for as many "rafts" as you want.

[GreyGeek]

LuciFer experiement continued - part II

I reformatted the 256GB LuciFer USB stick and then mounted it without using compression.
Whereas the compression allowed data to flow in at around 345Kbps for mp4 files, without using compression the mp4 files moved in at around 1MBps to 3MBps.

Over all, I'd say that it is 3X faster in copying stuff into @data than with using compression. It took only 4 hours to copy of what took over 12 hours using compression. But, the compression (using zlib) was about 3X, so my 256GB USB stick "appeared" to be about 700GB.

[jlittle]

... the compression allowed data to flow in at around 345Kbps for mp4 files, without using compression the mp4 files moved in at around 1MBps to 3MBps.

That's not good. IMO the compression should spot early that the files are already compressed and not try to compress them, and give similar performance as without compression. Using compression is meant to be a speed-up, because there's less i/o.

[GreyGeek]

That's not good. IMO the compression should spot early that the files are already compressed and not try to compress them, and give similar performance as without compression. Using compression is meant to be a speed-up, because there's less i/o.

That’s because I used “compress-force”. Had I used “compress” then compression would behave as you describe.

[jlittle]

That’s because I used “compress-force”. Had I used “compress” then compression would behave as you describe.

Good to know - that's a contribution you've made by experimenting and reporting, thank you.

Regards, John Little

[GreyGeek]

... I just bought a 256 GB USB 2.0 stick for $25 from Amazon. I plan to format it with Btrfs using compression to make it appear to be about 500GB as an experiment. The stick may be Chinese junk Either way I'll soon know.

With the massive price drops of SSD's I was hoping that USB sticks would show a similar decline. Well, the USB Stick, which had the brand name of LuciFer, turned out to be a demon. I formatted it using Btrfs with forced-compression and used mc under root to transfer several GB to it from my home accounts. While the first few GB moved over rather quickly, depending on the file type, at about 30MB/s, it gradually slowed to between 345KB/s to 1MB/s, depending on the file type, as I neared 50GB. As you'd suspect, highly compressed files like zips, mp4's, etc., took the longest time. I despaired of filling up the USB Stick within my remaining life time. So, I decided to reformat it with Btrfs use just the compress option to mount it. Transferring files, as I reported earlier, was about 3X faster but as the amount of files stored got around 30GB it resumed its snail-like behavior. So, I canceled mc and quit it. It dropped to the Konsole. I issued "sync", which took a full 30 minutes to come back.

I decided to test file transfer speed with EXT4 and reformatted the stick with that fs. I was up to around 26GB of file transfered and the process was getting so slow I began to suspect that it was a fake drive with much less capacity than was advertised. I quit mc and issued sync at the prompt. I never got the prompt back. So, attempted to umount the drive and got the "busy" warning. I couldn't umount it. I didn't want to reboot so I just pulled it out, waited a few seconds, and plugged it back in. It was dead.

The 2nd one I ordered came in the mail. I formatted it with EXT4 and installed f3 from the repository and ran f3write on it. It took 24 hours to do 55 writes of 2MB text files and the prompt said that it would require 43 more hours to test the entire disk. I didn't trust that number because as the writing continued on the predicted time continued to increase. And, it appeared to me that the stick was using memory cache to "enhance" its capacity.

At 2MB per file (if that is its size) it would take 128,000 files to fill that stick IF it were a 256GB USB stick. So, this morning, after 24 hours of f2write running, I canceled the operation and began running f3read on the stick. I got the following results:

As you can see, it got to only file 14 before the "corruption" began to show. The f3 app also has f3fix to reset the size of the stick to its real value, but I'm sending these fakers back to Amazon for a refund.

[TWPonKubuntu]

GG;

Thank you for taking the time to do this testing and reporting to us. I hope it is just a problem with that single brand of USB memory...

I wonder if Amazon will take a hint and drop that line from their inventory?

[GreyGeek]

Time will tell. Amazon was quick to accept the returns without question. The sticks are now on their way back to Bezos via UPS.

[SpecialEd]

Did you leave feedback at Amazon? When you first posted the link to those drives there was no feedback yet.

[GreyGeek]

Did you leave feedback at Amazon? When you first posted the link to those drives there was no feedback yet.

I did. They have yet to post it.
I wish they had zero stars.