Comparison of Ext4 and Btrfs

[GreyGeek]

The #2 pencil trick is a good idea. The problem could be temperature dependent because I let it run all night giving Kparted a chance to recognize it, which it finally did. It was after that, while it was still plugged in, that I used "mkfs.btrfs -f /dev/sdc" to format it and then copy and read some files to it. The next day, after I plugged it in, it wouldn't recognize the btrfs volume.

[oshunluvr]

Bummer your drive failed. Sure you already know this, but a pencil across a microscopic crack can restore a connection. You could just "color" over all the soldered connections with a #2 and see if it goes back to life.

I never trusted the pre-built USB drives. I have a Inateck USB 3.0 2.5" enclosure that I use. I have a 500GB laptop drive it it and it works great.

The btrfs format works because it really doesn't write to the entire disk like most formats do. It creates the headers and inode tables and then a very small amount of space for use. The "formatted" region grows dynamically as data is added. In fact, unless you use a specific tool to wipe a drive, you can reformat a btrfs file system with the files all still intact. Last time I reconfigured a btrfs partition, I had to run wipefs to clear the data inodes from the partition and free the space for use.

[GreyGeek]

After I booted up this morning I plugged in the Passport USB HD and tried to mount it. It failed.

1224.996256] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1224.996262] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1224.996266] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1224.996271] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 40 00 00 38 00
[ 1224.996274] blk_update_request: critical medium error, dev sdc, sector 64
[ 1228.350514] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1228.350520] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1228.350523] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1228.350527] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 80 00 00 90 00
[ 1228.350531] blk_update_request: critical medium error, dev sdc, sector 128
[ 1231.851278] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1231.851284] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1231.851287] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1231.851292] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 50 00 00 08 00
[ 1231.851295] blk_update_request: critical medium error, dev sdc, sector 80
[ 1231.851304] Buffer I/O error on dev sdc, logical block 10, async page read
[ 1235.236087] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1235.236093] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1235.236096] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1235.236101] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 50 00 00 08 00
[ 1235.236104] blk_update_request: critical medium error, dev sdc, sector 80
[ 1235.236112] Buffer I/O error on dev sdc, logical block 10, async page read
[ 1238.713461] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1238.713466] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1238.713470] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1238.713474] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 08 00
[ 1238.713478] blk_update_request: critical medium error, dev sdc, sector 512
[ 1242.081002] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1242.081008] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1242.081011] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1242.081016] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 02 00 00 00 08 00
[ 1242.081020] blk_update_request: critical medium error, dev sdc, sector 512
[ 1242.081028] Buffer I/O error on dev sdc, logical block 64, async page read
[ 1242.089252] BTRFS info (device sdc): disk space caching is enabled
[ 1242.089256] BTRFS info (device sdc): has skinny extents
[ 1245.590032] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1245.590037] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1245.590041] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1245.590045] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 a0 20 00 00 20 00
[ 1245.590048] blk_update_request: critical medium error, dev sdc, sector 40992
[ 1248.967198] sd 6:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1248.967204] sd 6:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
[ 1248.967208] sd 6:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[ 1248.967212] sd 6:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 e0 20 00 00 20 00
[ 1248.967216] blk_update_request: critical medium error, dev sdc, sector 57376
[ 1248.967349] BTRFS error (device sdc): failed to read chunk root
[ 1249.005002] BTRFS: open_ctree failed
root@jerry-Aspire-V3-771:~#

It will always recognize the existence of an HD at /dev/sdc and I can even "mkfs.btrfs -f /dev/sdc" and it will format. But when I attempt to mount it "mount /devsdc /backup" it fails. I've tried without a partition and with both a GPT and msdos type. All fail. I have a 50% failure rate with WD Passport USB drives. In researching the problem I find it is a rule instead of an exception, regardless of OS. So, I have a 50% failure rate with WD Passport USB HDs. Lesson learned.

[GreyGeek]

Comparison of Ext4 and Btrfs

Interesting thing happened, oshunluver. My 500gb 2.5" WD Passport USB drive failed to mount a couple of weeks ago. Fdisk-l failed to recognize its presence and neither would Kpartition, even after letting it scan overnight. Yesterday I thought that perhaps only the USB electronics were bad but the drive itself was good. I popped the cover off the drive and found that the drive was riveted to the circuit board and the USB circuitry was integrated into the drive connection circuitry. For grins and giggles I plugged it into my USB hub and fired up Kpartition. It suddenly "found" /dev/sdc and began scanning. It was stuck on 33% for several minutes but finally displayed it along with my other two drives. Apparently the problem was an intermittent connection (crack in soldering or trace?) which was re-established by my putzing around with it.

I tried to create a partition using GPT. When I applied the configuration it tried but shortly returned an error.

That's when what you wrote about "clean" HD's came to mind. I closed Kpartition and opened a Konsole. In it I issued:
sudo mkfs.btrfs /dev/sdc
It worked!
I immediately created a subvolume and cp'd some files to it from my home account. That worked nicely. I powered down last night. This morning comes the test to see if, when I plug Passport in, I can mount it to /mnt again.


Sent from my iPhone using Tapatalk

[oshunluvr]

If you get a new "clean" drive or use dd to zero out an entire device, you'll have to make a partition table before you'll be able to make partitions. If you use the Kubuntu installer or Kpartition Manager and click on a totally empty drive, you get a message asking you if you want to create a new partition table. Here's what my unpartitioned btrfs server drive shows with gdisk:

[FONT=monospace][COLOR=#000000]smith@server:~$ sudo gdisk /dev/sda[/COLOR]
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
 MBR: not present
 BSD: not present
 APM: not present
 GPT: not present

Creating new GPT entries.

Command (? for help): 
[/FONT]

verses a partitioned drive:

[FONT=monospace][COLOR=#000000]smith@server:~$ sudo gdisk /dev/sdb[/COLOR]
GPT fdisk (gdisk) version 0.8.8

Partition table scan:
 MBR: protective
 BSD: not present
 APM: not present
 GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): 

[/FONT]

[GreyGeek]

Comparison of Ext4 and Btrfs

BTRFS sanity check - it detected the partition table and prevented you from "accidentally" formatting a drive that was being used. Remember when you apply BTRFS formatting to an entire device, it uses the partition table and boot record space along with everything else.

Ah, so deleting the partition does not delete the partition table? I did not know that, after all these years. I thought creating the partition only created the table, which set the parameters for subsequent formatting, and when one deleted the partition the table was deleted as well.

[oshunluvr]

I have no clue as to what the "dos" partition was, since I deleted the only partition on the HD

BTRFS sanity check - it detected the partition table and prevented you from "accidentally" formatting a drive that was being used. Remember when you apply BTRFS formatting to an entire device, it uses the partition table and boot record space along with everything else.

[GreyGeek]

I finished the transformation. Here is a brief account.
I opened KPartition and deleted the sdb1 partition.
From within KPartition I could NOT format just /dev/sdb alone without creating sdb1, a partition.
So, I closed KPartition and opened a root Konsole.
Without attempting to mount /dev/sdb to /backup I issued
mkfs.btrfs /dev/sdb
It gave me an error, stating that /dev/sdb had a DOS partition on it.
So, I issued
mkfs.btrfs -f /dev/sdb
and it worked.

Then I mounted /dev/sdb to /backup
mount /dev/sdb /backup
and I also mounted /dev/sda1 to /mnt
mount /dev/sda1 /mnt

I then proceeded to delete the backup snapshots from /mnt and created fresh ro backups.

[FONT=monospace][B][COLOR=#000000]vdir /mnt/snapshots/[/COLOR][/B]
total 0
drwxr-xr-x 1 root root 256 Apr  8 16:53 @_bkup-2017-04-08
drwxr-xr-x 1 root root 256 Apr  8 16:53 @_bkup-2017-04-13
drwxr-xr-x 1 root root  10 Apr  8 16:54 @home_bkup-2017-04-13

root@jerry-Aspire-V3-771:~# [B]btrfs su delete -c /mnt/snapshots/@_bkup-2017-04-08 [/B]    
Delete subvolume (commit): '/mnt/snapshots/@_bkup-2017-04-08'

root@jerry-Aspire-V3-771:~# vdir /mnt/snapshots/                                
total 0
drwxr-xr-x 1 root root 256 Apr  8 16:53 @_bkup-2017-04-13
drwxr-xr-x 1 root root  10 Apr  8 16:54 @home_bkup-2017-04-13

root@jerry-Aspire-V3-771:~# [B]btrfs su delete -c /mnt/snapshots/@_bkup-2017-04-13[/B]
Delete subvolume (commit): '/mnt/snapshots/@_bkup-2017-04-13'

root@jerry-Aspire-V3-771:~#[B] btrfs su delete -c /mnt/snapshots/@home_bkup-2017-04-13[/B]
Delete subvolume (commit): '/mnt/snapshots/@home_bkup-2017-04-13'

root@jerry-Aspire-V3-771:~# [B]vdir /mnt/snapshots/[/B]
total 0

[/FONT]

Now create the new snapshots of @ and @home:

[FONT=monospace][COLOR=#000000]:~# [B]btrfs su snapshot -r /mnt/@ /mnt/snapshots/@_bkup-2017-04-14[/B][/COLOR]
Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@_bkup-2017-04-14'

root@jerry-Aspire-V3-771:~#[B] btrfs su snapshot -r /mnt/@home /mnt/snapshots/@home_bkup-2017-04-14[/B]
Create a readonly snapshot of '/mnt/@home' in '/mnt/snapshots/@home_bkup-2017-04-14'

root@jerry-Aspire-V3-771:~# [B]vdir /mnt/snapshots/[/B]
total 0
drwxr-xr-x 1 root root 256 Apr  8 16:53 @_bkup-2017-04-14
drwxr-xr-x 1 root root  10 Apr  8 16:54 @home_bkup-2017-04-14
[/FONT]

Then I created a subvolume on /backup called "snapshots"

root@jerry-Aspire-V3-771:~# vdir /backup/
total 0
root@jerry-Aspire-V3-771:~# [B]btrfs su create /backup/snapshots[/B]
Create subvolume '/backup/snapshots'

Then send & receive the ro snapshots on /mnt/snapshots to /backup/snapshots

root@jerry-Aspire-V3-771:~# [B]btrfs send /mnt/snapshots/@_bkup-2017-04-14 | btrfs receive /backup/snapshots[/B]
At subvol /mnt/snapshots/@_bkup-2017-04-14
At subvol @_bkup-2017-04-14

root@jerry-Aspire-V3-771:~# [B]btrfs send /mnt/snapshots/@home_bkup-2017-04-14 | btrfs receive /backup/snapshots[/B]
At subvol /mnt/snapshots/@home_bkup-2017-04-14
At subvol @home_bkup-2017-04-14

Now, let's check the usage of the HD space:

[FONT=monospace][COLOR=#000000]:~# [B]btrfs fi usage /backup/[/B][/COLOR]
Overall:
    Device size:                 698.64GiB
    Device allocated:             88.02GiB
    Device unallocated:          610.61GiB
    Device missing:                  0.00B
    Used:                         86.63GiB
   [B] Free (estimated):            611.19GiB[/B]      (min: 305.88GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              130.73MiB      (used: 0.00B)

Data,single: Size:86.01GiB, Used:85.43GiB
   /dev/sdb       86.01GiB

Metadata,DUP: Size:1.00GiB, Used:615.12MiB
   /dev/sdb        2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb       16.00MiB

Unallocated:
   /dev/sdb      610.61GiB


root@jerry-Aspire-V3-771:~# [B]btrfs fi usage /mnt
Overall:[/B]
    Device size:                 691.19GiB
    Device allocated:             95.07GiB
    Device unallocated:          596.12GiB
    Device missing:                  0.00B
    Used:                         88.05GiB
    [B]Free (estimated):            601.16GiB[/B]      (min: 303.11GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              148.78MiB      (used: 0.00B)

Data,single: Size:92.01GiB, Used:86.96GiB
   /dev/sda1      92.01GiB

Metadata,DUP: Size:1.50GiB, Used:557.42MiB
   /dev/sda1       3.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/sda1      64.00MiB

Unallocated:
   /dev/sda1     596.12GiB
[/FONT]

Now, THAT free space makes sense.
I have no clue as to what the "dos" partition was, since I deleted the only partition on the HD, sda1.

[oshunluvr]

LOL, ok glad it did some good. You weren't the target of that peice, but glad you found a tidbit in there.

Most of my desktop drives are partitioned for boot purposes but the server didn't need more than three bootable drives.

For awhile I didn't boot to my ssd's, using a stand alone grub partition on a hard drive instead. Last time I re-org'd my desktop storage the old hard drives on my desktop had reached 60,000+ power-on hours so I moved the grub partition to the ssd's.

[GreyGeek]

I just saw your find essay on Btrfs for noobs and you reminded me of something I did out of habit when I formatted my /dev/sdb drive -- I created a partition and didn't need to!

I'm going to blow away the partitions and give Btrfs /dev/sdb not /dev/sdb1, then redo the archival backups.
Thanks for the memory jog!

[GreyGeek]

Me too.
Going to continue to explore. I haven't installed natime yet, but if I'm going to do a lot of directory and file size comparisons I will have to do that.

[oshunluvr]

Well Jerry, I'm stumped

[GreyGeek]

try

vdir /backup/snapshots/@_bkup-2017-04-13/backup

should be empty.

It is.

[oshunluvr]

try

vdir /backup/snapshots/@_bkup-2017-04-13/backup

should be empty.

[GreyGeek]

So, just for grins, open the /backup folder and navigate to the install backup there, then open the /mnt folder inside the backup of @ and see if anything is in there.

Mmm... clever idea!
However, there is no "@" or "@home" on /backup, as the mc listing in a post above shows.

[FONT=monospace][COLOR=#000000]:/backup/snapshots/@_bkup-2017-04-13# vdir[/COLOR]
total 16
drwxr-xr-x 1 root root    0 Mar 29 11:16 backup
drwxr-xr-x 1 root root 2550 Mar 28 21:58 bin
drwxr-xr-x 1 root root  680 Mar 27 14:25 boot
drwxr-xr-x 1 root root    0 Mar 24 14:45 cdrom
drwxr-xr-x 1 root root  894 Mar 16 10:18 dev
drwxr-xr-x 1 root root 4358 Apr 12 18:45 etc
drwxr-xr-x 1 root root    0 Mar 24 14:44 home
lrwxrwxrwx 1 root root   32 Mar 26 13:16 initrd.img -> boot/initrd.img-4.8.0-42-generic
lrwxrwxrwx 1 root root   32 Mar 24 15:41 initrd.img.old -> boot/initrd.img-4.4.0-67-generic
drwxr-xr-x 1 root root  538 Mar 28 21:58 lib
drwxr-xr-x 1 root root 1304 Mar 27 14:22 lib32
drwxr-xr-x 1 root root  112 Mar 28 21:58 lib64
drwxr-xr-x 1 root root   18 Apr 13 14:16 media
drwxr-xr-x 1 root root    0 Apr 11 22:32 mnt
drwxr-xr-x 1 root root   52 Mar 28 21:59 opt
drwxr-xr-x 1 root root    0 Apr 12  2016 proc
drwx------ 1 root root  202 Apr  5 19:06 root
drwxr-xr-x 1 root root  162 Mar 16 10:19 run
drwxr-xr-x 1 root root 3980 Mar 31 20:28 sbin
drwxr-xr-x 1 root root    0 Feb 24 07:24 snap
drwxr-xr-x 1 root root    0 Mar 16 10:05 srv
drwxr-xr-x 1 root root    0 Feb  5  2016 sys
drwxrwxrwt 1 root root  992 Apr 13 14:35 tmp
drwxr-xr-x 1 root root   80 Mar 27 14:22 usr
drwxr-xr-x 1 root root  108 Apr  3 15:34 var
lrwxrwxrwx 1 root root   29 Mar 26 13:16 vmlinuz -> boot/vmlinuz-4.8.0-42-generic
lrwxrwxrwx 1 root root   29 Mar 24 15:41 vmlinuz.old -> boot/vmlinuz-4.4.0-67-generic

[/FONT][FONT=monospace]
root@jerry-Aspire-V3-771:/backup/snapshots/@_bkup-2017-04-13# vdir /mnt
total 0
[/FONT]

but I don't see anything there.