Compression relinking to new extents??

[DevTeam]

I have a BTRFS file system that was not mounted with compression, and later compression was set.
The issue is that in this case a single file has been copied A LOT post the compression being set; due to COW this meant that ~6T was reduced to ~400G; and now when defrag runs it wants to recreate the linked extents as it compresses them for some reason - that will blow out the space!

Luckily I realised this after only 966G was processed (screenshot below)

[Dev-FileServer serviceadmin]# btrfs fi df /Databases/; compsize /Databases/
Data, single: total=606.00GiB, used=560.86GiB
System, single: total=32.00MiB, used=96.00KiB
Metadata, single: total=11.00GiB, used=6.36GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Processed 1009 files, 15334110 regular extents (42415457 refs), 1 inline.
Type Perc Disk Usage Uncompressed Referenced
TOTAL 41% 560G 1.3T 6.0T
none 100% 386G 386G 5.0T
zstd 18% 174G 966G 966G

This caused our used space to INCREASE by ~20% and we would run out of space had I not noticed this!

Does anyone know how to do this and respect the links so that the existing extents are compressed?
Thanks in advance

[jlittle]

I'm guessing about the size of the file system, and to why you think the defrag is blowing out the space. Enclosing listings in code tags make them more readable:

Type Perc Disk Usage Uncompressed Referenced
TOTAL 41%     560G       1.3T       6.0T
none 100%     386G       386G       5.0T
zstd  18%     174G       966G       966G

The man page for btrfs-filesystem has this note on the df command:

The output needs to be further interpreted and is not suitable for quick overview.

The fi df says to me the usage is 560.9 GB data + 6.4 GB metadata = 567.3 GB. It does not anywhere say how big the fs is, or, confusingly, how much space is free.

I don't know how to interpret the compsize output; it's new to me. From a quick read of the man page, it's saying you have 560 GB of data.

What does sudo btrfs fi show /Databases say?

[GreyGeek]

Or, the usage command:

sudo btrfs filesystem usage /Database

I would also use the balance command

1854 btrfs

balance start -musage=0 /Database
1855 btrfs balance start -dusage=0 /Database
...
1856 btrfs balance start -musage=30 /Database
1857 btrfs balance start -dusage=30 /Database

Above 30 I'd use increments of 10 if the commands above give you lots of chunks recovery.

[GreyGeek]

Using code or quote that list of the balance commands gets split as shown

[DevTeam]

What does sudo btrfs fi show /Databases say?

It states that the size is 617 .... which you could have obtained from what I posted before ... the total+metadata (606+11)

[DevTeam]

Or, the usage command:

I would also use the balance command

Above 30 I'd use increments of 10 if the commands above give you lots of chunks recovery.

Regardless of how I check ALL methods point to increased usage ... the OP shows all of the values, and demonstrates that the compression that is occurring is not respecting the deduped extents from CoW.

I have used I did do a FULL balance (--full-ballance) after I did that and it made no difference!

Any other ideas?

[GreyGeek]

Do "all methods" include a scrub or check?

https://www.thegeekdiary.com/how-to-...-file-systems/

btrfs scrub start /Database

https://www.systutorials.com/docs/li...8-btrfs-check/
Did you run

btrfs check --subvol-extents /Database

to show the extent state for the given /Database ? Take note of the "SAFE OR ADVISORY OPTIONS"

If neither of those commands return successfully then backup your data and reformat the drive.

[jlittle]

It states that the size is 617 .... which you could have obtained from what I posted before ... the total+metadata (606+11)

No, I could not. For example, on this Kubuntu,

$ sudo btrfs fi show /
Label: 'main' uuid: 51f4dd3e-cd2c-4097-8864-d8e1223ea21e
Total devices 1 FS bytes used 153.01GiB
devid 1 size 216.52GiB used 162.02GiB path /dev/sda2
$ sudo btrfs fi df /
Data, single: total=157.01GiB, used=149.15GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=5.01GiB, used=3.86GiB
GlobalReserve, single: total=373.08MiB, used=0.00B

Total + metadata is 162 GiB, but the device has 216 GiB. The "total" numbers shown by btrfs fi df are high water marks; on a newly formatted fs they're quite small. Another drive I have total data + total metadata is 683 GiB, but the size is 932 GiB. (btrfs is a volume manager as well; there can be several drives of various sizes in a btrfs, so the relationship between the totals and drives is indirect.)

GreyGeek suggests the btrfs filesystem usage command; that gives a better overview.

[DevTeam]

It is re-writing the reflinks to point to new extents - I cannot have this...
I just found another post that has the same issue here: https://www.reddit.com/r/btrfs/comme...on_so_it_does/

To check you can create a directory - put a 10GB+ file in it ... copy the file (CoW) ... check the references (will be a lot). Then defrag the folder, the references will then be 0!
Apparently it has nothing to do with the compression!

[GreyGeek]

Do you use "chattr +C" on any folder other than the VM image folder?
Are you using autodefrag in your fstab file mountings? It is much better than manually defragging.

[DevTeam]

Do you use "chattr +C" on any folder other than the VM image folder?
Are you using autodefrag in your fstab file mountings? It is much better than manually defragging.

I have not set anything with +C.
I have autodefrag enabled; but ideally want to have the existing data be compressed now too!

[GreyGeek]

Did you use
"sudo btrfs filesystem defragment" with -vf or -rvf. And, do you have autodefrag as setting in your /etc/fstab in the line for the troublesome drive?