SSD performance problems -- config problem or old drive?

[jlittle]

The cost of several operations, including currently balance, device delete and fs resize

I don't normally do any of these. Maybe my next desktop will have a pair of storage devices, for which a regular balance would make sense, but it sounds like it would usually be a scheduled maintenance thing, so could be scheduled for quiets times.
Or am I missing something?

[chconnor]

Thanks!

I did run into an issue with btrfs on an old laptop with a small SDD (and I assume this is user error somehow because I barely know anything about btrfs) where the drive filled up for no apparent reason and it turned out to be the fact that apt was snapshotting every time it updated. Once I figured out what was happening it wasn't a big deal, but it was an unpleasant surprise. :-)

[GreyGeek]

You were probably reading "info" posted by ZFS fanboys relying on their experiences,or what they read, in 2012. Btrfs has been stable for me for 5+ years. I've never had any sort of failure mode while using it, and all the problems I had were of my own making while experimenting around. Because one can restore from snapshots within a couple minutes playing around with your system doesn't carry the penalties that activity used to carry. The Raid56 feature is the only instability ranking for Btrfs.

I will NEVER use a distro that does not allow setting BTRFS as the root filesystem.

For potential problems a "Gotcha" page is here.

When people first read about Btrfs and how snapshots created almost instantly and at first are essentially empty they go hog wild and use tools like TimeShift or Snapper, etc.., and create dozens of snapshots per day. A few days ago I was testing out the latest release of Snapper that's in the repository and edited the config down considerably. Even then, in the course of 10 hours, Snapper created 17 snapshots. When I did a single "sudo apt update && sudo apt full-upgrade" Snapper made three "Pre & Post" snapshots. After I was done experimenting with Snapper I deleted all of its snapshots and then uninstalled it. IMO, Snapper and TImeShift are more trouble than they are worth, since doing snapshots manually, and restoring if needed, are drop-dead easy. My only caveat, besides not using Raid56, is:

The FAQ says:

Having many subvolumes can be very slow

The cost of several operations, including currently balance, device delete and fs resize (shrinking), is proportional to the number of subvolumes, including snapshots, and (slightly super-linearly) the number of extents in the subvolumes.

This is "obvious" for "pure" subvolumes, as each is an independent file tree and has independent extents anyhow (except for ref-linked ones). But in the case of snapshots metadata and extents are (usually) largely ref-linked with the ancestor subvolume, so the full scan of the snapshot need not happen, but currently this happens.

This means that subvolumes with more than a dozen snapshots can greatly slow down balance and device delete. The multiple tree walks involve both high CPU and IOPS usage. This means that schemes that snapshot a volume periodically should set a low upper limit on the number of those snapshots that are retained.

This "slow down" can occur if snapshots get very old and eventually fill up to the point where they are almost as big as the @&@home subvolumes combined. Say @+@home takes 100GB. Your HD is 500GB. If you create 5 snapshot pairs and then ignore them they'll eventually consume 100GB each. When your HD free space drops below 10% your system will slow down. Below 5% it may appear to stop.

Good snapshot management may use about 7 rolling @ snapshots, one per day, deleting the oldest. Keeping a weekly snapshot would generate another 52 snapshots over the course of a year. The older snapshots could become as large as @, and one or a few may eat up all the drive free space. So, only 4 weekly rolling snapshots is more prudent. A couple monthly snapshots. One yearly snapshot kept on your system is probably one too many. A better scheme would be to keep the 7 daily snapshots and move the weekly, monthly and yearly snapshots to external subvolumes just after they are created, and then deleting them from your local subvolume. 7 daily + 4 monthly+1 yearly=12 snapshots for @. For SSD drives systemd sets cron to run fstrim about once a week, or what ever frequency you set it for. That recaptures extents.

Repeat those numbers for the @home subvolume, and you have a total of about 12 X2 = 24 snapshots.

I didn't see the sense in keeping @home separate from @ and creating @yyyymmdd & @homeyyyymmdd every night, etc. So I copied the contents of @home into @/home and then edited out the stanza in /etc/fstab that mounted @home to /home. Now I only have to snapshot @ and use one send & receive command using the "incremental" flag (-p) to move the snapshot to an external or remote subvolume.

[chconnor]

Thanks -- did some digging, and it sounds like that's about right (except RAID5/6 still sounds like a bad idea with btrfs: https://www.phoronix.com/scan.php?pa...ng-RAID5-RAID6 ).

Jim Salter is still pretty unexcited about btrfs compared to zfs, but sounds like it's not at all a unanimous opinion and sounds like the issues would be unlikely to affect me.

[oshunluvr]

Only thing that has been "sketchy" about BTRFS was RAID5/6 and full disk encryption, both of which have been resolved AFAIK, but I don't use either so worth checking on if those are things you need.

I have been using BTRFS full time since tools version 0.19 without any problems.

[chconnor]

Last I heard on btrfs (which was some years ago now) things were still a little sketchy... (I got all excited from Jim Salter's articles on arstechnica, but then he later advised against using it for important stuff until issues got worked out...) Is it all-clear now?

[GreyGeek]

What Oshunluver said, AND ... EXT4 is getting long in the tooth. Isn't it about time to move to a modern, advanced FS where you don't have to worry about partition sizes, and backups are almost instantaneous?

[oshunluvr]

Yeah, good conversation. Let us know if you get the slowness issue resolved.

[chconnor]

Thanks for the perspectives. I did read up on that "reserved block" issue as well, thanks.

[oshunluvr]

Pulled this from somewhere:

All SSD have overprovsioning. Samsung 840 Pro is sold with 256,000,000,000 Bytes space, but there are 256 GiBi build in which are 274,877,906,944 Bytes. There are 7% internal overprovisioning and there is no need to do overprovisioning. The Samsung tool only creates a partition which is smaller then the total size.
I would just use the SSD as a hard drive and try to keep at least 10% free or more.

[oshunluvr]

Overprovisioning will supposedly help performance, but a nearly full file system will definitely slow it. There USED TO BE a Samsung utility to allow the user to adjust the amount of provisioned space. Samsung has removed the over-provisioning tool from their software: https://community.spiceworks.com/top...d-space-on-ssd.

It's a dilemma for sure. Just my opinion here, but it you have been working to keep your root fs from filling up and are likely experiencing speed reductions due to it being over 90% capacity, https://www.howtogeek.com/165542/why...-fill-them-up/

Seems to me you've possibly overlooked your own actual experience in favor of advise from 2012 that probably doesn't even apply anymore, if it ever did. They knew very little about the actual use-cases for SSDs as they were still pretty new to the consumer market. About half the stuff I've read really applies to the enterprise environment. They used to (and some still do) advise against keeping swap on an SSD. Totally counter to logic IMO. Why spend the money to increase performance then purposely slow down the performance by not using the fastest device for the slowest operation? Makes no sense at all. They talk about wearing it out but I'm still using the first SSD I bought more than a decade ago as my server OS drive. It's way past it's usability date yet keeps on going. Why have something then not use it? /rant The highest wear factor is running trim to often.

For what it's worth, I have been using SSDs since 2010 or so and currently have more than 10 of them in various types, uses. and computers and have never left space any on them unformatted. My two 840 Pros are 100% partitioned and about 75% full and both outperform yours - not by much, but more than zero. I'm just saying in my experience a file system with free space is a larger benefit than unused space doing apparently nothing.

BTW, you seem like a sharp user but in case you weren't aware; the EXT file system by default keeps a 5% reserved amount of space for "system use" that really only apply to a file system that's has the OS on it. When I use EXT for data partitions like /home I reduce the reserved amount to zero and if you have an OS on it and it's a large partition, I shrink the reserved to 1-2%

To see what you've reserved:

sudo tune2fs -l /dev/<partition> | grep 'Reserved'

and to adjust it:

sudo tune2fs -m2 /dev/<partition>

The "-m2" meaning "set reserved at 2%"

[chconnor]

Yeah I keep a pretty tight control on the / size so I don't run out of space. But I didn't realize it could have a negative impact on speed, so I'll try to claw back some more free space and see if that helps.

Re: reserving space -- I was led to believe that it was still useful for the 840 Pro, at least of the era? E.g. this article (2012, which is roughly when I got the drive) seems to conclude that there are pretty clear benefits to it: "Vector and Samsung's SSD 840 Pro both deliver much better IO consistency if you simply set aside 25% of the total NAND capacity as spare area."

swap is encrypted, but on a slower HDD (I have 16GB RAM, so hopefully it's not used much?)

/tmp is already a ram disk, but thanks for that tip.

[oshunluvr]

Maybe it's just me, but unless your / partition is gigantic, 94% is danger close to a broken install IMO. It's a small matter to have a data partition fill up but a 100% / or even /home can wreak havoc. This might account for a bit of the slowness too.

There's no reason I know of to not allocate all the space on the drive to partitions. This is a years old held-over concept that no longer applies (and maybe never did). Unallocated space is not used by the system or drive and has no purpose. This is not the same as a drive with built-in over-provisioning in which the drive controller manages some extra sectors, but I don't think they even do that anymore. The drive controller will take care of bad sectors by marking them as unusable.

Unfortunately, as long as you continue to use EXT4 you will have to guess at your partition sizes. It's been years since I had to do that, but you could create an additional partition in that free space and move some folder onto it without having to re-partition the entire drive. /var and /opt were commonly on separate partitions in the "olden days" of Linux - from which I come, lol. My /var is about 8G right now. /opt can get large is you have lots of non-ubuntu based programs installed, but mines only about 2G

Another thing you can do is move /tmp to RAM. You don't mention a swap partition so if you're using a swap file, you can easily move that by deleting and re-creating it elsewhere.

[chconnor]

$ findmnt /dev/sda2
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda2 ext4 rw,noatime,nobarrier,errors=remount-ro,commit=60

...I expected "data=writeback" in there... /etc/fstab is:

UUID=1234 / ext4 noatime,data=writeback,barrier=0,nobh,commit=60,er rors=remount-ro 0 1

...any ideas? I notice that "nobh" also seems to be disappearing. Should I just ignore these issues and trust that data=writeback and nobh are in effect?

"journalctl -xb" seems to have answered these questions:

EXT4-fs (sda2): Ignoring removed nobh option
EXT4-fs (sda2): re-mounted. Opts: data=writeback,barrier=0,nobh,commit=60,errors=remount-ro

Seems odd that data=writeback doesn't show up in findmnt, but i dunno...

[chconnor]

Yes -- / is 94%, /home is 85%, the "other partition" mentioned above is 8% (all on the same physical drive). I suffer from chronic "partitions too small" syndrome. The next computer I make will have 10x too-large partitions.

The drive is under-partitioned though, for whatever it's worth: 128GB drive, with ~23 GB unallocated.