[SOLVED] ZFS file systems crashing

[CharlieDaves]

So. Many versions ago, ZFS worked perfectly. I could access each drive with SMART, and perform tests and checks on the vdev pools RaidZ1. NOW 22.04 it's all go to hell in a hand basket.
Has anyone read up or anything on upgrading ZFS, or Hardware Driver support?
I have an LSI 2024 8i flashed to IT-Mode (removes all hardware Raid controllers... Making it basically a JBOD controller. It supports (well did until 22.04) full HDD through put. If you go to FreeNAS, TrueNAS, ZFS forums, they all state similar things.

For me, all four of my vdev RaidZ1 pools have major issues, lost or damaged files. STATUS = Errors either read, write or Corruption.
AND ALL THIS HAS BEEN SINCE 22.04 FORCED UPGRADE.

Any solutions would be welcomed. Nothing worse than 40TB of data getting lost or damaged. Some cannot be replaced, as one VDEV is my BACKUP and it's damaged.

Thanks
system info and latest damaged files
2401117=zpool errors.txt

[CharlieDaves]

Question. Within the txt file, I copy and pasted the system Information PCI hardware. Does my LSI 8i SATA HBA card appear to be Okay?
Are there any other commands I could do to check that it's "loaded" or is using the correct system drivers
Many thanks

[claydoh]

I have absolutely zero knowledge of ZFS, and likely few of us in our tiny corner of the Linux sphere have any experience with it.

I'd suggest broadening your scope and include more general and wide ranging user bases, such as Linuxquestions and AskUbuntu.

22.04 FORCED UPGRADE

What does this even mean? It makes no sense, without any context.

[CharlieDaves]

I have absolutely zero knowledge of ZFS, and likely few of us in our tiny corner of the Linux sphere have any experience with it.

I'd suggest broadening your scope and include more general and wide ranging user bases, such as Linuxquestions and AskUbuntu.



What does this even mean? It makes no sense, without any context.

Context. That stupid popup demanding Upgrade now current version expires x.y.z date. It wouldn't go away. I could never disable it. AND when I did a command line update && upgrade guess *** what. Kubuntu decided it was good enough. And forced, my choice of words, forced me into 22.04. That's when things went sideways.
I have 2 internal connected via an LSI 2008 or 0223 IT MODE (means no raid and all disks are presented to the OS directly), and 2 external 4-bay sata boxes.
all 4 raidz1 have since the upgrade started to present massive errors, and now critical errors (unrecoverable files)
I have then found out that since this "forced" upgrade SMART is no-longer available. IT WAS BEFORE the upgrade. 22.04 doesn't appear to be using the correct drivers for the card, hence all the errors. I copy a new file over (as it's a file server / HTPC computer) and that's when errors start to pop up. I've lost 5 or more files. I have multiple damaged files.

My only solution is to export a Vdev Pool. Disconnect all the drives, and use my one and only last remaining motherboard sata connection to manually connect to each drive to run SMART and other tests on.
BUTT. I still have to physically replace this space saving card with a bunch of PCI-E X1 sata cards. Just so the OS can see the HDD's and ZFS can do it's job.

Claydoh. you are correct, I should be posting this stuff up to other locations. More spam for me by signing up

[jglen490]

In terms of the forced upgrade to 22.04, what Kubuntu version were you running before this happened?
Also, did you not have an external, independent backup solution in place?

[claydoh]

It is kinda sorta impossible to be forced to upgrade to a new release of Kubuntu (or most any distro), even if it becomes end-of-life and is no longer supported and no longer receives updates.
You my get a nag to do so -- which is completely configurable, or can disabled, via the Software Sources tool found in Discover's settings section, as well as from Muon Package Manager

There is no way to upgrade to a new release via apt, Discover, or Muon; it is physically impossible to do so without manually changing something yourself (and probably breaking your system), or by using the actual upgrade tools.

[CharlieDaves]

In terms of the forced upgrade to 22.04, what Kubuntu version were you running before this happened?
Also, did you not have an external, independent backup solution in place?

Previous version was 20.04.5
Backup, sort of. I have made one with Clonezilla, but was lost when a hdd decided to not play nice. I thought I had another backup copy, but it's on the external ZFS system that has also stuffed up.

My software updates page.
PS or FYI. I HAD 2 PC's running the same version and kept it that way. I've tried to (there is a forum post somewhere, not interested in searching for it) about this. It didn't work the way I was hoping, and lead to lots of problems on the second PC (clone)

[CharlieDaves]

FYI ALL. NEVER use USB and ZFS together. The USB protocol "clashes" with ZFS's ability to have direct access to the DRIVES.

I've heard back from Broadcom, whom make LSI HBA Cards. I asked about the IT-Mode compatible with ZFS files system. They responded (Just as others have on other forums)
>>> that IT MODE disables the onboard RAID (and I think JBOD??) and makes each PORT or DRIVE directly visible to the OS-System.
.
ZFS to work correctly needs to be able to access each DRIVE directly. THIS WORKED on 20.04.5.
It does NOT work on 22.04.
I am trying to find where I can post a "bug" report about it so it can be fixed.
The problem, is 90% the LSI HBA CARD driver. As listed above in post #1, TXT file, the system has found the card, the system can see the DRIVES, but SMART is disabled as the system isn't using/accessing/supporting the HBA Card correctly.

IN 20.04.5 I had no problems. I could Do a status, and maybe find an error. I would do a scrub (scan all data and check with Parity data). I could transfer all end-of-year files to backup and maybe get an error.... When I began to transfer 2023 end-of-year files, that's when I noticed in status, CRITICAL errors.
I found permanently damaged files
I've found damaged files, not reported as damaged.
I keep 3 copies on three RAIDZ1 Vdev's, one of which is USB (Yeah my bad)
All 3 RAIDZ1 Vdev's have ERRORS. All Errors are recent since the OS permanently decided to upgrade against my personal choice.
If my headaches calm enough I'll export (disconnect but keep RAIDZ1 Info) ZFS systems and wipe and reinstall 20.04.x
Then I'll complain if that crappy pop-up, pop ups. Shooting it will only damage my screen.
>
>
PS. This THREAD was about issues with ZFS, and how to find out what is wrong and a possible fix. PLEASE help me with that.
Can I get kubuntu to download and use the correct HBA driver ?
How do I achieve that?

[claydoh]

As I mentioned, you will need to ask around in a more general Ubuntu and general Linux venues, where you may find more people who actually use ZFS and your hardware. Our tiny corner doesn't seem to have any

AskUbuntu
Ubuntu's forums
Linuxquestions forum and its related Reddit sub

Since Kubuntu =Ubuntu 100% for anything related to this, as well as hardware/kernel.drivers, more general Linuxy places will be your best bet, since ZFS is not common in the desktop arena.

[CharlieDaves]

Something weird. I thought the HBA card was an LSI 2024 8i. I've pulled it out and it's labeled SAS9220-8i.
BUTT
The "System Settings" > About this system > PCI Info shows....
...
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
Subsystem: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at 4000 [size=256]
Memory at 518c0000 (64-bit, non-prefetchable) [size=16K]
Memory at 51480000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 51400000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas

Am I missing something here ?
Is there another command I could use to double check these results?
Maybe I flahsed the wrong IT-Mode version? Which is why the 22.04 is having difficulties with it.

[jglen490]

Not to beat a dead horse, but lessons learned:
- pay attention to EOL, even with LTS.
- if you need a reliability solution better than ext4, maybe use something more common, like BTRFS (there, I said it oshunluvr ).
- if your files are important, have an independent, external backup solution, that is not a part of your normal mounted set of drives; regardless of the normal mounted drive's FS.

And, I know nothing about ZFS.

[CharlieDaves]

BTRFS has been pitched to me, and I'm curious to try it. From what I've read, the Development of it is a little lagging, due to age, and preference to ZFS protocol. IT'S just what I've read.
I used to have a 8x2Tb WD Green HDD Hardware Raid then ZFS RAIDZ2, and all without issues considering those drives where purchased late Naughts and early 'teens. 10-14 Years old and there wasn't any suggestion of issues. There in a "travel box" somewhere.

Not knowing about ZFS, just Wikipedia it.

I am also asking about the HARDWARE issue. Why a LSI 9220-8i CARD is being listed by the OS as a 2008-8i. Is there a difference? Should I be concerned?
AND
That annoying paperclip is back
How do I completely get rid of it. It even POPPED-UP when I booted off of 20.04.6 USB and entered into "Test or Trial" not install.
If EOL is April this year, it's a false and misleading pop-up, and Unbuntu and/or KDE should be ashamed of themselves.

[claydoh]

How do I completely get rid of it.

Did you see my previous instructions above? The setting is in the Software Sources tool, which you can find in Discover's settings, as well as from Muon Package Manager
It is also very easily found on general web searches.

It even POPPED-UP when I booted off of 20.04.6 USB and entered into "Test or Trial" not install

Yes, 20.04 became EOL in April 2023. I will guess that it probably adds too much complexity to the install to have a different upgrade prompt setting on the live image - which is what is copied to the computer the OS install - then to go and change the setting post-install. Or installing a desktop OS after EOL is out of scope, and not worth making the effort and support requirements to change things for a very small number of people and use cases doing this.

But it definitely could be considered a bug, for sure.

If EOL is April this year, it's a false and misleading pop-up, and Unbuntu and/or KDE should be ashamed of themselves.

The popup is correct. Kubuntu 20.04 will not receive any Kubuntu updates. And 22.04 is the ONLY upgrade path. What is misleading here, other than maybe the layout possibly implying that 22.04 is ending its support life?


Luckily, there is a loophole in LTS support - the EOL of three years is only on the GUI bits - ie Plasma packages, for Kubuntu. The underlying OS bits still have the full 5 year lifespan. But I am sure you will be also mad at this long-time thing that has been around as long as there has been LTS.

I am also asking about the HARDWARE issue. Why a LSI 9220-8i CARD is being listed by the OS as a 2008-8i. Is there a difference? Should I be concerned?

Probably not. I imagine that the 2008-8i is the actual hardware chip or controller on the LSI 9220 But I really dunno. I am not into server hardware myself.

[CharlieDaves]

Oh! Set "Show new releases" to Never. Mine, in my image was LTS. I'll correct it
Thanks Claydoh