Kubuntu UBSAN array-index-out-of-bounds error bricked my system (update to kernel 6.5)

[sodgian]

Hi everyone,

On Wednesday January 10th, 2024 I upgraded through KDE Discover my current 22.04 LTS release to kernel 6.5 (suggested update by KDE Discover).

The upgrade bricked my system.

I got this error while booting:

UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-q7NZ0T/linux-hwe-6.5-6.5.0/include/linux/ieee80211.h:4304:28
index 9 is out of range for type 'u8 [1]'



I was able to start the system pressing several times the SHIFT key after the BIOS boot logo, selecting "Advanced options for Kubuntu" in the GRUB menu and then "Kubuntu, with Linux 6.2.0-39-generic".

Then, after a lot of research, I found the problematic update in /var/log/apt/history.log:

Start-Date: 2024-01-10 19:02:51
Commandline: packagekit role='update-packages'
Requested-By: goodkubuntuman (1000)
Install: linux-modules-extra-6.5.0-14-generic:amd64 (6.5.0-14.14~22.04.1, automatic), linux-hwe-6.5-headers-6.5.0-14:amd64 (6.5.0-14.14~22.04.1, automatic), linux-modules-6.5.0-14-generic:amd64 (6.5.0-14.14~22.04.1, automatic), linux-image-6.5.0-14-generic:amd64 (6.5.0-14.14~22.04.1, automatic), linux-headers-6.5.0-14-generic:amd64 (6.5.0-14.14~22.04.1, automatic)
Upgrade: linux-image-generic-hwe-22.04:amd64 (6.2.0.39.40~22.04.16, 6.5.0.14.14~22.04.7), libc6:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), libc6:i386 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), locales:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), linux-headers-generic-hwe-22.04:amd64 (6.2.0.39.40~22.04.16, 6.5.0.14.14~22.04.7), linux-generic-hwe-22.04:amd64 (6.2.0.39.40~22.04.16, 6.5.0.14.14~22.04.7), libc-dev-bin:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), libc-bin:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), libc-devtools:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), libc6-dbg:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6), libc6-dev:amd64 (2.35-0ubuntu3.5, 2.35-0ubuntu3.6)
End-Date: 2024-01-10 19:04:49

And a way to revert the upgrade using these commands:

sudo apt remove --purge linux-modules-6.5.0-14-generic linux-image-6.5.0-14-generic linux-headers-6.5.0-14-generic linux-modules-extra-6.5.0-14-generic

sudo apt remove --purge linux-image-generic-hwe-22.04 linux-headers-generic-hwe-22.04 linux-generic-hwe-22.04

sudo apt update

Then I rebooted the computer and everything seems fine again (with kernel 6.2).
Needless to say, I have lost total confidence in Kubuntu (probably I will be stuck forerver in kernel 6.2, no confidence in updating my system to newer kernel releases).

I have posted this message here to help other people with the same problem.

Regards

[oshunluvr]

Several of us have had issues of one form or another with kernel 6.5 and LOTS of people have issues with nvidia and it's CLOSED SOURCE drivers.

I'm not sure how that's Kubuntus fault since the Kubuntu team are not responsible for either of these. Did you need the 6.5 kernel for something? Lots of folks have given up on nvidia and all the breakages they cause. My AMD card works great.

[sodgian]

How do you know that the cause of the problem is the NVIDIA driver?

[claydoh]

How do you know that the cause of the problem is the NVIDIA driver?

Because the vast majority, if not all the people having this problem seem to have nvidia cards, using the proprietary drivers. It is not uncommon historically speaking, for the necessary driver shim between the kernel and the closed source code to fail to build for the new kernel, for example. No idea if this is the case here or not. But bugs like this are not at all unusual with these driver in Linux.

And to second oshunluvr here, Kubuntu isn't a separate distro at all. Kubuntu is the Ubuntu team that packages and maintains Plasma in Ubuntu. They do not handle things with drivers and kernels.

[oshunluvr]

How do you know that the cause of the problem is the NVIDIA driver?

Maybe because the first website in a google search of the error is actually ON nvidia's forum website?

https://forums.developer.nvidia.com/...kernels/271705

[Haggis]

I have an AMD Vega 56 and also can't use this kernel, it boots with the above error but any use of the desktop causes it to freeze and revert to the log in screen. Using kernel 6.2.0-39 in the meantime.

[sodgian]

What has to do IEEE 802.11 with NVIDIA? They do not appear to have any relation at all. By the way, the NVIDIA driver I am using is version 390.157, which is quite old (release date: November 22th, 2022).

I do not know whose fault is it but it is unacceptable that an upgrade breaks your system. I almost had a heart attack when my system did not boot; I could have lost a ton of work. I am pretty sure that this bug is affecting thousands of users worldwide (and a lot of them do not have the necessary knowledge to solve this issue). Why the update to kernel 6.5 is suggested and shown in KDE Discover? Not even a simple warning message alerting that it can be a risky upgrade!

I have lost total confidence in Linux.

[Snowhog]

I almost had a heart attack when my system did not boot; I could have lost a ton of work.

You don't use/have a backup strategy in place?

[dejjanku]

As stated previously, too frequent updates are breaking systems...

[jglen490]

As stated previously, too frequent updates are breaking systems...

Sorry, but the frequency of upgrades does not break systems. However, the quality of the updated drivers and other software can break a system.

My laptop is on kernel 6.5, I've installed every update since Day 1, zero problems. The BIG difference, my laptop does not have Nvidia graphics.

Quite often - probably not always - a user will just install some latest Nvidia driver just because it's the latest and end up with a brick. Use the recommended Linux driver for each specific Nvidia card and what Kubuntu driver manager recommends.

[oshunluvr]

Well said...

[jlittle]

The upgrade bricked my system.

BTW, IMO this is sort of an exaggeration. If you "brick" your system, it turns into a brick, a solid, inert lump, and does not turn on or do anything at all.

I think you meant you broke your system.

[oshunluvr]

LOL! "bricking" only really happens to cell phones

[dejjanku]

Sorry, but the frequency of upgrades does not break systems. However, the quality of the updated drivers and other software can break a system.

My laptop is on kernel 6.5, I've installed every update since Day 1, zero problems. The BIG difference, my laptop does not have Nvidia graphics.

Frequent, bad and unnecessary updates break systems that worked previously just fine. For old hardware it does not make sense to upgrade kernel every week, as long as it works.
That being said, 6.5 kernel is way better for my laptop as well. Kernel 5.x series had frequent issues last year, working on and off, intermittently during the year, dependent on sub version or what ever.
Also no problem with Nvidia on two laptops (one older and one newer, older i5 3rd gen even better with Fedora than K/Ubuntu). Both seems to be working at lower temperature and no unnecessarily CPU ramp up at idle with 6.5 kernel.

So, lets upgrade as frequently as possible, as long as it works...

[claydoh]

For old hardware it does not make sense to upgrade kernel every week, as long as it works.

Those updates are not new kernel versions, mostly. The version *upgrades* do come regularly, approx every 6 months on LTS, after their first year, for a period of time.
The weekly updates are not new versions at all, but security patches for the the existing kernel version(s), as are the vast majority of all the other updates one sees in an LTS release. This has been an ongoing feature of Ubuntu LTS for at least 12 years now, and enabled by default on desktop installs since 16.04.

One can revert to the original LTS kernel and never see full upgrades again, if they wish to do so. Basically, one removes installs linux-generic, uninstalls linux-generic-hwe-22.04 and then removes any linux-image and linux-header packages for anything but a 5.15 kernel.

[dejjanku]

I bet that --purge will remove break something. See you in a few days...

edit: no, purged 5* kernels just fine
5* kernels were reporting some errors while booting and refusing to shut down sometimes. I feel relief!

[claydoh]

I bet that --purge will remove break something. See you in a few days...

No, not really. the purge option just removes any system level configuration files specific to an application or something that would normally have been left behind. Here is a good example.