Large file issue causes system instability and slow down.

[shag00]

Operating System: Kubuntu 20.10
KDE Plasma Version: 5.19.5
KDE Frameworks Version: 5.74.0
Qt Version: 5.14.2
Kernel Version: 5.8.0-25-generic
OS Type: 64-bit
Processors: 16 × AMD Ryzen 7 3700X 8-Core Processor
Memory: 15.6 GiB of RAM
Graphics Processor: GeForce GTX 1080/PCIe/SSE2
Motherboard: MSI MEG X570 ACE

I have this and another PC and another which have both suffered mysterious issues since 20.04 which range from system slowdown to application freezing.
I did some more tests on large files and discovered that there is indeed some problem with Kubuntu that is directly affected by what size file it is working on. For the tests I used MKVtoolnix GUI Ver 50 once on Kubuntu 20.04 against the same files and MKVtoolnix on Windows 10 using the same hardware, including disks. The test involved re-muxing a video file from an m2ts container to an mkv container, therefore the data of the movie was not changed. File size is output size. This is repeatable.

File ...Time
Size ...Windows ...Kubuntu
1GB ...11 secs ...7 secs (45 min TV show)
6.2GBs ...2 mins 23 secs ...2 mins 30 secs (typical DVD)
18.8GBs ...6 mins 43 secs ...38 mins 45 secs (HD Blu Ray)
43.4GBs ...14 mins ...172 mins (UHD Blu Ray)

As would be expected the time increase for Windows is fairly linear being dependent on file size whereas Kubuntu needs fixing.

The above are onKubuntu 20.04 and I redid the tests on 20.10 and came up with very similar results except that if the test was conducted within the home directory (43.4 GB file) (both read and write) the time taken was 4 mins 25 secs which is as expected.

Does anybody work with large files, say 50GB+ and have experienced these issue or have a solution?

[oshunluvr]

I don't see where you talk about your file system, so I have to assume you're using plain vanilla EXT4. From your above times, your small file performance is better than NTFS and your large file performance is worse.

If you want a truer test, try disabling drive caching on NTFS. Also, assuming WIndows and Linux are on the same PC, try your test on the NTFS partition from Linux.

EXT4 has a lot of tune-able functions that may improve it's performance in your case. For example, "noatime" can increase multiple files access times by 30%. Or maybe switching to a file system more suitable to large files like XFS is the way to go.

[oshunluvr]

Also, this: https://www.phoronix.com/scan.php?pa...-Commit-Queued

[oshunluvr]

See the answer to this question: https://unix.stackexchange.com/quest...o-data-ordered

[shag00]

To give some further information: my system is indeed dual boot and of the 5 disks only the Kubuntu home drive is ext 4, the other 4 disks are NTFS so as to allow maximum interoperability between the different OSs. Each OS is installed on a separate disk. The 4-size test on 20.04 used an NTFS disk and I have edited my initial post to indicate that the 4 min 25 sec result on 20.10 was for a 43GB size file. This means that ext 4 is not the issue.

I was under the impression drive caching improved speed so can you confirm to disable it please as a new series of tests takes many hours. Additionally, I have reverted to mainly using Windows now as 20.04/20.10 are such a pain in the arse to use and I cannot do the things I need to do.

Perhaps the most important thing to note was that these issues were not present to anywhere near the extent they are now in 19.10.

Changing the FS to XFS is not an option as Windows cannot read it and as is shown currently I need Windows to be able to read my working data for when there are problems with Kubuntu.

[oshunluvr]

If you're using a Windows file system with Linux I wouldn't expect the same performance. It's the trade off you've chosen to keep using Windows. Basically, NTFS is native to Windows but not to Linux.

However, there are some things you can do to improve NTFS performance when using Linux: https://www.tuxera.com/community/ntfs-3g-faq/#highcpu

[shag00]

I think you may be missing the point. If I were talking about a 20 or 30% difference in performance then tweaks might be in order but I am talking about a difference well in excess of 10 times which is clearly not a tweaking issue especially as it only occurred with the introduction of 20.04. If it were just a read/write problem I would expect linear growth in the time not exponential as is experienced.