Kubuntu on proprietary NAS with software RAID and external SSD for system

[Krzysieq]

Hi all,
I'm looking to liberate my plenty good hardware NAS home server from it's proprietary linux-based operating system that is an absolute disgrace to linux, unix and mankind in general. The kit is as follows:

* Intel® Celeron® J1900 quad-core 2.0 GHz processor (burst up to 2.42 GHz) with Intel® HD Graphics for Intel Atom® Processor Z3700 Series clocked at 688 MHz with burst up to 854 MHz as per intel's specifications
* 2x 4gb DDR3L ram
* 2x 4TB HGST NAS drives
* most recent addition - 256 samsung 860 EVO NVMe SSD attached over a USB3.0 wire because of lack of space inside the box/on the motherboard to do otherwise.

I won't quote the manufacturer or model here, but can share on private message if someone's interested.

Question 1) Is this kit enough to run KDE smoothly? It has been enough to run Ubuntu 16.04 in a container running on the proprietary OS relatively painlessly in terms of overall GUI performance...

Out of the factory, the ssd wasn't there, the proprietary system would boot from a 512mb usb flash memory molded on to the motherboard (as if it made any difference that it can't be removed physically knowing that people can still switch boot order in the BIOS). The hdds were setup as mdraid RAID 1 array by that factory OS and that's my main pain point.

So last night I installed Debian 9.8 on that SSD but I'm having a hard time working around the fact that mdraid metadata is on the disks and there's lvm which prohibits the complete erasure and recreation of the RAID 1 array, which I wanted to partition differently as well.

So my thoughts turned to kubuntu, because that's a distro I used a good few years back at work and home, and I'm thinking I'll be able to enjoy the NAS a bit more with a more user-friendly distribution than raw Debian (all credit to Debian team, it's an amazing server distro I had the pleasure to work with years ago).

Question 2) Does Kubuntu 18.04 come with anything that will help me wipe out the hdds to factory state and create a new array? I don't want it bootable, I want the OS to live on the SSD for various reasons (including but not limited to speed and putting the hdds to sleep more easily which saves tons of electricity on this little box). Currently the server is a very much usable Debian machine so I can do any tweaks necessary prior to installing kubuntu afresh. I'm just kinda lost as to where to start and what kubuntu will have on offer for me.

Question 3) Is it possible to prevent KDE Plasma starting up automatically at reboot, but instead have it launched commandline as needed? Most of the time the NAS sits under the TV and works it's magic as DLNA/samba/app server, but occasionally it would be nice to be able to use it as a desktop, but I feel it's a waste of CPU&RAM to keep it running 100% of time.

[oshunluvr]

I won't quote the manufacturer or model here, but can share on private message if someone's interested.

Why not? You own it don't you? You'll get better advice if you do.

Question 1) Is this kit enough to run KDE smoothly? It has been enough to run Ubuntu 16.04 in a container running on the proprietary OS relatively painlessly in terms of overall GUI performance...

Short answer is "yes". I'm going to assume you mean Kubuntu 18.04 since you didn't specify. The longer answer is it will depend on what you ask it to do. Generally, if it's just a file server and used to share media, it'll be fine.

Question 2) Does Kubuntu 18.04 come with anything that will help me wipe out the hdds to factory state and create a new array? I don't want it bootable, I want the OS to live on the SSD for various reasons (including but not limited to speed and putting the hdds to sleep more easily which saves tons of electricity on this little box). Currently the server is a very much usable Debian machine so I can do any tweaks necessary prior to installing kubuntu afresh. I'm just kinda lost as to where to start and what kubuntu will have on offer for me.

If you have access to the drives, any distro will have the tools to reformat the drives. MDADM, if you need it, is not installed by default, but you can install it.

Question 3) Is it possible to prevent KDE Plasma starting up automatically at reboot, but instead have it launched commandline as needed? Most of the time the NAS sits under the TV and works it's magic as DLNA/samba/app server, but occasionally it would be nice to be able to use it as a desktop, but I feel it's a waste of CPU&RAM to keep it running 100% of time.

The answer is yes, but I question why you want a GUI at all if this is a server. What I mean is; What are you going to do with a server that requires a GUI? My server (Ubuntu Server 18.04) has no GUI, runs Plex media service, shares files via NFS and SAMBA, run Transmission (torrent server/client), stores my backups, hosts my family website, runs Nextcloud, and a few other things. It doesn't have any desktop and never has. I update it using the command line via SSH. If you really need some sort of GUI to do stuff on it, install Webmin and use that. No desktop required.

Servers, if they are just servers, are set-it-and-forget-it things. I have set mine up to do automatic backups and I upgrade packages manually from the terminal (remotely) and that's it. Once a month or so, I log in the terminal and clean up old backups, etc., but I'm going to automate that also when I get some time.

Do you really want a monitor and keyboard sitting under your TV so you can access your server? If not, why have a desktop environment?

[Krzysieq]

Thanks for replying back!

Why not? You own it don't you? You'll get better advice if you do.

I do indeed, but I don't really think it's relevant to the question and I don't want to be accused of advocating for or against a particular NAS vendor. Suffices to say their hardware is IMHO very good quality, while their software and - even more importantly - software update policies and habits - are shambles, which is why I'm replacing that itching part of the mix. My only regret is that I haven't researched this earlier, I've had the NAS for over 3 years now, and only recently it pushed me across the line to do that quest for alternatives to their bundled OS.

Short answer is "yes". I'm going to assume you mean Kubuntu 18.04 since you didn't specify. The longer answer is it will depend on what you ask it to do. Generally, if it's just a file server and used to share media, it'll be fine.

My use cases are for storing private files like documents, photos and other media, serving those over network shares and dlna, running a bunch of web applications (JIRA, Bitbucket, Bamboo + any others I develop myself and need to expose to the outside world for whatever reasons) behind a reverse proxy using apache, potentially also small time mail & sftp servers. But because it's usual resting place is under the TV, I might occasionally use it as a PC with a big screen. This is probably bottom of my list of important use cases, but it's there nonetheless.

If you have access to the drives, any distro will have the tools to reformat the drives. MDADM, if you need it, is not installed by default, but you can install it.

The answer is yes, but I question why you want a GUI at all if this is a server. What I mean is; What are you going to do with a server that requires a GUI? My server (Ubuntu Server 18.04) has no GUI, runs Plex media service, shares files via NFS and SAMBA, run Transmission (torrent server/client), stores my backups, hosts my family website, runs Nextcloud, and a few other things. It doesn't have any desktop and never has. I update it using the command line via SSH. If you really need some sort of GUI to do stuff on it, install Webmin and use that. No desktop required.

Servers, if they are just servers, are set-it-and-forget-it things. I have set mine up to do automatic backups and I upgrade packages manually from the terminal (remotely) and that's it. Once a month or so, I log in the terminal and clean up old backups, etc., but I'm going to automate that also when I get some time.

This Webmin thing - I saw it mentioned elsewhere too. Need to give it a shot I guess. I installed mdadm, but I'm a bit scared by the plethora of options that I will brick my hard drives.

Do you really want a monitor and keyboard sitting under your TV so you can access your server? If not, why have a desktop environment?

I don't need that - the NAS has a hdmi wire running to the TV through an audio receiver, all I need is a wireless keyboard with a touchpad, that I already own. Hence my last - however infrequent - use case on the list.

[oshunluvr]

I can't imagine how examining your drives would "brick" them. Try booting up - either to your Debian install or to a live USB stick, install mdadm and run;

sudo mdadm --assemble --scan

That should find all your mdadm RAID arrays and allow you to mount them - assuming of course nothing is wrong with them. I'm also assuming you have all your stuff backed up. If you have access to the original OS drive and they used mdadm, there should be a mdadm.conf file on there somewhere with the details you would need to re-assemble the RAID.

If that makes you nervous, you can see what's on the drives with;

sudo mdadm --examine /dev/sdb1

/dev/sdb1 is just an example. Obviously use whatever partition device name(s) you want to check out instead of /dev/sdb1.

You'll get a report like;

[COLOR=#242729][FONT=Consolas]/dev/sdb1:
[/FONT][/COLOR]          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f331560:fc85feff:5457a8c1:6e047c67 (local to host mserver)
  Creation Time : Sun Feb  1 20:53:39 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Sat Apr 20 13:22:27 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 6c8f71a3 - correct
         Events : 955190

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
   this  1       8       17        1    active sync   /dev/sdb1

   0     0       8       113       0    active sync   /dev/sdh1
   1     1       8       17        1    active sync   /dev/sdb1
   2     2       8       97        2    active sync   /dev/sdg1
   3     3       8       33        3    active sync   /dev/sdc1

As far as the desktop, I get that use-case. That TV will be a big monitor! Too bad the PC would be too weak for some awesome games.

Now, a new topic/suggestion: Assuming you have backed up all your data, why keep the old configuration at all? I dropped using mdadm years ago. It's not problematic, but there are more modern ways to create RAID or JBOD arrays. I've been using BTRFS since it was new and my server is super easy to maintain. My current configuration is 3 hard drives, one 10TB single drive and 2x6TB drives for backups and a small SSD for the operating system. Currently, they all stand alone - no RAID. My previous configuration was 2x6TB drives and 2x2TB drives. Each pair was configured in RAID1, but when I needed to retire the 2TB drives due to age, I bought the huge 10TB drive and moved all the data to it, keeping the 2x6TB drives as backup devices.

Here's the cool part: Converting to the new drive configuration and moving the data was done while the system was running normally and without even rebooting! Hard to believe, isn't it? BTRFS allows you to reconfigure RAID or even remove it on-the-fly without stopping anything - you can continue to use the computer during the reconfiguration. Since my server supports hot-swapping of drives, here's what I did;

I moved all the data off of the 2x2TB drives (there was enough room to use the 2x6TB array to store all the data).
I then unmounted the 2xTB array and remounted the data locations so the files from the 2x2TB array were now accessible on the 2x6TB array.
I removed the 2x2TB drives and inserted the new 10TB drive.
I formatted the 10TB drive to BTRFS (all the drives are BTRFS) and then copied all the data from the 2x6TB array (about 5.7TB) to the 10TB drive.
I remounted all the data folders on the new drive, thus leaving the 2x6TB array unused by the system, but still intact.
I then reconfigured the 2x6TB array into single disks resulting in all the data on a single 6TB drive and the second one empty.
I divided all the data folders (documents, pictures, movies, music, downloads, etc.) so that roughly half remained on one 6TB drive and the rest on the other 6TB.

This left me where I am now; All the data on a single 10TB drive with half backed up on each of the 6TB drives. I decided to not use RAID for the backup drives because I needed the whole 12TB of space to act as backup. That meant I would have to use RAID0 so why bother with RAID at all? In the future, when the 6TB drives need to be replaced, I'll add a second 10TB drive and convert back to RAID1.

[Krzysieq]

Clearly You've got way more exp on the matter than I do Are You using LVM? I'm trying to make some sense out of all the articles I'm reading, but they all seem to be covering use cases far more complicated than mine. As far as I read, BTRFS doesn't support encryption natively, so I think I'll pass for the moment. I think I'll go with the simplest thing I can imagine that gets the job done (RAID1 on whole disks -> encrypt with LUKS -> LVM -> partitions). If I find any problems with this setup, then I will worry, but currently I can't think of any TBH. My only gripe is I won't be able to do what I thought I would - use one of the disks from the old array to make a fresh filesystem, then use the other to copy the data, and create a new array, setup encryption and LVM. So I'm in for a lot of copying from usb sticks and other crappy media where I spread out all my data prior to doing anything. Most of that stuff is USB2.0 so that will take some time... Guess I can live with that though - if anything that will make me invest into more modern backup solutions

As far as the desktop, I get that use-case. That TV will be a big monitor! Too bad the PC would be too weak for some awesome games.

Not huge, just an old-school 42" Panasonic plasma, but good enough to watch things. For gaming I've got PS4, I ditched the idea of having a PC for gaming years ago when the graphics card had to be replaced every other year just to play the next iteration of FIFA

[oshunluvr]

Yes, you're correct BTRFS doesn't support full disk encryption, but it does do encryption:

Does btrfs support encryption?

There are several different ways in which a filesystem can interoperate with encryption to keep your data secure:

• It can operate on top of an encrypted partition (dm-crypt / LUKS) scheme.
• It can be used as a component of a stacked approach (eg. ecryptfs) where a layer above the filesystem transparently provides the encryption.
• It can natively attempt to encrypt file data and associated information such as the file name.

There are advantages and disadvantages to each method, and care should be taken to make sure that the encryption protects against the right threat. In some situations, more than one approach may be needed.
Typically, partition (or entire disk) encryption is used to protect data in case a computer is stolen, a hard disk has to be returned to the manufacturer if it fails under warranty or due to the difficulty of erasing modern drives. Modern hard disks and SSDs transparently remap logical sectors to physical sectors for various reasons, including bad sectors and wear leveling. This means sensitive data can remain on a sector even after an attempt to erase the whole disk. The only protection against these challenges is full disk encryption from the moment you buy the disk. Full disk encryption requires a password for the computer to boot, but the system operates normally after that. All data (except the boot loader and kernel) is encrypted. Btrfs works safely with partition encryption (luks/dm-crypt) since Linux 3.2. Earlier kernels will start up in this mode, but are known to be unsafe and may corrupt due to problems with dm-crypt write barrier support.
Partition encryption does not protect data accessed by a running system -- after boot, a user sees the computer normally, without having to enter extra passwords. There may also be some performance impact since all IO must be encrypted, not just important files. For this reason, it's often preferable to encrypt individual files or folders, so that important files can't be accessed without the right password while the system is online. If the computer might also be stolen, it may be preferable to use partition encryption as well as file encryption.
Btrfs does not support native file encryption (yet), and there's nobody actively working on it. It could conceivably be added in the future.
As an alternative, it is possible to use a stacked filesystem (eg. ecryptfs) with btrfs. In this mode, the stacked encryption layer is mounted over a portion of a btrfs volume and transparently applies the security before the data is sent to btrfs. Another similar option is to use the fuse-based filesystem encfs as a encrypting layer on top of btrfs.
Note that a stacked encryption layer (especially using fuse) may be slow, and because the encryption happens before btrfs sees the data, btrfs compression won't save space (encrypted data is too scrambled). From the point of view of btrfs, the user is just writing files full of noise.
Also keep in mind that if you use partition level encryption and btrfs RAID on top of multiple encrypted partitions, the partition encryption will have to individually encrypt each copy. This may result in somewhat reduced performance compared to a traditional RAID setup where the encryption might be done on top of RAID. Whether the encryption has a significant impact depends on the workload, and note that many newer CPUs have hardware encryption support.
Does Btrfs work on top of dm-crypt?

This is deemed safe since 3.2 kernels. Corruption has been reported before that, so you want a recent kernel. The reason was improper passing of device barriers that are a requirement of the filesystem to guarantee consistency.
If you are trying to mount a btrfs filesystem based on multiple dm-crypted devices, you can see an example script on Marc's btrfs blog: start-btrfs-dmcrypt.

I can't speak to it as I don't use encryption.