Why Systemd?

[GreyGeek]

I am VERY irritated to have to abandon Xorg for Wayland or some different Xmanagers!

My apologies! I've been using Linux since May of 1998 and your mention of Nutyx is the first I'd ever heard of it, even though I find it has been around since 2007. Probably because of its LFS base it doesn't draw much of a crowd. How many apps are in its repositories?

BTW, while I first dislike systemd a lot, I've come to appreciate its approach. The sooner Linux puts all of its controls into a single location using a single tool the better it will be, IMO. Or, adopt something like SNAP, where every app is complete within itself, carrying all the libraries and configs it needs, and is easily removed by deleting the directory it was placed in when it was installed.

[calligrafan]

Hi GreyGeek

However, when I began searching for a distro that did not feature Systemd or planned to go to it in a future release, I realized that there were NO other choices for me since I did not want to use Slackware or Gentoo.

I find it is a wrong information.

Regarding SystemD, you can get a systemD free pre compiled version of LinuxFromScratch distributed as binaries at nutyx.org !

More: Nutyx is not only precompiled like other popular distributions like Kubuntu. And Nutyx has predilection for KDE (the other environments were not all the time disponible, but KDE yes! Since years ago...). More, Nutyx allows both and make them easy: install a base system with binaries, continue to install from binaries from Nutyx, or compile yourself more as you want very easy as Nutyx has the vision and is proud to be after installation a fully normal LFS system with on top a packages management for the transactions with the Nutyx own binaries (made from LFS sources). Nutyx offers both: with and without systemD!

I am not irritated by systemD as long the possibility to install without is preserved.

I am VERY irritated to have to abandon Xorg for Wayland or some different Xmanagers!

[kubicle]

The others are not present. Signs marking future development of systemd?

The confdirs (*.conf.d) don't necessarily exist by default, and are reserved for the local administrator's own config snippet files (the confdirs under /etc) and for software packages (the confdirs under /lib or /usr/lib) in case a software package needs custom systemd config to function. Using confdirs, packages (or the local admin) can store their own config snippet files without the need to edit the main config file.

The conffiles in the confdirs will be read if they exist. otherwise they are obviously skipped

For example, I have made this little conffile for the systemd-logind (/etc/systemd/logind.conf.d/70-my-logind.conf):

[Login]
NAutoVTs=2
ReserveVT=2

I could have put the config in /etc/systemd/logind.conf, but since I made my own conffile in the confdir, systemd package upgrades won't prompt me for the possible changes in the main config file (it's also easier to keep track of config changes you've made this way).

[GreyGeek]

Is that you, Lennart?

I had already link the two sources you listed but haven't throughly studied them yet.

The SYNOPSIS you quoted, listing the source directories and config files for systemd I found interesting.Of all the paths listed, only two are on my system:

/etc/systemd/system.conf
/etc/systemd/user.conf

The others are not present. Signs marking future development of systemd?

[kubicle]

Table 2. Load path when running in user mode (--user)

This relates to (only) user systemd instances. Systemd commonly runs several instances, the system instance (pid 1) and one instance for every user (user processes). the system instance is responsible for starting up the system services necessary to run the system, and user instances can be used to start user services/units (things that run with the user's UID).

The system instance only reads system wide configuration, while user instances read both system wide (common to all users) and user specific configuration. You can check the system wide config file locations (for both system and user instances) and their precedence in "man systemd-system.conf"

/etc/systemd/system.conf, /etc/systemd/system.conf.d/*.conf, /run/systemd/system.conf.d/*.conf, /lib/systemd/system.conf.d/*.conf
/etc/systemd/user.conf, /etc/systemd/user.conf.d/*.conf, /run/systemd/user.conf.d/*.conf, /usr/lib/systemd/user.conf.d/*.conf

Like you posted, user instances also read user specific (and user controlled) configuration in user's $HOME according to the $XDG spec (variables like $XDG_CONFIG_HOME, and $XDG_DATA_HOME are only set if not using the "standard" default dirs $HOME/.config and $HOME/.local/share).

Of course, systemd configuration is not just the main config files, but also the targets (analogous to "old" runlevels) units (used to start the actual services, similar to old initscripts) and the sockets (used for inter-process communication)...and of course the different systemd child processes like timesyncd, resolved etc. have their own config files.

It will look like overkill at first but the "numerous" config paths of course give a much finer grained control of configuration of systemd.

If you aren't already familiar with them, the archwiki has some nice information on systemd basics:
https://wiki.archlinux.org/index.php/Systemd
https://wiki.archlinux.org/index.php/Systemd/User

[GreyGeek]

......
What kind of "inittab" edits where you looking for, similar configuration options can usually be made in systemd configuration as well (usually in config files in /etc/systemd/)

I wasn't looking for any inittab edits. When I was heavy into setting up Linux turnkey systems I used /etc/skel, /etc/inittab, fstab and mtab a lot for configuration, along with several kernel compiles to get things just right. I just happened to noticed that those files are either missing or have changed considerably while exploring how systemd works.

You mentioned /etc/systemd as a location for making configuration options. The "SystemdForUpstartUsers", which says nothing about "Load Path"s when discussion various locations of systemd conf files, links to a web page which does:


Table 2. Load path when running in user mode (--user).

$XDG_CONFIG_HOME/systemd/user	User configuration (only used when $XDG_CONFIG_HOME is set)
$HOME/.config/systemd/user	User configuration (only used when $XDG_CONFIG_HOME is not set)
/etc/systemd/user	Local configuration
$XDG_RUNTIME_DIR/systemd/user	Runtime units (only used when $XDG_RUNTIME_DIR is set)
/run/systemd/user	Runtime units
$XDG_DATA_HOME/systemd/user	Units of packages that have been installed in the home directory (only used when $XDG_DATA_HOME is set)
$HOME/.local/share/systemd/user	Units of packages that have been installed in the home directory (only used when $XDG_DATA_HOME is not set)
/usr/lib/systemd/user	Units of packages that have been installed system-wide

Apparently there are also other locations where conf files can be located:
"Additional units might be loaded into systemd ("linked") from directories not on the unit load path. See the link command for systemctl(1). Also, some units are dynamically created via a systemd.generator(7)."

On my system I noticed that $XDG_DATA_HOME is not defined:

:~$ env | grep XDG
XDG_VTNR=7
XDG_SESSION_ID=2
XDG_SESSION_CLASS=user
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session1
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_CONFIG_DIRS=/etc/xdg/xdg-/usr/share/xsessions/plasma:/etc/xdg:/usr/share/kubuntu-default-settings/kf5-settings
XDG_SESSION_TYPE=x11
XDG_SEAT=seat0
XDG_SESSION_DESKTOP=KDE
XDG_DATA_DIRS=/usr/share//usr/share/xsessions/plasma:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
[B]XDG_RUNTIME_DIR[/B]=/run/user/1000
XDG_CURRENT_DESKTOP=KDE

but XDG_RUNTIME_DIR is set. However, there are no conf files or a "user" subdirectory in /run/user/1000/systemd/. Just a couple of sockets. So, if the place to put user created conf files is at
$HOME/.config/systemd/user
then I would have to create the systemd and user subdirectories first. Putting them in /etc/systemd would require root permission. Have I figured it out right?

[kubicle]

Has that always been the case? I've read tons on how Linux worked over the years and do not recall reading that before.

The difference comesdown to BSD or GNU syntax, as mentioned on the ps man page:

The use of BSD-style options will add process state (stat=STAT) to the default display and show the command args (args=COMMAND) instead of the executable name.

if use "ps aux" (bsd options), it will default to bsd syntax...you can compare the output to "ps -ely", for example (standard GNU syntax).

I noticed that systemd does not use /etc/inittab

It doesn't (neither did upstart, by the way...IIRC).

What kind of "inittab" edits where you looking for, similar configuration options can usually be made in systemd configuration as well (usually in config files in /etc/systemd/)

[GreyGeek]

Systemd Concepts

The basics of systemd is given in the man files under the section titled "Concepts"
Note: This man page extract mentions 12 differrent "unit" types, but only lists 11. The one not listed is "snapshots".

"CONCEPTS
systemd provides a dependency system between various entities called "units" of 12 different types. Units encapsulate various objects that are relevant for system boot-up and maintenance. The majority of units are configured in unit configuration files, whose syntax and basic set of options is described in systemd.unit(5), however some are created automatically from other configuration, dynamically from system state or programmatically at runtime. Units may be "active" (meaning started, bound, plugged in, ..., depending on the unit type, see below), or "inactive" (meaning stopped, unbound, unplugged, ...), as well as in the process of being activated or deactivated, i.e. between the two states (these states are called "activating", "deactivating"). A special "failed" state is available as well, which is very similar to "inactive" and is entered when the service failed in some way (process returned error code on exit, or crashed, or an operation timed out). If this state is entered, the cause will be logged, for later reference. Note that the various unit types may have a number of additional substates, which are mapped to the five generalized unit states described here.

The following unit types are available:

1. Service units, which start and control daemons and the processes they consist of. For details, see systemd.service(5).

2. Socket units, which encapsulate local IPC or network sockets in the system, useful for socket-based activation. For details about socket units, see systemd.socket(5), for details on socket-based activation and other forms of activation, see daemon(7).

3. Target units are useful to group units, or provide well-known synchronization points during boot-up, see systemd.target(5).

4. Device units expose kernel devices in systemd and may be used to implement device-based activation. For details, see systemd.device(5).

5. Mount units control mount points in the file system, for details see systemd.mount(5).

6. Automount units provide automount capabilities, for on-demand mounting of file systems as well as parallelized boot-up. See systemd.automount(5).

7. Timer units are useful for triggering activation of other units based on timers. You may find details in systemd.timer(5).

8. Swap units are very similar to mount units and encapsulate memory swap partitions or files of the operating system. They are described in systemd.swap(5).

9. Path units may be used to activate other services when file system objects change or are modified. See systemd.path(5).

10. Slice units may be used to group units which manage system processes (such as service and scope units) in a hierarchical tree for resource management purposes. See systemd.slice(5).

11. Scope units are similar to service units, but manage foreign processes instead of starting them as well. See systemd.scope(5).

Units are named as their configuration files. Some units have special semantics. A detailed list is available in systemd.special(7).

systemd knows various kinds of dependencies, including positive and negative requirement dependencies (i.e. Requires= and Conflicts=) as well as ordering dependencies (After= and Before=). NB: ordering and requirement dependencies are orthogonal.

If only a requirement dependency exists between two units (e.g. foo.service requires bar.service), but no ordering dependency (e.g. foo.service after bar.service) and both are requested to start, they will be started in parallel. It is a common pattern that both requirement and ordering dependencies are placed between two units. Also note that the majority of dependencies are implicitly created and maintained by systemd. In most cases, it should be unnecessary to declare additional dependencies manually, however it is possible to do this.

Application programs and units (via dependencies) may request state changes of units. In systemd, these requests are encapsulated as 'jobs' and maintained in a job queue. Jobs may succeed or can fail, their execution is ordered based on the ordering dependencies of the units they have been scheduled for.

On boot systemd activates the target unit default.target whose job is to activate on-boot services and other on-boot units by pulling them in via dependencies. Usually, the unit name is just an alias (symlink) for either graphical.target (for fully-featured boots into the UI) or multi-user.target (for limited console-only boots for use in embedded or server environments, or similar; a subset of graphical.target). However, it is at the discretion of the administrator to configure it as an alias to any other target unit. See systemd.special(7) for details about these target units.

Processes systemd spawns are placed in individual Linux control groups named after the unit which they belong to in the private systemd hierarchy. (see cgroups.txt[1] for more information about control groups, or short "cgroups"). systemd uses this to effectively keep track of processes. Control group information is maintained in the kernel, and is accessible via the file system hierarchy (beneath /sys/fs/cgroup/systemd/), or in tools such as systemd-cgls(1) or ps(1) (ps xawf -eo pid,user,cgroup,args is particularly useful to list all processes and the systemd units they belong to.).

systemd is compatible with the SysV init system to a large degree: SysV init scripts are supported and simply read as an alternative (though limited) configuration file format. The SysV /dev/initctl interface is provided, and compatibility implementations of the various SysV client tools are available. In addition to that, various established Unix functionality such as /etc/fstab or the utmp database are supported.

systemd has a minimal transaction system: if a unit is requested to start up or shut down it will add it and all its dependencies to a temporary transaction. Then, it will verify if the transaction is consistent (i.e. whether the ordering of all units is cycle-free). If it is not, systemd will try to fix it up, and removes non-essential jobs from the transaction that might remove the loop. Also, systemd tries to suppress non-essential jobs in the transaction that would stop a running service. Finally it is checked whether the jobs of the transaction contradict jobs that have already been queued, and optionally the transaction is aborted then. If all worked out and the transaction is consistent and minimized in its impact it is merged with all already outstanding jobs and added to the run queue. Effectively this means that before executing a requested operation, systemd will verify that it makes sense, fixing it if possible, and only failing if it really cannot work.

Systemd contains native implementations of various tasks that need to be executed as part of the boot process. For example, it sets the hostname or configures the loopback network device. It also sets up and mounts various API file systems, such as /sys or /proc.

For more information about the concepts and ideas behind systemd, please refer to the Original Design Document[2].

Note that some but not all interfaces provided by systemd are covered by the Interface Stability Promise[3].

Units may be generated dynamically at boot and system manager reload time, for example based on other configuration files or parameters passed on the kernel command line. For details, see systemd.generator(7).

Systems which invoke systemd in a container or initrd environment should implement the Container Interface[4] or initrd Interface[5] specifications, respectively."

[GreyGeek]

Back to my "blog".

When I first started using Linux (RH 5.0 in May of 1998) the file /etc/inittab (initial table) held all the goodies. Things have changed. A comparison between sysvinit, upstart and systemd is here.

Systemd has evolved considerably since 2010. To determine which version your Kubuntu installation is currently running run:
systemd --version
and the result for my current state of Kubuntu 16.04 fully updated as of 10/13/2016 is:

:~$ systemd --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

When you are researching systemd make sure the documentation you are consulting is about version 229, or what ever version is on your box. A place to start is the onboard systemd 229 change log, which I located at /usr/share/doc/systemd/changelog.Debian.gz by running locate changelog.

systemd (229-4ubuntu11) xenial; urgency=medium

* 73-usb-net-by-mac.rules: Split kernel command line import line.
Reportedly this makes the rule actually work on some platforms. Thanks
Alp Toker! (LP: #1593379)
* fsckd: Do not exit on idle timeout if there are still clients connected
(Closes: #788050, LP: #1547844)
* libnss-*.prerm: Remove possible [key=value] options from NSS modules as
well. (LP: #1625584)
* Backport networkd 231. Compared to 229 this has a lot of fixes, some of
which we need for good netplan support. Backporting them individually
would be a lot more work and a lot less robust, and we did not use/support
networkd in 16.04 so far. Drop the other network related patches as they
are included in this backport now. (LP: #1627641)
* debian/tests/networkd: Re-enable the the DHCPv6 tests. The DHCPv6
behaviour is fixed with the above backport now.
* pid1: process zero-length notification messages again. Just remove the
assertion, the "n" value was not used anyway. This fixes a local DoS due
to unprocessed/unclosed fds which got introduced by the previous fix.
(LP: #1628687)
* pid1: Robustify manager_dispatch_notify_fd(). If
manager_dispatch_notify_fd() fails and returns an error then the handling
of service notifications will be disabled entirely leading to a
compromised system. (side issue of LP: #1628687)

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 04 Oct 2016 21:43:04 +0200

systemd (229-4ubuntu10) xenial; urgency=medium

* SECURITY UPDATE: zero-length notify message triggers abort/denial of
service
- systemd-dont_assert_on_zero_length_message-lp1628687.patch: change
assert to simple return + log (LP: #1628687)
- Thanks to Jorge Niedbalski <jorge.niedbalski@canonical.com> for
the patch.

-- Steve Beattie <sbeattie@ubuntu.com> Wed, 28 Sep 2016 14:21:42 -0700

systemd (229-4ubuntu8) xenial-proposed; urgency=medium

* Queue loading transient units after setting their properties. Fixes
starting VMs with libvirt. (LP: #1529079)
* Connect pid1's stdin/out/err fds to /dev/null also for containers. This
fixes generators which expect a valid stdout/err fd in some container
technologies. (LP: #1608953)
* 73-usb-net-by-mac.rules: Do not run readlink for *every* uevent, and
merely check if /etc/udev/rules.d/80-net-setup-link.rules exists.
A common way to disable an udev rule is to just "touch" it in
/etc/udev/rule.d/ (i. e. empty file), and if the rule is customized we
cannot really predict anyway if the user wants MAC-based USB net names or
not. (LP: #1615021)
* systemd-networkd-resolvconf-update.service: Also pick up DNS servers from
individual link leases, as they sometimes don't appear in the global
ifstate. (LP: #1620559)

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 06 Sep 2016 14:16:29 +0200

systemd (229-4ubuntu7) xenial-proposed; urgency=medium

* Add pre-dependency to dpkg >= 1.17.14 on udev, to ensure that
dpkg-maintscript-helper symlink_to_dir is available. (LP: #1585406)
* Add activation rate limiting for socket units. (LP: #1568094)
* Split out udev rule to name USB network interfaces by MAC address into
73-usb-net-by-mac.rules, so that it's easier to disable. (Closes: #824025)
* 73-usb-net-by-mac.rules: Disable when net.ifnames=0 is specified on the
kernel command line or if /etc/udev/rules.d/80-net-setup-link.rules is a
symlink to /dev/null, to be consistent with disabling the *.link files and
the documented way to disable ifnames. (Closes: #824491, LP: #1593379)
* coredump: Fix "Coredump file descriptor missing". (LP: #1602256)

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 12 Jul 2016 17:37:25 +0200

systemd (229-4ubuntu6) xenial-proposed; urgency=medium

* Add a DHCP exit hook for pushing received NTP servers into timesyncd.
(LP: #1578663)
* Revert "enable TasksMax= for all services by default, and set it to 512".
Introducing a default limit on number of threads broke a lot of software
which regularly needs more, such as MySQL and RabbitMQ, or services that
spawn off an indefinite number of subtasks that are not in a scope, like
LXC or cron. 512 is way too much for most "simple" services, and it's way
too little for the ones mentioned above. Effective (and much stricter)
limits should instead be put into units individually.
(Closes: #823530, LP: #1578080)
* debian/gbp.conf: Switch to ubuntu-xenial branch.

-- Martin Pitt <martin.pitt@ubuntu.com> Thu, 12 May 2016 10:39:30 +0200

systemd (229-4ubuntu5) xenial-proposed; urgency=medium

* debian/udev.postinst: Don't fail if /var/log/udev is a directory.
(LP: #1574004)
* Set MAC based name for USB network interfaces only for universally
administered (i. e. stable) MACs, not for locally administered (i. e.
randomly generated) ones. Drop /lib/systemd/network/90-mac-for-usb.link
(as link files don't currently support globs for MACAddress=) and replace
with an udev rule in /lib/udev/rules.d/73-special-net-names.rules.
(Closes: #812575, LP: #1574483)
* debian/extra/init-functions.d/40-systemd: Invoke status command with
--no-pager, to avoid blocking scripts that call an init.d script with
"status" with an unexpected pager process. (Closes: #765175, LP: #1576409)
* Add debian/extra/rules/70-debian-uaccess.rules: Make FIDO U2F dongles
accessible to the user session. This avoids having to install libu2f-host0
(which isn't discoverable at all) to make those devices work.
(LP: #1387908)
* On shutdown, unmount /tmp before disabling swap. (Closes: #788303)

-- Martin Pitt <martin.pitt@ubuntu.com> Mon, 02 May 2016 15:04:42 -0500

systemd (229-4ubuntu4) xenial; urgency=medium

* 73-special-net-names.rules: Further refine ibmveth naming.

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 12 Apr 2016 12:06:30 +0200

systemd (229-4ubuntu3) xenial; urgency=medium

* debian/tests/cmdline-upstart-boot: In test_rsyslog(), check for messages
from dbus instead of NetworkManager. NM 1.2 does not seem to log to syslog
by default any more.
* 73-special-net-names.rules: Refine ibmveth naming.

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 12 Apr 2016 10:43:34 +0200

systemd (229-4ubuntu2) xenial; urgency=medium
* debian/tests/unit-config: Call "daemon-reload" to clean up generated units
in between tests.
* debian/tests/unit-config: Check that enable/disable commands are
idempotent.
* debian/tests/unit-config: Detect if system units are in /usr/, so that the
test works on systems with merged /usr.
* debian/tests/unit-config: Use systemd-sysv-install instead of update-rc.d
directly, so that the test works under Fedora too.
* debian/tests/unit-config: Check disabling of a "systemctl link"ed unit,
and check "systemctl enable" on a unit with full path which is not in the
standard directories.
* Rename debian/extra/rules/73-idrac.rules to 73-special-net-names.rules, as
it is going to get rules for other devices. Also install it into the
initramfs.
* debian/extra/rules/73-special-net-names.rules: Add DEVPATH number based
naming schema for ibmveth devices. (LP: #1561096)
* Don't set SYSTEMD_READY=0 on DM_UDEV_DISABLE_OTHER_RULES_FLAG=1 devmapper
devices with "change" events, as this causes spurious unmounting with
multipath devices. (LP: #1565969)
* Fix bogus "No [Install] section" warning when enabling a unit with full
path. (LP: #1563590)

-- Martin Pitt <martin.pitt@ubuntu.com> Tue, 12 Apr 2016 09:17:35 +0200

systemd (229-4ubuntu1) xenial; urgency=medium

* Merge with Debian unstable. Remaining Ubuntu changes:
- Hack to support system-image read-only /etc, and modify files in
/etc/writable/ instead.

Upgrade fixes, keep until 16.04 LTS release:
- systemd Conflicts/Replaces/Provides systemd-services.
- Remove obsolete systemd-logind upstart job.
- Clean up obsolete /etc/udev/rules.d/README.
- systemd.postinst: Migrate mountall specific fstab options to standard
util-linux "nofail" option.
- systemctl: Don't forward telinit u to upstart. This works around
upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
that changes to point to systemd during the upgrade. This avoids running
systemd during a dist-upgrade. (LP: #1430479)
- Provide shutdown fallback for upstart. (LP: #1370329)
- Break lvm (<< 2.02.133-1ubuntu1) and remove our dummy /etc/init.d/lvm2
on upgrades, as it's shipped by lvm2 now.
- Make udev break on mdadm << 3.3-2ubuntu3, as udev's init script dropped
the "Provides: raid-mdadm".
- Clean up /var/log/udev on upgrade (which is written under upstart, but
not under systemd). (LP: #1537211)
- Migrate existing s390x network configuration to new names. (LP: #1526808)
- systemd.postinst: Bump Version comparison for migrating the UTC setting
from /etc/default/rcS to /etc/adjtime, to run it for upgrades to 16.04.
- VMWare BIOS reports implausibly high onboard numbers. This got fixed in
upstream commit 6c1e69f9. Migrate names in ifupdown accordingly.
(LP: #1550539)
- udev: Don't kill peer processes if we don't run in a cgroup. This happens
when running under/upgrading from upstart. udevd and everything else run
in the root hierarchy on all controllers then, and the alleged cleanup of
"our" cgroup becomes a system-wide killing spree. (LP: #1555237)

-- Martin Pitt <martin.pitt@ubuntu.com> Thu, 07 Apr 2016 09:53:25 +0200

# For older changelog entries, run 'apt-get changelog libsystemd0'

In that log there are a lot of references to LaunchPad (LP) bug numbers. The Systemd LaunchPad Bugzilla is here:
https://bugs.launchpad.net/ubuntu/+source/systemd.
There you will learn which Systemd bugs in Ubuntu/Kubuntu are open, and any possible workarounds till the bugs are fixed. There are currently 181 bugs listed there. Two are "Critical", about 15 are "HIGH", about 24 are "LOW" and the rest are either a "WishList" or "UNDECIDED". The undecided have not yet been confirmed as actual bugs.

The first "CRITICAL" is that in some conditions Virtual Terminals 1-6 get stuck or are blank. I've seen that problem posted here at least once but I've never experienced it on my installation.
The second "CRITICAL" is an occasional boot failure due to missing "base devices", reported for Yakkety.

Systemd main developer Lennart Poettering posted the 219 change log here but the entire Systemd change log for all releases is on GitHub here. Version 229 starts at line 666 (I know what you are thinking! ) and ends at line 881. That entire change log gives you the opportunity, at one location, to see what has been added and/or removed since version 229. Or before.

To give you an example of how much flux the Ubuntu Systemd information is just look at https://wiki.ubuntu.com/systemd, which is info on Systemd for Ubuntu. But, on that page is the warning that it is
"OUTDATED!
The information on this page is severely out of date. Don't use it! Have a look at SystemdForUpstartUsers instead." The date on that web page is January of 2015!
It leads you to a web page dated 10-11-2016, two days old.
https://wiki.ubuntu.com/SystemdForUpstartUsers.

That web page contains the the section that I used to attempt to permenantly switch 16.04 back to Upstart. The secttion begins with "If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will". When I tried it with 16.04 the process failed. So my conclusion was that it is not possible to swtich 16.04 back to Upstart. It also contained the fact that Systemd is started as PID=1 with /sbin/init, which never triggered any of my brain cells.

Lot's of reading ahead. Lot's of practice on commands.

[GreyGeek]

Ah, another CLI tool I was unaware of, even though I've used apt-get and apt-cache for years.

I've been using Linux since 1998, and beginning with the move to Upstart what I used to know about Linux has become outdated and replaced, and learning what is replacing past tools is becoming, for me, a full time job. Luckily, I'm retired and can devote more time to it. Unfortunately, my brain doesn't work as well as it used to.

:~$ apt-file search mysql.service
The program 'apt-file' is currently not installed. You can install it by typing:
sudo apt install apt-file

:~$ sudo apt install apt-file
[sudo] password for jerry:
Reading package lists... Done
Building dependency tree
Reading state information... Done
....
After this operation, 349 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libconfig-file-perl all 1.50-3 [9,722 B]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libregexp-assemble-perl all 0.36-1 [77.5 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 apt-file all 2.5.5ubuntu1 [21.6 kB]
Fetched 109 kB in 0s (233 kB/s)
....

:~$ apt-file search mysql.service
E: The cache is empty. You need to run 'apt-file update' first.

:~$ apt-file update
apt-file is now using the user's cache directory.
If you want to switch back to the system-wide cache directory,
run 'apt-file purge'
Downloading complete file http://us.archive.ubuntu.com/ubuntu/...tents-amd64.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 31.7M 100 31.7M 0 0 2091k 0 0:00:15 0:00:15 --:--:-- 2506k
......
Downloading complete file http://archive.ubuntu.com/ubuntu/dis...tents-amd64.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1378k 100 1378k 0 0 954k 0 0:00:01 0:00:01 --:--:-- 955k

:~$ apt-file search mysql.service
mysql-server-5.7: /lib/systemd/system/mysql.service
:~$

And there it is!

Thanks Sam, the info you supplied has cleared up a lot of confusion on my part. I never knew that

ps aux lists processes with the command used to start them, not the actual executable

Has that always been the case? I've read tons on how Linux worked over the years and do not recall reading that before.

I noticed that systemd does not use /etc/inittab, which I understood well and used, but which is no longer in my /etc directory. Changes like that made me feel like I am walking on quicksand, with Tux doing magic ps tricks with both sleeves rolled up! When I though about the process that led to my current ignorance of Linux, after having known it so well for over 10 years, I can trace it back to January of 2009, when I first installed Kubuntu 9.04 Alpha and decided to move away from the command line and use GUI config tools from then on. I never bothered to look into the internal workings of Upstart and felt justified in doing so because it came and went within a single computer generation.

I was going to remain with Kubuntu 14.04 till its EOL, which I thought I had read was 2019. When I learned it was 2017, I didn't see any reason not to switch to 16.04, except for the matte theme of Plasma5, which I hated. For three months earlier this year I ran Mint KDE and others as guest OS's, but all of them were moving to Plasma5. So, I came back to Kubuntu and installed 16.04 and set about making the theme as Plasma4-like as possible. That's when I ran into Systemd. At first it felt like I had encountered the Alien aboard spaceship Kubuntu, and it was eating my guts out.

At first I hated it but since almost EVERY major distro was switching to it I decided to ignore the dissenters and learn about it. My first learning problem was that Systemd started in 2010 and has had a rapid development cycle. There was a lot of conflicting information between then and now, and between various distros, which have varying degrees of implementation of systemd. I decided to limit my searches to articles written since January of this year, and to make this thread a sort of "blog" of my learning experiences, and a place to store links and info that I can refer to later as my memory continues to fade.
Thanks again, Sam!

[kubicle]

:-$ [B]ps aux | less[/B]
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         [B]1[/B]  0.0  0.0 185444  6044 ?        Ss   10:59   0:01 [B]/sbin/init[/B] splash

Yes, ps aux lists processes with the command used to start them, not the actual executable. You can check that systemd is actually pid 1 with:

$sudo ls -l /proc/1/exe

Now I am wondering why systemd-gui (or systemctl) doesn't seem know about mysql or mysqld?

Commonly (and in the default install) the mysql (or mariadb) systemd service unit is not installed (the service is in package mysql-server-{version}), and therefore mysqld is not started by systemd. A system wide mysqld instance (which would be started by systemd) is usually not necessary for desktop machines (unless you are running a web server on that machine, for example).

$apt-file search mysql.service
mysql-server-5.7: /lib/systemd/system/mysql.service

Instead, akonadi (the PIM backend) will start a local mysqld instance for every user (if you are running akonadi with the mysql backend...which is the default)

[GreyGeek]

It's a glitch in a very minor change I made to one of the vBulletin files some time back. Long story short, just click the Go Advanced button and scroll down and uncheck/recheck the Disable smilies in text (to have icon smilies not show in text) or uncheck to have icon smilies appear in text.
....

Ah! That fixed it! Thanks! Now I can wear my dunce cap proudly.

[Snowhog]

Edit: What's with the emoticons appearing as icons while editing and as text when posted?

It's a glitch in a very minor change I made to one of the vBulletin files some time back. Long story short, just click the Go Advanced button and scroll down and uncheck/recheck the Disable smilies in text (to have icon smilies not show in text) or uncheck to have icon smilies appear in text.

I did this to prevent iconized smiles from appearing in code tagged konsole output. However, the proffered fix (from over on vBulletin.com Forums) isn't entirely perfect.

Added:

Actually, I only made the change to style KFNBlue. If you aren't using that style, the unmodified behavior is in effect; the option is unchecked by default. Her is my Administrative note:

2016-07-31
Administrator: Snowhog
Edited KFNBlue's newpost_disablesmiliesoption template to enable 'Disable smilies in text' option in new posts. The change made for this on 2016-05-13 didn't actually work, so inquired as to why on vBulletin Forums and made a change recommended by MarkFL, changing {vb:raw checked="checked"} to just checked="checked". Typed smilies still show in the Your Message window if you click on Preview Post, but the actual post doesn't convert the typed smilies to graphical smilies. Acceptable, but I've also inquired if he knows why this happens.

Added:

If you're interested, this is the thread on vBulletin Forums.

[GreyGeek]

Not really, in recent versions of *buntus, /sbin/init is (should be) a symlink to /lib/systemd/systemd

$ls -l /sbin/init
lrwxrwxrwx 1 root root 1K 05.10.2016 03:12 /sbin/init -> /lib/systemd/systemd

And upstart can certainly be removed (and it shouldn't be installed at all by default)

You might be confused by the fact that systemd commonly runs several processes, only one of which is pid 1:

$pgrep -l systemd
1 systemd
243 systemd-journal
321 systemd-udevd
624 systemd-timesyn
741 systemd-logind
1111 systemd

pgrep is a new one on me. A process grep command. Nice.

I'm confused? Indeed! It looks like a shell game:

:-$ [B]ps aux | less[/B]
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         [B]1[/B]  0.0  0.0 185444  6044 ?        Ss   10:59   0:01 [B]/sbin/init[/B] splash
root         2  0.0  0.0      0     0 ?        S    10:59   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    10:59   0:00 [ksoftirqd/0]
...

I never noticed nor did it ever dawn on me that /sbin/init was a link to systemd.
Upstart/sysVinit was associated with init.d and I made too quick an assumption about /sbin/init. Wish I had known about pgrep.

I mistyped when I said Upstart cannot be removed. In my previous installation I was trying to switch from systemd to upstart and I installed upstart. It was when I attempted to remove systemd that I ran into a dependency roadblock. So, having installed upstart, I changed the grub kernel line on the run to fire upstart and my system when bye-bye.

That explains why my attempts to remove systemd borked my system. With Sam's input I realized that Upstart isn't really installed on my current system.

:~$ [FONT=courier new][B]locate upstart[/B][/FONT]
/etc/init/usb-modeswitch-upstart.conf
/etc/network/if-down.d/upstart
/etc/network/if-up.d/upstart
/lib/lsb/init-functions.d/01-upstart-lsb
/usr/share/upstart
/usr/share/doc/pulseaudio/examples/pulseaudio.upstart.example
/usr/share/locale-langpack/en@boldquot/LC_MESSAGES/upstart.mo
/usr/share/locale-langpack/en@quot/LC_MESSAGES/upstart.mo
/usr/share/locale-langpack/en_AU/LC_MESSAGES/upstart.mo
/usr/share/locale-langpack/en_CA/LC_MESSAGES/upstart.mo
/usr/share/locale-langpack/en_GB/LC_MESSAGES/upstart.mo
/usr/share/upstart/sessions
/usr/share/upstart/xdg
/usr/share/upstart/sessions/at-spi2-registryd.conf
/usr/share/upstart/sessions/dbus.conf
/usr/share/upstart/sessions/gpg-agent.conf
/usr/share/upstart/sessions/im-config.conf
/usr/share/upstart/sessions/no-pinentry-gnome3.conf
/usr/share/upstart/sessions/session-migration.conf
/usr/share/upstart/sessions/ssh-agent.conf
/usr/share/upstart/xdg/autostart
/usr/share/upstart/xdg/autostart/at-spi-dbus-bus.desktop
/var/lib/app-info/icons/ubuntu-xenial-main/64x64/upstart-monitor_upstart-monitor.png
/var/lib/app-info/icons/ubuntu-xenial-updates-main/64x64/upstart-monitor_upstart-monitor.png

:~$[FONT=courier new][B] whereis upstart[/B][/FONT]
upstart: /usr/share/upstart

:~$ [FONT=courier new][B]vdir /usr/share/upstart[/B][/FONT]
total 8
drwxr-xr-x 2 root root 4096 Sep  6 23:31 sessions
drwxr-xr-x 3 root root 4096 Sep  6 23:30 xdg

:~$ [FONT=courier new][B]vdir /usr/share/upstart/sessions/[/B][/FONT]
total 28
-rw-r--r-- 1 root root 401 Feb 24  2016 at-spi2-registryd.conf
-rw-r--r-- 1 root root 765 Apr  1  2016 dbus.conf
-rw-r--r-- 1 root root 992 Apr  8  2016 gpg-agent.conf
-rw-r--r-- 1 root root 935 Mar 25  2016 im-config.conf
-rw-r--r-- 1 root root 332 Jul  3  2015 no-pinentry-gnome3.conf
-rw-r--r-- 1 root root 159 May  7  2015 session-migration.conf
-rw-r--r-- 1 root root 792 Aug 11 11:24 ssh-agent.conf

:~$ [FONT=courier new][B]vdir /usr/share/upstart/xdg[/B][/FONT]
total 4
drwxr-xr-x 2 root root 4096 Sep  6 23:30 autostart

:~$ [FONT=courier new][B]vdir /usr/share/upstart/xdg/autostart/[/B][/FONT]
total 4
-rw-r--r-- 1 root root 135 Feb 24  2016 at-spi-dbus-bus.desktop

Now I am wondering why systemd-gui (or systemctl) doesn't seem know about mysql or mysqld?

:-$ systemctl status mysql.service
● mysql.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
:~$ systemctl status mysqld
● mysqld.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)

:~$ whereis mysqld
mysqld: /usr/sbin/mysqld /usr/share/man/man8/mysqld.8.gz
jerry@jerry-Aspire-7739:~$ ps -aef | grep [m]ysql
jerry 1789 1786 0 11:00 ? 00:00:03 /usr/sbin/mysqld --defaults-file=/home/jerry/.local/share/akonadi/mysql.conf --datadir=/home/jerry/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-jerry.HXGeTN/mysql.socket

I'm not going to temp fate by trying to disable it as a test
systemctl stop mysql(d) or systemctl disable mysql(d)

since KMail and akonadi are working well on my system.

Edit: What's with the emoticons appearing as icons while editing and as text when posted?

[kubicle]

PID=1 is /sbin/init

That is Upstart/SysVinit. So, systemd is setting on top of Upstart, which explains
Upstart has not been removed and can't be removed. Systemd is not yet ready to be PID=1.

Not really, in recent versions of *buntus, /sbin/init is (should be) a symlink to /lib/systemd/systemd

$ls -l /sbin/init
lrwxrwxrwx 1 root root 1K 05.10.2016 03:12 /sbin/init -> /lib/systemd/systemd

And upstart can certainly be removed (and it shouldn't be installed at all by default)

You might be confused by the fact that systemd commonly runs several processes, only one of which is pid 1:

$pgrep -l systemd
1 systemd
243 systemd-journal
321 systemd-udevd
624 systemd-timesyn
741 systemd-logind
1111 systemd