I think journald is by far the buggiest component systemd has. Log corruptions, the dubious Forward Secure Sealing mechanism, journal file fragmentation, the gateway daemon, coredump hooks. There's been several CVEs for it, too. Of course, you cannot disable it without source code modification.
Quoting from the systemd wiki [1]:
Note that it is now the journal that listens on /dev/log,
no longer the BSD syslog daemon directly. If your logging
daemon wants to get access to all logging data then it
should listen on /run/systemd/journal/log instead via
the syslog.socket unit file that is shipped along with
systemd. On a systemd system it is no longer OK to listen
on /dev/log directly, and your daemon may not bind to the
/run/systemd/journal/syslog socket on its own. If you do
that then you will lose logging from STDOUT/STDERR of
services (as well as other stuff).
...unless your syslog implementation is socket
activatable many services will not be able to log to your
syslog implementation and early boot messages are lost
entirely to your syslog implementation.
So the fact that keeping your regular syslogd wasn't as simple as merely symlinking syslog.service was publicly known, but often overlooked by proponents during discussions.
When someone comments about how overly complex and controlling systemd is for an init system, I think of all the hideous shell scripts that don't talk to their neighbors which end up forming most the sysvinit boot process on many distros (in the past?).
I wish there was something [lighter|simpler|more portable|more unix] than systemd.
Every time I seed something [good|bad] about systemd it makes me want to [comment|upvote|walk away].
Not just SysV - every attempt to move beyond BSD-style init has been a mess.
> I wish there was something [lighter|simpler|more portable|more unix] than systemd.
I think the trick with systemd is to realise that it's not unix. But it's something else that's really interesting in its own right.
Init/scheduling is a much more difficult problem than it first appears, and it has always and everywhere been tacked on to its host OS as an afterthought: init scripts, cron, windows services, system v init. All are poor, partial solutions. Yuck.
To fill the void, there has been a proliferation of application containers, each of which does a different 70% of the general problem. All of this stuff could just be done in the platform. And that's what systemd is.
There's going to be two separate camps at the end of this: a group led by the BSD platforms will stick to convenitional unix goals. On the other side, the linux kernel plays second-fiddle to systemd. The systemd service graph is a new operating system tradition.
[New, but not without precedent: in the debian world, the 'apt' package management system has become more significant than the kernel. You can create things that are not linux, but which are still debian. Both systemd and apt use a sophisticated data-graph to contain a difficult problem.]
There are exciting opportunities for developers on a systemd operating system: you'll be able to quickly put together application stacks using microservices in a way that hasn't been practical until now. It /is/ unixy in the sense that it gives the console operator a powerful new general-purpose tool.
I think the trick with systemd is to realise that it's not unix.
I keep coming back to that. It seems to me that there is a tension in the Linux world about people who want an open source UNIX and people who want an open source Windows. Systemd is much more a "Windows" approach to system configuration management than a "UNIX" approach.
That isn't "bad" per se but it makes for some confusing things. I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS. I believe such an explicit step would help folks move forward with "where" a certain idiom of OS management should land in the open source world. Of course I don't expect that to happen :-).
I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS.
This mindset should terrify everyone in the GNU foundation. systemd is the begining of the march away from both the GNU and GPL principles, into an incompatible fork of POSIX that uses RPC backdoors in order to circumvent GPL protections.
why doesn't anyone else view systemd as the direct assault on GNU that it is?
Perhaps you could ask yourself "how, as a non-free software developer, could I use systemd's RPC system to bypass the LGPL and GPL's requirements that all related software and derivative software must be licensed under the same license?"
The answer will come quickly, I bet. it's how you figure out magic tricks, as well. :)
> Systemd is much more a "Windows" approach to system
> configuration management
I can't see your Windows analogy. The Windows init system compares badly to everything in play here - it doesn't make the system observable or discoverable or give power to the user or do a good job of centralising control.
There would be a comparison to Mac OSX, which has rounded the corners of init with its own approach.
It's also worth mentioning that all mainstream unix (even OpenBSD) deviates from unix philosophy in places. A trivial example - the way that /bin/ls changes behaviour depending on whether it's outputting to stdout or to pipe. If there's a gold standard, it's Plan 9. And almost nobody uses that.
systemd is 100% free software, unlink Windows. systemd has a command line interface unlike most Windows tools. And I don't know what is the Windows init system but I doubt it is similar to systemd. It's probably much closer to the Mac OS X init than Windows'.
Seriously, it does not make any sense to say that systemd is a "Windows" approach.
Not only is it Free Software, but LP was nice enough to make the entire thing accessible over RPC, so that non-free software could easily integrate with the OS at the lowest levels (PID1!), bypassing GPL/LGPL restrictions, and removing much of the reason to write Free Software on Linux at all.
How this isn't worse than Tivo-ization (the reason the GPLv3 was written!), now how RMS/Moglen aren't screaming about this, I'll never know.
I like the rationale that services should be represented by a graph, and I'd like to see this pick up on BSD (it's not terribly complicated to implement on top of init).
However, systemd goes way beyond that, and tries to control everything too much, in too granular detail. I wish the move from init -> systemd had been more gradual, there was no need to rewrite absolutely everything around init.
Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box.
In fact, application container deployments frequently don't use an init and process supervisor at all, but when they do it's a bare minimum loop that reaps zombies and reexecs a process on SIGCHLD.
On the linux side, the kernel will start to play second fiddle to the systemd.
That much is true, for better or for worse. systemd and much of the new desktop userland is refactoring a lot of kernel components into userspace, creating the de facto equivalent of an unwieldy hybrid kernel. I suppose Tanenbaum did win that debate, after all. Sort of.
in the debian world, the package management system has become more significant than the kernel.
The package management infrastructure is basically what makes a Linux distribution a "distribution". It's the backbone. That you can create Debian stacks based on non-Linux kernels is because many Linux packages are cross-Unix.
as a development platform you'll be able to quickly put together application stacks using microserves in a way that hasn't been practical until now
Wikipedia defines microservices as:
In computing, microservices is a software architecture
style, in which complex applications are composed of
small, independent processes communicating with each
other using language-agnostic APIs. These services are
small, highly decoupled and focus on doing a small task.
That sounds a lot like... the Unix philosophy, which you openly admit systemd is in opposition to.
It seems like a daemontools-style toolkit approach (e.g. s6, nosh, perp, runit, etc.) fits the "microservices" umbrella far better. Indeed, I've been seeing plenty of container configurations based on those tools.
"Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box."
Given that multi-user and multi-tasking is pretty much a massive hack on what is at the core a single task, single user design, no surprises there.
I say a massive hack, as a CPU is in essence an assembly line. It takes data and processing directives in one end and spit processed data out the other. What multitasking does is basically to halt the assembly line mid job, documents its state, clear it out, and then load onto it the state of a different job. And so it goes, cycling through all the tasks.
Then again, the apparent reason for the discontinuation of Consolekit in favor of Logind was that the latter, via its ties to systemd-as-init-and-cgroup-manager-in-chief, was multi-seat.
A concept where one pretty much recreates the mainframe on the desktop by logically assigning disparate IO devices to a "seat", and so in essence creates terminals. Apparently this is supposed to work while still maintaining the ability to plug and unplug IO devices at will. Quite the rabbit hole.
Thanks for the links - I'm sticking with Debian for now so I guess I'm using systemd for the foreseeable future there, which I [do|dont] have a problem with. systemd hasn't failed me yet - it really does just make me feel 'weird'.
feeling uneasy about a software decision you didn't make is one of the reason many of us switched from Windows to Linux in the first place. It's a little weird seeing people make the same rationalizations I made with Windows.
For the servers I run, boot time isn't an issue. I might reboot a server once a year, and booting in 10 seconds vs 40 seconds isn't really an issue. What is an issue for me is stability. I don't really care about "hideous shell scripts" as long as they work.
no, there are almost no desktop users out there; certainly, far fewer, probably by two orders of magnitude, than server and embedded installations; and the divergence in favor of doing things the desktop developers' way is very much the tail wagging the dog.
The people who say "it's not that bad" are those who haven't interacted with these extremely reliable and integral parts of a Unix system for years and years and understand the ramifications of bad behavior.
So systemd has a few bugs that affects its interaction with syslog. I see nothing to indicate this is specifically because of some ill will towards it.
Is there a bug report with a dismissive reply? Or one being ignored?
(I do agree that it shows that systemd is too immature for critical production environments)
> Is there a bug report with a dismissive reply? Or one being ignored?
That's always my reaction to blog posts like this as well. A blog post is not a bug report. Given the mentioned bug in the systemd/syslog interaction that made it into RHEL, there should be a bug in the RHEL bug-tracker, and I'd hope it'd be a high priority one.
I think the issue is just stability in general. I just moved to a new server recently, and I decided to go with CentOS 6 instead of 7, as people were reporting that CentOS 7 is just a little less stable in general than 6. (Although systemd probably isn't entirely responsible for the lower stability in CentOS 7, it probably is one of the major changes in CentOS 7 so it likely takes a good share of the blame).
You always end up with bugs in released software, even CentOS. Some bugs only show up in certain situations, and might be difficult to track down. The only way to give a good guarantee of stability is for a large number of people to use the damn thing for years on end!
So this means any logging/auditing software that requires the data to be sourced from syslog is unreliable. This kind of behavior would make an auditor's head spin.
I only watched part of the systemd wars from the side line and to this day I haven't come up with a reasonable solution. Both sides have reasonable arguments.
Saying that a tool is unsupportive towards another tool because in the interaction two irregular bugs happen sounds like a weak claim though. Don't all tools have bugs? What makes these two special?
I use arch and still despise it. The reduced boot time isn't worth the headache the overly complex systemd suit of tools has introduced. BSD might have some issues, but it worked very well for my use-cases.
To me, as an Arch user, sytemd has been responsible for a series of btrfs kernel panics. I'll probably switch to Gentoo just to not have to deal with working around journald the next time I have a choice of distribution.
While I agree that systemd does not play nicely with syslog, you quickly realize that journald can be exported as JSON, filtered, described as time ranges, chunked and easily imported into ELK. My users (mainly developers) with less sed/awk fu have come to rely on these features heavily. I'm not really sure what there is to miss about syslog or syslog-ng when you have a robust ELK+journald system.
Unfortunately it's also very slow. Text files don't have the many advantages you outline, but in everyday use they are much faster than the C journal API.
Quoting from the systemd wiki [1]:
So the fact that keeping your regular syslogd wasn't as simple as merely symlinking syslog.service was publicly known, but often overlooked by proponents during discussions.[1] https://wiki.freedesktop.org/www/Software/systemd/syslog/