This is outstanding... I really wish I would have had this learning systemd as it would have saved me _hours_. Let's be real - a lot of Linux fundamental stuff is still pretty terse. Learning tools like this really smooth it out and optimize how long it takes to wrap your head around a concept. Also, I am a _huge_ advocate for hands-on in regards to retention.
Adding to my favorites and will be passing this on over the years - thank you for such a great resource.
If anything the biggest problem the systemd docs have is verbosity. It's all classic UNIX man pages, a billion pages of detail on every possible setting with no useful examples anywhere. Fortunately the core system is simple enough that the learning curve isn't too steep but I'd really hate to try and learn it from the official docs. They don't even apply CSS to the HTML versions of the docs.
This isn't a knock but it's exactly what I was talking about and why I find OP's learning tool to be so valuable.
Lots of folks learn by example and hands-on labs. Personally, I'd much rather learn the basic ropes by jumping into a tool like OP's vs. finding/digging through all of these resources. I'll also criticize to say you likely already know much about systemd, and were able to pull/filter these resources much easier vs someone completely new to the concepts.
To illustrate further: vim is another tool that has outstanding learning resources, everything from very quick "hey get started" examples docs all the way up to adventure games. If I had to go back and relearn vim I would absolutely do it this way vs. digging on man pages like when I was a kid in the 90's. Personally, I learn by doing.
---
Overall - OP's thingy is what I would call a "rich interactive learning tool." It's anecdotal, and obvious projection - but _for me_, interactive learning tools optimize the time it takes to fully "grok" a subject from scratch vs. jumping into a bunch of docs/man pages.
TIL about that manpage. I've always been guessing at which level in the hierarchy would contain a particular option ("Is it service-specific? Or maybe applies to all execs? Or maybe applies to all units?") and just giving up and opening them all in parallel.
Couldn't agree more. Once familiar with the big picture this page is the best starting point whenever to you want change something.
To get the big picture I read the systemd posts on https://0pointer.net/blog/archives.html But that's 10 years already! No idea what would be recommended today.
And almost none of the tools help you naturally discover the missing feature you didn't know to solve the problem in the most "systemd*" way.
The docs are not friendly enough to introduce you to new tools, and they're mostly shallow enough to not explain the technical details of how things interact or where all the defaults are stored. They exist but clearly this page exists because they need to improve.
And, as a Debian user who often finds himself on the Arch docs - there's a ton of distro-specific stuff in the Arch wiki. And rightfully so - it's a wiki for a distro.
These sorta landmines when trying to just research/digest a concept can really suck. OP's tool really eloquently breaks things down to _just_ core concepts so you can quickly start to grok what I consider to be a relatively complex tool.
In my experience very little of it is distro-specific, especially if you're on a distro that has similar components as Arch. I'm on OpenSUSE Tumbleweed, which also uses systemd, etc and is a rolling distro with the latest versions of everything, so a lot of the wiki articles apply directly as long as I translate the package names.
Agree - but there's still bits and pieces that don't translate over to other distros which can suck if you're jumping into to learning something. When I read the Arch Wiki as a Debian user I realize I'm reading it through the lens of someone using a different distro.
Overall I use the Arch wiki very often and it's because of the exact point you're making - I'm just being pedantic saying those slight distro differences can be a pain.
There is a trillion of tutorials for every young and hyped technology out there: Svelte/FastAPI/Elm etc. However try to learn how to maintain and secure a server or a database or anything fundamental... and it can be really hard to find good and updated material.
Agreed, I have generally avoided systemd as I don't feel it's fully baked yet (not trying to start a flame war, that is just my experience and I'm sure it's fine for others). Still, I need to really learn it both for when I am ready to use it daily and just to have it in my wheelhouse for supporting others. A guide like this seems like a great way to go "hands on" without spinning up an instance and fumbling through it blindly.
Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
>given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
I don't understand where anyone is getting this impression from, that describes the opposite of my experience with systemd. The authors publish stability guarantees, and they have done so for several years: https://systemd.io/PORTABILITY_AND_STABILITY/
I love systemd both for it's actually functionality and the fact that it's almost like the flagship of a whole movement to make Linux more consistent and integrated and less of a pile o scripts.
But for some reason, the authors occasionally say something that feels completely out of place for the project. They fix important stuff eventually, but sometimes it gets confusing to watch them discuss it.
The "Invalid username runs a service as root" bug is a good example of something that they didn't seem to take seriously. But it's fixed now.
Also, PulseAudio. Pulse is good. It's not that great and was deployed before it's ready on some distros. The whole Pulse/Jack split that is only just now being fixed was annoying.
Some people might still judge Poettering based on 10yo bugs in Pulse rollout?
After a lot of recent events, I don't feel comfortable judging any open source project on the speed at which they fix bugs or the way they communicate about it. In open source there is no obligation for anyone to fix anything or communicate anything. To me it's a miracle when anything gets fixed at all. It feels like the community is still largely held together with duct tape and string.
Yeah, I just expect all software to be buggy. To be honest my main way of evaluating software is by Googling stuff like "X ate my data" or "X stopped working" and seeing how many major complaints people have, relatively to how many users there are.
I think the duct tape and string feeling comes from the obsession with smallness and modularity and people trying to make "Linux not be like windows".
Half the FOSS scene WANTS things to be taped together because their main thing is random tinkering.
A guarantee is only as good as the willingness to honor it. There are great ideas and some good implementations in systemd, but when the people behind it have a pattern of refusing to fix (and sometimes even acknowledge) show-stopping bugs, it makes me apprehensive about relying on it.
Again, I didn't want my comment to start a heated discussion about systemd, but inevitably that happens. If I had praised it someone else would have come along to tell me how much it sucks (I don't think it sucks, personally, I just have issues with its current development process).
-----
Adding "hacks" but not testing to ensure said hacks don't cause issues:
Lennart Poettering 2014-02-21 13:49:25 UTC
To make this work we'd need a patch, as nobody of us tests this.
Comment 3 Michael Shigorin 2014-04-04 06:30:57 UTC
Hope all of you either test all the combinations or do not break at will those you don't have the time and inclination to test, at least in system-wide components that are not specific to systemd, while pushing the latter hard.
Comment 4 Lennart Poettering 2014-04-04 14:56:43 UTC
Well, cgroups-less kernels are explicitly not supported by systemd. However we added some hacks to allow it to boot to a certain degree even if a lot of things will not work correctly afterwards. In this mode when you boot you will actually get a warning on screen and bootup is delayed by 10s to make sure the user understands this.
Now, this mode recently broke, and it will segfault early on. I am happy to take a patch to 'fix' this again, but I will not work on this as i dont run kernels like this, and as mentioned its not really supported anyway...
Another option is to simply be honest amd stop supporting in entirely, and refuse booting completely. And I figure this is what I will eventually do if nobody cares enough to send me a patch to fix that segfault.
I can't figure out what the problem here is, are you trying to run systemd with a kernel with cgroups compiled out? I don't see how that's a show stopping bug, from the discussion there it was never supported.
No, I'm not. I'm satisfying the other person's request for sources with examples I've come across in the past. In this case it was not about cgroup-less kernels but about Poettering's and Sievers' attitudes towards fixing the issue.
This is why I don't like to talk about systemd on any internet forum; it always leads to people like you ready to attack any position around the subject, and me having to defend a position I don't even hold (I don't think systemd sucks, I think its developers could do better though). I am done with this thread.
I'm not attacking you or asking you to defend a position, I'm asking for clarification because I legitimately do not understand what the issue is. I totally agree with you, it is not helping anyone to become defensive over this. I don't understand what the attitude was, it seems to have been fixed 8 years ago? Did I miss something?
>Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
The core dependency principles around targets, services and startup does not see a lot of change, so this concern is not really realistic. You'll see more of this around the supporting utilities (networkd, resolved, and so on).
In fairness, quite often I've found this is often a case of exposing existing brokenness rather than creating new problems, but it doesn't make it any more comfortable when it happens.
That doesn't follow. If anything, I would expect knowing, for practical reasons, you have to learn something you think is dumb would make you even more angry about it, not less... or, maybe, you just believe something being either popular or common means it must be good? :(
It's reasonable to have the opinion that juggling chainsaws is a dumb idea, even before you've learnt the techniques for making it slightly less dangerous.
Is that really any dumber than doing any similarly dangerous industrial jobs that require a lot of training? Or something else dangerous in entertainment like being a Hollywood stuntman, or being Siegfried & Roy? I'm sure you could get a lot of money for having that skill if you did it in the right place.
It's on version 250. How many stable releases do you need to feel that it is "fully baked"? The software is of course evolving and new features are added but breaking changes are rare.
I just tried the first exercise of creating a default target and this web application is a great learning resource for Systemd. It should be noted that it works best as a complement to the author’s series on Systemd by example¹.
I also love how the web application works without requiring multiple JavaScript dependencies hosted on third-party servers.
I will up-vote every single one of these "_______ by example" sites that are made. I'd also love to see a "Buy a Coffee" or quick 5-10 USD donation button on each one - and if you want a commissioning for one throw a patreon or kickstarter up and I'll share it around.
I found the gobyexample site to be invaluable when I first encountered the language, and now I send people there BEFORE I send them to the official go docs (which is saying something - because the official docs are great!).
Here's hoping we can get well-structured documentation that scaffolds like these to replace most medium posts in the top results of most Google searches! :-)
I was very anti systemd initially but mostly because it meant the many initd scripts I had for different services would no longer be usable or the daemons I had would no longer need to self mange their own pid files. But mostly it was out of fear - knowing how many battle scars each of those shell scripts had learned or forking lines of code had adapted could be lost… fear that maybe systemd would not carry those lessons forward… I let go and it has been pleasantly easier and much less stressing then I imagined… more examples like this are great thanks!
Since systemd I never had to learn about my init system, it basically just works. For to reasons.
1. On desktop the old init systems were quite good already, before systemd was introduced. Systemd made it better, as I've been told: I never had any beef with it (oh boy did I have beef with init systems in the old days, not only on desktop...), I did not do any benchmarks to see that it shaved off a few seconds in startup time (and helps remove lots of fragile network mgmt code).
2. On server I now use Docker. It has no init system. And when I need one I use one that fits the docker world (i.e. supervisor).
My recollection is that it either adopted, or inspired, https://github.com/Yelp/dumb-init#readme which folks used to put into their Dockerfile as the init system back in the day
Folks (ahem, I'm looking at you, eks-anywhere[0]) who bundle systemd into a docker container are gravely misguided, and the ones which do so for the ability to launch sshd alongside the actual container's main process are truly, truly lost
I recently switched to multirun [1] which in my case is superior to supervisor. I don't have to create any config files anymore, and it behaves exactly as i want it to: If one on the processes dies the complete container will die and docker's restart-policy takes place.
I like all those 'learnXbyexample' sites, though I really hope I can have a drop down menu or side-bar to access the TOC instead of going back to homepage each time for next subject.
Maybe slightly off topic, but I'd love to hear about people's experiences using systemd as process manager inside a container, specifically for the unprivileged container case, where systemd would be launched in user mode, so only doing process management, without all the other kernel-hook stuff going on.
I really like systemd, and I really don't like supervisor. So I'd love to figure out how to make this work.
I know the easiest thing would be to ignore the user mode business and just use podman, where the work has already been done [1], but at least in the short term that doesn't help for more restricted environments like cloud kubernetes.
That's fascinating— thanks for pointing it out. I definitely assumed the issue was with the daemon itself, and not with the services it was trying to start.
This article led me to believe that I could run systemd in an unprivileged docker (not podman) container, but I did not find that to be the case. It was not happy until I added `--cap-add=SYS_ADMIN` capability, and then it started working.
May I ask why you'd want to use a process manager inside a container? I just consider Docker my process manager. If my application crashed, Docker will deal with it.
A valid question— certainly there's a school of thought that says one process per container, end of story.
However, there are lots of applications out there (especially legacy ones, think stuff like Zoneminder) which are made up of multiple small daemon-type processes sharing state across ports and maybe even the filesystem, and have a strong reliance on system services like cron or log rotation. Yes, with effort an application like this could be be fully "ported" to a container-native setup, but the path of least resistance is often to just make the container environment present as being more like a full VM.
I get the impression a lot of the entire point of containers in those things is to deal with those awful zoneminder style designs, PHP junk that needs specific Apache configs, etc.
I prefer to just avoid any application that requires a service to in any way be configured to account for the app.
It's bad enough that Zigbee2MQTT is a separate process, but it's acceptable because it is a pure layering model and as long as you are local only and not needing passwords, there's no manual config.
Things like ZM that need an actual database are just insane. Multi-page configuration instructions for something that the(Sadly closed source) AgentDVR does with one process, no containers, no major configuration to conflict with anything else.
But modular apps exist, so we need containers to make them act like monoliths.
I just learned the absolute basics of systemd the other day while trying to set up an auto-restarting game server on an RPi for my little sibling, and finding helpful examples was extremely difficult. This seems like a perfect solution!
Wow wonderful set of articles. A big thanks! Wanted to understand this stuff for a long time but could not give enough time to navigate the dense documentation.
Adding to my favorites and will be passing this on over the years - thank you for such a great resource.