From what I've experienced, it's mostly tools that target either sysadmins or sysadmins-turned-pseudo-devops, and those tools pretend that anything can be done with 'just configuration' as a marketing point. Programming is for programmers, you see.
It's easy for someone who never {wanted to, learned to} program to accept yet another configuration format (with all options specified up front) and then accidentally learn (bastardized) programming concepts. It'd take much more effort to turn them into a 'real' programmer, learning a language from scratch, practicing that, and only then introducing some tool in the form of a library.
I don't think this is fair to sysadmins, and I don't think sysadmins deserve your low opinion. The tools are designed the way they are not to appease lowly sysadmins, but because it is a pattern:
-Define the desired state of your system in a declarative way. This requires a DSL, and some tools such as Ansible and Kubernets use YAML as that DSL.
-The configuration engine (Kubernetes, Ansible, Chef, Salt) will converge the system to the desired state in an idempotent way using the DSL.
-Many systems have mostly the same config and many systems also run in environments that are mostly similar but with some differences. It doesn't follow the DRY principle to create separate but mostly identical desired state configurations for every system/environment, so we need programming primitives and templating tools to accomplish it. It's no less valid than a web app templating HTML for instance.
It's easy to conflate "mixing programming with configuration" with "using programming to solve configuration generation."
I think you're missing my point. I have nothing against using a DSL for declaratively defining an intended state of a system. It's just that this should be a proper programming language that allows to declare and use arrays, variables, and general logic.
Meanwhile, YAML is not a good DSL framework, it's not even a programming language, it's a markup language. Any logic defined in it will be inherently bolted on and likely based on text templating and full of stringly-typed confusion. A classic CM tool that does get this right is Chef, where normal Ruby code is used to build the declarative intended state. You can extremely easily implement any logic you want to create an intended state (even retrieve it from other components) and then emit a desired, declarative resource state (and --why-run will then show what steps will be taken to converge tit). You can do these things right, just don't try to turn a markup language into a turing-complete DSL.
You compare this to templating HTML - but nobody would ever seriously consider declaring such templating logic within HTML itself with some sort of Jinja2 templating engine duct-taped to the side. Instead, we use whatever is in control of serving HTML to render the final markup, with the bulk of the logic implemented in a normal programming language, while a templating engine takes care of rendering variables and loops.
Comparing Ansible's YAML and Kubernetes' YAML is also wrong. A better comparison would be comparing Ansible to Helm, since they both take the templated-YAML approach. Kubernetes by itself doesn't even let you just use YAML files as intent and idempotently apply them (some objects can be created from YAML using apply, but then not updated, and instead have to be recreated, and that's the logic that's pushed onto external tools like Helm, Kubecfg, ...).
And even instead of using a real DSL that's built into a particular tool, there's the second approach that I currently follow and recommend: use whatever programming language you're comfortable with to emit a desired state in whatever interchange format your management tools needs, and let those tools handle the domain-specific task of converging the state.
> A classic CM tool that does get this right is Chef, where normal Ruby code is used to build the declarative intended state.
If you're saying that, you should clarify for anyone not familiar with various config mgmt tools that in order to provide you with ruby, it compiles ruby in 1 pass, then evaluates the result in ruby again after enriching its environment with the results of the first pass.
This creates surprises when code apparently disappears (because it's been evaluated). In addition, to accomplish some of its other goals, chef performs an up-to 18-layer collapsing of various variables to make them available via the node object, which means that depending on which evaluation phase has completed and what else may have enriched the various layers, the state of the node object can be very hard to understand.
I say this as a big fan of chef - it's quite a lovely product. But on the other side, salt and ansible work to provide fewer footguns in the default case where the goal is to provide structured data to a function that will act on it. It allows you to footgun yourself differently, but your excitement about yaml vs. a programming language is understandable, but impractical since putting programming languages into the position of managing the messy reality of running a program on an operating system requires a lot of any programming language - it's where the rubber meets the road and the road is neither smooth nor straight.
And back in the day sysadmins wrote programs like tcp_wrappers, postfix, sendmail, etc. It was definitely common and encouraged if not expected.
The idea that system admins should be able to get away with never professionally stepping up to doing programming is a late-90s/early-00s aberration that was generated by the need to scale up during the dot com boom before good automation practices existed. And it is not a good thing and it stunts intellectual growth. Devops is in some sense back-to-the-future by tearing down an artificial wall that never should have been made in the first place.
Weitse, the author of 2 of those tools, is the only systems person at his university who I'm aware of who has done a lot of tooling. Allman, author of sendmail, was not a sysadmin by any stretch of the imagination. IIRC he was a grad student then systems hacker who was surrounded by unix and programming.
Sure, lots of systems people program and love programming, but the job isn't to write write tcp_wrappers, postfix, sendmail etc. it's to do what's needed to make sure that such things run, which is a different kind of programming.
Would it be beneficial? Yes. Is it necessary for most cases? No. I do think, however, the moment when you infrastructure becomes defined by code (eg. you start using any CM system) then you should be training your sysadmins to program more and duct tape existing solutions less. Operations become vastly easier to scale and maintain if you can defer their logic to actual services with clearly defined business logic rather than a mess of ansible configs and bash scripts.
> Admins that don't learn to program are left behind. Only those who have some cushy full time employment, were they can't be let go, will remain.
While I personally am a Software Engineer + Systems Engineer combination perfect for modern Devops... I do not agree with you one bit.
There will always be a place for non-programming systems administrators -- especially as we get more and more systems over time.
Will those administrators be able to /scale/ to managing 100s or 1000s of system without learning some sort of programming language or configuration management system? Probably not.
However, there are plenty of businesses out there that only have a handful of servers and services which can totally (and more cheaply) be managed manually.
Automation has an up-front cost that doesn't always make sense for the business to spend.
Because it's often really hard to know what aspects of your product needs the flexibility of code and what benefits from the simplicity of configuration.
I wish we started analyzing this as the hard problem it is rather than getting angry that people didn't always make the right decision by correctly anticipating how their software would end up being used.
Maybe jinja is well-suited to whatever its intended purpose is, but it's not well-suited for programmatically building markup. Using a dynamically typed scripting language to generate the markup at least has the benefit of allowing you to define appropriate semantics via functions ("give me a blob of YAML that represents a Kubernetes service with these parameters").
Sure, you can probably build your own concept of functions on top of Jinja, either by embedding them or building your own paradigm on top of Jinja's template inheritance, but it's a hack either way.
Well, I think it was originally designed with server-side html templating in mind. It works great for that purpose.
In the case of thread OP, that template is pretty messy, yes. In ansible, I would just create a custom module or action plugin in python to spit out whatever it is I need. I don't know how flexible salt is in this regard, but ansible is quite flexible.
For simple cases, jinja works great if you're just doing a find/replace type operation in yaml.
We don't have to--you can generate the markup language with a scripting language and end up with something that is quite a lot cleaner. I've been toying around with Starlark as a yaml generator for Docker Compose, CloudFormation, and Kubernetes. This ends up working out a lot more nicely in practice than Helm's templating solution or CloudFormation's nested stacks.
Thanks for pointing that out. It's exhausting reading people in this thread getting very excited and demonstrative about the fact that a flexible system like salt provides a way to do this that results in a serialized data stream. They assume that they know better how this should be done, and posture about their superiority vs. some kind of straw man.
Because their bosses dream up scenarios where customers will be generating config files to be used as templates, etc..., and they're just trying to solve whatever problem was put in front of them and "we'll use YAML", IME. Of course, those scenarios never play out, the thing is bastardized, some new responsibility gets created for some new role where it's their job to handle the template generation on a per customer / per site basis, and on and on we go.
You got me wrong here. I don't think markup languages are bad.
I just don't understand why someone would create such a "clusterfuck linked by my GP"
If you re-invent a programming language in markup, you basically get a new custom programming language that "probably" can do the same as another non-custom programming language, but without the benefit of many people already know them.
Why not integrate something like Lua or Dyon instead of YAML or XML?
I know what you mean, and agree with you ;) The inflationary use of the term "markup" just bothers me, when it denotes a very specific thing that's been in use since before computers existed.
You can use python to create that directly. Jinja is just another, default way to achieve this. Why? Because it turns out it's much cleaner that way. (I've done both)
I mean, sure, you can convert one into the other syntax-wise, both form trees, but what do you gain?