I used this for configuring RHEL / CentOS machines. Good stuff. Call via Python API and everything works nicely in one or two lines of code, as opposed to editing network and other various /etc config files.
Yep, I used it heavily three or four years ago for the same thing -- system configuration files on RHEL boxes -- when I was first getting involved with and deploying puppet.
Nowadays, though, I primarily use Ansible and its core "lineinfile" module gives me pretty much the same end result.
I currently prefer using Protocol Buffers for configuration. You can use their built-in text formatter [1] to do plain-text configs. The default format isn't terrible [2], but I've been thinking about using that code as the basis for a YAML or JSON version.
Then later when you want to automate generating bits of it, you can write simple Python (or whatever) scripts to transform the data. If you want the generated stuff to be less obvious or harder to change (just to stop people who can't help poking configs in production), write it in binary format.
So, one of the reasons why people like syntax standardization in the form of formats like s-expressions, JSON, XML, et al. is that the parsers are already there and you can just load them in, do some processing, and emit a transformed version.
But when developers go to write their own configuration format they often lean towards a custom syntax so that the end-user, who is expected to be a human being with a text editor, has a more comfortable working environment. And the resulting syntax is write-only, with comments, syntax sugars, etc. that would make it hard to impossible to automate data manipulations and also emit a document that resembled the original.
Tooling like this carves back a mix of both worlds - because it's just operating on the parse tree, it doesn't convey the final meaning of the data as it would be used by the thing being configured, just the "look" of it as a human would read it - but that's still enough to get an automation job done, for most kinds of job.
If you are a developer who wants to ease this job for your own configuration formats, of course, there are plenty of ways to go about this - provide more config formats, or your own extracted parse tree, a scripting API, or the ability to change parameters live. But for pretty much any solution, there is an engineering tax to be paid, and often a new surface area for security risk. So nobody goes about this stuff the same way, and many projects just never really solve it. Hence we end up with a third party like Augeas that comes in to fill the gaps.
My own policy tends to be to start with an XML format and then add a custom syntax to it later as necessary. The initial parsing job becomes straightforward with little chance of catastrophic error, and subsequent custom syntax can emit XML, so you get a twofer on the development time. The downside is basically that XML is a pretty big dependency and nobody is happy with it as a primary UI.
It provides a consistent interface for changing all kinds of configuration files, making Ansible-style automation easier and more robust.
"Changing" goes beyond "parsing and generating"; e.g. ordering, comments and formatting are retained.
The alternative would be search-and-replace style edits (not robust), custom implementations for every format (hard work to get right, especially without Augeas' infrastructure in place), or being limited to templating entire files.
The list of supported file formats is quite extensive: http://augeas.net/stock_lenses.html (note on naming: lens is a well-known theoretical concept)
It's sad that Augeas is necessary though -- in an ideal world, developers wouldn't come up with custom ad-hoc file formats or abuse 'sh' and /etc would be empty by default.
Yeah, one of the problems in the computer industry now is that people are generating config (and other) files with template systems that don't understand anything about the underlying syntax.
If you can turn the files into parse trees, manipulate them semantically, and then turn them back into files, that rocks. For instance, right now I am working with a database that has to have a lot of things configured -- one of them is how big the buffers should be, which is a function of how much RAM you have, which in turn depends on the kind of AWS instance my product runs on.
With a smart configuration API I could take a configuration file supplied by the user and set the buffers to the right size really easily.
> [...] I am working with a database that has to have a lot of things configured -- one of them is how big the buffers should be, which is a function of how much RAM you have, which in turn depends on the kind of AWS instance my product runs on.
Which has nothing to do with the configuration format itself, it only needs
some knowledge when building config files. It can be achieved as easily with
a simple template (that doesn't understand the config it produces!), only
supplied with information local to the system the config is going to work
under. The easiest way is to build the config from a template on the system
it's going to run, but masterful Puppet with its facts database can do it on
the side of commanding server as well.
If the template system doesn't understand the semantics of the configuration file, templates just work because you are lucky. In the case of blending configuration changes with a user-supplied file, the user can supply a configuration file instead of a template.
If the system does understand the configuration file, then you can get the full value out of having a "fact database" that describes the system.
> If the template system doesn't understand the semantics of the configuration file, templates just work because you are lucky.
Or because I put appropriate constraints on data filling the template. Integer
fields are the easiest format to handle.
Generating configs with template systems is surprisingly strudy in real world
scenarios, where at least half-competent programmer provides templates for at
least half-competent sysadmin to fill.
> If the system does understand the configuration file, then you can get the full value out of having a "fact database" that describes the system.
The problem is that this "full value" is not that much bigger, but the effort
in understanding config file formats is quite large.
The work is large because parser generators are designed for systems programmers and not for application programmers. They haven't advanced much in terms of usability, thus parsing seems like a much harder problem than it really is.
> The alternative [to Augeas] would be search-and-replace style edits (not robust), custom implementations for every format (hard work to get right, especially without Augeas' infrastructure in place), or being limited to templating entire files.
Precisely. CFEngine (3.x, but also venerable 2.x) does it the former way, but
there are templating engines (Template::Toolkit for Perl, on which I have
built my cfgen, and, less convenient to control, Jinja2 for Python, which is
the base for my current work with configs) that make it easy to do the latter
way.
CFEngine 3.x adopted also templating at some point, I believe (I remember
second-hand communication with CFE engineers that said our (my) templating
gave them some ideas, but it was long ago and may be invalid anyway), but
since I haven't used CFE 3.x, I can't tell much.
Apart from Augeas, I haven't seen the idea of working with configs that had
such a solid foundation. It's such a shame the gains from having
a semantically-valid configuration editing are so small against the cost of
developing and understanding the grammar of config syntax compared to
generating the said config from a template (at least that's what I think, that
the gains are so small). I certainly would love to use Augeas-like tool in my
daily work.
When changing configs, the unit of change is often less than a file.
Using diff/patch and things like Ansible's lineinfile can apply the change, but the result is not guaranteed to be semantically or even syntactically valid.
With Augeas, it is going to be valid.
Also the big promise with a tool like this is that changes are bidirectional.
See the Boomerang project for more on this. http://www.seas.upenn.edu/~harmony/
My own experience with Augeas has been mixed.
It's quite prickly and uninformative in use and probably unintuitive to many. For example augtool by default handles the entire tree of config files. So if you just want to change maybe /etc/fstab you might encounter errors on files that you don't care about. Also I've often found that the lenses aren't quite complete, at least in standard distributions. For instance, the Centos6 Augeas's (version 1.0.0) xml lens didn't handle PCDATA or the DOCTYPE declaration.
Luckily its improving and more examples are becoming available.
Well there's quite a few configure file formats in the world. Some are keypairs, some are hierarchical, there's the popular .ini format, and many others of course.
So say you don't really care what's in /etc/ssh/sshd_config, but want to make sure it doesn't have passwords enabled. With puppet you could:
So basically augeas provides set, unset, match, remove, append, and related functionality for pretty much all configuration types. Then you just each whatever tool you are using (like puppet) about augeas and you suddenly can easily manage 100s of file types.
I have been looking for this for AGES! I have been using template files and bash scripts that use sed to change values in config files all this time. I always wondered why there wasn't a plug-able interface for config files
