I propose a new term: Consultancy Driven Development. It goes like this:
- If it's too easy to set up, nobody will hire us to make it work.
- Implement a kickass setup dirt cheap for some big-name company, so we can claim they use it in production. Yeah we tweaked it so it bears little resemblance to the original product, and only fits an incredibly narrow use case, but nobody stands to benefit from blogging about that.
- Better ship with a configuration file that isn't production ready.
- Did I say one? Better have three configuration files, each duplicated in distribution-dependent directories (in some cases), needing manual sync between servers to prevent catastrophic data loss.
- Remember not to publish the program that checks for errors in configuration; half of our income would disappear.
- Benchmark with a configuration file that nobody would use in production, but looks really impressive when taken out of context.
- People want transactions, remember to claim support (and if you must, explain somewhere in the fine print that a transaction can only span a single operation on exactly one document, and btw. is precisely none of A, C, I or D).
- Somewhere on the front page, it should say how we can support petabytes of data (and it performs very well, as long as you write all your data in one batch, never modify it, keep it all in memory, and turn off persistence).
- Never give away answers online. Answer every question about configuration with "it depends".
- Don't release a new version without renaming a few configuration options. Be "backwards compatible" by ignoring unknown, obsolete and misspelled options.
What you are describing is basically Openstack. Although Hanlons razor applies, none of the current actors stands to benefit from improving the situation.
* Extremely difficult to set up.
* Claims that half of Fortune 100 uses it (read: many are required to support it; the rest have one guy with a toy installation in some branch office).
* Consists of dozens of components, each with several-thousand lines config files (actually Python code) that must be kept in sync between all nodes (yet have node-specific data).
* Claims to be "modular", but have complex interdependencies between each of the components.
* Upgrading is not officially supported, but some companies will help you.
* Will break in mysterious ways, and require you to backport bugfixes since you're stuck on an unsupported version after a year.
* Have unhelpful error messages (e.g. throw Connection Refused exception when you're actually receiving an unexpected HTTP return code).
* Write documentation in a way that appears OK to new users, but vague enough to be useless for those who are looking for specific information.
Yes, and this is why we had to invent a new job position and hire a huge workforce of "devops". A cost that tends to far outweigh the license costs associated with most commercial software because those people have to have the skills of a full blown software engineer, to preform what should be a sysadmin job.
I think if 15 years ago during the opensource vs close source wars you had mentioned that opensource projects would eventually decide that they wouldn't make any attempt at documenting the product, no one would create actual installers, and continuous development methodologies would leave 99% of opensource projects in a state of 0 testing before releases, it would have been a lot harder sell.
Part of this goes back to the early linux RPM/DEB decisions to refuse to follow in the footsteps of more traditional software installers and provide an interactive UI for configuration (see bottom about HPUX). Resulting in chef/puppet/etc. This has removed the onus on projects to dedicate resources to that portion of the project. I've worked at a few companies shipping commercial software and there were always either a team responsible for building an installer, or a part of the development cycle dedicated to it. There were also frequently technical writers (a job position that seems to be mostly gone). Now, with opensource/git its done when the code gets pushed to github. Forget any documentation more complex than a few docbook/etc comments scattered around the code base and a wiki so fragmented that going from one page to another switches the version of the code in question.
Its a pretty sad/depressing state, and sometimes I wonder how anything works anymore. Thank god for RH, which actually has testing labs, people to fix bugs, and strong opensource ethics about making sure the upstream/other distro's benefit from their work. But, then they go and behave like poo flinging CADT https://www.jwz.org/doc/cadt.html in other projects.
* See HPUX for a good example of how to do it on unix, without some of the problems windows installers frequently have. The packages have a standardized form description language that are picked up in a preinstall process, so the user can be prompted for install locations/configuration/etc before any of the packages are installed. The user runs though the entire dependent set of package forms before the install actually begins. Alternatively an ignite script (like kickstart/autoyast/etc only with most of the functionality of chef/puppet/etc) can be generated for automated deployments.
This is a problem with enterprise software as well as some Open Source software. IMO Open Source might sometimes even be simpler than some commercial stuff.
Jokes about SAP consultants moving in isn't funny anymore.
Former boss of mine told me every one of his customers that ever had decided to go with SAP had burned themselves. (IIRC the latest one I heard of started with an estimate of 5 illion monies and when I was there to install our stack they were at 25 and counting.)
I agree there is perverse incentive for open source companies to do this if their business model depends on support. Freely after Upton Sinclair: It is difficult to get a person to document something, when his salary depends upon his not documenting it!
Consultant ware has a storied history in our industry. SAP, Oracle & Peopleware are just some of the names you can think. Devops is clearly the next frontier in this movement.
- If it's too easy to set up, nobody will hire us to make it work.
- Implement a kickass setup dirt cheap for some big-name company, so we can claim they use it in production. Yeah we tweaked it so it bears little resemblance to the original product, and only fits an incredibly narrow use case, but nobody stands to benefit from blogging about that.
- Better ship with a configuration file that isn't production ready.
- Did I say one? Better have three configuration files, each duplicated in distribution-dependent directories (in some cases), needing manual sync between servers to prevent catastrophic data loss.
- Remember not to publish the program that checks for errors in configuration; half of our income would disappear.
- Benchmark with a configuration file that nobody would use in production, but looks really impressive when taken out of context.
- People want transactions, remember to claim support (and if you must, explain somewhere in the fine print that a transaction can only span a single operation on exactly one document, and btw. is precisely none of A, C, I or D).
- Somewhere on the front page, it should say how we can support petabytes of data (and it performs very well, as long as you write all your data in one batch, never modify it, keep it all in memory, and turn off persistence).
- Never give away answers online. Answer every question about configuration with "it depends".
- Don't release a new version without renaming a few configuration options. Be "backwards compatible" by ignoring unknown, obsolete and misspelled options.