Switching from Hadoop to Spark was clearly a good idea for our team, even though it required learning a new stack, but there isn't a strong reason to switch to Flink or start using Haskell.

Agile makes sense when your main risk is fine-grained details of user requirements, but not when you have other substantial risks, such as making sure a statistical algorithm is accurate enough.

Microservices probably reduces the asymptotic cost of scaling but add a huge constant factor.

Relational databases are the right choice 95% of the time, non-relational stores require a really specific use case.

TDD is good for fast feedback in some domains, but for others, manually investigating the output or putting your logic into types is better. E.g. a lot of my time comes from scaling jobs that work on 10gb of data but crash on 1tb, TDD is not that helpful here.

Continuous integration mostly makes sense when you're making a lot of small changes and can reliably expect a test suite to catch issues.

In short, ask the question "when is practice X useful?" instead of "is practice X a good idea?"

If this were Medium, I'd highlight the hell out of that.

That's so true, and so nicely, succinctly put - it ought to be the reply to end every argument about whether microservices are good or bad.

Meanwhile, the rest of our codebase (a monolith) crashed every few days for one reason or another. We had an on-call rotation not because that's what you're supposed to do, but because we actually needed it.

Now I'm not saying that microservices make sense for everyone. In general, I agree that they are used incorrectly. Microservices are hot and software developers, generally speaking, like to use hot technologies. Yes, moving to a microservice was costly. We had to re-write a lot of code, we had to set up our own servers, and we had to get permission from the guardians that be to do all of this. But, for our use case, and I assume there are other use cases too, the benefits of detaching ourselves from the company's monolithic codebase far outweighed the costs for doing so.

TL;DR No argument is the end to every conversation. Few things are so black and white.

Sooner or later you get a feel for which bits are becoming at least API stable and could run independently. That's when I split them out.

Do it too soon and you end up choosing the wrong boundaries and tying yourself up in knots, do it too late and your monolith can become a mess that's difficult to detach the pieces of.

I don't go too far with this, just avoid things like shared static state and other anti-patterns.

It does assume that you either start with devs familiar with OTP or you have generalist devs that can pick things up quickly.

Could you expand on how the architecture actually looked? What made it a monolith and what made it micro serviced?

You reap the scaling benefits way later, if ever.

When you need to scale, then you can easily throw your libraries behind an RPC framework and call it microservices, but there's no need to pay that cost until you actually face that problem.

That doesn't mean you can't scale up from a monolith (even one without clean interfaces) - every startup growth story is a testament otherwise - but it's never as as easy as strapping an RPC layer over your library.

This is a real problem - I recall that Sanjay Ghemawat et al was working on it when I left Google, though I dunno if the solution they came up with is public yet. It's unlikely to seriously affect you unless you're Google-scale, though, by which time you've probably divided everything up into services and aren't taking advice from the Internet anyway. For companies that are a few teams working on a single product, it's easy enough to send a company-wide e-mail saying "Rebuild & redeploy anything that depends upon library X", and if you're doing continuous deployment or deploy only as a single artifact, the problem never affects you anyway.

For starters the vast majority of code is developed in-house in non-software companies. The vast number of products are a single team working essentially in a silo, not "a few teams working on a single product".

When people are talking about small companies, it's misleading to think "smaller than Google". Smaller-than-Google is still an enormous quantity of development. Enterprisy practices make much sense in scaling software in companies that are smaller-than-smaller-than-Google. if you hear "small company", think multiple steps further from that, a smaller-than-smaller-than-smaller-than-smaller-than-smaller-than-Google company.

Sure, but if you do that in place it will still break stuff that assumes it works like the last version, and if you do a versioned API or the like you still can't force all teams to adopt the new version.

Does your CI system not automatically build dependent artifacts--

> It's unlikely to seriously affect you unless you're Google-scale, though

--okay, whew. ;)

Of course, there are other limitations this imposes, but it does make it very simple to deploy a new library to all code which uses it.

You can deploy a new version of the dll and applications can pick it up when they restart. Linux will apply security patches this way.

Microservices are only a step ahead.

That said, in many cases the cost of a full restart can be accepted.

It helps cognitive load because such apps can be reasoned about without reading code elsewhere.

Except splitting into microservices is an unnecessarily complex design choice. That's almost always worse, and the cognitive load comes in when you now need to figure out how to get this stuff right. The scaling benefits also require that you get it right, small flaws in your system become massive issues.

It can be as simple as a simple class, or maybe a larger class as a single-file service, or an entire namespace with a several classes, or a separate library easily referenced. All the "logic" split benefits without the ridiculous hassle of microservices.

However, exercising proper software discipline and using languages with good/existent module systems, like OCaml or Go, can lead to the same modular results without the fixed overhead. If you don't have a full-time ops person or team, you almost always have no business running microservices.

This too!

Relational databases are great, but I spent large parts of my life as a developer writing layers converting to/from SQL and later ORMs. There's a huge gain in just not translating data. I know Postgres (and others) deal with JSON, but I can't escape the feeling it's a bit shoe horned in there – basic SQL statements have strange new operators like ->> -> #>>.

Relational databases are great for, well, relational data with strong consistency requirements. The popularity of the original MyISAM tables without integrity checks baffled me at the time. Why spend time marshalling data in/out of table form when you don't gain the benefits of a RDBM?

Not doing data translation saves _a lot_ of time. Plain key-value stores are amazing, document stores like Elasticsearch are great, ultimately the choice comes down to requirements and time saving is often a very heavy argument, especially for small companies/startups.

I never found data translation/serialization to be a big pain (just rely on a framework/lib that does it for you). It's a bigger pain to hand-roll joins that would be a one-liner in SQL or deal with issues that arise from having your data (unnecessarily) reside in many systems.

> Things such as joins, transactions, and means of enforcing data integrity are useful

If the domain needs transactions I'm already speccing them regardless of general usefulness. Yes all that stuff is useful, but all abstractions have a cost which isn't free just because it's hidden in the DB.

For a problem that didn't need transactions, but for which they were useful, why would you automatically want to couple the solution with your storage layer? If you're looking for the ability to express business logic clearly without cluttering it with error handling, for instance, software transaction memory would probably be a better level to work at.

Considering we speak about daemon software (services exposing some API to readers and writers) that provides CRUD behavior to the user (end-user or other developer), isn't that nearly always the case to guarantee write access to concurrent writers without the risk of crippling your data?

Furthermore I am not sure how this relates to FP.

> It's much easier and safer to declare your constraints rather than trying to enforce them.

But that is a strong point of RDBMs implementing SQL. You have some kind of schema (think type) and use selected functions (select, update, delete, create, etc.) to transform the data.

> Furthermore I am not sure how this relates to FP.

I'm saying that FP is great for ensuring correctness.

> But that is a strong point of RDBMs implementing SQL.

Right. And if I didn't have other functional languages available that might be a bigger issue.

I'd definitely use a trusted DB for storing bank accounts. The consistency is pretty much the first requirement and the data maps perfectly to tables. And 7B checking accounts isn't that big, compared to some problems so it'd probably scale pretty well even worst-case.

But I probably wouldn't for an MMO. Or at least, it wouldn't be where I stored every little thing going on around them - just the events (xp and gold earned) that they'd freak out about if we lost. But even just a log-structured DB would work well for that.

If there's no contention for a resource (in the bank case - the value in the account) there's much less reason for a transaction. I want the system to make its best effort but I don't want to wait around for the message if there isn't anything I can do on a failure anyways.

I start with text files and for most purposes I do not bother with anything else.

Next is a key/value store. Simple.

Relational databases carry large overhead in translating data (as above) and also in design and maintenance of the structure and getting data into and out of them. I spent many years with them, like them a lot, but they are too much complication for most purposes.

Even with relational data RDBMS are only good if you are not certain of how you will be accessing the data. In most cases you are sure.

I am constantly stunned how people reach straight for MySQL or Postgres when flat text files with grep would work just as well and be much quicker to implement

Consider your process writing to the file and dying during write() - do you recover and repair the file after you reschedule?

All that said, doing this requires careful thought, and DB normalization when its discovered that there is a 1:1 relationship between rows in a table and a particular key/value table. So, its not something that should be taken to extreme, but I find it aids in quick development, as every time you discover you need to store another piece of data for some edge condition it doesn't require lots of DB normalization. Also, I wouldn't really consider making the "value" field a blob, rather a very limited int or string.

Most often I see TDD used as a couch for a really bad type system.

Can't think of any less functional languages than these.

If you need a template then think of scheme. It is the best example of a minimal functional language.

Pure functions help but the principles of generalisation are deeper than this.

Tests verify contracts offline. This will miss real issues.

Can you elaborate on that one? Sounds interesting to me.

When business rules are encoded in datatypes, it's easy to check that the encoding and the transformations are complete, and the logic can easily be mock-tested.

[1] http://degoes.net/articles/modern-fp

If it feels like translating into a foreign language, then it likely is that exotic or you are using a wrong language.

This can be more sophisticated, like "this function requires a sorted list" so lets make a sorted list type, or packaging things up into biz logic types (a cost type that contains an integer instead of just using integers), but you can catch a wide variety of errors with static analysis if you make your code and logic amenable.

We approached this first outlining the procedure and specifying the types involved, then outlining functions from each type to the next. You could essentially think of our types as tables, so our outline was f1: t1: -> t2, f2: t2 -> t3,...,fn: tn -> t_output (although not quite that linear.)

This let us split up work on the different functions across a team of 6 people. Specifying the input and output types was basically enough to make sure the functions were correct most of the time, and baking the interfaces into types enforced by the compiler made it easy to refactor & coordinate on changes when necessary. Feedback when we made an error was generally immediately available, because IntelliJ would highlight the function that produced an output value of the wrong type, or the compiler would catch it.

In contrast, if we had relied primarily on unit tests to check the functions, that would have made coordination more difficult, refactoring harder, and would have required us to either generate or acquire test data to feed through each function. But this architecture let us successfully build out most of the logic even while we had no access to real data & a different team was working on data ingestion.

Assuming you are talking about real types and having something like

  f1 :: t1 -> t2

  f2 :: t2 -> t3

  g :: t1 -> t3
  g = f2 . f1

Does f1 consume all input before f2 can run? Is it "streamed-through", like in `sh`, e.g.

  $ find /home/foobar | grep hs$ | xargs wc -l

I really like the concept and it works great, but for me it is simpler to apply to smaller constructs and I am still investigating how to apply it to more "business-logic".

g = f5 . f4 . f3 . f2 . f1

where f1 reads from s3 and f5 writes to a db. Then the implementation work mostly involved breaking these down further, e.g. f1 = h4 . h3 . h2 . h1, where only h1 is stateful and everything else is pure.

Spark is lazily evaluated, and in practice it will stream through many operations–f1 will generally not be done consuming the input by the time f2 starts, although sometimes we force it to for debugging purposes.

Lazy evaluation and the discarding of side effects make logging difficult, which is one of the downsides. There are various monitoring and debugging tools that help but it's still definitely harder than the single machine case.

But especially here, where we're talking about reducing complexity, it feels off to me. PostgreSQL and MySQL seem to me like incredibly complex packages. SQL, the language, is not easy to master either; most programmers I meet know mostly basics. On top of that, there's a long ongoing history of security malpractice.

When talking about reducing complexity, CouchDB and Redis are far easier alternatives, in my humble opinion, though they go slightly against 'use the tools developers know'.

SQL might take some getting used, but it is also not rocket science. It shouldn't take more than a week's study to master the basics. There is of course a lot of awful SQL code out there, exactly because most programmers don't even know the basics. You can do incredibly powerful things in it that would take 10x the code in an OO/procedural language. In my opinion dumping an ORM on top is also not the best way to leverage the strengths of an RDBM.

It is slightly ironic that you bring up security malpractice in the context of PostgreSQL, when in the next sentence you advocate Redis as a far easier alternative. As was recently in the news the Redis defaults were for a long time insecure (google for Fairware ransomware).

I agree. Unfortunately, the way I see people usually using them is pretty bad - you should not let ORM-generated stuff dictate your business model. Database is a database. A storage layer. Business objects will not map 1:1 to ORM objects. Approaches like "let's inherit from ORM class and add business-related methods", in my experience, lead to total disaster. One has to respect the boundary between storage layer and business model layer.

The problem I encountered in those projects is the mismatch between storage mental model and business mental model, which lead to explosion of crappy code (AKA technical debt). In particular:

1. the classes I need for business model may have initially mapped well to database tables, but over time they stop; business logic and model changes much faster than you'd like your DB schema to

2. since many things in AR can fire SQL queries, you have to keep in mind the workings of your database when doing almost every operation on your model; it's an abstraction leak

3. code shooting off SQL queries is randomly called from all over your codebase; it's harder to keep track of it and, if needed, optimize those queries

I like AR as a convenient API to get data from/to database, but given the point 1., I eventually learned to isolate AR layer as something below business model layer, so that the pattern is that business model is explicitly serialized and deserialized from database, instead of the database being coupled with the logic of your program.

Now I vaguely recall complaining about this before on HN and getting my ass handed back to me by someone who pointed out that these are all ORM n00b mistakes. I wish I could find that comment (pretty sure I noted the link down somewhere). Yeah, I admit - in those two projects I mentioned, we were all ORM noobs. So we've learned those lessons the hard way.

I don't disagree, in fact I'd go further and say that data and logic should not be coupled, but this is the Active record pattern which is far from the only way to use an ORM, most ORM's won't even support this pattern by default.

Key-value stores are "easy", but what I think isn't easy is to reduce your business domain to a simple key-value model without sacrificing promises and gaurantees offered by a good relational database system.

PostgreSQL and MySQL are very complex, but the complexity is entirely contained. Both are well-tested and reliable, so developers can deploy them without worrying much about them.

I would disagree about SQL being difficult to master. The basics are all most people need, and are not at all difficult to learn. The more advanced stuff (e.g. CROSS APPLY) is not necessarily standard across implementations, and can usually be replaced with application code.

>When talking about reducing complexity, CouchDB and Redis are far easier alternatives, in my humble opinion, though they go slightly against 'use the tools developers know'.

I can't say I'm that familiar with CouchDB, but Redis is entirely inappropriate for most SQL use cases. It's a key value store, and is not meant to do any sort of advanced queries.

Of course, if you regard sql as something you'd rather not "waste" time on, of course you're going to find those kind of subtle distinctions confusing - sort of like how people think css is difficult.

The more reasonably "complex" bits are the update visibility semantics, i.e. which transaction isolation levels mean what in various scenarios.

That's really complex, and it's truly somewhat unique to sql in that most alternatives simply don't bother trying to solve those problems at all - that can be a bad thing, but it is simpler.

A lot of other technologies drop complications like foreign keys which you won't miss when you start developing software but it gives guarantees you will miss sorely when you start seeing inconsistent data 6 months in.

Sometimes it is good to get a pickup truck ahead of time, but often a smaller less versatile car will suffice. But not quite a motorbike.

SQL is well worth its time to learn. It's a good DSL for relational data. Most programming languages used for regular code are not very convenient with relational data. As for its security issues, this is actually simple - one has to respect SQL as a real programming language with its own syntax and grammar, instead of resorting to idiocies like gluing strings together in an ad-hoc manner.

I'm going to use a car analogy here. Modern cars are incredibly complicated machines. They're also generally very reliable, thanks to about a century of development and engineering, and (relatively) inexpensive thanks to economies of scale.

If I want to transport myself to work on weekdays, and transport my girlfriend and maybe another friend on a weekend trip, and carry a bunch of groceries once a week, I can do all of that with a standard 5-person car. It isn't completely optimal for any of those tasks: it has more space than it needs for any of them, especially the weekday commuting. For my daily commute, I don't even reach highway speeds because I live close to work, so the engine is seriously overkill.

So I could have a custom-designed vehicle for each of these use-cases. Each vehicle would be a little less complex than my current car. But that's a lot to maintain, and would surely be far more expensive and less reliable, since each one is a one-off, requiring custom design and engineering, special parts, etc., and not benefiting from the economies of scale and engineering resources that a mass-market car gets. So instead, I just go buy a ready-made car and use it, and it works great and I'm happy.

Is PostgreSQL overkill for a lot of uses? Probably so. But it's designed to be used for all kinds of different tasks, and while it may not be quite as efficient for any of those tasks as some custom-designed solution, it's far more flexible, and it's readily available, plus it's benefitted from an enormous amount of engineering and debugging that a custom-designed solution would not. Things like CouchDB don't have nearly the number of users and amount of development, so while they may make sense for some tasks, the fact that PostgreSQL has more lines of code does not necessarily mean it's less reliable, in fact the opposite is likely true, just like an off-the-lot Honda or Toyota is likely much more reliable than some custom-designed car that someone built in their garage or some high-end limited-production exotic car like a Ferrari or Bentley.

The reason for using an off-the-shelf solution is because it's fast and easy and reliable. It doesn't matter if you're not making use of 80% of the features or capabilities. And software isn't like cars or engines; hard drive space is nearly free, and except for certain applications you're not likely to see a significant downside to just using a standard SQL database versus something more tailor-made. The main problem is cost (like with Oracle), but with PostgreSQL or MySQL this isn't an issue since they're free (and Free). It also helps that they use a standardized query language which makes them much more accessible.

Like most things in life there is no right answer, with tools like Terraform you can build a very complicated microservices system with not much effort, but only if someone on your team is experienced enough. If you're in a small team it's probably not worth the effort of learning the techniques and putting them in practice. We hate premature optimisation after all.

Agile methods are also useful. If you can't plan 2 weeks of work you can probably not plan 6 months.

When agile methods harden into branded processes and where there is no consensus on the ground rules by the team it gets painful. The underlying problem is often a lack of trust and respect. In an agile situation people will stick to rigid rules (never extend the sprint, we do all our planning in 4 hours, etc.) because they feel they'll lose what little control they have otherwise. In a non-agile situation people can often avoid each other for months and have the situation go south suddenly. In agile you wind up with lots of painful meetings instead.

Also I think it is rare for one language to really be "best for a job". If you want to write the back end of a run of the mill webapp, you can do a great job of that in any mainstream language you are comfortable in.

Hmmm. I was just thinking the opposite yesterday. I'm a performance engineer working closely with two teams. One doing Agile and the other basing on wikis and Adhoc in-person whiteboard discussions. I find the non agile team more productive, efficient and dare I say happy. The Agile based team makes me sit in on their daily scrum meetings. Although every one uses it to sync up on their dependancies, it just drags for an hour almost every day. I can visibly tell the devs walking out of the room spend more time worrying about "velocity" and "organisation of work" than the money making work that needs to be done. It almost feels like the agile process gives them "one more job" of picking the doable things from the list of stuff that needs to be done so they look better than their peers with better velocity.

Simply put, I was thinking if Agile is just not a good method when you can strive for good leadership and a healthy collaboration among individuals of the team?

Classic symptom of managers using agile as a (micro)management tool. Velocity, burndown charts, etc. are meant to be used by the team as a self-calibration tool. Managers do not get a say in what they think the velocity should be, either for the team or for individuals. If they do so, they create an incentive (let's be blunt, an overwhelming incentive) for the team/individuals to game them and that way lies madness.

(As an aside, the best response I've ever seen to this type of dysfunction is a team who simply decided to retcon the charts on the fly to make the work committed to match the work done. Management was happy that the burndown chart was right on target, developers were free to be fully productive instead of worrying about what their velocity looked like; it was a win-win solution all around.)

In large lines I agree with this comment. Micromanagement of Agile teams is detrimental. Implicitly, the message is that managers should leave their teams to work in peace?

The question I have: assume you are that manager. You have 5 agile teams working on 5 client projects. One team seems to get work done much slower than the other teams. What do you do? (And how does one actually track progress of an agile team to begin with? Story points can vary wildly across teams).

You've hit upon the key question: you want a progress metric that's in-line with productivity. IMO, assessing that comes down to evaluating whether functionality/code delivered (at a high level) per sprint is reasonable, evaluating whether the task breakdown the team is operating against is sensible and whether tasks are being accomplished in a reasonable amount of time relative to their difficulty, the skills of the person doing them, etc. In other words, the evaluation needs to be specific and include the circumstances: Saying "Why did implementing XYZ take longer than one would normally, even taking into account ABC?" is going to result in fixing the real issue whereas saying "Why is your velocity number so low?" is going to result in "fixing" the number.

That, in turn, requires the manager to either possess solid software engineering skills or have access to someone possessing those skills who can make the assessment in their place. And, yes, it's a lot more work. But, as has been amply documented, attempting to manage off a single number (a self-reported number, no less), simply doesn't work.

So long as it's one:many.

I will add that I was not on that team. Our scrum master, who was excellent, shielded us entirely from the management madness.

If they were really serious about "velocity" (for whatever reason; some are legit), they'd divide by man-hours, not weeks, anyway, and have actuals going back 3+ months (6+ is better) to baseline their sitrep before they started knob-twiddling.

There are either too many people in the room, or people are talking too much, and probably about the wrong things.

And, if they are doing standups wrong, I question how much else they are cargo-culting.

There is no one true way to organize development of software and generally shoehorning dogma without proof of value is counterproductive.

Standup daily meeting must be short and focused. Moreover, standup meeting is for developers only. No project manager, no customer, no QA team. Just developers talking about their problems. Everything else deserves it own separate, hour long meeting once a week or two.

Agree completely. When incredibly common complaints about a methodology are raised and the response is "you're doing the method wrong" you start to err towards dogma and a "No true Scotsman" approach to management.

Sticking to any technique, including agile, no matter what and with no modification is a symptom of a problem. Projects are unique, there's no one size fits all way to manage them.

The first thing we do after stand ups is have a bunch of quick one-on-one or small group meetings.

Then we got a new scrum master whose desired format was to talk about each item on the Kanban board each day, even the ones that weren't being worked on.

It should take 10m at most. Any issues (blockers, dependencies, etc) should be taken to separate ad-hoc discussions to let the team get back to work.

The standup performs an important social function of making everyone involved in the work of the team. People have to face each other, feel accountable to, and feel supported by the group.

And of course there's no room for synching up: The morning standup is meant to facilitate those synch-up meetings, synch-ups that don't have to involve everyone.

This is a fantastic description. And it goes a long way towards explaining why standup (even if I'm not directly involved) leave me feeling so uncomfortable.

I've been in well run Agile teams - and they're wonderful. I've been in badly run "Agile" teams and they're soul destroying. Either way agile is not the problem (or, I dare say, the solution).

I can see that the motivation is to avoid wasting the whole team's time on a discussion that only needs two or three people; but suppressing discussion can hurt too, as it stops people learning about tricky issues outside of their immediate work area.

It would be helpful to have some rules of thumb to show when you're doing Agile wrong. Probably those exist already -- anyone got a good link? And probably "too many people in the standup meeting" is a good rule of thumb!

Dragging on for an hour sounds absolutely awful. I'd even say more than about six people is too many.

I had one team of 10 that had a problem with our standups extending into half an hour once. We resolved that we'd make the standup one minute shorter each day. After a month and a half, we had it down to a one-minute standup (6 seconds per person). It was still useful, though a bit extreme - I'd target about 5 minutes for a 10-person standup (30 seconds per).

But we had a real agile guru on the team. We didn't do "Agile Methodology" exactly; we did Extreme Programming, and we kept tweaking it. Sometimes he'd say "let's try changing our approach in this way for the next two iterations, and see how it works out". We'd do the experiment, and keep the changes or not. We kept hacking and experimenting with the process, in a controlled way, but never in a "this is how it's done" way.

So if you have an "Agile is the one right way" person trying to run your team with big-A Agile, and he/she wants to do 30-person standups, you're probably in trouble...

The ad-hoc approach also sounds quite agile (at least with a small 'a'). It's certainly closer to Agile than to Waterfall, assuming they didn't do a big design up front before writing any code.

I think the ad-hoc agile approach can work very well with a good team. But Scrum fans always seem to warn against cherry-picking just the bits of Scrum you like and not using the whole process.

But of course. If you just cherry-pick and experiment, then you won't have any reason to pay an expensive expert to tell you how to do it right!

I'm a big Scrum fan (when it works), and my biggest takeaway is that it's exactly meant for cherry-picking and modifying. The best team I've ever been on was one where we were all using Scrum for the first time. We were constantly trying to mold it to fit us best, and it ended up looking nothing like the original model of Scrum. It was also the only time I've ever been on a Scrum team that did proper retrospectives, which I think is the biggest point!

Pretty much every other team has either ignored it ("Why do we need to discuss Scrum, it's in the book and laid out for us.") or merged it with the Review, so that managers, stakeholders, and people outside the team are involved in that. And no one wants to suggest changes or raise complaints with outsiders watching.

Too many people seem to read a book about Scrum, memorize all the concepts and rules and abide by it, without reading any of the justification behind it. If you swear we need story points, and they need to follow a fibonacci scale, but you can't tell me why story points are better than estimating hours, you're doing it wrong (and then points always get fucking conflated with hours anyways). If you understand that story points are just one way of estimating a task's effort relative to other tasks, and that relative estimates tend to be easier to make, and scale better with all the other estimates when things change, then you're allowed to make the call of whether story points are best for the team, or a different estimating system, or none at all. Even better than someone understanding that, everyone on the team should understand that and be able to weigh in.

This is one of the problems I have with these sorts of things. My company went Agile about two years ago, and lots of people like to rant about how much better everything is now and how much more productive we all are because of it. Except we actually have no way of knowing whether it made any difference at all.

But, looking at both teams from above, it feels like the non agile team is very simple and it works. The agile team is more complicated and works only on paper.

I still advocate agile for less homogeneous teams or in situations like other posts have highlighted but a team of more senior developers with a working process that is open to be improved (one of the cornerstones of agile) will thrive with less churn than when forced into a by-the-book agile process.

http://agilemanifesto.org/

Kanban board, prioritization, CI + CD, automated tests is probably about as much agile as most companies need.

In a daily scrum you cannot have conversations, its just everyone stating 3 things: What I was working on yesterday, what I will do today, and whether I need help today from someone. For a team of 10 people (a large agile team!) it should not last for more than 15 mins.

I guess other parts are broken too, if they dont even know how to do a standup.

Also extremely importantly is it brings you:

* Working tests. If you make changes and forget to or don't run all tests, the CI server will catch it and make you aware. You still have to write (useful) tests, of course, but that's a discrete problem.

* Entirely kills the excuse "Well, it builds on my machine". This means no undocumented dependencies, and the entire build is scripted.

* "Release builds" are a non-event. They just happen all the time, and your "release" is just the latest build that includes the changes you want and passes testing. This removes situations where there's only a limited set of people (eg: one) that can do a full release build.

Doing CI early on is much simpler. Aside from being beneficial from day 1, it is much easier to incrementally add to your build script/environment as needed than to try to create a script later based on a complicated manual process.

Not having CI is not nearly as bad as not using source control, but it's in the same ballpark.

The issue of people refusing to coordinate and working independently on two useless things for months at a time is an issue that transcends any popular software terminology. The role of a good manager is to see that units A and B are not in sync and resolve that. That's true in all disciplines. You can't attribute this to "agile" just because you meet every 2 weeks.

They use "agile" but I have budget and timeline constraints on every project.

Where it starts to be about how you conduct meetings then it tends to fall to pieces.

It's been that way long before agile methodology or microservices though. Complexity-for-the-sake-of-complexity EverthingHasToBeAnAbstractClass frameworks have been plaguing the software development business since at least the 1990s and I'm sure there are similar stories from the 80s and 70s.

It's hard to find a one-size-fits-all easy method for not falling into that over-engineering / over-management trap. I try to focus on simple principles to identify needless complexity:

- There is no silver bullet (see "microservices"): If the same design pattern is used to solve each and every problem there probably is something amiss.

- Less code is better.

- Favour disposable code over reusable code: Avoid the trap of premature optimisation, both in terms of performance and in terms of software architecture. Also known as "You aren't gonna need it".

- Code means communication: By writing code you’re entering a conversation with other developers, including your future self. If code isn't easily comprehensible again there's likely something wrong.

You sit a guy there 5 days a week for many years. He has to look busy, he has to do something with all of that time. He's not going to get paid if he writes the code in the most simple, concise, and straightforward way possible and then goes home until they're ready to make a new feature two weeks later. He has to sit around and make up something for himself to do.

Contrast with side projects. I have many simple weekend projects that continue to work well and provide their promised utility years later. Because you just write what you need and stop, you don't get sucked into the disastrous complexity spiral that every company-internal software project ends up as.

The other factor here is that people need some signal to say "I'm good at my job" (because no one can actually tell). That signal has to go to colleagues, superiors, and peers outside the workplace. People therefore invent artificial complexity or take intentionally convoluted approaches so they can sound fancy. In the most extreme cases, this is a conscious decision designed to block out "competitors" (colleagues). In many cases, it's a subconscious way to ego-stroke (and to mix in a little bit of variety per point one above).

This is especially true when a household brand like Google or Facebook pushes out some new esoteric thing; everyone wants to see themselves as a Google-or-Facebook-in-waiting and it makes it easy to pitch these things to the bosses, when the fact is that the kinds of things that work at large public companies like Google are probably not going to work in small companies.

I'd change this to "Write as little code as necessary, but no less." The problem with "Less code is better" is that some folks use that as justification to write clever one-liners that are difficult for other developers to read. That is not better.

That aside, I agree with everything else you said!

Seriously - Please, if you do this, stop it! You're just slowing everyone else down.

Some people call it "premature generalization". Relevant C2 page: http://wiki.c2.com/?PrematureGeneralization

My rule these days is that, in most cases, I shouldn't generalize before the same code exists in three places. Why not two? In my experience, things that are done twice aren't necessarily done three times. But things that are done three times are likely to be done a fourth.

I prefer this way of putting it over YAGNI. This makes it sound more like the trade-off it is.

I think I'm least productive in open source (again, in terms of felt accomplishment), because if one isn't the sole maintainer (like above), then it's a pretty safe bet that few changes take less than a certain baseline (eg. 1 hour) -- someone always has a nitpick, CI always takes it's sweet time, oh, did we discuss yet in which branches we wanna merge this? Ah, please avoid puns in documentation and comments. Do we want this? Can you write this differently, like ...? Did you manually test this or that scenario ...?

(Now this also has advantages in terms of stability, quality and consistency -- but it's also obviously far, far less efficient)

On the clock it's more like "Meh, change that and that, otherwise it's good, so merge it after these changes and tell ops to put it in prod"

It always amazes me, the instinct by engineers to overcomplicate things. They shoot themselves in the foot and curse when the inevitable subtle bugs start rolling in.

https://m.youtube.com/watch?v=ubaX1Smg6pY

Note that the laptop he is presenting on is not running Linux/Windows/OSX and that the presentation software he is using is not OoO/PowerPoint/Keynote. Instead, it is a custom productivity suite called "Frank" developed entirely by his team, running on a custom OS, all compiled using custom languages and compilers. And, the total lines of code for everything, including the OS and compilers, is under 100k LOC.

This is fine and everything, but it's bad self-promotion. If you want your bosses to give you a raise, you need them to think that you have a unique, difficult-to-acquire skillset and that it's worth going to lengths to keep you happy.

Unfortunately, modest behavior rarely results in recognition. Bombast is a very effective tool, and at some level, you always have to compete against someone.

The dilemma re-emerges as one asks himself whether it is right to sit by and allow the dangerously incompetent to ascend based on mind games.

My answer used to be "Yeah, I'll just go to a place where that doesn't happen". I no longer believe such places exist.

Having a lab and doing experimental stuff is great, but choosing to stake your company's products on it should be a much weightier consideration. In practice, we see that this weight is apparently not felt by many.

So far, I know about this: https://harc.ycr.org/project/

Hopefully, they're shooting for something like this: https://www.youtube.com/watch?v=gTAghAJcO1o

The whole vertical software stack sitting on top of the hardware of a PC is generally considered a massive towering best with layers of abstractions, and armies of programmers needed to implement and maintain each layer. To say that this does not need to be so, would be taken as theoretical, impractical nonsense without any proof. Which actually would be a valid position, because doing software is so hard that generally you can't guarantee will something work without actually doing it.

So, yes, to make that point, and to have it taken seriously, he really needed such an example.

There is also a more recent example of Arthur Whitney writing a C compiler in <250 lines of C. Remarkable how productive a programmer can be when he chooses not to overcomplicate.

2) Conway's Law applies in reverse here: If your organization consists of a lot of rather disjoint teams, then microservices can be quite beneficial because each team can deploy independently. If you're one cohesive team, there is not much benefit, only cost.

3) Depends. If you have a well-designed distributed system, it can be amazingly resilient and reliable without introducing much administrative overhead. (From my experience, OpenStack Swift is such a system. Parts may fail, but the system never fails.) There are two main problems with distributed systems: a) Designing and implementing them correctly is really hard. b) Many people use distributed systems when a single VM would do just fine, and get all the pain without cashing out on the benefits. See also http://idlewords.com/talks/website_obesity.htm#heavyclouds

4) Continuous integration was not meant to help with complexity. Its purpose is to reduce turn-around time for bugfixes and new features. If your release process is long and complicated, the increased number of releases will indeed be painful for you. Our team sees value in "bringing the pain forward" in this way. Your team obviously puts emphasis on different issues, and that's okay.

A REST API will.

Plus, once you've debugged a problem that involves crossing 5 microservice boundaries you'll start to wonder if it was all worth it.

Monolith is also a wrong (and somewhat derogatory) word to describe a non-microservice architecture. There's nothing monolithic about loosely coupled code running on the same machine.

I really think that microservices are a hack to deal with conway's law in large corporations. Operationally it's inefficient but it fixes a nexus of technical and political problems when the correct boundary is picked.

The only difference I noticed with respect to rube goldberg (what you call "microservices") systems and coupling is that tight coupling between components of rube goldberg systems was much more painful: particularly debugging across multiple service boundaries.

About that points. Scaling developments across many devs is a very difficult problems. It just doesn't scale.

A lot of organizations recruit/grow a lot of people and they try to get away from the human scaling problem by having them work independently/in their corner.

This allows people execute a lot of stuff... usually the same stuff 10 times, with little coordination or collaboration between them to the point it can be felt it in the resulting system(s).

Advice tends to be cherrypicked to suit an agenda they already have (with your example on microservices, the vast amount of resources saying they're very difficult, should be driven by a monolith first approach, and solve a specific set of problems is largely brushed under the rug).

I think because our industry moves so fast there's a fear of becoming irrelevant. Ironically companies are so scared of not being able to employ developers that they're also onboard with complicating their platform in the name of hiring and retention. I think this is down to the sad truth that most developer roles offer very little challenge outside of learning a new stack.

This is a gem observation from this thread. In my own tech sphere the first thing developers are talking about with each other is the new x,y,z lib or framework they're using to accomplish something relatively banal. There's still a lot of work out there that really boils down to basic CRUD and reporting at the end of the day, and developers naturally begin to invent complexities on top of that CRUD to make the work interesting and challenging. I'm absolutely guilty of this first hand.

I've found personally it also doesn't help that past work on projects e.g. large Rails apps that were never architected well turn out to be such nightmares to work on. The memory of the end state of these projects lingers with developers as they move onto the next piece of work, and they're inclined to say "no that doesn't work" and pick up shiny new-tech to do the old job instead.

As a side analogy: most small business construction jobs, e.g. building a timber frame house, don't involve the builders arriving on site and are stumped by the challenge of how to put up the framing for the bedroom walls - there's also very little challenge in these projects, yet the reward is in the completion.

I'd go so far as to say that _most_ work today (at least in startups) is building CRUD apps. The technology has changed, but the work hasn't. Inside of building CRUD apps in Rails, we now build them in React.

The thing is, there is so much other stuff to learn/fix. Where I'm working now most of my day is trying to understand the layers and layers of code they built. If they'd stuck to simpler code I could be learning more about the business and the users workflows. I could be improving the UI with that knowledge, I could be optimizing the business. Instead all my efforts go into understanding the code base.

Buzzwords can mean freaking anything. I've seen great Agile teams that don't look anything like textbook Agile teams. Microservices can be a total clusterfuck unless you know what the hell you're doing -- and manage complexity. (Sound familiar?) CI/CD/DevOps can be anything from a lifesaver to the end of all life in the known universe.

So yes, we are over complicating software development, but the way we do it isn't through slapping around a few marketing terms. The way we do it is not understanding what our jobs are. Instead, we pick up some term that somebody, somewhere used and run with it.

Then we confuse effort with value. Hey, if DevOps is good, the more we do DevOps, the better we'll be, right? Well -- no. If Agile is good, the more Agile stuff we do the better we'll be, right? Hell no. We love to deep dive in the technical details. If there aren't any technical details, we'll add some!

Software development is too complicated because individual developers veer off the rails and make it too complicated. That's it. That's all there is to it. Throw a complex library at a good dev and they'll ask if we need the entire thing to only use 2 methods. Throw a complex library at a mediocre Dev and they'll spend the next three weeks writing 15 KLOC creating the ultimate system for X, which we don't need right now and may never need.

It has nothing to do with the buzzwords, the tech, or software development in general. It's us.

I think keeping things as simple as possible and always going for that goal will increase velocity overall. Everything should be subject to scrutiny for promoting productivity and open to modification or removal. I know there is a balance where you have to increase complexity in a team environment but keeping friction as low as possible in terms of process and intellectual weight couldn't hurt.

The most productive place I've seen so far is a huge athletic brand I worked for where they kept teams at max 5 people in mini projects. This forced the idea of low overhead and kept the scale of management needed small. The worst place I worked for in terms of unnecessary complexity is a well known host, although it is the best place to work in terms of people, hired offshore that has a one-size fits all mentality and layered in as much shit as possible to slow down development to a mud crawl. I don't buy into process over productivity.

Default form elements with some basic, nice styling to fit your theme? Form done in one hour. Special snowflake version of the same thing from the design team, which has no idea what the platform can and cannot easily be made to do, but the client is absolutely in love with? Two days, a third party dependency or two, some extra environment-specific bugs to track down later, and generally increased fragility (so more time lost in the future). This has slowed you down now and increased the resources required for the project indefinitely. But the client looooooves it.

Support Android pre-5.0, at the cost of 20% more development time, a pile of extra bug reports, an uglier, harder-to-maintain codebase, and a much much longer testing cycle, for a side project? Hell no. Client says that will cost them $4 million/yr not to support those? Ugh. FINE.

And so on, and so on.

Where I might agree with you more are on points 3 and 4: 'Advanced reliability' and 'Microservices'. While I have no doubt that these are useful to solve specific problems, I think as a profession we tend to over-estimate the need for these things and under-estimate the costs for having them. To me this implies that there needs to be a very clear empirical case that they support a requirement that actually exists. I'd also make the argument that the drive for microservices within an organization has to come from a person or team that has the wherewithal to commit resources over the long-term to actually make it happen and keep it maintained. (ie: probably not an individual development team.)

I don't agree on point 4 though - CI can be something as basic as running a monolith's tests on each commit, which makes sure that builds are reproducible (no more "works on my machine").

When you practice good seperationof concerns, specific choice in different areas can be more easily fixed later. It requires having decent APIs and being thoughtful on the interaction of different components, but it helps immensely in the long run.

Microservices are one way to practice seperation of concerns, but it can also be practiced in monolithic software as well, by having strong modular systems (different languages are stronger at this than others).

1 - Do not pick a new language for an urgent project. Do look at them when you have some leeway.

2 - Yep.

3 - There's something wrong with your ops. That happens often, and it is a bug, fix it.

4 - If CI is making your ops more complex, ditch it. If less complex, keep it. In doubt, choose the safest possible way to try the other approach, and look at the results.

5 - Do not listen to consulting experts, only to technical experts. The agile manifesto is a nice reading, read it, think about it, try to follow, but don't try too hard. Ignore any of the more detailed methodologies.

Understanding the possible benefits and drawbacks of any solution is important. It's important in whether or not that solution is selected, but also to make sure that the implementation actually delivers those benefits.

It's very common in our industry to use "best practices" without understanding them, and therefore misapplying the solutions.

Facebook == respected tech brand == someone I should copy. The end.

Guy I know uses Cassandra == developed by hot tech brand Facebook == cool by mental association with Facebook.

Guy I know uses MSSQL or Oracle == developed by crusty old Evil Empire Company that cool people don't want to work for == bad.

Conclusion: We must use "big data" so we can be like the cool people -- err, because we really have some big data.

This doesn't sound like the outcome we'd expect from technical people making these decisions, but we can obviously see that it's what we're getting.

1) Without a unified technology stack and a common framework we would not be able to build and maintain our applications. We decided on C# as it works best for us. Currently we are 5 developers. Not a single one of us has ever written a line of C# code before entering the company - learning the language from ground up enables us to pick up patterns that our colleagues who joined the company earlier found to be best practices.

2) If you are not introducing a whole new stack with every micro service that you develop the devops costs are quite low.

3) I agree with you on that - I think redundancy always introduces more complexity. However there are systems that handle that job quite well (e.g. SQL Server). For application servers we use hot-spares and a load balancer that only routes traffic to them, when the main servers are not reachable. This works for us, as all our applications are low traffic applications.

4) Continuous integration works brilliant for our unified stack. In the last two years we went down from 1d setup + 20min deploy to 10min setup + 20s deploy.

5) We use agile methodology whenever possible and it works like a charm. However we had a lot of learnings. Most recent example: Always have at least one person from all your target groups in any meeting where you try to create user-stories.

Planning our software architecture has been a key element in my teams success and I do not see a point where we are going to cut it.

2. Microservices vs monoliths is a see-saw. You build a monolith, find it's a brittle, incomprehensible hairball, and you break out microservices. You build microservices, find that operational headaches are killing you, and start consolidating them into monoliths. Which kneecap do you want the bullet in?

3. Fix what breaks.

4. Continuous integration is vital. But it needs to be evolved along with the system. There's this thing I say... "Have computers do what computers do well, have humans do what humans do well". Handling complex and repeatable behavior (i.e. builds and test suites) should absolutely be automated as much as possible. Think continuous integration sucks? Try handing it off to humans for a while! You'll learn whole new levels of pain.

5. All process is about (or should be about) specific, discrete communications issues.

Funniest thing I've heard all day!

I occasionally refer to the final steps of a project as "bayoneting the wounded" too.

Rob Gingell at Sun stated it as a form of uncertainty principal. He said, "You can know what features are in a release or when the release will ship, but not both." It captured the challenge of aspirational feature development where someone says "we have to have feature X" and so you send a bunch of smart engineers off to build it but there is no process by which you can start with an empty main function and build it step by step into feature X.

That said, it got worse when we separated the user interface from the product (browser / webserver). And you're rants about microservices and continuous integration are really about releases, delivery, and QA. (the 'delivery time' of Gingell's law above).

These are complexities introduced by delivery capabilities that enable different constructions. The story on HN a few days about about the JS graphics library is a good example of that. Instead of linking against a library on your computer to deliver your application with graphics, we have the capability of attaching to a web service with a browser and assembling on demand the set of APIs and functions needed for that combination of client browser / OS. Its a great capability but to pull it off requires more moving parts.

95% of the time, a language that your developers are familiar with is the correct tool for the job simply for that reason! There are cases where it is not the case but those involve special case languages and special case systems. If you don't know what special case means then you're situation is almost certainly on that list.

- 2) Avoid microservices where possible, the operational cost considering devops is just immense

"If your data fits on one machine then you don't need hadoop ..." Same thing applies here. Microservices have place and putting them in the wrong one will bite you bad.

- 3) Advanced reliability / redundancy even in critical systems ironically seems to causes more downtime than it prevents due to the introduction of complexity to dev & devops.

Then there's probably something wrong or limited with the deployment that needs to be reviewed (2 node when you need a 3 node cluster, bad networking environments, etc.) If you have a reasonable setup with solid tech under it, deployed per specs then this should not be true. If, on the other hand, something is out of whack (say running a 2 node cluster with Linux HA and only a single communication path between them) you're going to have problems and the only way to fix them is to get it done right.

- 4) Continuous integration seems to be a plaster on the problem of complex devops introduced by microservices.

I'm not sure about this but, if your deployment system requires CI you have a problem. An individual, given hardware and assets/code, should be able to spin up a complete system on a fresh box cleanly and in a reasonable timeframe. (Fresh data restores can take longer of course but the system should be runnable barring that.) If this requires (i.e. it can't reasonably be done manually) something like an CI script or ansible/chef/etc. script then you're deployment process is probably too complex and needs to be re-evaluated.

- 5) Agile "methodology" when used as anything but a tool to solve specific, discrete, communications issues is really problematic

Agile is commonly used to gloss over a complete lack of structured process or a broken. Even with Agile there should be some clean process and design work that goes into things or you're hosed.

1. Writing code while using as many useful libraries and tools as possible to avoid recreating wheels

2. Continuous integration set up early on to handle the menial work and to let me concentrate on 1.

3. Constantly evaluating and researching what technology is available and newly appearing to give me an edge, because having an edge is never a bad thing in this field.

Agree with some of what OP said, especially with methodologies become hindrances and HA tools becoming points of failure.

If you write code like that, you get many of the benefits of unit tests, whether or not they are actually written.

Perhaps it's a good idea to write a test harness (e.g. larger integration tests) for old so that you have a reasonable chance of catching it if it becomes broken, and focus on writing new code in a testable fashion.

It's too easy to fool yourself into thinking you've written testable code.

Proper unit testing also complements integration testing in that corner cases can be handled at the unit test level, therefore reducing the amount of integration test code which arguably is much more brittle, runs slower and more complicated to write.

Only case where I'd support "unit tests" as typically practiced (small units, isolated functions/classes) is around core competence (defined as narrowly as possible). But then I'd argue that this functionality should be put into a library anyways, which is used by products codebases. And then the tests are tests for the functionality promised to the products.

I guess I shouldn't have used the word 'brittle', but this is what I was thinking of.

And of course, I think unit testing anything and everything is absurd and not a good use of developer time.

*Logical meaning only test for one thing, not a single assert statement. So testing for null and testing if a value is set is fine, but testing if 10 values are set correctly is not.

The discipline of unit testing forces me to think about what risks I'm introducing with my new code.

Unit tests drive better design - smaller classes, looser coupling, better separation of concerns, functions that don't have side effects.

Best of all, unit tests reduce regressions. I can't count the times my test suite has prevented me from introducing a bug in my app.

I can refactor code with much more confidence than if I did not have 400 unit tests checking my work.

Most recently, these tests proved their worth when upgrading my app to Swift 3.0.

CI can be a way of enforcing the simplicity of the others - it can be a way of tunneling the build process into assuredly straightforward steps and preventing individual team members from arbitrarily (or even accidentally) adding their own complications into build requirements.

Other than that, I think you are definitely on to something here.

It proposes how to write C++-ish (it's an extremely minimal subset of C++ proper) code in a mathematical way that makes all your code terse. In this talk, Sean Parent, at that time working on Adobe Photoshop, estimated that the PS codebase could be reduced from 3,000,000 LOC to 30,000 LOC (=100x!!) if they followed ideas from the book https://www.youtube.com/watch?v=4moyKUHApq4&t=39m30s Another point of his is that the explosion of written code we are seeing isn't sustainable and that so much of this code is algorithms or data structures with overlapping functionalities. As the codebases grow, and these functionalities diverge even further, pulling the reigns in on the chaos becomes gradually impossible. Bjarne Stroustrup (aka the C++ OG) gave this book five stars on Amazon (in what is his one and only Amazon product review lol). https://smile.amazon.com/review/R1MG7U1LR7FK6/

This style might become dominant because it's only really possible in modern successors of C++ such as Swift or Rust that have both "direct" access to memory and type classes/traits/protocols, not so much in C++ itself (unless debugging C++ template errors is your thing).

http://www.vpri.org/pdf/tr2012001_steps.pdf

"If computing is important -- for daily life, learning, business, national defense, jobs, and more -- then qualitatively advancing computing is extremely important. Fro example, many software systems today are made from millions to hundreds of millions of lines of program code that is too large, complex and fragile to be improved, fixed, or integrated. (One hundred million lines of code at 50 lines per page is 5000 books of 400 pages each! This is beyond humane scale.)

What if this could be made literally 1000 times smaller -- or more? And made more powerful, clear, simple, and robust? This would bring one of the most important technologies of our time from a state that is almost out of human reach -- and dangerously close to being out of control -- back into human scale."

...and of course if you haven't seen it, you'll want to check out the Forth guys who want to do everything with 1000 times less code:

http://www.ultratechnology.com/forth.htm

Microservices work fine in some situations, agile works fine in some situations, but until you find that you are in one of those situations trying to bend your deliverables to meet a sprint-cycle or some other nauseating jargon will cause, as you put it, over-complication or just poorly targeted effort. (It can also cause enough stress to dramatically affect your health, I know better than most)

Those moments of solidarity between product and effort are real gems that I've only recognized in hindsight.

First focus on identifying the primary job to be done: build a valuable piece of software with as little effort as possible given your current team and existing technology.

Second, consider how valuable the existing software is and whether it really needs to be rewritten at all. Prefer a course that retains the most existing value. It is work you won't have to repeat.

Third, choose tools that maximize the value produced per hour of your team. CI, Devops, Microservices, Languages all promise productivity and reliability benefits but will incur complexity and time costs. Choosing the right mix is part of the art of software management.

We've been burned by the microservice hype, and it took a while for us to realize that most of the touted benefits are for larger organizations. These "best practices"" rarely include organizational context.

My major issue for startup software development is that often software is developed too discretely - with a utopian 'final version' in mind. Developers don't think holistically enough - they focus on details at the expense of design. "current accepted wisdom" is intangible, ever shifting, whereas the failure of a system is very real and can lead to loss of income etc...

Lots of start up companies don't design systems with humans in them, they write code as if it was a standalone thing - they often leave out the human bits because they are hard to evaluate, measure and control - variety of skill, ideas, approaches, mistakes, quality of life etc.

In my experience, this variety (life) often comes back to bite companies that can't handle eventual variance because of poor system design - not because of a choice of platform / provider / software etc.

I have been reading a lot around the viable system model (VSM) for organising projects. It seems to fit with what my view on this is. I am trying to implement a project using this model currently.

https://en.wikipedia.org/wiki/Viable_system_model

E.g let's make an online shop.

It has browsing, purchasing and admin sections.

Browsing is simple. Query dB and show html. It's probably the most used as well and needs to be reliable. Having as different service means admin section could break while users are still able to browse. Same for payments. Sometimes it's crazy complicated. I think of microservices as big product feature boundaries that can work independently. A failure in one doesn't affect the other.

Continuous integration: once you have your tests, and some auto deploy scripts, you have an engine. You push code, tests auto run, a live staging is created for latest code, you play with it. Looks good? Merge with Master. It's deployed to production. The idea is deployment is effortless and you can do it multiple times a day just like git push. Tests just don't only have to be unit. We run integration testing features on dummy accounts periodically from different regions in the world on production. This means you are alerted as soon as something breaks. Fast deployment and great telemetry mean you can always revert to last known good state easily.

Investing in tests is a pain but it pays off in the long run. Especially if you have other developers working on same code base.

Just don't over do it. I believe these ideas came from pain developers actually faced and they used then to solve it. If you're not feeling the pain or won't feel it then you don't need the remedy.

I've had my fair share of cases where I ended up implementing something needlessly complicated, only to later realize my approach was terribly misguided. I'd like to think I'm slowly improving on this as time goes on.

The software world has a big discoverability problem. Even though I know there's probably prior art of what I'm working on, I don't always know where to look for it.

Imagine the solution space as some multidimensional space where there is somewhere an optimal solution. The dimensions include the habits of your programmers, the problem you are trying to solve, and the phase of the moon. Microservices, a special form of redundancy, continuous integration, agile development are all extreme solutions to specific problems. Solutions which are extreme in that they are somewhere in the corner of your multidimensional solution space.

They are popular because they are radical in the way they conceptualize the shape of the problem and attempt to solve it. Therefore they seem like optimal solutions at first glance when really they only apply really well to specific toy models.

Take e.g. microservices. Yes, it's really nice if you can split up your big problem into small problems and define nice and clean interfaces. But it becomes a liability if you need too much communication between the services, up until the point where you merge your microservices back together in order to take advantage of using shared memory.

Don't believe any claims that there is a categorically better way to do everything. Most often, when you see an article about something like that, it is "proved" by showing it solves a toy model very well. But actual problems are rarely like toy models. Therefore the optimal solution to an actual problem is never a definite answer from one of the "simplified corner case scenarios" but it is actually just as complex as the problem you are trying to solve.

Certainly the language your developers already know is better than one they don't, all things being equal. But your rule is way too simplistic.

2) Of course. Avoid every complex thing where possible.

3) This means the cost/benefit ratio was not considered closely enough when planning these features. Again, avoid every complex thing where possible.

4) this is a strange one. Most people doing CI are not building microservices. CI is really more about whether you have different, independently moving pieces that need to by integrated. Could be microservices, could be libraries, could be hardware vs software. If you only have a single active branch everyone's merging into regularly, you're doing CI implicitly. You just might not need it automated.

5) take what you can from the wisdom of agile, and then use your own brain to think. And don't confuse agile with scrum.

    * Was the "best tool" what the devs thought it was?
    
    * Was it something they would hate using? Say, Java for Perl devs?
    
    * Was there a steep learning curve? An obscure language?

3) You (or someone) must know how much system failure would cost.

4) CI can help with your devops, but its main point is to help with your software quality. See #2.