The article title gives me flashbacks to when I had to work on a project that was all-in on AWS serverless stuff, and we had a single dev environment (there could be only one person at a time working on a specific service), that was also used by other teams (so, many times when we tested stuff we received complains that it was broken because other teams' services were pointing to ours).
And no, there was no way to test locally and "no time" to even try.
There was also a test/staging environment that required waiting a few days between the moment you request a deployment until the moment it was actually deployed.
And the best part, all environments had very different Terraform files, so there's some stuff you could only test when you deployed to the staging environment, and some stuff you could only test when deploying to production. Each deployment with a waiting time measured in days.
"Deployment speed" is mentioned there. Many devs forgot (or never knew in the first place) that local development used to be the default and that the waiting time in the cycle "write code, run it, find a problem, write code" should be measured in seconds, not tens of minutes or hours (as is usually the case when devs don't have a local environment), or even days.
This is another (from a long list) hidden cost of complicated "webscale" architectures (not just serverless, but often the case for the plain microservice architecture).
> the waiting time in the cycle "write code, run it, find a problem, write code" should be measured in seconds, not tens of minutes or hours
I've run across developers who have never seen this in their career; they simply don't believe it or write it off as another old-timer story about punch cards. The bureaucracy of modern software is so pervasive that it's ruined our expectations of basic things like being able to run the software you're developing. That really shouldn't be controversial, yet here we are.
I worked at a “serverless” shop and ran into people who would actually edit code in the AWS console. They’d then copy-paste into their local editor before committing to git (usually. Stuff sometimes got overwritten!) It was baffling that no time was spent on local environments.
Is this how serverless really is? I just inherited someone's lambda project and basically I have the same experience. Like no testing, just deploy to production and see how it goes. Idk how this didn't set off red flags to anyone.
Not always. Check out how Oracle Cloud does it. It's a hosted version of an open source stack called fn, which you can run fully locally via a simple CLI tool.
Sadly, yes. You can build an app that runs both locally and on Lambda / serverless. I've done it and seen it done. However, in my experience, this is generally beyond the capabilities of most individuals. It's more often a mess than not.
Yes it's a Kafka-esque soul-deadening trial. Either you: use some OSS tool that emulates AWS infra locally (never exactly, and always missing features), run the aforementioned tool in a container because why not (looking at you, GitHub runners), or simply rely on manual integration testing to somewhat emulate the environment (and provision and point to blackbox cloud resources when you have a lot of moving pieces).
The list of companies the devs surveyed came from are all in the same subculture of the software industry. It would have been more interesting (and more meaningful) if there was a more representative sampling.
> …the biggest thing was we didn’t have good tests or good types, so I had to do a lot of work to do e2e testing of stuff whenever I wanted to ship stuff
Shouldn't solve that one by itself after a couple iterations? If one's really testing what they wrote + surroundings because there were no tests, there will be tests over all of the codebase very fast.
IME, it just becomes about: "I shouldn't have to do all this to make my one little change."
You need the team on board that there could be unexpected slowdowns, due to the scope of the work being greater (or I guess you can look at test coverage during planning stage).
I've been on one team that had the philosophy: "if we touch a file, all the technical debt becomes the responsibility of the team implementing"
Not to say it was perfect, because if people skip technical debt once, they're likely too again later.
However, so many managers in some fields would get all wound up about the potential for "unexpected issues", that we lost site of the big picture. This is how change works efficiently.
Since the work was organized by file it actually becomes common sense to make everything right in the file, including in some cases unrelated changes to meet static (lint) analysis (though of course this leads to even more testing work).
If it was something small, it takes so much more effort to plan a cleanup story, and execute that. Additionally, it's highly likely to be de-prioritized. Basically you want to clear the sink after you start doing dishes.
Lot of BS excuses in there. The ones about nitpicking in reviews too, if the comment is a tiny nitpick then surely it's trivial to fix? I regularly fix a whole bunch of minor things people comment on in minutes, it only takes long if it's a big change.
You can also work on other stuff while waiting for reviews. Similarly one guy complained about pipeline runs, like what? Does it take hours? Are you just waiting for it to finish? Are you not running tests locally first?
At one former gig we pulled telemetry from devs machines and lo and behold - the biggest complainers ran the pipeline twice a week or something. At this point i am convinced surveys is the worst way to go about “measuring” productivity. People are just going to tell you what they think you want to hear mixed with some self-serving bs and you will end up pouring resources into fixing the wrong things
Most of these complaints are from people that are essentially causing the problems they complain
About. They just can’t see past “must write garbage code as fast as possible”
Writing docs? Waste of time
Reviews? Waste of time.
Usability to other engineers? waste of time.
Waiting for CI because they can’t be bothered to test thing locally first? Waste of time.
Stakeholder approval? lol waste of time, we don’t care if people want the software we write.
Having any idea how anything works outside of whatever tiny thing being worked on? Waste of time.
> Most of these complaints are from people that are essentially causing the problems they complain About. They just can’t see past “must write garbage code as fast as possible”
+1. The article summarizes well as: reasons not to ship code asap.
Do it right the 1st time. Failing that: make sure any fixes absolutely, 100% fix the problem, are well tested & don't cause regressions.
Isn't [determine requirements], running builds, QA etc supposed to take time? Is development velocity even a goal worth striving for?
Whatever you kick out the door, could be on others' systems for ages.
I think our programming languages haven't caught up to an ideal notation of semantics of computer system behaviour at a macro level.
The design patterns (Gang of Four etc) and software architectures that are so complicated they require extreme discipline to keep understandable. (And cognitive load)
Which pattern did you find complicated? Most of them just come down to "wrap it/them with a new object to encapsulate the new/shared state or provide the new behavior". The rest are probably made obsolete by modern programming languages.
This is a very interesting article, thank you for the research!
I am very happy to work solo to prevent as much of these things as possible.
Everyone is using Typescript these days it seems instead of plain JavaScript but the 10 seconds + compile times and the extra layers in debugging make it a pain for me instead of a benefit. So I am using ES6 with Rollup. Rollup is great and my only dev dependency really. However great it is, if one day I could do without, I would :-)
The article title gives me flashbacks to when I had to work on a project that was all-in on AWS serverless stuff, and we had a single dev environment (there could be only one person at a time working on a specific service), that was also used by other teams (so, many times when we tested stuff we received complains that it was broken because other teams' services were pointing to ours).
And no, there was no way to test locally and "no time" to even try.
There was also a test/staging environment that required waiting a few days between the moment you request a deployment until the moment it was actually deployed.
And the best part, all environments had very different Terraform files, so there's some stuff you could only test when you deployed to the staging environment, and some stuff you could only test when deploying to production. Each deployment with a waiting time measured in days.
[thousand yard stare] So much time wasted.