Just another reminder of the absolutely terrible state of our industry. Imagine anyone in any other industry taking this ridiculous stand. "Oh we don't fit roofs on Fridays, in case our roof leaks and we need to bring the roofer back in on the weekend". "We don't do Friday wisdom tooth removals, would hate for the dentist to have to fix a problem in the weekend". "No we don't fit tyres on Fridays, our tyre guy doesn't want to fix your tyre problem on the weekend".
The problem is the bugs, not the Friday releases. This industry is so utterly depressing to me. We get paid like kings to deliver like monkeys. Clown industry.
The consequence of a bad deploy can be millions of dollars for a client or business. Depending on the industry, it can be a million dollars in the course of an hour. If a dev team works Monday to Friday, then it's likely they're going to be harder to reach and coordinate with during the weekend. It usually costs less money to wait an additional business day than to get a team of engineers in on a weekend to try and fix a problem in production.
If a roof leaks on a Sunday and the roofer doesn't work Sundays it's going to be similarly difficult to get the roofer to come and fix it. The difference is the order of magnitude of money lost for a leaky roof vs a software fuckup at a pharma company or a business conglomerate.
It's ultimately the responsibility of the developers to test their code sufficiently so that there never is a weekend problem to begin with, but the reality is there will always be bugs that crop up in production, and you can only control that so much. You can usually control deploying on a Friday though.
I really don't agree with any of your examples here, or the combativeness of your comment.
Roof work is usually a fair bit more deterministic than software.
For car work, it's not uncommon where I am for the tire guy, the vehicle inspector guy, or the guy who knows how to work on my brand of car to have unavailability.
My dentist has scheduled work earlier in the week with the explicit intent that if something goes wrong, I can be seen quicker. I appreciated this gesture, even if it wasn't exercised.
Roofing isn’t unskilled, but building codes make it sort of a commodity. If you just invented a new type of roof, I would not want you to install the very first one on Friday night and then send home every available expert to party or travel or whatever for the next two days.
The core of your argument is that it's the worker deciding this, and that they don't want to fix issues over the weekend.
That's not true.
It's the business deciding that they don't want to ship bugs on Fridays and then take the hit for customers having a bad experience over the weekend if the issue can't be resolved. What if you let someone deploy on a Fri and then they go skiing for the weekend? Sure, someone else can step in, but maybe it takes them an extra 2-3 hours to solve the problem. That might cost you a renewal.
Indeed, it's the business making the (IMHO poor) decision that the best way to address this issue is more manual oversight, and preventing release at certain times.
There are often other, better ways to address the underlying issue; "no releases on Friday" is a band-aid fix.
First off, we're still saying we deploy 16 days out of the month. I'm sorry that's not good enough for you? Do you want cookies from management for deploying every single day and on christmas eve? We'll host an extra pizza party in your honor (no bonuses this year, even though you did 33 deploys on Fridays).
Not to mention I worked at places that deployed twice a month, or deployed once a month, only with CTO intervention, which took four+ hours and multiple devops and engineering people doing manual shit, even with Kubernetes etc. So individual engineers deploying by themselves 16 days out of the month is "manual oversight" and "poor business decision". what a joke.
Also, what, a band-aid fix? I wouldn't work at a place that made me deploy major stuff on Fridays, first off. I have a life, so I'll take my bandaids, thanks. You think because people write tests and follow a spec that you can deploy MyBigFeature on a Friday without issue? absolutely not.
There are befits to small batches, short cycle time and completing at will.
They are nothing do do with what management wants or rewards.
I'm sorry to hear that you have only worked at places that are unaware of or unwilling to do good practices.
Your commentary on the whole is defensive, ignorant, rambling, ill-tempered and borderline insulting. It lacks the basic respect necessary to engage meaningfully, rather is more likely to create conflict. No further response is needed, thanks.
> There are befits to small batches, short cycle time and completing at will.
That has nothing to do with deployments on Fridays.
> you have only worked at places that are unaware of or unwilling to do good practices
I didn't say that. I also worked at places that did small deploys daily. But we still didn't deploy on Fridays. I deploy with my own projects, to production, many times each day. But I still try to avoid Fridays.
> No further response is needed, thanks.
I disagree. You complain about the industry as a whole but can't deal with basic criticism to back it up.
> Just another reminder of the absolutely terrible state of our industry. Imagine anyone in any other industry taking this ridiculous stand. "Oh we don't fit roofs on Fridays, in case our roof leaks and we need to bring the roofer back in on the weekend"
Agreed, the industry operates on a much more janky level than many other industries, but this syndrome is by no means limited to the field of software development.
If your roof is functioning perfectly well, fixing a non-critical problem on a Friday is less safe than fixing it on a weekday because if something goes wrong the next day, it's better that the "next day" be a day when the roofers are available. All other things being equal, I'd say fix your roof on a Monday so you have the next four days to call the roofers back if things go wrong.
It's the same reason you're safer booking an early flight, or scheduling surgery early in the day: if something goes wrong, at least you will get bumped to later in the day. If your originally scheduled flight or surgery was later, and something goes wrong, you will likely get bumped to the next day.
Obviously this rule doesn't apply to emergency fixes: if your roof has a bad leak, and it's a Friday, and you know it's going to rain on Saturday, you're still better off having it fixed on Friday.
Actually what's happening in those examples is that issues DO happen on the weekend, the customer just suffers or calls someone else to patch things up till it can be addressed correctly. Difference with tech work is that your employer typically suffers.
The thing is that the one roof doesn’t affect all your entire customer base.
That isn’t quite a fair comparison to make due to the nature of software scale, but I can relate to the sentiment.
I don’t think this take is complete unless you include complexity and the risk of unintended consequences. We pretty much understand roofing, but software doesn’t struggle with the chains of the physical world and the base case can be significantly more complex.
It depends on the type of software. A B2C SaaS that has continuous small releases with easy incremental deployments and rollbacks is incomparable to a bespoke B2B release that integrates with dozens of the customer's other systems.
The latter are typically deployed over the weekend with multiple companies/teams on standby as unexpected interactions are at a certain point of complexity a given.
Your quality aim is for instrumentation and verification to rapidly detect and access and triage issues since assuming you can hit 'deploy' and walk away because your code is perfect (delusional) ignores that 95% of the systems you integrate with aren't even under your control, and 'it worked in test and was signed off' is not an acceptable answer for a client's production system being down on Monday.
There are three maturity levels, do not confuse them:
0: We ship to production whenever, why not? If stuff breaks, oops. People make mistakes.
1: We have processes in place to avoid bad releases. These include such measures as people agreeing to avoid Friday releases, and requiring manager approval of releases.
2: The processes are automated, and high enough quality that we can be confident that clicking the "go" button" will either result in the build/test/deploy pipeline halting due to a problem, or a good release going to production. We have no issue with Friday releases, or releases whenever.
Allen Holub is broadly correct. And if that author's "bigger release" happens a lot, then release size is more the issue.
3. The processes are automated, and high enough quality that we can be confident that clicking the "go" button" will either result in the build/test/deploy pipeline halting due to a problem, or a good release going to production. But we still prefer not to release on Friday, because no CICD methodology is perfect.
There’s another maturity level: blah blah blah, but our CICD setup and overall code quality has long proven itself to be of such quality that we as a team / organisation have opted to assume this risk.
My real point is that, at a point, reasonable people can disagree about this, and calling these “maturity levels” strays further from reality and closer to the usual pass-ag nerd habit of mistaking personal preference for absolute objective truth.
I've seen all three in use, and while agree that "maturity level" is not "absolute objective truth" - that's your overstated reading, but IMHO dismissing them as mere "personal preference" is also wrong in the other direction.
These are not personal - they are are the product of the interactions and agreements of a large number of co-operating people who look for ways to deal with release quality concerns with instruments that they are familiar with: signoffs, manual gates, controls; vs. automation and small batches.
Also dismissing them as "mere preference" implies that the outcomes will be broadly similar, in my experience this is not at all so. The industry has uncovered ways to proceed both fast and safely, but these are not widespread, partly because of the myopia that they are mere "preferences".
Same for me. I think we are all talking like we work on the exact same product. I work at an agency with many codebases, from different clients. Some we inherited that don't yet have good coverage, some with notoriously brittle third party integrations we can't truly test until it is in the live data environment, and some with clients who are very unresponsive and if we planned a Friday deploy the chances of a no-go because we couldn't get sign off or testers is pretty high. It is just all around easier to do midweek deploys in such an environment.
Do I envy the fully automated Friday deployment teams? Sure! But this is my commercial reality, and the pragmatic approach is to not deploy on Fridays.
It sounds like you're deep into scenario 2. The mention of "sign off or testers" is a dead giveaway. "testers" being people not automation. likewise "sign-off".
It also sounds like while you know that better exists, the "commercial reality" is "not on this project". I've been there too. I don't see this as being in conflict with what I said above.
At my workplace we release small fixes on Fridays, but prefer to wait until Monday.
I don't see how we could trust ci/cd to catch all errors. Tests are great at detecting expected errors, but it's not possible for us to get 100% confidence.
We also need some sort of tradeoff. Creating a perfect system with few bugs take a lot of time and resources that a startup does not have. We accept that our releases may include bugs, because we can respond to angry customers quickly.
In our B2B industry, most of our customers sales happen during weekends. A bug has a much higher financial impact.
A few years ago, I was working on a legacy system that required preparing changesets for a release by hand. There was no version control. The entire release workflow was manual. The release sizes were massive. During those days, it was given that a release would take an entire day. We were averse to Friday deployments then.
Fast forward a few years, we now have version control, a proper CI/CD with rollbacks for failure and small well-scoped releases. Cutting down the release sizes had two positive effects: easier rollbacks and, quick and precise feedback about the release's status. These days, we release things whenever they are ready and Friday is no different than any other day.
Completely agree. Also, we have significantly lower usage on Friday into the weekend. Would rather catch a bug and quickly rollback Friday with fewer users impacted, rather than arbitrarily force devs to wait until Monday to queue up their releases during our peak usage.
As in most things, context matters. Tired of "always do x" and "never do y" opinions.
"Never ship on Fridays" is pretty high up on my antipattern lust. It's a nice way to phrase "how about we work 20% less at the same pay? It will help reduce the number of bugs / incidents we ship each year!". And it will help with that.
But why do I say 20% reduction in work?
Bugfixes and features that get fixed/finished on Fridays will ship 3 days later. Do your average lead time increases by >0.5 days.
This impacts anyone relying on you, too. So if this happens org wide then features and projects may ship days to weeks later.
Oh and what are you doing on Fridays then? Less relevant work? If you stick with priorities then you are just queuing up more changes for Mondays, leading to nice higher risk deploy Monday. You simply can't gain stability with skipped Friday deploys without changing / ignoring work priorities for Fridays.
So yes, this should be comparable to such a reduction.
Oh, and Friday deploys ruin your weekend? That means you have really high detection and mitigation time.
With 1h detection time and 2h recovery I could still with low risk deploy Fridays, 5PM, wait 1h and enjoy a weekend after wrapping up my work.
I have always shipped on Fridays. It never ruined my weekend. Even though I did have a few fun screwups (which sometimes required help from my coworkers).
This is really really ridiculous. I've never worked for a company that has releases every day, even every day of the week except Friday. That's not how it works. The most I've seen was 2 days (Tue, Thu) but 3 was fine if you insisted too too much. In my current company we have only 1 day a week release: Tuesday. No other day is allowed or is even possible without someone spending tons of time doing manual things that are supposed to be automated. Moreover, if your code is allowed to be released next Tuesday it has to be merged by 4PM this Thursday. I still spend 100% of my time working, including Fridays. Because not all coding is releasing, and not all software engineering is coding. It's hilarious that someone would think if you're not releasing on day X you're not working. Sounds like straight out of Dilbert cartoon where some management type thinks I'm releasing my code every day.
What is ridiculous is when not releasing on Friday means that "your metrics go up and that is bad". This is one of the reasons I hate metrics for software engineers. I should be able to put up a PR any day of the week without holding back committing and PR'ing just because there's a natural 2 day delay that will fuck up my metrics.
I will decide if this is something that needs merging now and be out in an hour even though it's Friday or if it can wait. And for that I need to not have perverse incentives. Just like I need to not have perverse incentives to make your Thursday cutoff by bypassing good engineering. If I can merge now or in two hours and it makes just a two hour difference and nobody cares for some dang metric, then I am more likely to take the time to add that one extra test that catches the bug. Before it hits a customer.
But in no way do I ever want to be restricted ever again from having a fix roll out within the hour. Our hourly deploy process (fully automated) is awesome. Never. Again. Without. It.
When you control the devices to which you're deploying to, there is little reason why you wouldn't deploy as often as you can. It helps a great deal in isolating bugs to keep your changesets small, and you can either do that by slowing down the product iterations (and getting poor feedback from each), or releasing more often. This is ubiquitous with web development.
Weekly releases (or slower) is appropriate when you rely on users to update their software or firmware. Most mobile app development does this.
> I've never worked for a company that has releases every day, even every day of the week except Friday. That's not how it works.
I'm sorry you've had bad experiences but it absolutely is how it works. When it comes to web based SaaS software, all high functioning organisations release multiple times a day. Daily releases are a bare minimum.
I dunno man. I work for a fintech that does tons of volume and we release safely multiple times a day. We’re working towards releasing on every commit for true continuous delivery. Once you’ve worked that way you’ll never want to do it any other way.
The only way to continuously ship in many B2B or enterprise environments is with copious feature flags. You can’t just change shit out from under you multimillion dollar clients — that’s part of the deal. So you flag it all out. Then changing the flag is itself a deploy by another name, and you are damn sure you don’t do it on Friday.
I think the author read "inadequate" in a different way than it was written. The way I read the tweet is not "then your practices are shit", but rather "then your practices are not ready yet". I.e. as an assessment, not judgement. It's a bit of a "you must be this high to ride" situation for Friday deploys. With 64 releases, I don't expect the release pipeline to even be ready for this - we're doing a few releases a day and I'll still choose not to do some of them on Friday afternoon. (Especially dns changes that could bring down everything)
Then there's a risk of something going wrong regardless of how well you're prepared. Don't release on Friday is stupid, ignore Fridays is stupid. How about "work on your release processes, so that the scope of what you're comfortable releasing on Fridays grows over time"?
There's tech perspective tunnel vision here. There can also be an issue with the requirements the perfect code was written for, or some other non-tech downstream process.
A risk calculation exists here and included in mine is our on-call support team. I prefer no Friday releases.
100% agree with Allen Holub. If your policy is no-ship-fridays because you're worried about things breaking and having to work the wkd, then your code quality is shit and you need to get better at being a software engineer.
We tend to avoid Friday deployments because they give us no tangible benefit while increasing the chance of an out of band issue. At a practical level, our customers literally do not notice if something ships at 4pm on a Friday or 10am on a Monday.
As for velocity, we can move way way faster by risking some bad deployments. Catching bugs is an exponential time trade off. We could catch most bugs, but we’d move like molasses. Instead, we catch what’s reasonable then ship during periods where it’s likely for us to recover quickly and easily.
Most companies will benefit from fastest shipping velocity at the trade off of occasional bad deployments.
Until we have formal verification that works across an API boundary where the parties are at different companies or much more extensive than usual test suites, changes to one end of an API endpoint will pass all manner of testing and still fail a customer use case. On Sunday afternoon in industries that ship over the weekend and start working again on Sunday.
And on what grounds/experience can you so quickly hand-wave all code that doesn't ship on Fridays as "shit"? This is lacking in nuance that anyone with experience would otherwise have.
Bugs happen. Sometimes bugs take a day or two to surface and aren't caught in dev/QA.
I would suspect there is a planning issue somewhere if an engineering team is brought to a standstill if they prefer not to ship on 1 workday of the week.
We save boring work for Fridays. We knock out quality-of-life patches/scripts/cleanup tasks. We'll review external and internal docs. We'll even explicitly verify the functionality of sufficiently large features, which has saved us a few times. Friday is our day to do tasks that improve quality.
I've seen enough weekends saved by not deploying on Fridays for me to be skeptical of any contrarian dogma that demands it's universal removal.
Now, if employees were to be paid 1.5x (+vacation) to work on weekends, then company would prioritize the quality and resiliency enough that Friday deployments won't become an issue
Yet here we are, ignoring the elephant in the room.
As a customer I prefer you not to release on Friday, then go home for weekend and maybe fix stuff on Monday... Specially when you have repeatedly proven that you don't know how to test things you are releasing...
If you call people back to work on Friday after release and keep them at office until things are fixed on your own dime go ahead. Feel free to release on any day.
Probably someone who never had to fix a critical bug on a friday.
I mean it's like wearing a helmet on your motorbike, 99% of the time it is useless, but the day you crash, you better have it and not need it than the opposite.
- the quality of your tests sucks, fix that with high priority
- your CI/CD process sucks, fix that quickly
There’s absolutely nothing wrong with releasing on Friday if you’re doing things correctly. And the meme that we shouldn’t ship on Friday needs to die.
I’ve literally never seen a codebase with 100% branch coverage from integration tests. There’s always some risk of handing a corner case badly with invalid writes. Now you need experts to write ad hoc tools to repair what’s in your datastores, and you’re really screwed if enough of those experts are unreachable until Monday.
Oh, they mean to never release on Fridays. I first thought that it is about shipping. I once bought something a little temperature sensitive and asked the seller to not ship on Friday so that the parcel has less time to be exposed to the temperature.
The problem is the bugs, not the Friday releases. This industry is so utterly depressing to me. We get paid like kings to deliver like monkeys. Clown industry.