Our setup is mixed, all our own servers are virtual on rented space at a local server rental shop. I’m not a hardware tech, so I’m afraid that’s as technical I can get.

So our SQL cluster, and most of our web applications run on our own iron. Anything external as well as a lot of managed services run in Azure where the added security and operations dev-tools are invaluable compared to what we had when we self-hosted.

We do make a conscious choice about everything that moves to the cloud. We’ve spent a lot of time figuring out if buying new rental space for our SQL cluster would be cheaper than moving it to Azure. With the current political climate, we’re also a little more hesitant about moving things to clouds operated by American companies, because there is risk attached, in that we may have to move it back rather suddenly. Not something we expect will ever happen, but we don’t like risk in the public sector.

So my post wasn’t so much a “we run everything in Azure” as it was “well Azure is the obvious choice for the things we do operate in the cloud”. The relationship Microsoft has with enterprise, and the expertise staff has with their products means they become the best choice for a lot of enterprise. Unless Microsoft solutions are radically more expensive than their competitors, they simply have an advantage by already being a big, and typically well liked, player in most large organisation.

I say this as someone who used to firmly agree with this position, but in the last year my mind has been changed without reservation.

The time where this was a good idea has genuinely passed. There is no earthly way that any reasonable organization will be able to provide even the security that is present by relying on a large-scale cloud provider.

There is no way they will come out on top money-wise either. The big cloud players have a _absurd_ number of servers that _teams_ of some of the best CA talent the globe has to offer stressing to improve KPIs for.

10k employees in a municipality is small potatoes to what azure / AWS / GCP have dedicated to their products.

I have literally never seen a setup where cloud came out less than ~twice the cost of dedicated hosting. I have seen instances where dedicated hosting comes out cheaper than putting your own equipment in a colo, but even that depends on being in a location where energy and property prices (and so colo rental costs) are high.

And this does factor in devops - when doing consulting I earned consistently more (because of more hours) from clients that went for cloud setups; often they'd end up spending more times solving problems that generally didn't exist in the first place in a dedicated setup.

I do see lots of people that keep assuming that the cloud players must be cheap because they're so big, but I just have never seen that bear out in practice - in part because of that attitude, the margins they can charge are far higher.

Cloud providers are great for ease and for the number of services they provide, but they are generally an expensive step up.

I suspect you’re doing it wrong, or there’s some niche type of computing you specialize in.

In general purpose IT, looking across an enterprise portfolio of applications, we consistently see customers of Tidal Migrations replatform their applications to cloud and save 95+% in OpEx vs dedicated on-premise hosting.

IMO, The first step to realizing those cost benefits is recognizing that the cloud is not your datacenter and you need to architect differently.

Yes, cloud spend can grow as you open up access to more developers, but that’s why we have a plethora of tools and governance people to help make that manageable. I believe the business benefit of the agility gains that come from instant and decentralized resource provisioning will always trump any cloud bill... especially if you’re in a competitive industry & don’t want to get left behind.

> IMO, The first step to realizing those cost benefits is recognizing that the cloud is not your datacenter and you need to architect differently.

I see you've never setup or had to deal with setting up SAP. There are a ton of legacy line of business applications which, won't be close to "cloud" any time in the near future. And are all run on if you're lucky, vm clusters, if you're unlucky on bare iron due to silly crap like per cpu licensing on where it "might" be run. Or if virtualized, a sum of all the physical hardware cpu's.

"Enterprise" software running on premises is... problematic at best. Good luck replatforming something like this. They ask for your arm, leg, first unborn child, and your great grandkids children for the opportunity to run their software.

I'm avoiding talking about the vendors that require up to or over a month to have a contractor on site helping you "integrate and install" their application on your systems. That crap is so far removed from instant and decentralized resource provisioning its like being in another universe. God help you if you need to change anything.

Parent is probably comparing the cost of on prem to running VMs 24x7. Most IT departments are running software they did not write and don’t have the luxury of even getting access to the source code. If these customers want to do cloud they have to do it ‘wrong’

First of all:

> dedicated on-premise hosting

I did not say on-premise for a reason. Most people are not well placed to host on-premise. For starters it tends to require ops staff on site, which in many countries means a minimum of 3 shifts of a minimum of 2 people. On-premise deployments rarely makes sense.

I said dedicated hosting, which implies renting servers from providers like e.g. Hetzner.

But that said, you can replatform to anything from anything and save money in most organizations, because most organizations tends to be very bad at optimizing cost, so this to me says very little.

Most of the systems I've moved over the years were on the other hand carefully architected to be "cloud friendly" to start with. Some of them started out on cloud platforms and were migrated off to save money.

When you on the other hand start comparing the amount of compute and bandwidth you can get for the same prices, it becomes very clear how overpriced they are.

You can easily find bandwidth at less than 1/10th the price of AWS for example, and in fact I've had clients where their bandwidth bill alone at AWS was bigger than the total hosting bill after I'd moved them elsewhere. No amount of architectural change of their systems will change that - at a minimum you need to reduce the data transfer from their AWS setup. Now, you don't need to move everything out of AWS to fix that - often the savings you can achieve by cutting the AWS bandwidth bill can pay for an entire CDN....

Dedicated hosting also tends to give you far more flexibility in the precise hardware configuration to the point where savings can be similarly huge by substantially reducing the number of instances.

> I believe the business benefit of the agility gains that come from instant and decentralized resource provisioning will always trump any cloud bill...

Nothing prevents you from spinning up cloud instances when needed. Most dedicated hosting providers today also offers cloud instances, so you can typically do that even with a single provider. In practice, the cost difference between dedicated and cloud typically allows a substantial overprovisioning and still saving money, but if you're prepared to use cloud to handle spikes, you can save even more by using dedicated by going closer to the wire, because you know you can spin up cloud instances to take the peaks.

I've set up and operated systems like that which balanced loads over both colo's, dedicated hosting and cloud instances seamlessly several times.

It's amazing how people fail to see that public cloud is the equivalent of a hotel.

It's far cheaper than buying a house... in the short term.

Cloud providers are not getting hardware, datacenters, electricity and labor for free. You are still paying for it.

Public cloud is only the equivalent of renting a hotel if buying a house means hiring the builders full-time for continued maintenance.

I used to provide devops consulting services exactly because nobody but large organizations "hires the builders full-time for continued maintenance" for dedicated servers any more than for cloud, because it takes really large systems before you need hardware intervention very often.

Even when working clients that had multiple racks of hardware they owned, I spent on average a couple of days a year dealing with that.

On the contrary, clients with cloud setups "hired the builder" for far more hours on average than those with dedicated setups. For my billable hours it'd have been far more beneficial if more people went to cloud setups.

Hiring the builders full-time is only the equivalent of building a private data center if building a private data center means buying the entire companies of Intel and Supermicro.

Nothing stops you from doing infra as code on dedicated hosting. All of my setups for the last decade or so have been built around VMs and/or containers with deployment systems where we spun up containers across multiple datacenters on servers we had full control over.

Many dedicated hosting providers now provide APIs for deploying servers, as well, so you can handle even deployment of the underlying servers in an automated way.

Several have combined cloud deployments with deployments to dedicated servers from the same container images, bound together in a single virtual network. E.g. I had client that hosted across AWS, GCP and Hetzner, and migrated services between them zero-downtime. Eventually they moved everything to Hetzner because it cost them about 1/10th of AWS and GCP given their bandwidth use (at the time outbound bandwidth at AWS cost 50x what it cost at Hetzner).

If organizational dysfunction means you're not allowed to order the resources you need, then that is of course a problem, but a very different one.

Maybe not own dc, but colocation could still make sense? You don't need to be more efficient than Azure & Co. They have pretty solid profit margins, even being 30% less efficient should still be cheaper for you. And beyond a few thousand servers, I'm not even sure if scale matters that much (for server virtualization only).

This varies by case - Office365 phishing breakins have been a bad epidemic for a long time now and the anti-phishing measures have not kept up well enough. I think MS still doesn't support any phishing resistant 2FA method there...

FIDO is supported, which is cred phishing resistant - but Oauth permissions phishing obviously can't be prevented if it's all 'legitimate' traffic to a bad app.

Haha. Here's another one: "At this scale, why not write your own OS, since you already have a lot of servers to maintain?"

An potential problem is that now half of your IT recruits go to reinventing this stuff instead of working on domain problems. There aren't that many it staff per 10k municipal employees.

(Of course this is assuming Azure specific hassles take much less staff time than running your own infra, not a given...)

One point(though I will admit this is the cynical part of my brain speaking) immediately jumps to mind.

If the server goes down, you can blame Microsoft. Even the least technical person can’t blame you for that. If you create your own data center however, if it goes down, you may potentially be on the chopping block if service is interrupted.

Self preservation is a strong motivator, perhaps the strongest in a business environment.

> Self preservation is a strong motivator, perhaps the strongest in a business environment.

And it's the most bullshit one. Outsourcing the risk does not mean avoiding it, it means however putting it out of control.

This is a typical manager bullshit attitude "nobody got fired to buy IBM" that generally lead to adopt unadapted, bloated, overpriced solutions to trivial problems. Just because they do not have the balls to do things properly.

Running out of its responsibilities should be a criteria to get fired when things go badly wrong.

The outsource of the 737 MAX MCAS code to India today is a perfect example of that

Agreed that outsourcing risk does not free you from the responsibilities. However, for many many services you have to rely on others to provide the service better than you could do yourself, as you are limited in time, money and other resources. For running production grade databases at startups, it's much more cost-efficient to run on AWS RDS than to hire a systems engineering team with 24/7 standby. For more trivial systems you might be right and managers might be cowards. However, you have to realize it's a spectrum.

Whether you're running it yourself or you outsource it, things will go wrong at some time. When this happens, and you run it yourself, you also have to explain why it went wrong in a post-mortem or RCA. The problem is that your customers have no point of reference for your explanation (how likely is it that this occurs again?). In my experience, the following message goes down a lot smoother: "This is an Azure/AWS/GCP outage which affected not just us, but 1000s of other companies. We rely on {provider} to continuously learn from their mistakes and improve their service and they've shown this in the past. Here's their post-mortem report about this outage.". Note that not all cloud providers fit this bill.

> However, for many many services you have to rely on others to provide the service better than you could do yourself, as you are limited in time, money and other resources.

I do not criticized the fact of outsourcing when an outsourced service does a better job. This is normal and should be like that.

What I am criticizing however, is outsourcing even when local/home made/OSS solutions are a better and cheaper fit even considering SLA.... just to avoid responsibilities.

This is in my experience common, especially if the management has no technical background, no trust in his team and fear its upper layer.

Boeing didn’t outsource MCAS to India. The outsourcing contract was for display software.

Yes, surprisingly enough, people look out for their own self interests.

What makes outsourcing hardware different from the dozens of other software as a service vendors that most companies depend on?

Why spend the time developing competency in managing servers if that doesn’t give you a competitive advantage.

But, going with the biggest most stable vendor is usually good. If you bought IBM hardware in the 70s you can still buy new hardware that supports your software. If you went with their competitors - not so much.

> But, going with the biggest most stable vendor is usually good. If you bought IBM hardware in the 70s you can still buy new hardware that supports your software. If you went with their competitors - not so much.

For 20 times the cost of commodities x86 that you should have bought by doing the right thing.

And this 20 times the cost will very likely also please your own competitors because you will yourself be less competitive.

That's how you finish with entire airline industry or bank still running on COBOL with no possibility of migration.

IBM itself is very happy about it however.

How much will it cost to rewrite everything? What were they suppose to choose in the 70s? Do you think that all of the people still using IBM and running legacy code are dumb or just maybe they did a cost benefit analysis and decided they didn’t need to rewrite everything in Node and React?

Maybe you should ask why some sector that "fear" failures more than anything else (banks, aviations) are still trapped into these systems while everyone else is not.

This is exactly related to what I was saying before.

>that generally lead to adopt unadapted, bloated, overpriced solutions to trivial problems.

There's almost no meaningful business downside for choosing poorly.

Most businesses just don't care that someone else thinks they didn't use the right tool for the job.

Should it be that way? Definitely not. But it won't change unless there's consequences (and when that happens, those products will die off quickly)

My experience with home-rolled solutions suggests there are plenty of hidden costs. Got a new project, and want to spin up some experimental servers? Gotta wade through a bunch of IT guys. Documentation and training around Azure, etc is also going to be better than homegrown alternatives, most likely. Things like that.

It’s hard to put a monetary figure on the frictions costs, but they are there.

You're comparing one extreme (public cloud) to another extreme (sysadmin-managed infrastructure). The alternative, today, is either a private or a hybrid cloud, which have all the advantages of short-circuiting sysadmins that public clouds have.

Public institutions usually have a lot of systems but not a lot of traffic (if we compare to tech companies), so it makes sense for them to put things in the cloud instead of building everything themselves. Also public institutions rarely get top talent and instead do everything via contractors, do you really think it is cheaper to get government contractors to build and run infrastructure instead of letting Microsoft do it?

Active Directory, Domain Services, Exchange Server and SharePoint Servers (which can use integrated Windows auth).

Also many internal Enterprise apps are integrated with AD for auth and permissions.

As someone who worked in the very same municipality... one word; bureaucracy.

More specific reasons were. Legal boundaries between ownership and taxpayer money separation. Of cause GDPR is also a big obstacle in joining forces on this scale.

I worked on a project which tried to make a joined venture between several municipalities. I'm not sure if that partnership has ended yet, but overall you could only progress if there were people on both sides collaborating with a "better ask forgiveness, than permission" mindset. As soon as the municipality lawyers or their security officers got involved, all progress stopped immediately.

Or not dealing with infrastructure gatekeepers.