More than a decade has passed since then so I am stretching my memory. At peak we were serving in the region of 10 million page views per day which made us one of the most popular websites on the internet (Minecraft was a phenomenon and every Minecraft player needed the wiki). We were probably the highest traffic Wiki after Wikipedia. Nowadays Cloudflare could absorb most traffic because of the highly cacheable nature of it, but at the time, Cloudflare didn't exist, and every request hit our servers.
Yeah, Wikia in aggregate was in the top 50, maybe a top 20 site at various points. Wikia was built on caching. From my memory, about 99% of page views hit some kind of cache. If that dropped down to 97%, servers started to suffer. It's good to remember that the Fastly CDN company is a spinoff of Wikia, it was developed internally there first. Without that (varnish cache plus lots of memcache) Wikia would not have been able to handle the traffic. Mediawiki is horribly inefficient and one reason why Wikia was attractive as a host was that we had figured out a bunch of tricks to run it efficiently. The default configuration of mediawiki/wikipedia is real bad. Bigger independent wikis just couldn't handle the scale and many of the best independent wikis moved there for that reason. Just as one example, every link/url on a page hits a hook/callback that can call into an extension literally anywhere in the code base, which was several million lines of PHP code. I remember the "Batman" page on the DC wiki used to take several minutes to render a new copy if it fell out of the cache. That was one page I used for performance optimization tests. The muppet wiki and the lyrics wiki also had huge performance issues and fixing them was some of the most fun engineering work I've done. Every useful feature had some kind of horrible performance side effect, so it was always a fun puzzle. I also hate landing on a Fandom wiki now but thanks to the actual editors, it's still got some good content.
I bet, being by some counts the most popular video game ever - but which also makes it kind of a bad example to use when talking about wikis.
By definition, very few wikis will have to deal with becoming one of the most popular websites. (And as you say, at that point one should be able to figure out funding.)
Cloudflare get the best deals on bandwidth. It will usually be cheaper to serve a terabyte from Cloudflare than to do it yourself: you could probably run the wiki on the free plan!
Perhaps, but VPS traffic prices are also already a lot better than "big cloud" traffic prices, especially if you choose your VPS provider with that in mind. And once your traffic is large enough there are also options where you pay for a fixed pipe instead of a transfer amount.
If you want to pay for bandwidth then yeah, CloudFlare is a great option.
Otherwise, if you like the experience of not paying per GB/TB, go for a dedicated server with unmetered connection that has the same price every month, regardless.
Cloudflare don't charge per GB/TB. You get unlimited bandwidth even on their free plan. The problem with paying per GB is that it's in the CDN's interest for you to get a DDOS attack so they can charge you for all the bandwidth. It's in Cloudflare's interest to reduce DDOS attacks and unwanted bot traffic because it costs them bandwidth, not you.
I moved a few of my personal websites to AWS's CloudFront and it cost me like a buck a month, way cheaper than maintaining a virtual server to do it. Except that somebody somewhere decided to try their DDOS tool on one of them for a few hours in the middle of the night, and I got a bill for $2541.69.
The whole point of systemic incentives is that there is no conspiracy. Nobody wants a DDOS and every large provider will have people genuinely working to avoid them. But every time there is an opportunity to allocate resources, the team that gets to frame their return on investment in terms of real dollars will always have an edge over one whose value is realized only in murky customer satisfaction projections. Over the lifetime of a company, the impact of these decisions will add up with no need for any of the individuals involved to even be aware of the dynamic, much less conspire to perpetuate it.
That's sound logic. In this specific case of capitalistic incentives, I haven't noticed that it's working out in a way that make one more vulnerable to DDoS when one pays for bandwidth
It's more like: if you have a website that (sometimes) gets a lot of traffic, do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests?
> do you want Cloudflare to cache it and serve it with very few hits to your cheap server, or do you want your compute costs to expand to cope with the requests?
Usually you have something like a platform/tool/service that is mostly static requests that could be cached, with some dynamic requests that couldn't, as they're CRUD requests or similar.
If your struggling to serve static content, then do go ahead and slap Cloudflare on top of that bad boy and probably your visitors will be a bit happier, instead of upgrading from a cheap VPS.
If you're struggling to serve the dynamic requests, Cloudflare/CDN won't matter because these things actually need to be processed by your backend.
So instead of trying to shave 50ms off from my simple static requests with a CDN, I'd much happier to optimize for all the requests, including the "dynamic requests" that need to hit the backend anyway.
I'll still go for a dedicated server with proper connection and performance rather than a shitty cheap VPS with a CDN in front off it.
Hetzner is pretty cheap, but only offers Europe location for their dedicated servers last time I checked. For more locations, DataPacket is nice, although a bit more expensive.
If you can run your application on Cloudflare Pages / Workers with Cloudflare's storage/DB things, it really gets dirt cheap (if not free) and very fast. And even without that, Cloudflare's caching CDN is very good, very cheap and very easy.
Ten years ago bandwidth was expensive. Still is, even if not as much. A simple VPS gets overwhelmed, but a simple VPS behind cloudflare can do quite well.
Cloudflare caches pages at many many datacenters, often colocated with large ISPs.
This lets Cloudflare deliver pages from their local cache over local links (which is fast and cheap), instead of fetching the data every time across the world from wherever the VPS is located.
sidetrack but how does cloudflare make things cost effective? wouldn't it be cheaper if i just hosted the wiki on a simple vps?