He calls out my post so I'm obliged to respond. First I know my first post there was kind of arrogant and comes off sounding like an armchair expert. But I have scaled similar code, by having extremely efficient barebones code. And I stand 100% behind that approach as valid, and something many websites do. Look at instagrams architecture for an example of simple infrastructure doing massive workloads. I gave my thoughts as a way to get discussion over what the challenges of scaling a site like 4chan would be, not because I meant to be arrogant about it.
4chan sounds like its based off some php/perl scripts +databases, and I think that approach is fundamentally bad for high performance sites with very well defined usage patterns. You wouldn't cut a steak with a plastic knife, therefore you shouldn't be writing a website that's meant to scale in php or perl. Sure you can do it with a lot of effort, but it's ass backwards.
The advantage of databases and scripting languages is flexibility with how you can add features and do queries on the data. Once you have a well-defined usage model and queries, it makes complete sense to change to the customized highly optimized approach with java servlets or something for the queries that need to be fast.
Also if you have a small enough data set that can live in memory like 4chan does (text), it makes complete sense to have it all in memory ready to serve up on demand super quickly.
I didn't mean to trivialize the efforts to scale such a website, but on a conceptual level, if you can fit all your data in memory, and you don't need to customize pages per user as 4chan doesn't, it really isn't that complex to scale.
Instagram infrastructure engineer here. Our infrastructure is far from simple. However, I confirm that it does, indeed, handle a fairly large workload.
4chan sounds like its based off some php/perl scripts +databases, and I think that approach is fundamentally bad for high performance sites with very well defined usage patterns. You wouldn't cut a steak with a plastic knife, therefore you shouldn't be writing a website that's meant to scale in php or perl. Sure you can do it with a lot of effort, but it's ass backwards.
The advantage of databases and scripting languages is flexibility with how you can add features and do queries on the data. Once you have a well-defined usage model and queries, it makes complete sense to change to the customized highly optimized approach with java servlets or something for the queries that need to be fast.
Also if you have a small enough data set that can live in memory like 4chan does (text), it makes complete sense to have it all in memory ready to serve up on demand super quickly.
I didn't mean to trivialize the efforts to scale such a website, but on a conceptual level, if you can fit all your data in memory, and you don't need to customize pages per user as 4chan doesn't, it really isn't that complex to scale.