I read this essay in 2003 and it influenced how my business partner and I built our startup. It influenced what possibilities we chased after. But Shirky.com is now off-line:
This is yet another example of the history of the early Web disappearing. Shirky's early essays were fundamental to way we understood the potential of the Web back in 1999-2008.
I actually discovered this while looking for another old Shirky essay which, as far as I can tell, is now entirely gone from the Web.
This whole thread is giving me feels, but to the basics, I'll tell you how it happened.
I'd been writing about what we came to call social media since the early 90s (alt.culture.usenet and alt.folklore.urban ftw), but by the middle of last decade, all anyone wanted to talk to me about was marketing on Facebook, which was the boringest possible topic.
At the same time, my wordpress host had lousy security, and my site was getting frequently disabled because of some malicious javascript uploaded through some hole they hand't patched. I wasn't writing there anymore, so it was pure cost at that point, and cost of my time, not just dollars.
Then I moved to Shanghai for several years, working on other stuff, and fixed the site a couple of times again, and one time, my host was like "We disabled your site!" because of their own security flaws had let it get hacked again, which, the whole thing had entered 'ugh field'territory.
I never decided to let the site lapse, I was just tired of dealing with it, and the political circumstances in both China and the U.S. seemed much more urgent than rescuing some historical essays, so one day at a time of not dealing with it became years.
And here we are, me reading my own eulogy. Which is incredibly flattering and touching, I have to say.
I'm not even sure what of it can be resuscitated -- maybe if I want it back, I'll have to copy it from Wayback (and will say "Thank you Brewster", not for the first time), but if anyone here has advice about competent and secure hosting for an old Wordpress blog, hmu at cshirky@gmail.com, because reading this, it makes me embarassed not to have just fucking fixed this a year or two ago.
I used to work on a lot of US Department of Defense projects, mostly stuff I can't talk about. One very notable project I CAN talk about was an initiative (pushed by utterly clueless, insular, and frankly corrupt academics) to spend billions of dollars in 2008-2010 timeframe on implementing Semantic Web technologies in various military business systems across the DoD.
As an actual technologist who knew how to build things, I was perpetually in the awful position of having to explain to leadership that these highly credentialed academics were selling garbage. I had tried to implement systems according to their design. The graph databases they pushed (they hated Neo4J, for reasons of purity because it didn't actually use RDF/OWL in the database...... i get a headache just talking about this...) were slow piles of dogshit that couldn't scale. No amount of reality could dissuade the academics. They had their theories, and any collision with reality was merely an implementation detail that I and my team were simply too incompetent to overcome in their eyes. Almost none of them had actual technical experience. A smattering of Comp Sci folks, and a ton of "Library Science" idiots.
Your essays on why the SemWeb was utter bullshit were a potent weapon I used with the generals the academics were pushing, and I eventually got the generals funding the project to see the light. Got them cancelled, and sent the idiot egg-heads packing. I still see them on LinkedIn to this day. They desperately continue trying to push that rock up the hill, and only recently warmed to more practical graph database solutions.
They HATED YOU. It was hilarious, watching them try to refute your obvious points and clear writing with jargon and hand-waving. Utterly unconvincing to the generals.
You cannot possibly know how much this delights me!
Most of my writing was about social media, back when the web was young, but "Ontology is Overrated" is actually my favorite thing I ever wrote, and it makes me happy beyond measure to know that it helped someone manage an actual argument over whether to buy into the semantic web!
I have never been talented enough to write production code, but I often thought of myself as trying to provide ammunition to people like you who are, when talking to bosses who didn't understand that the phrase "Now it's just a simple matter of programming!" was a bitter, sardonic joke, not an upbeat assessment of possibility.
Thank you for telling this story! This whole thread has been like hearing my own eulogy, but this in particular is just :chefs_kiss:
Awesome! Yeah man, you nailed it all on the head back then, and yes, saved my ass. Sorry the thread felt like a eulogy. You've got a lot of mileage left in you my friend, and I'm looking forward to your future insights on the next big trends none of us are predicting yet.
The response of the academics to your writings was actually a master class for me in the nature of academic corruption and groupthink. What I learned from that experience was that (contrary to my prior beliefs) high IQ individuals are actually far more susceptible to cognitive dissonance than others are, not less so. They are far more adept at mentally constructing rationalizations and false realities that bolster and protect their existing belief systems from new information than most people are. Add to that the fact that they are extremely economically vulnerable to reputational damage, and you have a really toxic recipe.
Library Science is what librarians learn. It's a real thing for a real job. Like most credentials and like most jobs, some people try to over-fit experience and knowledge in one field to another.
You see the same with CompSci/tech people treating data like there's no bias in its collection.
As I learned from the people on that project with lib sci degrees, the employment prospects are predominantly low-paying, but these ones found a new boondoggle to employ them as "ontologists" where they could get 6 figure salaries to sit around and build models all day in a piece of software called TopBraid Composer. (GUI program built in Eclipse, where users would create diagrams that would then be translated to an XML offshoot called OWL, a W3C standard that's never been successfully used in any meaningful project I've seen) I witnessed these people sit around and create business models and knowledge graphs of arcane Air Force business processes for 3 years (there were literally 9 of them doing this) before the project was cancelled due to its technical impossibility. The ontologies they created were never used once, and when I actually tried to provide them (in PDF form) to a separate project where Air Force personnel were trying to map out business processes, the personnel stated to me (in writing, with a Colonel CC'd) "These are so inaccurate that they are frequently misleading, and cannot be trusted." The Colonel later pulled me into his office and stated (rather comically):
"You mean to tell me I've been paying people to draw cartoons for 3 years? We're not goddamned Disney here."
If it's just a collection of essays, what about a static site? You can set one up on Github Pages or Gitlab Pages with a minimum of coding. There are also virtually no security concerns and maintenance is minimal.
You'd have to take all of the text from the Wordpress blog and format it into Markdown but that shouldn't take a huge amount of time unless there is a lot of weird formatting or different media types.
This is absolutely the easiest solution. We did this for open.media.mit.edu in order to archive the old wordpress site. Used wget to fetch all of the HTML and ran a script over it to remove some obvious clutter. Happy to help out. Your writing really shaped the way I thought about the potential of an open internet.
Hi Clay, Kevin Marks here. I an help you scrape it out of wordpress into a static site if you like, I've done similar for Dan Gilmor and other old bloggers. Ping me - kevinmarks@gmail.com
You could scrape the site, either from Wayback or from WordPress, if you manage to get it working just briefly, and then host the site statically out of an S3 bucket. wget has an option to recursively crawl and save a site, but there are other tools.
> You’ll never maintain a Wordpress site long term securely. Need to convert it to static html one way or another.
I'm in favor of static HTML myself where possible, but it's not hard to maintain a secure Wordpress install. Keep automatic updates enabled and don't install any third party plugins.
It's that second part that most people screw themselves with.
It may not be very hard to maintain, but you still have to maintain it. Whereas if you just have a collection of articles that you want to keep around as an archive, if you convert them to a static site, you can basically forget about them afterward...
> It may not be very hard to maintain, but you still have to maintain it.
When the maintenance is "ensure auto updates are on, and don't do anything that would not get updated automatically" it's not like it requires regular effort.
> Whereas if you just have a collection of articles that you want to keep around as an archive, if you convert them to a static site, you can basically forget about them afterward...
Your web server, your operating system, etc. still require at bare minimum the same level of maintenance.
You can outsource that maintenance to someone else of course, but you can do the same with WP as well.
--
My point is that WP alone doesn't massively increase the maintenance burden, it's what people tend to do with (to?) WP that increases the burden and eventually leads to unmaintained sites.
>When the maintenance is "ensure auto updates are on, and don't do anything that would not get updated automatically" it's not like it requires regular effort.
no dog in the fight here but I felt impelled to point out that ensuring auto updates are on solves almost all security holes except for the security hole it opens up.
> no dog in the fight here but I felt impelled to point out that ensuring auto updates are on solves almost all security holes except for the security hole it opens up.
In almost any computing context, but especially in the context of a personal blog, the vast majority of exploits are against known security holes for which patches have already been released and those with automatic updates enabled are already safe from.
Yes, hypothetically updates can deliver new flaws of their own and even potentially intentional malicious code, but from a practical sense it's not worth worrying about if you're using mainstream software packages on a major OS.
Right. I should have clarified it’s unlikely to happen if you want to be hands off for years at a time. If that’s the goal the ideal state is to convert it to static.
WP security has come a long way. I've had a site up for over a decade, and while I used to be VERY nervous, now with automatic updates and a fair amount of code-hardening, it really hasn't been a problem.
What strikes me is how timeless your writing is. My own stuff is hit-and-miss, the bulk of it has a best before date measured in a few weeks or months but yours is just as relevant today as when you wrote it and that takes some real talent, not just in writing but also in the depth of insight that you have on this things.
I only know of you because of The Shirky Principle, which was very valuable to me in figuring out what's wrong with homeless services and that we need to focus on fixing housing instead. Thank you for your work and for stopping by to respond here.
Try to take a look at wpengine.com; they're not the cheapest, but I heard they have very good support people, and via some business relations I had with them, I know they have a system for auto-updating wordpress plugins.
He explained that an expanding __________ (I can't remember this word. Maybe he said "network"?) meant that each person in the __________ was exposed to a greater variety of choices, however, the total number of choices decreased. His point was that separate niches each maintained some unique offerings, but when everything was combined together into a single market, and attention focused on certain items, there would be a winnowing effect. The paradox was that everyone could legitimately feel they were enjoying an abundance of options, greater than ever before, while the total number of options decreased.
"Now that more than a billion people have access to the Web, there is no longer a trade-off between size and specificity. The basic math is simple: A tiny piece of an immense pie is huge. A decade ago, reaching one-tenth of 1 percent of Web users amounted to 36,000 people, a number that compared favorably with the circulation of, say, the daily newspaper in Bridgewater, New Jersey. Back then, reaching a million users required a decidedly mainstream offering (Amazon.com and MSN come to mind). Now, getting niche can be the path to getting big; one-tenth of 1 percent of today's Web audience is a million people."
I was just thinking about this - I can go to the store and have fifty variations of peanut butter at my fingertips - but I get basically the same fifty variations anywhere in the US, and much of them anywhere in the world.
Whereas in the past, my store might have 1 or even 0 options on peanut butter, but travel 10/50 miles away and there'd be an entirely different option.
We have a little bit left of this with beer, as most places have a "local" beer available.
Someone once described the general difference between European and U.S. grocery stores: They said that in Europe, a store would have every kind of product, but not very many brands of any particular thing, maybe two or three for most non-staple products. But in the U.S., a store would either have like 20 brands of something or have none of them; i.e. not carry that kind of product at all. It seems to me like this points to “choice” being an important scale by which stores are measured in the U.S., but no, or little, negative associations are made with no products of a particular kind being available at all.
There's some of that European approach creeping into US markets thanks to Aldi Nord and Aldi Sud - we now have both Aldi and Trader Joe's which primarily have their own versions of products and carry only that brand (or occasionally a second 'mainstream' brand as special items). Costco is similar as well in typically having only a single brand of any given type of product.
Ever stopped to ponder the laundry detergent aisle at your local grocery store? Or the potato chip aisle? Especially on detergent, what's there is a huge proportion of the shelf space devoted to a couple of brands (which may, in fact, be from the same parent company) with very large containers taking up a lot of space for relatively little actual product, but lots of different choices for those brands. Or, as I like to say, 31 flavors of Tide Pods. Same goes for potato and corn chips. Lays and Ruffles dominate, which oversize bags that are half air, relegating other brands to lower shelves on one end. But consumers can "choose" from half a dozen different flavors of the same brands.
And it's the same all over the US. The only variation tends to be that some flavors Flamin' Hot Dill Pickle or Wasabi Ginger, e.g., appear more often in some markets than others.
Good summary. The other day I was hearing a political example mentioning that every little local detail, now with social-media, it becomes potentially "a national issue" but that never happen before. Surprisingly, the Founding Fathers were aware of this issue back then and tried to solve (actually mitigate) the problem by keeping the central power as small as possible.
The title of that article is now itself a comment on the passage of time and the fading of things. How old do you have to be today to get the "I've fallen and I can't get up!" reference?
My 27yo ("reference Millennial") even has a mental image of the video (although has never seen it on a Tele-Vision Device). It may have entered the meme reservoir on its own at this point.
I wasn't quite sure when he wrote this. The earliest capture in the Internet Archive was January 2000, but it could be from earlier than that maybe? I do remember that it was quite prescient when I first read it, although it's sort of obvious in 2022.
> People who work on social software are closer in spirit to economists and political scientists than they are to people making compilers. They both look like programming, but when you're dealing with groups of people as one of your run-time phenomena, that is an incredibly different practice. In the political realm, we would call these kinds of crises a constitutional crisis. It's what happens when the tension between the individual and the group, and the rights and responsibilities of individuals and groups, gets so serious that something has to be done.
> And the worst crisis is the first crisis, because it's not just "We need to have some rules." It's also "We need to have some rules for making some rules." And this is what we see over and over again in large and long-lived social software systems. Constitutions are a necessary component of large, long-lived, heterogenous groups.
We focused on the "unfairness" and we thought there might be a technical fix, so we focused on new discovery mechanisms. But our solution did not get traction, so we eventually pivoted towards tools for building commerce sites, and there we had more success.
http://www.shirky.com/
I'm not sure when this happened but I see that Wikipedia has adjusted to this and now links to the archive.org link:
https://en.wikipedia.org/wiki/Clay_Shirky
This is yet another example of the history of the early Web disappearing. Shirky's early essays were fundamental to way we understood the potential of the Web back in 1999-2008.
I actually discovered this while looking for another old Shirky essay which, as far as I can tell, is now entirely gone from the Web.