Question: Towards the end of your post you mention your amazement regarding how much of non-image content is accounted for by JavaScript code, citing 200K Facebook libraries and such. Could you comment a little bit about the mix between CDN-hosted JS and locally-hosted JS?
From your data, would you be able to speculate on the amount of efficiency gain (in bytes transfered and average latency across the visitors to the top 1000) that could be had if we aggressively re-configured this entire mass of sites to use jQuery/Prototype/etc from a single location rather than on their own sites?
I don't have a dead easy way to extract that information in general, unfortunately. It's something that I do look at the jQuery and YUI (to see whether they are taking the hosted or self-hosted versions), but for generic JavaScript I don't have CDN information.
Addendum: Another interesting thing would be to load every single piece of js from all of these sites and compare it against a minified version of itself.
Perhaps you could count how many references you see to, for example, jQuery (without worrying where it's hosted by those sites). Then assume all of those sites loaded it from Google. Then take what you know about the traffic to those sites and obtain a rough estimate of two extremes: everybody hosting locally vs. everybody hosting in a single location.
Interesting upward kink starting in August/September in the average number of JS tags on a page. Any idea why? Any particular tag in the data showing a similar trend?
I wrote my own spider in RoR (to manage a database of historical information since I run it weekly). The spidering is done by controlling Firefox using Selenium from the RoR application.
I'd love to have a copy of this spider if you ever decide to open source it. I've built WATIR-based robots to drive a browser to do useful stuff before, but have not yet tinkered with Selenium.
This is awesome and thanks for posting it.
Question: Towards the end of your post you mention your amazement regarding how much of non-image content is accounted for by JavaScript code, citing 200K Facebook libraries and such. Could you comment a little bit about the mix between CDN-hosted JS and locally-hosted JS?
From your data, would you be able to speculate on the amount of efficiency gain (in bytes transfered and average latency across the visitors to the top 1000) that could be had if we aggressively re-configured this entire mass of sites to use jQuery/Prototype/etc from a single location rather than on their own sites?
Thanks
[edit: formatting]