> Could you say more about this? I wasn't involved, but my recollection is that Google pioneered using a zillion low-cost nodes for search. E.g., I remember that AltaVista was using a big iron approach partly as a way of showing off DEC's wares. I can believe everybody had to try to keep up, but I am skeptical that Google slowed down enough for everybody else to catch up and/or surpass them.
It's the nature of innovation that something that puts you ahead doesn't create a sustainable advantage (e.g. AltaVista as you highlighted), and there tends to be some marginal advantage for second movers of having the benefit of hindsight. I don't know that anyone knows the why of how this played out in Google's case, basically no one is privy to all the information about each search engine's operations, and to the extent they do, they likely cannot share what they know.
Certainly, on the cost side, Google's approach wasn't entirely novel in the sense that there was a lot of expertise already out there on how to run low-cost nodes at scale. Reverse indexes are actually a pretty "embarrassingly parallel" problem, and Page Rank style algorithms don't really make it much more difficult. In short: once Google demonstrated the value of the approach, it was pretty easy to figure out how to implement it. Google certainly continued to innovate on their data center designs, but if anything were at a disadvantage in terms of rolling it out because they already had a substantial investment in their existing designs that were working pretty well for them. There were also startups like FAST (AllTheWeb) that sprung up after Google and were able to leapfrog Google much as Google leapfrogged its competitors.
In terms of relevancy, we did do double-blind style evaluations of the relevancy of search engines as well as of the consumer behaviour with search engines. What we found was that by 2004-2006, search engine competitors had largely closed the relevancy gap, and in some cases reliably exceeded Google. What we also found though was that it largely didn't matter. Even when presented with terrible results, most of the population would not switch search engines. Indeed, most of the growth in market share amongst search engines came through distribution deals where users would be presented with a search engine as the default.
Now, that's not to say that people didn't switch because they found a search engine to be better. They absolutely did.. but only 10-30% of the users would ever engage in switching behaviour for any reason. Getting those users certainly would help you, but the vast majority of the market was driven by other factors.
It's the nature of innovation that something that puts you ahead doesn't create a sustainable advantage (e.g. AltaVista as you highlighted), and there tends to be some marginal advantage for second movers of having the benefit of hindsight. I don't know that anyone knows the why of how this played out in Google's case, basically no one is privy to all the information about each search engine's operations, and to the extent they do, they likely cannot share what they know.
Certainly, on the cost side, Google's approach wasn't entirely novel in the sense that there was a lot of expertise already out there on how to run low-cost nodes at scale. Reverse indexes are actually a pretty "embarrassingly parallel" problem, and Page Rank style algorithms don't really make it much more difficult. In short: once Google demonstrated the value of the approach, it was pretty easy to figure out how to implement it. Google certainly continued to innovate on their data center designs, but if anything were at a disadvantage in terms of rolling it out because they already had a substantial investment in their existing designs that were working pretty well for them. There were also startups like FAST (AllTheWeb) that sprung up after Google and were able to leapfrog Google much as Google leapfrogged its competitors.
In terms of relevancy, we did do double-blind style evaluations of the relevancy of search engines as well as of the consumer behaviour with search engines. What we found was that by 2004-2006, search engine competitors had largely closed the relevancy gap, and in some cases reliably exceeded Google. What we also found though was that it largely didn't matter. Even when presented with terrible results, most of the population would not switch search engines. Indeed, most of the growth in market share amongst search engines came through distribution deals where users would be presented with a search engine as the default.
Now, that's not to say that people didn't switch because they found a search engine to be better. They absolutely did.. but only 10-30% of the users would ever engage in switching behaviour for any reason. Getting those users certainly would help you, but the vast majority of the market was driven by other factors.