Does remind me a broader question -- why is there no popular open source search engine? This seems much more tractable than an open source social network. I wouldn't be surprised if it could money/resources from major players like Facebook, Apple, Oracle. Apache has a lot of hte pieces, but no consumer facing front-end that ties it all together to search the web (AFAIK).
I'd say there's a combination of factors, the first (and most important) being that Google is good enough for most people.
You'd need to coordinate crawling so as not to turn it into a giant DDoS machine; speed will be an issue due to geo distribution, variable hardware and result sets.
Validity and reliability of the data would also be issues, and would probably require several peers to "agree" to consistency, but in a way that does not allow easy gaming of results.
I suppose they're all solvable, though I think there would have to be a powerful incentive to do so. I imagine it'd be quite pricey too for the individual, though perhaps Gabriel Weinberg* could weigh in there.
I'd say there's a combination of factors, the first (and most important) being that Google is good enough for most people.
This is actually why I think it's important to have a good open source alternative. Google is good today. And frankly, if Bing wasn't around, Google could probably stop doing any work on search for the next five years with impunity.
Does remind me a broader question -- why is there no popular open source search engine? This seems much more tractable than an open source social network. I wouldn't be surprised if it could money/resources from major players like Facebook, Apple, Oracle. Apache has a lot of hte pieces, but no consumer facing front-end that ties it all together to search the web (AFAIK).