DuckDuckGo continues to impress. And sometimes editorial decisions are better than algorithms: since most users view most Demand Media articles as spam, DDG and Blekko get good results without investing a lot of engineering investment simply by banning them. Meanwhile Google had to work out algorithmic changes and test them at scale -- with the risk that they unintentionally get rid of stuff from other sites that most people actually do want to see. So it's a tough situation for Google, especially when you factor into account potential impact on all the advertising revenue they're generating from these spam sites.
I think long term Google will benefit from their actual changes vs. what DDG and Blekko have done, in 2 years when the sites they've banned or selected get better or turn to shit they'll have to rework for that, Google won't. I guess DDG and Blekko are gambling on the fickle nature of "tech" people, but they'll get bitten in the ass by it when the sites they've chosen change and they lose users because suddenly they suck and Google will be there with their proper ranking which can adapt to sites shutting down.
Also the idea that there are people "curating" the way my search works is... unsettling I guess, they're deciding based on arbitrary criteria whether or not they think a user wants a site. What if I like the content DM produce?
I agree that it's harder for Google, because we don't have the luxury of hard coding rankings, but I take big issue with this:
> So it's a tough situation for Google, especially when you factor into account potential impact on all the advertising revenue they're generating from these spam sites.
Adsense revenue does not enter in to ranking decisions. Our evaluation process is far from perfect, but revenue is not part of it. We're not blind to the fact that a lot of scummy sites run adsense, but the even scummier ones have already been kicked out of adsense and now use other ad networks or affiliate programs. "Denying spammers revenue" has been at times the explicit goal of projects that launched.
The tough situation I was talking about is that the net result of investing all this engineering effort to keep up with Duck Duck Go and Blekko still is likely to wind up as a revenue loss for Google.
In terms of AdSense revenue influencing search results, it would be great if Google published their ranking algorithms and implementations so that people could verify it for themselves. [Ditto for Microsoft of course.] But okay, I'll take your word for it that it's not factored explicitly into the ranking calculations and that you've done the analysis to make sure that it doesn't indirectly influence calculations. Even so, it may have affected resource decisions. At the organizational level, did the specter of losing tens of millions in ad revenue had something to do with why Google waited so long to start to address the problem?
> Even so, it may have affected resource decisions. At the organizational level, did the specter of losing tens of millions in ad revenue had something to do with why Google waited so long to start to address the problem?
At the organizational level, Google is essentially chaos. In search quality in particular, once you've demonstrated that you can do useful stuff on your own, you're pretty much free to work on whatever you think is important. I don't think there's even a mechanism for shifting priorities like that.
We've been working on this issue for a long time, and made some progress. These efforts started long before the recent spat of news articles. I've personally been working on it for over a year. The central issue is that it's very difficult to make changes that sacrifice "on-topic-ness" for "good-ness" that don't make the results in general worse. You can expect some big changes here very shortly though.
that's really the problem. Adsense doesn't factor into it... but it should. negatively. truth is, Google has created a superb "signal" that a website is crummy: Adsense. it would be funny if it wasn't so sad. and they can't do anything about it, since it would hurt the bottom line.
Good point. Google will be challenged to handle these issues, and certainly can't be so direct about it. But if removing them draws visitors to the smaller search engines, Google will need to respond in some fashion.
Yep. "Don't be evil" doesn't scale with company size and success: the bigger you get, the more likely you are to be in situations where others are less evil than you.
Conversely, "Do good (and minimize evil)" scales positively with company size and success. But it's nowhere near as catchy.
