Hacker News new | past | comments | ask | show | jobs | submit login

Just curious what have you seen crawlers do to make you conclude they're up to nonsense?



Well, from amazonaws.com, there are so many requests for wp-login.php!

And then all the off-brand scraping companies use amazonaws.com.


What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?


Some AWS visitors:

Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/

That last one is a "SEO platform".




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: