You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved.

1: http://doc.scrapy.org/en/0.24/intro/tutorial.html