It is quite hard, but I think it is impossible if you are not doing TDD and asserting throughout the process.
Basically, write a whole bunch of rules, fail safes, etc. So for a twitter crawler the tweet should always be there, the username should always be there, and the response should always be more than 400 characters.
Then any response that fails doesn't bring down your app, it just goes into the "needs review pile" and your app continues unless it gets 50 or so of these errors in a row. Then it SMSes or emails you to let you know it shut down.
It isn't like normal programming, it is dirty and any analysis you plan to do has to take the assumption that you will never get all of the data.
In terms of cleaner code, TDD will help here, but yeah, you will need dozens if not hundreds of methods while you're trying to classify a response.
Basically, write a whole bunch of rules, fail safes, etc. So for a twitter crawler the tweet should always be there, the username should always be there, and the response should always be more than 400 characters.
Then any response that fails doesn't bring down your app, it just goes into the "needs review pile" and your app continues unless it gets 50 or so of these errors in a row. Then it SMSes or emails you to let you know it shut down.
It isn't like normal programming, it is dirty and any analysis you plan to do has to take the assumption that you will never get all of the data.
In terms of cleaner code, TDD will help here, but yeah, you will need dozens if not hundreds of methods while you're trying to classify a response.