Technology companies selling a solution to a problem created by technology companies instead of coming together to fix the original problem. This is the world we live in.
The original problem is that people want to Google for recipes and not pay for them.
As a cook there are several solutions to this:
1) pay for a subscription from vendors who are in business to sell recipes, such as Cooks Illustrated
2) get recipes from companies that provide them as a complement because they sell something else, generally food-related products. For instance King Arthur Baking has good recipes, and many are available from food companies such as Tyson, Kraft, and Betty Crocker. Grocers also have many recipes. These sources aren't interested in spamming users with ads because the website is one big ad.
3) Buy cookbooks, they're not expensive.
But yeah as long as people want to crank search terms into Google and get "free" stuff, it's going to be ad-infested, and then other ad-seeking folks are going to run their shakedown operations just like the adblock extensions charge money to advertisers to let their ads through.
Hi, currently there is nothing of this sort. The user agents are random. I have a couple of servers doing the scraping in real-time. The IPs are not static.
Let me see if I can build a opt-out list. But wouldn't it beat the entire purpose of this app?
You should just declare your bot as a user-agent. Most publishers won't even bother to do it, but leaving a publisher an option is the correct etiquette for any bot. Random user agents is cloaked scraping.
