As both a website host and website scraper I can see both sides of it. The website owners have very little interest in opening their data up; if they did they'd have made an API for it. In my case it's scraping supermarket prices so obviously big-grocery doesn't want a spot light on their arbitrary pricing patterns. It's frustrating for us scrapers but from their perspective opening up to bots is just a liability. Besides bots just spamming the servers getting around rate limits with botnets and noise any new features added by bots probably won't benefit them. If I made a bot service that would split your orders over multiple supermarkets, or buy items temporally as prices drop that wouldn't benefit the companies. All the work they've put into their site is to bring them to the status quo and they want to keep it that way. The companies don't want an open internet, only we do. I'd like to see some transparency laws so that large companies need to publish their pricing.