Business idea: build a firewall for AI crawlers that variably gives them access to customer’s sites based on traffic they send. The more traffic they send, the more they are allowed to crawl.
Yeah that was actually what I was thinking of for the idea, but the problem with their firewall is that it doesn’t penalize the big crawlers, like Google’s bot, because Google still sends traffic. Even if Google is sending less and less traffic over time.
That just gives the biggest companies the prize of being able to crawl all the web for free, while they offer less over time in exchange.
