Hacker News new | past | comments | ask | show | jobs | submit login

Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will have an advantage, but I don't give swindlers a pass just because they rob me to my face. I'm okay with defaults that punish all bots.





You can have a bot allow list. I think it's also being planned as a subscription service (not sure about this part).



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: