Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
qwertox
on Aug 7, 2023
|
parent
|
context
|
favorite
| on:
GPTBot – OpenAI’s Web Crawler
I wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.
Gasp0de
on Aug 7, 2023
[–]
I mean, technically you can just disallow all robots in your robots.txt and only allow those of which you are aware and you like?
Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
