Hacker News new | past | comments | ask | show | jobs | submit login

Distributed bot and scraper networks. Thousands of IPs geographically dispersed throughout the world. There is only so much you can do with rate limiting.



They asked about LinkedIn, where the content is gated behind a login. If it was a rate limiting problem, that would be trivial.

Needing to be logged in as the same user defeats the purpose of proxying to hide your physical origin.

Registering thousands of different users to use in a distributed way is hard now that they require a text message verification for new accounts.


Public LinkedIn profiles (which is many of them) are open to scrapers and they lost a court case about it.

https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...


I go to LinkedIn without being logged in and nearly always get a login gate instead of the profile.

They were ordered to unblock hiQ specifically, they were not ordered to open up content to scrapers generally.

They can still throttle high volume traffic and put up captchas. I think the only specific thing the court ordered was for them to unblock hiQ IP ranges.


Proxies can also work well for cheaper than buying distributed compute.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: