Hacker News new | past | comments | ask | show | jobs | submit login

Obviously scraping logic using puppeteer, but there are many other tooling aspects that are critical to bypass bot prevention.

one is signature / fingerprinting emulation. It helps to run the bot in a real browser and export the fingerprint (e.g. UA, canvass, geoloc etc) into JS object . Add noise to the data too.

Simulate residential IPs by routing through a residential proxy. If you run bots from cloud you will get blocked.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: