Hacker News new | past | comments | ask | show | jobs | submit login

A crawler should be limited by the network not CPU. Outside of the language making it easier to handle multiple concurrent connections, I doubt speed would be much of a consideration.



I'm not sure, it might be. If you're not careful, CPU can become bottleneck when you have a 10Gbit network card.




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: