Hacker News new | past | comments | ask | show | jobs | submit login

> using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily

Can you please explain how you were able to better the performance of aws cli.




you need more connections than what a single AWS CLI process open to saturate network on a big box. You can achieve the same by doing some "xargs | aws cli" trickery but then error handling becomes harder.

Our S3 client just handles multiple worker processes correctly with error handling.





Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: