That's an interesting claim. I make range requests to 100GB+ files (genomics) al...

bdon · 2024-09-25T02:16:44 1727230604

You can test this claim directly against a AWS S3 bucket.

First 100KB of a 100GB+ file:

curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"

First 100KB at the 100GB mark:

curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"

hyperknot · 2024-09-25T08:53:16 1727254396

Here the requests are really really small, on average 405 bytes each. I guess in your genomics work you are making larger requests, so probably it's not so much of an issue.

BTW, we are discussing latency with bdon in this issue, it seems to be specific to Cloudflare: https://github.com/hyperknot/openfreemap/issues/16

apitman · 2024-09-26T15:08:14 1727363294

I just tried @bmon's curl examples above with 100 byte requests. Similar results. I think the Cloudflare explanation is more likely.