This should work pretty well for tc/iptables shaping where you count the bytes in the connection and move to a lower class.

But I guess most of the time something like aria2[1] works better for real downloads - if you shape only for a single tcp stream this should also defeat this or at least speedup the download enough that the rate limit doesn't matter.

On the server side it's probably easy to stop this - nginx seems to have all you need[2],[3]. Just set a unique cookie for the download and deny access otherwise - not sure what the shared hosters are doing but likely something similar (mandatory waiting time before the cookie or url-hash is set, limit access based on connections/hash)

    limit_conn_zone $uid_got zone=cookie:10m;

    server {
        mp4;
        limit_conn cookie 1;
       	limit_rate_after 10m; 
        limit_rate       512k;

    }

On the other hand it probably screws users for mostly no reason most of the time.

1: https://aria2.github.io/

2: http://nginx.org/en/docs/http/ngx_http_limit_conn_module.htm...

3: http://nginx.org/en/docs/http/ngx_http_userid_module.html

Can you use aria2 to abuse the network shaping for something like SFTP? Or does the handshake dwarf the performance benefit?

Good question. I think the only thing aria2 does is using multiple connections - that should work on SFTP.

I could TIAS later but does it somehow carry your login between connections?

The commit messages for this are gold. https://github.com/matildah/hellcat/commits/master

"mkdir src, put stuff in it."

"we need 4 argz"

"i fail at operators"

I've seen other scripts utilizing curls --speed-limit and --continue-at for http to restart the download after the throttling kicks in and resume the download on a new connection. Really nice!

I wrote a while ago a shell script utilizing curl to do something similar. But many pages stop high speed download after about 100mb. Not just a couple of kb. So using this approach is actually quite nice. https://github.com/BrandiATMuhkuh/downloadAccelerator

Could anyone with a custom OS make their own TCP stack that violates all rate limiting and always sends data as fast as possible / reports a humongous window size?

>violates all rate limiting

Rate limiting is performed upstream; the network isn't just asking nicely. Your packets will be either queued or dropped if you send them faster then allowed.

Some ISPs allow high speeds for the first N megabytes (often 20 or so), then throttle the shit out of the rest. This trick presents an entire stream as a bunch of little streams, so that you get the "first 20MB" treatment for the whole thing.

Depending on circumstances, the "upstream" place where rate limiting is performed may be your cable modem. People have been known to reflash their cable modems with cracked firmware that doesn't honor the rate limit. This is easy enough to detect and punish further upstream.

That won't defeat QoS shapers/policers that will simply drop that traffic.

Traffic shapers usually aren't actually inspecting or tracking the internal state of the TCP stack at either end of the connection. They just count bytes and credit them to a particular bucket by identifying the flow/protocol/class based on port numbers, IPs, etc.

There are a lot of ways to do packet prioritization rules, and only some of them can be gamed this trivially. For example, a large hierarchical HTP or HFSC ruleset may prioritize new flows over existing flows but still apply an aggregate limit on the whole protocol. More modern methods like fq_codel automatically give new connections an advantage, but it only applies to a very small amount of data and doesn't treat new flows any better than sparse flows.

Is it really faster to do a whole FIN / ACK / FIN / ACK / SYN / SYNACK / ACK every N bytes? How much does this sort of traffic shaping typically throttle, and how long does it typically give a connection?

Can this be easily parallelized? I'll bet it could.

If N is large enough, maybe.

However this assumes that the state of the traffic shaping system is per connection. If it's per site<>client or just client<>net over a duration then this wouldn't help.

Reminds me a lot of hierarchical token bucket filters and letting clients burst over their normal allocation as long as there was spare bandwidth (then squeezing them when others woke up and wanted to use bandwidth as well).

Comcast had a feature called "Powerboost" (http://www.dslreports.com/faq/14520) that would let you burst up 100mbit or something for the first 20MB of a file. I'm not sure if this is still being offered, but I bet it would be awesome with Hellcat.

Because Powerboost and similar systems are per-customer instead of per-TCP-connection, Hellcat cannot bypass them.

I think that might be a bit outdated since it refers to DOCSIS1, and I haven't ever received a package offer from Comcast before below 200Mbps or so

In my area, I have the highest tier of Comcast service (residential) at 75 mbps.

This exact tool is one I've wanted for a while, since I've had the exact scenario of trying to download a file only for the download to slow by 10x after several seconds. In such scenarios, I'm able to download the file 10x faster by manually restarting with --continue every time the speed drops.

Not only is the handshake expensive, but TCP has something called "slow start" which effectively artificially limits the rate of data in the beginning of the connection till the network capacity is discovered.

An ideal N would be high enough where the cost of the TCP handshake and slow-start are minimal, but small enough not to trip the traffic shaper to downgrade the connection from the fast flow to the slower one.

Is slow start really still in use? I'm not too familiar with congestion control algorithms, but I thought that TCP stacks have moved beyond that...

You can make up for that with parallizing

Is this a problem where multiple connections wouldn't help?

I can add an option to saldl[1] to use a new connection with each chunk. But I'm not sure there are real world examples where this would help.

[1] https://github.com/saldl/saldl

The HN comments suggest there are other ways to do this. Is this another one?

http://lftp.yar.ru/lftp-man.html

See pget.

I really like the simplicity of hellcat, but IMO original netcat's brilliance is partly due to its portability and resistance to bit rot; it does not use getaddrinfo and I cannot think of any good reason one needs to use it.