I have spent hours arguing with someone at my work that the issue we are experiencing at our remote locations is not due to bandwidth, but latency. These graphics are exactly what I've been looking for to help get my point across.
People do a speedtest and see low (sub-100) numbers and think that's why their video call is failing. Never mind the fact that Zoom only needs 3 Mbps for 1080p video.
Latency is a cruel mistress. Had a Customer who was using an old Win32 app that did a ton of individual SELECT queries against the database server to render the UI. They tried to put it on the end of a VPN connection and it was excruciating. The old vendor kept trying to convince them to add bandwidth. Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.)
I was involved in some physical network inventory software a dozen years ago. One team produced some new web-based software for it, for use by field agents and such. The first version we got to review was rather bad in various important areas; my favourite was that search would take over thirty seconds in common real-world deployment environment: it was implemented in .NET stuff that makes server calls easy to do by accident and in unnecessarily blocking fashion, and searched by each of the 34 entity types individually, in sequence; and some .NET session thing meant that if the connection had been idle for ten minutes or something, it would even then need to retry every request that got queued while the session was “stale”, which was all of them. So you ended up with 68 sequential requests, on up to half a second’s latency (high-latency 3G or 2G or geostationary satellite)… so yeah, 30 seconds.
They’d only developed it with sub-millisecond latency to the server, so they never noticed this.
I don’t think it was a coincidence that the team was US-based: in Australia, we’re used to internet stuff having hundreds of milliseconds of latency, since so much of the internet is US-hosted, so I think Australians would be more likely to notice such issues early on. All those studies about people abandoning pages if they take more than two seconds to load… at those times, it was a rare page that has even started rendering that soon, because of request waterfalls and high latency. (These days, it’s somewhat more common to have CDNs fix the worst of the problem.)
I think that latency people grow up with has a huge impact on how careful they are when using the internet.
Having gotten my hands on an experimental 128kbps link early on, but later and moving to the countryside with a 56kb-1Mbps really spotty connection made me really appreciate local state as every time things blocked on the internet made it pretty notorious.
I'm glad there's a push for synchronized, local-first state now, as roaming around on mobile or with a laptop hopping on wifi can only perform nicely with local state.
> I think that latency people grow up with has a huge impact on how careful they are when using the internet.
This reminds me a ton about the studies showing harm to children's future mental health as a result of growing up poor. I definitely "hoard" data locally because I grew up with accessing BBSs at 1200 baud, moving up thru dial-uo Internet, and eventually shitty DSL. Services and products that rely on constant access to the Internet strike me as dodgy and likely to fail as a result of my "upbringing".
I test all my web dev on the other side of a 4G modem connected to an MVNO. It forces you to be considerate of both bandwidth and latency as it's about 5-20Mbps in the city with 120ms average latency.
It's not at all impossible to design fast and responsive sites and single page applications under these constraints, you just have to be aware of them, and actively target it during the full course of development.
Many years ago I was called into troubleshoot the rollout of a new web based customer management application that was replacing a terminal green screen one. I was flummoxed finding all the developers had only ever tested their app running from workstations on 100Mbps switches where the target offices for this application were connected by 128kbps ISDN lines. I was able demonstrate how each 12kB of their application was going to take 1 second. (It was amazing to see their HTML still full of comments, long variable names, etc). I don't even think they had discovered gzip compression. This was after many millions of dollars had already been spent on the development project.
part of the reason why modern software is so crappy, is because developers often have thee most powerful machines (MacBook pro like) and don't even realize how resource hungry and crrappy their software at lower end devices
Another part of the reason is that in every company I have ever worked for some SEO dude insists we need to add literally 10 different tracking scripts and insists they need to load first and consume megabytes of data. At my last gig the landing page was 190kb of HTML, CSS and JS from us, 800kb of images and literally 8mb in vendor scripts we were mandated to load as early as possible. Of course we fought it, of course nobody cared.
GitHub and everything Atlassian deserve to be thrown in the pit-of-shame and laughed at for perpetuity.
Jira is so agonisingly slow it’s a wonder anyone actually pays for it. Are the devs who work on it held against their will or something? It’s ludicrous.
GitHub gets worse every day, with the worst sin being their cutesy little homegrown loading bar, which is literally never faster than simple reloading the page.
These days people on HN love to advocate sockety frontend solutions like Phoenix LiveView, where every stateful interaction causes a roundtrip, including ones that don’t require any new data (or all required data can be sent in a batch at the beginning / milestones). It’s like they forgot the network exists and is uneven and think everything’s happening in-process.
To ward off potential criticism: I know you can mix client-side updates with server-side updates in LiveView and co. I’ve tried. Maintaining client-side changes on a managed DOM, sort of like maintaining a long-lived topic branch that diverges from master, sucks.
In the Google office we had (perhaps they still have) a deliberately crappy wifi you can connect to with your device, to experience extra latency and latency spikes and random low bandwidth. All in the same of this kind of testing.
Yes, Bloomberg had fun with latency because of their datacenter locations (about a decade ago they still only had two and a half close to New York). Pages that would paint acceptably in London would be unacceptable in Tokyo as when poorly designed they would require several round trips to render. Once the page rendered there was still the matter of updating the prices, which was handled by separately streaming data from servers close to the markets to the terminals. A very different architecture but rather difficult to test because of the significant terminal-side functionality.
Multiple requests sent in series (e.g. because each depends on the previous one) is also a problem with web apps. When you're developing locally or testing on a staging server near you, the latency is negligible and you might not notice.
Chrome's developer tools allow you to disable caching and choose to simulate a specific network type. Most people know those settings restrict the throughput. But they also increase the request latency. They do this on a request-by-request basis, not at packet level, so it's only an approximation. Still a good test.
Had some devs in another country complaining that their database query was taking hours to complete but doing it from a server in the same datacenter took a few minutes. Took some weeks of emails and a meeting or two until they understood that we couldn't do anything, I had to actually say that we couldn't do anything about latency unless they physically move their country closer to us.
I have been there many times. While the network guy can solve some of the latency in things like TCP handshakes and use compression and caching with magic black boxes you can't fix the actual application query and acknowledgement requirements that might be there.
> Fortunately the Customer asked the question "Why does the app work fine across the 1.544Mbps T1 to our other office?" (The T1 had sub-5ms latency.)
That reminds me on the atrocious performance of Apple's TimeMachine with small files. Running backups on SSDs is fast, but cable ethernet is noticeably worse, and even WiFi 6 is utterly disgraceful.
To my knowledge you can't even go and say "do not include any folder named vendor (PHP) or node_modules (JS)", because (at least on my machine) these stacks tend to be the worst offenders in creating hundreds of thousands of small files.
That is why I've been successfully working from home for almost a decade starting on an LTE connection that was 5Mb up and 10Mb down(notice this is small b as in bits). No problem at all... Why because most of the time latency was good.
I'm still on the same LTE connection, but everyone kept telling me how my speeds are crap and how I should update to a new LTE cat 21 router. So I got one of more popular models ZTE MF289F. And the speed increased to 50Mb up 75Mb down on a speed test. But all my calls suddenly felt very choppy and the perceived Web browsing was unbearably slow... What happened? Well, the router would just decide every day or so to up it's latency to Google.com from 15ms to 150ms until it was restarted. But that is not all. Even when the ping latency was fine it still felt slower than my ancient tplink lte router... So the zte went into a drawer waiting for the times I'll have time to put Linux on it. And the tplink went back on top of my antenna mast.
speedtest.net added support for tracking latency under load a few years ago. they show ping during up/dl now. That's the number to show your colleague.
However they tend to use something like the 75th percentile and throw out real data. The waveform bufferbloat test does 95% and supplies whisker charts. cloudflare also.
No web test tests up + down at the same time, which is the worst scenario. crusader and flent.org's rrul test do.
Rathan than argue with your colleague, why not just slap an OpenWrt box as a transparent bridge inline and configure CAKE SQM?
Funnily enough I have found since moving from London to Sydney that people here are far more understanding of the difference between latency and throughput. Being 200ms from anyone else on the internet will do that to you!
Not just latency, but also jitter. Jitter was the biggest issue we had when broadcasting and streaming video. You don't need a lot of bandwidth, latency can be surivived... but jitter will ruin your experience like nothing else.
People do a speedtest and see low (sub-100) numbers and think that's why their video call is failing. Never mind the fact that Zoom only needs 3 Mbps for 1080p video.