Hacker News new | past | comments | ask | show | jobs | submit login

Verizon FIOS in my area was throttling down Youtube and Netflix to unusable speeds. I was paying for a higher end connection speed and getting constant speed issues.

So then switched to a cable provider, pay half what was paying with Verizon, get lower advertised speeds yet have a much more reliable and consistent connection.

Also Verizon had free installation at that time when I signed up, so they installed a bunch of their equipment on premises, even a phone (including a battery pack) and now after I dropped them they come to door, in person, every other month begging me to reconsider.




Verizon did not throttle speeds. Instead, Verizon wanted to be paid for peering link upgrades. Such links are only free when two backbone providers feel that they both benefit from making them free. The division of Verizon that manages backbone is different from the one that runs Verizon Fios, so the fact that the other division had already been paid should not matter very much in that mindset. It was the refusal of both sides to budge and the growth of Netflix traffic that caused bandwidth requirements to exceed the network capacity. That is what people incorrectly called throttling. Setting up the Hurricane Electric tunnel broker so that Netflix traffic would be carried on IPv6 was a reliable workaround for me in New York.

Netflix might have been able to get Verizon to be more reasonable if they modified their streaming clients to upload random numbers back to them in excess of the amount downloaded. Then Verizon's logic would have dictated that Verizon owed Netflix's backbone provider money and Verizon would have wanted to avoid paying rather than be paid. Such a situation should have caused the links that Netflix traffic used to be upgraded without much negotiation. I was surprised that Netflix did not do that as a workaround. That would have made anyone even superficially familiar with game theory smile.


However they structured the deal, as an end user, it seemed like throttling to me. I could download ISOs and other large binary files ok. But somehow streaming from a few sites seemed broken.

That is one thing to expect that on a low end 2Mbps connection, but I was paying for a 100Mbps wouldn't expect to have any issues at rate I was having them. How they decided to structure their payments downstream is their problem so to speak (putting on my non-technical consumer hat on).

I am glad I had a choice to switch and now with half the price or so (and also half the advertised bandwidth (65Mbps) I seem to get a much better user experience.


I fully expect to be down voted for saying this, but the way that the Internet works means that you have no guarantee that communication between two computers via Internet Protocol packets will operate at either's last mile connection speed or even operate at all. When you get an Internet connection from an ISP, what you are getting is a connection to their network. Access to the other networks that make up the Internet is something that they both do not guarantee and cannot guarantee. The only reason that you get it at all is that if most of the others were not typically reachable from them, they would lose business to an ISP from which most of the others are typically reachable. If the links degrade, it only matters to them when they perceive inaction as impacting their profits more than doing something. In some cases, the ISPs cannot reasonably do anything about because the owner of the other network refuses to provide sane terms for doing something or even allow access at all. For example, China and the US were connected by a 155Mbps link 10 years ago because Chinese ISPs wanted non-Chinese ISPs to pay for transit connections at >10 times the going rate. In the case of the 25.0.0.0/8 block, the owner is the British government, which refuses to allow interconnection. Consequently, it is simply not possible for any commercial ISP to provide quality access to every other network on the Internet or even provide access at all, even before we get to the cases that cerrain ISPs are jerks, like Verizon was with Netflix.

I was affected by that peering dispute too. That lead me to research it, understand it and workaround it. Consequently, I use that knowledge to implement workaround whenever a peering issue affects me.

To name one, I observed 1.3 second average RTTs between Shanghai and New York when I was visiting family there in 2014. My research into the Verizon peering disputes prepare me fairly well for dealing with it. I initially used OpenVPN over TCP and my pings were running inside the VPN tunnel. Packet loss was typically in the range of 10% to 20%. There were occasional latency spikes to 30 seconds. Switching to UDP lowered latencies to ~300 ms in either case, but bandwidth was rather low. I forget how bad it was in absolute terms, but I can say that it was too bad to use YouTube or even VoIP. What helped was routing through servers in Japan.

The trick was to do a perverse variation on a NAT where the LAN and WLAN interfaces are the same and the NAT is effectively a proxy for a port on a remote system. It works well for routing around peering disputes when you can get a VM in a place where there are not peering disputes between the local ISP and the VM provider and the VM provider and the destination. Trace route results at the time suggested that Japanese ISPs appeared to maintain both sides of the underwater link between them and China while US ISPs only control the network sockets at the Internet exchanges in the US. Routing through the Japanese internet backbone presumably eliminated the congested US-Chinese peering links from the path, which showed because packet loss went into the single digits. I do not remember the exact numbers, but it also put speed between 1Mbps and 10Mbps. That was enough for me to make VoIP calls to NY while I was visiting family in Shanghai and YouTube was usable and even load YouTube videos.

However, the connection was still bad enough to make Verizon's peering dispute with Netflix look mild in comparison. There were occasional latency spikes to 30 seconds, which appeared to be from buffer bloat in the great firewall. Google search results on the Chinese backbone at the time suggested that the entire county also appeared to have a star-like network topology for its backbone, with the exception being that Hong Kong was on the outside of that. My trace route results from Shanghai then were consistent with it.

Another OSS developer in Shanghai informed me at the time that the three major ISPs in China have peering disputes among themselves despite the Chinese government owning all three. He claimed that communications over his 100Mbps last mile connection at his house with his employer's office operated at dial up speeds because of that. I did not confirm that, but I believed him. What I saw of Chinese Internet connectivity while I was ther struck me as being incredibly terrible.

I still remember how bad things were with Verizon and Netflix. However, the reality is that things could have been many times worse like they were in China when I was there in 2014. At least with Verizon and Netflix, you had the option of the HE tunnel broker. In China, I saw ISPs using carrier grade NAT, such that using IPv6 via the tunnel broker is impossible. That is not to say that IPv6 via the tunnel broker is some sort of magic cure for issues. The only reason IPv6 via the tunnel worked during that dispute was because there was no congestion between Hurricane Electric and Verizon as well as between Hurricane Electric and Netflix. If many people had tried using it, it would likely have degraded too.


You are technically correct, but but missing the real issue.

I have 0 interest in buying a fast line to comcasts internal services, they are selling me a way to connect to the rest of the internet.


This.

Whether it's what they intended to sell or not, ISPs, through their own marketing material, sell a connection to "the Internet", and unfortunately for them that term cannot fairly be applied to just Comcast's network anymore (though it would make for a comical court case).

If they can't provide what is advertised, specifically "Internet access", then it's false advertising and I expect, and will fight for, my money back.


The real issue is that those of us that are not ASN operators are at the mercy of those who are. The only true solution is finding an ASN operator that has user interests at heart. Municipal fiber is probably a good example of that. There are other ways (all involving getting an Internet backbone connection), but they are not cost competitive with last mile distribution.

A workaround would be to rely on an IPv6 tunnel broker or a VPN tunnel to a VM in some datacenter. As long as the tunnel is unaffected by peering disputes, your traffic over it will live at the other side of the divide between content producers and consumers. That gives a far better experience than relying on a last mile operator to give you access to the broader Internet.

Speaking of which, does anyone know when hacker news will support IPv6?


I am paying to connect to the Internet, if it's slow Verizon fucked up end of story. If Verizon needs to pay netfix extra to get done then fine. Instead Verizon assumes they have full monopoly power and does not care about service, which suggests they need to be broken up.


Such problems are enabled by the mix of having multiple organizations responsible for the path, there being more traffic than it can handle and the organizations not coming to terms on how to handle that. In the case of Verizon and Netflix, the division between Verizon's enterprise and residential services meant that the enterprise division thought that they had not been paid for the traffic.

How would breaking up Verizon solve a problem caused by Verizon's internal divisions being separate?


The garbage return traffic scheme would only work if Verizon and every other ISP were completely asleep at the wheel. The public has enough trouble understanding the network neutrality debate without reading the headline that Netflix is intentionally wasting petabytes of backbone bandwidth every day.


The idea of ingress and egress being roughly equal as suggesting that upgrades should be done at no charge by either party to the other is how peering agreements often work. If the traffic levels are equal (or even skewed in the other direction), on what basis would they ask for money?


I just want to note that Verizon LTE on my mobile has incredibly stable and fast streaming from Netflix and others....

I never have buffering and I use it a TON.... I'm convinced that they want my data consumption over anything


Netflix throttles Verizon Wireless to 600kbps at the A/V codec level to minimize data usage:

http://arstechnica.com/business/2016/03/netflix-throttles-vi...


Well of course they do, the number of people for whom heavy LTE usage is an economically feasible prospect (i.e. unlimited usage) is vanishingly small. Since their network can't tell who pays per GB and who doesn't, yes they want people to consume as much data as possible.


Verizon FIOS was doing this about a year ago in the NYC area. I was getting crappy YouTube and Netflix playback but everything else was fast. My solution was to use a VPN service, and suddenly I was magically streaming Netflix and YouTube in crystal clear HD.

I think what was happening is that the VPN service had servers in the NYC area that had great connections to the local ISP network, and they also had a great connection to the Netflix servers, vs Verizon was deliberately keeping their pipe to the Netflix servers small so that there was limited Netflix bandwidth entering the local ISP network.


Setup the hurricane electric IPv6 tunnel broker and you should be immune to disputes between Verizon and Netflix or YouTube as long as the peering links between Hurricane Electric and Verizon remain congestion free. Both Netflix and YouTube traffic will run over IPv6 and HE is a tier one ISP known for aggressively peering with everyone. They would likely be the last ISP to try to charge Netflix and YouTube for connectivity.


They're not actually begging you to come back. They just pay someone shit wages and commission to engage in real-life spam. If they really gave a shit, they would not have lost you as a customer in the first place.

Why improve business fundamentals when it's cheaper to manipulate human emotion through marketing?


Well they paid enough someone to have them drive out to my house. I can see how in their eyes they lost a deal because house already has their equipment installed and I only was a client for 2 years. But well it is their fault.


I guess my point is, how does a company prevent its customers from turning into statistics on a report? You were part of the churn on some report somewhere in their giant number crunching machine.

It's cheap to have someone pester you and 100+ other people that day to see if you'll budge, but it's far from a personal effort to win you back.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: