Hacker News new | past | comments | ask | show | jobs | submit login
Netflix replies to Verizon cease and desist letter (cnbc.com)
442 points by hornokplease on June 5, 2014 | hide | past | favorite | 349 comments



Awesome response to the C&D, via spokesman Jonathan Friedland:

"This is about consumers not getting what they paid for from their broadband provider. We are trying to provide more transparency, just like we do with the ISP Speed Index, and Verizon is trying to shut down that discussion."


Playing Devil's advocate here for a second:

If Netflix is 35% of global internet traffic at peak capacity (as per the Akamai CEO's comments at a number of events), is it really fair to treat them like every other company? That is to say, if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?

I'm not taking the stance that this is correct, only querying as to why the dynamics here are such that we automatically assume Netflix is in the right. Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.

Yes, ISP customers are paying for access but the business reality is that the ARPU per subscriber is decreasing every year at a rate lower than additional subscriber acquisition can sate. As Wall Street demands growth, it has to come from somewhere, and since Carriers are not able to sell consumers additional services (no matter how hard they try), they need to find another set of wallets (content providers). They're not utilities and have a profit motive, right?

Again, I'm not saying this is correct, simply trying to add another viewpoint to the conversation.


An ISP's job is to provide internet access to last mile users, such as ourselves.

If they choose to reduce that speed for any reason, or throttle any kind of connection they are entitled to do that but they can't complain when someone else says that is what they are doing.

Personally, I think given how much public money has gone into infrastructure and the amount of money I pay in taxes it seems insane for me to first pay for the ISP's infrastructure with taxes then have to pay them again. So while they have a right to throttle, they don't have a right to throttle my connection on infrastructure that they did not pay 100% for. Which probably accounts for the majority of their infrastructure.


s/last mile/first mile/


As the issue is data flowing FROM Netflix TO the end user, I think your 'correction' may be incorrect, as it is indeed the 'last mile' that is in discussion.


"last mile" doesn't have to do with which direction the bits are flowing (although there may be a loose correlation). It has to do with the fact that the customer is on the outside edge of the network. A connection from Verizon to Cogent isn't considered last mile because there are other networks on the other side of Cogent. Since you aren't providing a connection from your ISP to some other network, you are the last mile.


From the point of view of a carrier, yes. I think you missed the point.


At the risk of diving too deep into a semantic discussion...

http://en.wikipedia.org/wiki/Last_mile

"Last Mile" is a common telco industry term. It's not about framing, it's the correct term for the connection. It does not connote any specific value to the connection relative to others, nor is it derogatory. It's simply saying where in the network the connection lives. A "last mile" connection is not an inter-network connection, it's a specific link within the network that your ISP has built from the rest of their network to you, the individual customer.


Saying "first mile" re-frames the conversation from the point of view of the customer, and is important because the Verizons and Comcasts of the world are very good at framing the debate in terms that benefit them.

Last mile is from the POV of the carrier, and implies that something is being delivered to you.

First mile is from the POV of the customer, and implies that this is the first mile of your internet connection.

The direction data is flowing is irrelevant.


An ISP's job, as a private company, is to maximize shareholder value. The circuit consumers are sold is a "best-effort" connection in stark contrast to, say, a T1, which is a dedicated circuit.

It's also very hard to prove that Verizon is throttling, versus, say, simply physically underprovisioning the connection.

With respect to public money, I think you've hit the nail on the head. The crux of the Net Neutrality debate, to my mind, is whether these networks are public or private. If they're private, Net Neutrality is bullshit. If they're public, it's obvious.

It is in consumer interest for the networks to be public, but the same could be said about the socialization of any number of industries. I'm constantly fascinated by the way folks consider socializing the telecom industry to be incredibly reasonable and socializing, say, the healthcare industry to be insanity.

Edit: Maybe I need to tag all of these posts with #DevilsAdvocate to avoid being downvoted into oblivion.


Oh no, not the "maximising shareholder value" argument again! This argument has three fundamental problems. First, it is difficult to tell what would "maximise shareholder value". Second, it misses the fact that something that could "maximise shareholder value" on short term could very well minimise "shareholder value" on long term (or the other way around). Third, and this is the biggest one, people involved in a private company (employees, directors, shareholders) often have an agenda that goes beyond simply making more money. That's because they are human beings.


1) Agreed.

2) Agreed.

3) All of the big Telcos are public, but your point stands.

To be explicit, I wholeheartedly agree with you about humans making pseudo-random decisions. I'm trying to explain the dynamics of the Telco position, from their vantage point because I think it helps the discussion to view things from other angles.


The telcos are publicly traded, but not publicly owned in the sense of being government-owned and regulated. And that's one of the bigger problems here - the last-mile physical infrastructure should probably be publicly owned.


> I'm constantly fascinated by the way folks consider socializing the telecom industry to be incredibly reasonable and socializing, say, the healthcare industry to be insanity.

Can't I want both to be socialized?


Where do you draw the line? What shouldn't we socialize?


Excellent question. Anything that doesn't have a natural monopoly. I can say with certainty that the following should be socialized or delivered in non-profit ways:

* Water

* Sewer

* Power

* Internet

* Healthcare

* Fire

* Police

* Roads

* Education (added per /u/CodeMage)

Feel free to chime in if you can add anything I forgot.


Not a bad list.

Cheers.

I'll note that we should have explicit public subsidies for Internet and that I would much prefer to have public networks with private operators (to my mind that's heaven for competitive markets but politically impossible in the US).


I completely agree with you. The last mile should be public, with anyone permitted to provide service to customers (similar to the old DSL/CLEC/ILEC model).


Or you could have public last miles with openaccess requirements providing competition on the parts that can see competition.


I'd just like to point out that Adam Smith himself (the "Carl Marx of Capitalism") agrees with you for fire, police, and roads at the very least (I haven't read everything he wrote). If he were born today rather than in 1723 I suspect his list would look very much like yours.


I will take that as a compliment! Thank you!


It's interesting that you seem to view providing Internet access as a natural monopoly. While (from what I see as an outsider) it seems to be common in the US for households to have only one practical ISP available, that is certainly not the only model possible.

For example, here in the UK, we have the formerly nationalised telephone network provider required to allow individual ISPs access to key parts of its infrastructure via local loop unbundling, so those ISPs compete for the contracts with individual customers. And while that infrastructure is indeed common and so in a sense a monopoly at that level, there are also multiple cable companies and mobile networks with completely independent infrastructure all the way down. There have even been a couple of impressive success stories where installing high speed broadband infrastructure was too expensive or risky in a certain geographic area for any of the big commercial providers to take it on but the local community devised their own solution, sometimes even finishing up with a better connection than what most of us get in big cities.

In other news, prices for Internet access in the UK seem to be dramatically cheaper than most places in the US.


The reason that Americans view Internet access as a natural monopoly is because there is a monopoly--the city. ISPs have to negotiate rents from the city government. While it is possible for a city to offer fair-access terms they have largely discovered that they can demand rents much higher than you would expect by keeping the number of tenants small.

These rents comprise something like 5-10% of city revenue. And so everybody gets what they want:

* The city gets a free 7% budget bump, and doesn't care who pays for it as long as it's too complex to be an election issue

* ISPs get a monopoly, and don't care that they play the scapegoat because consumers have no other choices anyway

* Citizens get better government services at lower tax rates. Sure, they pay for it via a backdoor tax on internet access but nobody really notices that and they direct their anger at the ISP rather than the elected government

Really the only way to throw a wrench in this system is something like Google Fiber. Nobody wants to be the elected official that turned Google away, because that is actually a sound-bytey enough issue that could come up at re-election. And Google, as a company that doesn't really care about being a profitable ISP, will only move in to a city if it gets a pretty good deal on the rents.


> And while that infrastructure is indeed common and so in a sense a monopoly at that level, there are also multiple cable companies and mobile networks with completely independent infrastructure all the way down. There have even been a couple of impressive success stories where installing high speed broadband infrastructure was too expensive or risky in a certain geographic area for any of the big commercial providers to take it on but the local community devised their own solution, sometimes even finishing up with a better connection than what most of us get in big cities.

This is what I'm advocating for. A public last mile that any service provider can use.


What you describe is the definition of a natural monopoly. However, the monopoly comes from physical wires, not whatever flows over them. So, the infrastructure should be a public good (most infrastructure, really, not just wires, since it's more efficient and less wasteful to build a single network and keep it in excellent condition than to build multiple networks providing the same service at a lower quality), but whatever flows over it can be provided privately and competitively (also, each ISP would have to build it's own servers/routers, etc.)


What you describe is the definition of a natural monopoly. However, the monopoly comes from physical wires, not whatever flows over them.

Part of my point was that there is no physical monopoly here, even on the wires.

BT (formerly the national telephone provider) has one set of infrastructure.

Cable companies have their own, literally installing a different box in your home with an independent physical connection to their own network.

If you use one of the mobile providers, which is viable for a home connection in the 3-4G era if you're only using a modest amount of data, there is no physical wiring to the home at all.

There are even a few special cases where people have literally built their own infrastructure as a community, only hooking in to some big provider's network higher up the hierarchy, presumably on some sort of customised, one-off deal.

So while we do have local loop unbundling and a shared-last-mile kind of deal, and we do have ISPs competing for that same physical connection, even that is not a monopoly provision.


Education, Police, Fire, Healthcare do not have natural monopolies.

Still a good list to have around so we can argue about where the line should be drawn. Kudos!


I'd definitely argue that Police and Fire have natural monopolies. It's definitely more efficient for one agency to manage the entire police force for a city, rather than having multiple. Imagine having cops from 5 different services go out to one call, possibly mistaking the others for an armed civilian. You can see where that would lead, given how trigger happy some cops are already. Imagine 5 different fire trucks arriving at a scene. They wouldn't have enough space to do their job with all the others around them. Keeping track of the people at both of these scenes is also nearly impossible with several forces (and it's critical to know who's in the burning building or who's armed in a crime scene.)


However both police and firemen already deal with those issues, especially at the edges of cities. E.g. where I live, at the edge of two, police from both regularly handle incidents.

Firemen are even more heterogeneous, in that they have arrangements for big fires and catastrophes like tornadoes where adjacent units come to help. After my city got hit by an EF-5 tornado, as I was exiting my trashed apartment complex, there were fire units varying from a small Indian tribe that's right across a state border from us (and the city e.g. treats the sewage of their casino) to one 40 miles away. Next day, the markings on my apartment door were from a fire unit the came from 160 miles away.

What is needed is something like unity of command/hierarchy. Having two or more units responsible for the same territory leads to the sorts of problems you cite, e.g. a friend told me the NYC city police and subway police once got into a big firefight. This would have been in the '70s or earlier, now they've been unified.


Healthcare is a highly imperfect market, and that's what makes it subject to either huge amounts of regulation, or outright external management. Consumers are very ill informed. Ccosts are not displayed, treatments options are not well defined, and knowing all of the options and understand them requires more time and education than is reasonable for most cases.

Education is tricky, and honestly, I don't know any better than the next guy how to solve the issue. What's important to one parent may not be important to another. Second language? Art? Music? Sports? Recess? Lectures? History? Religion? Critical thinking? What outcomes are important? Good test scores? Writing?


None of these attributes are incumbent in the nature of delivering health care. They're just things that are part of the system today. As you stated, in today's health care system, true costs are obscured. The exact problem is that nefarious corporate forces have removed most of the real market elements by adding layers of indirection (health insurance), which allows everyone in the industry to maximize their dollars billed and push some paper around, and leaves a great many people totally screwed. The issue is that there's just enough variance and division that there isn't a unified, whole-hearted call for reform, because it works great for 30% of people, works mediocre for 40% of people, and completely leaves the last 30% destroyed. I believe that those ratios are calculated and intentionally kept in place to prevent unification and meaningfully addressing the issue on a social scale.

Health care has become a perverse industry because its market properties were destroyed not by government, but by malicious corporate conglomerations out to maximize their own profit at the expense of human life. People forget that one of the purposes is government is to prevent private organizations from growing too powerful and interfering with the general welfare of the people, because the narrative is rarely framed this way, but it's a very important governmental function. I don't think socialization is the answer, but something has to be done.

I think health care needs to be brought back down to earth, and we should outlaw all non-catastrophic health insurance products. The insurance racket has completely destroyed that market. There are other elements that have caused the health market to get completely out of whack, but insurance is at the very core of the issue, and it has to go away.

When almost half of the country's bankruptcies are due to health care debt, simply sick people trying to stay alive, we have a real social problem that we need to address.

Of course, we did the opposite of this and made it illegal not to have health insurance, only further inflating the size of the monstrosity that has leeched and destroyed the system in the first place. I don't believe writing a blank check, like the other countries have done, is the way out. I believe the way out is going back to a what-you-see-is-what-you-get approach, and forcing medical providers to charge rates people can actually afford, as their business will immediately plummet if they don't seriously rework their billing structure to become affordable again since people will actually be paying it now.


I disagree about health insurance. I think that in the absence of health insurance, health care wouldn't work. Imagine getting in an accident, you're unconscious, you need emergency assistance. If the hospitals weren't reasonably sure you're insured and can pay for the services, they wouldn't be motivated to help you.

Most European countries offer health insurance, and total healthcare costs are much lower (compared to the GDP), so I don't think insurance is the problem (also, AFAIK most countries are bankrupt because of bad bank loans, falling GDP/taxes, and social payments (pensions), not health care costs).

The real problem is that doctors get paid when people are sick, so they are not incentivized to heal them, long-term (without regulation). If doctors were paid when the patients were healthy, you wouldn't have this problem. But I don't know how to implement that.


There's a question though here as to how to define insurance: in your example, everything still works if everyone has catastrophic insurance.

Insurance, classically understood, is risk sharing, not cost sharing. In other words taking it from a "not everyone will use this service" risk sharing model to "everyone will use this at some point" cost sharing model is what causes the problem of hidden pricing.

You can have it both ways (risk sharing and visible prices) because most healthcare costs aren't unforeseen. In other words, the number of accidents where the hospital needs to be relatively sure insurance is involved pales in comparison to the cost of routine visits/care which could be paid for out of pocket and prices would then be visible allowing for competition.


That's a good point, which is why there were some suggestions in some European countries about having a "paid first visit" that is refunded if there is actually something wrong with you. I in general support things like this. I also support some parallel "private" medical system, where you can pay to "skip the line" (but e.g. in Slovenia, only certain procedures can be performed in the private system).

However, there are also many medical costs that can be foreseen (even planned for) but are either "beneficial" for the society (e.g. childbirth) or still risky and essentially no-one's fault (e.g. cancer - most people will get it, but some will get it at 50, others at 95). Incidentally, it's cancer and the catastrophic treatments (e.g. heart surgery) that are most expensive. Possibly also because it's in the big-pharma's interest to keep it this way (at least for cancer).


This is exactly the root of the issue with medical "insurance". We know that everyone will need medical care at some point. Insurance models work only in circumstances where the policy is rarely used; the concept is that everyone wants to be protected in case this rare, almost anomalous event happens, so they're all willing to put up a small payment to pay for that coverage in the extremely unlikely event that it will be utilized. However, in today's society, medical care will be needed by everyone at one point or another, and many people will need it consistently or in large amounts.

Insurance is absolutely the wrong model for something with this kind of usage pattern. The insurance industry is notorious for its antics precisely because they're one of the only industries that can only make money when they don't provide the service you're paying them for. The fact that medicine is so widely used makes this especially egregious with regard to medical insurances. They'll do anything they can to deny your claim because aside from basically accidentally having a healthy group of people in the insurance pool (due to the way corporate policies work, and the new health legislation), denying claims is how they make money.


Police have a natural monopoly because they're agents of the government, and the government by definition has a monopoly on violence.


Education doesn't, but police and fire definitely do: police because beat cops reduce crime for everyone, so they're a natural monopoly. Fire as well, since stopping houses burning down helps stop the entire neighborhood (especially when you have close-together row houses) burning down.

Healthcare is arguably a public good, at least as far as vaccinations and contagious diseases go, since herd immunity is a large way to fight them (especially for contagious diseases for which we have 80%-90% effective vaccines). Not so sure about other illnesses (like cancer) though. I'm personally for socialized medicine for other reasons (humanitarian), but I agree that the natural monopoly argument is not applicable there.


A natural monopoly doesn't mean it provides a public good, it means that a single entity can provide the service more efficiently than multiple competing entities.


Yeah you're right. I'm conflating the two and being confused. I apologize.


Excellent list. I'd like to add education to it.


Prisons

Orphanages

Mental institutions

Defense


I draw the line at things that people need in order to fully participate in society. If a good or service doesn't meet that threshold, a private market only solution is likely just fine. Otherwise there ought to be at least some socialized option available.

My list would include: non-elective healthcare, food, housing (same as for food), electricity, water, local/regional transportation (e.g. for commuting to work), internet access and possibly basic internet-capable mobile phones.

Basically "natural monopolies" are certainly all candidates for complete socialization.


Things that don't have a natural monopoly.


Things that work better when they are socialized than when they are privatized. Like healthcare.


Health care only works "better" when socialized if you believe that health care really should be so expensive that they have to print extra money explicitly to pay for it. There's no reason that has to be an integral property of health care. Turns out that most of the time, we don't need hospitals the size of college campuses (incidentally another thing that is completely unaffordable). The reason things cost so much is because true costs are obscured, just as they would be in a socialized system, by insurance.

IMO non-catastrophic insurance should be abolished, the government should overcome the supply chain cycle ("Sorry, I have to charge $500 for this MRI because this machine cost me $4 million") by producing its own functional medical equipment and selling it at or under cost, and health providers need to get back to reality and charge face rates that real people can pay directly. And we all need to come to terms with the fact that some of the extravagance probably needs to be toned down.


"Turns out that most of the time, we don't need hospitals the size of college campuses (incidentally another thing that is completely unaffordable)."

However people are not willing to accept the consequences of hospitals that can't surge in catastrophes. When an EF-5 tornado hit my city 3 years ago, it took out one of our two hospital complexes. Things got ... rather intense at the other (http://stormdoctor.blogspot.com/2011/06/first-response-mode-...) but it being big, and stocked with otherwise excess inventory of all sorts of things, kept the death toll at ~160 instead of a likely 1,000 or so.


We tried both in the UK. Conclusions, roughly, it works for health, not so well for telecoms. I think the best is some sort of hybrid.


I think the issue is more that the ISPs are demanding payment at both ends of the tunnel. They charge you and I for transit (best-effort though it may be) and then when a bulk of their customers traffic turns to a popular network, attempt to charge that network to upgrade their port speed. (something which is customarily handled for free between peered carriers since both are charging their customers) Think of it this way - it's as if I pay UPS to send packages and then they request fees from the cities they deliver to, or they only deliver 100 packages a day to the municipality. That would be insane, and laughed out of court, but that's what Comcast/Verizon are doing.

Also, intentionally physically underprovisioning the connection is throttling, plain and simple. You are limiting the amount of traffic that can traverse that network boundary to 10G/40G/Whatever your port speed is. What difference does it make if I do it in software or simply refuse to bind more ports to the team?


> I'm constantly fascinated by the way folks consider socializing the telecom industry to be incredibly reasonable and socializing, say, the healthcare industry to be insanity.

Is it really the same people arguing for that? First and foremost, the telecom industry is nothing like the healthcare industry, and they shouldn't be reasoned about in the same way. Common carrier status has precise laws and legal meaning surrounding it. Please don't confuse it with socialized medicine.


Agreed, it's not necessarily an issue of full socialization. The government can issue rules that ISPs must abide without taking them over completely.


I just wanted to call out where you say that it's "hard to prove" that ISPs are purposefully degrading service.

Hard to prove in a court of law, perhaps, but the speeds of access to the businesses being extorted can tell a pretty convincing story.

This chart: http://appledailyreport.com/backend/wp-content/uploads/2014/... paints a nice picture of how ISPs want to extort businesses that are in more profitable lines of work: notice how speeds change from before Comcast started trying to extort from Netflix, to during the negotiations, and after Netflix payed their protection money.

Level3's article also called this out: the ISPs are purposefully, physically, underprovisioning their connections, because it gives them leverage over companies like Netflix.

As to why the internet is different: the internet is not like healthcare. Socialized or not-socialized medicine is obviously not that big of a deal to the future of a nation; both the US and Britain were economically competitive while having very different systems.

In contrast, the Internet obviously matters a great deal to the future. And if various players (hollywood; last-mile ISPs) can manage to appoint themselves gatekeepers and engage in rent-seeking behavior on the internet (in the United States, at any rate), well, HN is acutely aware of what that will mean for the future of innovation, and the future of nations as well.


Sorry I was speaking from a court of law viewpoint. It's really obvious if you look at the network charts but it would be really hard to get those into a court for a lot of reasons.

I understand that it would be a negative impactor on the future of nations and innovation; I'm asking if that's enough to actually drive change. My sense is that, historically, with respect to ISP behavior, over time they get what they want. I'm not saying this is right, I'm simply noticing a trend, just as many have noted that less-private social networks win over more privacy focused businesses.


At the bottom of this you can find a description of how Netflix at the moment is detecting these problems: http://blog.streamingmedia.com/2014/06/netflixs-network-cong...

Sounds sufficiently sophisticated.


As to the shareholder--Max/profits argument I use my ISP as an example. I own shares of stock. I will never by a share of Comcast stock for these reasons: I hate the company. I have hated the company since it's inception. I believe it should be split up by the federal government-- it's a monopoly in my house. Comcast employees hate Comcast. Comcast got much worse when they found out they could overcharge/lie for accesses to the Internet. Many investors don't invest in companies they Hate.


Maximizing shareholder value is less important than offering good customer service. Corporate greed is so disgusting.


Except where the executives view the shareholders as their customers and the end users as the product.


Which nobody does.


> It's also very hard to prove that Verizon is throttling, versus, say, simply physically underprovisioning the connection.

It doesn't matter either way, in the end the experience is the same.


> An ISP's job, as a private company, is to maximize shareholder value.

You lost me there.

"There is only one valid definition of a business purpose: to create a customer." ~Peter Drucker (whose writings contributed to the philosophical and practical foundations of the modern business corporation)

“On the face of it, shareholder value is the dumbest idea in the world. Shareholder value is a result, not a strategy… your main constituencies are your employees, your customers and your products. Managers and investors should not set share price increases as their overarching goal. … Short-term profits should be allied with an increase in the long-term value of a company.” ~Jack Welch (former very successful CEO of GE)


That's true except in markets where saturation has been achieved. In a market with 80-100%+ saturation, the per subscriber revenue does not increase year over year. In these cases, creating a customer doesn't actually increase shareholder value because there aren't enough customers that can be created to combat the atrophy of existing revenue.

Jack Welch was hugely successful in creating and competing in new markets specifically because competing in saturated markets is not profitable.

Those models are great for industries in the 0-80% saturation growth period, but aren't really applicable to high saturation markets.


The issue is that value doesn't necessarily mean money. In terms of money, is Apple really the most profitable company in the world? We know that it's not, so why do they have the highest market cap? Because people value their shares for more than just the present dollar value. People want to be part of Apple regardless of its profitability.

The negative shareholder value argument depends on a summary characterization of all investors as people who only care about money. Not all investors are like that. You can create shareholder value out of anything that your shareholders value, which is not to say that "shareholder value" should even be a serious metric. It's just a narrative that anti-corporate people have drummed up.


> "There is only one valid definition of a business purpose: to create a customer."

And yet financial statements are reported in dollar units, not in customer units.


A business should be solvent, so of course they're in dollar units.

A business is a balancing act of many "should"s and "could"s.

But when your top priority, or god forbid, sole priority, becomes "maximizing" money, versus contributing in a meaningful way to society, your business no longer deserves to exist.


"why does Verizon have to subsidize their costs of business?"

Because I, a Verizon customer, have paid to access Netflix.

I pay monthly for bandwidth to use as I see fit. It doesn't matter if all of the customers are using their bandwidth to go to one source. It doesn't matter: it's our bandwidth to use how we see fit.

It is Verizon's responsibility to provide us with the service that we are paying for. They advertise a service, they market this service, and they charge us for the service: but then fail to deliver on their promises because they're charging someone else too?

They sell us capped packages and tell us we have a set amount of data. We cannot exceed the data, because the infrastructure can't take it and we have to pay for our usage. We have to pay for our demand, that's the point of caps. Make the users pay for using.

That to me implies that we are paying for a set amount of data! That we are paying for it, not anyone else.

We pay for the upgrades that are necessary to deliver data to our homes. That is the point of a subscription.

This is like saying: all of the telephone customers are calling Pizza Hut, so Pizza Hut needs to pay the phone company to upgrade the switches at the Phone Company. What???

We've already paid for the infrastructure required to deliver the data we're paying for.


If you want to play devil's advocate then let's advocate for all the devils.

If Comcast wants to offer their customers video over a traditional cable network, they pay content providers.

Why should the Internet be different? Shouldn't ISPs be paying people who provide the content their customers are buying?


Remember that when the cable companies started they were competing with over-the-air TV. To convince people to switch to cable, they had to provide something valuable enough to justify the cost. Content was that thing.

The reason ISPs can charge both the customers and the content providers is because we have no other choice. Where I live I have a choice of two providers: Comcast or Verizon. My choices are: use one of these two, or have no Internet. Not much of a choice.


Actually, it was both content and quality. They used to advertise that a cable signal was clearer than one from an antenna and stood up to weather, etc.


I've never thought of it that way. That's a very interesting way to look at it! Isn't netflix driving business up for ISPs? The faster ISP gets business? I could be mistaken though.


Aren't content providers always driving up prices for content carriers? Consumers are paying for content delivery and if your business can't deliver the quantity or quality they want and someone switches to your competitors because of it that's your problem.


If you consider everything on the Internet to be content, then would that require ISPs to be paying every website that they load? They would, of course, pass that cost along to us with whatever markup they want. Enjoy your $24.99/month Netflix, delivered to you by Time Warner Cable! Want to use reddit too? That's another $3.99/month, but look at all those grumpy cats you get for that! Facebook is another $4.99/month, but you can't put a price on virtual friendship, right? :)


I think that the way the cable network works relative to the internet is a historical artifact versus modern parlance. My opinion is that precedent is not actually a guiding factor in how future arrangements should be determined.

The Internet is different in many ways, but notably because cable companies aren't doing peering and don't have to deal with the complex settlement of international data trafficking.


This is going under the false assumption that when I bought bandwidth from my ISP they didn't _really_ need to provide me with that bandwidth for realzies. This isn't netflix driving network upgrades this is Verizon customers driving network upgrades. If VZ didn't promise the customers the bandwidth they wouldn't need to upgrade their backhaul to provide it. I mean it would be like Verizon wireless charging apple to connect iPhones to their network because "I phones drive up the amount of data customers consume"

How is that viewpoint even remotely considered valid ?


... Your wireless network service is best effort. This is why your service provider doesn't have to compensate you for dropped calls or dropped packets or towers with insufficient capacity.

The word "promise" is not an accurate reading of the legal concepts at play in bandwidth purchase. Dedicated bandwidth for consumers is expensive and completely unavailable on cellular networks.

Verizon also doesn't have to let iPhones connect to their network and only does so because they benefit financially from the relationship. Trust me, putting a device onto a major cellular network in the US is highly political, full of peril and financial alchemy.

Source: I do lots of telecom stuff specifically around wireless.


> Your wireless network service is best effort.

Wouldn't intentionally throttling and/or not keeping up your capacity intentionally be a violation of this "best effort" doctrine? Not arguing, honest question.


To the best of my knowledge, Net Neutrality doesn't apply to wireless networks.


However, false advertising laws do apply. If they advertise "up to 40Mb/s", then they can't configure their routers to throttle you to 10Mb/s - they can only drop below 40Mb/s due to real contention for limited bandwidth as a result of high utilization.


Ah, I misread that. Sorry.


Amusingly, I've heard carriers use that justification before for pushing phones other than iPhone. Well, that and Apple demands the largest subsidy of all phones. It's a position Apple can sustain because people walk in to the store and ask for Apple and want nothing else, but the carriers really want competition so they will encourage other phones when they can.


> the carriers really want competition so they will encourage other phones when they can.

I've yet to meet any carrier who wants competition, for anything. Their ideal world would be that they would decide which phone the customer can have, what limited service the customer can buy, and how much it is, and take-it-or-leave it. That's how AT&T worked, and every major carrier today is a spin-off from AT&T and still thinks pretty much the same way.


This is a two way street. Verizon customers are demanding content from Netflix. Peering agreements are based on a mutual benefit, in that both sides will use relatively the same amount of bandwidth. Now the bandwidth usage is lopsided towards Netflix, and Verizon is stating this is no longer a mutually beneficial relationship.


Sorry this perfectly valid and relevant quote is being down voted. Disagreeing with the hive mind on HN has consequences now days unfortunately.


It's not valid, as has been discussed about a dozen times in the thread. Peering arrangements to consumer ISPs are not symmetrical, and never have been.


That's what peering is. Both parties agree to connect their networks for mutual benefit of each other. If one side is lopsided on traffic usage, this really is no longer a mutually beneficial relationship. See Wikipedia for more definition information.


Netflix isn't sending their data to customers unsolicited.

Customers are requesting that data, through a pipe which they have already paid for.

It is unfair for Verizon, or any other ISP, to ask to be paid twice for the same bits. What bits I am asking for, as a customer, shouldn't make a damn bit of difference.


There's two ends to the pipe though.

To my understanding, the upstream ends are linked largely thanks to peering, which is based on mutual benefit. At the upstream level, nobody charges other providers because everybody is interacting with everybody- sort of like, "I bought this round, but I know you'll be buying the next" so you don't worry about balancing the books.

Then Netflix comes along, chews up 35% of internet traffic, and basically takes a disproportionate piece of the pie. If I've got everything in my head right, this is against the spirit of how peering is supposed to work, and so it doesn't surprise me if spats start to develop upstream.

If Verizon is unhappy because they have to develop their last-mile network to meet promised speeds, that's Verizon's problem. But if Verizon is unhappy because the upstream is becoming distorted and they are becoming beholden to just one peer (Netflix) instead of an environment of free, balanced exchange- maybe that's not Verizon's problem.


The only reason we're having this discussion is because it's "one peer".

A thought experiment: Let's pretend it's not Netflix for a moment. Let's pretend there's a bunch of competing streaming media services which happen to take up, combined, the same amount of bandwidth that Netflix does.

In this case, Verizon would still have the same problem (if you buy that they have a problem at all, which I personally do not), except they couldn't play at being mafiosos, holding one in particular to what would be a protection racket in any other context since the load is the same amount, just distributed among many much smaller players.

What would they do in this case?

The fact that it's one large company gives them the ease of threate^H^H^H^H^H^H leveragi^H^H^H^H^H^H^H^H negotiating with one entity instead of upgrading their damn network like they're expected to do.



> If Netflix is 35% of global internet traffic at peak capacity..., is it really fair to treat them like every other company?

Yes it is.

If you solve their problems in the general case, then that solution will also work for the next big user, and for the small users as well. If you special-case handling for them, then it will be harder for the next new entrant (who could even be a Netflix-upending upstart) to become established.

They do use more resources, there is no question about that, but they also have higher costs to their ISPs. This cost could be mitigated by them running their own internal network and peering directly with other backbone providers (they already do this to some extent). But in an architectural sense that option is available to anyone (in practice, of course, nobody is doing BGP for their home network).

If you believe in the original promise of the end to end network (a promise which is currently lying in the street gasping for breath) then you don't want netflix special cased.

It's no surprise that Verizon (with POTS roots) and Comcast (hub and spoke cable provider) don't have end-to-end in their DNA, but rather a tollbooth model of the world. It doesn't mean we have to go along with it.


Netflix is half the reason why I'm paying Verizon anything. If Netflix is paying for the network, why should any of my money go to Verizon?

Well, actually, none of my money goes to Verizon. My provider manages to avoid all warnings from Netflix and the graphs from Google show basically 100% of users getting the HD stream. They give me a legitimate 100mbps symmetric (well, almost: http://www.speedtest.net/my-result/3546047591) and I'm paying under $40/mo.

If my niche provider can offer that service, Verizon can offer that at scale. The fact that their service is a) worse and b) more expensive means either a) they're less competent in running the business or b) they're overcharging their customers. Either way, I have zero sympathy for them. They're a network provider and people are paying them to connect to the internet. People aren't paying them to connect to the subset of the internet that's willing to pay Verizon. It's already bad enough that TV viewers are seen as the product rather than the customer. Verizon's stance here is, once again, treating their subscribers as a product to be leveraged rather than a customer who's interests matter. That they can do it is due to their mono/duo-poly that keeps customers from having a choice of providers. But that doesn't make it any less offensive.

Basically, it boils down to this: if the service costs you more to provide, then charge more to your customers. You can even have a streaming plan that charges customers who stream more than customers who don't. But treating your subscribers as a resource that can be pimped out to all the services on the internet is offensive.


For a simple reason. Netflix has to pay for enough bandwidth outbound to supply their customers, regardless of anything else. Their customers are also paying for a certain amount of bandwidth.

ISPs typically possess less total bandwidth then they sell. The problem is that the typical users usage is rising faster than the ISPs are willing to handle. If I pay for 50Mb/s, and the other end of my communication has enough bandwidth to provide it, then I should be able to get to that speed.

As far as I am aware, there has been no accusations that Netflix does not have enough bandwidth to supply their customers, but that the intermediaries have oversold their networks to the point where they cannot deliver on their promises.


That's actually a really good point.

Ideally, you're right -- if netflix (and internet video in general) drives growth in internet traffic, maybe it isn't fair to require service providers to upgrade their service out of hand.

But then I think that rule primarily applies for a free market. I'd argue that internet providers are not currently participating in a free market and it causes real problems. Without a free market... companies need more supervision.

I'll put it this way: 2 competing providers. Average internet usage keeps rising exponentially. People want faster internet. The providers either lower prices or upgrade infrastructure. Netflix is not forcing them to do anything. But then it's probably reasonable for Netflix to say "Your connection is crappy because of your internet bandwidth, it's not us..." But hey, consumers can choose.

Problem here is... what are their choices? Even just one other company participating in the market that can make a similar offering (eg cable speeds) would change the whole conversation I think. Suddenly they'd be working harder to keep your business (or take you from the competitor) and they'd stop talking about more and more ways of screwing their customer and taking advantage of de facto monopoly status.

The incentives are all messed up and it encourages gouging instead of innovation.


The question is: are the networks public, and therefore conduits for competition, or private, and therefore competitive infrastructure. In some capacity, the networks have to peer to hit all of the IP blocks, but that's pretty much one of the better examples of free market success (although that might be changing) and arguably quite private. In another narrative, the networks were laid with private investment (in many cases partially public funded as well) and so the question is moot (if it's all private money).

Are the networks public or private and why can't we decide?


>If Netflix is 35% of global internet traffic at peak capacity (as per the Akamai CEO's comments at a number of events), is it really fair to treat them like every other company?

I don't see why not. I think it might make more sense to think of it as streaming video, the category of goods, rather than Netflix, the company. So 35% of global internet traffic is people streaming videos. And who cares that it's video; from Comcast's or Verizon's perspective, it's just an awful lot of bytes. So if the issue is the amount of traffic, it's just Verizon complaining that we're using too much bandwidth (yes, us; we're the ones watching Netflix). I get that's expensive to deliver all those bytes to my home, but that's what precisely what I'm paying for. That a huge number of those bytes comes from the same originator shouldn't enter into it. It's just a site on the internet.

EDIT: Also, I wanted to add, that all of the traffic comes from the same source actually helps Verizon from a practical standpoint, since it can solve a lot of it with only a small number of interconnects. Compare to a more fragmented market, where Verizon might have to make specific peering provisions for a bunch of different 5%-market-share content providers to fulfill its customer obligations.

> Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.

All sorts of businesses benefit from the ecosystem of other businesses around them. If I run a store in rural Iowa and Acme Co opens a huge factory there, then I stand to make a bunch of money from all these newly employed customers. Should I subsidize Acme? [1] After all, hiring people is expensive. If I open a hardware store near a bunch of planned construction, should I pay those real estate developers? You can expand this to the whole economy being basically just a set of interdependent positive externalities. Maybe Verizon should pay Netflix for making their service more valuable?

Or we can use the simpler model, in which you get to charge your customers. Verizon charges us for delivering the content we request. Netflix charges us for the content itself. Cogent (or whomever) charges Netflix for its bandwidth. Everything works.

> They're not utilities and have a profit motive, right?

They could charge more, and they could even charge per unit bandwidth (or tiered bandwidth) if they wanted. "My business isn't working that well" is not a good excuse to hold your subscribers hostage. You could equally use "Wall Street demands growth!" to defend Comcast's new practice of robbing banks.

Imagine I bought a popular TV set in 1980 to watch broadcast content, but after I got it home, the TV maker remotely activated a previously undisclosed filter that would only show certain channels. Then they went to ABC, NBC, and FOX and said, "So guys, you want your content seen? My margins on these TVs keep going down, so somebody's gotta pay." Not cool, right?

EDIT: Another point, probably the most important. The "problem" here is ridiculous: it's that demand for Verizon's product (i.e. internet access for end-users) is increasing a lot because it is becoming more useful to its customers. That's what every business wants! "We sell internet access and everyone is now using so much more internet! Whatever shall we do?" The idea that Verizon is up against some sort of profitability wall because of how popular its product is becoming and thus needs to charge the companies responsible for that increased popularity...it's simply not credible. Not sure why I didn't think of this point earlier.

Edit: s/Comcast/Verizon, since that's specifically who we're talking about.

[1] This actually does happen in the form of tax incentives for companies opening factories. It's a bit messed up.


To your last point, the problem with a subscription service (any subscription service) is that the more it's used, the less its profit margins. A company that is subscription based actually has an interest in their customers using the service as little as possible. It's completely backward (see: gyms, insurance, etc.)

The only way to incentivize a company to provide the best service possible is to tie the use of the service to its cost. I wouldn't mind paying, say $2/GB for service if it meant that Comcast's incentives were in line.

That would even have an external benefit on creating competition among streaming companies to have the best compression algorithms. Efficiency and speed become directly correlated to the service and its value.


Every company has an incentive to provide lower quality goods for an higher price. There's nothing special about ISPs, their quality just includes bandwidth. The way you align the incentives is by having competition, so that the ISPs with bigger pipes gain customers. Paying more would solve nothing, though I'm sure they would tell you otherwise.


$2/GB

That's a ridiculously high price. You'd pay $2 for every hour of streaming-quality HD video watched, quite a bit more for download-quality HD.


I guess I'm probably an outlier. I don't watch a lot of TV. I was thinking in terms of my current bill, which is about $125/mo for internet and cable, and supplanting all my cable viewing with internet streaming. That's about 60 hours of television (I watch maybe 2 hours a night on the regular). So yeah, I guess it's probably a little high.

My main point was that this charging scheme incentivizes companies to provide better service, not set up monopolies or short change you service.


I pay $25/14 GB pretty consistently here in Singapore. Ironically, in Singapore, it's cheaper for me to pay for traffic on a pre paid SIM, than it is to pay for it with a pre-paid account. Works out great when we're downloading 50-60 GBytes/month. $100/month gets us 56 gigabytes @ about 20-30 mbits/second.


Is wired bandwidth that expensive in Singapore?


It's too slow. Everywhere I go - St. Regis, Fusionopolis, Fraser Place - Lucky to get more than 3 Mbits/second. I have to rely on wireless for performance. Singtel + LTE gives me about 30 Mbits.


Interesting. I find it fascinating that your situation seems to be the exact opposite of the US.


Keep in mind that I'm staying at Hotels, and they cap their wired service to guests. On the plus side, for $100/month, you've got 56 Gigabytes of bandwidth anywhere in the country. It's refreshing being able to not worry about a wired connection.


$2 per gigabyte is absolutely astronomical, the actual cost to major ISP's per gigabyte is literally measured in pennies. I used nearly 2 terabytes last month, so my new bill should be maybe $40 at most. I paid $63, and Comcast just raised me up to $85 3 days ago which I haven't called and bitched about yet. Anyone using <50 gb/month is getting bent over so incredibly hard by the ISPs it's honestly mindblowing.


Country: Belgium - ISP: Telenet:

100 gb. / month (included) for 47,5 € monthly (65 $). Extra bandwidth @ 0,5 € (0,7 $) / GB, so it's an additional 1,5 € extra for your ISP, if you want to watch a netflix HD movie (2,3 GB).

Don't worry, i know i'm getting bend over :)


Not questioning, just asking: Could you please cite data source? I would like to understand this better.


What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network.

Then Netflix comes along, and suddenly 35% of Comcast's customer traffic is all going to a handful of Netflix DSNs? Suddenly Comcast has to redo the whole connection. They have to buy top-shelf hardware just for Netflix.

I don't know if this has happened, but it's an interesting thought experiment to see how "just one company" could place disproportionate burden/cost on the ISP, as compared to the case where traffic travels to & from many different entities.


It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude.

And then there's this: http://arstechnica.com/information-technology/2014/04/bittor...

"[W]hen we ask [ISPs] if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence," Netflix CEO Reed Hastings wrote last month. "That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs."


> It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude.

That's because around the time Comcast was building up their networks customer demand was wildly asymmetrical. People simply download far more than they upload and Comcast wisely dedicated more of it's bandwidth to downloads.


I got the data in the following paragraph from a talk on the subject at an Outerz0ne or maybe a Phreaknic years ago:

At the time all cable companies were were building up their data network, they were building that network over top of their existing video distribution network. That network was designed [ages prior to this point] to have tiny, tiny, upstream channels for use by system control and maintenance devices, and huge downstream channels to shove 100+ analog TV channels. Because it would have cost a metric shitload to replace all of that hardware, the cable companies were forced to give their data subscribers pretty large downstream links and relatively meager upstream links.[0]

Frankly, the nerds of the day (remember, this is the mid-to-late 1990's) would all have been running home servers, and would have been demanding a symmetric connection so that they would have a substantially cheaper alternative to ISDN.

(Note that nerds that remember those days consider asymmetric home Internet connections to have been more harmful than the Eternal September.)

[0] Note that this property is pretty meaningless in an HFC network where the cable company could run fiber directly to the home, but chooses not to. [See also ATT U-Verse.]


But then doesn't that mean that it's also easier to put a cache in front of those overused switches and solve the problem much more easily?

I guess the core issue here is that, from a customer's point of view, the ISP is selling me inet access, regardless of where the data actually goes, so it is the ISP's responsibility to keep their end of the bargain (to provide inet access). As a customer I should not care how is their topology planned, what kind of equipment they use, etc.

So basically the ISP though that they could offer T amount of bandwidth, advertised it as U (with U > T) because most customer won't use even close to U, and now that more and more users are actually using U amount of bandwidth (or closer to it) the ISP's infrastructure is suffering because they simply where not prepared for it, i.e. they oversold their capacity.

OT: By the way, why downvote the parent? I really thought downvoting is for when the post is off-topic or offensive or something... but more and more I see HN modders downvote because they disagree... really? so you make the post go into an ureadable color (grey) just because you disagree? I don't think it's supposed to be that way, is it?


Good point about U > T. It also happens in Gym business where they oversell their gym capacity betting that 80% of their customers will be too lazy to come exercise everyday. Now imagine if I sell a motivation pill that makes you go to gym everyday. It would be absurd if Gym companies send me a cease and desist letter because i am overburdening their facilities because everyone who takes my pill wants to use their gyms everyday.


Actually, now that I think of it, we could probably name a lot of other services that rely on this business model and do a thought experiment like the gym one.

An airline for example; now everyone actually caches their flight because Google's self-driving cars always make it on time, and then they demand that Google stops bringing passengers on time?

Or a buffet restaurant, since now everyone goes to the gym, they can actually eat a lot more, thus restaurants complain about, either the gym or the pill maker?

I don't know, it would be a strange world to live in if this happened in other industries. Maybe prices would rise too much?


Playing the devil's advocate: what if the higher contention ratios lead to 50% reduction in gym membership fee? All other gyms follow suit and consumers and gyms both benefit from the realization that not all members will be in the gym at the same time. Marketing departments start vying for consumers who they know are unlikely to attend everyday and lure them with annual sign up discounts.

Fast forward 3 years later, you come out with your pill and throw a wrench in the works. Now gyms realize they have to go back to the old pricing model because the economics have changed. Consumers will no doubt become upset at the prospec of having to pay more. Some gyms take a wait and see approach to see what other gyms will do. Your pill started it but really, it's not your fault or your problem.

What should the gyms do?


>>What should the gyms do?

Be Honest. They can invent a new business model based on transparency. Imagine if price is not fixed but you are charged per use based on the traffic at the moment. It could be similar to what Uber does, price based on demand.

In terms of ISPs, I think there should be a micro transaction exchange where I can buy X minutes of data transfer. If Verizon is clogged I switch my router to connect with ATT etc. All networks should create a single broker device thats given to each consumer.


Netflix even makes it relatively easy to do this as an ISP too

https://www.netflix.com/openconnect


Netflix isn't "doing" anything to Comcast. Comcast's customers are demanding (and paying for) the asymmetrical traffic. If they have to redo the connection, so be it -- they promised their customers connectivity, and if that means installing new hardware to achieve connectivity so be it.


Edit: rewritten because I misunderstood your point.

Funny, I actually just added an edit to my post above that discusses this. Comcast adds interconnect capacity as needed. So in at least one way, it's actually easier this way, right? Just one peer to make sure they have capacity for.

But either way, it's Comcast's job to run a network that's optimized for its customers needs, not some abstract concept of fairness. And in this hypothetical, it's not on Netflix that streaming lots of content is not compatible with how Comcast happened to build its network. Comcast needs to fix its network so that its customers can get their data.


I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.


> just a small handful of nodes

But Netflix is not "just a small number of nodes". They're basically their own CDN; they have servers all over the Internet, precisely in order to not have the problem you describe.

What's more, Comcast has a deal with Netflix to directly connect Netflix nodes to Comcast's network; in other words, Comcast directly controls how many paths there are for Netflix content to get into its network. If there aren't enough such paths, they can just open up more.


Thankfully netflix expose their stuff via bgp (as well as other nice things) so you can have multiple paths into their network (and out of yours) without any single giant router. allowing you to balance your traffic.

You can even ask them for a freeish box to run inside your network to reduce requests to their CDNs


Oops, my bad. Rewrote my response.


What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network.

Consumer ISPs have never expected symmetric bandwidth. If they had, they would sell us 50/50 connections instead of 50/10 or 50/2.


Note: It is definitely not built under that premise, hence why your consumer bandwidth is asymmetrical...

The Tier 1 always wants to ask for more bits than a lower tier operator. If a lower tier operator asks for more bits and tilts the traffic balance, things get interesting. Cogent is not a tier 1 but slings 35% of the internet around at will.

It does cost a lot of money to route Netflix's bits and the narrative is hugely helpful to Netflix. I'm not saying this is correct, I'm simply trying to point out that Netflix has a ton to gain from carriers upgrading their network connections and routers.

Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.


Netflix is demanding huge network upgrades without any additional revenue source.

I think a lot of us would have more sympathy with the ISPs' situation if they hadn't consistently and for a very long time been promoting their services as ever more valuable by offering ever higher bandwidth, yet keeping quiet about little things like massive over-subscription and high contention ratios that meant if all their users actually tried to use the service that was purportedly being offered then no-one would get anywhere near the advertised speeds they were paying for.

With the rise of things like streaming video services and Internet telephony, some people are actually trying to pull significant volumes of data down the pipes relative to the advertised size of pipe available. Everyone's known this was coming for a long time, and no-one more so than the ISPs. The way I see it, if the current situation hurts ISPs that failed to invest or set a realistic pricing model, they'll just have to take their medicine and offer a more realistic deal to customers next time (or fail because they can't compete effectively).


If Netflix closed its doors today, presumably Amazon Prime and Hulu would pick up the slack. The problem is not Netflix providing content; the problem is users consuming it. Users want better service than the ISPs are inclined to provide.


> Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.

Sounds to me like just another way of rephrasing "ISP Customers are demanding huge network upgrades without any additional revenue source."

Netflix does not send packets to you unless you request them.


silverstorm didn't say anything about symmetrical. If Comcast assumes that the various interconnect points are relatively even in traffic usage, but then the few that go to Netflix start getting congested (leaving the rest relatively light), they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic.

Edit: not sure why I'm getting downvotes since silverstorm clarified exactly what I wrote

I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.


> they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic

You're missing a key element of the situation: Comcast made a deal with Netflix precisely to address this problem. Netflix now has direct connections to Comcast's network, so Netflix traffic does not have to go through a third party. In other words, Comcast now directly controls how they connect to Netflix, how many connection points there are, what the average network distance is between the closest Netflix connection point and their customers, etc. So they have a much cheaper way to even out Netflix traffic: open up more connection points.

(Which, btw, AFAIK they already did as part of the deal; they certainly did not come out of the deal having "just a few spots" where they are connected to Netflix. Netflix basically is their own CDN; they have servers with their content on them all over the Internet. So Comcast has all kinds of ways to shorten the average network distance between each of their customers and the closest Netflix endpoint.)


Asymmetrical peering has been the rule since the 90s. Think of it as fan-in and fan-out, each of which has different dimensions.


>Verizon have to subsidize their costs of business

I don't see that argument going very far. Netflix has an ISP and pays for their own bandwidth. Why should they pay twice for their bandwidth?

If verizon has a problem with the amount of data their customers are requesting from netflix they could always disallow that or institute data caps etc.


Yep, I still don't understand why land line ISPs don't charge by the MB like cell services do.


Yes, because I am paying Verizon to connect me to Netflix. If Netflix's traffic is so high, it's because lots of people want to pay Verizon to connect them to Netflix. If Netflix is so expensive, Verizon would need to raise their prices for customers; possibly offering low-bandwidth customers lower prices. If Verizon wants to give me free internet access, then it would make perfect sense to charge Netflix to deliver to me. Otherwise, Netflix paying large sums to Verizon is only going to ensure that we won't get alternatives to Netflix, because the barrier to entry will be too high for smaller startups. And if bandwidth is the issue, do you think Verizon should pay companies it primarily uploads, but not downloads, to (such as cloud hosting services)?

Personally, I'd really love to see Netflix start using encrypted peer-to-peer to distribute the bandwidth. See if Verizon would be willing to cripple all VPN and HTTPS users to try and shake down Netflix.


> is it really fair to treat them like every other company?

Interesting question. I guess from my perspective I buy broadband from AT&T and they tell me under no uncertain terms that I have a cap at x amount of gigabytes. As a customer I expect to use all of that.

>Guilt tripping Verizon into adding more routers is a major net positive to Netflix's business.

As a customer it shocks me they aren't doing so. Yesterday it was Netflix, today its every kid with an Xbox1 or PS4 buying games digitally as buying physical media is out of style. What's a typical game nowadays? 20gb? That's the equivalant of many netflix movies right there, all at once.

What does tomorrow hold? Why aren't they prepared for this?


You only really pay AT&T for one end of the pipe though. If your end can deliver 100Mbit/s, how responsible are they for the other end of the pipe? Have they met their commitment?

This is the biggest part I think people are missing- they are not thinking about the upstream end of the pipe. Maybe it's the same story even when you do consider the upstream, but I'd at least like people to acknowledge that aspect.


Netflix is saying that they have sufficient bandwidth on the other side, but that Verizon has a bottleneck in their connection to the backbone.


I'm a customer, I'm entitled for 500gb as per our agreement. I'm paying for both ends of the pipe.


I'll reiterate the problem carriers face that I stated earlier.

Customer revenue is decreasing and because the market is saturated they cannot add enough customers to combat the atrophy of revenue. As shareholders expect growth every year, ISPs are forced to look for new revenue streams.

If you don't think they should be doing this as a private company, reclassify them as a utility. Otherwise shareholder demand is what gets satisfied.

Carriers have very specific equipment amortization patterns in their heads and they are financial models that banks accept. If carriers have to change their amortization patterns (more routers, faster) they have to change their financing which damages the value of the stock.

Sure we can do this, but what does "fair" mean here? I'm trying to say that the argument is not as black and white as people make it out to be.


>If carriers have to change their amortization patterns (more routers, faster) they have to change their financing which damages the value of the stock.

Welcome to the world of business. Change is constant, you can't just tell your customers, "Whoops we didn't plan for that."

I also disagree that its as hopeless as you make it out to be (in many nations 100mbps or even 1gbps unlimited internet is trivial to get). They can raise prices on customers, lower caps, throttle netflix users, etc if they're in such financial dire straits. Their competitors will love this as customers will migrate to them. Their competitors may have a more nimble management structure and can provide the data at this level.

What I don't get, assuming they really are in dire straits and not trying to monetize artificial scarcity, is why don't they just push out a notice, "Due to network limitations, we are throttling streaming video users to 4mbps which is enough for medium quality 720p streaming. The investment to support 1080p streaming is impossible for us. Love Verizon."

Actually, I know why they won't do this, because Comcast will pay for the router upgrades and steal all their customers once they find out. So they go around their customers' backs and try dirty dealings with Netflix and others. These guys being duopolies in most markets (substitute AT&T with Verizon in some states) just decided to not compete and simply change the rules at the FCC, which is pretty much a textbook description of collusion.


You're looking at this all wrong. It's not that Netflix is taking up 35% of all bandwidth. It's that CONSUMERS are choosing to spend 35% of their consumption on Netflix.

The ISPs are punishing consumers for their choices - it's just ridiculous.


On the other hand, can you swear that Netflix is doing all that it can to minimize bandwidth?

There's a conflict of interest: Netflix has a vested interested in provided lightly-compressed, very-high-resolution, bandwidth-hungry movies to its subscribers, especially if it obtains 100% free peering that it demands, as bandwidth would be free for them. On the other hand, ISPs are forced to upgrade network any time Netflix unilaterally decides to increase the bitrate of the movies, without this bringing any benefit to their bottom line.

For instance, if Netflix halved the bitrate of all their movies, subscribers will complain, but surely they would be able to deliver them without any buffering to everybody, and at the same time to keep free peering agreement and whatnot; they would go from 35% of total US bandwidth to 17%, and ISPs would be satisfying their customers' choice.

So where do you draw the line? Surely Netflix can increase the bitrate at will, at any time; should they be "allowed" to do it any time they want, while keeping a free peering agreement? Should the ISPs be forced to upgrade their network every time?


> can you swear that Netflix is doing all that it can to minimize bandwidth?

This is an excellent point. Why doesn't NetFlix at very least offer the ability for their users to queue content for download / local caching? If customers had a slow Internet connection at least they could plan ahead to avoid suffering streaming problems.

Ironically isn't streaming-only perhaps a violation of net neutrality? NetFlix artificially limits the speeds my ISP can deliver by only offering streaming. I have a 100Mbit/sec connection yet NetFlix chooses to limit me to streaming speeds.


>Why doesn't NetFlix at very least offer the ability for their users to queue content for download / local caching?

Because this would almost certainly result in the content industry screaming bloody murder.


> Ironically isn't streaming-only perhaps a violation of net neutrality?

No, not at all.


Netflix does allow for caching at the ISP level.


It isn't Netflix's job to minimize their bandwidth usage for the sake of the ISPs. If I pay for 500GB/month at 20Mb down, that's what I should get. If Netflix does decide to increase the bitrate of their videos, than I'm the one that will have to deal with potential buffering issues and/or going over my cap.

As others have mentioned, the ISPs woefully miscalculated the growth of the internet and consumer bandwidth usage and as is the case with capitalism, those mistakes should cost them.


> Should the ISPs be forced to upgrade their network every time?

Forced to upgrade? Maybe not.

OTOH, should people be made aware that the ISPs failure to do so is a contributing factor to the unsatisfactory results they are getting so that they can make informed market (and political, insofar as government policy impacts the competitive landscape) decisions? Absolutely.


> Should the ISPs be forced to upgrade their network every time?

Come on, they have been doing this since the internet started. And in the beginning of internet some services were hugely popular as well. Keep in mind that upgrading the network costs money, but once the infrastructure is in place, moving bits is really cheap.


The difference is that now there is a single commercial player, with the ability of filling up whatever bandwidth the ISPs allocate for their customer base, with a single switch, and a commercial interest in increasing bandwidth (to improve quality to its subscribers).

If Verizon tripled its total bandwidth today, how long before Netflix begins raising the bitrate because "the Internet has evolved and we can now deliver bluray-quality streaming at the same $9/mo?". And then they fill up pipes again.

The whole issue here is cause by the impedance between the cost structure (pay by the gig transferred) and the price structure (pay by the peak speed, and I approximate the average gig transferred).

In a perfect utopic world where everybody was willing to pay by the gig transferred as that is the real cost factor, we wouldn't be here having this discussion.

> moving bits is really cheap.

Moving bits is cheap WITHIN the ISP network, but once you go outside they have a cost (depending on the peering agreements).


Fair point. The issue obviously isn't 100% black and white. But what the ISPs are currently doing is not the way to go about it, IMO.


Does the devil need any help? Does Verizon's PR department? You might ask yourself why you feel obliged to take the side of an enormously wealthy oligopoly.

But to answer your question: I am already paying my ISP to deliver all packets I request to me, up to certain limits that my ISP agreed to when they started my service. If my ISP is failing to live up to that bargain, I'd like to know.

Also, the "poor Verizon, helpless in the hands of Wall Street" thing verges on obscene. That they are in a position to shake down people for more money does not justify them doing so. If they don't want to be in the ISP business, they can get out of it. They are also welcome to find new ways to create value. But running a protection racket is not in any way morally defensible.


> If Netflix is 35% of global internet traffic at peak capacity (as per the Akamai CEO's comments at a number of events), is it really fair to treat them like every other company?

If the ISP's customers are making requests to Netflix' servers, then yes.


Netflix is not using the capacity, they are selling access to the content. The customers are using 35% of the traffic. The customers who pay for access to content they want. Most people want Netflix because it is a sea/wonderland of content and is replacing tv. Netflix is not consuming the content, they are providing it, the customers, who pay for the access, are consuming the content and bandwidth.

Content services should not be harmed for being successful. If the bandwidth customers are desiring certain content providers it is the bandwidth providers responsibility to make sure it is available and fast, per their marketing and products.


"Most people want Netflix because it is a sea/wonderland of content and is replacing tv."

And that's in direct competition with these big last mile providers. Cablecos by definition, and the two big telcos have their own FiOS and U-verse "TV"/"cable" video offerings (even if AT&T's coverage is meager, and both stopped their build outs except for contractual requirements when the Great Recession started).

Which provides additional strong motivation to provide less than stellar service when handling Netflix packets.


If it wasnt't Netflix, it would be another streaming service or a hundred. The total capacity required would still be the same.


This is evidently not true. Streaming services are a dime a dozen but their distribution of bit volume is not even. Netflix is far and away the biggest user of bits on the Internet and other streaming services are an order of magnitude smaller.

It's possible that if Netflix didn't exist someone would fill that vacancy but it's also possible that the aggregate total volume of bits transferred during streaming would be lower if Netflix did not exist.


Customers should pay for the bandwidth they use.

ISPs have been miss-selling Internet connectivity for many years - "UNLIMITED!!" "FAST!!" "CHEAP!!" - and now they ( and their customers) are reaping that harvest.


I've seen this argument about Netflix's oversized traffic slice, but it always seems to me like it should mean the opposite: if Netflix is 35% of the reason people sign up for broadband, isn't Netflix making broadband more valuable? People are willing to pay more for broadband because Netflix makes it worth more to them. So shouldn't the broadband providers be paying Netflix rather than the other way around? You never hear the cable TV providers complaining that a particular channel has gotten too popular. What's different here?


>> If Netflix is 35% of global internet traffic at peak capacity (as per the Akamai CEO's comments at a number of events), is it really fair to treat them like every other company? That is to say, if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?

Netflix isn't the driver, the Verizon customers who want to use netflix are the driver. They're paying Verizon to get them access to cool stuff like Netflix.

It sounds to me like those customers are not getting the service they expect.


It's not just Netflix, it's also YouTube (http://www.google.com/get/videoqualityreport/), and Level3 (http://blog.level3.com/global-connectivity/observations-inte...) and you can be sure those aren't the exception.

Instead of fulfilling their obligations to provide advertised bandwidth to their paying customers (who only pay them in order to access the content provided by someone else), the last-mile providers are using their customers as hostages while they negotiate ransom from the content providers. While they also report record profits (sending their customer's payments back to stockholders instead of into the network), because the economics of their business are compelling on both the consumer side (low expectations, minimal support costs, essentially no competition) and on the technology side (network technologies increase capacity-per-$ as fast or faster than traffic grows).

Oh, also, 99% of the last-mile providers operate a sanctioned monopoly that has let them build and grow their networks with essentially guaranteed income from rate-payers and essentially no competition. Hence adding Internet is relatively low-cost and highly-profitable, but apparently not profitable enough.


You're not as far off as you think. It seems very few commenters to this thread understand how the internet actually operates from a business perspective.

When you install a server in a colo, you don't get free transit. You find a transit provider (like Level3, Cogent, or NTT) and you pay them for bandwidth. You can understand this very intuitively as most hosting services (like AWS) will charge you for bandwidth.

As a service provider (like Weebly), you are paying your own transit provider for a certain amount of bandwidth, and they are covering the costs of delivering that bandwidth to (usually) anybody globally and the costs of maintaining their network.

If a network gets large enough, they may often negotiate peering agreements with other networks. So Level3 and Cogent may have a peering agreement, or Level3 and Comcast may have a peering agreement. These are generally set up when two networks deliver roughly the same amount of traffic to each other, or it's considered beneficial to both parties to get the networks closer, and it simplifies the working relationship.

These peering agreements can often become contentious when they become imbalanced. Very early on, I remember a fight Cogent and France Telecom had over peering -- FT felt like they were getting the raw end of the deal, and de-peered with Cogent. Cogent refused to pay another transit provider for access to FT's network and so traffic from the Cogent network to FT would drop. They resolved about a week later, with lots of unhappy customers caught in the middle.

As you can see, it's simply not the case that (a) service providers never pay for bandwidth because the end-user already pays for bandwidth and (b) peering / free transit is guaranteed.

Maybe I'm not understanding the nuances here, but it seems to me like Netflix is taking an ideological stance to hide behind the fact that they are trying to negotiate a better business agreement for themselves.


It's probably both. Having an incentive and an ideological position that align doesn't automatically invalidate the ideological position. I don't think that the net neutrality position is nonsensical, personally so I'm glad to have companies with an interest in it champion it.

I also think that there these peerage agreement create an untruthful dynamic with their customers, the point about transparency is valid. People think they are buying "internet" access at a certain speed, not different speeds for different sites. If Verizon thinks it's OK for the speed of their customers' connection to be dependant upon "interconnect agreements" or whatever, then letting their customers know about this isn't defamation. If they do have a problem with their customers knowing, then maybe they should be making sure their customers get what they expect (or expect what they get).

In any case, I don't see how this is foul play on Netflix' part. As long as the message they display is true, why should't they be able to display it? Even if the message openly advised their customers to switch away from Verizon, I don't see a foul.


If there is imbalance between the traffic Comcast is sending to other networks versus what they receive that's because of the policy to forbid residential customers from hosting their own servers. Comcast created this situation because they wanted to force the internet to behave more like television.


Exactly. The "equal traffic" concept totally fails for consumer last mile providers. In fact, it's technically more expensive for them to provide symmetrical service due to head end noise, but it also fits with their "you're supposed to consume content, especially ours, not supply it" attitude.

The latter is something that more people ought to be thinking about, consumers using Netflix are not just consuming more peak bandwidth, they're competing with the video offerings of all of the big last mile providers.

Although. AT&T doesn't actually offer U-verse video service to very many of their subscribers; don't know about Verizon's FiOS vs. plain DSL offerings (except that they're shedding small population states and markets as fast as they can), but the others are cable companies so they obviously do.


If Netflix is paying Level3 enough to send the data to users, but Verizon is not paying Level3 enough to get the data on behalf of their users then it sounds like Verizon's fault.

Peering agreements should be about saving costs on both sides, but if Netflix is willing to pay Level3 for sufficient bandwidth then Verizon should have the ability to handle that for their users.


Verizon doesn't pay Level3 for the traffic, the person sending the traffic always pays, unless there is a peering agreement.

That's how it's always worked. These fights and skirmishes over peering agreements happen often. That's the reality of how things have always been up until today, it's not a change in the status quo.


This is also kind of relevant given that Cogent bears Netflix's bits and Cisco benefits from that bearing:

http://news.cnet.com/2100-1038_3-1019435.html


> That is to say, if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?

Because that's Verizon's business: to provide internet. That's not up to Netflix, who is an internet-based content provider. If Verizon wants to charge the fuck out of their customers to beef up their network so they are the best Netflix streamer out there, then they're free to do that.

People want Netflix because Netflix is doing its job. People can't get Netflix because Verizon's not doing its job. Simple as that.

To put it another way, if Netflix was the bottleneck (and thus, the problem), Verizon would only have to show that their bandwidth from Netflix is bottlenecked by Netflix' lack of serving capacity. Of course they can't show that, because that's not the case, and thus not the problem.


If it not Verizon's responsibility to provide a internet connection, to the internet, then what exactly are their internet service customers paying for.

Also, I find it interesting that netflix charges ~10-15 bucks a month, while the average internet only service plan in the 40-50$ range per month. This is a guess, but I would also say there are more Comcast/Verizon/Time Warner customers than netflix customers. The amount of money these large service providers rake in must be an order of magnitude higher than Netflix.

To me, this money that the big ISP's are trying to get from Netflix looks like some sort of fee applied for forcing them to actually provide the service they promise to their customers.


Some of these issues feel like a grey area to me, such as peering agreements. However, this move from Verizon felt very black and white. Thanks for trying to be Devil's advocate!

However, I feel that this issue is one of misrepresentation. Netflix wants Verizon to be a common carrier, and have its revenue increase linearly with the amount of data it transfers. In that case, Verizon would happily increase their revenue by providing actually faster Internet, because customers would pay more per day if they downloaded more per day, since they'd then pay a fixed amount per GigaBit.

On the other hand, I don't understand what Verizon wants to be. I feel like the type of company it wants to be is only sustainable in the very short run, highly dependent on other companies such as Netflix, and incapable of good PR because what clients want, and what they subscribe for, is impossible to deliver (the amount of data clients want to transfer is always above whatever they can promise to offer). That market positioning is tolerable for instant Wall Street growth, but it is detrimental to customers, to website owners, to peering partners, and to the long-term future of the company, were a reasonable competitor to enter the market.

I also feel there could be a middle ground, such as Verizon asking customers from every city to tip in for a massive city-wide network upgrade. That would be good for PR and customers would happily donate. It seems fitting, because from what I understand, the real money eater is a one-off upgrade; it doesn't increase maintenance costs much, if at all.


> the business reality is that the ARPU per subscriber is decreasing every year at a rate lower than additional subscriber acquisition can sate...Carriers are not able to sell consumers additional services (no matter how hard they try)

In other words, the ISPs are doing a bad job of running their businesses, but rather than do a better job, they want others to pay them more without offering any additional value.


> Yes, ISP customers are paying for access but the business reality is that the ARPU per subscriber is decreasing every year at a rate lower than additional subscriber acquisition can sate.

ARPU is going up while Verizon's bandwidth costs are going down (peer with Netflix/Google/YouTube/Facebook and a huge chunk of your traffic is taken care of).

http://newscenter.verizon.com/corporate/news-articles/2013/0...

"FiOS revenues grew 14.7 percent, to $2.7 billion in second-quarter 2013, compared with $2.4 billion in second-quarter 2012."

vs.

"Verizon added 161,000 net new FiOS Internet connections and 140,000 net new FiOS Video connections in second-quarter 2013. Verizon had a total of 5.8 million FiOS Internet and 5.0 million FiOS Video connections at the end of the quarter, representing year-over-year increases of 12.2 percent and 12.6 percent, respectively."

(Revenue growing faster than new customer growth and a very high ARPU.)


Just imagine that you had more than one choice of high speed Internet provider (unlike 95% of Americans who are stuck with the one cable company that happened to successfully bribe their local municipality) and think about what would happen to this issue. Competition would force the ISPs to knock this shit off. It's a product of their entrenched geographic monopolies. Period.


I am also frustrated with people who blur a) net neutrality with b) monopolies/undercapacity issues, and so I agree that people should not be so quick to demand ISP-paid extra capacity as solutions to this, especially on grounds of net neutrality.

However, I don't think it follows that Netflix must be treated any differently. The customers are the ones requesting the data. To the extent that anyone (once at capacity) should bear a cost, it should be the requester -- either through a higher fee or a smaller fraction of the pipe.

In practice, that will mean "the more Netflix [or anything] you use, the slower your service". But you shouldn't "short-circuit" the process and just start punishing Netflix's packets directly just for being Netflix. Rather, it should be "this customer endures this cost [broadly defined] for using this much, regardless of what it was for".


But net neutrality comes into play because these big last mile providers are also offering their customers video services. That's what cable companies started out doing, after all, and Verizon and to a lessor (much lessor, I gather) extent AT&T have implemented the same business model with their FiOS and U-verse offerings.

People getting their video from Netflix threatens this business, and it's pretty obvious to me that's one of the reasons we're seeing all this now, especially with Netflix getting big enough in streaming to a) pose a greater threat b) potentially have a lot of voters on their side (particularly relevant because the last mile providers are all government enabled monopolies, cable companies at the municipality level, telcos at the state and Federal level).


I'm more concerned about the fact that everyone seems to give priority to speedtest traffic, then abruptly chokes streaming HD.

I'd really like Netflix to offer their own speedtest site that looks to the network almost exactly like streaming video, so the ISPs couldn't game the measurements.



>why does Verizon have to subsidize their costs of business?

Verizon isn't some kind of customer owned co-op. It is a profitable for-profit business that has (over)sold a service. They need to deliver that service, not complain when people use it.


>That is to say, if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?

Because Netflix has a higher brand affinity and customers of Verizon will be outraged due to what they perceive is shitty customer service buy a quasi monopoly.

The consequences is damage to Verizon's business because customers would switch if they could, but they can't.

Verizon doesn't HAVE to subsidize Netflix's business but they should if they don't want a legion of pissed off Netflix customers who feel trapped by their ISP.


"if they don't want a legion of pissed off Netflix customers who feel trapped by their ISP"

This is supremely dangerous to Verizon.

Taking the US political area I know best, gun owning households are a majority in the US, but of course those who vote, vote first and foremost on this issue, let alone be politically active are the usual successively smaller fractions of that huge number.

And starting in the mid-80s we've been able to move mountains (details on request). The last thing the big incumbent landline last mile providers want is a large percentage of the population getting pissed off and doing the above, voting when they might not otherwise bother, especially in state and local elections, voting on this issue, and becoming politically active on it.

I think the incumbents' behavior is so outrageous that this is eventually inevitable, Netflix's PR attack, in part enabled by their being so big, has the potential to substantially move that day of reckoning forward, and is therefore probably the biggest thing to happen in the net neutrality fight since AT&T's Edward Whitacre opened his foolish mouth and really got the issue started 8 years ago.


> if Netflix is really the sole driver of Network upgrades, why does Verizon have to subsidize their costs of business?

Verizon is paying to provide its own customers with what they want (speed that enables access to Netflix and other entertainment), not to subsidize Netflix.

Netflix pays its for its own high-speed internet access already. Verizon offers high-speed access and then doesn't pay for it. Therein lies the conflict.


Quick question. When you signed up to your say 20meg broadband contract, did the contract exclude certain services - YouTube, amazon Prime, Netflix?


Your argument is invalid, because Verizon holds monopoly in many areas for high-speed broadband.

I would entertain your argument only if there was consumer choice. In such case we could perhaps experiment with different business models where content provider subsidizes last mile. Without it, consumers are essentially hostage to a monopoly, which is unacceptable.


My ISP has provided 100 Mbps for years, and has recently brought 1 Gbps fiber. They've never complained about video services or torrents eating away their bandwidth. Guess why? Because when you're able to provide 100-1000 Mbps to each and every customer, the ~5 Mbps required for Netflix streaming is nothing.


Why add a viewpoint that is not valid? That's like saying climate change deniers have a “valid” viewpoint when you can find 3 crackpots in a sea of 100,000 scientists.

You buy an Internet connection from an Internet service provider to consume the Internet. Until the word Internet is dropped from “ISP”, that's the service they sell.

They have no more rationale to charge Netflix to “access” Internet service purchasers than they have to charge your grandmother to deliver you her email.

And about that profit motive, cable already raises prices at 4x consumer price inflation:

“Since the FCC began compiling these reports 19 years ago, the cost of basic, expanded basic and the next-most-popular tier of service have risen at average annual clips of 4.3 percent, 6.1 percent and 5.1 percent, respectively. Inflation has grown at an average annual pace of just 2.4 percent over that time period.”

http://www.siliconbeat.com/2014/05/19/pay-tv-prices-continue...


If anything ISP's should be thankful that NetFlix exist.

They have subscribers because those subscribers want to use stuff like NetFlix.

Take away all the good 'content' and why bother having an ISP? Mobile phone can check emails too.


Thank you for the position, this comes up regularly, and I love seeing it discussed and dismissed.

It's not that Netflix is 35% of global internet traffic. 100% of my ISPs "global bandwidth" should be content of the end users choosing (usually requests). The user is paying for the right to request this content, and have it delivered. Wether its Netflix, gaming, news, poltiics or Zombo.com doesn't enter into it. At 35% though, it might make sense for Netflix to enter the ISP business.

This framing is a tool that ISPs and their supporters use to excuse charging twice for the same service, and more importantly, controlling users access to content through discrimination.

Thanks again for the writing prompt though.


What about fedex or ups? The customer is paying for x bandwidth and not getting it.


Am I the only one who thinks that this could hugely backfire from a PR perspective?

The message is "The Verizon Network is Crowded Right Now." not "Verizon is Deliberately Slowing This Right Now." (or some less loaded version of that)

It's bad PR for the ISPs, but combined with the fact that most people can hardly spell ISP, and that Comcast and their FCC homies keep describing the new toll booth as a fast lane, if I were a nontechnical consumer my first response would be, "What the hell are they waiting for? Put in the fast lane! I wanna watch my damn movies!"


Which is why it's so important they used the ISP's name there. Instead of saying "congestion", they say "(ISP)'s congestion".

Considering that customer satisfaction with the average telecom company is somewhere between that of the DMV and a crooked tax collector, it plays into the confirmation bias people have that they don't love their ISP, they tolerate their stupidity because its the only way to get online.


> "What the hell are they waiting for? Put in the fast lane! I wanna watch my damn movies!"

Telling a customer they're not getting what they paid for from their service provider is about as plain English as you can get. I daresay we need this sort of rhetoric, and I doubt their response will be levied toward Netflix unless they start getting letters from their ISPs saying "We're throttling you because you're watching Netflix and they won't pay up."

This situation could get ugly, though.


I hope it does get ugly, because that's the only way this 'fast lane' horseshit is going to stop.


Or perhaps it'll just make ISPs start charging by bandwidth instead of offering "unlimited" and hoping most people don't use it.


That doesn't address the net neutrality aspect of this. Above at https://news.ycombinator.com/item?id=7858919 we see quoted from a Verizon press release that "Verizon had a total of 5.8 million FiOS Internet and 5.0 million FiOS Video connections at the end of the [2nd] quarter [of 2013]"

Verizon doesn't want more "cord cutters" of a sort to drop their FiOS video accounts in favor of a la carte offerings from Netflix et. al. AT&T's U-verse isn't as big a thing for them, but they use vicious caps to protect it, and of course all the cable companies are by definition video service providers.


The question is why didn't they try this with Comcast before shelling out millions of dollars.


> "This is about consumers not getting what they paid for from their broadband provider"

If I pay for a 100Mbit/sec connection from my ISP why doesn't NetFlix send me 100Mbit/sec of data? They choose to limit my speeds (via streaming) and do not offer any options to download or pre-cache video content I plan to watch. Is this a violation of network neutrality? If I ask NetFlix for 4GB of data why are they allowed to limit the speeds?


AFAIK no where in your Netflix terms of use is Netflix obligated to give you content at a specific speed or amount. They are obligated to stream you movies, not stream you movies at 100mb/sec. This is in stark contrast to ISPs who advertise 100Mbit/sec down.


This is a genius move. It's exactly what needs to happen to get consumers to realize that it's their ISP fucking them over, not Netflix.

The average joe isn't outraged enough about net neutrality. If only they'd start doing this to other known bad actors coughcomcastcough, that might just be what the doctor ordered.

I wonder why they ponied up the money to the protection rackets first, and only then started pointing fingers. Maybe the agreement gave them access to some better data? If not, this should have been done months ago!

By the way, don't bother with the comments on the article page unless you want to lose all faith in humanity :(


From a strategic perspective, I think paying off one of the ISPs in this case strengthens their case by showing that the problem is a shakedown for money. If they didn't pay up and the issue was never fixed then the ISPs could attempt to play it off as some larger-than-money issue. They've proven the problem is greed.


Even better, it attacks Verizon using their own words against them.

Verizon: Netflix performance sucks because their traffic is congesting our network!

Netflix: Our performance sucks because their network is congested!

Verizon: OMG WTF you can't say that!


I never thought of this, but it's quite genius when laid out this way.


It could be so that the ISP can't claim "if you had peered with us, we wouldn't be so congested" or some other BS.


Sure, but Netflix did "peer" with Verizon (read: put their servers on the Verizon network). At this point, anything happening is either Netflix's fault (for not buying enough bandwidth) or Verizon's fault (for having a congested internal network), and we don't have enough information to know which.

Could also be that the agreement had an anti-defamation clause, and Verizon was counting on that to avoid upgrading their internal network.


In Germany, this would actually be a great thing! If the German Telekom (biggest provider here, formerly owned by the state) played that card, other providers could start to demand peering with them. The strange thing with German Telekom is not that peering with them would be too expensive, it is that they don't take part in the big peering points at all. As a "smaller" provider, you can connect with Telekom on by becoming a Telekom customer, but not on eye level as a peering partner.


We have a similar problem in australia. The ACCC (consumer competition watchdogs) forced the 4 largest at the time to peer together (which was a good move) but they refuse to peer with other australian ISPs.

They also control a fair chunk of the overseas connections.


If I had to guess, I would say that Comcast had a huge proportion of 'users paying Netflix money and which live in an area where there is no broadband competition'.

If the problem with Comcast is that their network is congested and users can't switch to another provider then it doesn't matter whose fault it is, those people are going to leave regardless.

Note that Netflix is only showing these notices in areas where Verizon has competition to which their customers can switch, which lends a lot of credence to the idea that 'ISP switchability' is a key factor guiding Netflix's behaviour.


Re: comments, I didn't believe you, now I wish I had.

> NetFlix will lose huge. Obama is protecting the Taliban now. That's the only reason Silicon Valley soared. Democrats (from Jan 2007) and Obama 'til his Taliban kiss.


I wonder why they ponied up the money to the protection rackets first, and only then started pointing fingers.

One possible reason is that it makes it easy to quantify their economic injury.


It's frustrating that Verizon and friends can make grandiose statements in their advertising about "unlimited bandwidth", "faster wifi" ,"stream 5 things at a time", etc, etc. Or lie to customers, My mother just called Comcast to downgrade her service and the CS Rep said "WiFi won't work with our Economy Plus internet plan"

Yet when another entity does something as simple as showing an error message that casts them in a negative light, they are willing and able to threaten legal action.

They stretch the truth as far as they can, yet give not an inch when confronted with truths they don't like.

Remove government so we can add customers! But we need government so we can slap Nextflix when they say something we don't like!

I don't have a good answer how to fix it, it's just very plain to see, and frustrating.


What do they mean WiFi won't work? How is that anything other than a blatant lie?

Reminds me of the late 90's when my ISP told me I couldn't get cable Internet because I didn't have a 486. I lied and told them I did to have the modem sent to me to hook up myself. It's a computer network, once it's in my house I can expose it as I like.


How many of us had to bluff our way through the first gatekeeper on tech support like this?

   Tech: "Could you please unplug your computer for 20 seconds and restart it?"
   Me: "Okay. <does nothing for 60 seconds>.  Okay, it's back"
Or this gem?

   Tech: "What browser are you using?"
   Me: "<Looks at Ubuntu desktop>  Internet Explorer 6".


I had a fun time with a Comcast installation tech a year or two ago:

  Tech: "Looks like we're about done.  Now I just need to
         use your computer to activate the connection."

  Me:  <looks at my Debian laptop> "I don't have a computer,
       I only use my internet for tablets and my roku"

  Tech: <looks at my laptop> "I need to use your computer to
        activate your connection..."

  Me: "I don't have a computer, and of course you do not."

  Tech: "Okay, just a second, I have to make a call..."

  [one minute later]

  Tech: "Okay, your connection is activated, you should be
         good to go."

  Me: "Thanks!"
If there was really a hard and fast requirement that they use a computer to activate the connection, they would bring a computer with them. They just want to install their shitware on your computer.


Wait, Comcast tries to install software on your own computer without asking?


I mean, technically they are asking, except they phrase it in a way that tricks people into believing that it is required.

I feel comfortable lying to their techs about a computer that is in plain sight because they feel comfortable lying to my face that they must use my computer to complete the installation.


Wow, that's some real Jedi action going on there. Nicely done.


I just want to point out that Teksavvy have in my experience proven to be FANTASTIC at helping with linux internet-related problems.

I now refuse to subscribe to any ISP who does not offer Linux support, and will in fact check to make sure that Teksavvy (or an equivalent as described above) is available in an area before moving to it.

Full Disclosure, I am a Teksavvy customer but not a Teksavvy employee.


As a Sonic customer, I agree with you: small ISPs can be awesome.

However, over 90% of American homes have at most 2 ISP choices. At most! So we are in a small but happy minority.

I lay most of the blame here on the FCC and the FTC, who have totally failed to regulate in a way that maintains a healthy market.

But I also have some blame left over for all the people who signed up with major ISPs while the small ones went out of business. It has always cost me more money to go with a local ISP rather than whatever the megacorp's latest intro rate would be. Now everybody is shocked that Comcast, Verizon, et al are giving shitty service and being total dicks about it. But it turns out I can't go down to the corner and shout, "WELL DUH! WHAT DID YOU EXPECT?!? TO THEM YOU ARE THE VEAL!" because nobody likes hearing I told you so. That or they don't like crazy people shouting on the corner.


Must be nice to have a choice of ISPs...


stop it, you're giving me flashbacks


What do they mean WiFi won't work? How is that anything other than a blatant lie?

They probably mean the WiFi built into the cable modem.


I'm honestly not sure what the Representative meant. This is just what my mother told me (non tech person, she could have misunderstood). I left out the part where the rep told my mother she was already on the cheapest plan. Fortunately they had just mailed their yearly "price sheet" that listed Economy Plus Internet, with prices. Magically when my mother mentioned this by name, it became available.

My general sense is that these types of conversations happen daily at Comcast and such...I've heard too many similar stories over the years.


Yes, whenever I miss the payment on my TalkTalk bill, the whole local network goes down, not just Internet access. I've been meaning to get a cheapo router to remedy.

People just don't know that you can have wifi for free, especially when you've got adverts from O2 and such that 2gb 3G and " unlimited wifi".

I dunno if it's the same in the US but apparently it's some typically restricted resource that many providers in the UK are lifting the limits off.


>I dunno if it's the same in the US but apparently it's some typically restricted resource that many providers in the UK are lifting the limits off.

No, this isn't too typical. Most people have their own router offering WiFi in front of the cable modem. It wasn't until recently that cable companies started bundling wi-fi and NAT into the modem.


Really? I had Time Warner Cable providing a Netgear modem+router+802.11g AP combo box back in 2004, maybe 2003. (It was shit.) I think the only thing that's changed is that nowadays even non-technical users have more than one device that needs a connection, so attaching the modem directly to the PC by USB doesn't cut it.


I know it's not typically restricted, I've enjoyed "unlimited wifi" for over a decade! The thing that bothers me is some providers in the UK say they give away unlimited wifi as part of the package.


> " unlimited wifi".

I just felt a disturbance in the force. Damn, that sentence triggers a whole range of evil ideas I hadn't yet thought of.

Hmm, hold on while I check with my ISP whether I can downgrade my WiFi data limit for cheaper service... or would that just be giving them ideas? :p


How would that be anything other than a blatant lie? Are they going to mail the customer a different model of modem? Do any current cableco-provided modems not have built-in wifi?


> Do any current cableco-provided modems not have built-in wifi?

No, but all of their features can be enabled/disabled by the ISP remotely.


I know for my ISP the user names and passwords used by support are all freely available online. I needed a tech to come in because they couldn't troubleshoot my issue (magically fixed the next day) and he was flummoxed when he couldn't log in because his passwords didn't work.


I have a Ubee cable modem from Comcast that doesn't have wifi, and I think had an Arris before that that also didn't have wifi.


Can't it be remotely disabled because it's not a part of the cheaper plan?

Similarly to how you only get the ability to tether your data with more expensive plans. It's a completely artificial client-side restriction that can be circumvented by rooting, etc. Not unlike DRM.


Yeah: a few months ago Charter upgraded our cable modem, and neither the new one nor the old one has wifi (they want to sell a router for $12/month).


Or their comcast/xfinity hotspots, which is true.[1] Their lowest "Performance Starter" package does not have access to those hotspots, while the next higher "Performance" package does.

[1] http://www.comcast.com/internet-service.html


Similarly in college, when setting up TWC account after some friends and I moved into a house. I clearly remember them saying that our wifi would only work with 1 device if we didnt get the "Fastlane" or whatever package they where trying to up-sell at the time.

I just played it dumb and said it was just me. We had about 10+ wireless devices connected.


I suppose that if you had it hooked to a single router, that could be your one device. ;)


Nothing new about tech support not knowing what they're talking about. After all, it's written into the script that they're told to read for that question.

A few years ago I was setting up a new internet provider for my mother, so I called to get the info to get email: Support: "You're using Outlook, right?" Me: "No, Thunderbird." Support: "We only support Outlook, so I can't help you." Me: "It's email, it won't matter if I use Thunderbird." Support: "Our email does not work with anything other than Outlook." Me: "Okay, I'll get rid of Thunderbird and use Outlook." I entered the information into Thunderbird as he gave it to me and then hung up the phone.


At the earlier 2000's my ISP sent a "drivers" with malware that would disable brigding and NATing at the OS.

But, of course, it only run on Windows, and I was lying telling that I had Windows, as every ISP would refuse to serve somebody that uses Linux...


Or when your ISP 'requires' Windows


And modem/router/switch boxes that come with Windows 7 logos.


Maybe he was talking about their public WiFi hotspots that are here http://hotspots.wifi.comcast.com/?


Story about Shaw (in canada, BC):

I had one of their built in modem/router combos that provides internet and wifi, me and a room mate had a minecraft server set up on a box wired into the router, and we played locally, but for some reason every 20 minutes, our connections to the server, the internet, and eachother would just die.

leaving an active traceroute on a spare computer suggested that the router just stopped routing for a period of 20-30 seconds, every 20-30 minutes.

Angry and annoyed, I called in a tech from Shaw to look at and replace the hardware. Standard operating procedure here: "We will show up some time between 8am-1pm". Of course he showed up at 3pm, with no call that he was going to be late. I had to call them.

Unbelievably, he logged into the router (with no permissions given, so those routers are backdoored... goody) and actually identified that the router WAS resetting every 20 minutes or so. Yes, the problem was identified! The problem was that my internet plan was not sufficient enough to support online games, and that if I didn't get another better one, the problem couldn't be fixed. (For the record, I had a 30 mbps down, 3 mbps up, cable connection.)

Jaw agape, I tried to explain to him that A. The game server he was referring to was literally right next to his feet, and B. that even if it wasn't, game data is minuscule compared to video data, something I am a prolific consumer of with no obvious problem there. All that combined with the subtle hint that the router literally just told him it was resetting, the conclusion clearly was that the router is broken, and my internet is fine.

His response to that was:

   ...
that's a direct quote. I'm not paraphrasing.

He didn't want to say anything because I had the audacity to say something that went contrary to the marketing line, and the only option he had was to either say I was wrong, or accept it.

He wouldn't replace the router regardless.

So I told him to take his modem/router, put in my old modem, get it registered, and buzz off (though maybe not so colourfully).

When I talked to a Shaw customer representative about having a Tech show up late, lie to me, and not fix the problem without an upgrade, I got this response:

   uh, ok?
That's a direct quote. I'm not paraphrasing.

Fun part though: that transgression was actually considered a transgression by Shaw's regulatory agency, So I reported it to them after I got piddly squat from customer service.


oof :(


I feel like Verizon's treading VERY dangerous waters here. If they sue Netflix for libel over this, then they're going to have to go through discovery. Since the claims center around network congestion, that means it'd be fair game for Netflix to go after every scrap of paper they have about the state of their internal network, oversubscription strategies, data about advertised vs. actual customer performance, any history of traffic shaping, and they'd be able to depose employees about all of these things.

As much as I'd love to see all of that information come to light in a court battle, somehow, I don't think Verizon would...


Wow, what a stupid move on Verizon's part. I have to believe the Streisand effects will far and away outpace any remedial action they might try to achieve here. While I get that they are irritated by their customers calling them up knowing more about how broken their network is, than the tech answering the phone, but this is going to be the new reality. With Google, Netflix, and no doubt Amazon providing more information to their customers about exactly why their product is having issues.


Until the law firm responsible for their client getting Streisanded suffers significant and public repercussions for doing so, they'll continue to advise this kind of action because billable hours.


That's why I just tweeted this @Verizon:

.@Verizon if it weren't for that C&D you sent to @netflix I would never have known how shitty you've been to your customers. THANKS!


It's more likely that the executives are telling their (in-house) lawyers 'you need to shut this down' and leaving it at that.


I'm so outraged that I'm going to switch my provider from Verizon to....nobody. Since there's no competition and it looks like the providers have formed a cartel to collude on selectively restricting bandwidth to Netflix.

They've got us right where they want us.


Verizon fios customer here. There has been a noticeable drop in the quality of streams in the past couple of months. The picture went from excellent > barely functional(after the FCC ruling) > watchable(after the peering agreement) still not great. Kudos to Netflix.


Yeah, same here. I was watching a movie on Roku/Netflix 6 weeks or so ago. Quality was 2* (out of 4, meaning OK, but visible encoding artifacts now and again) and yet it still fell back to the buffering screen 8 times in 30 minutes. I got fed up and rented the same movie on Amazon (still on Roku) and watched it in HD without a single instance of buffering. After it was done I switched back to Netflix and tried to watch something else, and again hit buffering every couple of minutes. It's not usually that bad, but it clearly illustrates that somewhere in the Netflix -> Verizon connection that something is badly wrong.


That's funny. I'm using a Roku as well and had pretty much the same experience except I rented a movie from M-Go. Of note, I'm paying for "Quantum" speeds 75/35.


Yeah, likewise. On speedtest.net I'm 57 down, 38 up.


Funny how Amazon works fine but Netflix doesn't, yet the groupthink here blames Verizon rather than Netflix.


Note that I said "somewhere in the Netflix -> Verizon connection".


Verizon FIOS customer here (with a "business" link, I wanted a static IP, which is not offered with "residential").

I've seen no change in download/upload performance myself.

So I suspect they prioritize the "business class" customers above the "residential" customers.

[EDIT]: fix a spelling typo.


The parent was referring directly to Netflix performance, not raw upload/download speeds...


I've read elsewhere that business class internet connections are regulated by a different set of law than last mile residential connections.


For scientific purposes, could you buy a VPN service for a month and see if that makes it work better? Because then it's just super evil anti-net neutrality on Verizon's part and not actual congestion.


I noticed that Youtube performance went up around the time the peering agreement happened.


In the cease and desist, the Verizon's lawyers allege that Netflix can't possibly know if the network slow-down is coming from Verizon's network or other parts of the internet. I actually chuckled when I read this -- because after reading Netflix Tech blog (here: http://techblog.netflix.com/ ) and seeing Netflix's open source contributions (here: https://github.com/Netflix) I have a feeling Netflix probably has a LOT of data to back up these assertions.


I love what Netflix is doing with this. Not so much because I am totally convinced that they are 100% right and the Cable companies are 100% wrong.

Mostly just because I 100% hate all the cable companies.


Do you consider Verizon a cable company? I don't.

Come to think of it, what really separates cable and telephone companies these days? Most of them sell the same things as the others (TV, internet, phone), and the distinction is largely historical and technology based.


I believe that the solution is forcibly separate infrastructure from service. The same company that provides the infrastructure should not be the same company that provides the service to the customer.

There are several nations around the world where this is the case. It reduces the barriers to entry for ISPs, creating an environment where there are dozens of ISPs to choose from with differing levels of customer service and pricing plans.

The primary problem with Comcast and Verizon is that they can leverage their customer base as a negotiation tactic rather than solely on the state of the network.


Netflix is angry because Verizon isn't giving Netflix the bandwidth that Netflix paid for[0]. Netflix should ignore this C&D on free speech grounds, and tell Verizon to get lost, because corporations are people[0] and have free speech.

[0] I wish I was joking about that.


How do we get ISPs classified as common carriers?

Sue their pants off for the copyright infringement of their customers (and all the other illegal things customers can do). This why common carriers exist. They trade ultimate control over the transfer of content and treat it equally for the protections of being indemnified from the content itself. This is why the U.S. Postal Service can't be prosecuted for transporting a death threat letter. Make it about money and they'll jump on the common carrier bandwagon before you can blink.


They already have immunity for users' infringements via the Safe Harbor provisions of DMCA. The safe harbor provisions requirements are a lot less stringent then Common Carrier.

They must register as an ISP, can't modify data, and can't directly profit from infringement or promote it. Certain ISPs already have done things that should get them deregistered though, such as compressing images, replacing ads or injecting content. For example, comcast injects their copyright warnings into html pages transfered.




It's interesting how Verizon talks about the "panoply of middle-man networks" when that is really the Internet.

Verizon is a walled city. Verizon entices people to live in it's city and pay its rent because it says you can get deliveries from all over the world in the blink of an eye if you live in our city. It has high speed trains, and wide thoroughfares.

It also has a small gate that can let a few cars in at a time.

Outside the gates is another series of superhighways and supersonic trains that connect the whole world.

Netflix says "We can't deliver our movies because the gate to Verizon is too crowded."

Verizon says "You can't claim that, that's unfair. We offered to let you operate inside our walls if you wanted to pay our rent, but you didn't want to."

The problem is that when Verizon got all of its citizens, it promised that it would give them fast INTERNET access. It didn't promise them that it would give them fast access to things on the Verizon network, or things with special arrangements that bypass the internet and use a special toll entrance to the city.

There's no reason they can't widen the gate. The highways outside are absolutely fast enough to send as much traffic as they need into Verizon. But Verizon doesn't want to widen that gate, because they know if they don't, Netflix will have to pay them in order to deliver to their residents at a reasonable rate.

But they absolutely don't want their residents to think that they aren't getting their advertised fast access to the super-highways outside.


WRT that "panoply of middle-man networks" ... it pretty much never works that way. Certainly not with a big player like Netflix, they're going to be buying from providers that peer with Verizon, vs. ones that have to pay transit fees to providers that then have peering with Verizon. The latter would make absolutely no sense in any way, technical or financial.

They're also doing this in concert with AWS, right? Another huge player.


Interesting observation re: FiOS's performance with other video services. I'm not a huge fan of the incumbent ISPs in this country, but they do have something of a point there.


That's sort of the whole point... Lack of net neutrality leads to ISPs slowing down specific traffic (such as Netflix but not, say, Youtube).


I don't think that is how it works. The letter pretty clearly explains that Netflix uses for delivery third parties that have insufficient bandwidth to Verizon. Verizon did not -- at least according to this letter -- actively DO anything to slow Netflix down.


I haven't been following this too closely (I don't live in the US) so I apologise if what I'm about to say it absolute rubbish but... my understanding is Netflix use Level3 for transit and the problem is Verizon refuse to upgrade their peering capacity with Level3.


Yes. There was a post on Level3's blog explaining that they have zero peering problems worldwide with the exception of a tiny group of ISPs in the US, and this group is coincidentally ranked dead last in customer satisfaction.


It doesn't state it anywhere but I assume Netflix can be pretty sure of where the bottleneck is happening before transmitting that message correct? Given all the other factors that can contribute to a slow down.


> ...I assume Netflix can be pretty sure of where the bottleneck is happening ...

It depends on how fancy they get with their detection routines. They can certainly see which networks the traffic is flowing over, but then there are multiple devices (and networks) in play to deliver that traffic to them.

For instance, just a one-way trip showing the devices that could slow down a packet of data can give you an idea of the number of potential bottlenecks:

    -> Chromecast CPU
    
     -> Chromecast WiFi chipset
    
      -> User's home network router (usually, but not always, provided by ISP)
    
       -> Cable/DSL/FiOS modem
    
        -> [[ ISP devices they directly control ]] 
    
         -> Transit provider's network
    
          -> Netflix's network.
Netflix could certainly jump through hoops to isolate the bottleneck to the ISP network, but are they? Who knows?

I'm not sure they really need to anyway. The hockey stick graph they have showing throughput before and after payment looks like a classic mafia shakedown.


Those factors are smoothed out in aggregate, though. You can't make assumptions about individual cases, but you can look at the whole picture and compare different large ISPs pretty accurately.


If a large majority of customers that use a particular ISP but are having streaming problems across different devices and if Netflix has no load on their network, then you can assume its a problem with the middle-man.


The point Verizon is making here is that Netflix has not done due diligence to verify that the specific customer they are showing the "Verizon Sucks!" message to is having connection issues that are provably Verizon's fault.

It seems to me that they're stopping one foot shy of calling Netflix out for libel. It's not libel if it's true, so it might be better for Netflix to only show the message when they can prove that the specific connection is being slowed because of the interconnect (or lack thereof).


I think it would be better put as "claim", vs. point.

At the bottom of this item you can see in a lot of detail how Netflix is making this judgement: http://blog.streamingmedia.com/2014/06/netflixs-network-cong...

Sounds sufficient to me.


Two ugly chickens are coming home to roost here.

1) ISP's insisted on selling home bandwidth as functionally unlimited. They did this primarily so they would only have to offer expensive plans. Grandma can't get the 300meg email plan for $8, only the $80 plan. And then everybody tried to use it like it really was unlimited.

2) DRM. Drm makes it so that the net can't cache. 90% of netfilx bandwidth is likely the same 10% of videos. They all have to make the full trip through the wires. A local cache at the ISP to improve efficiency is impossible.


Regarding 2): Don't Netflix provide content servers / boxes / network devices that ISP's can put in the ISP's own data centres to help with exactly this issue?

https://www.netflix.com/openconnect


Netflix is paying someone for their traffic. Verizon customers are paying for broadband which includes streaming movies.

The ISP need to quit whining. They need to quit trying to get laws passed in every state that keep competition out.

Verizon 2014 1st quarter. $30.8 billion revenue $7.2 billion operating income http://www.verizon.com/investor/news_verizon_reports_fifth_c...

no sympathy


And I'll bet landline CAPEX, even adjusted for getting rid of various territories, is flat or declining. It certainly is for AT&T.


Error message should include links to better ISPs in your area.


Unfortunately, my choices are:

1) cable internet service from TWC (fortunately - they have decent service in my area so Netflix won't be displaying the "your provider sucks here is a better one" link for me)

2) 1.5mbits/s DSL from AT&T

Some might argue that satellite and wireless providers like Clear are viable alternatives. They are not as wireless has spotty connectivity and satellite broadband has unacceptable latency.

So, my choices are either broadband through the local cable monopoly or almost-broadband DSL through the telco that refuses to upgrade their infrastructure in the area that I live.

Free market indeed. :-/


In a lot of cases, I'd imagine it already does.


At most I've only had two choices for ISP - Comcast and Time Warner. Currently I have one "option" - Charter.

I guess you could add mobile providers to the list if you count mobile hotspots as broadband. That would be an additional 1-4 providers in most areas.


It should also include the support number for the ISPs number... you know, keep it handy just in case the user wants to call and complain.


Or an affiliate link to sign up for another ISP in their area. Or Netflix should start their own ISP (as a reseller or white label).


In most cases, you don't have a choice of ISP.


That as well as contact info for your city councilors and example legislation on use of eminent domain for last-mile cables.


"Click here to show your interest in Google Fiber".


Maybe Verizon would prefer the more accurate "Verizon is deliberately slowing down your Netflix experience."


1) Why are ISPs not classified as Common Carriers? This needs to happen like now.

2) This makes me want to get a Netflix subscription just to support them, knowing I would never watch a single video from them. I'm signing up right now. Go Netflix!


Anyone have a screenshot of what the error message actually says?


"The Verizon Network is Crowded Right Now."

http://arstechnica.com/information-technology/2014/06/netfli...


That seems like a very fair and accurate claim. They should've done it to Comcast, too, before they paid up, and when they did this to them:

http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...



Here's a tweet including a screen capture that was widely cited two days ago: https://twitter.com/yurivictor/status/473978204852453376


This is the image that I've seen posted to several news articles. https://twitter.com/yurivictor/status/473978204852453376

Edit: interesting, this seems to be the only screenshot anyone has?




Im sure most of the internet providers cap their bandwidth to services like this. I am confident that the main Swedish ISP provider Telia is capping as we speak.

I can see that ping times increases and it's always reoccurring around 9-10pm Fridays. Around 9:30 buffering speeds up, meaning most of the people giving up and switching to regular TV. One could argue that this is the time that Netflix/Youtube/Hbo/ViaPlay or any other service is used the most, but my suspicion is that capping occurs at those times, to validate the Fastlane/slowlane argument when the it arrives here in Sweden too.

I am currently collecting statistics to validate this. However, its insane that a ISP can cap and provide lesser service's on certain ports and argue that the fault is at the entertainment provider. Its also insane that when going from a "torrent way of life" to becoming a paying stream customer, I cant even watch my favourite show, when i want, on 100/100Mbit line.

ISP's are in the business of providing fast connections and high bandwidth, and that's what I'm paying for, however I want to use it. It shouldn't matter if I'm streaming from YouTube or downloaded illegal porn. Though it seems it looks like they want to sell me bad gasoline that doesn't take me anywhere, and say my engine is at fault.


Someone please clarify my doubts. When I run a server, I have to pay for bandwidth costs. For eg. My website hosted on linode gets me 20TB of data transfer limit. I expect end users to be able to view content worth 20 tb of up/down data transfer. Netflix runs its own servers but the ISP providing connectivity must be already charging Netflix for a certain bandwidth and data transfer limit. If the ISP is charging for data transfer already, I expect the ISP to provide the entire service. For eg. Let us assume that Verizon charges netflix 1 usd per tb of data transfer. And Netflix uses 20000tb of data in a month. Then Netflix owes Verizon 20000 usd a month. And Verizon has to serve the data according to the bandwidth agreed to.

Is Netflix not paying Verizon for the bandwidth and data transfer?

If Netflix is not paying Verizon then why does my hosting provider charge me for bandwidth and data transfer?


Netflix has their own ISP.

Netflix's ISP and Verizon pay eachother nothing to exchange traffic (called peering).

Verizon wakes up one day and says, "Shit out of all the ISPs we peer with, Netflix and their ISP is responsible for 35% of our traffic, we have to do huge upgrades on the link to them, and they aren't paying us a dime"

That's my understanding of what is going on.


The traffic exchange would only occur once a customer routing data over a Verizon network uses Netflix, no? Meaning that the 35% of the Verizon network usage that Netflix uses, is due to customers actually using the connection they pay for, no?

As an ISP, I think if you are offering 100Mbps unlimited in/out, you should Expect full thrust from every connection, or at least be capable of delivering that. I know it's not really realistic, they need to assume that some people, sometimes are disconnected but if I am paying for a connection and a certain level of throughput, I expect to get that.

Just my point of view, correct me if my assumptions are incorrect!


Over subscription is what drives economies of scale in IaaS and PaaS. It allows companies to drive down prices and increase performance for the p99 customers.


Do you have any idea what internet service would cost if they built every node to handle 100% of the Internet's traffic?


Hence why I said "I know it's not really realistic". To answer your question though, I have no idea. Do you? I'd love to know what an ISP builds to expect in the terms of constant traffic..


Does Netflix's ISP pay anything to anyone? If no why does linode make me pay for data transfer?

Thanks for replying (Linode is just an example)


To prevent you from abusing the network and keep their data center not congested, and have other customers service not degraded by your uses.

Linode isn't an ISP, they still have to deal with ISPs for the end user, which is why it is different.


There is an easy and practical solution to this whole problem: If Verizon doesn't like the amount of bandwidth their customers use watching Netflix, they can always choose to block Netflix completely on their network, and inform their customers that they've done so. It would be honest, clean, fair, etc. Customers would be paying for exactly what they are getting, everyone could be happy with the arrangement.

Of course there would be a massive revolt of their customers if they did so. Because their customers WANT Netflix. So instead, Verizon will try very hard to make it "sorta" work and blame Netflix while trying to get money from them.


Perhaps I've misunderstood the situation in the US, but isn't the problem that Verizon are a de-facto monopoly in many markets? If customers can't say "screw you" then all Verizon need to do is avoid getting hammered by the DOJ anti-trust department.


Many customers, but perhaps not many markets.

I'm in the same position with AT&T, in a residence that's at the edge of two cities, but is unincorporated (not part of any city, just the county, state, and the USA itself). No cable service is available, the deals they cut with the two cities don't cover me. Those deals make them local monopolies, long ago the original AT&T (which got split up, and then recombined into AT&T, Verizon and Century) arranged a Federal and state level monopoly, except of course for many rural and otherwise expensive to serve markets, which are served by smaller telcos.

I have at least one option for a WISP, but it's awful; at least AT&T's ADSL here is reliable. For now, AT&T really doesn't want it's landline business, wants to drop it for more profitable wireless. Ditto Verizon.




It's funny that Verizon thinks Netflix is damaging the Verizon brand. As if there's anything more they could do to lower people's opinions of Verizon.


I love it. Fuck you Verizon/Comcast/AT&T.


What is the point of Verizon sending such a letter?

Do they think Netflix will comply? "Oh Jeez we hadn't thought this through but now with this letter and all, we understand! We'll just start blaming our own servers instead. Thanks."

Or is it simply a necessary step before they sue? But if they sue, isn't it likely they'll lose? (Sue over what anyway?)


Clearly they are just preparing ground for possible future legal actions against Netflix. I don't think on their sane minds that Netflix would ever comply to that.


I certainly hope that if they sue, Netflix countersues and gets attorneys fees and punitive damages.


I wonder why ATT didn't C & D Netflix as well? Netflix had that notice for ATT too, just like Verizon. AT & T screenshot from a few days ago: https://twitter.com/kunjanshah/status/473152026147557376


Whatever happened to the right to free speech? There are of course restrictions to free speech, such as you cannot yell fire in a crowded building unless there really is a fire, but this is not one of those scenarios. The behavior of Verizon and the other ISPs are grossly offensive!


Don't forget about defamation. Sure, you can spread lies in order to do me harm. But I can sue you for the damage done by those lies. Why would you think that lies that damage a reputation are substantially different from vandalism that damages a building?

Not that I think Verizon actually has a strong case for defamation. Netflix should have little difficulty arguing that they had good reason to believe that their claims are true. I don't think this is going to end well for Verizon.


This seems very naive on VZW's part its extremely easy to show that there is congestion at the ingress of one of your peers. Even if VZW is not a direct peer its pretty easy for them to get metrics off their clients deployed at the last hop.


Netflix should also be including the phone numbers of city council members in the zip code of the account owner, and a link to example legislation for use of eminent domain seizure on last-mile cables.


> "So Netflix is throwing up a 'The AT&T network is crowded right now' message. Meanwhile, Youtube is playing 1080p no problem."

Netflix should try to explain that in the error messages too.


AT&T can choose to throttle specific services.


Netflix could also make their buffering settings more advanced. i want a bigger buffer if my internet is unstable, and it should be transparant how netflix buffers..


If it takes 2 hours to send you 1 hour of content, no amount of buffering is going to give you the instant gratification you want.


It already does that, but there's only so much buffering you're willing to wait for.


Thank you Netflix for telling Verizon to go fuck themselves.


I support this in principle, but I would bet a good portion of the buffering that happens for Verizon customers is due to issues with their own wifi network.


I'm on CenturyLink DSL and they don't show that message for me when there are speed issues... they apparently only show it to Verizon customers?


If you expend the list (include smaller ISPs) provided here(http://ispspeedindex.netflix.com/usa), yours is still above Verizon


Netflix is testing it in select markets.


Hopefully YouTube will follow the lead.


I thought they were paying fees to Verizon.


Wow, Verizon... dumb move.


Streisand Effect.


Well I think we should introduce a tax on commuters because as it turns out they are the 90% of the highway traffic.If they want to get to work on time they can use the fast lane for a little extra and if they don't pay we just make them drive 10 miles per hour. I think this is fair. Ohh btw. if they complain about it, it is just them trying to influence policy that was set by good corporations so I don't understand why we should change it. :)

(please look up irony on wikipedia before downvote, thanks)


Awww, poor Verizon...


The heat is on!


Is Verizon actively slowing the connection to Netflix.com itself or is the connection between Verizon and Netflix's provider (Cogent?) simply limited? In other words, are all Cogent customers suffering poor performance with Verizon customers?

Because the former is wrong and demanding more money from a single company hosted on a service provider you have specific peering agreements with is extortion.

The latter however, well, no one said there were unlimited pipes between every transit provider on the planet and if you're someone large like Netflix, sometimes you need to pay for transit on more than one provider to get the performance you need when your own provider can't or won't do so themselves. It has been like that for decades.


Netflix may be reaping what they have sown here. They agreed to buy paid peering from Verizon, so if they don't have enough capacity into Verizon the onus is on them to buy more. OTOH if there is some kind of problem inside Verizon then the error message is justified.


But peering really only links the network at the "top level". Verizon could still have lots of congestion within their network which would cause lack of end-to-end available bandwidth.


This is exactly why I dropped Netflix. I want a high-def movie, not their whining about someone else's network. I have no problems streaming from Amazon. I have a choice of streamers. Other streamers work better. I have less choice of wired ISPs and they're harder to switch. Netflix would rather whine and preserve its profit margins than pay up. OK, fine, but since their whining does not get me the movie I want, I cut it off.

Netflix will find that most other people also do not care to hear their whining. I can understand why Netflix does it though...it's $8 a month. Between production costs, licensing costs, marketing, and profits, all $8 a month leaves room for is crappy movies, poor investment, and a continuous drive to shift costs to ISPs and whine about it.


You are paying your ISP to move bytes to and from your home. It doesn't matter where those bytes come from or go to; you are paying for that bandwidth. It's already paid for. You either get that bandwidth per your agreement with them or you don't.

It's not like Netflix is causing you to magically download more bytes. You chose to do this, and that's fine, because that's what you're paying your ISP for. Comcast, et al want to double-dip on this transaction and you should be extremely not OK with that. Why wouldn't the ISPs do the exact same thing to Amazon that they did to Netflix as soon as they become a big enough target and the legal precedent has been established? This is something that needs to be nipped in the bud, like NOW, before this becomes an established way for ISPs to do business.

Also, it's not like all of your favorite services that will someday be paying Comcast protection money not to throttle them are just going to swallow those costs. Those costs will get passed on to you one way or another. No good can possibly come out of any of this.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: