I sympathize with why the mods changed the title from "Porn" to "Video", but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall. That's not an ideal usability decision. Why not use good ol square brackets to maintain the integrity of the title:
How I Made [Porn] Video Streaming 20x More Efficient with Python and C
>but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall.
If you get heat about things like that in your workplace, then run, don't walk.
I think your sympathy is overapplied here... that's very clearly an important bit of information for anyone considering clicking on this link. Thanks for putting the warning up for those of us who would not otherwise have known thanks to a poor title edit.
I don't mean to be a hardass, but... citation needed.
Seriously, we have no data on the number of mod edits made vs. complaints voiced. As such it seems premature to conclude that mods mostly do a good job with these arbitrary edits.
Agreed that RTMP is an abomination that needs to be exorcised from the Earth as soon as possible. Unfortunately, it is probably here to stay until something like WebRTC gains critical mass.
It's not clear what the article means by "repackaging" a stream or "pointers" to tags (especially in the diagram that shows tag pointers being transported between users). While RTMP is cumbersome, shoving media data (tags) under a per-session protocol header is essentially the standard way of moving data from one session to another.
So I'm not really following this. Is this cutting out the RTMP entirely for receiving clients, and instead sending the FLV down via another transport, like HTTP or whatever? Or is it more of, "I wrote my own RTMP streaming server in C and Python", along with some implementation details which I'm not understanding? (Not that there's anything wrong with doing so. Options are limited in the streaming-server space.)
>It's not clear what the article means by "repackaging" a
stream or "pointers" to tags (especially in the diagram
that shows tag pointers being transported between users).
By repackaging I meant extracting the FLV tags which pretty much travel in the same format as in an FLV stream (.flv file) if memory serves well. Pointers to tags refer to the internal implementation. I took the FLV tags out of the RTMP stream, which resulted in an almost complete FLV stream. With the proper header prepended to it you could save it as a file and play it or stream it and play it. That's just what I did, created a header for every new user, and after that was sent I could just stream the FLV tags from a common buffer. The users had pointers that pointed in this buffer so after the header is sent it was true multicasting.
> So I'm not really following this. Is this cutting out the
RTMP entirely for receiving clients, and instead sending
the FLV down via another transport, like HTTP or whatever?
Or is it more of, "I wrote my own RTMP streaming server in
C and Python", along with some implementation details
which I'm not understanding? (Not that there's anything
wrong with doing so. Options are limited in the streaming-
server space.)
Yes. From the source RTMP stream I extract the FLV tags, which I could use to multicast. Sending the same RTMP stuff to every user would not work, but I can easily send the FLV tag stream over HTTP if I send the crafted header first.
> I extract the FLV tags, which I could use to multicast
I assume you don't really mean IP multicast (which would, incidentally, be one approach to edge-origin mirroring for the popular marketing campaigns you mentioned, at least within a data center).
Anyway that makes sense, simply sending the FLV is clever. What's the method of delivery -- chunked HTTP? Does it play well with proxies?
Oh, please. The article uses the word porn exactly three times in the introductory paragraphs. Four if you count the title.
The comments here have double that, including your own use. If you were genuinely that paranoid about being "logged" for having visited a page that used the word porn (really?!) I doubt you'd be using it yourself.
I apologize if it sounded that way, but I wasn't insinuating that I'm going to get into trouble.
I'm just saying that it could get someone into trouble.
As far as the "logging" goes, it had more to do with the word porn in the title of the page, because many content-filters just parse the title for blacklisted words.
I love reading articles about technical issues and solutions in the Porn industry. It's like getting a peek inside a youtube scale company as they grow.
Disregarding the morality and ethics that are often center-stage, they are faced with really interesting technical hurdles.
It's an old article, but I was fascinated to hear about some of the challenges they face as well: http://highscalability.com/blog/2012/4/2/youporn-targeting-2...
"In short, porn sites cope with astronomical amounts of data. The only sites that really come close in term of raw bandwidth are YouTube or Hulu, but even then YouPorn is something like six times larger than Hulu."
Probably. Netflix has the big constraint of availability that only extends to the US. Porn is global. Even though the US has a above-average bandwidth usage (guesstimated) it's still only a small(ish) drop in the global network.
I'm astonished by some of the responses here, that could be summarised thusly:
>"I work on workplace, in a modern western society, not some theocratic backwater, that monitors my web activity and would frown if I visited an article with the word porn in it. This on 2013. I find this OK, and won't quit my job or raise hell protesting this degrading treatment, but would rather complain for HN titles".
In an age where people fight for LGBT rights, this is what the American workplace has come to?
Unfortunately, the majority of jobs in the United States are run with this kind of degrading treatment. Many even require you to randomly pee into a cup on short notice, to make sure you didn't do any drugs recently. Some of them also monitor your Facebook accounts (as far as permissions allow) to see what you're up to there. A few require you to hand over your Facebook passwords (!) to the boss to make that easier. The corporate world is weird and scary, but not always easy to avoid.
>A few require you to hand over your Facebook passwords (!) to the boss
That was a poorly-source, probably-made-up story that now one could verify yet quickly became accepted truth. (Unless it's meant to refer to cases involving heavy security clearances, in which cases it's ho-hum)
Since when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary?
I work at a small company where no one would care. They also don't care if I spend time on Facebook, Twitter, etc. etc. But not everyone is in a company like that. Big legacy companies hire from the HN crowd too. Whether or not their policies are fair is not always a simplistic argument. However, what is as sure as rain is that trivial violations of those policies can be used as leverage to punish people, when the office politics get dirty and desperate.
It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it.
>Since when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary?
Since people let them get away with it. The "agreement that involves paying that employee's salary" does not meant they should get away with treating him less than a civilized society accept.
Signing a contract to work on some place doesn't give the employer any inherent rights over the employee, besides those that the society is willing to accept.
Hitting an employee was once tolerated. Not so anymore. One time not hiring or paying blacks less was allowed. Not so anymore (not explicitly at least). Child labour was allowed. Not anymore. Racial or sexist slurs were allowed. Not anymore. Lax safety at work was tolerated. Not anymore.
So it's not like there is some undeniable inherent right of an employeer to "have the employee pee in a cup" or to "check his Facebook profile". It's just that people haven't protested enough to make it into law that it's not his fucking business what the employee does at his own time.
>It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it.
Sure, but at least some anger should be directed against businesses having those policies, not just on HN titles, as if the policies are OK.
> In an age where people fight for LGBT rights, this is what the American workplace has come to?
Yes. With a lack of privacy expectations in 2013, any visited website is permanent added to the record. Even with no paranoia, it does add an unnecessary risk to ones career.
It's not that simple when you have a family to provide for. The job you have might be the best job for many miles.
I don't agree with the decision not to protest, and I don't think HN should really care about catering to that demographic, but I understand their motive.
The link is that sex (even gay/lesbian etc sex) is not a sin, what people like sexually is not the business of their employeer, and if we can accept people doing parades for the right to fuck each other in the ass or similar, we sure should be able to accept people reading the word "porn" on a website at work.
That's the implicit link. The explicit link is that LGBT advocates and activists have frequently defended porn and freedom of sexual expression through it, too.
trigger-happy, i guess. Given that "Porn" definitely implies a much larger scale than merely "video streaming", I think the title change does a disservice
If anyone is interested in an alternative to the usual RTMP servers (FMS, Red5, and Wowza), I highly recommend EvoStream (http://www.evostream.com). Compared to the alternatives, EvoStream is much more efficient. I believe TinyChat published a whitepaper discussing their transition from Red5 to EvoStream, which resulted in a decline in the number of required servers.
EvoStream is a highly scalable streaming media server written in C++ based off of the open source RTMPD (http://www.rtmpd.com). The commercial company, also called EvoStream, is a relatively new startup and they do great custom work for those not familiar with streaming media/RTMP.
It's easy to underestimate the power of switching to a better language by just doing it - guess at the syntax until it works, then refactor as you start to understand the language and its culture more. In fact I've found I learn faster this way than any other.
You also have Red5 which is open source and rocks very much. It is just a -tad- less reliable than Wowza which you can't really sell to paying customers, but it's much more fun and customizable.
And to answer your question, Wowza is just a cheaper FMS which also supports other platforms next to RTMP. This article is basically taking the more performant HTTP download mechanism for static content (like YouTube uses) and then hacking it to put a live stream in it instead.
To be fair I clicked on the article and immediately came back to the comments expecting the absolutely inevitable sprawl of comments about the title edit. :P
As someone who built the infrastructure for serving porn for Kink.com, I'd say that this was a total waste of time. Spend the money on a third party CDN and serve from there.
Take a look at KinkLive.com. We were the first porn company to do live streaming in HD using a CDN (Bitgravity), all paid for, by the minute, with a micro currency system (kinks) that we built.
I've posted about small parts of it in various places, but not the whole experience. I no longer work there (since April 2010), so while I know they still use quite a bit of the serving infrastructure that I built, my knowledge is now quite a bit out of date.
One fun bit that I built is called the cockblocker (as you can imagine, we used all sorts of fun names for internal projects). People who repeatedly attempt to hack the system (usually through various forms of abuse like failed login attempts) would automatically get their IP address routed to /dev/null.
I'm not really sure what you mean, but absolutely using a CDN product is the way to go when you want to sell your content on a global scale. Otherwise, you are going to deal with having to build / maintain your own and the cost of doing that is far higher than just letting other experts deal with the issues that surround it.
I know others already said this, but I clicked on this link from work and immediately became appalled when realizing it was about porn and quickly backed out. I'd gladly read this from the comfort of my own home and I'm sure the content itself is SFW but still the point is I was mislead. Can we please [mods] not change titles in cases like this?
Thanks. This is the only article on the blog? I was actually intrigued by the site and was wondering if anyone on HN has used this service as a dev or a client....
OP here, and Director of Community at Toptal. This is the first live article, but we've got loads more in progress covering a bunch of different projects and technologies.
Ballsy to lead with an article about porn. I'm glad you did because I know they must face scaling challenges that only a handful of services have to deal with, but rarely do I see articles about how they overcome the challenges. Thanks.
I'm glad you liked it, it was a lot of fun. I also implemented on the fly transcoding, but that's probably for a different article.
For the tech, well really at that place we used pretty much what everyone else did. Not a lot of big secrets, as far as I know the porn driving the tech revolution thing is ridiculous.
I was in talks with them as a dev. Their requirements were a bit weird, so that's why I didn't sign a contract in the end. I think there was one requirements which asked that the developer answer the employer's message within a pretty short time frame, not matter the timezone difference.
Hey Elbear, that's not in our contracts, at all. We do make sure our developers communicate within 10 hours regardless of the timezone. We feel that is more than reasonable and it's worked extremely well for Toptal. However, that's not in our contract, it's actually something we simply stay on top of as a company. Our requirements are not "weird" at all, they enforce high integrity, and in most places in the world, the concept of high integrity is "weird". The type of people who will not conform to such standards are precisely the types of individuals who we would never want to work with in a million years, and that is precisely why freelancing platforms have such a pervasively bad reputation.. because they're filled with low or even medium integrity individuals. To us, that's unacceptable. As it is to any A-player team. -Taso, CEO, Toptal
You're right that it wasn't in the contracts. It was in an internal rulebook for developers. Too bad I didn't keep it, so I could quote from it exactly.
Anyway, my final experience with TopTal was that I asked some questions about the contract that I was about to sign and never heard back.
Your version of "on call" is not congruent with ours or most of the world. When a doctor or DBA is "on call", they have to get up, fix something, and spend hours if not days doing it. We simply enforce communication. If you want to call that "on call" then your decision, and we disagree with that definition. To answer you question, yes, if you're responsible (in our subjective definition of what constitutes responsibility), you will answer within around 10 hours. In practically all communication you can reply "I got this I'll answer you later." or something similar to show responsiveness. The word responsibility stems from "response ability" and we believe that everyone should be... responsible. That's in our DNA.
"For example, if the in-house engineers start their workday at 11am PST, our Eastern European toptal engineers will start working at 7pm EET and work through the night."
"During work hours, they are expected to respond to any client communication within 10 minutes; during non-work hours, they are expected to respond within 3 hours."
Great work. I worked with RTMP for some time and I recall the pain to reverse-engineer their protocol. One other comment, maybe the title should be more like: "How I improved a slow/inefficient RTMP video streaming service by 20x". Just a thought.
Unfortunately that's the case. I heard really good things about wowza these days. Not open source but a lot of things are open and flexible. Also back then red5 was also an ok alternative, not sure if that's still the case.
How is it NSFW? It is an article that is about video streaming on scale. Yes it does mention it was for a porn site, but it really does not have much to do with the article. If you work somewhere where this would be viewed as inappropriate I am not sure what can be viewed at that employer.
Because, like it or not, the porn industry dealt with a lot of the technical issues of selling video, pictures, and video streams on the internet before anyone else. They needed it to be profitable / have actual revenue.
Nothing. I can talk about my work on a blog or to a technical person without mentioning once that it's porn related and I've been a programmer in the porn industry for over 10 years. They used it as a way to gain viewers.
Actually this was a website which streamed online broadcasts live from people, like ustream does but with adult content. Some broadcasts could be hours long.
How I Made [Porn] Video Streaming 20x More Efficient with Python and C