> Uptake of the open-video codec AV1 has been slow, with major video providers waiting for broader device support.
The people involved in AV1 had a software decode plan B that has worked better than most of them hoped.
If you combine that with encoding choice based on software decode impact, most big players have started with software rollouts, but still seen gains.
"You need hardware support for a codec" is often the same kind of delaying tactic as "renewables need storage". You do both in parallel, not enter an infinite loop where you don't do either because the other isn't complete yet.
This is just lazy and risk-averse management. With the list of companies behind AV1 is certainly looked like the codec of the future several years ago. While predicting the future can be dicey, switching to AV1 should have been fairly obvious. Everyone waiting to see if it will actually take off has simply delayed the takeoff. Software decode has been around for years, so content providers could have encoded and used that for desktop users, or even people on mobile - half of them will blame their devices when the battery doesn't last.
> switching to AV1 should have been fairly obvious
Things are moving, but not quickly.
VLC shipped an optimized x86 software decoder very recently, but Handbrake has yet to find an open source software encoder in good enough shape that they can add it for general consumer use.
The entrenched interests are adverse to wide adoption of a royalty-free codec. Even some of the parties involved with AV1 have a conflict of interest as they are patent holders for other technologies that would exclude AV1's wider usage.
> The entrenched interests are adverse to wide adoption of a royalty-free codec.
I see this claim again and again on HN, but in reality "entrenched interests" ship with support for lots of royalty-free audio, video, and image formats.
The reality is that AV1 is just not ready for prime-time yet. As TFA notes, it's a chicken-and-egg problem, and few companies have stepped up to be among the first chickens.
If this rumor is correct, the first phones with Qualcomm-powered AV1 support will ship in 2023. At that point, expect the AV1 IP wars to begin. AV1's future depends on how that goes.
> If this rumor is correct, the first phones with Qualcomm-powered AV1 support will ship in 2023. At that point, expect the AV1 IP wars to begin. AV1's future depends on how that goes.
Intel added AV1 decoding to their onchip GPU's in 2020 [0]. Why didn't it begin then?
> Intel added AV1 decoding to their onchip GPU's in 2020 [0]. Why didn't it begin then?
I don't have any special insight into Sisvel's strategy, but I'd guess there are a few factors. This assumes that Intel isn't already a licensee — Sisvel has 32 AV1/VP9 licensees last I looked, and I believe only a few are public.
- Very little content is delivered as AV1, and where it is used it's not exclusive, so content distributors could easily kill Sisvel's golden goose today. They can't move too soon.
- I imagine Sisvel is happy to focus on low-hanging fruit while AV1's popularity (presumably) soars. They'll continue to build their stable of voluntary licensees until AV1 use hits a tipping point where the ROI for harder targets makes sense. The day Apple announces support for AV1, Sisvel will be celebrating harder than HN'ers.
- When push comes to shove, I'd guess that Sisvel will most likely build a start building legal wins by going after smaller companies first, then work their way up to AOM members.
Sisvel needs AV1 to be successful in order to extract maximum revenue. Qualcomm's support increases the odds that will happen, but it doesn't guarantee it. That necessary success probably happens a minimum of 2 years after broad device support.
Hardware support for a codec is a business decision decision, it doesn’t require a tech/physics breakthrough as renewables storage at city and country scale does, it’s not remotely comparable.
> renewables storage at city and country scale does
That's also a business decision, just with a bigger number attached. And hitherto it's been far cheaper to use gas. But that's now a "conflict mineral".
Renewable storage is also a business decision? Like we know how to build enough storage, it's just very expensive using current technology. In the limit, you can just make a lot of these:
That article is pretty confusing, since it only talks about the power of those batteries, not how much energy they can store. Or is it confusing MWh with MW?
Note, no support for hardware encoding was announced by Qualcomm. Decoding only. This seems to repeat the trajectory of VP9, where it exists and is supported solely thanks to YouTube, while the whole world is utilising H.265 for everything else, including recording video on smartphones.
It very well may be AV1 and H.266 will play the same roles in the future.
> the whole world is utilising H.265 for everything else.
This is very hyerbolic. There's plenty of places that the HEVC licencing catastrophe stopped its use. The release of 3 more MPEG codecs that compete with it is not a good sign for example.
> This seems to repeat the trajectory of VP9, where it exists and is supported solely thanks to YouTube, while the whole world is utilising H.265 for everything else, including recording video
VP9 came too close on the heels of VP8. Thanks to WebRTC in particular, VP8 encoding support is everywhere. AV1 has a lot more backing, thanks in no small part to the H.265 licensing mess.
Google bought a chip design company and has been releasing codec chip designs for years, probably during the VP8 era but certainly VP9. It's now at the stage when they don't really need to it mostly just happens.
The reference software implementation (libaom) is royalty-free. Any company can come up with their own proprietary implementation, including in hardware. There are no royalty-free hardware designs at the moment.
Well, the actual standard is royalty-free and that's what we usually mean by royalty-free in this context. A royalty-based standard requires you to purchase a license for developing your own implementation of it.
Yes. The codec itself and the reference implementation are both royalty-free, so we’re both right. The implementation part is more relevant here since gp was asking about it.
AV1 has had several issues plaguing its uptake (as a close observer of the technology):
- The reference silicon design for an AV1 decoder was very space-inefficient, and so many chip vendors refused to use it. This, however, necessitated the development of internal or 3rd-party decoding IP which slowed uptake considerably.
- AV1, partially due to it's patent-free design, has an extremely complicated compression process. So complicated, that even in software it is still a few dozen times slower than H.264 encoding. This, plus the first issue mentioned above, means on-chip AV1 encoding is a ways away.
- AV1, despite being "patent-free," has had a patent troll (Sisvel) come along and claim that they have a bunch of patents applying to it. They want 20 cents per device. AV1's founding group claims to have a legal defense fund for members but the terms and coverage are unclear. Sisvel wants 20 cents per device using AV1, so now you need to get the lawyers involved to figure out whether to bet on paying Sisvel or bet on taking the AV1 founder's view of the matter.
- Finally... there's AV2 coming. Though a ways away, it will use recently-expired patents to further help its compression capabilities, new techniques that couldn't be completed on time for AV1, and will feature a more optimized reference silicon implementation that might be actually easily usable by chip designers this time.
"While still working on the format, the encoder was not targeted for production use and speed optimizations were not prioritized. Consequently, the early version of AV1 was orders of magnitude slower than existing HEVC encoders. Much of the development effort was consequently shifted towards maturing the reference encoder. In March 2019, it was reported that the speed of the reference encoder had improved greatly and within the same order of magnitude as encoders for other common formats."
Other encoders also exist that are much faster, listed in a different section of the wiki.
I didn't know about Sisvel and the hardware design problems, although Sisvel is in the wiki. The specific license fees that you listed are interesting.
I'm hoping future Apple chips have AV1 decode support as well. If Intel, Google, Qualcomm, and Apple all add AV1 support in hardware, it would become the gold standard as far as codecs go. Right now it's all a mixed bag.
Considering Apple's love for proprietary everything and their large market share, I wouldn't be surprised to see them roll out their own codec in SW and HW, same how they developed Metal instead of using the open Vulkan.
a) Apple has always been an active part of the MPEG standards group for decades so not sure why they would roll their own.
b) CoreGraphics and CoreAnimation are underpinned by Metal. If you were building an OS that was going to be used by every device you make (computer, phone, tablet, monitor, watch, TV streamer, headset) would you rely on an open source project that you don't control and may have different values, wishes, roadmap etc than you.
> If you were building an OS that was going to be used by every device you make […] would you rely on an open source project that you don't control and may have different values, wishes, roadmap etc than you.
Apple's XNU kernel is a derivation of open source OSFMK and FreeBSD kernels, and Apple's Darwin has large components of FreeBSD userland. They use (se)L4 for running their Secure Enclave.
I remember in the early days of MacOS where entries from the FreeBSD release notes would appears as word-for-word copies in the Apple ones.
Apple wouldn’t do their own video codec, they’d adopt VVC[0], which is what looks like what will become the standard for non-web video (e.g. DVB over the air broadcasts in Europe)
[0] https://en.m.wikipedia.org/wiki/Versatile_Video_Coding
I don't think the Vulcan/Metal comparison is useful here. Metal shipped before Vulcan so it was never an option for Apple. The case of AV1 is different.
Not sure why that would matter. Apple is currently a "Promoter Member" of Khronos Group (the non-profit consortium behind Vulkan) but still did their own thing with Metal instead of going with Vulkan.
No. ProRes is designed to be a good compression scheme for editing. That means it has to be designed in a way that make jumping to arbitrary frames and scrubbing very fast, as well as it needs to focus very hard on near-elimination of compression artifacts.
In comparison compression systems designed for playback, like AV1, can have more occasional key-frames (points that you can easily jump to, then work your way to other frames), and it is more ok to have compression artifacts, especially ones that are just going to look like motion blurs when played back.
So the compromises between features and size of the output stream are different.
Out of curiosity, what is the big difference of AV1 vs h264 and HEVC / x265 ?
As someone who mainly cares about blu-ray / large movies, I was under the impression that HEVC was becoming the gold standard. Is that not the case, and/or is AV1 the next iteration, and/or do they solve different problems?
There are just so many patents on things both specific and vague, that you can't ever be sure you're not infringing on some. As far as I know, all serious companies using HEVC pay to licence relevant patents, while the big players using AV1 don't. If I was building a business than had a choice of codec, I'd rather bet on AV1. Worst case, you get dragged into a legal battle, but so do Youtube and Netflix.
> Worst case, you get dragged into a legal battle, but so do Youtube and Netflix
Except that's not how legal cases work.
Just because all three of you infringe doesn't mean lawsuits will result for all three.
It's quite likely they would allow Youtube and Netflix to continue infringing to promote the growth and adoption of the codec whilst they go after smaller players who won't be able to put up a fight.
Sorry, but what you say doesn't make any sense. You think that YouTube and Netflix will just sit by as precedent is decided next to them? And the right owners having already won in court would just ignore the biggest "offenders"?
Google and Netflix would have no choice but to sit by as you can't force yourself to be sued.
You can try and invalidate the patent but in a situation like this where the patent holders are serious companies and legitimate innovators it's unlikely to get you far.
And yes it is very common to bully lots of smaller players rather than get into an expensive protracted lawsuit with a large one.
> Google and Netflix would have no choice but to sit by
In a lot of jurisdictions a 3rd party can join a lawsuit, it's called an intervention [1].
I'm not fully aware on US law, but e.g. in NL all large internet providers joined a lawsuit as defendants when a copyright enforcer wanted 1 to block The Pirate Bay.
First you bully enough small-scale offenders who don't fight back to establish precedence and only then you go after the big fishes.
This is why grassroots-scale efforts to defend against patent trolls are so important - you absolutely have to prevent precedence case building for patent trolls or it becomes so much harder down the road.
The patent situation is always unclear: Companies were gladly licensing the MP3 patent portfolio to the best of their abilities for their portable audio players and yet their shelves were cleared by customs because they missed licensing patents that were not part of the "patent pool". See https://edri.org/our-work/edrigramnumber6-5german-police-ceb...
Paying off _some_ patent holders doesn't put you in the clear for all the others. No licensing pool indemnifies you against all other claims. There's never an exhaustive list of patents that apply to any particular product. "Competing product's patent situation is unclear" is always true, no matter the product (unless it's at least some 20+ years old, which should make it _somewhat_ safe because applicable patents expired). Bringing it up to favor one product over another is FUD though because it's true for the other product just as well.
The patent regime is an extortion racket: patent offices hand out exclusive rights for money, without accepting any responsibility that these rights are warranted. They don't even guarantee that their own catalog is free from conflicting rights.
> For head content, it's okay to streaming multiple formats because the viewership is huge, so streaming multiple formats, although increase our cost, it still actually save our traffic cost, so it's still worth while. But for the tail content, it's very different. We can only afford streaming one single format, so our strategy is currently still doing H.264 using hardware, high-density hardware solution, but we're hoping towards 2024, 2025, the AV1 ecosystem is ready, we want to switch to AV1 100%.
>
> Jan Ozer: Did you say 2024 and 2025?
>
> Yueshi Shen: 2024, this is our projection right now. But on the other hand, so as I said, our AV1 release will be, for the head content will be a lot sooner, we are hoping 2022-2023 we are going to release AV1 for the head content. But for the head content, we will continue to stream dual-format, AV1, H.264. But for the tail content, we are hoping towards five years from now to AV1, whole eco, every five-year-old device supports AV1. Then, we will be switching to AV1 100%.
The initial roll-out will probably just involve transcoding at sub-source quality settings to save bandwidth with no encoding to be done by the streamer.
Right now, platforms have only talked about AV1 for transcoding purposes, so there's not the same urgency to add encoding support on consumer-level GPUs. Even thought it'd be amazing to see AV1 streams and YouTube videos, it's mostly on the server-end to cut costs and make things easier on people with data plans and cellular.
You want low latency encoding. It doesn't matter if you can half the bitrate of Video but requires 3s delay to encode it. Something which as far as I am concern is done better with VVC.
And that's exactly why OP asked for GPU encoder, those are low-latency and if the resulting stream requires less bandwidth it means that there can be fewer packets on the network, meaning lower absolute re-transmits (the networks relative error rate is the same independent of the data transferred) and faster transmission for slower networks, i.e., in general also lower latency as by-product; latency, available bandwidth and packet amount are often coupled.
Low-latency encoder are not exactly known for producing bit-efficient streams.
Their purpose is to dump the frame into compliant stream in least possible time, so the compliant decoder can decode it back. It doesn't concern itself with using less bits; if it does, it is nice, but not deal breaker.
So if the hardware encoders can produce VP9/HEVC/AV1 at roughly the same bit-budget, it doesn't make sense to use the more complicated one. It makes sense to use the one, where you do have to pay less fees, though.
Some codecs might be more suitable for encoding video streams which are known ahead of time and utilize similarity of frames both before and ahead to maximize compression (multi pass encoding); they might also require larger buffer. With real time video calls you can't really do that. There is no codec-fits-all solution unfortunately.
In live video, latency from encoding is a limiting factor for what types of compression you can use. Its not like bandwidth is the sole limiting factor here.
I can still see compression artifacts, so it's clearly not using enough bandwidth. It could probably compress it better if it had hardware available, rather than needing to do real-time encoding in software (since it doesn't have a VP9 encoder either).
> If you mean AV1 specifically, do you think software encoders won't get anywhere near x264, which can do it in less than one core?
From what I understand, AV1’s encoding complexity is more like H.265 than H.264. It’d be more appropriate to look at the performance of x265 to see what might be possible.
Sofware decode is broadly comparable so should be about the same.
Later generations often have more tools, which can lead to bigger die size, but this can also lead to energy gains from those tools depending on content/usage.
See cisco's presentation about adding AV1 to Webex and how it could outperform H264 in this niche due to having specific tools for it.
In Netflix type content the grain emulation for example could make a big difference.
The people involved in AV1 had a software decode plan B that has worked better than most of them hoped.
If you combine that with encoding choice based on software decode impact, most big players have started with software rollouts, but still seen gains.
"You need hardware support for a codec" is often the same kind of delaying tactic as "renewables need storage". You do both in parallel, not enter an infinite loop where you don't do either because the other isn't complete yet.