Frustrating to hear it called bizarre. Arithmetic encoding should've been the default if IBM hadn't sat on a patent for mathematics. A huge amount of bandwidth has been wasted sending images around the world because Huffman encoding avoided legal headaches.
Be that as it may, but I can also feel the pain of implementing an old-as-balls image standard which has lots of historical options which account for, let's say, 70% of the effort of implementing the full standard, but are only used in .5% of the images available "in the wild". If these parts are then left out of implementations, it's regrettable from an academic point of view, but still understandable...
AC was way slower than Huffman back when JPEG was first defined in 1992, so it would have taken many years to see AC-encoded JPEG in the wild even without a patent concern.
> if you see a white box, it’s because you don’t support the bizzare Arithmetic coding extention
On an Intel Mac running Big Sur, that image (title "444-fp-acode-prog.jpg") is only solid white in Chrome; in Safari, it's solid black and in Firefox it doesn't load at all, there's the broken image placeholder.
Core 2 Duos predate Intel adding any hardware decoding for JPEG. Some, but probably not all, of the rendering failures people are seeing may be due to software handing off JPEG decoding to hardware but some hardware not being able to handle the Arithmetic coding extension.
My guess is your JPEG rendering is being done by libjpeg, which has supported Arithmetic coding since 2009.
My guess why Intel added hardware JPEG decoding is HD webcams. Most of those do MJPEG, so with a hardware JPEG decoder and a hardware h.264 encoder videoconferencing can almost completely avoid the CPU. Most external higher-res webcams also only do YUV (for low fps / resolution, as it quickly exceeds USB 2.0 bandwidth) or MJPEG, some newer ones do h.264 though. Incidentally, all those cheap HDMI USB capture sticks (I think most have the MS2109 in them) also use MJPEG.
I have never seen a jpeg decode fail until the above website fed it some “obscure” format. The fact it works or doesn’t based on hardware is pretty crazy. If I didn’t know that hardware was responsible for decoding the jpeg it would have been one hell of a bug to diagnose.
Since 2019, iPads no longer run iOS, they run iPadOS. Of course, there's still a lot of overlap between the two but differences can creep in. A difference that shouldn't be relevant to JPEG rendering is iPadOS Safari loads the desktop versions of websites by default.
iPhone SoCs have had hardware JPEG decoding since basically forever. However, I'd be surprised if that hardware block suddenly got updated to support AC.
If I had to guess, A14 devices aren't using the hardware decoder any more for some reason, and that's why it works.
> that there is actually no consistent way to decode a H.264 video stream. Don’t ever expect two decoders (especially hardware assisted) to give you identical outputs.
I'd really like to see a citation for the claim there is "no consistent way to decode a H.264 video stream."
H.264 decoding should be fully deterministic and consistent, I believe. That said, bugs exist, shortcuts are taken, and not all decoders are truly compliant. But "some decoders are non-compliant" is very different from "it's impossible to consistently decode H.264".
Yeah, having done some work on this in the past I think both are true: the standard should decode deterministically, but it's also quite complicated, so most of them are buggy.
(A colleague of mine filed something like 30% of all the bugs on the H265 bug tracker, for instances where the reference implementation produced different results than the textual specification. Because we'd written a parser for the English text ..)
Whenever I hear things like this I’m surprised there’s no set of reference videos & reference decoded frames. Like: “this video file should decode into this uncompressed video. This video file should decode into this other uncompressed video” - etc, with a large combinations of encoder features. Like a video equivalent of the commonmark test suite[1]
I was playing with email parsers a couple of years ago and I still don’t know if my cobbled together email parser works correctly with all emails. 40 years after RFC822 it seems there’s still no standard, reference set of email message test cases to process to verify an email parser is correct & robust.
Back in the day when I was involved with video codec fw, I remember we were having trouble with some codec's (VP8?) deblocking pass, We had to implement that in SW because the particular method that codec used wasn't possible with our HW.
A very legitimate question we asked ourselves was whether we could skip it altogether: the pass did smooth out edges between MacroBlocks, but on our test streams you really had to look closely at still frames to see the difference.
So I can imagine different codecs implementing different parts of standard, as long as you could produce a valid stream/image that was acceptable to the human eye.
(I don't remember what the final decision was for our deblocking. I would guess that certified compliance required it to be implemented)
Any H.264 decoder that doesn’t produce bit identical output is non-conformant to the spec; this was one of the big lessons of MPEG4 part 2, that you really don’t want the drift that comes from repeatedly using “near enough” reconstruction as prediction in a loop.
Well, he could be including YUV -> RGB conversion, which while the conversion matrix is specified, the required precision and chroma upsampling method are outside the scope of the spec.
Guess what: in general, no two CPU floating-point architectures produce exactly same results.
EDIT: Also, try converting JPEGs to uncompressed PNG using Python with Pillow or OpenCV, then try on C using libjpeg and the stb_image header library. All four create slightly different output. I discovered this evaluating the accuracy of a MobileNet implementation. The accuracy of the model changed based on what language/library converted the JPEGs.
For the JPEG part: That is not surprising, for several reasons:
- Depending on if the IDCT (2D cosine-based frequency domain => time domain) is implemented using floating point or fixed-point, the ways roundings are done, 0.5 offsets or not etc, the number of bits of precisions, you will have small errors.
- Most JPEGs are in i420 format, that is "for a 16x16 Y block, there is one 8x8 Cb block and one 8x8 Cr block". The upsampling algorithm may differ both in precision and on the choice of the algorithm itself.
- Even in YCbCr => RGB conversion, there may be 1-bit differences. More if they used other constants.
Note that even between two versions of the same library, you can get different results.
Denormals are part of IEEE 754, but in practice are more complicated for hardware to figure out. So as a common efficiency trick, you disable the processing of denormals entirely (IE: all denormals are seen as "0" to the hardware, speeding up processing severely).
On Intel processors, a configurable register can set up how your floating-point units act when they come across a denormal. So an inline-assembly statement can change your results, even if it was no where "close" to your code.
Do you know what are typically circumstances under which this becomes problematic? Clearly these issues don't impact FPS games, or music quality, but do they start becoming problems in weather forecasting or particle simulations? What kind of applications are impacted?
This stuff happens in pretty much every video game. You "solve" it by...
1. In FPS games, most _clients_ serve as the source of truth. This means that in most FPS games, you allow the client to "hack" the location data for clipping, speed-hacks, and the like. (Server calculated you at point X. Client claims to be at point X+.05. Well, client wins the argument).
2. In games like Factorio, an extraordinary amount of effort is placed upon finding all sources of desync (https://wiki.factorio.com/Desynchronization) and squashing them. Protocols communicate with integers as much as possible, avoiding any floating point issues.
> but do they start becoming problems in weather forecasting or particle simulations? What kind of applications are impacted?
I'm no supercomputer expert. But for reliability, a lot of supercomputer simulations step through twice to "verify" the results. That means both simulations must be setup with the same settings, so that your floats all line up.
In my undergraduate studies, my professor for numeric computing stressed the importance of sorting numbers: not only to avoid floating-point cancellation error, but also to have a precisely defined ordering for supercomputer applications.
You want to do things like add / subtract numbers from smallest to biggest, and all the positive numbers add together, then all the negative numbers add together and only do one subtraction. Cancellation error is serious business!!
The Factorio Friday Facts dev diaries are a great read, and they’ve talked about the effects of floating point numbers multiple times. Sometimes in reference to multiplayer, but even bugs in single player have been tracked back to floating point numbers: https://www.factorio.com/blog/post/fff-297
> I'm no supercomputer expert. But for reliability, a lot of supercomputer simulations step through twice to "verify" the results. That means both simulations must be setup with the same settings, so that your floats all line up.
In such cases, the result of one simulation is not enough, and we do several ones with slightly different initial conditions, just to make sure that we don’t spend time analysing an outlier. Exact reproducibility is important to validate the codes running test suites, but for the purpose of using simulations to make predictions, it does not really matter as long as each simulation is physically accurate enough.
AFAIK IEEE 754 does allow for some variation in the results of operations like log and exp. I recall this being an issue with the LHC@Home BOINC project, where simulations computed on Intels would run for a long time before reaching the termination condition (beam hitting wall) while when computed on AMDs would terminate quickly.
For a project relying on multiple runs on different computers for verification of correctness, this was a serious issue. IIRC before they solved it properly, they had to increase the number of validations per piece of work, as well as trying to segregate the work so a work unit was only issued to either Intel or AMD but not both. This only reduced the impact somewhat though.
From their[1] page:
Besides helping build the LHC, LHC@home has proved to be an invaluable, even essential, tool for investigating and solving fundamental problems with floating-point arithmetic. Different processors, even of the same brand, all conforming more or less to the IEE754 standard, produce different results, a problem further compounded by the different libraries provided for Linux and Windows. In particular, a major problem was the evaluation of the elementary functions, such as exponent and logarithm, which are not covered by the standard at all (apart from the square root function). This problem was solved by the use of the correctly rounded elementary function library (crlibm) developed by a group at the Ecole Nationale Supérieure in Lyon. Due to the often chaotic motion of the particles even a one unit of difference in the last digit of a binary number produces completely different results, as the difference grows exponentially as the particle circulates for up to one million turns round the LHC. Thus it is extremely difficult to find bugs or to verify new software and hardware platforms. This methodology has allowed us to detect failing hardware, otherwise unnoticed, and to find compiler bugs. With your help, it is being extended to ensure identical results on any IEE754 compliant processor with a standard compliant compiler (C++ or FORTRAN) which will allow us to work on any such system and even on Graphical Processing Units (GPUs).
As long as the you use only the four elementary arithmetic operations in the same order with the same types, and use compiler flags that disables any optimisations, then you should get the same result. (knock on wood)
Different standard libraries for different processors could have different precision for transcendental functions though.
There are those that are written to always produce reproducible results though; and in the case of CRlibm, proven correctly rounded.
Older compilers for x86 used the x87 FPU which used a 80-bit type internally but all modern compilers for x86 and x86-64 use SSE/AVX instead which supports IEEE types.
> Optimizations in FMACs can lead to LSB differences. IEEE doesn't specify how an FMAC is to be implemented, only the format of the number.
Quoth the spec:
> Each of the computational operations that return a numeric result specified by this standard shall be performed as if it first produced an intermediate result correct to infinite precision and with unbounded range, and then rounded that intermediate result, if necessary, to fit in the destination’s format (see 4 and 7).
Not really. There’s a lot of post processing going on to make up for the lossy part of the compression. It’s actually quite wonderful that the technology and methods can improve over time.
I work for a company that stores and displays images for customers, and converts them to various formats for them... CMYK JPEGs are the bane of my existence. A recent annoyance was a realization that they don't render consistently in different browsers, so some work I put into making sure that CMYK JPEGs always get CMYK JPEG thumbnails to keep colors consistent was basically a waste of time because the end-user can still see inconsistent color previews.
In the end the idea of a CMYK JPEG makes no sense to me, because if you're generating content specifically to print, especially if it contains vector-like content, why are you storing it in a lossy format? But, whatever, that is less annoying than people who ask for the ability to modify the DPI of an image to 72 for "web exports". I built it in instead of trying to explain to them that there's no need to turn a 300 dpi JPG into a 72 dpi JPG for use on social media.
I don't even want to get into the joy of CMYK SVGs with transparency, and trying to explain to customers that since JPG doesn't support transparency I have to use PNG for generated preview images, which necessitate a conversion to PNG, and SVG doesn't have any sort of color profile support, so converting them from CMYK to RGB is kind of just lucky dip.
But in general when you're dealing with user generated content you see a lot of weird stuff... we get a lot of JPG uploads that are actually MP4 video files, and sometimes the reverse, MP4 uploads that are actually still frame JPEGs.
Frustrating to hear it called bizarre. Arithmetic encoding should've been the default if IBM hadn't sat on a patent for mathematics. A huge amount of bandwidth has been wasted sending images around the world because Huffman encoding avoided legal headaches.
https://en.wikipedia.org/wiki/Arithmetic_coding#History_and_...
This is also the reason for the 2 in bzip2.