The past, present and future of image formats

maskros 2021-08-19 11:14:19 +0000 UTC [ - ]

"Ultimately is was probably patents and the associated license costs that prevented JP2 from going mainstream, a lesson which would later be learnt."

Not even remotely the reason. Lack of high quality decoders and absolutely terrible performance are what kept it back.

The format is ridiculously complicated (design by committee meant it had to include every single bad idea everyone could ever come up with) and there were no good open source implementations. Even the best commercial implementations have decoding performance that is orders of magnitude slower than JPEG. I don't think it's possible to decode JPEG2000 with good performance even if you tried.

JPEG2000 is plain and simple a terrible format. Adobe shoe-horned JPEG200 into PDF, and tried to make it common, so all PDF viewers have to support it. Nobody with half a brain likes to use JPEG2000 images in PDF though, since all files with embedded JPEG2000 images are ten times slower to open and use ten times as much memory as the equivalent PDF with JPEG images would.

acdha 2021-08-19 12:17:12 +0000 UTC [ - ]

I think you’re arguing the same cause from different angles: JPEG2000 had no good implementations prior to OpenJPEG because the spec required a license to read and there wasn’t even a compliance suite for the first 15 years or so.

I think the quote is also more correct than you give it credit for because IP concerns kept it out of browsers. I don’t think you can get traction for a new image format without browser support — _maybe_ that could be a robust WASM polyfill now but I really think anyone making a new format needs to focus how to get over that hump.

Performance is complicated: it’s good enough to be used in digital cinema systems but it’s definitely trading CPU for size, just as we see in the modern video codecs which are also orders of magnitude slower than JPEG. The one area where it can be faster is progressive decoding when you don’t want the entire image resolution - I work with systems which serve lots of thumbnails & similar lower-resolution derivatives and this is the main selling point for the format. I’m still not convinced it’s worth the hassle but I do look for it in other formats.

ZeroGravitas 2021-08-19 12:22:45 +0000 UTC [ - ]

I don't disagree, but on the other hand, the web has taken many horrible standards, made them ubiquitous and used their collective intelligence and spare time to push them to places that no one expected.

This ironically demonstrates the absurd nature of patents in networked contexts, because not having patents made those things more valuable (to the world at large) since most of the value comes from the adopters and the ecosystem, not the specific patented spec.

JP2 would have had some niches that it could have grown from, scanned books for example.

pornel 2021-08-19 14:33:45 +0000 UTC [ - ]

In the end, JPEG2000 is not even good at compressing images.

It uses wavelets which are overfitting the PSNR metric. In purely numeric benchmarks (without humans judging the images) it makes JPEG2000 seem way better than it really is. Wavelets may look impressive when you compress an image to blurry death, but that's a niche use-case. It falls apart in the more relevant high and nearly-lossless quality ranges. It handles sharp edges very poorly, and struggles to preserve textures of things like skin or wood — either everything looks like plastic, or you get poor compression.

DCT-based codecs can generate realistically-looking textures with fewer bits. For web uses even de-blocking filters of WebP and AVIF are not that useful, because people want to use qualities at which blocking isn't a problem in the first place.

veltas 2021-08-19 12:12:49 +0000 UTC [ - ]

> Lack of high quality decoders and absolutely terrible performance are what kept it back.

Isn't that also a symptom of lack of adoption?

zokier 2021-08-19 10:29:55 +0000 UTC [ - ]

> HEIC, based on H.265 video, means it's encumbered with patents (moreso then anything before it). So it has no support outside of Apple.

Notably Canon and Sony support HEIC natively in their cameras. It makes probably lot of sense to them, as they need h265 encoders anyways for video capture, so they get HEIC pretty much for free.

Also Android supports HEIC these days.

Together with Apple also supporting HEIC, it pretty much guarantees that HEIC is here to stay.

Meanwhile I haven't heard any camera manufacturer to support either AVIF or JPEG-XL. Even Android doesn't support AVIF yet. So outside web bubble, it's AVIF that has little to no support.

Youden 2021-08-19 11:01:45 +0000 UTC [ - ]

> Also Android supports HEIC these days.

The situation isn't quite so simple. Android devices are only required to support HEIC if they also support HEVC decoding [0]. Interestingly, this seems to be less strict than the Android 10 CDD [1].

[0]: https://source.android.com/compatibility/11/android-11-cdd#5...

[1]: https://source.android.com/compatibility/10/android-10-cdd#5...

2021-08-19 15:41:44 +0000 UTC [ - ]

ksec 2021-08-19 15:37:20 +0000 UTC [ - ]

>So outside web bubble

Yes. A lot of the conversation talk about image and video as if Web is the only platform. Just like how AV1 is suppose to be everything because of Youtube while completely neglecting the rest of the video industry from Broadcasting to Movies etc.

skyfaller 2021-08-19 12:52:02 +0000 UTC [ - ]

If JPEG XL actually delivers the ability to serve multiple screen sizes / resolutions from the same image as proposed in FUIF: https://cloudinary.com/blog/introducing_fuif_responsive_imag...

instead of requiring storing many copies of the same image for responsive design, that would be a godsend and would immediately convert me to JPEG XL for graphics on websites. I don’t want the complication of generating, storing, and serving the right image to the right client, there should be one image and clients should only download the amount they want.

Santosh83 2021-08-19 14:13:10 +0000 UTC [ - ]

But this still doesn't take away your need as webdev to specify the exact sizes for different screen/container dimensions and aspect ratios. The only difference is you don't have to generate several images and clutter your directories.

edflsafoiewq 2021-08-19 13:21:42 +0000 UTC [ - ]

Couldn't JPEG2000 do that? I think it's more a question of if browsers will actually implement it.

oenetan 2021-08-18 12:19:26 +0000 UTC [ - ]

JPEG XL is easier to encode and make than AVIF in my opinion.

For more info on jpegxl, go to https://jpegxl.info/

ksec 2021-08-19 17:01:17 +0000 UTC [ - ]

> To do these tests I took this photo taken by Alessio Soggetti. I then down-scaled and cropped it. I started at 0 quality for all encoders and worked my way up in 10 step increments. I tried to aim for a little over 100KB. I did bump JPEG XL a few steps (50 to 53) to keep it competitive with AVIF (JXL came in at 100KB at q50 while AVIF came in at 110KB at q40).

Correct me if I am wrong on the calculation.

The down-scaled PNG is showing 5 images at 6106 x 1200. At 100KB per one image would be at bpp of ~0.55. So despite the favourable results, I need to state again JPEG XL isn't yet optimised for low bpp yet ( bpp below 1 ).

Reposting My previous comment ( https://news.ycombinator.com/item?id=27577701 ) on JPEG XL from a few months ago.

Comment from Facebook Infra [1],

>Erik Andre from the Images Infra team at Facebook here. I'd like to share a bit of our view on JPEG XL in the context of new image formats (e.g AVIF, JPEG XL, WEBP2, ...) and how browser adoption will let us move forward with our plans to test and hopefully roll out JPEG XL.*

>After spending the last 5 months investigating and evaluating JPEG XL from both a performance and quality point of view, it's our opinion that JPEG XL has the most potential of the new generation of image formats that are trying to succeed JPEG.

>This opinion is based on the following findings:

>JPEG XL encoding at speed/effort 6 is as fast as JPEG encoding (using MozJpeg with Trellis encoding enabled). This means that it's practical to encode JPEG XL images on the fly and serve to client. This can be compared with the encoding speed of AVIF which necessitates offline encoding which offers much less flexibility when it comes to delivering dynamically sized and compressed content. Depending on the settings used, JPEG XL can also be very fast to decode. Our mobile benchmarks show that we can reach parity with JPEG when using multiple threads to decode. This matches and in many cases surpasses the decoding performance of other new image formats. The JPEG XL image format supports progressive decoding, offering similar gains in perceived image load performance we are already benefitting from with JPEG. This is a feature lacking in the other new image formats which are all derived from Video codecs where such features makes little sense to support.

>Having browser support from all the major browsers is going to make our lives a lot easier in upcoming WWW experiments and ensure that we can deliver a consistent experience across platforms and browsers.

Blink Tracking Bug [2] currently behind a flag ,Firefox on both [1] and [3], currently in about:preferences#experimental on Firefox Nightly. If I remember correctly it is supported on Edge behind a parameter as well. I thought it was all very quiet after the standard was published, turns out both Chrome and Firefox intent to support it.

AFAIK, neither Webkit nor Safari has any plan or intention to support JPEGXL. I think ( someone correct me if I am wrong ) Safari uses macOS image decoding library. So supporting JPEGXL may come from an OS update and not browser?

Finally, an open standard, Royalty Free, open-source reference implementation, and it is nearly better than all other alternative. As an image format on the web it is quite possibly close to perfect [4]. It is exciting and I hope JPEG XL will succeed.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1539075

[2] https://bugs.chromium.org/p/chromium/issues/detail?id=117805...

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1707590

[4] I remember the conversion from a little more than 6 months ago current encoder is not optimised for image quality below bpp 1.0, those are going to be the focus once all the initial reference encoder and decoder standards and issues are done. So in case anyone wondering it doesn't look as good as other competitors ( but still a lot better than JPEG ), those improvements are coming later.

imaginariet 2021-08-19 10:10:11 +0000 UTC [ - ]

From a linked article:

> It would be great if web browsers would accept in an <img> tag all the video codecs they can play in a <video> tag, the only difference being that in an <img> tag, the video is autoplayed, muted, and looped. That way, new and masterful video codecs like VP9 and AV1 would automatically work for animations, and we can finally get rid of the ancient GIF format.

Of course, the typical rebuttal on HN will be "I hate autoplay, nothing should autoplay without an explicit click", which is why we'll be stuck with GIF for another 10 years.

gardaani 2021-08-19 10:48:56 +0000 UTC [ - ]

Safari already accepts MP4 in an <img> tag and it is autoplayed, muted and looped [1]. Sadly, Chrome doesn't want to implement it [2].

[1] https://developer.apple.com/documentation/webkit/delivering_...

[2] https://bugs.chromium.org/p/chromium/issues/detail?id=791658...

acdha 2021-08-19 12:27:40 +0000 UTC [ - ]

This is a great example for the “Chrome is the new IE” debate: that’s a feature which would immediately make the web better but they held back support for a standard, hardware-accelerated format trying to force mainstream adoption of their own format which had limited tool support and never performed enough better than JPEG/PNG to be worth the trouble for most sites.

pwdisswordfish8 2021-08-19 13:56:34 +0000 UTC [ - ]

s/better/worse/

acdha 2021-08-19 15:16:14 +0000 UTC [ - ]

You think it's good that people keep using gigantic GIF files rather than smaller videos? There's no technical reason to prefer the GIF format — it burns data transfer, requires more CPU, and has terrible quality.

Zababa 2021-08-19 11:41:06 +0000 UTC [ - ]

> Of course, the typical rebuttal on HN will be "I hate autoplay, nothing should autoplay without an explicit click", which is why we'll be stuck with GIF for another 10 years.

Why? If you believe nothing should autoplay GIFs are also part of the problem.

rhdunn 2021-08-19 15:08:38 +0000 UTC [ - ]

I personally find animated gifs on things like YouTube community posts visually distracting, as my eyes will continually move to look at what is moving. That's also why I turn off things like the moving tile images on the Windows start menu.

chrismorgan 2021-08-19 12:30:01 +0000 UTC [ - ]

Absolutely; I consider GIFs autoplaying to be a bug—unfortunately a bug that would be rather difficult to fix.

idoubtit 2021-08-19 14:22:20 +0000 UTC [ - ]

It's easy to fix with the right browser. For instance with the Chromium-based Vivaldi, I configured "Image Animation" to "Once", with the other possible values being "Loop" and "Never". You may define a shortcut key to change this setting on the fly.

chrismorgan 2021-08-19 15:16:58 +0000 UTC [ - ]

That doesn’t fix undesired animated image autoplay, it breaks animated images completely, since you can’t trigger playback on demand like you can with videos.

magicalhippo 2021-08-19 10:30:29 +0000 UTC [ - ]

There are many valid reasons why autoplaying videos should be up to the user.

I'm generally against autoplaying, but I do accept that in some cases it is a nice thing.

So this proposal seems reasonable, if the browsers in addition get a separate option to disable this behavior.

It's important though that javascripts can't override this behavior to prevent abuse (say by unmuting), so browsers would have to take some precaution against that.

ThePadawan 2021-08-19 13:43:59 +0000 UTC [ - ]

I really can't understand why we now have both autoplaying video against the users' wishes and incredibly complicated configuration whether to allow autoplay [0].

[0] https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_...

hulitu 2021-08-19 15:18:30 +0000 UTC [ - ]

Because the ad industry pays for the browser.

vbezhenar 2021-08-19 11:50:10 +0000 UTC [ - ]

AFAIK Autoplay works fine in modern browser for video tags, just without sound. Enabling it for images does not alter web in any meaningful way IMO.

unwind 2021-08-19 11:58:33 +0000 UTC [ - ]

I hope most would be okay with it due to the forced (by the browser) muting, i.e. images may animate but they may not make sound, even if they are displaying what is in reality a video stream containing sound.

At least I would be fine with that, and I can be quite annoyed with auto-playing video.

2021-08-19 12:24:49 +0000 UTC [ - ]

Santosh83 2021-08-19 12:37:57 +0000 UTC [ - ]

Both JPEG XL and AVIF support is already there in Chrome and Firefox, and can be enabled with about:config or command-line flags. AVIF will become supported by default in the very next version of Firefox and is already supported by Chrome.

Edge mysteriously does not yet support AVIF despite building on the same Chromium base that does support it, which means the AVIF code is deliberately compiled out by MS for some reason.

Safari is a different kettle of fish. It supports neither format yet, but there seem to be vague indications on the issue tracker that one day they may land AVIF support. For that matter even Webp support is confined to only Safari on Big Sur and later, according to caniuse. It truly is shaping up to take over IE soon, as the browser for which one has to go to great lengths to code fallbacks.

IvanK_net 2021-08-19 12:11:50 +0000 UTC [ - ]

At the end, they have a photo example of various formats.

Here is the GIF version of their final image (249 kB) https://i.imgur.com/8yt7NS0.gif

kettleballroll 2021-08-19 12:00:19 +0000 UTC [ - ]

I think it would be interesting to have progressive decoding (and, for those who want it, autoplay) of videos, too.

ggm 2021-08-19 11:12:31 +0000 UTC [ - ]

No jxl in Android Chrome on Samsung tablet. Taking me back to early jpeg days when gifs worked and not much else.

londons_explore 2021-08-19 10:03:08 +0000 UTC [ - ]

With todays phones and network connections, downloading and rendering images actually constitutes only a small proportion of the total loading time.

Of that small proportion, the cpu time of rendering the images, relayout of the page for changed image dimensions, and re-rendering the image and all overlapping text and layers each time a bit more of it partially loads, is more substantial than the network delay of loading the data.

The TL;DR: is that optimising images isn't the place to start to make most websites load faster, especially on mobile.

Obviously if CPU speeds increase faster than network speeds, image size may matter again in the future. But today the size is mostly irrelevant as long as you aren't serving your photographs as multi-megapixel images or gifs.

cryptonym 2021-08-19 11:37:44 +0000 UTC [ - ]

A new format might come with improved decoding performance/memory usage with hardware support or improved algorithm. More than layout, the real issue around web performance usually is JavaScript / Third Parties. Browser is only a tiny part of potential benefits.

What if your business relies heavily on images and can save 30% of your bills on image storage and bandwidth? On your camera, maybe you will take more pictures without switching SD-Card...

imaginariet 2021-08-19 10:14:06 +0000 UTC [ - ]

For something like Instagram, where most images (and videos) are viewed only once, size is very important.

And way more images are viewed on Instagram than on mobile websites.

nyx-aiur 2021-08-19 10:53:26 +0000 UTC [ - ]

What an idiotic attitude from a guy who thinks the world is connected to 5G and everyone has an M1 Macbook.

q-rews 2021-08-19 13:00:10 +0000 UTC [ - ]

> a finely tuned JPEG encoder will do just as well as WebP

That clearly isn’t the case in any of the tests that I’ve seen so far. Without proof of these statements, this article is garbage and should be ignored.

pornel 2021-08-19 14:55:31 +0000 UTC [ - ]

People usually make a mistake of performing lossy re-encoding, seeing the file size drop, and concluding it's a success. That's meaningless for a format vs format comparison, because it doesn't control for equivalent quality. You can re-encode any lossy format, even back to itself, and always see significant file size reduction due to nearly-imperceptible quality loss (but that "nearly" matters a lot: see heap paradox).

Proper comparisons need to control for visual quality, but this very hard to do properly, because human range of "looks the same to me" is bigger than compression efficiency difference between WebP and JPEG. So you need either synthetic "objective" metrics that are not exactly human judgement (problematic, because e.g. PSNR likes blur more than people do), or test with lots of images and lots of people.

Anyway, WebP has also clear limitations compared to JPEG. For example 4:2:0 chroma subsampling of VP8 means it automatically loses whenever an image needs sharp colors, because it just can't do it, but JPEG can. VP8 also uses "studio range", so it has less than 8 bits of color precision (224 intensity levels instead of 255).

dusted 2021-08-19 10:32:40 +0000 UTC [ - ]

This too shall pass, whatever will eventually become the standard, will be lossless, as it became with audio. An 2 hour uncompressed 3840x2160 24 bit image stream at 120 fps is just shy of 20 TiB, which seems like a lot, it's more than some people have in their systems.. Let's say that 4 TiB is reasonably widespread.. Then that super extravagant movie clip is about 5 times the size of your average drive, what a waste!!!1one

well, sure, right now, but not in the future.. Point: in 1996 my harddrive had a formatted capacity just shy of 1 GiB of space Now my copy of the 1979 movie "Alien" takes up 16 GiB, or 16 times the size of my 1996 drive, which is even more extreme.

So sure, clever (lossy) image compression has its place in history, but it is going to fade, just as with audio, a lossless compression format will be the future.

kettleballroll 2021-08-19 11:59:43 +0000 UTC [ - ]

> whatever will eventually become the standard, will be lossless, as it became with audio

At least ony bubble, the standard for audio is still lossless: I'm fairly sure 99.9% of all audio i consume is using something lossy:streamed over Spotify, in movies like Netflix, clips like YouTube, video Chats... Is that just my impression and technology moved to FLAC without me noticing?

yitchelle 2021-08-19 10:37:15 +0000 UTC [ - ]

While the master should be lossless, for us who are just watching it, does it really have to be lossless?

I think that we have reached a point where it is fine to consumed media as it is now, personal view.

mchusma 2021-08-19 13:50:25 +0000 UTC [ - ]

I believe that AVIF and JPEGXL are going to be the last "traditional codecs".

But there had been good research on using neural nets for compression and decompression, and I expect the next codec to have massive improvements over these formats.

dusted 2021-08-19 10:42:42 +0000 UTC [ - ]

Need is defined as much by what's economical. Do we NEED flac files instead of 320kbit mp3's ? many people can't tell the difference, it's certainly "good enough" (tm), it's just that, at some point, it makes no economical difference, and then there's just no reason anymore to use a lossy format.

dangravell 2021-08-19 10:51:50 +0000 UTC [ - ]

That's not necessarily the reason people choose FLAC. Some people, myself included, simply feel it's "wrong" to throw data away. Remember you might need that data later for re-encoding or otherwise changing the audio.

hulitu 2021-08-19 15:27:46 +0000 UTC [ - ]

We do not have flacs or 320kbps mp3s. We have crappy 158kbps webm (youtube) which has the same audio quality like an 64 kbps mp3.

hulitu 2021-08-19 15:25:00 +0000 UTC [ - ]

Why not ? why do we have to watch crap even when the impact is so low ?

imaginariet 2021-08-19 12:35:14 +0000 UTC [ - ]

Even professional video production (movies, ...) shoot the raw material in lossy formats, albeit, with minimal compression.

Not even they can afford lossless capture. Because while storage capacities do increase, so does the used video size, bit depth and framerate, and number of takes.

jl6 2021-08-19 10:42:26 +0000 UTC [ - ]

I no longer believe storage capacity growth will follow an exponential increase curve like it did in previous decades. If you look at the last 5-10 years you’ll see far slower growth than in the decade prior to that.

So I believe we must still strive for a degree of storage efficiency.

blitzar 2021-08-19 12:02:23 +0000 UTC [ - ]

Is there an environmental argument (reduce waste, save the planet) to be made at some point? There are many, possibly exagerated, pieces about the energy savings from something as simple as 'dark-mode'. These small changes if scaled to billions of users and hours represent a meaningful impact.

edflsafoiewq 2021-08-19 12:33:32 +0000 UTC [ - ]

Lossless video is a ridiculous idea, but video encoding suffers hard from Jevon's paradox. The demand for video in pixel-hours explodes way faster than coding efficiency increases, and increasing efficiency expands the market even further.

To put it differently, you'll never optimize your way ahead of ten million people watching cat videos.

nyx-aiur 2021-08-19 10:49:19 +0000 UTC [ - ]

If you think the past is an indicator for the future I got some bad news for you.