The number is an index, not a measurement
Set a JPEG encoder to 80 and you have not asked for a picture that is eighty per cent as good as the original. You have selected a row in that encoder’s quantisation tables — a set of divisors applied to frequency coefficients before they are rounded. Set a WebP encoder to 80 and you have selected something else entirely: a target for a rate-control loop inside a codec built on a different transform, different prediction and different tuning.
The two numbers share a scale the way two countries share a currency symbol. They are both written from 0 to 100 because that is what the first implementations exposed, and that is the whole of the relationship.
This matters because almost every published comparison of image formats is built on it. Encode a corpus at quality 80 in two encoders, plot the file sizes, and you have measured which encoder is more conservative at the number 80. You have not measured which format stores a picture more efficiently, and the two results can point in opposite directions.
What this bench does instead
It fixes the appearance and lets the setting fall where it must. For each lossy candidate it runs a bisection search over that encoder’s own parameter, encoding, decoding what it encoded, and measuring the result against your source. It keeps the lowest setting whose result still clears the threshold you selected, and reports both the setting it landed on and the score it achieved.
The steps, in order
- Decode your file once, with any EXIF orientation already applied.
- Build a reference copy at 512 pixels on the long edge, so the cost of scoring does not grow with the size of your picture.
- For each lossy candidate, encode at the ceiling and measure. If even that misses the target, stop and report honestly what the closest setting achieved.
- Otherwise bisect downwards, keeping every setting that clears the target and discarding every one that does not, for up to seven encodes.
- Report the byte count of the file the search produced, the setting, the measured score, and how long the final pass took on this device.
Lossless candidates skip the search, because there is no parameter to search. They are still measured, and the score is still printed, because “lossless” is a claim about an encoder and the number is a fact about a file.
What the measurement is
Multi-scale structural similarity. It compares local means, variances and covariances between the two pictures over small overlapping windows, at several scales, and combines the results into one number where 1 means identical by that definition. This bench computes it on the brightness channel with a smaller colour term added, in a worker thread, on the 512-pixel reference.
It is a better proxy than the alternatives that are cheap enough to run seven times per format in a browser. Mean squared error, the obvious choice, treats a uniform brightness shift and a burst of blocking as the same magnitude of error, which is not how anyone looks at a picture. Structural similarity at least asks whether the local structure survived.
Where it misleads
It is a statistic, and it has the failure modes of a statistic. Fine film grain reads as structure, so an encoder that smooths grain away can score worse than one that replaces it with noise of its own, even where most people prefer the smoothing. Hard synthetic edges — the edge of a button, text on a screenshot — sit at the scale where the windows are least forgiving, so a small amount of ringing costs more score than it costs a viewer at normal size.
And it says nothing about colour shifts that are uniform across a region, which is exactly where chroma subsampling does its damage. The chroma term in this implementation catches some of that and not all of it.
That is why the two panes are the larger half of this page rather than an illustration beneath it. The number decides which candidate is recommended; your eyes decide whether to accept the recommendation, and the divider, the 800 per cent zoom and the held space bar exist so that decision takes seconds rather than a download and a round trip through some other program.
A note on what would be better
Metrics that model human vision more closely exist and are used in the codec research this site’s formats came out of. They are also, generally, far too expensive to run seven times per format inside a browser tab on a phone, and several are not available in a form that could be shipped here at all.
So the honest position is the one stated on the bench itself: this compares encoders by measurement, the measurement is a model, and the model can be a little generous to one format or another on some pictures. If a stronger metric becomes practical here, the number on every card will change and this page will say so.