2026-09-12 Image Compression of the Leica M8

The quirky camera with a square-root companding table sitting in a DOOM WAD

Everyone nowdays is all the rage for CCD cameras, from pocket shitcams from Konica, Canon, and Sony, to high-end CCD cameras like the Contax TVS Digital. Many have raged (snobbishly) that they have a filmic look, without the film tax attached to it. Despite my tone, i must admit that i participate in this debauchery myself.

Released in 2006, the Leica M8 was the company’s first foray into digital cameras. Built from the body of the Leica M7 and a Kodak KAF-10500 APS-H sensor, the camera was a hot mess but has since gotten a cult following for many people swearing by its color science which mimic Kodak’s now discontinued Kodachrome.

Within Leica’s marketing and Kodak’s documentation, the M8 was supposed to have 16-bit color, which would rival the top of the line cameras from firms like Canon (the 1D Mark III claims only 14-bit RAW image files). But just checking M8 files in applications like RawDigger reveals something weird.

The M8’s sensor readout is 14-bit (values 0 to 16383). The Leica M8 uses an analog-to-digital converter (ADC) system that Leica marketed as a 16-bit process, but the final DNG raw files store data in an 8-bit color. But why? We can see after we look into the histogram of a typical M8 RAW file.

The compressed RAW seems to be losing data, as more than half of its histogram data seems to be gone. Imagine buying a new camera for $4,795 ($7,942 adjusting for inflation) and finding out you’re getting half of the sensor output you are promised with. This always rattled me the wrong way, as i like the camera but its poor performance in low-light photography has always rubbed me the wrong way.

Its Always DOOM

Behind the KAF-10500 developed by Kodak for the Leica M8, there is the Analog Devices ADSP-BF561 Blackfin Digital Signal Processor (DSP) that is responsible for analog-to-digital conversion of images from the sensor to the onboard Xilinx FPGA. The DSP sits directly on the digital signal processing board beneath the camera’s top cover.

The ADSP-BF561 is a fixed-point DSP lacking a native hardware floating-point unit (FPU), so my first assumption was that the square root had to be either some fixed-point iterative routine or a lookup table somewhere in the Blackfin Processor code, and that the whole reason the M8 is slow to write DNGs is that a fixed-point DSP is grinding through 10 million square roots per frame. i had a firmware file sitting in my downloads folder (m8-2_024.upd, the last firmware Leica ever released for it), so i figured i would just go look. Unfortunately, I was wrong about basically all of it.

The first four bytes of the .upd are PWAD, and the file command on macOS tells you that the Leica M8 firmware file is a “doom patch PWAD data containing 8 lumps”. Whoever wrote Leica’s updater in the mid 2000s needed a container format with a directory of named blobs, and id Software had already published one in 1993, so they used it.

$ file m8-2_024.upd
m8-2_024.upd: doom patch PWAD data containing 8 lumps

$ xxd m8-2_024.upd | head -4
00000000: 5057 4144 0800 0000 0c00 0000 8c00 0000  PWAD............
00000010: dc0b 0000 5255 4c45 5300 524f 680c 0000  ....RULES.RO h...
00000020: a5cd 0200 4c55 5453 004f 5250 10da 0200  ....LUTS.ORP....
00000030: b01b 0d00 5058 4100 4752 4f55 c0f5 0f00  ....PXA.GROU....

So the M8 is not a Blackfin camera with some glue around it. It is a PXA270 camera. That is the same ARM core that was in every Windows Mobile phone and every Palm of that era, and it runs eCos with a RedBoot bootloader (the strings eCos : and \REDBOOT.BIN are right there). The Blackfin is a coprocessor on the sensor board, and the analog-to-digital conversion itself happens in an analog front end chip before either of them sees a pixel.¹

The PXA lump is where the interesting stuff lives, and it is a flat ARM binary that loads at 0x20000. You can tell because it starts with an ARM exception vector table, eight copies of ldr pc, [pc, #0x18], followed by the absolute addresses those jumps go to, and they are all 0x0002xxxx.

If the camera uses a lookup table, the table has to be in the firmware somewhere. It would look something like “16,384 bytes where byte x is round(2·√x)” which i thought shouldn’t be very hard to look for. So I did something along the lines of :

import math
d = open("m8-2_024.upd", "rb").read()
lut = bytes(min(255, int(2 * math.sqrt(x) + 0.5)) for x in range(16384))
print(hex(d.find(lut[4096:4160])))        # -> 0xf03a0
print(d[0xef3a0:0xef3a0 + 16384] == lut)  # -> True

Inside the Firmware

Knowing where the table is, radare2 can find the code that touches it. There are exactly three places in the ARM image that load the address 0xE1990 from a literal pool, and two that load 0xE1790.

The first two are a constructor. It sets a vtable, then stores the table address into the object at offset 0xb4:

When the mode flag is 1 it copies the whole table out of flash into 0x5D000000 and repoints the object at the copy. That address is not in the PXA270’s physical memory map, so it is something the MMU has mapped, and the obvious candidate is the chip’s 256 KB of internal SRAM.² This is the reason why the camera operates reasonably fast.

Then, right after it is the entire compression algorithm. The function loads a 16-bit pixel, use it as an index into the table, store a byte. It is unrolled sixteen times, twice, with pld cache prefetches on both buffers every 32 bytes, which is the kind of thing you only bother writing if you have read the XScale optimisation guide.

In C it is nothing:

static const uint8_t lut[16384];            /* round(2*sqrt(x)), shipped in flash */

void compand(uint8_t *dst, const uint16_t *src, size_t n)
{
    for (size_t i = 0; i < n; i++)
        dst[i] = lut[src[i]];
}

From this we can start to understand the design limitations that Leica and its partners was trying to work with. Ten million table lookups on a 520 MHz ARM with the table sitting in the on-chip SRAM is equals to a few tens of milliseconds. So the bottleneck was never the internals of the M8, but the SD cards that were available at the time.

For completeness, the inverse also exists, at 0x784b4. It walks a buffer backwards, reads a byte, doubles it to make a short index, and looks up the 16-bit value, in place. That is what the camera uses to get a DNG back to linear when you review it on the back screen.

The cmp ip, 1 below is checking a flag at offset 0xb8 in the same object. The DNG writer checks the same flag. Around 0x766f4 there is a function that emits the TIFF IFD tag by tag, and this is the part that handles the linearization.

0x000766e8   ldr r3, [r6, 0xb8]        ; mode flag
0x000766ec   cmp r3, r4                ; == 1 ?
0x000766f0   beq 0x76780               ; yes: go write the table
...

0x00076780   mov r1, 0xc600
0x00076784   add r1, r1, 0x18          ; tag 0xC618 = LinearizationTable
0x0007678c   mov r2, 3                 ; type SHORT
0x00076790   mov r3, 0x100             ; count 256
0x00076798   bl  write_tag
0x000767a0   ldr r1, [0x000767b8]      ; 0xE1790, the inverse table
0x000767a4   mov r2, 0x200             ; 512 bytes
0x000767a8   bl  write_bytes

Tag 0xC618 is LinearizationTable in the DNG spec.³ When the flag is 1 the camera compands the pixels and writes the 256-entry inverse into the file so Lightroom can undo it. When the flag is anything else it does neither, and you get 16-bit samples with no table.

That second path is the “uncompressed RAW” that many users found in the Leica M8 Debug Menu thats accessible using a specific button sequence (right ×4, left ×3, right ×1, SET, from a powered-on body). The code path was always there, sitting behind a flag the service menu happens to toggle. This debug mode however does not survive a power cycle because the state isn’t saved to the ROM itself.

Behind the Mathematics

Now that we know exactly what the camera does, the math for why it is not stupid is short. Call the linear 14-bit sample $x \in [0, 16383]$ and the stored byte $y \in [0, 255]$. The table is

and the factor of 2 is only there so that $2\sqrt{16383} \approx 255.99$ lands on the top of a byte.

Photons hitting a CCD are a Poisson process, so a pixel that collected $x$ units of signal has shot noise with standard deviation $\sqrt{x}$. Now look at how far apart adjacent stored codes are in linear units. Differentiate $x = y^2/4$:

Within the highlights the codes are 64 levels apart, and the sensor could not have told those 64 levels apart anyway because the noise is bigger than that. In the shadows the codes are 1 or 2 levels apart, which is where the sensor actually can. If Leica had just dropped the low six bits the step would be 64 everywhere, which is fine in the sky and terrible in the shadows. This is known as the Anscombe transform, minus a small offset.⁴

Conclusion

So Leica didn’t exactly lie in its marketing through a technicality as the 8-bit DNG is not half the sensor thrown away. Rather with this, the images are re-quantised so that every step is one noise-width wide, and for a normally exposed frame you genuinely cannot see the difference (especially in low ISO images). But if you miss your exposure by a few shots or try to push the camera’s ISO for use in low-light environments, even DNG files won’t be able to save you in post.

As postulated before, this engineering design was likely made due to the limitation of portable storage mediums at the time. While i compared Leica’s claim of 16-bit images with Canon’s claim of 14-bit images on the 1DX Mark III (released around the same time in 2007), the 1DX uses Type I or II CompactFlash cards instead of regular SD cards.

The CompactFlash Association adopted the CF 4.0 specification in 2007, which introduced UDMA 133 (Ultra Direct Memory Access). Top-tier professional cards were rated around 266x to 300x multipliers, translating to peak read/write speeds of roughly 35 MB/s to 45 MB/s. Meanwhile, Traditional SD and the newly emerging SDHC (Secure Digital High Capacity) standard (introduced in 2006/2007). A Class 6 SD card guaranteed a minimum sustained write speed of 6 MB/s, though premium high-speed consumer cards pushed actual read/write performance closer to 12 MB/s to 20 MB/s.

The Leica M8 followed the size and form factor of the M7, putting a CF slot would be cumbersome and likely required bulkier internals and batteries that will make the camera significantly larger. I personally think it was the right decision, as Leica snobs are not really well known to care for specs anyways.

In conclusion, I think this limitation adds to the quirk of the M8. Real Kodachrome, being slide film, also requires very accurate metering and are rarely offered at high ISO speeds. So I guess its part of the gimmick.