2026-10-16 Decrypting FortiOS 8.0.x Firmware Image
The materials here were partly given out in my talk “Cracking Hard(ware) : The World of Network Appliance Forensics” at STANDCON 2026 in Singapore, this is a partial writeup for those who missed it.
The materials here were partly given out in my talk “Cracking Hard(ware) : The World of Network Appliance Forensics” at STANDCON 2026 in Singapore. This is a partial (but more focused and indepth, i think, on the important stuff) writeup for those who missed it. I will be mainly focusing on my work in the Fortigate FortiOS appliance as its one of the few parts of the presentation that alot of audience members had found to be interesting and novel enough to current industry applications.
Decrypting a firmware image’s root filesystem is usually the first move in a white-box audit of an embedded device, and thats no different with Fortigate devices which are especially well researched as they are extensively used in enterprise environments and are frequent targets to APT hacks. Bishop Fox and Optistream had done early works in decoding the FortiOS firmware, and in April 2024 GreyNoise Labs documented a way to decrypt FortiOS 7.0.x. There the firmware used stock ChaCha20, and the whole cipher state (a 32-byte key and 16-byte IV) sat in the clear in the kernel (flatkc). You could recover it with lief to resolve a symbol offset and objdump to dump the static memory.
The newer FortiOS 8.0.x seems to introduce stronger protections against firmware decryption, as searching for the usual ChaCha20 fingerprint (the constant "expand 32-byte k" (0x61707865 0x3320646e …)) turns up nothing. The secure-boot and integrity path of the OS has seemingly been rebuilt around a root of trust
- The symmetric key is no longer stored in the clear, it is RSA-wrapped and appended to
rootfs.gz, and the public key used to unwrap it is itself obfuscated in the kernel - ChaCha20 and AES were dropped for a customized RC4 variant whose keystream matches no reference implementation
The verifier sub_FFFFFFFF81710483() delivers the 32-byte disk key as a 256-byte block appended to the end of rootfs.gz. Since the image can only be verified with the public exponent, that block was produced with Fortinet’s private key. This is RSA signing with message recovery, not encryption. The decryption core sub_FFFFFFFF81710334() initializes a standard 256-byte S-box but passes the output through an obfuscated keystream function. The customization lives entirely in the output stage.
However, on 8.0.x the symbols are gone. The reliable pivot is the fail-closed invariant of secure-boot code: if verification fails, the device must brick. So the verifier sits one or two hops up the call graph from a halt. Search the kernel image in IDA for machine_halt() / panic() / emergency_restart(), then cross-reference backward. One short walk lands on sub_FFFFFFFF81710483(). The function is full of unnamed variables, but there seems to be a 270-iteration for loop carrying an & 0x1F (mod-32) XOR.
rsa_parse_pub_key(v36, n6291648_1, 270);
That line resolves the loop’s purpose — the XOR reconstructs the RSA public key the next call parses. Double-clicking the two source pointers gives both inputs: a 32-byte de-obfuscation seed at byte_FFFFFFFF8179A2C0 in .init.data (e.g. 5C 19 C6 E1 …), and a 270-byte obfuscated certificate at the adjacent byte_FFFFFFFF8179A1A0.
270 is not just a random byte size, it is the exact length of a DER-encoded PKCS#1 RSAPublicKey (the bare SEQUENCE { modulus, publicExponent }, not a SubjectPublicKeyInfo wrapper) for a 2048-bit modulus with a 3-byte exponent.
So the 270-byte blob is a raw DER public key, XOR-masked byte-for-byte, and it must begin 30 82 01 0A once unmasked. That gives a free correctness check before touching RSA at all. The & 0x1F loop is a repeating 32-byte XOR keyed by the seed.
uint8_t pubkey_der[270];
for (int i = 0; i < 270; i++)
pubkey_der[i] = obf_cert[i] ^ seed[i & 0x1F]; /* seed = 32 B @ 8179A2C0 */
rsa_parse_pub_key(&rsa_ctx, pubkey_der, 270);
So in a script we can do something like :
def deobfuscate_pubkey(obf_cert: bytes, seed: bytes) -> bytes:
der = bytes(obf_cert[i] ^ seed[i & 0x1F] for i in range(len(obf_cert)))
assert der[:4] == bytes.fromhex("3082010A"), "seed or offset is wrong"
return der # load with RSA.import_key(...) as a DER RSAPublicKey
The seed at …8179A2C0 is 32 bytes and lives in kernel data, exactly like the disk key, so the tempting wrong guess is that it is the key for the big blob. It isn’t; it only unmasks the public key in memory. The DER-header assertion above disproves the misread — a wrong 32-byte value won’t XOR the certificate into a valid SEQUENCE. The second trap is an off-by-one in the payload slice. After grabbing the 256-byte tail and running m = c^e mod n, you recover a structured 256-byte block. The C code reads the key at a fixed struct offset into that buffer (v11 + 223), anchored from the front. Reproducing that naïvely in Python breaks:
pow(...).to_bytes(256, "big") renders the recovered integer big-endian, and any recovered value whose top byte is zero — the PKCS#1 leading 00, or simply m < 2^2040 — shifts a front-anchored slice by one byte. The key lands one byte early and the output is uniform garbage. The fix is the general rule for PKCS#1 message recovery: anchor to the tail, not to a hardcoded front index. recovered[-32:] and the disassembly’s v11 + 223 (byte 224 of a 256-byte struct) then coincide.
With the slice corrected, the trailing 96 bytes resolve into three aligned fields:
recovered = pow(c_tail, e, n).to_bytes(256, "big")
# [00 01] [FF … FF] [00] || <---------- payload (96) ---------->
# SHA256(body) | field_B (32) | RC4 key (32)
# [160:192] [192:224] [224:256]
expected_sha256 = recovered[-96:-64] # 160:192
rc4_key = recovered[-32:] # 224:256
The middle 32 bytes ([192:224]) are recovered but unused by the decryptor. Given the layout, the candidates are a second digest (SHA256 of the decrypted body, as a post-decrypt check), a per-image nonce, or a build/model tag binding the key to one firmware. Diffing that field across two images settles it: constant means a tag or reserved field, per-image means a digest or nonce. The loop itself is self-verifying, you only trust the extracted key if SHA256(encrypted_body) equals recovered[160:192], which confirms the offsets. The key schedule uses key[i & 0x1F], which is ordinary RC4 key scheduling with the key length hardcoded to 32 (% keylen collapses to & 0x1F). In the x86_64 disassembly of sub_FFFFFFFF81710334() it shows up as an and eax, 0x1F inside a 256-iteration loop that also swaps S-box entries. That pairing is the signature for “RC4-family, 32-byte key,” and it’s the first thing to grep for.
The custom part seems to be in the PRGA output, and the classic RC4 byte is still in there. After the swap, S[i] = old_sj and S[j] = old_si, so the standard RC4 output S[(S[i] + S[j]) & 0xFF] is really just S[(old_sj + old_si) & 0xFF] — which is comp_z in the code. So Fortinet does the normal RC4 thing first, computes the plain RC4 byte, and then whitens it by running it through two extra table lookups:
The disassembly tells for the added layer are shr …, 3 / shl …, 5 (the >>3 / <<5 on the counters) and a literal xor …, 0xAA, none of which appear in stock RC4.
; sub_FFFFFFFF81710334
shr eax, 3 ; j >> 3 ┐ idx1 = (j>>3) ^ (i<<5)
shl edx, 5 ; i << 5 ┘
xor eax, edx
movzx ecx, byte [S+eax] ; S[idx1]
; … idx2 = (i>>3) ^ (j<<5) ; S[idx2] the same way …
add ecx, esi ; S[idx1] + S[idx2]
xor ecx, 0AAh ; ← whitening tell, absent from stock RC4
movzx ecx, byte [S+ecx] ; comp_y = S[(S[idx1] + S[idx2]) ^ 0xAA]
The motive is the same as dropping "expand 32-byte k": keep a proven cipher core, add whitening so the emitted keystream matches no reference RC4, and defeat constant- and signature-matching without designing a cipher from scratch. Once the key boundary is aligned and the PRGA operator precedence is reproduced, the SHA256 computed over the firmware body matches the digest recovered from the RSA block.
import hashlib
from Crypto.PublicKey import RSA
CERT_FILE = "forti_pubkey.der" # de-obfuscated DER RSAPublicKey (270 B)
ROOTFS_ENCRYPTED_FILE = "rootfs.gz" # original encrypted firmware
ROOTFS_DECRYPTED_FILE = "rootfs_decrypted.gz"
# 256-byte RSA-wrapped block from the tail of rootfs.gz
rootfs_tail_hex = (
"31C6C1E3E33F19F477E38CB15845DFC919C60D0CE33975E34A63815BB8BCB47C"
"4FDCA8AC8F30C64053C00A045FA876F2B3C2D0AA2D82B64842E2C6B43C7A4A6C"
"169577A860171DE17019EE1CE7C89E6F44833B959104E4E0B61525C214EA68AE"
"D4557BC902597E15B8C5A38C18217FB0E9F77EE1F3E50CEE4D57166DC6FA358B"
"5E6FAF48B11D26399209D792018729E1AF9040C14026250EF18DA11EBE14A8F9"
"B9D2F002B2772DA82796E755DCFD238DD563611C323175ED4E238B23EB76D6FA"
"EE0A6FB93EA9CC5149A62AB2685B44ECFB43BE3185D3976B0CE58E6BC9A2B01F"
"0D8D2BCA4CD928C9F874E4F6D5A81460257C31C6CE2CC09B49F772878E08BEA0"
)
def deobfuscate_pubkey(obf_cert: bytes, seed: bytes) -> bytes:
"""Rebuild the DER RSAPublicKey from the 270-B obfuscated cert (byte_..A1A0)
and the 32-B seed (byte_..A2C0). Must start with SEQUENCE 30 82 01 0A."""
der = bytes(obf_cert[i] ^ seed[i & 0x1F] for i in range(len(obf_cert)))
assert der[:4] == bytes.fromhex("3082010A"), "seed or offset is wrong"
return der
def forti_custom_rc4_decrypt(ciphertext: bytes, key: bytes) -> bytes:
# --- KSA: standard RC4 schedule, key length hardcoded to 32 (i & 0x1F) ---
S = list(range(256))
j = 0
for i in range(256):
j = (j + S[i] + key[i & 0x1F]) & 0xFF
S[i], S[j] = S[j], S[i]
# --- PRGA: classic RC4 byte (comp_z) whitened by two extra lookups ---
plaintext = bytearray(len(ciphertext))
i = j = 0
for idx, c in enumerate(ciphertext):
i = (i + 1) & 0xFF
old_si = S[i]
j = (j + old_si) & 0xFF
old_sj = S[j]
S[i], S[j] = old_sj, old_si
comp_x = S[(j + old_sj) & 0xFF]
idx1 = ((j >> 3) ^ (i << 5)) & 0xFF
idx2 = ((i >> 3) ^ (j << 5)) & 0xFF
sum_idx = (S[idx1] + S[idx2]) & 0xFF
comp_y = S[(sum_idx ^ 0xAA) & 0xFF]
comp_z = S[(old_sj + old_si) & 0xFF] # == standard RC4 output
keystream = comp_x ^ ((comp_y + comp_z) & 0xFF)
plaintext[idx] = c ^ keystream
return bytes(plaintext)
# --- Load the reconstructed public key -------------------------------------
print("Parsing the RSA public key...")
with open(CERT_FILE, "rb") as f:
pub = RSA.import_key(f.read())
n, e = pub.n, pub.e
print(f" N (first 16 hex): {hex(n)[:18]}...")
# --- RSA message recovery over the tail block ------------------------------
print("Recovering the wrapped payload (m = c^e mod n)...")
c_tail = int.from_bytes(bytes.fromhex(rootfs_tail_hex), "big")
recovered = pow(c_tail, e, n).to_bytes(256, "big")
# Trailing 96 bytes = [ SHA256(body) | field_B | RC4 key ], tail-anchored
expected_sha256 = recovered[-96:-64]
rc4_key = recovered[-32:]
print(f" expected SHA256 : {expected_sha256.hex()}")
print(f" core RC4 key : {rc4_key.hex()}")
# --- Self-verifying integrity check ----------------------------------------
print("Reading and verifying firmware body...")
with open(ROOTFS_ENCRYPTED_FILE, "rb") as f:
full = f.read()
encrypted_body = full[:-256]
actual_sha256 = hashlib.sha256(encrypted_body).digest()
print(f" actual SHA256 : {actual_sha256.hex()}")
if actual_sha256 == expected_sha256:
print(" SHA256 match — offsets and key are correct.")
else:
print(" SHA256 MISMATCH — check tail slice / payload offsets.")
# --- Decrypt ---------------------------------------------------------------
print("Decrypting the filesystem with the RC4 variant...")
decrypted = forti_custom_rc4_decrypt(encrypted_body, rc4_key)
with open(ROOTFS_DECRYPTED_FILE, "wb") as f:
f.write(decrypted)
print(f"Done — written to {ROOTFS_DECRYPTED_FILE}")
Why does Fortinet do This?
In the posts by Bishop Fox and GreyNoise, both were unable to figure out why Fortinet designed their firmware encryption scheme like this. The truth is that a FortiGate boots unattended with no password or network, so it has to carry everything needed to decrypt its own rootfs.gz, which means you can never get real confidentiality of the filesystem against someone with the image, because the decryption path ships inside the image. In the 7.0.x firmware version, the stock ChaCha20 with the key in the clear was pure obfuscation meant only to keep binwalk and casual readers out. Which is probably why the key was simply hardcoded (it has to live on the device somewhere, and the kernel is the easy spot).
The 8.0.x seems to try to wrap the key instead of leaving it as a plaintext (so only Fortinet’s private key could have produced it), the public key is XOR-obfuscated so the anchor is harder to find, and ChaCha20 is swapped for a whitened RC4 so there’s no magic constant or reference keystream to match. Crucially it’s RSA signing with message recovery, not encryption because the goal was to prove the image came from Fortinet. In this case i think they did well enough, but this demonstrates that firmware compromise of perimeter devices such as network appliances are becoming more common and defenders need to actually start preparing not only to defend their networks from these attacks but learn how to properly respond and investigate these types of incidents.
So what did we learn from all this? Probably that all of this is just very hard. Cryptographically function cracking, firmware decryption, linux kernel analysis, if you’re able to do all of that you won’t settle working in incident response. You would be working in vulnerability research to find zero days and easily make ten times more. The unfortunate truth is that so far there hasn’t really been an effective way to detect, analyze, and tackle these types of attacks.