Multi-interpretable tarballs

Oct 22nd, 2026

When a new version of a package is released, package managers typically record a cryptographic hash of the release tarball for security and reproducibility purposes:

$ sha256sum foo-1.2.3.tar.gz
6b6bb19ded6dd37dc...5c1c4e30aa64afee  foo-1.2.3.tar.gz

Future builds use it to verify the integrity of the sources, and provided a sufficiently equivalent build environment, you get a reproducible build, right?

This post is meant to point out a subtlety: you can speak of integrity of the tarball, but that does not translate to integrity of the sources. Why? Because tarballs can be multi-interpretable[1].

I took tricky archives from the test suites of a dozen tar implementations, plus some of my own after fuzzing, and found that various tools extract the same tarball differently, without producing errors or warnings.

To keep things short, this blog post compares just GNU tar 1.35 and Python's tarfile module (from CPython 3.12 to 3.14, to be precise).

TL;DR on security

If you cannot establish trust in release tarballs other than extracting and scanning their sources, keep in mind that source extraction is interpretation of untrusted data. Trusting the extracted files is not the same as trusting the tarball that you downloaded.

If you're reading this and the first thought that comes to mind is “but Nix / Guix solves this!”, the answer is “to some extent”. About a quarter of nixpkgs and a third of Guix, including popular GNU packages like glibc, gcc and openssl, still hash and verify just the tarball, and extract with a pinned extraction tool. If that tar implementation is switched, sure, it's reflected in the build's hash, and sure, the new build is reproducible—but the trusted tarball hash is unchanged, while the effective sources upon extraction may suddenly be different. The best guard is to record both the tarball hash and the tree hash post extraction.

Tarball crash course

I'll briefly explain the tar file format, with just enough information so you can build a strong intuition of all the things that can go wrong.

A tar file is a sequence of 512 byte blocks. Each member is a header block followed by its data, padded to a multiple of 512 bytes. The archive ends with two blocks of zeros:

A tar archive: the directory foo/ is only a header, the file foo/bar of 800 bytes a header and two data blocks, the second holding 288 bytes and 224 bytes of padding, the symlink foo/baz to bar only a header, and the archive ends with two blocks of zeros.

The header has fields of fixed width: a name (or path) of 100 bytes, a type (file, directory, symlink, hardlink), a link target, the size, a checksum of the header itself, and a magic value that says which flavor of header it is. In practice you'll see three layouts of the header block: the original V7 one without magic, POSIX ustar, and GNU's own. There are more fields, but none are relevant for this post. Values like size only matter for members that have data, like regular files; directories and links are just a header.

Numbers are encoded as ASCII text, in octal digits, ended by a NUL or a space. For example, a file size of 800 is stored as 00000001440. Because that only allows files smaller than 8 GiB, GNU tar added a binary, base-256 encoding for larger numbers too.

Very old tar implementations did not have a type for directories, so they distinguished directories from files by checking whether the name had a trailing slash: so foo/ is a dir, foo is a file.

Tar files are meant for streaming, so unlike zip files, there is no index of all entries. To find where headers and files start, you have to do bookkeeping based on these size fields and use the header checksum to see if you're doing OK.

Long paths

If you've followed along so far, you must have realized that 100 bytes for a path is not very generous. Fortunately, the file format was extended in 3 ways to deal with that:

Three ways to store a long name. A ustar header splits a/very/long/file over its prefix and name fields, in the same block. For a name of 700 bytes, GNU puts a header of type L with size 700 before the file, followed by the 700 byte name in two blocks, the second padded. Pax puts a header of type x with size 710 before the file, followed by the record path=... in two blocks. In both, the file's own header has a truncated name.

The ustar format of POSIX.1-1988 added a 155 byte prefix field in the second half of the header block itself; those bytes were unused or reserved for extensions. Paths are computed as prefix / name and in a perfect world it gives you a solid 256 bytes. In practice it's fewer, because you have to split on a path separator /. Obviously ustar is not much of an improvement, because there's still a hard limit on the path length.

GNU tar introduced long names in 1992 in its own tar format that derived from an early POSIX draft. It puts an extra member before the file: a header of type L, whose data is the long name, in as many blocks as necessary.

POSIX.1-2001 generalized this idea in the pax format: a header of type x, whose data consists of records that override fields of the next header. A single pax header can have any number of them, for example:

32 path=foo-1.2.3/data/huge.bin
19 size=8589934592
30 mtime=1700000000.123456789

Each record is a key=value pair, preceded by its own length in bytes, up to and including the last newline character. It eliminates all fixed size limitations, including symlinks to long paths.

Structurally, GNU long names and pax headers are ordinary members: a header block with type L or x, followed by data. You could think of them as a special file type whose contents are extra metadata for the next member.

Although pax is standardized, GNU tar still uses its own format when creating tar files. Since everybody creates tarballs with their preferred tools, you basically need to support all formats when implementing a tar file parser; following just POSIX is practically insufficient.

These new formats solve problems with plain tar, but also lead to redundancy and associated parsing issues: a header field is now specified once in a GNU/pax header and then again in the traditional header, and nothing prevents one from repeating say size=...\n in the pax data. What's more, why not put a GNU long name in front of a pax header in front of a traditional header? That's clearly out of scope of POSIX (it does not even recognize GNU long names) and solidly within the domain of ambiguity.

How to extract the same tarball differently

Armed with this knowledge and the acquired sceptical standpoint that it's a miracle that any software works at all, we can now consider a few multi-interpretable tarballs.

Example 1: empty size

I'm starting with the simplest example I found. Technically the “justification” for this one is a historical curiosity in GNU tar, but it's still based on a clear ambiguity in the encoding of numbers as ASCII text.

What if the size field starts with a NUL byte?

A header a, then a header and data of a member called hidden. The size field of a, magnified, is a NUL, the octal digits 0000002000, and a terminating NUL.

POSIX only says that numeric fields are octal numbers “terminated by one or more <space> or NUL characters”, not what to do with one that's terminated before its first digit. Now Python defensibly stops at the first NUL, like at the end of a C-string, and interprets the “empty” or zero-digit number as 0. So, in the example above, the file a is empty, and the next block is the header of a member called hidden. GNU tar on the other hand skips the leading NUL and reads octal 2000, so the file a has 1024 bytes of data, containing all of hidden in the figure.

GNU tar reads a with 1024 bytes of data, which hold the header and data of hidden. Python reads an empty a followed by the member hidden.

What you get when you extract the tarball depends on the implementation:

GNU tar     Python
.           .
└── a       ├── a
            └── hidden

Now why the hell does GNU tar skip the leading NUL? According to comments in its sources, it does this on purpose since 1999, to “accommodate buggy tar of unknown vintage, which outputs leading NUL if the previous field overflows”.

Example 2: whitespace sensitivity

When parsing pax headers there is some room for interpretation. For example the record 11  size=0 has an extra space before the key: is this an unknown key ' size', different from 'size'? Python takes it literally and ignores it, but GNU tar interprets it as a size field that overrides the next header's size field.

A pax header with the record ' size=0', a header a of size 1024, and the member hidden. GNU tar applies size=0 and also reads hidden; Python ignores the record and reads a with 1024 bytes.

There are variations on the theme with ' path=', leading to disagreement between Python and GNU tar on where to extract the file.

Example 3: repeated headers

As already alluded to, nothing prevents you from creating a tarball with two pax headers, or two GNU long names, for a single tar member. GNU tar uses the last name, while Python uses the first. Intuitively I would say GNU tar is “right”, but Python's interpretation makes sense too: the left-most pax header overwrites all headers on the right-hand side.

Two pax headers, with path=a and path=b, followed by a header c of size 4 and its contents. GNU tar uses the last path and extracts b; Python uses the first and extracts a.

I must say though, it's not hard to imagine a world in which this was simply a parsing error.

Example 4: old-style directories

Remember that old tar distinguished directories from files by a trailing slash, like foo/? When only the pax path ends in a slash, while the traditional header's name does not, GNU tar interprets it as a directory without data. Python on the other hand interprets it as a file with 1024 bytes of data:

A pax path dir/ for a header of type NUL with size 1024, then the member hidden. GNU tar makes a directory dir and reads hidden; Python makes a file dir of 1024 bytes.

Addendum: upon further inspection, GNU tar 1.35 is internally inconsistent in this example: tar -tv does not show the hidden file, but tar x does extract it. Upstream has since made tar t match tar x, but to add to the confusion, Ubuntu 26.04 patched its tar 1.35 for CVE-2026-5704 the other way around, such that tar x matches tar t: it creates the directory dir, and skips hidden. So that's a third interpretation.

Example 5: repeated files and symlinks

Even when GNU tar and Python parse the tar file exactly the same way, there remains sufficient room for interpretation on how to extract them, still leading to different files on disk. Take a file g, a symlink f->g, and then a regular file f:

A file g with contents "orig", a symlink f pointing to g, and a file f with contents "new".

If you list the contents with GNU tar and Python, it looks the same, but extract them and you get:

GNU tar           Python
.                 .
├── f  "new"      ├── f -> g
└── g  "orig"     └── g  "new"

GNU tar replaces the previously extracted f with a new file, whereas Python opens the path for writing and follows the symlink, so g gets overwritten. With a hardlink instead of a symlink, Python writes into the shared inode, and again g changes. The data extraction filter, which Python 3.14 made the default to harden extraction, doesn't help here: it only prevents writes outside the destination.

Example 6: magic

The last case does not exactly qualify as a multi-interpretable tarball. It's rather a bug in Python that, in Python's defense, is an easy mistake to make, related to nearly identical magic bytes.

I mentioned that ustar gets you longer file paths by putting an extra prefix field in the second half of the header block. How to interpret that second half depends on the 8 magic bytes at its start, at byte 257. To clarify at the risk of confusing you further: this is a different “dimension” than GNU long names versus pax headers; those are extra members, a custom “file type” for additional file metadata.

So, when parsing the second half of a header, you check its magic bytes, and only if they match ustar's do you continue to parse its prefix field. It so happens that GNU tar's own header layout, which it uses by default, has magic bytes marginally different from those of ustar, and puts other fields where ustar puts its prefix. In incremental archives those are timestamps; in ordinary GNU archives they are fortunately NUL bytes, so nothing goes wrong there.

POSIX ustar   u s t a r \0 0  0
GNU           u s t a r ␣  ␣  \0

Now, Python has an unfortunate bug: it does not properly check the magic bytes, and always interprets these timestamps as a ustar prefix:

Byte map of a tar header. The magic at byte 257 is ustar NUL 0 0 in ustar headers and ustar space space NUL in GNU headers, so the last three bytes differ. In ustar headers bytes 345 to 500 are the name prefix; in GNU incremental headers they start with the access and change times, which Python reads as a prefix anyway. GNU tar checks the magic and reads d/f.txt, Python ignores it and reads 15266226341/d/f.txt.

You can easily construct this type of tar file using just GNU tar:

$ tar --format=gnu --listed-incremental=snap -cf incr.tar d
$ tar -tf incr.tar
d/
d/f.txt
$ python3
>>> import tarfile
>>> tarfile.open("incr.tar").getnames()
['15266226341/d/', '15266226341/d/f.txt']

At the time of writing, this issue is known as gh-155629, with an associated fix.

What to make of all this

Since release tarballs are ubiquitous, it's unlikely they will go away any time soon. Alternative file formats exist, like nar, which are better suited for distribution than tar (for various other reasons beyond those mentioned in this post[2]), but they are rather niche.

The hope is that with more awareness of these issues, there could be a coordinated effort to make tar parsers strict by default: a tar file that is ambiguous should just be rejected. That's a non-trivial amount of work: in this post I focused on GNU tar and Python's tarfile library, but pretty much every programming language has its tar library and every platform its own command line utility, each of which would need fixing and at the very least would trigger discussions about breaking backward compatibility.

Regarding backward compatibility, you could maybe allow certain ancient “features” such as “foo/ is a dir” behind a flag, but I would argue there's a good case for dropping those altogether, even if very old archives would no longer extract.

If you'd like to discuss this, send me an email at firstname[at]lastname.ch.

[1] Multi-intarpretable? A missed opportunity in the title of this post.

[2] I cannot help myself, let me mention just one: when run as root, GNU tar by default changes ownership of every extracted file to the user that was recorded in the archive. This could make sense for backups, but for distribution across machines, it makes no sense at all, and is a major liability.