Working draft
tar-minefield F17 pax-global-comment › Tar file extraction test suite
This project provides an extensive test suite documentation for tar file extraction, with the goal of ensuring that implementations extract tar archives the same way.
There are many implementations of tar, from command line tools to libraries for programming languages. Because the tar file format leaves room for interpretation, different implementations can extract certain crafted tar files in different ways.
This is a security concern for file exchange: for example, when a security scanner and the extractor use different tar implementations, the scanner could inspect other files than the ones that end up on disk.
By assigning identifiers to these ambiguities, and providing test cases and usage data, this project aims for tar implementations to converge on strict parsing and identical behavior.
This project proposes a default policy accepts 99.77% .
Comparison of different tar implementations¶
This table shows how different tar implementations behave on each test case. A green ✓ is the
strict behavior that the default policy asks for; click a row for details
about the case.
What implementations make of every case of the test suite GNU tar bsdtar BusyBox tar 7-Zip Python tarfile, filter=data, as in pip Python tarfile, filter=tar Python tarfile, filter=fully_trusted node-tar node-tar, strict tar-fs Java Commons Compress .NET System.Formats.Tar PHP PharData, as in Composer Go containerd archive.Apply Rust tar Rust astral-tokio-tar Rust tar-codec, as in uv RubyGems R untar, internal Perl Archive::Tar Haskell tar Julia Tar.jl live-bootstrap untar live-bootstrap untar, --non-strict all 78 cases GNU tar: 26 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 26 GNU tar: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 GNU tar: 42 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 42 bsdtar: 28 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 28 bsdtar: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 bsdtar: 40 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 40 BusyBox tar: 15 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 15 BusyBox tar: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 BusyBox tar: 53 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 53 BusyBox tar: 1 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it 7-Zip: 19 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 19 7-Zip: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 7-Zip: 49 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 49 7-Zip: 1 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it Python tarfile, filter=data, as in pip: 25 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 25 Python tarfile, filter=data, as in pip: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 Python tarfile, filter=data, as in pip: 1 of 78, rejects a tolerated case, which its verdict allows too Python tarfile, filter=data, as in pip: 43 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 43 Python tarfile, filter=tar: 24 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 24 Python tarfile, filter=tar: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 Python tarfile, filter=tar: 44 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 44 Python tarfile, filter=fully_trusted: 22 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 22 Python tarfile, filter=fully_trusted: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 Python tarfile, filter=fully_trusted: 43 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 43 Python tarfile, filter=fully_trusted: 3 of 78, writes outside the destination node-tar: 14 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 14 node-tar: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 node-tar: 54 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 54 node-tar: 1 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it node-tar, strict: 25 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 25 node-tar, strict: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 node-tar, strict: 1 of 78, rejects a tolerated case, which its verdict allows too node-tar, strict: 43 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 43 tar-fs: 26 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 26 tar-fs: 8 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 8 tar-fs: 2 of 78, rejects a tolerated case, which its verdict allows too tar-fs: 42 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 42 Java Commons Compress: 23 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 23 Java Commons Compress: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 Java Commons Compress: 45 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 45 Java Commons Compress: 1 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it .NET System.Formats.Tar: 45 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 45 .NET System.Formats.Tar: 8 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 8 .NET System.Formats.Tar: 2 of 78, rejects a tolerated case, which its verdict allows too .NET System.Formats.Tar: 23 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 23 PHP PharData, as in Composer: 18 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 18 PHP PharData, as in Composer: 7 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 7 PHP PharData, as in Composer: 1 of 78, rejects a tolerated case, which its verdict allows too PHP PharData, as in Composer: 49 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 49 PHP PharData, as in Composer: 3 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it Go containerd archive.Apply: 26 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 26 Go containerd archive.Apply: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 Go containerd archive.Apply: 42 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 42 Rust tar: 22 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 22 Rust tar: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 Rust tar: 46 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 46 Rust astral-tokio-tar: 31 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 31 Rust astral-tokio-tar: 9 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 9 Rust astral-tokio-tar: 1 of 78, rejects a tolerated case, which its verdict allows too Rust astral-tokio-tar: 37 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 37 Rust tar-codec, as in uv: 55 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 55 Rust tar-codec, as in uv: 5 of 78, accepts a tolerated case and extracts its tree, which its verdict allows Rust tar-codec, as in uv: 5 of 78, rejects a tolerated case, which its verdict allows too Rust tar-codec, as in uv: 13 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 13 RubyGems: 19 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 19 RubyGems: 7 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 7 RubyGems: 2 of 78, rejects a tolerated case, which its verdict allows too RubyGems: 47 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 47 RubyGems: 3 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it R untar, internal: 17 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 17 R untar, internal: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 R untar, internal: 51 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 51 Perl Archive::Tar: 19 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 19 Perl Archive::Tar: 8 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 8 Perl Archive::Tar: 1 of 78, rejects a tolerated case, which its verdict allows too Perl Archive::Tar: 48 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 48 Perl Archive::Tar: 2 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it Haskell tar: 32 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 32 Haskell tar: 6 of 78, accepts a tolerated case and extracts its tree, which its verdict allows Haskell tar: 3 of 78, rejects a tolerated case, which its verdict allows too Haskell tar: 36 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 36 Haskell tar: 1 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it Julia Tar.jl: 38 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 38 Julia Tar.jl: 10 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 10 Julia Tar.jl: 30 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 30 live-bootstrap untar: 19 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 19 live-bootstrap untar: 8 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 8 live-bootstrap untar: 40 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 40 live-bootstrap untar: 8 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it 8 live-bootstrap untar: 3 of 78, writes outside the destination live-bootstrap untar, --non-strict: 10 of 78, as the verdict says: rejects a case to reject, extracts the tree of a case to accept 10 live-bootstrap untar, --non-strict: 8 of 78, accepts a tolerated case and extracts its tree, which its verdict allows 8 live-bootstrap untar, --non-strict: 49 of 78, accepts a case to reject, which a strict parser rejects, or crashes on it 49 live-bootstrap untar, --non-strict: 8 of 78, rejects a case to accept, or extracts another tree than that of the case, or crashes on it 8 live-bootstrap untar, --non-strict: 3 of 78, writes outside the destination to accept F17 pax-global-comment ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ F31 pax-xattr-not-utf8 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ N01 gnu-long-name ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗ ✗ N02 name-fills-field ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ N03 pax-long-name ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✗ N04 ustar-prefix ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ S01 tree-dir-after-contents ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ S02 tree-duplicate-dir ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ tolerated F01 number-leading-space + + + + + + + + + + + + + + + + − + + + + + + + F02 end-data-after + + + + + + + + + − + + + + + + + + + + − + + + F05 magic-version-nuls + + + + + + + + + + + − + + + − − + + + − + + + F14 pax-size-disagrees + + ✗ + + + + + + + + + ✗ + + + + ✗ + ✗ ✗ + ✗ ✗ F18 pax-global-comment-after-members + + + + + + + + + + + + + + + + + + + + + + ✗ ✗ F32 checksum-leading-space + + + + + + + + + + + + + + + + − + + + + + + + N05 name-dot + + + + + + + + + + + + − + + + + − + + + + + + N06 name-leading-dot-slash + + + + + + + + + + + + + + + + + + + + + + + + N07 symlink-absolute + + + ✗ − + + ✗ − − ✗ − ✗ + + + − − + − − + + + N15 name-control-character + + + + + + + + + + + + + + + + − + + + + + + + to reject: framing F03 end-lone-zero-block ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ✓ ! ! ! F04 end-missing ! ! ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ✓ ! ✓ ✓ F06 checksum-signed ! ! ! ! ! ! ! ✓ ✓ ✓ ! ! ✓ ! ✓ ✓ ✓ ! ! ! ✓ ✓ ! ! F07 end-archive-after ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! F08 end-truncated-in-member ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F09 end-zero-block-then-member ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ✓ ! ! ! ✓ ! ! ! F10 extended-pax-size-before-gnu-long ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ! ✓ ✓ F11 magic-odd-version ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ✓ ✓ ! ! F12 number-size-gap ! ✓ ! ! ✓ ✓ ✓ ! ! ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ! ! ! ! ! F13 pax-newline-in-path ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ! ! ! F15 pax-size-on-dir ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ✓ ! ! F16 type-unknown ! ! ✓ ! ✓ ! ! ! ! ✓ ! ✓ ! ✓ ! ! ✓ ! ✓ ! ! ✓ ! ! F19 pax-global-between-extended-and-member ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ! ! ! F20 pax-global-size ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! F21 pax-size-empty ✓ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ✓ ! ! ! ! ✓ ! ! F22 pax-size-negative ✓ ✓ ! ! ! ! ! ! ! ✓ ✓ ✓ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ! ! F23 pax-size-plus ✓ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ✓ ! ! ! ! ✓ ! ! F24 pax-size-leading-space ✓ ! ! ! ! ! ! ! ! ! ✓ ! ! ✓ ! ✓ ✓ ! ! ! ! ! ! ! F25 pax-size-not-ascii-digits ✓ ! ! ! ! ! ! ! ! ✓ ! ✓ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ! ! F26 pax-size-leading-zero ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! F27 pax-size-too-large ✓ ✓ ! ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ! ! F28 pax-newline-in-comment ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! F29 number-size-base256-negative ✓ ✓ ! ✓ ✓ ✓ ✓ ! ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ! ✓ ✓ ✓ F30 number-size-base256-too-large ✓ ✓ ! ✓ ✓ ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ! ! ✓ ✓ ✓ ! ! ✓ ✓ ✓ F33 size-on-dir ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ✓ ! ! F34 size-on-hardlink ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ! ✓ ✓ ! F35 pax-size-on-hardlink ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ! ✓ ✓ ! F36 pax-record-length-short ✓ ✓ ! ! ✓ ✓ ✓ ! ! ! ✓ ✓ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ! ! to reject: names N08 extended-pax-path-and-gnu-long-name ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ✓ ! ! ! ✓ ! ✓ ✓ N09 extended-two-pax-headers ! ✓ ! ! ! ! ! ! ! ! ! ✓ ! ! ✓ ✓ ✓ ! ! ! ✓ ! ! ! N10 format-star ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! N11 gnu-prefix-field ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! N12 name-absolute ! ! ! ! ! ! ↑ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ✓ ! ✓ ✓ ✓ ↑ ↑ N13 name-backslash ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! N14 name-bytes-after-nul ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! N16 name-dotdot ✓ ✓ ! ! ✓ ✓ ↑ ! ✓ ! ✓ ✓ ! ! ! ! ✓ ✓ ✓ ✓ ✓ ✓ ↑ ↑ N17 name-empty ✓ ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ! ! ✓ ✓ ✓ ✓ ✓ ✓ ! ! N18 name-not-utf8 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ✓ ✓ N19 pax-nul-in-path ! ! ! ! ✓ ✓ ✓ ! ! ✓ ✓ ! ! ✓ ✓ ✓ ✓ ! ! ! ✓ ✓ ! ! N20 pax-repeated-key ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ! ! ! N21 pax-global-path ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! N22 pax-path-empty ! ! ✓ ! ✓ ✓ ✓ ! ✓ ! ! ✓ ! ! ! ✓ ✓ ! ✓ ! ✓ ✓ ! ! N23 extended-gnu-long-then-pax ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ✓ ✓ N24 pax-hdrcharset-binary ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! N25 symlink-empty-target ✓ ✓ ! ! ! ! ! ✓ ✓ ! ! ✓ ! ✓ ✓ ✓ ✓ ✓ ! ✓ ✓ ✓ ✓ ! N26 pax-global-path-then-empty ! ✓ ! ! ✓ ✓ ✓ ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ✓ ! ! to reject: kinds K01 kind-trailing-slash-file ! ! ! ! ! ! ! ! ! ✓ ! ✓ ! ! ! ! ✓ ! ! ! ✓ ✓ ! ! K02 linkname-on-file ! ! ! ! ! ! ! ✓ ✓ ! ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! K03 symlink-trailing-slash ! ! ! ! ! ! ! ! ! ✓ ! ✓ ! ! ! ! ✓ ! ! ✓ ✓ ✓ ✓ ! to reject: order and state S03 hardlink-missing-target ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ! ! ✓ ✓ ✓ ✓ ! S04 hardlink-to-itself ! ✓ ! ✓ ! ! ! ! ✓ ✓ ! ✓ ! ✓ ✓ ✓ ✓ ! ! ! ! ! ✓ ! S05 hardlink-to-later-member ✓ ✓ ! ! ✓ ✓ ✓ ! ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ! ! ! ! ✓ ✓ ! S06 hardlink-to-symlink ! ! ! ✓ ! ! ! ! ✓ ! ! ! ! ! ! ! ✓ ! ! ! ✓ ✓ ✓ ! S07 tree-duplicate-file-differs ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! S08 tree-duplicate-file-dir ! ! ! ✓ ✓ ✓ ✓ ! ! ✓ ✓ ✓ ! ! ✓ ✓ ! ✓ ! ✓ ✓ ! ! ! S09 tree-duplicate-file-identical ! ! ! ! ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ! ! ! ! S10 tree-symlink-then-file ! ! ! ✓ ! ! ! ! ! ! ! ✓ ! ! ! ! ! ! ! ! ✓ ! ! ! S11 tree-through-symlink ✓ ✓ ✓ ✓ ✓ ✓ ↑ ! ✓ ✓ ✓ ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ↑ ↑ S12 tree-under-file ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ! ! ! S13 hardlink-empty-target ✓ ✓ ! ! ! ! ! ! ✓ ! ! ✓ ✓ ✓ ✓ ✓ ✓ ! ! ✓ ✓ ! ✓ ! ✓ as the verdict says: rejects a case to reject, extracts the tree of a case to accept + accepts a tolerated case and extracts its tree, which its verdict allows − rejects a tolerated case, which its verdict allows too ! accepts a case to reject, which a strict parser rejects, or crashes on it ✗ rejects a case to accept, or extracts another tree than that of the case, or crashes on it ↑ writes outside the destination Each case has a permanent identifier, such as F15, whose letter is its category:
F framing: which members an archive has.N names: what path each member has.K kinds: the kind of each member: file, directory or link.S order and state: how members combine into a directory tree.
A case's page shows its archive block by block, what every implementation makes of it, and how
common it is in release tarballs. An implementation's page, linked from its name above the matrix,
lists the cases where it does not do as the policy says.
Policy¶
This section defines a working draft of the
default policy 1 data from the 82,586 distinct release
tarballs of Spack, nixpkgs, Guix and live-bootstrap, and samples of PyPI, crates.io and RubyGems.
The prevalence column is the share of these tarballs that have a case: a first-order
approximation of how common it is in general.
1 A policy can be stricter and also reject the tolerated
cases, at the cost of rejecting more published tarballs: some tolerated cases are common, such as
numbers padded with spaces (F01), and others rare. That suits special-purpose tools,
such as an extractor for a single ecosystem, where it can be checked against every tarball ever
published there.
To accept¶
identifier name what it is prevalence F17pax-global-comment The global pax header with only a comment, as git archive writes it. not measured F31pax-xattr-not-utf8 An xattr in a pax header whose value is not UTF-8, as libarchive and star write them. 0.04% N01gnu-long-name A name over 100 bytes, in a GNU long name. not measured N02name-fills-field A name of exactly 100 bytes, which fills the name field without a NUL. 9.53% N03pax-long-name A name over 100 bytes, in a pax path. not measured N04ustar-prefix A long name split between the ustar prefix and name fields. 7.65% S01tree-dir-after-contents A directory after its contents. 1.22% S02tree-duplicate-dir The same directory twice, as tar cf x.tar d d/sub writes. 0.19%
Tolerated¶
identifier name what it is prevalence F01number-leading-space Numeric fields padded with spaces rather than zeros, as old tars wrote them. 0.99% F02end-data-after Data after the end-of-archive marker that is not a header, which is ignored. 0.03% F05magic-version-nuls The POSIX magic with NULs for its version, ustar\0\0\0, as some Java archivers write. 0.07% F14pax-size-disagrees A pax size that disagrees with the header's, so that a reader of the header's misses a member. none F18pax-global-comment-after-members A global pax header with only a comment, after the first member. 0.10% F32checksum-leading-space A checksum padded with spaces rather than zeros, as Perl's Archive::Tar writes it. 0.99% N05name-dot A member named ./, the root of the tree, as tar cf x.tar . writes first. 0.08% N06name-leading-dot-slash The name ./a, which is the path a. 0.30% N07symlink-absolute A symlink to an absolute path, created as it is. 0.11% N15name-control-character A control character in a name, here a newline, created as it is. 0.01%
To reject: framing¶
identifier name what it is prevalence F03end-lone-zero-block An end-of-archive marker of one zero block, not two. <0.01% F04end-missing No end-of-archive marker: the archive ends after a complete member. 0.03% F06checksum-signed A checksum that only matches as the signed sum of the header. none F07end-archive-after Another archive after the end-of-archive marker. <0.01% F08end-truncated-in-member An archive that ends inside the data of a member. not measured F09end-zero-block-then-member A zero block, then another member: the end of the archive, or not? <0.01% F10extended-pax-size-before-gnu-long A pax size followed by a GNU long name: which of them does the size apply to? none F11magic-odd-version The POSIX magic with a version that is neither 00 nor NULs. none F12number-size-gap A size with a space among its digits, which reads as two sizes. none F13pax-newline-in-path A newline in a pax path, which split on lines is another record. none F15pax-size-on-dir A pax size on a directory, which takes the next header for its data. 0.01% F16type-unknown An unknown typeflag with a size: data, or no data? none F19pax-global-between-extended-and-member A global pax header between an extended header and its member. none F20pax-global-size A global pax header with a size, which frames every member after it, or none. none F21pax-size-empty An empty pax size, which in POSIX removes the size rather than giving one. none F22pax-size-negative A negative pax size, which a member cannot have. none F23pax-size-plus A pax size with a plus sign, +512, which conversions of strings to numbers accept. none F24pax-size-leading-space A pax size with a leading space, 512, which conversions of strings to numbers skip. none F25pax-size-not-ascii-digits A pax size in Arabic-Indic digits, ٥١٢, which some conversions of strings read as 512. none F26pax-size-leading-zero A pax size with a leading zero, 0777: 777 in decimal, as POSIX says, or 511 in octal. none F27pax-size-too-large A pax size of 2^64 + 512, which wraps around to 512 in 64 bits. none F28pax-newline-in-comment A newline in a pax comment, which split on lines leaves a size record. none F29number-size-base256-negative A negative base-256 size in the header, -512, which a member cannot have. none F30number-size-base256-too-large A base-256 size of 2^64 + 512 in the header, which wraps around to 512 in 64 bits. none F33size-on-dir A size in the header of a directory, which takes the next header for its data. none F34size-on-hardlink A size in the header of a hard link, in a pax archive, which takes the next header for its data. none F35pax-size-on-hardlink A pax size on a hard link, which takes the next header for its data. none F36pax-record-length-short A pax record whose length is one byte short, so that it ends just before its newline. none
To reject: names¶
identifier name what it is prevalence N08extended-pax-path-and-gnu-long-name A pax path and a GNU long name for one member, and a third name in its header. none N09extended-two-pax-headers Two pax extended headers for one member, with different paths. none N10format-star A ustar prefix whose end reads as star's times, with star's tar\0 at offset 508. <0.01% N11gnu-prefix-field The prefix field of a header with GNU magic, which GNU tar uses for times in incremental archives. none N12name-absolute An absolute name, which points outside the directory the archive is extracted to. <0.01% N13name-backslash A backslash in a name: a character on Unix, a separator on Windows. <0.01% N14name-bytes-after-nul Bytes after the NUL that ends a name. none N16name-dotdot A name with .., which climbs out of the directory the archive is extracted to. <0.01% N17name-empty A member with an empty name. none N18name-not-utf8 A name that is not UTF-8. <0.01% N19pax-nul-in-path A NUL inside a pax path. none N20pax-repeated-key One pax key twice in one extended header. <0.01% N21pax-global-path A global pax header with a path. none N22pax-path-empty An empty pax path, which in POSIX removes the path rather than giving one. none N23extended-gnu-long-then-pax A GNU long name, then a pax header, then the member: which of them does the long name apply to? none N24pax-hdrcharset-binary hdrcharset=BINARY, which declares a pax path that is not UTF-8 as bytes.none N25symlink-empty-target A symlink with an empty target. none N26pax-global-path-then-empty A global path, then a global header that removes it with an empty value. none
To reject: kinds¶
identifier name what it is prevalence K01kind-trailing-slash-file A regular file whose name ends in a slash: a file, or a directory? <0.01% K02linkname-on-file A regular file with a link target. <0.01% K03symlink-trailing-slash A symlink whose name ends in a slash. none
To reject: order and state¶
identifier name what it is prevalence S03hardlink-missing-target A hardlink to a member that is not in the archive. none S04hardlink-to-itself A hardlink to itself. 0.03% S05hardlink-to-later-member A hardlink to a member that comes later. none S06hardlink-to-symlink A hardlink to a symlink: a link to the symlink, or to its target? <0.01% S07tree-duplicate-file-differs The same file twice, with different contents: the first copy, or the last? not measured S08tree-duplicate-file-dir A file, then a directory of the same name. none S09tree-duplicate-file-identical The same file twice, with the same contents. 0.11% S10tree-symlink-then-file A symlink, then a file of the same name: replace the link, or write through it? none S11tree-through-symlink A member written through a symlink, which points out of the destination. none S12tree-under-file A member under a file. <0.01% S13hardlink-empty-target A hardlink with an empty target. none
Data to support the default policy¶
The default policy accepts 99.77% of the 82,586 distinct release tarballs of Spack, nixpkgs, Guix and live-bootstrap, and samples of PyPI, crates.io and RubyGems. See the list of the 192 it rejects , and why.
ecosystem tarballs rejected share Spack 37,553 70 0.19% nixpkgs 18,990 78 0.41% Guix 10,168 34 0.33% PyPI 10,000 19 0.19% crates.io 5,000 0 none RubyGems 5,000 6 0.12% live-bootstrap 171 0 none total (distinct) 82,586 192 0.23%
The tarballs by the median mtime of their members, a rough check that the data spans enough years, leaving out the 14,361 whose mtimes are fixed for reproducibility or missing:
Tarballs by the median mtime of their members, in bins of two years 0 5,000 10,000 15,000 1981–1989: 1 tarball 1990–1991: 2 tarballs 1990 1992–1993: 4 tarballs 1994–1995: 18 tarballs 1996–1997: 10 tarballs 1998–1999: 34 tarballs 2000–2001: 64 tarballs 2000 2002–2003: 148 tarballs 2004–2005: 283 tarballs 2006–2007: 286 tarballs 2008–2009: 662 tarballs 2010–2011: 956 tarballs 2010 2012–2013: 1,751 tarballs 2014–2015: 3,847 tarballs 2016–2017: 6,008 tarballs 2018–2019: 8,440 tarballs 2020–2021: 10,896 tarballs 2020 2022–2023: 12,704 tarballs 2024–2025: 14,107 tarballs 2026–2027: 7,966 tarballs Motivating examples¶
case the archive what implementations make of it N09 two pax headers for one file, with path=first and path=second GNU tar, BusyBox, 7-Zip, node-tar, tar-fs, Go, R and Julia extract second; Python and Java extract first; PHP, RubyGems, Perl and live-bootstrap extract header-name, the name in the ustar header; bsdtar, Rust, .NET and Haskell reject it S10 a symlink a to target, then a file a most replace the link with the file; Python's tarfile and live-bootstrap write through the link, to target; 7-Zip, .NET and Haskell reject it K01 a regular file named a/ GNU tar, bsdtar, BusyBox, 7-Zip, node-tar, Java, Perl and live-bootstrap create a directory a; Python, Go, Rust's tar and astral-tokio-tar, PHP's PharData, RubyGems and R an empty file a; tar-fs, .NET, Haskell, Julia and tar-codec reject it F12 a size field of 000000 1000, with a space GNU tar, BusyBox, 7-Zip, node-tar, tar-fs, Perl, Haskell, Julia and live-bootstrap read 0, and find a second file smuggled in the data; R reads 512, and extracts it as the data of a; bsdtar, Python, Go, Rust, Java, .NET, PHP and RubyGems reject it
Contributing¶
This project is looking for contributions from people who work on package managers, registries and tar implementations, in the form of discussions in issues or pull requests on GitHub or by email to harmen at stoppels dot ch, so that the working draft of the default policy can become a reference.
You can also contribute by testing other implementations, or by crafting a tarball that is particularly hard to parse. A case that shows a vulnerability that is not yet disclosed is reported privately first: see SECURITY.md .
Contributors: