N31 pax-linkpath-not-utf8 · Tar file extraction test suite

N31 pax-linkpath-not-utf8

A pax linkpath that is not UTF-8.

Reject(draft) A strict parser must reject this archive.

Category

name

Default policy

rejects it

Files

`N31-pax-linkpath-not-utf8.tar`	test archive, 2,560 bytes sha256 `79c9ac02aa5e39dcbf5d2ac455a191bcb50eaeb85f203893fe3a39fe690d5280`
`N31-pax-linkpath-not-utf8.json`	expected result
`N31-pax-linkpath-not-utf8.txt`	annotated blocks
`N31-pax-linkpath-not-utf8.py`	generator script

The case¶

The archive has a symlink link, whose pax header has linkpath=caf\xe9, café in Latin-1, and whose header has the target from-header.

POSIX says that pax records are UTF-8, apart from the values of gname, linkpath, path and uname records under a hdrcharset record, such as hdrcharset=BINARY, which this archive does not have. A link name that is not UTF-8 can be seen as one that cannot be translated, for which a diagnostic is required, and by default the file is bypassed. This case is ambiguous because the target is the bytes as they are, caf\xe9; a decoding of them, such as café or caf followed by U+FFFD; the header's from-header, to a reader that drops a record it cannot decode; or there is no symlink at all, as the reader bypasses the member, the default in POSIX, or rejects the archive as invalid.

Decision point: what a pax value that is not UTF-8 means.

Seen in: none of the scanned tarballs.

The archive¶

Every block of 512 bytes, to scale, with its offset. Hover over a field for its bytes.

N31-pax-linkpath-not-utf8: must reject A pax `linkpath` that is not UTF-8. The archive has a symlink `link`, whose pax header has `linkpath=caf\xe9`, `café` in Latin-1, and whose header has the target `from-header`. POSIX says that pax records are UTF-8, apart from the values of `gname`, `linkpath`, `path` and `uname` records under a `hdrcharset` record, such as `hdrcharset=BINARY`, which this archive does not have. A link name that is not UTF-8 can be seen as one that cannot be translated, for which a diagnostic is required, and by default the file is bypassed. This case is ambiguous because the target is the bytes as they are, `caf\xe9`; a decoding of them, such as `café` or `caf` followed by U+FFFD; the header's `from-header`, to a reader that drops a record it cannot decode; or there is no symlink at all, as the reader bypasses the member, the default in POSIX, or rejects the archive as invalid. Decision point: what a pax value that is not UTF-8 means. Seen in: none of the scanned tarballs. See also: N32, the same target in the header, and N24, a pax `path` that is not UTF-8, declared as bytes with `hdrcharset=BINARY`. 0x000000 header: pax extended header, a target that is not UTF-8 name [ 0,100) "PaxHeader" + 91 NUL mode [100,108) "0000644" + NUL 644 (octal) uid [108,116) "0000000" + NUL 0 gid [116,124) "0000000" + NUL 0 size [124,136) "00000000021" + NUL 17 mtime [136,148) "00000000000" + NUL 0 chksum [148,156) "011072\0 " 4666: the unsigned sum of the header, this field as spaces typeflag [156,157) "x" pax extended header linkname [157,257) 100 NUL magic [257,263) "ustar" + NUL version [263,265) "00" uname [265,297) 32 NUL gname [297,329) 32 NUL devmajor [329,337) "0000000" + NUL 0 devminor [337,345) "0000000" + NUL 0 prefix [345,500) 155 NUL pad [500,512) 12 NUL 0x000200 data: pax records 17 bytes of pax records, then 495 NUL "17 linkpath=caf\xe9\n" 0x000400 header: symlink name [ 0,100) "link" + 96 NUL mode [100,108) "0000644" + NUL 644 (octal) uid [108,116) "0000000" + NUL 0 gid [116,124) "0000000" + NUL 0 size [124,136) "00000000000" + NUL 0 mtime [136,148) "00000000000" + NUL 0 chksum [148,156) "012167\0 " 5239: the unsigned sum of the header, this field as spaces typeflag [156,157) "2" symlink linkname [157,257) "from-header" + 89 NUL magic [257,263) "ustar" + NUL version [263,265) "00" uname [265,297) 32 NUL gname [297,329) 32 NUL devmajor [329,337) "0000000" + NUL 0 devminor [337,345) "0000000" + NUL 0 prefix [345,500) 155 NUL pad [500,512) 12 NUL 0x000600 2 zero blocks: the end-of-archive marker 0x000a00 end of file

00000000: 5061 7848 6561 6465 7200 0000 0000 0000 PaxHeader....... 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000060: 0000 0000 3030 3030 3634 3400 3030 3030 ....0000644.0000 00000070: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000 00000080: 3030 3030 3032 3100 3030 3030 3030 3030 0000021.00000000 00000090: 3030 3000 3031 3130 3732 0020 7800 0000 000.011072. x... 000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000100: 0075 7374 6172 0030 3000 0000 0000 0000 .ustar.00....... 00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000120: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ 00000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000200: 3137 206c 696e 6b70 6174 683d 6361 66e9 17 linkpath=caf. 00000210: 0a00 0000 0000 0000 0000 0000 0000 0000 ................ 00000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000250: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000270: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000280: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000290: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000002f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000300: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000310: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000320: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000330: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000340: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000350: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000360: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000370: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000380: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000390: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000400: 6c69 6e6b 0000 0000 0000 0000 0000 0000 link............ 00000410: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000420: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000430: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000440: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000460: 0000 0000 3030 3030 3634 3400 3030 3030 ....0000644.0000 00000470: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000 00000480: 3030 3030 3030 3000 3030 3030 3030 3030 0000000.00000000 00000490: 3030 3000 3031 3231 3637 0020 3266 726f 000.012167. 2fro 000004a0: 6d2d 6865 6164 6572 0000 0000 0000 0000 m-header........ 000004b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000004f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000500: 0075 7374 6172 0030 3000 0000 0000 0000 .ustar.00....... 00000510: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000520: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000530: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000540: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 00000550: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ 00000560: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000570: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000005f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000600: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000610: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000620: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000630: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000640: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000650: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000660: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000670: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000680: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000690: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000006f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000700: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000710: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000720: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000730: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000740: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000750: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000760: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000770: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000780: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000790: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000007f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000800: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000810: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000830: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000840: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000850: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000860: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000870: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000880: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000890: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000008f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000900: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000910: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000920: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000930: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000940: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000950: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000960: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000970: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000980: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000990: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000009f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................

What implementations make of it¶

In the outcome column, ✅ means that the implementation follows the default policy.

implementation	version	outcome	what it makes of it
GNU tar	tar (GNU tar) 1.35	⚠️ accepts	tree A: `link` → `caf\xe9`
bsdtar	bsdtar 3.8.11 - libarchive 3.8.11	✅ rejects	error: bsdtar: Linkpath can't be converted from UTF-8 to current locale bsdtar: Error exit delayed from previous errors
BusyBox tar	BusyBox v1.38.0, defconfig	⚠️ accepts	tree A: `link` → `caf\xe9`
7-Zip	7-Zip (z) 26.05 (x64)	⚠️ accepts	tree A: `link` → `caf\xe9`
Python tarfile, filter=data, as in pip	CPython 3.15.2	⚠️ accepts	tree A: `link` → `caf\xe9`
Python tarfile, filter=tar	CPython 3.15.2	⚠️ accepts	tree A: `link` → `caf\xe9`
Python tarfile, filter=fully_trusted	CPython 3.15.2	⚠️ accepts	tree A: `link` → `caf\xe9`
node-tar	tar 7.5.22	⚠️ accepts	tree B: `link` → `from-header`
node-tar, strict	tar 7.5.22	⚠️ accepts	tree B: `link` → `from-header`
tar-fs	tar-fs 3.1.3, on tar-stream 3.2.2	⚠️ accepts	tree C: `link` → `caf�`
Java Commons Compress	Commons Compress 1.28.0, Expander, on Java 27.0.1.0.2	⚠️ accepts	tree D: `link` (0 B, e3b0c442)
.NET System.Formats.Tar	System.Formats.Tar of .NET 10.0.14, ExtractToDirectory	⚠️ accepts	tree C: `link` → `caf�`
PHP PharData, as in Composer	PharData of PHP 8.5.12, as Composer's TarDownloader calls it	⚠️ accepts	tree D: `link` (0 B, e3b0c442)
Go containerd archive.Apply	containerd v2.4.2, archive.Apply, of go1.27.3	⚠️ accepts	tree A: `link` → `caf\xe9`
Rust tar	tar 0.4.46, unpack	⚠️ accepts	tree A: `link` → `caf\xe9`
Rust astral-tokio-tar	astral-tokio-tar 0.7.1, unpack	⚠️ accepts	tree A: `link` → `caf\xe9`
Rust tar-codec, as in uv	tar-codec 0.0.16, with the policies of uv 0.13	✅ rejects	error: at byte 0: pax records contain invalid UTF-8 text
RubyGems	RubyGems 4.0.20 of Ruby 4.0.9, extract_tar_gz	⚠️ accepts	tree E: nothing
R untar, internal	utils::untar of R 4.6.2, tar = "internal"	⚠️ accepts	tree B: `link` → `from-header`
Perl Archive::Tar	Archive::Tar 3.12 of Perl v5.44.0	⚠️ accepts	tree B: `link` → `from-header`
Haskell tar	tar 0.7.2.0, extract, of GHC 9.14	⚠️ accepts	tree F: `link` → `café`
Julia Tar.jl	Tar 1.10.0 of Julia 1.13.3	⚠️ accepts	tree A: `link` → `caf\xe9`
live-bootstrap untar	mescc-tools-extra a151c245e512, as live-bootstrap pins it	⚠️ accepts	tree G: `PaxHeader` (17 B, e581ebae), `link` → `from-header`
live-bootstrap untar, --non-strict	mescc-tools-extra a151c245e512, as live-bootstrap pins it	⚠️ accepts	tree G: `PaxHeader` (17 B, e581ebae), `link` → `from-header`

Prevalence in release tarballs¶

The table below shows how many release tarballs have the features of this test case.

ecosystem	tarballs	share
Spack	37,553	none
nixpkgs	18,990	none
Guix	10,168	none
PyPI	10,000	none
crates.io	5,000	none
RubyGems	5,000	none
live-bootstrap	171	none
total (distinct)	82,586	none

ecosystem

tarballs

with it

Spack

37,553

none

nixpkgs

18,990

none

Guix

10,168

none

PyPI

10,000

none

crates.io

5,000

none

RubyGems

5,000

none

live-bootstrap

171

none

total (distinct)

82,586

none

`N31` pax-linkpath-not-utf8

The case¶

The archive¶

What implementations make of it¶

Prevalence in release tarballs¶

See also¶