TAR/GZ Archive Format: The Unix Compression Standard Explained

TAR/GZ Archive Format: The Unix Compression Standard Explained

TAR (Tape ARchive) is the foundational archive format of Unix and Linux systems, with origins dating to 1979. Unlike ZIP or RAR which combine archiving and compression, TAR separates these concerns: TAR handles file bundling and metadata preservation, while external compression programs (gzip, bzip2, xz, zstd) handle compression. This Unix philosophy approach has made the TAR family of formats — .tar, .tar.gz (.tgz), .tar.bz2, .tar.xz, .tar.zst — the standard for software distribution, system backups, and Linux package management.

TAR File Structure

A TAR file is a sequential format: file headers and data blocks are written one after another with no central index. Each archived file consists of:

[512-byte POSIX header block]
  Filename:      100 bytes (null-terminated, or GNU extended for >100 chars)
  File mode:     8 bytes (octal: 0000755 = rwxr-xr-x)
  UID:           8 bytes (numeric user ID)
  GID:           8 bytes (numeric group ID)
  File size:     12 bytes (octal)
  Modification time: 12 bytes (Unix timestamp, octal)
  Checksum:      8 bytes
  Type flag:     1 byte (0=regular, 2=symlink, 5=directory, 3=char dev, 6=FIFO)
  Link name:     100 bytes (for symlinks)
  [POSIX extension: UStar indicator "ustar", version, uname, gname, device numbers]

[File data, padded to 512-byte boundary]
[Next header block...]
[Two 512-byte zero blocks = end-of-archive marker]

This sequential structure has important implications:

• No random access: to extract file #1000, you must read through all 999 preceding headers
• Streaming-friendly: TAR can be piped directly to compression and network tools without temporary files
• Append only: adding files to an existing TAR requires reading the entire file to find the end marker

TAR Header Formats

Three main header formats exist:

GNU tar and PAX are the formats produced by modern tar implementations on Linux. PAX (--format=posix) is the most portable for archiving very long filenames or very large files.

Compression Algorithms

TAR itself stores no compression — it is a pure archive format. Compression is applied by piping TAR output through a separate compressor:

# Create archives (all equivalent, different compression)
tar -czf archive.tar.gz directory/        # gzip
tar -cjf archive.tar.bz2 directory/       # bzip2
tar -cJf archive.tar.xz directory/        # xz
tar -c --zstd -f archive.tar.zst directory/  # zstd

# Extract any tar archive (modern tar auto-detects compression)
tar -xf archive.tar.gz
tar -xf archive.tar.bz2
tar -xf archive.tar.xz

# List contents without extracting
tar -tf archive.tar.gz

# Extract single file
tar -xf archive.tar.gz path/to/specific/file.txt

# Create with verbose output
tar -cvzf archive.tar.gz directory/

# Extract to specific directory
tar -xzf archive.tar.gz -C /destination/path/

Preserving Metadata

TAR's key advantage over ZIP for Unix/Linux use is metadata preservation:

# Preserve permissions, ownership, timestamps
tar -cpzf backup.tar.gz /etc/

# Restore with original ownership (requires root)
tar -xpzf backup.tar.gz -C /

# Verify archive integrity
tar -tzf archive.tar.gz > /dev/null && echo "OK"

Preserved metadata includes:

• File permissions (read/write/execute for owner/group/other)
• Owner and group (by numeric UID/GID in standard TAR, by name in PAX)
• Modification timestamps (with nanosecond precision in PAX)
• Symbolic links (as link entries, not copies of targets)
• Hard links
• Device files (character and block devices)
• Named pipes (FIFOs)
• Extended attributes (SELinux contexts, ACLs in PAX extended headers)

Incremental and Differential Backups

GNU tar supports incremental backups through snapshot files:

# Level 0 (full) backup
tar --listed-incremental=/backup/snapshot.snar \
    -czf /backup/full_$(date +%Y%m%d).tar.gz /home/

# Level 1 (incremental — only files changed since last backup)
tar --listed-incremental=/backup/snapshot.snar \
    -czf /backup/incr_$(date +%Y%m%d).tar.gz /home/

# Restore from incremental chain (full, then each incremental)
tar --listed-incremental=/dev/null -xzf full_20240101.tar.gz
tar --listed-incremental=/dev/null -xzf incr_20240102.tar.gz

Streaming TAR Over SSH and Pipes

TAR's sequential format makes it ideal for streaming:

# Archive and transfer over SSH in one command (no temp file on source)
tar -czf - /data/ | ssh user@remote "cat > /backup/data.tar.gz"

# Transfer directory tree via SSH without creating archive file
tar -cz /source/ | ssh user@remote "tar -xz -C /destination/"

# Archive from remote, extract locally
ssh user@remote "tar -cz /remote/data/" | tar -xz -C /local/data/

# Create archive and immediately verify integrity
tar -czf archive.tar.gz files/ && tar -tzf archive.tar.gz > /dev/null

# Archive entire disk to remote using netcat (faster than SSH for LAN)
# On receiving machine:
nc -l 9000 | tar -xz -C /destination/
# On sending machine:
tar -cz /data/ | nc receiving_host 9000

Compression Comparison for TAR

For a typical mixed-content directory (source code + binaries + data):

Recommendation: .tar.zst (Zstandard) offers the best balance for modern systems — compression ratio near bzip2, speed near gzip. .tar.xz is preferred for software distribution where archive size matters more than encoding time.

Linux Package Management and TAR

Most Linux package formats are TAR-based:

# Inspect a .deb package contents
dpkg -c package.deb
# or: ar x package.deb && tar -tzf data.tar.xz

# Inspect a .rpm package contents
rpm -qlp package.rpm

TAR vs. ZIP for Different Use Cases

Summary

The TAR family's dominance in Linux/Unix environments stems from its metadata fidelity, streaming capability, and composability with the Unix pipe model. By separating archiving from compression, TAR lets you choose the right compressor for your use case — gzip for universal compatibility, xz for maximum compression, zstd for modern speed. For Windows-primary environments, ZIP's native OS support makes it the practical choice. For any Linux workflow, system backup, or software distribution, TAR with an appropriate compressor remains the correct tool.

Related conversions

Archive format conversions used most often:

• RAR to ZIP
• 7Z to ZIP
• TAR to ZIP
• GZ to ZIP
• ZIP to RAR
• ISO to ZIP