Zstandard & LZ4: The Modern Compression Formats Replacing Gzip
For decades, gzip dominated compression for Linux packages, web servers, and data pipelines. Bzip2 offered better ratios but was painfully slow. Two algorithms changed everything: LZ4 (2011) and Zstandard (2016) — both achieving near-gzip or better compression ratios while being 3–10× faster.
LZ4 (2011): Speed Above All
Created by Yann Collet (the same person who later created Zstandard), LZ4 was designed with a single priority: decompression speed. On modern hardware it decompresses at roughly 4 GB/s — fast enough to saturate a RAM bus.
- Compression ratio: ~1.7:1 (below gzip — larger files)
- Compression speed: ~750 MB/s (6–8× faster than gzip)
- Decompression speed: ~4,000 MB/s
Where LZ4 is used today:
- Linux kernel (zram RAM compression, squashfs)
- LVM snapshots
- Docker layer caching
- Redis, Cassandra, and other in-memory databases
- Real-time data pipelines where latency matters more than file size
Zstandard / Zstd (2016): The New General-Purpose Standard
Created by Yann Collet at Facebook (now Meta) and open-sourced in 2016, Zstandard's design goal was to match or beat gzip's compression ratio while being much faster.
- Configurable levels 1–22: level 3 ≈ gzip level 6 in ratio, but 3× faster
- At level 19+: beats bzip2 in compression ratio
- Facebook uses it for billions of internal compressions per day
Zstd is now standard in:
- Fedora and Arch Linux packages (
.tar.zst) - Ubuntu apt packages (experimental)
- OpenSSH, Samba, Btrfs, FreeBSD base system
- Linux kernel (
CONFIG_KERNEL_ZSTD)
Compression Comparison Table
| Algorithm | Ratio | Compress speed | Decompress speed | Use case |
|---|---|---|---|---|
| LZ4 | ~1.7:1 | 750 MB/s | 4,000 MB/s | Real-time, RAM compression |
| Gzip (level 6) | ~2.8:1 | 120 MB/s | 400 MB/s | Web (Content-Encoding: gzip) |
| Zstd (level 3) | ~2.9:1 | 400 MB/s | 1,400 MB/s | Linux packages, general use |
| Bzip2 | ~3.1:1 | 20 MB/s | 90 MB/s | Legacy (being replaced) |
| XZ/LZMA | ~3.4:1 | 10 MB/s | 50 MB/s | Debian packages, maximum ratio |
Common File Extensions
| Extension | Meaning |
|---|---|
.zst |
Zstandard compressed file |
.lz4 |
LZ4 compressed file |
.tar.zst |
Tar archive compressed with Zstd |
.tar.lz4 |
Tar archive compressed with LZ4 |
Working with Zstd
# Compress
zstd file.txt # creates file.txt.zst
zstd -19 file.txt # maximum compression
zstd -T0 file.txt # multi-threaded (use all cores)
# Decompress
zstd -d file.txt.zst # or: unzstd file.txt.zst
# Compress directory (tar + zstd)
tar -I zstd -cf archive.tar.zst ./folder/
# Extract tar.zst
tar -I zstd -xf archive.tar.zst
Install zstd:
- Linux (Debian/Ubuntu):
sudo apt install zstd - Linux (Fedora/RHEL):
sudo dnf install zstd - macOS:
brew install zstd - Windows: download from the Zstd GitHub releases page
Working with LZ4
# Compress
lz4 file.txt # creates file.txt.lz4
# Decompress
lz4 -d file.txt.lz4
# Tar + LZ4
tar --use-compress-program=lz4 -cf archive.tar.lz4 ./folder/
Converting from Legacy Formats
If you have old archives in gzip, bzip2, or XZ format and want to recompress them with Zstd:
# Gzip to Zstd (recompress)
gzip -d -c file.gz | zstd -o file.zst
# Bzip2 to Zstd
bzip2 -d -c file.bz2 | zstd -o file.zst
# XZ to Zstd
xz -d -c file.xz | zstd -o file.zst
These commands decompress the original format via pipe and recompress to Zstd without creating an intermediate uncompressed file on disk.
Why Package Managers Switched
- Fedora 31 (2019): Moved RPM packages to
.tar.zst— package install speed improved ~15% due to faster decompression - Arch Linux (2020): Migrated
.tar.xzpackages to.tar.zst - Docker: Uses Zstd for faster image layer transfer (eStargz / OCI image spec)
- npm: Under consideration for registry tarball compression
When to Use Which Format
| Situation | Recommended format |
|---|---|
| Maximum decompression speed (database snapshots, RAM caches, real-time streaming) | LZ4 |
| General-purpose replacement for gzip/bzip2 | Zstd (level 3) |
| Maximum compression ratio, speed less important | Zstd (level 19+) or XZ |
| Web servers, HTTP Content-Encoding | Gzip (universal compatibility) |
| Legacy tools that only understand gzip | Gzip |
The key takeaway: Zstd is the new default for most use cases. LZ4 wins when sub-millisecond latency is critical. Gzip stays relevant only where compatibility is non-negotiable.
Related conversions
Frequent conversions across the catalogue: